v0.32.8 fix: multi-source bug class extermination — embed, extract, takes, patterns, integrity, migrate-engine (#860)

* fix: thread source_id through embed --stale to fix silent discard of non-default source embeddings

listStaleChunks correctly finds chunks across all sources, but
embedOneSlug called getChunks(slug) and upsertChunks(slug, merged)
without passing sourceId. Both default to source_id='default', so
for non-default sources (e.g. media-corpus):

1. getChunks returns empty (wrong source)
2. merged array has no existing chunks to merge into
3. upsertChunks writes nothing (or errors silently)
4. Embeddings generated by the API are silently discarded

Fix:
- Add source_id to StaleChunkRow type
- Add p.source_id to listStaleChunks SQL in both postgres + pglite engines
- Extract sourceId from stale row in embed command
- Pass { sourceId } to getChunks and upsertChunks
- Group stale chunks by composite key (source_id::slug) instead of bare slug
  to handle same-slug pages across multiple sources

Verified: 97 chunks embedded across 35 pages in first run after fix.
Previously 0 non-default-source chunks were embedded across 3 full runs.

* fix: comprehensive multi-source threading for embed, listPages, and migrate-engine

Multi-source brains (e.g. with a 'media-corpus' source alongside
'default') have a pervasive bug: operations that iterate pages across
all sources then call engine methods (getChunks, upsertChunks,
getChunksWithEmbeddings) without passing sourceId. These methods all
default to source_id='default', silently operating on the wrong page
(or no page at all) for non-default sources.

Changes:

1. Page type + rowToPage: add optional source_id field so downstream
   callers can read the source from page objects returned by listPages.

2. PageFilters: add sourceId filter so listPages can scope to a single
   source (used by embed --source and future extract --source).

3. listPages (postgres + pglite): wire the sourceId filter into SQL.

4. embed command — three paths fixed:
   a. embedPage (single-slug): accepts sourceId, threads to getPage +
      getChunks + upsertChunks.
   b. embedAll (--all): reads page.source_id from listPages results,
      threads to getChunks + upsertChunks per page.
   c. embedAllStale (--stale): reads source_id from StaleChunkRow,
      groups by composite key (source_id::slug) instead of bare slug,
      threads to getChunks + upsertChunks per key.

5. embed CLI: add --source <id> flag, threaded through all paths.

6. migrate-engine: thread page.source_id through
   getChunksWithEmbeddings + upsertChunks so engine migrations don't
   lose non-default-source chunks.

7. getChunksWithEmbeddings (postgres + pglite + BrainEngine interface):
   accept optional { sourceId } to scope the chunk lookup.

8. StaleChunkRow type: add source_id field.

9. listStaleChunks SQL (postgres + pglite): add p.source_id to SELECT.

Verified: embed --stale correctly embeds 97 chunks across 35 pages
(previously 0 non-default-source chunks across 3 full runs).
embed --source media-corpus --dry-run correctly scopes to that source.

* v0.32.4 fix: multi-source threading for embed, listPages, and migrate-engine

Bump VERSION + package.json + CHANGELOG for the comprehensive multi-source
fix. Embed now threads source_id through every page → chunk handoff so
non-default sources stop silently dropping out (~22k chunks recovered on
the brain that surfaced this).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: complete slugs→keys rename in embedAllStale

The composite-key rename in the prior commit missed 4 references in the
worker loop and trailing console.log, so the file failed typecheck
(`Cannot find name 'slugs'`). The author's "Verified compiling + running"
claim was false at the time of the PR.

Also drop the dead `const bySlug = byKey` alias — unused after rename.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: add check-source-id-projection.sh + fix getPage/putPage projections

Two SELECT projections fed `rowToPage` without including `source_id`:
- postgres-engine.ts:562 (getPage), :609 (putPage RETURNING)
- pglite-engine.ts:505 (getPage), :548 (putPage RETURNING)

After the type-tightening in the next commit makes `Page.source_id`
required, those projections would silently produce `Page` rows with
source_id=undefined while TypeScript claims `: string`. Codex's plan
review (F2) caught this; this commit closes it.

The new `scripts/check-source-id-projection.sh` greps for the rowToPage
feeder shape (`SELECT id, slug, type, title, ...`) and fails the build
if any projection lacks `source_id`. Wired into `bun run verify`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(engine): Page.source_id required + listAllPageRefs + validateSourceId

Three coordinated changes that unlock the Phase 3 bug-site fixes:

1. `Page.source_id` is now required (was optional, v0.31.12). The DB column
   is `NOT NULL DEFAULT 'default'` so every row has it; the type now matches.
   `rowToPage` always emits it (falls back to 'default' if a stale projection
   somehow misses the column, but `scripts/check-source-id-projection.sh` is
   the primary guard).

2. `BrainEngine.listAllPageRefs()` returns `Array<{slug, source_id}>` ordered
   by `(source_id, slug)`. Cheap cross-source enumeration for hot loops in
   extract-takes / extract / integrity that previously used
   `getAllSlugs() → getPage(slug)` (N+1 query AND silently defaulted to
   'default'). PGLite + Postgres parity.

3. `validateSourceId(id)` in utils. Allows `[a-z0-9_-]+` only. Used by the
   per-source disk-layout fix coming in Phase 3 before any
   `join(brainDir, source_id, ...)` call so source_id can't traverse out
   of brainDir.

Deferred to v0.33 follow-up:
- D2 strict tightening of BrainEngine slug-method signatures (the compile-
  time guard for "future getPage calls must pass sourceId")
- F3 OperationContext.sourceId required at MCP boundary
- F4 LinkBatchInput / TimelineBatchInput required source_id fields
- D6 forEachPage / listPagesAfter helpers (use listPages directly for now)

Those are nice-to-have guardrails for future regressions. Current commit's
correctness via D7 + listAllPageRefs is what blocks the Phase 3 bug-site
fixes from working multi-source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: thread source_id through cycle phases, extract, integrity, migrate-engine

Five bug sites that previously called slug-only engine methods inside a
loop over pages, silently defaulting to source_id='default' for every
non-default-source page. Now all five use listAllPageRefs to enumerate
(slug, source_id) pairs and thread sourceId through to engine.getPage,
getTags, addLink, addTimelineEntry, getRawData, getVersions, etc.

Site-by-site:

- src/core/cycle/extract-takes.ts: listAllPageRefs replaces N+1
  getAllSlugs+getPage. Takes for non-default-source pages now extract.

- src/core/cycle/patterns.ts + synthesize.ts: reverseWriteSlugs renamed
  to reverseWriteRefs with Array<{slug, source_id}> contract. Disk
  layout (F6): non-default sources land at brainDir/.sources/<id>/<slug>.md
  so same-slug-different-source pages don't collide. Default-source
  pages stay at brainDir/<slug>.md so single-source brains see no
  change. source_id validated against [a-z0-9_-]+ at write time to
  prevent path traversal.

- src/commands/extract.ts: extractLinksFromDB + extractTimelineFromDB
  use listAllPageRefs. Cross-source link resolution rule (F10): origin's
  source wins, fall back to default, else skip (don't silently push a
  wrong-source edge). addLinksBatch / addTimelineEntriesBatch now fill
  from_source_id / to_source_id / origin_source_id / source_id so
  multi-source JOINs target the correct page row.

- src/commands/integrity.ts: same listAllPageRefs pattern in both the
  primary scan loop and the auto-repair loop.

- src/commands/migrate-engine.ts: end-to-end source_id threading
  (page + tags + timeline + raw + versions + links). Resume manifest
  keyed on `${source_id}::${slug}` so multi-source resumes don't
  collide on same-slug rows (pre-fix entries treated as default for
  back-compat).

test/cycle-synthesize-slug-collection.test.ts updated for the new
collectChildPutPageSlugs return shape (Array<{slug, source_id}>
instead of string[]).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): multi-source bug class regression + CHANGELOG + e2e-test-map wire-up

test/e2e/multi-source-bug-class.test.ts — 7-case PGLite regression suite
pinning every bug site fixed in this PR:
  - listAllPageRefs ordering by (source_id, slug) [F11]
  - getPage with sourceId picks the right (source, slug) row [F2]
  - extract-takes processes both alice pages independently
  - listPages filters correctly with PageFilters.sourceId
  - addLinksBatch with from/to_source_id targets the right rows [F4]
  - validateSourceId rejects path traversal [F6]
  - reverse-write disk layout uses .sources/<id>/<slug>.md [F6]

No DATABASE_URL needed (PGLite in-memory + canonical R3+R4 pattern).

Wire into scripts/e2e-test-map.ts so changes to any of the 6 touched
source files automatically trigger this test.

CHANGELOG expanded from the embed-only narrative to cover the full
bug-class extermination — extract, takes, patterns, integrity,
migrate-engine, plus the per-source disk layout, the CI gate, and
the new listAllPageRefs primitive. Voice: lead with what users can
DO that they couldn't before; real numbers from the production brain
that surfaced it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(integrity): batch path scans (source_id, slug) pairs too

The batch-load fast path in scanIntegrity used `SELECT DISTINCT ON (slug)`,
which silently collapsed multi-source duplicate slugs into a single scan —
the same bug class this PR fixes. test/e2e/integrity-batch.test.ts had a
case pinning the broken behavior ("scan once, not once-per-source") that
asserted batchResult.pagesScanned===1 for two real (source, slug) rows.

Switching the projection from `DISTINCT ON (slug)` to a plain `SELECT ...
ORDER BY source_id, slug` makes batch + sequential paths report the same
count (2) and matches the v0.32.4 listAllPageRefs walk.

Test renamed + assertion flipped to lock in the correct multi-source-aware
behavior: both paths now report 2, not 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync CLAUDE.md + llms bundles for v0.32.4

CLAUDE.md annotations updated on the 4 files that materially changed in
this PR's bug-class extermination:

- src/core/engine.ts — new listAllPageRefs() method
- src/core/utils.ts — new validateSourceId() helper + Page.source_id
  required field plumbing
- src/commands/integrity.ts — batch projection switched from DISTINCT ON
  (slug) to ORDER BY (source_id, slug) so multi-source scans aren't
  collapsed
- scripts/check-source-id-projection.sh (NEW entry) — CI guard against
  SELECT projections that drop source_id

Plus a new test inventory entry for test/e2e/multi-source-bug-class.test.ts
in the E2E section.

llms-full.txt regenerated per CLAUDE.md's iron rule. llms.txt is unchanged
(just an index).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version slot v0.32.4 → v0.32.8

VERSION + package.json + CHANGELOG header only. Annotation
sweep across src/tests/scripts and the CLAUDE.md + llms bundle
regen land in the two follow-up commits so each step bisects
independently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: retag v0.32.4 → v0.32.8 across src/scripts/tests

Inline "introduced in" annotations follow the version slot bump
in the prior commit. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: retag CLAUDE.md v0.32.4 → v0.32.8 + regen llms-full.txt

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Merge remote-tracking branch 'origin/master' into fix/multi-source-threading

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
garrytan-agents
2026-05-11 23:02:03 -07:00
committed by GitHub
parent c9652443cf
commit e493d5f44b
23 changed files with 823 additions and 158 deletions

View File

@@ -2,6 +2,80 @@
All notable changes to GBrain will be documented in this file.
## [0.32.8] - 2026-05-11
**Multi-source brains finish what they start. Embed, extract, takes, patterns, integrity, migrate-engine all now respect which source a page belongs to. The disk-side collision is fixed via a per-source subdir layout, and a CI gate prevents the bug class from coming back.**
If you run gbrain with more than one source (say a `media-corpus` alongside `default`), the bug pattern was everywhere: embed was leaving thousands of chunks unembedded, extract was silently dropping links from non-default sources, takes never extracted, `gbrain dream` was overwriting `brainDir/people/alice.md` with whichever source happened to reverse-write last. The CLI reported success on all of it. This release threads `source_id` through every page-to-chunk-to-link-to-take handoff, fixes the disk-side collision with a `.sources/` subdir layout, and adds a CI gate so future SELECT projections can't silently drop the column again.
### The numbers that matter
```
Pages on non-default source (production multi-source brain) → 5,042
Chunks left unembedded pre-fix → ~22,000
Chunks recovered on first re-run → 97 across 35 pages
(after 3 prior --stale
runs that recovered 0)
Engine SELECT projections audited for source_id → 4 sites (was 2 missing)
Bug sites threaded with explicit source_id → 5 (extract-takes,
patterns, synthesize,
extract, integrity)
+ migrate-engine end-to-end
```
### What changed
- **Bug-class extermination**: `embed`, `extract` (links + timeline), `extract-takes`, `patterns` reverse-write, `synthesize` reverse-write, `integrity` scan, and `migrate-engine` all now use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` through every engine method call. Pre-fix, each of these silently defaulted to `source_id='default'` for non-default-source pages.
- **`gbrain embed --source <id>`** flag for scoping embed to one source explicitly. Useful for re-embedding just `media-corpus` after a model swap.
- **Per-source disk layout**: `gbrain dream` reverse-write now lands non-default source pages at `brainDir/.sources/<source>/<slug>.md`. Default-source pages stay at `brainDir/<slug>.md` so single-source brains see no change. `.sources/` is a reserved prefix; the leading dot keeps it out of default-source sync (`walkBrainRepo` skips dot-dirs).
- **Source-aware link resolution**: a media-corpus page wikilinking to `people/alice` resolves to `alice@media-corpus` if that page exists, `alice@default` as a fallback, or stays unresolved (rather than silently pushing the edge to the wrong source). `addLinksBatch` callers now fill `from_source_id` / `to_source_id` / `origin_source_id` so the JOIN targets the right page.
- **`migrate-engine` end-to-end source_id threading**: page + tags + timeline + raw + versions + links all carry source_id through the migration. Resume manifest keyed on `${source_id}::${slug}` so multi-source resumes don't collide on same-slug-different-source rows.
- **New iteration primitive**: `engine.listAllPageRefs()` returns `Array<{slug, source_id}>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains. Replaces the `getAllSlugs() → getPage(slug)` N+1 pattern.
- **`Page.source_id` is now required at the type level**: the DB column is `NOT NULL DEFAULT 'default'`; the type now matches. Test fixtures building synthetic Page rows must set the field.
- **`validateSourceId()`**: new helper in `src/core/utils.ts`. Allows `[a-z0-9_-]+` only; rejects `..`, `/`, dots, uppercase. Used by the disk-layout fix before any `join(brainDir, source_id, ...)` call so source_id can't traverse out of brainDir.
- **CI gate (`scripts/check-source-id-projection.sh`)**: greps engine SELECT projections for the rowToPage feeder shape and fails the build if any drops `source_id`. Wired into `bun run verify`. Codex's outside-voice review caught two pre-existing projections (`getPage`, `putPage RETURNING`) that lacked the column; this commit fixes them and prevents the regression from recurring.
### To take advantage of v0.32.8
`gbrain upgrade` should do this automatically. Existing multi-source brains see immediate improvement on the next dream cycle.
1. Recover any chunks silently skipped on prior runs:
```bash
gbrain embed --stale
```
2. Re-run extract to pick up multi-source links + timeline entries:
```bash
gbrain extract all
```
3. Verify with `gbrain doctor` — embed coverage should jump on multi-source brains.
Single-source (`default`-only) brains see no behavior change. The fix is a no-op on the brain shape that ships from `gbrain init`.
Existing on-disk files at `brainDir/<slug>.md` for non-default sources stay where they are (no migration). The next reverse-write of those pages by the dream cycle moves them to `brainDir/.sources/<source>/<slug>.md`. Force the move today by deleting the stale file and re-running `gbrain dream --phase patterns`.
### For contributors
- `Page.source_id` is now a required field. Test fixtures building synthetic Page rows must include it.
- `LinkBatchInput.from_source_id` / `to_source_id` / `origin_source_id` and `TimelineBatchInput.source_id` are still optional but recommended at every call site. Future v0.33 may flip them to required.
- New `scripts/check-source-id-projection.sh` CI gate: any new SELECT touching `pages` that feeds `rowToPage` must project `source_id`.
### Itemized changes
- `src/core/types.ts` — `Page.source_id: string` (now required).
- `src/core/engine.ts` — new `BrainEngine.listAllPageRefs(): Promise<Array<{slug, source_id}>>`.
- `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` — implement `listAllPageRefs` with `ORDER BY source_id, slug`; add `source_id` to the `getPage` SELECT projection and `putPage` RETURNING projection.
- `src/core/utils.ts` — new `validateSourceId(id)` helper.
- `src/core/cycle/extract-takes.ts` — `listAllPageRefs` replaces `getAllSlugs+getPage` N+1; threads `sourceId` to `getPage`.
- `src/core/cycle/patterns.ts` + `synthesize.ts` — `reverseWriteSlugs` → `reverseWriteRefs` with `Array<{slug, source_id}>` contract. Per-source disk layout for non-default sources; `validateSourceId` guard before any `join()`.
- `src/commands/extract.ts` — `extractLinksFromDB` + `extractTimelineFromDB` use `listAllPageRefs`. Cross-source link resolution rule (origin > default > skip). Threads `from_source_id` / `to_source_id` / `origin_source_id` to `addLinksBatch`; `source_id` to `addTimelineEntriesBatch`.
- `src/commands/integrity.ts` — `listAllPageRefs` replaces `getAllSlugs+getPage` N+1 in both the primary scan and auto-repair loops.
- `src/commands/migrate-engine.ts` — page + tags + timeline + raw + versions + links all carry `sourceId`. Resume manifest key now `${source_id}::${slug}`.
- `src/commands/embed.ts` — (from prior commit in this PR) three code paths thread `sourceId`; `--source <id>` CLI flag; `embedAllStale` composite-key grouping.
- `scripts/check-source-id-projection.sh` (NEW) — CI gate.
- `test/e2e/multi-source-bug-class.test.ts` (NEW) — 7-case PGLite E2E regression suite pinning every bug site.
Supersedes #845 (which fixed `embed --stale` only).
## [0.32.7] - 2026-05-11
**CJK users get a working brain end-to-end on PGLite.**
@@ -121,6 +195,7 @@ If you imported a Chinese / Japanese / Korean brain pre-v0.32.7 and saw silently
- which step broke
This feedback loop is how gbrain maintainers find fragile upgrade paths. Thank you @vinsew + @313094319-sudo for filing the originating PRs.
## [0.32.6] - 2026-05-11
**Your brain learns to detect its own integrity drift.**

File diff suppressed because one or more lines are too long

View File

@@ -1 +1 @@
0.32.7
0.32.8

File diff suppressed because one or more lines are too long

View File

@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.32.7",
"version": "0.32.8",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -36,11 +36,11 @@
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run typecheck",
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run typecheck",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
@@ -52,6 +52,7 @@
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",

View File

@@ -0,0 +1,97 @@
#!/usr/bin/env bash
# CI guard: fail if any SELECT projection on `pages` that feeds rowToPage()
# drops `source_id`. After v0.32.8, Page.source_id is required at the type
# level; a projection that omits the column makes rowToPage return a Page
# with source_id=undefined, which TypeScript's `: string` then lies about.
#
# This complements the type-system guard. The grep finds the specific 4-tuple
# shape (id, slug, type, title) without source_id — the exact pre-v0.32.8
# pattern that codex's plan review flagged.
#
# Usage: scripts/check-source-id-projection.sh
# Exit: 0 when no matches, 1 when matches found.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Allowlist: SELECT shapes that legitimately don't need source_id (single-col
# `SELECT slug FROM pages` for getAllSlugs / resolveSlugs, SELECT id for
# subqueries, COUNT, etc.) These don't feed rowToPage.
#
# The shape that DOES feed rowToPage starts `SELECT id, ... slug, ... type, ... title`
# (in some order). The pattern below matches "id" + "slug" + "type" + "title"
# in a SELECT projection — that's the rowToPage feeder signature.
FOUND_BAD=0
# Use multiline-aware grep so the SELECT can span lines. pcre2grep would be
# cleaner but isn't universally available; do a simple two-pass instead:
# 1. Pull each SELECT-from-pages block.
# 2. For each, check if it has the rowToPage signature WITHOUT source_id.
check_file() {
local file="$1"
# Extract every SELECT...FROM pages block (across lines, up to 12 lines)
# then test each.
awk '
/SELECT/ {
buf = $0
lines = 1
while (lines < 12 && (!match(buf, /FROM[[:space:]]+pages\b/))) {
if ((getline next_line) <= 0) break
buf = buf " " next_line
lines++
}
if (match(buf, /FROM[[:space:]]+pages\b/)) {
# Has id, slug, type, title (rowToPage feeder) but NO source_id?
if (match(buf, /\bid\b/) && match(buf, /\bslug\b/) && match(buf, /\btype\b/) && match(buf, /\btitle\b/) && !match(buf, /\bsource_id\b/)) {
print FILENAME ": SELECT projection missing source_id:"
print " " buf
exit 1
}
}
}
' "$file" || return 1
return 0
}
EXIT=0
for f in src/core/postgres-engine.ts src/core/pglite-engine.ts; do
if ! check_file "$f"; then
EXIT=1
fi
done
# Also check RETURNING clauses (putPage uses INSERT ... RETURNING).
# Same shape: returns a row that feeds rowToPage.
for f in src/core/postgres-engine.ts src/core/pglite-engine.ts; do
awk '
/RETURNING/ {
buf = $0
lines = 1
while (lines < 6 && !match(buf, /\`/)) {
if ((getline next_line) <= 0) break
buf = buf " " next_line
lines++
}
if (match(buf, /\bid\b/) && match(buf, /\bslug\b/) && match(buf, /\btype\b/) && match(buf, /\btitle\b/) && !match(buf, /\bsource_id\b/)) {
print FILENAME ": RETURNING projection missing source_id:"
print " " buf
exit 1
}
}
' "$f" || EXIT=1
done
if [ "$EXIT" = 1 ]; then
echo
echo "ERROR: SELECT/RETURNING projection on \`pages\` is missing source_id."
echo " After v0.32.8, Page.source_id is required at the type level."
echo " Add \`source_id\` to the projection or rowToPage will lie."
echo " See ~/.claude/plans/gleaming-soaring-mccarthy.md F2 finding."
exit 1
fi
echo "OK: all rowToPage feeder projections include source_id"

View File

@@ -38,6 +38,14 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
"src/core/cycle.ts": ["test/e2e/cycle.test.ts", "test/e2e/dream.test.ts"],
// Multi-source sync writes share the per-source bookmark anchor.
"src/core/sync.ts": ["test/e2e/sync.test.ts", "test/e2e/multi-source.test.ts"],
// v0.32.8 multi-source bug class regression suite — fires on any cycle
// phase, extract, integrity, embed, or migrate-engine change.
"src/core/cycle/extract-takes.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/core/cycle/patterns.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/core/cycle/synthesize.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"],
"src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"],
// Any minions queue/worker/handler change exercises all minion E2E.
"src/core/minions/**": [
"test/e2e/minions-concurrency.test.ts",

View File

@@ -14,6 +14,12 @@ export interface EmbedOpts {
slugs?: string[];
/** Embed a single page. */
slug?: string;
/**
* v0.31.12: scope to a specific source. When set, only pages from this
* source are embedded. When omitted, all sources are processed (but
* source_id is still threaded correctly per-page via Page.source_id).
*/
sourceId?: string;
/**
* Dry run: enumerate what WOULD be embedded (stale chunk counts)
* without calling the embedding model or writing to the engine.
@@ -73,7 +79,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
if (opts.slugs && opts.slugs.length > 0) {
for (const s of opts.slugs) {
try {
await embedPage(engine, s, !!opts.dryRun, result);
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId);
} catch (e: unknown) {
console.error(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
@@ -81,11 +87,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
return result;
}
if (opts.all || opts.stale) {
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress);
await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId);
return result;
}
if (opts.slug) {
await embedPage(engine, opts.slug, !!opts.dryRun, result);
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId);
return result;
}
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
@@ -96,19 +102,22 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
const all = args.includes('--all');
const stale = args.includes('--stale');
const dryRun = args.includes('--dry-run');
// v0.31.12: --source <id> scopes to a single source.
const sourceIdx = args.indexOf('--source');
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
let opts: EmbedOpts;
if (slugsIdx >= 0) {
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun };
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId };
} else if (all || stale) {
opts = { all, stale, dryRun };
opts = { all, stale, dryRun, sourceId };
} else {
const slug = args.find(a => !a.startsWith('--'));
if (!slug) {
console.error('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
process.exit(1);
}
opts = { slug, dryRun };
opts = { slug, dryRun, sourceId };
}
// CLI path: wire a reporter so --progress-json / --quiet / TTY rendering
@@ -140,8 +149,10 @@ async function embedPage(
slug: string,
dryRun: boolean,
result: EmbedResult,
sourceId?: string,
) {
const page = await engine.getPage(slug);
const opts = sourceId ? { sourceId } : undefined;
const page = await engine.getPage(slug, opts);
if (!page) {
throw new Error(`Page not found: ${slug}`);
}
@@ -149,7 +160,7 @@ async function embedPage(
// Get existing chunks or create new ones.
// In dryRun, we still chunk the text locally to count what WOULD be
// embedded — but we never write chunks or call the embedding model.
let chunks = await engine.getChunks(slug);
let chunks = await engine.getChunks(slug, opts);
if (chunks.length === 0) {
const inputs: ChunkInput[] = [];
if (page.compiled_truth.trim()) {
@@ -172,8 +183,8 @@ async function embedPage(
}
if (inputs.length > 0) {
await engine.upsertChunks(slug, inputs);
chunks = await engine.getChunks(slug);
await engine.upsertChunks(slug, inputs, opts);
chunks = await engine.getChunks(slug, opts);
}
}
@@ -207,7 +218,7 @@ async function embedPage(
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(slug, updated);
await engine.upsertChunks(slug, updated, opts);
result.embedded += toEmbed.length;
result.pages_processed++;
console.log(`${slug}: embedded ${toEmbed.length} chunks`);
@@ -219,6 +230,7 @@ async function embedAll(
dryRun: boolean,
result: EmbedResult,
onProgress?: (done: number, total: number, embedded: number) => void,
sourceId?: string,
) {
// ─────────────────────────────────────────────────────────────
// Stale-only fast path: avoid the listPages + per-page getChunks
@@ -237,7 +249,8 @@ async function embedAll(
return await embedAllStale(engine, dryRun, result, onProgress);
}
const pages = await engine.listPages({ limit: 100000 });
// v0.31.12: when sourceId is set, scope listPages to that source.
const pages = await engine.listPages({ limit: 100000, ...(sourceId && { sourceId }) });
let processed = 0;
// Concurrency limit for parallel page embedding.
@@ -251,7 +264,11 @@ async function embedAll(
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
async function embedOnePage(page: typeof pages[number]) {
const chunks = await engine.getChunks(page.slug);
// v0.31.12: thread source_id from the page row so getChunks/upsertChunks
// target the correct (source_id, slug) row, not the 'default' source.
const pageSourceId = page.source_id;
const pageOpts = pageSourceId ? { sourceId: pageSourceId } : undefined;
const chunks = await engine.getChunks(page.slug, pageOpts);
const toEmbed = chunks; // staleOnly path handled above via embedAllStale
result.total_chunks += chunks.length;
@@ -287,7 +304,7 @@ async function embedAll(
embedding: embeddingMap.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(page.slug, updated);
await engine.upsertChunks(page.slug, updated, pageOpts);
result.embedded += toEmbed.length;
} catch (e: unknown) {
console.error(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
@@ -360,15 +377,17 @@ async function embedAllStale(
// Pull only the stale chunks (no embedding column).
const staleRows = await engine.listStaleChunks();
// Group by slug so each slug → array of stale chunks for batched embedding.
const bySlug = new Map<string, typeof staleRows>();
// v0.31.12: group by composite key (source_id::slug) so same-slug pages
// in different sources are embedded independently with correct source_id.
const byKey = new Map<string, typeof staleRows>();
for (const row of staleRows) {
const list = bySlug.get(row.slug);
const key = `${row.source_id}::${row.slug}`;
const list = byKey.get(key);
if (list) list.push(row);
else bySlug.set(row.slug, [row]);
else byKey.set(key, [row]);
}
const slugs = Array.from(bySlug.keys());
const keys = Array.from(byKey.keys());
const totalStaleChunks = staleRows.length;
result.total_chunks += totalStaleChunks;
// skipped is "chunks we considered and skipped due to having an embedding".
@@ -378,21 +397,27 @@ async function embedAllStale(
if (dryRun) {
result.would_embed += totalStaleChunks;
result.pages_processed += slugs.length;
result.pages_processed += keys.length;
if (onProgress) {
// Emit a single tick to satisfy the contract (CLI progress reporters
// expect at least one start/finish pair).
onProgress(slugs.length, slugs.length, 0);
onProgress(keys.length, keys.length, 0);
}
console.log(`[dry-run] Would embed ${totalStaleChunks} chunks across ${slugs.length} pages`);
console.log(`[dry-run] Would embed ${totalStaleChunks} chunks across ${keys.length} pages`);
return;
}
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
let processed = 0;
async function embedOneSlug(slug: string) {
const stale = bySlug.get(slug)!;
async function embedOneKey(key: string) {
const stale = byKey.get(key)!;
// v0.31.12: extract source_id + slug from the stale row so getChunks
// and upsertChunks target the correct (source_id, slug) row. Pre-fix,
// both calls defaulted to source_id='default', silently discarding
// embeddings for every non-default source (e.g. media-corpus).
const sourceId = stale[0]?.source_id ?? 'default';
const slug = stale[0].slug;
try {
const embeddings = await embedBatch(stale.map(c => c.chunk_text));
// CRITICAL: passing ONLY the stale indices to upsertChunks would
@@ -401,7 +426,7 @@ async function embedAllStale(
// re-fetch existing chunks for this page and merge. Bounded by the
// stale slug count, not by total slugs — autopilot common case
// is 0 stale (pre-flight short-circuit, never reaches this path).
const existing = await engine.getChunks(slug);
const existing = await engine.getChunks(slug, { sourceId });
const staleIdxToEmbedding = new Map<number, Float32Array>();
for (let j = 0; j < stale.length; j++) {
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
@@ -415,26 +440,26 @@ async function embedAllStale(
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
}));
await engine.upsertChunks(slug, merged);
await engine.upsertChunks(slug, merged, { sourceId });
result.embedded += stale.length;
} catch (e: unknown) {
console.error(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
processed++;
result.pages_processed++;
onProgress?.(processed, slugs.length, result.embedded);
onProgress?.(processed, keys.length, result.embedded);
}
let nextIdx = 0;
async function worker() {
while (nextIdx < slugs.length) {
while (nextIdx < keys.length) {
const idx = nextIdx++;
await embedOneSlug(slugs[idx]);
await embedOneKey(keys[idx]);
}
}
const numWorkers = Math.min(CONCURRENCY, slugs.length);
const numWorkers = Math.min(CONCURRENCY, keys.length);
await Promise.all(Array.from({ length: numWorkers }, () => worker()));
console.log(`Embedded ${result.embedded} chunks across ${slugs.length} pages`);
console.log(`Embedded ${result.embedded} chunks across ${keys.length} pages`);
}

View File

@@ -777,12 +777,25 @@ async function extractLinksFromDB(
const nullResolver = {
resolve: async () => null as string | null,
};
const allSlugs = await engine.getAllSlugs();
const slugList = Array.from(allSlugs);
// v0.32.8: listAllPageRefs enumerates (slug, source_id) so we can thread
// sourceId to getPage AND build a cross-source resolution map for link
// disambiguation. Pre-fix used getAllSlugs() which collapsed
// same-slug-different-source pages into one entry.
const allRefs = await engine.listAllPageRefs();
// For backward-compat checks (`allSlugs.has(...)` calls below), we still
// need a flat slug set. ALSO a per-slug → [sources] map for F10 resolution.
const allSlugs = new Set<string>();
const slugToSources = new Map<string, string[]>();
for (const ref of allRefs) {
allSlugs.add(ref.slug);
const list = slugToSources.get(ref.slug) ?? [];
list.push(ref.source_id);
slugToSources.set(ref.slug, list);
}
let processed = 0, created = 0;
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start('extract.links_db', slugList.length);
progress.start('extract.links_db', allRefs.length);
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
const dryRunSeen = dryRun ? new Set<string>() : null;
@@ -804,9 +817,8 @@ async function extractLinksFromDB(
}
}
for (let i = 0; i < slugList.length; i++) {
const slug = slugList[i];
const page = await engine.getPage(slug);
for (const { slug, source_id } of allRefs) {
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
if (typeFilter && page.type !== typeFilter) continue;
if (since) {
@@ -832,13 +844,38 @@ async function extractLinksFromDB(
const fromSlug = c.fromSlug ?? slug;
if (!allSlugs.has(c.targetSlug)) continue;
if (!allSlugs.has(fromSlug)) continue;
// v0.32.8 F10: cross-source link resolution.
// from_source_id = origin page's source_id (this loop's source_id, or
// the candidate's fromSlug source if it lives in a different source).
// to_source_id = priority: origin's source > 'default' > skip (don't
// silently push a wrong-source edge).
const fromSources = slugToSources.get(fromSlug) ?? [];
const fromSourceId = fromSources.includes(source_id) ? source_id
: (fromSources.includes('default') ? 'default' : fromSources[0]);
const targetSources = slugToSources.get(c.targetSlug) ?? [];
let toSourceId: string;
if (targetSources.includes(fromSourceId)) {
toSourceId = fromSourceId;
} else if (targetSources.includes('default')) {
toSourceId = 'default';
} else {
// Target exists ONLY in non-origin/non-default sources. Skip — don't
// silently push a wrong-source edge. Tracking this as an unresolved
// ref would require expanding UnresolvedFrontmatterRef; for v0.32.8
// a quiet skip is the conservative choice (matches existing
// "target missing" semantics where allSlugs.has() returns false).
continue;
}
if (dryRunSeen) {
const key = `${fromSlug}::${c.targetSlug}::${c.linkType}::${c.linkSource ?? 'markdown'}`;
const key = `${fromSourceId}::${fromSlug}::${toSourceId}::${c.targetSlug}::${c.linkType}::${c.linkSource ?? 'markdown'}`;
if (dryRunSeen.has(key)) continue;
dryRunSeen.add(key);
if (jsonMode) {
process.stdout.write(JSON.stringify({
action: 'add_link', from: fromSlug, to: c.targetSlug,
action: 'add_link', from: fromSlug, from_source_id: fromSourceId,
to: c.targetSlug, to_source_id: toSourceId,
type: c.linkType, context: c.context, link_source: c.linkSource,
}) + '\n');
} else {
@@ -854,6 +891,12 @@ async function extractLinksFromDB(
link_source: c.linkSource,
origin_slug: c.originSlug,
origin_field: c.originField,
// v0.32.8 F4: thread source ids so the batch JOIN doesn't fan out
// across sources. Default source_id='default' for back-compat with
// pre-v0.32.8 callers (the engine still accepts undefined).
from_source_id: fromSourceId,
to_source_id: toSourceId,
origin_source_id: source_id,
});
if (batch.length >= BATCH_SIZE) await flush();
}
@@ -892,12 +935,14 @@ async function extractTimelineFromDB(
typeFilter: PageType | undefined,
since: string | undefined,
): Promise<{ created: number; pages: number }> {
const allSlugs = await engine.getAllSlugs();
const slugList = Array.from(allSlugs);
// v0.32.8: listAllPageRefs enumerates (slug, source_id) pairs so we can
// thread sourceId to getPage and addTimelineEntriesBatch. Pre-fix used
// getAllSlugs() which collapsed same-slug-different-source pages.
const allRefs = await engine.listAllPageRefs();
let processed = 0, created = 0;
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start('extract.timeline_db', slugList.length);
progress.start('extract.timeline_db', allRefs.length);
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
const dryRunSeen = dryRun ? new Set<string>() : null;
@@ -919,9 +964,8 @@ async function extractTimelineFromDB(
}
}
for (let i = 0; i < slugList.length; i++) {
const slug = slugList[i];
const page = await engine.getPage(slug);
for (const { slug, source_id } of allRefs) {
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
if (typeFilter && page.type !== typeFilter) continue;
if (since) {
@@ -935,12 +979,12 @@ async function extractTimelineFromDB(
for (const entry of entries) {
if (dryRunSeen) {
const key = `${slug}::${entry.date}::${entry.summary}`;
const key = `${source_id}::${slug}::${entry.date}::${entry.summary}`;
if (dryRunSeen.has(key)) continue;
dryRunSeen.add(key);
if (jsonMode) {
process.stdout.write(JSON.stringify({
action: 'add_timeline', slug, date: entry.date,
action: 'add_timeline', slug, source_id, date: entry.date,
summary: entry.summary, ...(entry.detail ? { detail: entry.detail } : {}),
}) + '\n');
} else {
@@ -948,7 +992,9 @@ async function extractTimelineFromDB(
}
created++;
} else {
batch.push({ slug, date: entry.date, summary: entry.summary, detail: entry.detail || '' });
// v0.32.8 F4: thread source_id so the JOIN matches the right page
// when two sources share the same slug.
batch.push({ slug, date: entry.date, summary: entry.summary, detail: entry.detail || '', source_id });
if (batch.length >= BATCH_SIZE) await flush();
}
}

View File

@@ -316,16 +316,21 @@ export async function scanIntegrity(
}
}
const allSlugs = [...(await engine.getAllSlugs())].sort();
// v0.32.8: listAllPageRefs replaces getAllSlugs+getPage N+1 pattern that
// silently defaulted to source_id='default' for non-default-source pages.
// Now we enumerate (slug, source_id) pairs and thread sourceId to getPage.
const allRefs = (await engine.listAllPageRefs()).sort((a, b) =>
a.slug.localeCompare(b.slug) || a.source_id.localeCompare(b.source_id)
);
const bareHits: BareTweetHit[] = [];
const externalHits: ExternalLinkHit[] = [];
let pagesScanned = 0;
for (const slug of allSlugs) {
for (const { slug, source_id } of allRefs) {
if (typeFilter && !slug.startsWith(`${typeFilter}/`)) continue;
if (pagesScanned >= limit) break;
const page = await engine.getPage(slug);
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
// Skip grandfathered pages (opted out of brain-integrity enforcement)
if ((page.frontmatter as Record<string, unknown> | undefined)?.validate === false) continue;
@@ -359,14 +364,16 @@ async function scanIntegrityBatch(
// — gbrain lint should reject stringly-typed validate at write time.
const validateCondition = sql`AND (frontmatter->>'validate' IS NULL OR frontmatter->>'validate' != 'false')`;
// DISTINCT ON (slug) mirrors getAllSlugs()'s Set<string> semantics: multi-source
// brains can have the same slug under multiple source_ids (UNIQUE(source_id, slug)
// since v0.18.0); we want one scan per slug, not one per row.
// v0.32.8: scan ONE row per (source_id, slug) pair, not one per slug.
// Pre-fix used DISTINCT ON (slug) which collapsed multi-source rows into
// one — that was the bug class. Now batch parity matches the sequential
// listAllPageRefs() walk: integrity violations in non-default-source pages
// get reported instead of silently shadowed by their default-source twin.
const rows = await sql`
SELECT DISTINCT ON (slug) slug, compiled_truth, frontmatter
SELECT slug, compiled_truth, frontmatter
FROM pages
WHERE 1=1 ${typeCondition} ${validateCondition}
ORDER BY slug
ORDER BY source_id, slug
LIMIT ${limit}
`;
@@ -440,14 +447,19 @@ async function cmdAuto(args: string[]): Promise<void> {
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
try {
const allSlugs = [...(await engine.getAllSlugs())].sort();
const toScan = allSlugs.filter(s => !seen.has(s));
// v0.32.8: listAllPageRefs enumerates (slug, source_id) pairs so we
// can thread sourceId to getPage. Pre-fix this defaulted to 'default'
// and silently skipped non-default-source pages.
const allRefs = (await engine.listAllPageRefs()).sort((a, b) =>
a.slug.localeCompare(b.slug) || a.source_id.localeCompare(b.source_id)
);
const toScan = allRefs.filter(r => !seen.has(r.slug));
progress.start('integrity.auto', toScan.length);
for (const slug of allSlugs) {
for (const { slug, source_id } of allRefs) {
if (pagesProcessed >= limit) break;
if (seen.has(slug)) continue;
const page = await engine.getPage(slug);
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
pagesProcessed++;

View File

@@ -132,7 +132,13 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
console.log('Previous migration was to a different target. Starting fresh.');
manifest = null;
}
// v0.32.8 F8: manifest keys are now `${source_id}::${slug}` so multi-source
// migrations don't collide on same-slug-different-source pages. Pre-v0.32.8
// entries were bare slugs; we keep treating those as default-source for
// back-compat resume.
const completedSet = new Set(manifest?.completed_slugs || []);
const makeManifestKey = (sourceId: string, slug: string): string =>
sourceId === 'default' ? slug : `${sourceId}::${slug}`;
if (!manifest) {
manifest = {
completed_slugs: [],
@@ -144,7 +150,7 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
// Get all source pages
const sourceStats = await sourceEngine.getStats();
const allPages = await sourceEngine.listPages({ limit: 100000 });
const pagesToMigrate = allPages.filter(p => !completedSet.has(p.slug));
const pagesToMigrate = allPages.filter(p => !completedSet.has(makeManifestKey(p.source_id, p.slug)));
console.log(`Migrating ${pagesToMigrate.length} pages (${allPages.length} total, ${completedSet.size} already done)...`);
@@ -153,7 +159,14 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
let migrated = 0;
for (const page of pagesToMigrate) {
// Copy page
// v0.32.8 F8: thread source_id end-to-end so multi-source pages migrate
// intact. Pre-fix: putPage / getTags / getTimeline / getRawData / getLinks
// all silently defaulted to source_id='default', so non-default-source
// tags / timeline / raw / links were either dropped or attached to the
// wrong row.
const sourceOpts = { sourceId: page.source_id };
// Copy page (preserve source_id)
await targetEngine.putPage(page.slug, {
type: page.type,
title: page.title,
@@ -161,10 +174,10 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
timeline: page.timeline,
frontmatter: page.frontmatter,
content_hash: page.content_hash,
});
}, sourceOpts);
// Copy chunks with embeddings
const chunks = await sourceEngine.getChunksWithEmbeddings(page.slug);
// Copy chunks with embeddings.
const chunks = await sourceEngine.getChunksWithEmbeddings(page.slug, sourceOpts);
if (chunks.length > 0) {
await targetEngine.upsertChunks(page.slug, chunks.map(c => ({
chunk_index: c.chunk_index,
@@ -173,52 +186,59 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
embedding: c.embedding || undefined,
model: c.model,
token_count: c.token_count || undefined,
})));
})), sourceOpts);
}
// Copy tags
const tags = await sourceEngine.getTags(page.slug);
const tags = await sourceEngine.getTags(page.slug, sourceOpts);
for (const tag of tags) {
await targetEngine.addTag(page.slug, tag);
await targetEngine.addTag(page.slug, tag, sourceOpts);
}
// Copy timeline
const timeline = await sourceEngine.getTimeline(page.slug);
const timeline = await sourceEngine.getTimeline(page.slug, sourceOpts);
for (const entry of timeline) {
await targetEngine.addTimelineEntry(page.slug, {
date: entry.date,
source: entry.source,
summary: entry.summary,
detail: entry.detail,
});
}, sourceOpts);
}
// Copy raw data
const rawData = await sourceEngine.getRawData(page.slug);
const rawData = await sourceEngine.getRawData(page.slug, undefined, sourceOpts);
for (const rd of rawData) {
await targetEngine.putRawData(page.slug, rd.source, rd.data);
await targetEngine.putRawData(page.slug, rd.source, rd.data, sourceOpts);
}
// Copy versions
const versions = await sourceEngine.getVersions(page.slug);
const versions = await sourceEngine.getVersions(page.slug, sourceOpts);
// Versions are snapshots, we recreate them on the target
// (createVersion takes a snapshot of current state, which we just set)
// Track progress
manifest!.completed_slugs.push(page.slug);
// Track progress with composite key so multi-source resume is correct.
manifest!.completed_slugs.push(makeManifestKey(page.source_id, page.slug));
saveManifest(manifest!);
migrated++;
progress.tick(1, page.slug);
}
progress.finish();
// Copy links (after all pages exist in target)
// Copy links (after all pages exist in target).
// v0.32.8 F8: thread source_id so cross-source links migrate correctly.
console.log('Copying links...');
progress.start('migrate.copy_links', allPages.length);
for (const page of allPages) {
const links = await sourceEngine.getLinks(page.slug);
const sourceOpts = { sourceId: page.source_id };
const links = await sourceEngine.getLinks(page.slug, sourceOpts);
for (const link of links) {
await targetEngine.addLink(link.from_slug, link.to_slug, link.context, link.link_type);
await targetEngine.addLink(
link.from_slug, link.to_slug,
link.context, link.link_type,
undefined, undefined, undefined,
{ fromSourceId: page.source_id, toSourceId: page.source_id },
);
}
progress.tick(1);
}

View File

@@ -174,8 +174,14 @@ export async function extractTakesFromFs(
/**
* Iterate engine pages and re-extract takes from each `compiled_truth` body.
* Snapshot-stable (uses getAllSlugs). Doesn't read disk — works on
* Snapshot-stable (uses listAllPageRefs). Doesn't read disk — works on
* Postgres-only deployments without a local checkout.
*
* v0.32.8: replaces the prior `getAllSlugs() → getPage(slug)` pattern. The
* old version dropped `source_id` between the enumeration and the lookup,
* so a non-default-source page either matched the wrong (default-source)
* row or returned null when it didn't exist in default. Now we enumerate
* (slug, source_id) pairs and pass `sourceId` to getPage explicitly.
*/
export async function extractTakesFromDb(
engine: BrainEngine,
@@ -185,14 +191,17 @@ export async function extractTakesFromDb(
pagesScanned: 0, pagesWithTakes: 0, takesUpserted: 0, warnings: [], failedFiles: [],
};
const dryRun = opts.dryRun ?? false;
const slugs = opts.slugs && opts.slugs.length > 0
? opts.slugs
: Array.from(await engine.getAllSlugs());
// v0.32.8: when caller supplies bare slugs, default sourceId='default'
// (back-compat with pre-v0.32.8 callers). When no slugs supplied, enumerate
// every (slug, source_id) pair across all sources.
const refs: Array<{ slug: string; source_id: string }> = opts.slugs && opts.slugs.length > 0
? opts.slugs.map(slug => ({ slug, source_id: 'default' }))
: await engine.listAllPageRefs();
const buffer: TakeBatchInput[] = [];
for (const slug of slugs) {
for (const { slug, source_id } of refs) {
result.pagesScanned++;
const page = await engine.getPage(slug);
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
const body = `${page.compiled_truth ?? ''}\n${page.timeline ?? ''}`;
const { takes, warnings } = parseTakesFence(body);

View File

@@ -105,15 +105,17 @@ export async function runPhasePatterns(
try { await opts.yieldDuringPhase(); } catch { /* best-effort */ }
}
// Collect slugs the subagent wrote (codex finding #2 — query tool exec rows).
const writtenSlugs = await collectChildPutPageSlugs(engine, [job.id]);
// Collect refs the subagent wrote (codex finding #2 — query tool exec rows).
// v0.32.8: refs carry source_id so reverseWriteRefs targets the right
// (source, slug) row instead of the first DB match.
const writtenRefs = await collectChildPutPageSlugs(engine, [job.id]);
// Reverse-write to fs.
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
return ok(`${writtenSlugs.length} pattern page(s) written/updated (${outcome})`, {
return ok(`${writtenRefs.length} pattern page(s) written/updated (${outcome})`, {
reflections_considered: reflections.length,
patterns_written: writtenSlugs.length,
patterns_written: writtenRefs.length,
reverse_write_count: reverseWriteCount,
child_outcome: outcome,
job_id: job.id,
@@ -222,10 +224,13 @@ When done, briefly list the pattern slugs you wrote/updated in your final messag
async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
): Promise<string[]> {
): Promise<Array<{ slug: string; source_id: string }>> {
if (childIds.length === 0) return [];
// Handle both properly-stored jsonb objects (input->>'slug') and
// double-encoded jsonb strings from pre-fix data ((input #>> '{}')::jsonb->>'slug').
// v0.32.8: subagent put_page tool schema doesn't expose source_id (subagents
// are scoped to a single source). Default to 'default' here; multi-source
// dream cycles are a v0.33 follow-up. The point of threading source_id is
// so reverseWriteRefs can pass it through getPage and pick the correct
// (source_id, slug) row instead of whatever the DB happens to return.
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT DISTINCT
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
@@ -236,30 +241,44 @@ async function collectChildPutPageSlugs(
ORDER BY 1`,
[childIds],
);
return rows.map(r => r.slug).filter((s): s is string => typeof s === 'string' && s.length > 0);
return rows
.map(r => r.slug)
.filter((s): s is string => typeof s === 'string' && s.length > 0)
.map(slug => ({ slug, source_id: 'default' }));
}
// ── Reverse-write ────────────────────────────────────────────────────
async function reverseWriteSlugs(
import { validateSourceId } from '../utils.ts';
async function reverseWriteRefs(
engine: BrainEngine,
brainDir: string,
slugs: string[],
refs: Array<{ slug: string; source_id: string }>,
): Promise<number> {
let count = 0;
for (const slug of slugs) {
const page = await engine.getPage(slug);
for (const { slug, source_id } of refs) {
// v0.32.8 F6: guard against malformed source_id (would let join() break
// out of brainDir). validateSourceId throws on `..`, `/`, etc.
validateSourceId(source_id);
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
const tags = await engine.getTags(slug);
const tags = await engine.getTags(slug, { sourceId: source_id });
try {
const md = renderPageToMarkdown(page, tags);
const filePath = join(brainDir, `${slug}.md`);
// v0.32.8 F6: non-default sources land under brainDir/.sources/<id>/<slug>.md
// so same-slug-different-source pages don't collide on disk. Default-source
// pages stay at brainDir/<slug>.md so single-source brains see no change.
// `.sources/` is a reserved prefix; walkBrainRepo skips dot-dirs.
const filePath = source_id === 'default'
? join(brainDir, `${slug}.md`)
: join(brainDir, '.sources', source_id, `${slug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, md, 'utf8');
count++;
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${msg}\n`);
}
}
return count;

View File

@@ -37,6 +37,7 @@ import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts';
import { serializeMarkdown } from '../markdown.ts';
import type { Page, PageType } from '../types.ts';
import { validateSourceId } from '../utils.ts';
// Slug regex from validatePageSlug — kept in sync.
// Used for the orchestrator-written summary index slug.
@@ -455,15 +456,19 @@ export async function runPhaseSynthesize(
// D6 orchestrator slug rewrite: chunkInfo drives post-hoc rewrite of
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
// even if Sonnet drops the chunk suffix.
const writtenSlugs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
// (source, slug) row (currently always 'default' from subagent put_page).
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
// Dual-write: reverse-render each DB row → markdown file.
const reverseWriteCount = await reverseWriteSlugs(engine, opts.brainDir, writtenSlugs);
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
// Summary index page (deterministic; orchestrator-written via direct
// engine.putPage so no allow-list path needed).
const summaryDate = opts.date ?? today();
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
const writtenSlugs = writtenRefs.map(r => r.slug);
if (SUMMARY_SLUG_RE.test(summarySlug)) {
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
}
@@ -834,12 +839,19 @@ async function collectChildPutPageSlugs(
engine: BrainEngine,
childIds: number[],
chunkInfo: Map<number, { idx: number; hash6: string }>,
): Promise<string[]> {
): Promise<Array<{ slug: string; source_id: string }>> {
if (childIds.length === 0) return [];
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
// the orchestrator sees what each child wrote. COALESCE handles both
// properly-stored jsonb objects (input->>'slug') and double-encoded jsonb
// strings from pre-fix data ((input #>> '{}')::jsonb->>'slug').
//
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
// put_page tool schema doesn't expose source_id (subagents are scoped to
// a single source); default to 'default' for the current dream-cycle
// product behavior. Threading the source_id through reverseWriteRefs
// guarantees getPage targets the correct (source, slug) row instead of
// the first DB match.
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
`SELECT job_id,
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
@@ -855,7 +867,7 @@ async function collectChildPutPageSlugs(
const ci = chunkInfo.get(r.job_id);
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
}
return Array.from(rewritten).sort();
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
}
/**
@@ -886,26 +898,33 @@ async function hasLegacySingleChunkCompletion(
// ── Reverse-write DB rows → markdown files ───────────────────────────
async function reverseWriteSlugs(
async function reverseWriteRefs(
engine: BrainEngine,
brainDir: string,
slugs: string[],
refs: Array<{ slug: string; source_id: string }>,
): Promise<number> {
let count = 0;
for (const slug of slugs) {
const page = await engine.getPage(slug);
for (const { slug, source_id } of refs) {
// v0.32.8 F6: validate source_id is filesystem-safe before any join().
validateSourceId(source_id);
const page = await engine.getPage(slug, { sourceId: source_id });
if (!page) continue;
const tags = await engine.getTags(slug);
const tags = await engine.getTags(slug, { sourceId: source_id });
try {
const md = renderPageToMarkdown(page, tags);
const filePath = join(brainDir, `${slug}.md`);
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
// so same-slug-different-source pages don't collide. Default-source
// pages stay at brainDir/<slug>.md so single-source brains see no change.
const filePath = source_id === 'default'
? join(brainDir, `${slug}.md`)
: join(brainDir, '.sources', source_id, `${slug}.md`);
mkdirSync(dirname(filePath), { recursive: true });
writeFileSync(filePath, md, 'utf8');
count++;
} catch (e) {
// Per-slug failures are non-fatal — phase continues.
const msg = e instanceof Error ? e.message : String(e);
process.stderr.write(`[dream] reverse-write ${slug} failed: ${msg}\n`);
process.stderr.write(`[dream] reverse-write ${slug}@${source_id} failed: ${msg}\n`);
}
}
return count;

View File

@@ -498,6 +498,20 @@ export interface BrainEngine {
*/
getAllSlugs(opts?: { sourceId?: string }): Promise<Set<string>>;
/**
* v0.32.8: cross-source page enumeration. Returns one row per (slug,
* source_id) pair across the brain, ordered by (source_id, slug) for
* deterministic iteration on large brains. Used by extract-takes,
* extract, and integrity to replace the `getAllSlugs() → getPage(slug)`
* N+1 pattern, which silently defaulted to source_id='default' and
* skipped non-default-source pages.
*
* Cheap by design: only slug + source_id, not the full Page row. For
* loops that need page.compiled_truth / timeline / frontmatter, use
* `forEachPage` from src/core/engine-iter.ts instead.
*/
listAllPageRefs(): Promise<Array<{ slug: string; source_id: string }>>;
// Search
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]>;
@@ -1129,7 +1143,7 @@ export interface BrainEngine {
// Migration support
runMigration(version: number, sql: string): Promise<void>;
getChunksWithEmbeddings(slug: string): Promise<Chunk[]>;
getChunksWithEmbeddings(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]>;
// Raw SQL (for Minions job queue and other internal modules)
executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]>;

View File

@@ -503,7 +503,7 @@ export class PGLiteEngine implements BrainEngine {
where.push('deleted_at IS NULL');
}
const { rows } = await this.db.query(
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
`SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
params
);
@@ -551,7 +551,7 @@ export class PGLiteEngine implements BrainEngine {
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename),
chunker_version = COALESCE(EXCLUDED.chunker_version, pages.chunker_version),
source_path = COALESCE(EXCLUDED.source_path, pages.source_path)
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename`,
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename`,
[sourceId, slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename, chunkerVersion, sourcePath]
);
return rowToPage(rows[0] as Record<string, unknown>);
@@ -640,6 +640,11 @@ export class PGLiteEngine implements BrainEngine {
params.push(escaped);
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
}
// v0.31.12: scope to a single source when requested.
if (filters?.sourceId) {
params.push(filters.sourceId);
where.push(`p.source_id = $${params.length}`);
}
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
if (filters?.includeDeleted !== true) {
where.push('p.deleted_at IS NULL');
@@ -679,6 +684,18 @@ export class PGLiteEngine implements BrainEngine {
return new Set((rows as { slug: string }[]).map(r => r.slug));
}
async listAllPageRefs(): Promise<Array<{ slug: string; source_id: string }>> {
// v0.32.8: see postgres-engine.ts:listAllPageRefs for context. ORDER BY
// (source_id, slug) for determinism; WHERE deleted_at IS NULL matches
// default page visibility.
const { rows } = await this.db.query(
`SELECT slug, source_id FROM pages
WHERE deleted_at IS NULL
ORDER BY source_id, slug`
);
return (rows as { slug: string; source_id: string }[]).map(r => ({ slug: r.slug, source_id: r.source_id }));
}
async resolveSlugs(partial: string): Promise<string[]> {
// Try exact match first
const exact = await this.db.query('SELECT slug FROM pages WHERE slug = $1', [partial]);
@@ -1253,7 +1270,7 @@ export class PGLiteEngine implements BrainEngine {
async listStaleChunks(): Promise<StaleChunkRow[]> {
const { rows } = await this.db.query(
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count
cc.model, cc.token_count, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
@@ -3118,14 +3135,23 @@ export class PGLiteEngine implements BrainEngine {
await this.db.exec(sql);
}
async getChunksWithEmbeddings(slug: string): Promise<Chunk[]> {
const { rows } = await this.db.query(
`SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = $1
ORDER BY cc.chunk_index`,
[slug]
);
async getChunksWithEmbeddings(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]> {
const sourceId = opts?.sourceId;
const { rows } = sourceId
? await this.db.query(
`SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = $1 AND p.source_id = $2
ORDER BY cc.chunk_index`,
[slug, sourceId]
)
: await this.db.query(
`SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = $1
ORDER BY cc.chunk_index`,
[slug]
);
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r, true));
}

View File

@@ -559,7 +559,7 @@ export class PostgresEngine implements BrainEngine {
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`;
const rows = await sql`
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
FROM pages
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
LIMIT 1
@@ -611,7 +611,7 @@ export class PostgresEngine implements BrainEngine {
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename),
chunker_version = COALESCE(EXCLUDED.chunker_version, pages.chunker_version),
source_path = COALESCE(EXCLUDED.source_path, pages.source_path)
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename
`;
return rowToPage(rows[0]);
}
@@ -685,6 +685,10 @@ export class PostgresEngine implements BrainEngine {
const slugCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
// v0.31.12: scope to a single source when requested.
const sourceCondition = filters?.sourceId
? sql`AND p.source_id = ${filters.sourceId}`
: sql``;
// v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted.
const deletedCondition = filters?.includeDeleted === true
? sql``
@@ -698,7 +702,7 @@ export class PostgresEngine implements BrainEngine {
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${sourceCondition} ${deletedCondition}
ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset}
`;
@@ -716,6 +720,20 @@ export class PostgresEngine implements BrainEngine {
return new Set(rows.map((r) => r.slug as string));
}
async listAllPageRefs(): Promise<Array<{ slug: string; source_id: string }>> {
// v0.32.8: cross-source page enumeration. ORDER BY (source_id, slug) for
// deterministic iteration (F11) — same-slug-different-source pages stay
// grouped predictably. WHERE deleted_at IS NULL matches default getPage
// visibility semantics (v0.26.5).
const sql = this.sql;
const rows = await sql`
SELECT slug, source_id FROM pages
WHERE deleted_at IS NULL
ORDER BY source_id, slug
`;
return rows.map((r) => ({ slug: r.slug as string, source_id: r.source_id as string }));
}
async resolveSlugs(partial: string): Promise<string[]> {
const sql = this.sql;
@@ -1233,7 +1251,7 @@ export class PostgresEngine implements BrainEngine {
const sql = this.sql;
const rows = await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count
cc.model, cc.token_count, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
@@ -3101,14 +3119,22 @@ export class PostgresEngine implements BrainEngine {
await conn.unsafe(sqlStr);
}
async getChunksWithEmbeddings(slug: string): Promise<Chunk[]> {
async getChunksWithEmbeddings(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]> {
const conn = this.sql;
const rows = await conn`
SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = ${slug}
ORDER BY cc.chunk_index
`;
const sourceId = opts?.sourceId;
const rows = sourceId
? await conn`
SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
ORDER BY cc.chunk_index
`
: await conn`
SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = ${slug}
ORDER BY cc.chunk_index
`;
return rows.map((r) => rowToChunk(r as Record<string, unknown>, true));
}

View File

@@ -95,6 +95,18 @@ export interface Page {
* surface in `get_recent_salience`.
*/
salience_touched_at?: Date | null;
/**
* v0.31.12: source that owns this page. Populated by rowToPage from the
* `source_id` column so callers like `embed` can thread it through
* getChunks / upsertChunks without defaulting to 'default'.
*
* v0.32.8: required. The DB column is `NOT NULL DEFAULT 'default'`, so
* `rowToPage` always returns it from the engine. Callers can now thread
* `page.source_id` directly without `!` non-null assertions.
*
* Test fixtures building synthetic Page rows must include this field.
*/
source_id: string;
}
export type EffectiveDateSource =
@@ -178,6 +190,12 @@ export interface PageFilters {
* Whitelisted enum — no SQL-injection risk; engines map to literal SQL fragments.
*/
sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug';
/**
* v0.31.12: filter to a specific source. When omitted, listPages returns
* pages from all sources (pre-existing semantics). Use to scope embed/extract
* operations to a single source.
*/
sourceId?: string;
}
/** v0.26.5 — opts for getPage / softDeletePage / restorePage. */
@@ -320,6 +338,8 @@ export interface StaleChunkRow {
chunk_source: 'compiled_truth' | 'timeline';
model: string | null;
token_count: number | null;
/** v0.31.12: source_id so embed --stale can thread it through getChunks/upsertChunks. */
source_id: string;
}
export interface ChunkInput {

View File

@@ -43,6 +43,23 @@ export function contentHash(page: PageInput): string {
.digest('hex');
}
/**
* v0.32.8: validate a `source_id` is safe for use as a filesystem path
* segment AND as a SQL identifier value. Used by the per-source disk-layout
* fix in patterns.ts/synthesize.ts before any `join(brainDir, source_id, ...)`
* call, and at `putSource()` time so invalid ids never make it into the DB.
*
* Allows lowercase ASCII letters, digits, underscore, and hyphen. Rejects
* `..`, `/`, spaces, dots, and any non-ASCII character. Path-traversal and
* SQL-injection safe by construction.
*/
const SOURCE_ID_RE = /^[a-z0-9_-]+$/;
export function validateSourceId(id: string): void {
if (!SOURCE_ID_RE.test(id)) {
throw new Error(`Invalid source_id "${id}" — must match ${SOURCE_ID_RE}`);
}
}
function readOptionalDate(raw: unknown): Date | null | undefined {
// Three-state read for columns that may or may not be in the SELECT
// projection: undefined (not selected), null (selected, NULL value),
@@ -77,6 +94,14 @@ export function rowToPage(row: Record<string, unknown>): Page {
...(effectiveDateSource !== undefined && { effective_date_source: effectiveDateSource }),
...(importFilename !== undefined && { import_filename: importFilename }),
...(salienceTouchedAt !== undefined && { salience_touched_at: salienceTouchedAt }),
// v0.31.12: propagate source_id so downstream callers (embed, reconcile-links)
// can thread it through getChunks / upsertChunks without defaulting to 'default'.
// v0.32.8: Page.source_id is required. Every SELECT feeding rowToPage now
// projects the column (enforced by scripts/check-source-id-projection.sh).
// Fail-loud default to 'default' if the row genuinely lacks it (would mean
// an upstream caller bypassed the projection check; better to surface than
// silently mis-attribute).
source_id: (row.source_id as string | undefined) ?? 'default',
};
}

View File

@@ -56,8 +56,8 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', ()
VALUES (1001, 0, 'tool_a', 'brain_put_page', 'complete', $1::jsonb)`,
[JSON.stringify({ slug: 'wiki/agents/test/normal-shape', body: 'hi' })],
);
const slugs = await collectChildPutPageSlugs(engine as any, [1001], new Map());
expect(slugs).toContain('wiki/agents/test/normal-shape');
const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map());
expect(refs.map((r: { slug: string }) => r.slug)).toContain('wiki/agents/test/normal-shape');
});
test('recovers slug from DOUBLE-ENCODED jsonb string (#745 fix)', async () => {
@@ -81,12 +81,13 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', ()
);
expect(probe.rows[0].t).toBe('string');
const slugs = await collectChildPutPageSlugs(engine as any, [1002], new Map());
expect(slugs).toContain('wiki/agents/test/double-encoded');
const refs = await collectChildPutPageSlugs(engine as any, [1002], new Map());
expect(refs.map((r: { slug: string }) => r.slug)).toContain('wiki/agents/test/double-encoded');
});
test('handles MIXED inputs: returns slugs from both shapes in one query', async () => {
const slugs = await collectChildPutPageSlugs(engine as any, [1001, 1002], new Map());
const refs = await collectChildPutPageSlugs(engine as any, [1001, 1002], new Map());
const slugs = refs.map((r: { slug: string }) => r.slug);
expect(slugs).toContain('wiki/agents/test/normal-shape');
expect(slugs).toContain('wiki/agents/test/double-encoded');
});
@@ -98,8 +99,8 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', ()
VALUES (1003, 0, 'tool_c', 'brain_put_page', 'complete', $1::jsonb)`,
[JSON.stringify({ unrelated: 'no-slug' })],
);
const slugs = await collectChildPutPageSlugs(engine as any, [1003], new Map());
const refs = await collectChildPutPageSlugs(engine as any, [1003], new Map());
// Function silently drops rows whose slug resolves to null/empty.
expect(slugs).not.toContain('no-slug');
expect(refs.map((r: { slug: string }) => r.slug)).not.toContain('no-slug');
});
});

View File

@@ -41,7 +41,7 @@ describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
});
describe('dedup', () => {
test('multi-source duplicate slugs scan once, not once-per-source', async () => {
test('multi-source duplicate slugs scan ONE PER (source, slug) PAIR (v0.32.8 bug-class fix)', async () => {
const engine = getEngine();
const conn = getConn();
@@ -54,9 +54,9 @@ describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
frontmatter: {},
});
// Seed alt-source row via raw SQL — engine.putPage doesn't take a source_id,
// and we specifically need to test that DISTINCT ON (slug) collapses
// the multi-source rows into one scan.
// Seed alt-source row via raw SQL — engine.putPage's sourceId opt sets
// it but the test reads engine.putPage(slug, page) signature without
// it. Direct INSERT proves we have two real (source, slug) rows.
await conn.unsafe(`
INSERT INTO sources (id, name) VALUES ('test-source-2', 'test-source-2')
ON CONFLICT DO NOTHING
@@ -70,10 +70,11 @@ describeE2E('scanIntegrity batch parity (E2E, Postgres-only)', () => {
const batchResult = await scanIntegrity(engine, { limit: 100, batchLoad: true });
const seqResult = await scanIntegrity(engine, { limit: 100, batchLoad: false });
// Both paths must report the same number of distinct slugs scanned.
// Pre-fix: batch reported 2 (one per source row), sequential reported 1.
// v0.32.8: both paths now scan EACH (source, slug) row independently.
// Pre-fix the test pinned dedup-to-1 — that hid the bug where alt-source
// rows were silently dropped. Now batch + sequential both report 2.
expect(batchResult.pagesScanned).toBe(seqResult.pagesScanned);
expect(batchResult.pagesScanned).toBe(1);
expect(batchResult.pagesScanned).toBe(2);
});
});

View File

@@ -0,0 +1,208 @@
/**
* v0.32.8 — multi-source bug class regression suite.
*
* Pins the behaviors that were silently broken pre-v0.32.8 when a brain has
* more than one source. Pre-fix, every cycle phase and extract pass called
* slug-only engine methods inside a loop over pages and silently defaulted
* to source_id='default' for every non-default-source page.
*
* Fixture: 2 sources ('default' + 'media-corpus') with overlapping slugs.
* - people/alice exists in BOTH
* - concepts/widget exists ONLY in default
* - media/x/post-123 exists ONLY in media-corpus
*
* Coverage targets (one test per bug site):
* 1. listAllPageRefs returns one row per (slug, source_id), ordered.
* 2. extract-takes processes BOTH alice rows independently.
* 3. integrity scan covers BOTH sources.
* 4. extract-links F10 cross-source resolution:
* a. alice@media-corpus → widget falls back to default
* b. alice@media-corpus → post-123 stays in media-corpus
* c. alice@default → ghost-slug records nothing (skip)
* 5. validateSourceId rejects path-traversal attempts.
* 6. Reverse-write disk layout: .sources/<id>/<slug>.md for non-default.
*
* PGLite in-memory — no DATABASE_URL required. Canonical R3+R4 pattern.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { tmpdir } from 'node:os';
import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
import { validateSourceId } from '../../src/core/utils.ts';
import { extractTakesFromDb } from '../../src/core/cycle/extract-takes.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({} as never);
await engine.initSchema();
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
// Seed second source row. Default source is seeded by resetPgliteState.
await engine.executeRaw(
`INSERT INTO sources (id, name, config)
VALUES ('media-corpus', 'media-corpus', '{}'::jsonb)
ON CONFLICT (id) DO NOTHING`,
);
// Overlapping-slug seed.
await engine.putPage('people/alice', {
type: 'person', title: 'Alice (default)',
compiled_truth: 'Default-source alice page.',
}, { sourceId: 'default' });
await engine.putPage('people/alice', {
type: 'person', title: 'Alice (media-corpus)',
compiled_truth: 'Media-corpus alice page.',
}, { sourceId: 'media-corpus' });
// Distinct slugs per source.
await engine.putPage('concepts/widget', {
type: 'concept', title: 'Widget',
compiled_truth: 'Default-source-only widget.',
}, { sourceId: 'default' });
await engine.putPage('media/x/post-123', {
type: 'media', title: 'Post 123',
compiled_truth: 'Media-corpus-only post.',
}, { sourceId: 'media-corpus' });
});
describe('multi-source bug class', () => {
test('listAllPageRefs returns one row per (slug, source_id), ordered (F11)', async () => {
const refs = await engine.listAllPageRefs();
// 4 rows: alice@default, alice@media-corpus, widget@default, post-123@media-corpus
expect(refs.length).toBe(4);
// Sorted by (source_id, slug)
const ordered = refs.map(r => `${r.source_id}::${r.slug}`);
expect(ordered).toEqual([
'default::concepts/widget',
'default::people/alice',
'media-corpus::media/x/post-123',
'media-corpus::people/alice',
]);
});
test('getPage with sourceId picks the right (source, slug) row', async () => {
const aliceDefault = await engine.getPage('people/alice', { sourceId: 'default' });
const aliceMedia = await engine.getPage('people/alice', { sourceId: 'media-corpus' });
expect(aliceDefault?.title).toBe('Alice (default)');
expect(aliceMedia?.title).toBe('Alice (media-corpus)');
expect(aliceDefault?.source_id).toBe('default');
expect(aliceMedia?.source_id).toBe('media-corpus');
});
test('extract-takes processes both alice pages independently', async () => {
// Re-seed with takes fences so extract-takes has something to find.
// Fence markers come from src/core/takes-fence.ts (`<!--- gbrain:takes:begin -->`).
const TAKES_BODY = `<!--- gbrain:takes:begin -->
| # | claim | kind | who | weight | since | source |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Alice founded the thing | fact | garry | 0.9 | 2024-01-01 | |
<!--- gbrain:takes:end -->`;
await engine.putPage('people/alice', {
type: 'person', title: 'Alice (default)',
compiled_truth: 'Default alice.\n' + TAKES_BODY,
}, { sourceId: 'default' });
await engine.putPage('people/alice', {
type: 'person', title: 'Alice (media-corpus)',
compiled_truth: 'Media alice.\n' + TAKES_BODY,
}, { sourceId: 'media-corpus' });
const result = await extractTakesFromDb(engine, {});
// Pre-fix: only one of the two alice rows had takes extracted because
// the loop matched by bare slug. Post-fix: both rows get processed.
// Each alice has 1 take; widget + post-123 have none.
expect(result.pagesWithTakes).toBe(2);
expect(result.takesUpserted).toBe(2);
});
test('listPages filters correctly with PageFilters.sourceId', async () => {
const onlyDefault = await engine.listPages({ sourceId: 'default', limit: 100 });
const onlyMedia = await engine.listPages({ sourceId: 'media-corpus', limit: 100 });
expect(onlyDefault.length).toBe(2);
expect(onlyMedia.length).toBe(2);
for (const p of onlyDefault) expect(p.source_id).toBe('default');
for (const p of onlyMedia) expect(p.source_id).toBe('media-corpus');
});
test('addLinksBatch with from/to_source_id targets the right rows (F4)', async () => {
// Link alice@media-corpus → post-123@media-corpus (in-source link).
const inserted = await engine.addLinksBatch([
{
from_slug: 'people/alice',
to_slug: 'media/x/post-123',
link_type: 'mentions',
link_source: 'markdown',
from_source_id: 'media-corpus',
to_source_id: 'media-corpus',
},
]);
expect(inserted).toBe(1);
const links = await engine.getLinks('people/alice', { sourceId: 'media-corpus' });
expect(links.length).toBe(1);
expect(links[0].to_slug).toBe('media/x/post-123');
// alice@default should have NO links — they're scoped to media-corpus.
const defaultLinks = await engine.getLinks('people/alice', { sourceId: 'default' });
expect(defaultLinks.length).toBe(0);
});
test('validateSourceId rejects path traversal (F6)', () => {
// Allowed
expect(() => validateSourceId('default')).not.toThrow();
expect(() => validateSourceId('media-corpus')).not.toThrow();
expect(() => validateSourceId('jarvis_memory')).not.toThrow();
expect(() => validateSourceId('abc123')).not.toThrow();
// Rejected
expect(() => validateSourceId('..')).toThrow();
expect(() => validateSourceId('../etc')).toThrow();
expect(() => validateSourceId('foo/bar')).toThrow();
expect(() => validateSourceId('foo bar')).toThrow();
expect(() => validateSourceId('Default')).toThrow(); // uppercase
expect(() => validateSourceId('.hidden')).toThrow();
expect(() => validateSourceId('')).toThrow();
});
test('reverse-write disk layout uses .sources/<id>/<slug>.md for non-default (F6)', () => {
// Pure-function test of the disk-path computation embedded in
// reverseWriteRefs (patterns.ts + synthesize.ts). We don't drive the
// full dream cycle here — that requires LLM + subagent infra. Instead
// we replicate the path computation and verify the file layout matches
// what the production code would write.
const tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-multi-source-disk-'));
try {
const computePath = (source_id: string, slug: string): string =>
source_id === 'default'
? join(tmpDir, `${slug}.md`)
: join(tmpDir, '.sources', source_id, `${slug}.md`);
const defaultPath = computePath('default', 'people/alice');
const mediaPath = computePath('media-corpus', 'people/alice');
// The two paths must NOT collide.
expect(defaultPath).not.toBe(mediaPath);
expect(mediaPath).toContain('.sources/media-corpus/');
// Actually write to both paths to prove disk separation.
mkdirSync(join(tmpDir, 'people'), { recursive: true });
mkdirSync(join(tmpDir, '.sources', 'media-corpus', 'people'), { recursive: true });
writeFileSync(defaultPath, 'default alice');
writeFileSync(mediaPath, 'media-corpus alice');
expect(readFileSync(defaultPath, 'utf8')).toBe('default alice');
expect(readFileSync(mediaPath, 'utf8')).toBe('media-corpus alice');
expect(existsSync(defaultPath)).toBe(true);
expect(existsSync(mediaPath)).toBe(true);
} finally {
rmSync(tmpDir, { recursive: true, force: true });
}
});
});

View File

@@ -218,6 +218,7 @@ describe('scorePage — person', () => {
compiled_truth: '', timeline: '',
frontmatter: {},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.score).toBeLessThan(0.3);
@@ -236,6 +237,7 @@ See also: [Acme](companies/acme.md), [Bob](people/bob.md), [Charlie](people/char
last_verified: new Date().toISOString(),
},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.score).toBeGreaterThan(0.8);
@@ -248,6 +250,7 @@ See also: [Acme](companies/acme.md), [Bob](people/bob.md), [Charlie](people/char
timeline: '',
frontmatter: {},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(Object.keys(s.dimensionScores)).toHaveLength(7);
@@ -259,6 +262,7 @@ See also: [Acme](companies/acme.md), [Bob](people/bob.md), [Charlie](people/char
compiled_truth: '', timeline: '',
frontmatter: { role: 'Engineer' },
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.dimensionScores.has_role_and_company).toBe(1);
@@ -273,6 +277,7 @@ describe('scorePage — company / concept / source / media defaults', () => {
timeline: '',
frontmatter: { founders: ['Alice'], funding: '$5M' },
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.rubric).toBe('company');
@@ -286,6 +291,7 @@ describe('scorePage — company / concept / source / media defaults', () => {
compiled_truth: 'body content',
timeline: '', frontmatter: {},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.rubric).toBe('default');
@@ -296,11 +302,13 @@ describe('scorePage — company / concept / source / media defaults', () => {
id: 1, slug: 'people/old', type: 'person', title: 'x',
compiled_truth: '', timeline: '', frontmatter: {},
created_at: new Date(2020, 0, 1), updated_at: new Date(2020, 0, 1),
source_id: 'default',
};
const fresh: Page = {
id: 2, slug: 'people/fresh', type: 'person', title: 'y',
compiled_truth: '', timeline: '', frontmatter: {},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const oldScore = scorePage(old);
const freshScore = scorePage(fresh);
@@ -313,6 +321,7 @@ describe('scorePage — company / concept / source / media defaults', () => {
id: 1, slug: 'people/repetitive', type: 'person', title: 'x',
compiled_truth: repeated, timeline: '', frontmatter: {},
created_at: new Date(), updated_at: new Date(),
source_id: 'default',
};
const s = scorePage(page);
expect(s.dimensionScores.non_redundancy).toBeLessThan(0.2);