699db50a3d
* feat(engine): add addLinksBatch + addTimelineEntriesBatch via unnest()
Multi-row INSERT...SELECT FROM unnest() JOIN pages ON CONFLICT DO NOTHING
RETURNING 1. 4 array-typed bound parameters (links) or 5 (timeline)
regardless of batch size, sidesteps Postgres's 65535-parameter cap.
Returns count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist).
Per-row addLink / addTimelineEntry signatures and SQL behavior unchanged.
All 10 existing call sites compile and behave identically.
Tests: 11 PGLite cases (empty batch, missing optionals, within-batch dedup,
JOIN drops missing slug, half-existing batch, batch of 100) + 9 E2E
postgres-engine cases against real Postgres+pgvector.
* fix(migrate): pre-create btree helper in v8 + v9 dedup; bump phaseASchema timeout
Production bug: v0.12.0 schema migration timed out at Supabase Management API's
60s ceiling on brains with 80K+ duplicate timeline rows. The DELETE...USING
self-join was O(n²) without an index on the dedup columns.
Fix: pre-create idx_links_dedup_helper / idx_timeline_dedup_helper on the
dedup columns BEFORE the DELETE, drop after. Turns O(n²) into O(n log n).
On 80K+ rows the migration completes in <1s instead of timing out.
Also bumps the v0.12.0 orchestrator's phaseASchema timeout 60s -> 600s as
belt-and-suspenders for unforeseen slowness.
Exports MIGRATIONS for structural test assertions.
Tests: 2 structural assertions (helper-index DDL must appear in v8/v9 SQL
in the right order — catches regression even at 0-row scale) + 2 behavioral
regression tests (1000-row dedup completes <5s).
* perf(extract): kill N+1 dedup pre-load; switch to batched writes
Production bug: gbrain extract hung 10+ minutes producing zero output on
47K-page brains. The pre-load loop called engine.getLinks(slug) (or
getTimeline) once per page across engine.listPages({limit: 100000}) — 47K
serial round-trips over the Supabase pooler before the first file was read.
Both engines already enforced uniqueness at the SQL layer
(UNIQUE(from, to, link_type) on links, idx_timeline_dedup on timeline_entries).
The in-memory dedup Set was redundant insurance that became the bottleneck.
Fix: delete the pre-load entirely. Buffer 100 candidates per file walk,
flush via engine.addLinksBatch / engine.addTimelineEntriesBatch. ~99% fewer
DB round-trips per re-extract.
Also fixes counter accuracy: 'created' now counts rows actually inserted
(via batch RETURNING 1 row count). Re-run on a fully-extracted brain
prints 'Done: 0 links' instead of lying.
Dry-run mode keeps a per-run dedup Set so duplicate candidates from N
markdown files print exactly once, not N times.
Batch errors are visible in BOTH json and human modes — silent loss of
100 rows is worse than per-row error visibility.
Tests: extract-fs.test.ts (idempotency + truthful counter + dry-run dedup
+ perf regression guard <2s).
* chore: bump version + changelog (v0.12.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md for v0.12.1 (batch engine API, test counts)
Reflect what shipped in v0.12.1:
- New engine methods addLinksBatch + addTimelineEntriesBatch (PGLite via
unnest() + manual $N, postgres-engine via INSERT...SELECT FROM
unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING).
- extract.ts no longer pre-loads dedup set; candidates are buffered 100
at a time and flushed via the new batch methods.
- v0.12.0 orchestrator phaseASchema timeout bumped 60s to 600s.
- Test counts 1297 unit / 105 E2E to 1412 unit / 119 E2E.
- New test/extract-fs.test.ts covers the N+1 regression guard.
- BrainEngine method count 37/38 to 40.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4.0 KiB
v0.12.1 Migration: Extract Performance + Migration Timeout Fix
This release is a pure performance bug fix. No manual steps are needed for most
users — re-run gbrain init --migrate-only and re-run gbrain extract all if
desired. Both should now complete successfully on any brain size.
What changed
Two production-blocking bugs are fixed:
-
gbrain extractno longer hangs on large brains. The N+1 dedup pre-load that ran 47K serialgetLinks()calls before any work started is gone. Both engines already enforced uniqueness at the SQL layer; the in-memory dedup was redundant. Combined with new batched 100-row INSERTs, a full re-extract on a 47K-page brain drops from "10+ min hang then ~minutes more" to "immediate work, ~30-60s total." -
v0.12.0 schema migration no longer times out on duplicate-heavy brains. Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create a btree helper index before the
DELETE ... USINGself-join, then drop it after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the migration completes in under a second instead of timing out at 60 seconds.
What you need to do
If your v0.12.0 upgrade succeeded — you're already done
The migration is idempotent. The fix only matters if your migration FAILED on
the v0.12.0 upgrade attempt. Run gbrain init --migrate-only once to confirm
your schema is at version 10, then run gbrain extract all --dir <brain> if
you want to reuse it now that it's fast.
If your v0.12.0 upgrade FAILED on idx_timeline_dedup creation
You may have the brain in a partial-migration state with duplicate rows in
timeline_entries (or links). Run:
gbrain init --migrate-only
Migration v9 will pre-create the helper index, dedup any existing duplicates (now sub-second instead of timing out), drop the helper, and create the unique index. Idempotent — safe to re-run.
If you previously ran the manual CREATE TABLE _clean AS SELECT DISTINCT ON ... + table swap workaround Garry posted, your schema should already be at
version 10. Confirm with:
gbrain config get version
If it shows 10, you're done.
If you previously did manual SQL surgery on duplicate timeline rows
The unique index idx_timeline_dedup should now be present after the workaround.
Re-running gbrain init --migrate-only is a no-op for v9 (uses
CREATE UNIQUE INDEX IF NOT EXISTS). The new migration code adds the helper
btree on the dedup columns first — but the helper is dropped at the end of the
migration, so even if v9 re-ran (it won't, version is already at 10), it would
leave your schema unchanged.
Re-running gbrain extract on a previously-stuck brain
This is the most common case. Run:
gbrain extract all --dir <brain-dir>
# or for live brains with no local checkout:
gbrain extract all --source db
Expect immediate output (Links: created N from M pages lines streaming as files
process), not a 10-minute hang. On a re-run of a fully-extracted brain you should
see Done: 0 links, 0 timeline entries from N pages — that's the truthful counter
at work, confirming nothing changed.
New engine API (informational, optional)
For plugin authors building integrations on the BrainEngine interface, two
new methods are available:
addLinksBatch(LinkBatchInput[]) → Promise<number>addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>
Both return the count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist). Existing per-row addLink /
addTimelineEntry are unchanged — no migration required for plugin code.
Verification
After the migration completes:
# Confirm schema version
gbrain config get version
# Expect: 10
# Confirm the unique indexes exist (Postgres / Supabase only)
gbrain stats
# Expect: link_count and timeline_entry_count both populated, no duplicates
If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the
output of gbrain init --migrate-only and gbrain config get version.