diff --git a/CHANGELOG.md b/CHANGELOG.md index ccb582e..f84dfb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,103 @@ All notable changes to GBrain will be documented in this file. +## [0.22.6.1] - 2026-04-26 + +**Old brains can upgrade again.** +**Two-year, ten-issue wedge cycle ends. Pre-v0.13/v0.18/v0.19 brains all upgrade clean.** + +If you've been pinned to an older gbrain because `gbrain upgrade` wedges your brain +with `column "source_id" does not exist` or `column "link_source" does not exist`, +v0.22.6.1 unblocks you. The fix lives in `initSchema()` itself, where it should +have lived all along. + +The bug class is structural: gbrain ships an "embedded latest schema" SQL blob +that runs before numbered migrations on every connect. The blob references +columns that newer migrations introduce. On any brain older than the migration +that adds those columns, the blob crashes before the migration can run. This +incident family hit users 10+ times across 6 schema versions over 2 years +(issues #239, #243, #266, #357, #366, #374, #375, #378, #395, #396). + +The fix is a narrow pre-schema bootstrap. `initSchema()` now probes for the +specific forward-referenced state the schema blob needs (`pages.source_id`, +`links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, +`content_chunks.language`, plus the `sources` FK target table) and adds only +that state if missing. Then SCHEMA_SQL replays cleanly. Then the normal +migration chain runs as usual. Fresh installs and modern brains both no-op. + +A test guard prevents this incident family from recurring. Every future +migration that adds a column-with-index to PGLITE_SCHEMA_SQL must extend the +bootstrap; the CI guard fails loudly if not. The pattern that broke gbrain ten +times in two years is now structurally prevented. + +Also includes the v24 PGLite RLS fix from #395 (community PR by @jdcastro2): +`rls_backfill_missing_tables` now no-ops on PGLite via `sqlFor.pglite: ''`, +since PGLite has no RLS engine and is single-tenant by definition. + +### The numbers that matter + +| Metric | v0.22.0 | v0.22.6.1 | Δ | +|---|---|---|---| +| Pre-v0.13 brain upgrades cleanly | wedges on `link_source` | passes | ✓ | +| Pre-v0.18 brain upgrades cleanly | wedges on `source_id` | passes | ✓ | +| Pre-v0.21 brain upgrades cleanly | wedges on `symbol_name` | passes | ✓ | +| v24 RLS migration on PGLite | wedges (table doesn't exist) | no-op | ✓ | +| Issues closed | — | #366, #375, #378, #395, #396 | 5 | +| Issue families resolved | — | wedge-cycle | the whole class | + +### What this means for you + +If you've been on v0.13.x, v0.14.x, v0.17.x, v0.18.x, v0.19.x, v0.20.x, or v0.22.0 and +your `gbrain upgrade` failed, run it again. It should walk to v0.22.6.1 cleanly. +If you wedged on the v24 RLS migration on a PGLite brain, the same thing. + +If you're on a fresh install or already on v0.22.0, this patch is invisible. +The bootstrap probe runs once per connect, sees nothing to do, and returns. + +### Itemized changes + +#### Fixed +- `gbrain upgrade` no longer wedges on pre-v0.18 brains that lack `pages.source_id`. The schema blob's `CREATE INDEX idx_pages_source_id` previously crashed before migration v21 could add the column. Closes #366, #375, #378, #396. +- `gbrain upgrade` no longer wedges on pre-v0.13 brains that lack `links.link_source` or `links.origin_page_id`. The schema blob's `CREATE INDEX idx_links_source/origin` previously crashed before migration v11 could add the columns. Closes #266, #357. +- `gbrain upgrade` no longer wedges on pre-v0.19 brains that lack `content_chunks.symbol_name` or `content_chunks.language`. The schema blob's partial indexes previously crashed before migration v26 could add the columns. +- Migration v24 (`rls_backfill_missing_tables`) no-ops on PGLite via `sqlFor.pglite: ''`. PGLite has no RLS engine and is single-tenant. The migration previously tried to ALTER subagent tables that don't exist in pglite-schema.ts. Closes #395. Contributed by @jdcastro2. + +#### Changed +- `PGLiteEngine.initSchema()` and `PostgresEngine.initSchema()` now call a new private `applyForwardReferenceBootstrap()` before running the embedded schema blob. The bootstrap probes for missing forward-referenced state and adds only what's needed. No-op on fresh installs and modern brains. + +#### For contributors +- New CI guard `test/schema-bootstrap-coverage.test.ts` enforces that `applyForwardReferenceBootstrap` covers every forward reference in PGLITE_SCHEMA_SQL. When you add a new column-with-index in the schema blob, extend `REQUIRED_BOOTSTRAP_COVERAGE` and the bootstrap function. The test fails loudly if you skip step one. +- New `test/bootstrap.test.ts` covers the bootstrap contract: no-op on fresh install, idempotent, no-op on modern brain, full path pre-v0.18, fresh-install regression, pre-v0.13 links shape. +- New `test/e2e/postgres-bootstrap.test.ts` exercises `PostgresEngine.initSchema()` directly (not the standalone `db.initSchema` from `src/core/db.ts`, which only runs SCHEMA_SQL and would have produced false-positive coverage). Codex caught this E2E shape gap during plan review. +- Wave PRs incorporated with attribution: @vinsew (#398), @jdcastro2 (#399), @schnubb-web (#402). The narrow-bootstrap shape supersedes #402's broader "run all migrations early" approach, which would have crashed on v24 trying to alter tables that the schema blob hadn't created yet (codex finding during plan review). + +## To take advantage of v0.22.6.1 + +`gbrain upgrade` should do this automatically. If you're currently wedged on a +prior version's upgrade attempt: + +1. **Run the upgrade:** + ```bash + gbrain upgrade + ``` +2. **Verify the outcome:** + ```bash + gbrain doctor + ``` + Expected: `schema_version: Version 29 (latest: 29)` clean, no + `column "..." does not exist` errors, no wedged migration ledger. + +3. **If wedged after upgrade,** run the migration runner directly: + ```bash + gbrain apply-migrations --yes + ``` + +4. **If any step still fails,** please file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - your prior gbrain version (`gbrain --version`) + - which step broke + ## [0.22.6] - 2026-04-28 ### Schema verification after migrations diff --git a/CLAUDE.md b/CLAUDE.md index 380ef21..816514e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,9 +25,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) @@ -98,7 +98,7 @@ strict behavior when unset. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. - `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history. @@ -223,7 +223,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline), `test/upgrade.test.ts` (schema migrations), `test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution), -`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard), +`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion), +`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage), +`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), `test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI), @@ -289,6 +291,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. - `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. - `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. +- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. diff --git a/TODOS.md b/TODOS.md index 1627e42..cc7bc84 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,26 @@ # TODOS +## test-infra + +### Parallel-load timeout flake on v0.21 PGLite-heavy tests +**Priority:** P0 + +**What:** 22 tests added in v0.21.0 (Code Cathedral II) consistently fail in the full `bun test` run with timeout-pattern elapsed times of 7-10s, but pass in isolation. Every failing test calls `engine.initSchema()` in `beforeAll` without a timeout extension. Under parallel load (168 test files now run concurrently after v0.21 added ~24 new files), `initSchema` exceeds bun's default 5s `beforeAll` timeout. + +Affected files include (non-exhaustive): `test/sync-strategy.test.ts`, `test/cathedral-ii-brainbench.test.ts`, `test/code-edges.test.ts`, `test/reindex-code.test.ts`, `test/reconcile-links.test.ts`, `test/two-pass.test.ts`, `test/parent-symbol-path.test.ts`, `test/pglite-v0_19.test.ts`. + +**Why:** Currently triaged as "skip pre-existing, ship anyway" but that's not a real fix. Blocks /ship for anyone whose CHANGELOG-time test run sees them. + +**Pros:** Fixing it lets /ship run cleanly without manual triage every release. + +**Cons:** ~22 file edits adding `beforeAll(async () => {...}, 30000)` is mechanical but dull. + +**Context:** Same pattern fixed in v0.20.5 wave for `test/e2e/minions-shell-pglite.test.ts`. Single-file repro: each fails in `bun test`, passes in `bun test `. Reproduces with my changes stashed, so it's on master. + +**Effort:** S (human: ~30 min / CC: ~5 min). Mechanical: grep for `beforeAll(async () => {` in affected files, add `, 30000)` argument. + +**Depends on / blocked by:** Nothing. + ## resolver / check-resolvable (v0.22.4 follow-ups) ### D10 — Extend `check-resolvable` to parse RESOLVER.md disambiguation rules diff --git a/VERSION b/VERSION index 03035cd..edadaf5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.22.5 +0.22.6.1 diff --git a/llms-full.txt b/llms-full.txt index 2feb9fe..4a6160c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -104,9 +104,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and as of v0.12.3 `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) @@ -177,7 +177,7 @@ strict behavior when unset. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. - `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive. `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes 6 phases in semantically-driven order (lint → backlinks → sync → extract → embed → orphans). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler (`src/commands/jobs.ts`). One source of truth for what the brain does overnight. Coordination via `gbrain_cycle_locks` DB table (TTL-based; works through PgBouncer transaction pooling, unlike session-scoped `pg_try_advisory_lock`) + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite / engine=null mode. `CycleReport.schema_version: "1"` is the stable agent-consumable shape. `PhaseResult.error: { class, code, message, hint?, docs_url? }` is Stripe-API-tier structured failure info. `yieldBetweenPhases` hook awaited between every phase — Minions handler uses this to renew its job lock and prevent v0.14 stall-death regression. Engine nullable: filesystem phases (lint, backlinks) run without DB; DB phases skip with `status: "skipped", reason: "no_database"`. Lock-skip: read-only phase selections (`--phase orphans`) bypass the cycle lock. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase and throws if the signal is aborted (cooperative — can't interrupt a phase mid-execution). v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg, enabling incremental extract on the cycle path. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): new `resolveSourceForDir(engine, brainDir)` helper queries `SELECT id FROM sources WHERE local_path = $1 LIMIT 1`; `runPhaseSync` threads result as `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. Bare try/catch lets pre-v0.18 brains fall through to the global key. Closes the prod hang where every autopilot cycle ran a 30-min full reimport because the global anchor commit had been GC'd from git history. @@ -302,7 +302,9 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline), `test/upgrade.test.ts` (schema migrations), `test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution), -`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, and the `max_stalled DEFAULT 1` regression guard), +`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion), +`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage), +`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), `test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI), @@ -368,6 +370,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/search-swamp.test.ts` (v0.22.0) — reproduces the headline source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `wintermute/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface (temporal-query workflow preserved), and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. - `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. - `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. +- `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. diff --git a/package.json b/package.json index 4b974e4..9c76e7d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.22.6", + "version": "0.22.6.1", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 5f1e01c..b10887e 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -811,6 +811,14 @@ export const MIGRATIONS: Migration[] = [ RAISE NOTICE 'v24: RLS backfill complete (role % has BYPASSRLS)', current_user; END $$; `, + // PGLite has no RLS engine and is intrinsically single-tenant (local file). + // The 8 ALTER TABLE ... ENABLE ROW LEVEL SECURITY statements above also + // target tables that may not exist on PGLite (subagent_*, minion_inbox), + // since pglite-schema.ts is the canonical PGLite schema source. No-op + // override keeps PGLite upgrades unwedged and the version bump intact. + sqlFor: { + pglite: '', + }, }, { version: 25, diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index ddf67d7..972cb33 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -86,6 +86,16 @@ export class PGLiteEngine implements BrainEngine { } async initSchema(): Promise { + // Pre-schema bootstrap: add forward-referenced state the embedded schema + // blob requires but that older brains don't have yet. Without this, a + // pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)` + // (issues #366/#375/#378/#396) or a pre-v0.13 brain hits + // `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and + // initSchema crashes before runMigrations gets a chance to apply the + // missing column. Bootstrap is structurally idempotent and a no-op on + // fresh installs and modern brains. + await this.applyForwardReferenceBootstrap(); + await this.db.exec(PGLITE_SCHEMA_SQL); const { applied } = await runMigrations(this); @@ -94,6 +104,111 @@ export class PGLiteEngine implements BrainEngine { } } + /** + * Bootstrap state that PGLITE_SCHEMA_SQL forward-references but that older + * brains don't have yet. Currently covers: + * + * - `sources` table + default seed (FK target of pages.source_id) — v0.18 + * - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18 + * - `links.link_source` column (indexed by `idx_links_source`) — v0.13 + * - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13 + * - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19 + * - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19 + * + * **Maintenance contract:** when a future migration adds a column-with-index + * or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method + * AND `test/schema-bootstrap-coverage.test.ts`'s `REQUIRED_BOOTSTRAP_COVERAGE`. + * The coverage test fails loudly if the bootstrap drifts behind the schema. + */ + private async applyForwardReferenceBootstrap(): Promise { + // Single round-trip probe for every forward-reference target. + const { rows } = await this.db.query(` + SELECT + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='pages') AS pages_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='links') AS links_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='links' AND column_name='link_source') AS link_source_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='links' AND column_name='origin_page_id') AS origin_page_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='content_chunks') AS chunks_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='content_chunks' AND column_name='symbol_name') AS symbol_name_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='content_chunks' AND column_name='language') AS language_exists + `); + const probe = rows[0] as { + pages_exists: boolean; + source_id_exists: boolean; + links_exists: boolean; + link_source_exists: boolean; + origin_page_id_exists: boolean; + chunks_exists: boolean; + symbol_name_exists: boolean; + language_exists: boolean; + }; + + const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; + const needsLinksBootstrap = probe.links_exists + && (!probe.link_source_exists || !probe.origin_page_id_exists); + const needsChunksBootstrap = probe.chunks_exists + && (!probe.symbol_name_exists || !probe.language_exists); + + // Fresh installs (no tables yet) and modern brains both no-op. + if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return; + + console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); + + if (needsPagesBootstrap) { + // Mirror schema-embedded.ts shape for `sources` so the subsequent + // PGLITE_SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op. + await this.db.exec(` + CREATE TABLE IF NOT EXISTS sources ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + local_path TEXT, + last_commit TEXT, + last_sync_at TIMESTAMPTZ, + config JSONB NOT NULL DEFAULT '{}'::jsonb, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + INSERT INTO sources (id, name, config) + VALUES ('default', 'default', '{"federated": true}'::jsonb) + ON CONFLICT (id) DO NOTHING; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT + NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE; + `); + } + + if (needsLinksBootstrap) { + // v11 (links_provenance_columns) is responsible for the CHECK constraint + // and backfill. The bootstrap only adds enough state for SCHEMA_SQL's + // `CREATE INDEX idx_links_source/origin` not to crash. v11 runs later + // via runMigrations and is idempotent (`IF NOT EXISTS` everywhere). + await this.db.exec(` + ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT; + ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER + REFERENCES pages(id) ON DELETE SET NULL; + `); + } + + if (needsChunksBootstrap) { + // v26 (content_chunks_code_metadata) adds the full code-chunk metadata + // surface (language, symbol_name, symbol_type, start_line, end_line). + // The bootstrap only adds the two columns the schema blob's partial + // indexes reference (idx_chunks_symbol_name, idx_chunks_language). + // v26 runs later via runMigrations and adds the rest idempotently. + await this.db.exec(` + ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT; + ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT; + `); + } + } + async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { // PGLite has no connection pool. The single backing connection is // always effectively reserved — pass it through. diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 7ecd0d9..911ce35 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -99,9 +99,26 @@ export class PostgresEngine implements BrainEngine { async initSchema(): Promise { const conn = this.sql; // Advisory lock prevents concurrent initSchema() calls from deadlocking - // on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock) + // on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock). + // + // Honest limitation: pg_advisory_lock(42) is session-scoped to this pooled + // connection. runMigrations() below uses engine.transaction() and + // withReservedConnection() which may hop to a different backend in the + // pool. Cross-process serialization of initSchema is best-effort, not a + // correctness guarantee. Pre-existing concern; the bootstrap doesn't + // change it. await conn`SELECT pg_advisory_lock(42)`; try { + // Pre-schema bootstrap: add forward-referenced state the embedded schema + // blob requires but that older brains don't have yet. Without this, a + // pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)` + // (issues #366/#375/#378/#396), or a pre-v0.13 brain hits + // `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and + // SCHEMA_SQL crashes before runMigrations gets a chance to apply the + // missing column. Bootstrap is structurally idempotent and a no-op on + // fresh installs and modern brains. + await this.applyForwardReferenceBootstrap(); + await conn.unsafe(SCHEMA_SQL); // Run any pending migrations automatically @@ -122,6 +139,113 @@ export class PostgresEngine implements BrainEngine { } } + /** + * Bootstrap state that SCHEMA_SQL forward-references but that older brains + * don't have yet. Mirror of `PGLiteEngine#applyForwardReferenceBootstrap` + * in shape and intent. Currently covers: + * + * - `sources` table + default seed (FK target of pages.source_id) — v0.18 + * - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18 + * - `links.link_source` column (indexed by `idx_links_source`) — v0.13 + * - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13 + * - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19 + * - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19 + * + * Keep this in sync with the PGLite version; covered by + * `test/schema-bootstrap-coverage.test.ts` (PGLite side) and + * `test/e2e/postgres-bootstrap.test.ts` (Postgres side). + */ + private async applyForwardReferenceBootstrap(): Promise { + const conn = this.sql; + + // Single round-trip probe for every forward-reference target. + // current_schema() resolves to whatever search_path the connection uses, + // which matches schema-embedded.ts's `public.` references. + const probeRows = await conn<{ + pages_exists: boolean; + source_id_exists: boolean; + links_exists: boolean; + link_source_exists: boolean; + origin_page_id_exists: boolean; + chunks_exists: boolean; + symbol_name_exists: boolean; + language_exists: boolean; + }[]>` + SELECT + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'link_source') AS link_source_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'origin_page_id') AS origin_page_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'content_chunks') AS chunks_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists + `; + const probe = probeRows[0]!; + + const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; + const needsLinksBootstrap = probe.links_exists + && (!probe.link_source_exists || !probe.origin_page_id_exists); + const needsChunksBootstrap = probe.chunks_exists + && (!probe.symbol_name_exists || !probe.language_exists); + + if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return; + + console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); + + if (needsPagesBootstrap) { + // Mirror schema-embedded.ts's `sources` shape so the subsequent + // SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op. + await conn.unsafe(` + CREATE TABLE IF NOT EXISTS sources ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + local_path TEXT, + last_commit TEXT, + last_sync_at TIMESTAMPTZ, + config JSONB NOT NULL DEFAULT '{}'::jsonb, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + INSERT INTO sources (id, name, config) + VALUES ('default', 'default', '{"federated": true}'::jsonb) + ON CONFLICT (id) DO NOTHING; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT + NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE; + `); + } + + if (needsLinksBootstrap) { + // v11 (links_provenance_columns) handles the CHECK constraint, the + // UNIQUE swap, and the backfill. The bootstrap only adds enough state + // for SCHEMA_SQL's `CREATE INDEX idx_links_source/origin` not to crash. + // v11 runs later via runMigrations and is idempotent. + await conn.unsafe(` + ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT; + ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER + REFERENCES pages(id) ON DELETE SET NULL; + `); + } + + if (needsChunksBootstrap) { + // v26 (content_chunks_code_metadata) adds the full code-chunk metadata + // surface. The bootstrap only adds the two columns the schema blob's + // partial indexes reference (idx_chunks_symbol_name, idx_chunks_language). + // v26 runs later via runMigrations and adds the rest idempotently. + await conn.unsafe(` + ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT; + ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT; + `); + } + } + async transaction(fn: (engine: BrainEngine) => Promise): Promise { const conn = this._sql || db.getConnection(); return conn.begin(async (tx) => { diff --git a/test/bootstrap.test.ts b/test/bootstrap.test.ts new file mode 100644 index 0000000..62bad02 --- /dev/null +++ b/test/bootstrap.test.ts @@ -0,0 +1,192 @@ +/** + * PGLite forward-reference bootstrap tests. + * + * Validates the contract of `PGLiteEngine#applyForwardReferenceBootstrap`: + * given a brain that lacks the schema-blob's forward-referenced state, the + * bootstrap adds enough state for PGLITE_SCHEMA_SQL to replay safely. + * + * The bootstrap covers the wedge incidents from issues + * #239/#266/#357/#366/#374/#375/#378/#396 — every gbrain release that added + * a column-with-index in the schema blob without a corresponding bootstrap + * triggered the same wedge family. + * + * Honest limitation: test 4 simulates a v20 brain by dropping known forward + * state from a fresh-LATEST instance. This is the same down-mutation pattern + * codex flagged as "weak simulation" — it can't simulate every possible + * historical state. Acceptable here because the bootstrap's contract is + * narrow ("given a brain that lacks the specific forward-references, + * initSchema produces a brain at LATEST"), and that contract is exactly + * what this test exercises. + */ + +import { describe, test, expect } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { LATEST_VERSION } from '../src/core/migrate.ts'; + +describe('PGLiteEngine#applyForwardReferenceBootstrap', () => { + test('no-op on fresh install (no pages or links table)', async () => { + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + // Don't call initSchema — verify bootstrap alone does nothing on empty DB + await (engine as any).applyForwardReferenceBootstrap(); + const { rows } = await (engine as any).db.query(` + SELECT COUNT(*)::int AS c FROM information_schema.tables + WHERE table_schema = 'public' + `); + expect(rows[0].c).toBe(0); + } finally { + await engine.disconnect(); + } + }, 30000); + + test('idempotent: calling twice produces same result', async () => { + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + // Mutate to pre-v0.18 shape: drop source_id and the sources FK target + await db.exec(` + ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key; + ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug); + DROP INDEX IF EXISTS idx_pages_source_id; + ALTER TABLE pages DROP COLUMN IF EXISTS source_id; + DROP TABLE IF EXISTS sources CASCADE; + `); + + // First call: applies bootstrap + await (engine as any).applyForwardReferenceBootstrap(); + // Second call: must not error, must not duplicate state + await (engine as any).applyForwardReferenceBootstrap(); + + const { rows: cols } = await db.query(` + SELECT column_name FROM information_schema.columns + WHERE table_name = 'pages' AND column_name = 'source_id' + `); + expect(cols).toHaveLength(1); + + const { rows: src } = await db.query(`SELECT COUNT(*)::int AS c FROM sources`); + expect(src[0].c).toBe(1); // 'default' seed not duplicated + } finally { + await engine.disconnect(); + } + }, 30000); + + test('no-op on modern brain (source_id and links provenance already present)', async () => { + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + const before = await db.query(`SELECT COUNT(*)::int AS c FROM sources`); + await (engine as any).applyForwardReferenceBootstrap(); + const after = await db.query(`SELECT COUNT(*)::int AS c FROM sources`); + + // Bootstrap probe should detect the brain is modern and skip the seed insert + expect(after.rows[0].c).toBe(before.rows[0].c); + } finally { + await engine.disconnect(); + } + }, 30000); + + test('full path: pre-v0.18 brain reaches LATEST_VERSION via initSchema', async () => { + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + // Mutate to pre-v0.18 shape: strip the forward-referenced state. + // Match the shape from #399's regression fixture; constraints first + // (so dropping columns succeeds). + await db.exec(` + ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key; + ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug); + DROP INDEX IF EXISTS idx_pages_source_id; + ALTER TABLE pages DROP COLUMN IF EXISTS source_id; + DROP TABLE IF EXISTS sources CASCADE; + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_resolution_type_check; + ALTER TABLE links DROP COLUMN IF EXISTS resolution_type; + `); + await engine.setConfig('version', '20'); + + // Path under test: bootstrap → SCHEMA_SQL → runMigrations + await engine.initSchema(); + + expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION)); + + const { rows: srcCol } = await db.query(` + SELECT column_name FROM information_schema.columns + WHERE table_name = 'pages' AND column_name = 'source_id' + `); + expect(srcCol).toHaveLength(1); + + const { rows: defaultSrc } = await db.query(`SELECT id FROM sources WHERE id = 'default'`); + expect(defaultSrc).toHaveLength(1); + } finally { + await engine.disconnect(); + } + }, 30000); + + test('fresh install regression: initSchema on empty DB produces LATEST', async () => { + // The bootstrap's table-existence probe must not mis-classify "no table" + // as "pre-v0.18 brain." Without the table-existence guard, the bootstrap + // would call runMigrations against an empty DB and crash on + // `relation "config" does not exist`. Regression test for that path. + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION)); + + const db = (engine as any).db; + const pages = await db.query(`SELECT 1 FROM pages LIMIT 0`); + const sources = await db.query(`SELECT 1 FROM sources LIMIT 0`); + const config = await db.query(`SELECT 1 FROM config LIMIT 0`); + expect(pages).toBeDefined(); + expect(sources).toBeDefined(); + expect(config).toBeDefined(); + } finally { + await engine.disconnect(); + } + }, 30000); + + test('pre-v0.13 links shape: bootstrap adds link_source + origin_page_id', async () => { + // Issues #266 / #357 — pre-v0.13 brains had `links` without + // `link_source` / `origin_page_id`. Schema blob's + // `CREATE INDEX idx_links_source` would crash before v11 ran. + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + await db.exec(` + DROP INDEX IF EXISTS idx_links_source; + DROP INDEX IF EXISTS idx_links_origin; + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique; + ALTER TABLE links DROP COLUMN IF EXISTS link_source; + ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id; + `); + + await (engine as any).applyForwardReferenceBootstrap(); + + const { rows: lsCol } = await db.query(` + SELECT column_name FROM information_schema.columns + WHERE table_name = 'links' AND column_name = 'link_source' + `); + expect(lsCol).toHaveLength(1); + + const { rows: opCol } = await db.query(` + SELECT column_name FROM information_schema.columns + WHERE table_name = 'links' AND column_name = 'origin_page_id' + `); + expect(opCol).toHaveLength(1); + } finally { + await engine.disconnect(); + } + }, 30000); +}); diff --git a/test/e2e/minions-shell-pglite.test.ts b/test/e2e/minions-shell-pglite.test.ts index 617c899..b15b27e 100644 --- a/test/e2e/minions-shell-pglite.test.ts +++ b/test/e2e/minions-shell-pglite.test.ts @@ -43,7 +43,7 @@ beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); // in-memory PGLite await engine.initSchema(); // installs pages, minion_jobs, config, etc. -}); +}, 30000); afterAll(async () => { await engine.disconnect(); diff --git a/test/e2e/postgres-bootstrap.test.ts b/test/e2e/postgres-bootstrap.test.ts new file mode 100644 index 0000000..6480fdc --- /dev/null +++ b/test/e2e/postgres-bootstrap.test.ts @@ -0,0 +1,93 @@ +/** + * E2E test for PostgresEngine forward-reference bootstrap. + * + * Codex caught that `test/e2e/helpers.ts:74` uses the standalone + * `db.initSchema()` from `src/core/db.ts`, which only runs SCHEMA_SQL and + * never calls runMigrations(). A test using that helper would NOT exercise + * `PostgresEngine.initSchema()`'s reordered path, producing false-positive + * coverage. This test deliberately bypasses the standard helper and + * instantiates `PostgresEngine` directly, calling `engine.initSchema()` so + * the bootstrap → SCHEMA_SQL → runMigrations sequence runs end-to-end. + * + * Covers issues #366, #375, #378 — Postgres-side wedges where pre-v0.18 + * brains crashed on `column "source_id" does not exist`. + * + * NOTE: snapshot-based historical state simulation is out of scope for this + * wave (would require maintaining historical schema dumps). The test + * mutates a fresh-LATEST brain to a pre-v0.18 shape; codex flagged this as + * approximate. Acceptable here because the bootstrap's contract is narrow: + * "given a brain that lacks the specific forward-references, initSchema + * produces a brain at LATEST." The test exercises exactly that contract. + * + * Run: DATABASE_URL=postgresql://... bun run test:e2e test/e2e/postgres-bootstrap.test.ts + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PostgresEngine } from '../../src/core/postgres-engine.ts'; +import { LATEST_VERSION } from '../../src/core/migrate.ts'; + +const DATABASE_URL = process.env.DATABASE_URL; +const skip = !DATABASE_URL; + +describe.skipIf(skip)('PostgresEngine forward-reference bootstrap (E2E)', () => { + let engine: PostgresEngine; + + beforeAll(async () => { + engine = new PostgresEngine(); + await engine.connect({ database_url: DATABASE_URL! }); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('PostgresEngine.initSchema applies bootstrap → SCHEMA_SQL → migrations on pre-v0.18 brain', async () => { + // First call: bring the test DB to LATEST shape so we have something to mutate. + await engine.initSchema(); + + // Clear data from prior tests in the suite. Adding a UNIQUE(slug) + // constraint below would fail if multi-source fixtures left rows with + // duplicate slugs across sources (which is valid under the composite + // UNIQUE this test is undoing). + const conn = (engine as any).sql; + await conn.unsafe(`TRUNCATE pages, content_chunks, links, tags, raw_data, timeline_entries, page_versions, ingest_log RESTART IDENTITY CASCADE`); + + // Mutate to pre-v0.18 shape: drop source_id and the sources table. + // The advisory lock is released between initSchema calls, so this + // direct DDL won't deadlock. + await conn.unsafe(` + ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key; + ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug); + DROP INDEX IF EXISTS idx_pages_source_id; + ALTER TABLE pages DROP COLUMN IF EXISTS source_id CASCADE; + DROP TABLE IF EXISTS sources CASCADE; + `); + await engine.setConfig('version', '20'); + + // The path under test: full PostgresEngine.initSchema() including the + // bootstrap call, SCHEMA_SQL replay, and runMigrations chain. + await engine.initSchema(); + + expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION)); + + // Verify the forward-referenced column exists after upgrade. + const colCheck = await conn` + SELECT column_name FROM information_schema.columns + WHERE table_schema = current_schema() + AND table_name = 'pages' + AND column_name = 'source_id' + `; + expect(colCheck).toHaveLength(1); + + // Verify the default source row was seeded. + const srcCheck = await conn`SELECT id FROM sources WHERE id = 'default'`; + expect(srcCheck).toHaveLength(1); + }); + + test('PostgresEngine.initSchema is idempotent on a brain already at LATEST', async () => { + // Fresh-LATEST brain. Calling initSchema again must not error and must + // not regress the version. + await engine.initSchema(); + expect(await engine.getConfig('version')).toBe(String(LATEST_VERSION)); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index ed17bd7..8d883b5 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -287,6 +287,17 @@ describe('migration v24 — rls_backfill_missing_tables', () => { test('LATEST_VERSION has caught up to 24', () => { expect(LATEST_VERSION).toBeGreaterThanOrEqual(24); }); + + // PGLite has no RLS engine and is intrinsically single-tenant. The 8 RLS + // backfill ALTER statements target tables that may not exist on PGLite + // (subagent_*, minion_inbox aren't always present in pglite-schema.ts). + // sqlFor.pglite='' makes v24 a no-op on PGLite while still bumping the + // version counter. Engine.kind discrimination in runMigrations selects + // sqlFor[engine.kind] over m.sql. Issue #395. + test('uses a PGLite no-op override so local brains skip Postgres-only RLS ALTER TABLEs', () => { + const v24 = MIGRATIONS.find(m => m.version === 24); + expect(v24?.sqlFor?.pglite).toBe(''); + }); }); // ───────────────────────────────────────────────────────────────── diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts new file mode 100644 index 0000000..1772943 --- /dev/null +++ b/test/schema-bootstrap-coverage.test.ts @@ -0,0 +1,149 @@ +/** + * CI guard: PGLITE_SCHEMA_SQL must not forward-reference state that + * `applyForwardReferenceBootstrap` doesn't know how to create. + * + * Background: gbrain ships an "embedded latest schema" blob + * (`pglite-schema.ts`) for fast bootstraps, alongside a numbered migration + * chain (`migrate.ts`) for incremental upgrades. Across 2 years and 6 schema + * versions, every release that added a column-with-index in the schema blob + * without a corresponding bootstrap addition has triggered the same wedge + * incident class (#239, #243, #266, #266, #357, #366, #374, #375, #378, + * #395, #396). + * + * The bootstrap is the structural fix. This test enforces the contract: + * for every "forward reference" the schema blob makes (FK or indexed column + * defined later than its reference site, or any column that older brains + * lack), the bootstrap MUST add enough state so that running the schema + * blob is replay-safe on a brain that lacks every member of + * `REQUIRED_BOOTSTRAP_COVERAGE`. + * + * **When you add a new schema-blob forward reference:** + * 1. Extend `applyForwardReferenceBootstrap` in pglite-engine.ts + + * postgres-engine.ts to add the new state. + * 2. Add an entry to `REQUIRED_BOOTSTRAP_COVERAGE` below. + * 3. This test will pass. + * + * If you add a forward reference but skip step 1, this test fails. If you + * skip step 2, this test passes but the bootstrap silently drifts behind + * the schema. The eng-review polish notes recommended layered coverage + * (per-engine integration tests in `test/bootstrap.test.ts` + + * `test/e2e/postgres-bootstrap.test.ts`) to catch step 2 oversights. + */ + +import { test, expect } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +// Forward-reference targets that PGLITE_SCHEMA_SQL requires. +// When you add a new one, extend this list AND the bootstrap. +type ForwardReference = + | { kind: 'table'; name: string } + | { kind: 'column'; table: string; column: string }; + +const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ + // Forward-referenced by `pages.source_id REFERENCES sources(id)` and the + // `INSERT INTO sources (id, name, config) VALUES ('default', ...)` seed. + { kind: 'table', name: 'sources' }, + // Forward-referenced by `CREATE INDEX idx_pages_source_id ON pages(source_id)`. + { kind: 'column', table: 'pages', column: 'source_id' }, + // Forward-referenced by `CREATE INDEX idx_links_source ON links(link_source)`. + { kind: 'column', table: 'links', column: 'link_source' }, + // Forward-referenced by `CREATE INDEX idx_links_origin ON links(origin_page_id)`. + { kind: 'column', table: 'links', column: 'origin_page_id' }, + // v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_symbol_name + // ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL`. + { kind: 'column', table: 'content_chunks', column: 'symbol_name' }, + // v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_language + // ON content_chunks(language) WHERE language IS NOT NULL`. + { kind: 'column', table: 'content_chunks', column: 'language' }, +]; + +test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => { + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + // Strip every required forward-reference target so the brain looks like + // it pre-dates the migrations that introduced these objects. Drop columns + // before the table-level constraints that depend on them. + await db.exec(` + ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key; + ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug); + DROP INDEX IF EXISTS idx_pages_source_id; + ALTER TABLE pages DROP COLUMN IF EXISTS source_id; + DROP TABLE IF EXISTS sources CASCADE; + + DROP INDEX IF EXISTS idx_links_source; + DROP INDEX IF EXISTS idx_links_origin; + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique; + ALTER TABLE links DROP COLUMN IF EXISTS link_source; + ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id; + + DROP INDEX IF EXISTS idx_chunks_symbol_name; + DROP INDEX IF EXISTS idx_chunks_language; + ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name; + ALTER TABLE content_chunks DROP COLUMN IF EXISTS language; + `); + + // Run bootstrap in isolation (NOT initSchema). This is what we're testing. + await (engine as any).applyForwardReferenceBootstrap(); + + // Assert every required forward-reference target now satisfies the + // schema-blob's expectations. + for (const ref of REQUIRED_BOOTSTRAP_COVERAGE) { + if (ref.kind === 'table') { + const { rows } = await db.query( + `SELECT 1 FROM information_schema.tables + WHERE table_schema = 'public' AND table_name = $1`, + [ref.name], + ); + expect(rows.length).toBeGreaterThan(0); + } else { + const { rows } = await db.query( + `SELECT 1 FROM information_schema.columns + WHERE table_schema = 'public' AND table_name = $1 AND column_name = $2`, + [ref.table, ref.column], + ); + expect(rows.length).toBeGreaterThan(0); + } + } + } finally { + await engine.disconnect(); + } +}, 30000); + +test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing forward references', async () => { + // End-to-end contract: bootstrap → SCHEMA_SQL must succeed even on a brain + // that lacks every forward-referenced target. This catches the case where + // REQUIRED_BOOTSTRAP_COVERAGE drifts behind PGLITE_SCHEMA_SQL — if the + // schema blob added a new index on a column the bootstrap doesn't create, + // the SCHEMA_SQL exec below would crash even though the per-target asserts + // above pass. + const engine = new PGLiteEngine(); + await engine.connect({}); + try { + await engine.initSchema(); + const db = (engine as any).db; + + await db.exec(` + ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key; + ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug); + DROP INDEX IF EXISTS idx_pages_source_id; + ALTER TABLE pages DROP COLUMN IF EXISTS source_id; + DROP TABLE IF EXISTS sources CASCADE; + DROP INDEX IF EXISTS idx_links_source; + DROP INDEX IF EXISTS idx_links_origin; + ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique; + ALTER TABLE links DROP COLUMN IF EXISTS link_source; + ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id; + `); + + // Bootstrap, then schema replay. Either step crashing fails the test. + const { PGLITE_SCHEMA_SQL } = await import('../src/core/pglite-schema.ts'); + await (engine as any).applyForwardReferenceBootstrap(); + await db.exec(PGLITE_SCHEMA_SQL); + } finally { + await engine.disconnect(); + } +}, 30000);