2 Commits

Author	SHA1	Message	Date
Garry Tan	e60b60244f	v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (#1165) ... * feat(cross-modal/0): batched multimodal + query helpers + SSRF helper Commit 0 of the cross-modal search wave. Foundation for Phase 1-3: - embedMultimodal accepts MultimodalInput text variant + EmbedMultimodalOpts with inputType: 'document' | 'query' (D22-2). Default unchanged so importImageFile keeps document-side embedding. - embedQueryMultimodal(text) + embedQueryMultimodalImage(input) wrappers for hybridSearch + searchByImage query paths. - embedMultimodalSafe binary-search retry on transient batch failure + failed_indices surfacing. Phase 3 reindex uses this so a single bad chunk doesn't discard the 31 in-flight embeddings around it. - Voyage path: text + image inputs in one batch via content arrays. - openai-compat path: text + image inputs in one request per input. - src/core/ssrf-validate.ts (D19): DNS-resolve-and-fetch-by-IP defense for redirect chains. Closes the DNS-rebinding gap that url-safety.ts' static check leaves open. Uses node:dns/promises with {all: true, family: 0} to inspect every A and AAAA record before connecting. fetchWithSSRFGuard helper validates per-redirect-hop and limits chain depth (default 3). - Re-exports from src/core/embedding.ts public seam. Tests: - test/embed-multimodal-batching.test.ts (13 cases): text variant, query inputType discipline, mixed text+image batches, embedQueryMultimodal, embedQueryMultimodalImage, embedMultimodalSafe happy/empty/all-fail/ mid-batch-recovery/permanent-misconfig. - test/ssrf-validate.test.ts (20 cases): static rejections via isInternalUrl, scheme + credentials rejection, DNS rebinding defense (single-record + multi-record), public happy path, IPv6 literals, malformed URLs. No regression in existing voyage-multimodal.test.ts or openai-compat-multimodal.test.ts (33 cases all pass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/1): Phase 1 text→image routing + knobsHash + RRF + backfill Phase 1 of the cross-modal search wave. Wires the existing 1024d Voyage multimodal embedding space (already populated for image chunks via importImageFile) into the user-facing query path. Text queries that match cross-modal intent regex route through Voyage multimodal-3 instead of the text embedding model, then search content_chunks.embedding_image. - query-intent.ts: new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope CROSS_MODAL_PATTERNS regex array (D15 — compiled once at module load). Conservative on purpose; LLM intent escalation (Commit 4) catches genuinely ambiguous phrasings. - query-intent.ts: new `isAmbiguousModalityQuery(query)` pure heuristic for Commit 4's escalation gate. Returns true ONLY when regex misses AND a visual noun + reference marker both fire. - types.ts: `SearchOpts.crossModal: 'text' | 'image' | 'both' | 'auto'` + `SearchResult.modality: 'text' | 'image'` for downstream renderers. - mode.ts: 7 new knobs in ModeBundle (D2): cross_modal_both_text_weight, cross_modal_both_image_weight, image_query_text_refinement_weight, image_query_image_refinement_weight, unified_multimodal, unified_multimodal_only, cross_modal_llm_intent. All three mode bundles default to the same values (cross-modal is opt-in). - mode.ts: D2 cache-key fix — KNOBS_HASH_VERSION bumped 2→3, all 7 new knobs participate in knobsHash so a text-mode cache hit can't be served to an image-mode caller. - mode.ts: D3 registry — all 7 keys land in SEARCH_MODE_CONFIG_KEYS so `gbrain search modes` / `stats` / `tune` see them. - hybrid.ts: routing branch at the embed step. Resolves effective modality from (per-call opts → suggestions → 'text'). Image route: embedQueryMultimodal + searchVector(embedding_image), skip expansion + keyword (D9 mode-bundle override). Both route: parallel text + image vector searches merged via weighted RRF (D6) with cross_modal_both_* weights. Fail-open: multimodal misconfigured → structured warn + text fallback. 'auto' literal normalized to undefined (D22-1). - operations.ts: thread `cross_modal` param through `query` op. - backfill-registry.ts: new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` (D22-7 defensive guard). Idempotent. - doctor.ts: `cross_modal_modality_backfill` check surfaces unflagged image-asset chunks with paste-ready `gbrain backfill modality` hint. Tests: - cross-modal-phase1.test.ts (45 cases): regex classification (positive + negative + plural-safe), isAmbiguousModalityQuery, D3 registry, D2 knobsHash diffs across all 7 new knobs, MODE_BUNDLES defaults, resolveSearchMode precedence chain. - cross-modal-hybrid-integration.test.ts (7 cases): PGLite + stubbed gateway. Verifies image-modality calls Voyage and not OpenAI, text calls OpenAI and not Voyage, 'auto' literal normalizes, 'both' mode hits both endpoints, fail-open routes to text on multimodal misconfig. - search-mode.test.ts: updated MODE_BUNDLES + KNOBS_HASH_VERSION assertions (148 cross-suite tests still pass; no regression). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/2): Phase 2 image-as-query + D18 path ban + D23-#6 spend cap Phase 2 of the cross-modal search wave. Adds the `search_by_image` MCP op, the SSRF-defended image loader, and the daily per-OAuth-client spend cap on paid Voyage multimodal calls. D17 honest framing applied: Phase 2 ships image→similar-images + image-OCR-text retrieval. True image→full-text- knowledge requires Phase 3's unified column. - src/core/search/image-loader.ts: loadImageInput accepts local path, data: URI, or http(s):// URL. Magic-byte sniff for PNG/JPEG/WebP (no other formats). Hard size cap (10MB local default, 2MB remote default). http(s) path uses fetchWithSSRFGuard from Commit 0: every redirect hop re-resolved via DNS lookup + every record checked against the internal IP deny list. Max 3 redirect hops. 5s total fetch timeout. Pre-flight Content-Length check + post-fetch size guard for lying servers. - src/core/search/by-image.ts: searchByImage runs the image branch always; D13 hybrid intersect runs a parallel text branch when `query` is provided, merged via weighted RRF. Phase 3 will widen the column routing to embedding_multimodal once that lands. - src/core/operations.ts: new search_by_image op (scope: read, NOT localOnly). D18 P0 — when ctx.remote === true AND image_path is set, rejects with permission_denied at handler entry (validateParams would catch it again at dispatch). D5 source-id thread via sourceScopeOpts. D12 per-param length cap enforced via remote-vs-local maxBytes config read at handler entry. D23-#6 pre-flight checkBudget + post-call recordSpend (best-effort; failures don't block response). - src/core/spend-log.ts: BudgetExceededError + checkBudget + recordSpend + getTodaySpendCents. UTC day-aligned aggregation so the cap rolls over deterministically. Local CLI callers (no clientId) bypass the gate entirely. Pre-v0.36 brains without the mcp_spend_log table fail open to spend=0; the migration brings the table in on first start. - src/core/migrate.ts: new migration v67 mcp_spend_log table + indexes for the (client_id, day) and (token_name, day) hot reads. PGLite parity via sqlFor.pglite. - src/core/search/hybrid.ts: RRF_K constant exported so by-image.ts can share the same effective-K math as the main hybrid path. Tests: - cross-modal-phase2.test.ts (15 cases): magic-byte sniffing (PNG + JPEG + WebP positive, GIF rejection), oversized rejection (default + custom cap), data: URI happy path + malformed + decoded-non-image + oversized, invalid input shapes (empty + ftp), SSRF defense via DNS rebinding stub. - search-by-image-op.test.ts (7 cases): D18 remote image_path rejection + local CLI accepts; input validation (missing all three / multiple together); D23-#6 budget block-at-cap + allow-under-cap + local-CLI-bypass; migration v67 mcp_spend_log table applied cleanly. All 166 tests across the cross-modal suite pass; no regression in existing voyage-multimodal / openai-compat-multimodal / search-mode suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/3): Phase 3 unified column + reindex + D8 fail-open + D23-#2 Phase 3 of the cross-modal search wave. Adds the unified multimodal column on content_chunks + the `gbrain reindex --multimodal` sweep + the `search.unified_multimodal` routing flag with D8 source-aware coverage guard + fail-open behavior. D17 honest framing: this is the phase that unlocks true image→full-text-knowledge — Phase 2's searchByImage transparently upgrades to the richer retrieval once the unified column has coverage. D10 reindex-core extraction filed as a follow-up TODO. The existing markdown reindex walks pages and re-imports via importFromFile; this walks content_chunks and re-embeds via the gateway. Patterns rhyme but cores diverge enough that extraction balloons the diff. Both commands stand alone with their own checkpoint + cost-prompt logic. - migrate.ts v68 (embedding_multimodal_column): column-only ALTER on content_chunks. HNSW partial index deferred to post-reindex build (D20: pgvector docs recommend post-load build for HNSW). Both engines. - types.ts SearchOpts.embeddingColumn type widened to include 'embedding_multimodal'. - postgres-engine.ts + pglite-engine.ts searchVector: route to embedding_multimodal column when opts.embeddingColumn set. NO modality filter (unified column carries both text + image content). - hybrid.ts unified routing branch: when search.unified_multimodal=true, bypasses dual-column branching and runs embedQueryMultimodal + searchVector(embedding_multimodal). D8 fail-open: zero rows + not strict-mode → falls through to dual-column text path with structured warning. search.unified_multimodal_only=true bypasses the fallback. - src/commands/reindex-multimodal.ts: `gbrain reindex --multimodal`. D7 lock via tryAcquireDbLock('gbrain-reindex-multimodal'); 6h TTL. Cost prompt + 10s Ctrl-C grace window in TTY; auto-proceeds non-TTY. GBRAIN_NO_REEMBED=1 bypass. Checkpoint at ~/.gbrain/reindex-multimodal-checkpoint.json for resume. D23-#2 auto-flip prompt at coverage=100% completion. - cli.ts: `gbrain reindex --multimodal` dispatch with --limit, --dry-run, --cost-estimate, --no-embed, --yes, --json flags. - doctor.ts: unified_multimodal_coverage check (D21 source-aware) + reports per-source % when search.unified_multimodal is on. Warns at <95% lowest source; fails when unified_multimodal_only=true AND lowest source <99%. Falls open to OK when column not yet present. Tests: - unified-multimodal.test.ts (8 cases): schema migration v68 applies, reindex --dry-run + --cost-estimate + GBRAIN_NO_REEMBED bypass + zero-pending fast-path, hybridSearch unified routing forces voyage endpoint, D8 fail-open routes to text on empty unified, D8 strict blocks text fallback. All 211 tests across the cross-modal + related suite pass; no regression in voyage-multimodal / openai-compat-multimodal / search-mode / intent / search base suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/4): LLM intent escalation for ambiguous modality Commit 4 of the cross-modal search wave (opt-in default off). When `search.cross_modal.llm_intent` is true AND the regex classifier returned 'text' AND `isAmbiguousModalityQuery(query)` fires, hybridSearch awaits a Haiku tie-break via gateway.chat() before routing. The ambiguous-modality gate (introduced in Commit 1) ensures the LLM call only fires on the narrow band where regex misses but a visual noun + reference marker both fire — roughly <1% of queries with the flag on. - src/core/search/llm-intent.ts: new module. `classifyModalityWithLLM` routes through gateway.chat() with a fixed system prompt ("Output exactly one word: text, image, or both"). 1s timeout via AbortController. `parseModality` is a pure exported helper that tolerates trailing punctuation + casing. Fail-open on every error path (gateway unavailable, timeout, parse failure, unrecognized output). - src/core/search/hybrid.ts: escalation branch slots BEFORE the unified routing branch. Gated by: no explicit per-call crossModal opt, regex result == 'text', config flag on, ambiguity heuristic fires. Fail-open to regex result on any error from the LLM tie-break. Tests: - llm-intent-escalation.test.ts (14 cases): parseModality tolerance matrix (text / image / both / trailing punct / whitespace / unrecognized / empty), classifyModalityWithLLM happy paths for all 3 outputs, fail-open on throw / unrecognized output / gateway-not- configured, explicit-fallback-honored. - llm-intent-hybrid-integration.test.ts (6 cases): hybridSearch escalation gate fires ONLY when flag-on + ambiguous; off when flag-off, unambiguous, regex-confident, or explicit per-call opt set; fail-open on LLM throw. All 231 tests across the cross-modal + related suite pass; no regression in voyage-multimodal / openai-compat-multimodal / search-mode / intent / search base suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cross-modal/3): verify-gate fixes for full test suite Three small fixes to pass the full unit + E2E sweep after the cross-modal wave commits land. - migrate.ts v67: drop date_trunc('day', created_at) from mcp_spend_log indexes. TIMESTAMPTZ truncation depends on session timezone and isn't IMMUTABLE, so Postgres rejects the function in the index expression with SQLSTATE 42P17. BTREE on (client_id, created_at) covers the per-day rollup query via range scan on created_at — same performance, no IMMUTABLE constraint. - pglite-schema.ts + src/schema.sql: shorten the embedding_multimodal column comment. The longer version contained a comma inside a SQL line comment ("...search.unified_multimodal=true, all queries..."), which broke parseBaseTableColumns in test/schema-bootstrap-coverage (the parser splits on commas at depth-0 before stripping comments, so the comma inside the comment shortened the column-definition part and an "all" token from "all queries" got picked up as the next column name — silently hiding embedding_multimodal from coverage). - schema-embedded.ts: regenerated via `bun run build:schema`. - test/e2e/v030_1-integration-pglite.test.ts: listBackfills assertion extended to include the new `modality` entry registered in src/core/backfill-registry.ts as part of Commit 1. - test/search/knobs-hash-reranker.test.ts: KNOBS_HASH_VERSION assertion updated from 2→3 to match the cross-modal-wave hash-key extension (D2 cache contamination fix). Same shape as the prior v0.32→v0.35 bump. - test/unified-multimodal.test.ts: migrated process.env mutation to withEnv() helper to satisfy the scripts/check-test-isolation R1 rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cross-modal): VERSION + CHANGELOG + CLAUDE.md + spec doc + llms regen Final docs commit for the cross-modal wave (v0.36.0.0). - VERSION + package.json: bump 0.35.5.1 → 0.36.0.0 - CHANGELOG.md: full Garry-voice release entry with five-commit breakdown, the-numbers-that-matter table, what-this-means-for-you, and the required to-take-advantage-of-v0.36.0.0 block - docs/issues/cross-modal-search.md: cherry-picked from PR #1127 head (164 lines, the original spec doc preserved as historical reference for Phase 2 + 3 background) - CLAUDE.md: Key Files entries for src/core/ssrf-validate.ts, src/core/search/image-loader.ts, src/core/search/by-image.ts, src/core/search/llm-intent.ts, src/core/spend-log.ts, src/commands/reindex-multimodal.ts, plus extension annotations on src/core/search/query-intent.ts, src/core/search/mode.ts, src/core/search/hybrid.ts, src/core/backfill-registry.ts, src/core/migrate.ts (v67 + v68) - llms-full.txt + llms.txt: regenerated via `bun run build:llms` `bun run verify` clean (privacy + proposal-pii + test-names + jsonb + source-id-projection + progress + test-isolation + wasm + admin-build + admin-scope-drift + cli-exec + system-of-record + eval-glossary + typecheck). `bun test test/build-llms.test.ts` clean (7/7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cross-modal): renumber migrations 67→69 + 68→70 post-master-merge Master shipped its own v67 (`facts_typed_claim_columns`) during the cross-modal wave's review cycle. The merge picked up both side's v67 entries, breaking the migration-distinct-versions test. Renumbering moves cross-modal's table + column ALTER off the collision: - v67 mcp_spend_log → v69 mcp_spend_log - v68 embedding_multimodal_column → v70 embedding_multimodal_column References updated in CHANGELOG, CLAUDE.md, pglite-schema.ts, schema.sql. schema-embedded.ts regenerated. llms-full.txt regenerated. 7006 unit tests pass, 0 fail. No test code touched — just version renumbering plus comment refs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version 0.36.0.0 → 0.36.4.0 Bumping to v0.36.4.0 to land in the queue slot the user requested. No behavior change; pure version bump across VERSION, package.json, CHANGELOG.md header, llms-full.txt regen. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 17:14:53 -07:00
Garry Tan	dffb607ef7	v0.30.1 feat: operational hardening — make upgrades just work on Supabase (#750) ... * v0.30.1 Lane A: connection-manager foundation + X1 initSchema routing Routes Postgres queries by query type: - read() goes to the Supabase pooler (port 6543, fast) - ddl() and bulk() go to direct (port 5432, 30min stmt timeout, mwm 256MB) Auto-detects Supabase via hostname pooler.supabase.com or port 6543. Override with GBRAIN_DIRECT_DATABASE_URL. Kill-switch via GBRAIN_DISABLE_DIRECT_POOL=1 falls back to single-pool legacy path. Foundation modules (Lane A scope): - src/core/connection-manager.ts: read/ddl/bulk/healthCheck, parent-CM inheritance (T5/X1), cached Promise<Sql> lazy init (A1), kill-switch inheritance (A2), Supabase URL auto-derivation - src/core/url-redact.ts: redactPgUrl + redactDeep (F3) - src/core/retry-matcher.ts: typed predicates for stmt-timeout / lock / conn errors (C4) - src/core/connection-audit.ts: ~/.gbrain/audit/connection-events JSONL with ISO-week rotation; doctor tail-reads last 5 errors (F8) - scripts/check-pg-url-redaction.sh: CI grep guard against unredacted postgresql:// URL leaks (F3) Engine integration: - PostgresEngine.connect: instantiates instance-owned ConnectionManager, inherits from parentConnectionManager when set (worker engines, sync, cycle), shares pool with module-singleton path - PostgresEngine.disconnect: tears down direct pool first - PostgresEngine.initSchema: routes DDL through connectionManager.ddl() when dual-pool active (X1 part 1; lock semantics replacement is Lane B) - cli.ts:connectEngine(opts): probeOnly skips initSchema entirely (X1 part 2 — get_health, upgrade --status will use this) Tests added (51 new cases): - test/url-redact.test.ts: 11 cases - test/retry-matcher.test.ts: 13 cases - test/connection-manager.test.ts: 27 cases (URL detection, derive, kill-switch, parent inheritance, dual-pool routing modes) Foundation for Lanes B-E. Sequential lane work continues. Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-wadler.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane B: migration runner retry + verify hooks + namespaced --force flags Adds Migration interface fields: - idempotent: boolean (default true; explicit false blocks verify-hook re-runs on destructive migrations) - verify: optional post-condition probe; runs after migration claims success Migration retry wrapper (Cherry D3 / Finding F2): - 3 attempts with 5s/15s/45s backoff (env GBRAIN_MIGRATE_BACKOFF_MS=0 for tests) - Retries only on statement_timeout (57014) or connection-reset patterns - Pre-attempt: logs idle-in-transaction blockers via getIdleBlockers - On exhaustion: throws MigrationRetryExhausted with named PID + suggested pg_terminate_backend() recovery command Verify-hook self-healing (Cherry D6 / Codex X3): - On verify=false + idempotent=true → re-runs migration once silently - On verify=false + idempotent=false → throws MigrationDriftError - --skip-verify CLI flag bypasses for operator override withRefreshingLock helper (Cherry T4 / Codex A4 / X1 part 3): - setInterval refresh every TTL/6 ms during long-running work - SELECT 1 backend-alive heartbeat per refresh tick - Heartbeat hang past 30s → log + clear interval; lock TTL auto-expires - LockUnavailableError when acquire fails (caller decides retry) - buildTenantLockId(scope) appends current_database() suffix for multi-tenant safety (Cherry D4) Namespaced --force flags (Codex T5): - --force-orchestrator: write 'retry' markers for ALL wedged orchestrators - --force-schema: re-runs runMigrations against current config.version - --force / --force-all: both - --force-retry vX.Y.Z: existing single-version reset (preserved) - --skip-verify: bypass verify-hook drift detection on a single run Test additions: - test/migrate-extensions.test.ts: 14 cases (idempotent default, error envelopes, MIGRATIONS contract) - test/db-lock-refresh.test.ts: 10 cases (LockUnavailableError, buildTenantLockId multi-tenant, opts shape) - test/migrate.test.ts: updated 2 existing cases (PR #356 retry shape + function-name anchor) for v0.30.1 retry-wrapper semantics 156 unit tests passing across the v0.30.1 surface so far. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane C: backfill primitive + registry + X4 + X5 First-class generic backfill runner (Fix 3). Generalizes the keyset+checkpoint+adaptive-batch pattern from src/core/backfill-effective-date.ts so future backfills (embedding_voyage in v0.30.2, etc.) reuse one tested runner. NEW src/core/backfill-base.ts: - runBackfill() with keyset pagination, config-table checkpoint, adaptive batch halving on stmt timeout, conn-drop reconnect, max-errors bail - ensureBackfillIndex() verifies/creates partial index CONCURRENTLY (P2/X4) - clearBackfillCheckpoint() for --fresh path - T3 fix: writes go through engine.withReservedConnection so BEGIN / SET LOCAL / UPDATE / COMMIT execute on the SAME backend (otherwise SET LOCAL evaporates between pooled executeRaw calls) NEW src/core/backfill-registry.ts: - effective_date: implemented (wraps existing computeEffectiveDate) - emotional_weight: implemented (wraps computeEmotionalWeight + stamps new emotional_weight_recomputed_at column) - embedding_voyage: declared-only in v0.30.1 (multi-column embedding schema lands in v0.30.2) NEW src/commands/backfill.ts: - gbrain backfill <kind> [--batch-size N] [--concurrency N] [--resume] [--fresh] [--dry-run] [--keep-index] [--max-errors N] - gbrain backfill list — shows registered backfills + status - X5 admission control: clampConcurrency() forces --concurrency to GBRAIN_DIRECT_POOL_SIZE - 1 ceiling (always reserves 1 conn for HNSW + heartbeat + doctor probes). Loud-warns when user requests above. Schema migration v44 (X4 / Codex C8 fix): - pages.emotional_weight_recomputed_at TIMESTAMPTZ - emotional_weight = 0 is a VALID steady-state value per migration v40, so the original P2 predicate ("WHERE emotional_weight = 0") would have been a permanent large index over normal data. The corrected backlog predicate is "emotional_weight_recomputed_at IS NULL"; the partial index drops naturally as the cycle phase + this backfill stamp the column over time. - idempotent: true (ADD COLUMN ... NULL is metadata-only) CLI integration: - src/cli.ts: registers `backfill` subcommand - reindex-frontmatter stays as thin alias for v0.30.1 back-compat; canonical entrypoint is now `gbrain backfill effective_date` Test additions: - test/backfill-base.test.ts: 11 cases (keyset, checkpoint, dry-run, resume/fresh, maxRows cap, withReservedConnection routing, error paths, clearCheckpoint, ensureBackfillIndex) - test/backfill-concurrency-clamp.test.ts: 6 cases (X5 admission control) 173 unit tests passing across Lanes A+B+C of v0.30.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane D: HNSW lifecycle manager + A3 atomic-swap Extends src/core/vector-index.ts with the v0.30.1 lifecycle layer. The original chunkEmbeddingIndexSql / applyChunkEmbeddingIndexPolicy contract is preserved unchanged. New surfaces: - checkActiveBuild(engine, indexName): probes pg_stat_activity for an active CREATE INDEX or REINDEX on the named index. Used as pre-op guard so dropAndRebuild doesn't compete with a build already in flight (Supabase auto-maintenance, parallel gbrain procs). - dropZombieIndexes(engine, tableNames): startup sweep of indisvalid=false rows on gbrain tables. Drops them with DROP INDEX IF EXISTS, BUT skips any zombie that has an active build still in pg_stat_activity (codex Fix-5 in-progress-build guard). Wired into PostgresEngine.initSchema() — runs after migrations + verifySchema, best-effort, never blocks engine.connect(). - dropAndRebuild(engine, spec, opts): A3 atomic-swap pattern: 1. checkActiveBuild → bail if another build is active (--force overrides) 2. CREATE INDEX CONCURRENTLY <name>_rebuild_<unix-ms> via engine.withReservedConnection (CONCURRENTLY can't run in a txn) 3. Atomic swap inside engine.transaction: DROP INDEX <old-name> ALTER INDEX <temp-name> RENAME TO <old-name> 4. If step 2 fails (OOM, timeout, conn drop), the OLD index stays intact and search keeps serving queries. This is the headline A3 win — no production-degraded silent failure mode. - monitorBuild(engine, indexName, onProgress, opts): poll pg_stat_activity every 30s; emit elapsed_ms + size_bytes (via pg_relation_size) + pid. Used by gbrain backfill embedding_voyage when batch > 1000 triggers a rebuild. - isSupabaseAutoMaintenance(active): predicate on application_name (matches "supabase" / "postgres-meta"). Used by dropAndRebuild to log + back off when Supabase auto-maintenance is doing the rebuild. Engine integration: - PostgresEngine.initSchema() calls dropZombieIndexes after verifySchema. Surfaces zombie counts via console.log. - Best-effort wrapped in try/catch: pg_stat_activity / pg_index access can be restricted on managed Postgres tiers; gbrain shouldn't fail engine.connect() over diagnostic queries. Test additions (18 cases): - test/vector-index-lifecycle.test.ts: * chunkEmbeddingIndexSql contract (3 cases) — pre-existing behavior preserved * applyChunkEmbeddingIndexPolicy contract (1 case) * checkActiveBuild (4 cases, including PGLite no-op + best-effort failure) * isSupabaseAutoMaintenance (3 cases) * dropZombieIndexes (4 cases, including in-progress-build guard) * dropAndRebuild atomic-swap (3 cases, including PGLite + active-build bail + temp-name format assertion) 191 unit tests passing across Lanes A+B+C+D of v0.30.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane E: upgrade pipeline checkpoint + brain_id binding + get_health migrations NEW src/core/upgrade-checkpoint.ts: - Cherry D5: persists step-by-step progress through gbrain post-upgrade so partial failures can be resumed via gbrain upgrade --resume. Steps: pull → install → schema → features → backfills → verify. - Codex X2: checkpoint binds to brain identity via sha256(database_url) (userinfo stripped before hashing so cred rotations don't invalidate). PGLite uses sha256(database_path). Cross-brain checkpoint application is now refused with reason='brain_mismatch'. - F4 fall-through: validateCheckpoint returns reason='no_checkpoint' when none exists, enabling silent fall-through to a full upgrade. - All-complete detection: stale checkpoints (every step done) return reason='all_complete' so the next run clears + re-runs from scratch. - markStepComplete + markStepFailed maintain the partial-state shape. T2 preserved: upgrade.ts still re-execs `gbrain post-upgrade` so the NEW binary's migration registry runs (the existing re-exec pattern is correct per codex round 1's plan-breaking finding). The checkpoint module is the substrate that Lane E's --resume / --status surfaces will plumb through in v0.30.2. D7 + C3 contract committed: - BrainHealth.schema_version: '1' (literal type) — additive-only contract pinned for MCP get_health consumers. - BrainHealth.migrations: { schema, orchestrator } — explicit two-ledger diagnostic surface (codex T5 namespacing). Both fields are OPTIONAL in v0.30.1 — engines can populate them in v0.30.2 without a contract bump. Backwards/forwards compat: clients default-handle missing fields. VERSION: 0.30.0 → 0.30.1 package.json: synced Test additions (18 cases): - test/upgrade-checkpoint.test.ts: * computeBrainId: userinfo strip, DB-distinct hashes, stable hex (5 cases) * write/load round-trip: roundtrip, missing file, malformed JSON, clear (4 cases) * validateCheckpoint: F4 no_checkpoint, X2 brain_mismatch, partial → resumeAt, all_complete, first-step pending (5 cases) * markStepComplete/markStepFailed: append, idempotent, clear-failed, failed-state shape (4 cases) 209 unit tests passing across all 5 lanes of v0.30.1 (Lanes A-E core foundations). Plumbing into upgrade.ts CLI + doctor checks + get_health() implementation is layered in via follow-up commits within this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 e2e + test isolation: integration smoke + serial quarantine NEW test/e2e/v030_1-integration-pglite.test.ts (14 cases): PGLite integration smoke proving Lane A-E surfaces work together. Lane B: migration runner applies v44 (emotional_weight_recomputed_at) cleanly; config.version reaches LATEST_VERSION Lane C: backfill registry resolves all 3 entries; emotional_weight + effective_date backfills on empty brain return examined=0 cleanly Lane D: dropZombieIndexes / checkActiveBuild on PGLite are no-ops Lane E: upgrade-checkpoint round-trips with brain_id; X2 mismatch refused; F4 fall-through detected via reason='no_checkpoint'; full step progression to all_complete Test isolation hygiene (scripts/check-test-isolation.sh): - test/connection-manager.test.ts → connection-manager.serial.test.ts - test/backfill-concurrency-clamp.test.ts → .serial.test.ts - test/upgrade-checkpoint.test.ts → .serial.test.ts All three files mutate process.env (kill-switch, GBRAIN_DIRECT_POOL_SIZE, GBRAIN_HOME) which would race other tests in the parallel runner. *.serial.test.ts quarantine ensures they run at --max-concurrency=1. Choice between withEnv() refactor and serial quarantine made on the side of preserving existing well-formed test code. E2E coverage status: - v030_1-integration-pglite.test.ts (this commit): 14 cases, all green - backfill-perf-pglite.test.ts: 1 case, green (no regression) - cycle-recompute-emotional-weight-pglite.test.ts: green (no regression) - multi-source-emotional-weight-pglite.test.ts: green (no regression) - dream-synthesize-pglite.test.ts: 14 cases, green (no regression) - anomalies-pglite.test.ts + salience-pglite.test.ts: 6 cases, green Postgres-only E2Es (migration-flow, http-transport, hnsw-lifecycle, connection-routing) require DATABASE_URL + a real Postgres+pgvector container per the CLAUDE.md E2E lifecycle. They land as separate DATABASE_URL-gated work — not regressed by v0.30.1 changes; their preconditions just aren't met in the current run environment. `bun run verify` (typecheck + 4 shell pre-checks + test-isolation lint) passes cleanly. Final v0.30.1 unit + integration test count: 4547 pass, 0 regressions. Two pre-existing flaky failures (BrainRegistry serial test + warm-create perf gate under shard contention) confirmed unrelated to this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.30.1) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-08 13:25:48 -07:00