triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
318 Commits 2 Branches 0 Tags
03ffc6ebdbc7dd8b29e5bfd0c3a9a6c983b54f01
Commit Graph

4 Commits

Author SHA1 Message Date
Garry Tan
03ffc6ebdb v0.42.37.0 fix(jobs): reap stale locks, bound disconnect, complete cooperative-abort (#1972) (#2015) 
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972)

A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row
until something contended for it — reclaim was on-contention only. Add a
host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder
PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle*
namespaces only (never elections/supervisor/reindex), with a snapshot-matched
delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse.
Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now
auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now
back inspectLock + listStaleLocks (killed the triplication).

* fix(db): bound pool disconnect so teardown can't eat CLI output (#1972)

pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. #1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.

* fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972)

v0.42.29 made only the embed phase honor the abort signal; a 24h pull still
showed force-evicts from a long non-embed phase ignoring it. Thread the signal
into every cycle-reachable long loop: extract (extractForSlugs + the full-walk
extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop +
embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket
loop. Add a terminal abort check so an aborted cycle never stamps
last_full_cycle_at as a completed run (Codex #9). lint now yields + checks
abort every 200 pages (it's synchronous; the yield is what lets the signal
land). New phase-duration force-evict attribution log names any phase that
crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start.

* chore: bump version and changelog (v0.42.38.0)

#1972 — stale-lock reaper, bounded pool disconnect, and complete
cooperative-abort coverage across cycle phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage

document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY),
db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/
extract_facts/consolidate/lint + terminal guard) entries to current truth.

* test(isolation): fix shard-order flakes exposed by #1972's new test files

Adding 3 new test files reshuffled the hash-based shards, exposing two
pre-existing test-isolation bugs:

- cycle-consolidate.test.ts assumed the global legacy-embedding preload's
  1536-d gateway config still held at initSchema, but a co-sharded test that
  calls resetGateway() in teardown nulls it, so initSchema fell back to the
  1280-d default and built a halfvec(1280) facts column its 1536-d fixtures
  can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern
  legacy-embedding-preload.ts documents for 1536-d fixture tests).
- db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated
  process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation
  rule R1. Convert to withEnv().

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-09 22:28:44 -07:00
Garry Tan
1666ec427e v0.39.2.0 feat(autopilot): per-source fan-out + cycle lock primitive + phase taxonomy (#1295) 
* feat(source-id): canonical dependency-free source_id validator module

New src/core/source-id.ts consolidates the three regex sites that drifted
across the codebase (utils.ts permissive, sources-ops.ts strict,
source-resolver.ts strict). Exports:
  - SOURCE_ID_RE: ^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$ (strict kebab,
    1-32 chars, no underscores, alphanumeric boundaries)
  - isValidSourceId(s): boolean — for silent-fallback tiers (dotfile,
    brain_default config)
  - assertValidSourceId(s): void, throws — for explicit-validation tiers
    (explicit --source flag, GBRAIN_SOURCE env, cycleLockIdFor primitive)

Dependency-free by design (no engine imports), so both PGLite and
Postgres engines can pull it without circular-import risk. Replaces
the soon-to-be-removed local validators in utils.ts and sources-ops.ts;
preserves both call shapes (boolean + throwing) per the codex outside-voice
finding that resolver tiers need both.

19 unit tests covering valid ids, length boundary (32-char max), underscore
rejection, path-traversal shapes, edge hyphens, whitespace, non-ASCII,
non-string inputs, and TypeScript narrowing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(source-id): migrate three regex sites to canonical source-id.ts

Consolidates source_id validation through src/core/source-id.ts:

- src/core/utils.ts: validateSourceId is now a back-compat re-export of
  assertValidSourceId. Regex TIGHTENS from permissive ^[a-z0-9_-]+$ to
  the strict kebab. The path-safety boundary now matches what sources-ops
  enforces at source creation time; no production source IDs break because
  sources-ops always rejected underscored IDs at creation. Picks up the
  blast-radius callers in cycle/patterns.ts and cycle/synthesize.ts
  reverse-write paths.

- src/core/sources-ops.ts: deletes local SOURCE_ID_RE + validateSourceId;
  imports isValidSourceId from source-id.ts. Keeps the thin SourceOpError-
  wrapping validator so `gbrain sources add` keeps its user-facing error
  envelope.

- src/core/source-resolver.ts: imports SOURCE_ID_RE + isValidSourceId from
  source-id.ts. Per codex outside-voice P1-F, silent-fallback tiers
  (dotfile read at tier 3, brain_default config at tier 5) use
  isValidSourceId so an invalid dotfile/config value falls through to the
  next resolver tier instead of throwing. Explicit + env tiers keep their
  inline regex-test-and-throw shape because they need tailored error
  messages.

Behaviour: validateSourceId('snake_id') NOW THROWS where pre-PR it accepted.
Documented as the intentional tightening; no existing IDs in production
contain underscores.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): per-source lock primitive + db-lock consolidation + PGLite ordering

Three intertwined cycle.ts changes that landed as one logical unit:

1) DELETE acquirePostgresLock + acquirePGLiteLock (~75 LOC of duplicated
   UPSERT-with-TTL SQL). Replace with tryAcquireDbLock from
   src/core/db-lock.ts, which was extracted in v0.22.13 and should have
   been adopted here at that time. New acquireDbCycleLock(engine, sourceId)
   is a 6-line adapter that keeps cycle.ts's LockHandle shape.

   Deliberately uses tryAcquireDbLock NOT withRefreshingLock (codex r2 P0-A):
   - tryAcquireDbLock returns null on busy → cycle returns
     {status:'skipped', reason:'cycle_already_running'} (existing contract)
   - withRefreshingLock throws → would convert busy cycles into failures
   - withRefreshingLock's background timer would skip Minion job-lock
     renewal (codex r2 P0-B) and add in-phase DB traffic on PGLite's
     single connection (codex r2 P1-A)

2) Add cycleLockIdFor(sourceId?: string) primitive:
   - undefined → 'gbrain-cycle' (legacy default, back-compat for autopilot
     and every existing caller)
   - valid kebab → 'gbrain-cycle:<source_id>' (per-source DB lock row)
   - invalid → throws via assertValidSourceId (codex r2 P1-B defense-
     in-depth at the primitive layer, since CycleOpts.sourceId is a new
     direct API surface that becomes part of a DB lock ID AND a PGLite
     file path component)

   Add CycleOpts.sourceId; thread through to acquireDbCycleLock. Documents
   that this only scopes the LOCK — embed/orphans/purge/etc remain
   brain-global per PHASE_SCOPE.

3) PGLite file+DB ordering invariant (codex r2 P0-C + P0-D):
   - PGLite engines acquire the GLOBAL file lock (cycle.lock, no source
     suffix) BEFORE the per-source DB lock. PGLite's process-level
     write-lock is the single-writer guard; per-source DB lock IDs
     alone would let two PGLite cycles run concurrently.
   - File lock release on DB acquisition failure (cleanup guarantee)
   - Compose both handles into one LockHandle whose release() is
     reverse-of-acquire (DB first, file last) so file lock isn't released
     while DB lock is still live.
   - Postgres engines skip the file lock entirely — per-source DB IDs
     are the full granularity.

13 unit tests in test/cycle-lock-per-source.test.ts pin the back-compat
default, per-source ID shape, distinct-ID property, and the internal-
validation throws.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): PHASE_SCOPE taxonomy + doctor cycle_phase_scope check

Static documentation of each cycle phase's scope: 'source' (safe to
parallelize per source), 'global' (must serialize brain-wide), or
'mixed' (per-phase decomposition needed before parallelizing).

The PHASE_SCOPE record is the load-bearing input for any future
autopilot fan-out wave. It surfaces what codex round-1 P0-1 was
warning about: not all 14 cycle phases are source-scoped today.
embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile
walk brain-wide regardless of sourceId. Per-source cycle LOCKS (this
PR) let two cycles RUN concurrently, but global-scoped phases inside
each will still touch the same rows.

The taxonomy is documentation, not runtime enforcement (runtime
enforcement deferred per plan; filed as TODO).

New doctor check cycle_phase_scope renders the taxonomy as an
operator-facing message AND surfaces phase_scope_map under
Check.details for JSON consumers. Added optional Check.details field
to the doctor types — mirrors PhaseResult.details. Additive; no
schema_version bump.

11 unit tests across phase-scope-coverage + doctor-cycle-phase-scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(source-id): blast-radius regression for strict-regex callers

Pins the codex round-2 P1-D finding: utils.validateSourceId is also
used in cycle reverse-write paths at patterns.ts:263 and
synthesize.ts:909. Pre-PR they used the permissive regex; post-PR
they share the strict kebab regex with sources-ops creation-time
validation. Existing underscore IDs would fail at THOSE cycle sites,
not just at source add/remove.

Structural assertions guard against future drift:
- utils.ts validateSourceId === assertValidSourceId from source-id.ts
- patterns.ts + synthesize.ts both import validateSourceId from utils
- validation call precedes the join() at both reverse-write sites
- utils.ts no longer contains the inline ^[a-z0-9_-]+$ permissive regex
- utils.ts re-exports assertValidSourceId-as-validateSourceId from source-id.ts

9 cases pin the contract. IRON-RULE: source-text grep regressions land
on the offending refactor first.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(engine): listAllSources + updateSourceConfig for per-source autopilot

Two lean engine-layer methods that the v0.38 per-source autopilot wave
consumes. Both have parity implementations on Postgres + PGLite.

listAllSources(opts?):
- Returns the bare SourceRow shape (id, name, local_path, last_sync_at,
  config) without sources-ops.listSources's per-source page_count
  enrichment (N+1 expensive; out of scope for hot-loop callers).
- `includeArchived` defaults false (matches sources-ops semantics).
- `localPathOnly` filters local_path IS NOT NULL so autopilot fan-out
  doesn't dispatch jobs for pure-DB sources whose handler would fall
  back to global sync.repo_path (codex r1 P1-4).
- Ordering: (id = 'default') DESC, id — same as sources-ops for
  operator-output stability.

updateSourceConfig(sourceId, patch):
- Atomic JSONB merge via Postgres `config || $patch::jsonb` operator.
  No read-modify-write race; same-key overwrites (no deep merge —
  flat patches only, matches the v0.38 use case of last_full_cycle_at).
- Returns true when a row was updated, false when sourceId doesn't
  exist (best-effort no-op; caller decides how to handle).
- Postgres: sql.json(patch) per the canonical pattern; PGLite:
  JSON.stringify + ::jsonb cast on positional param.

New SourceRow type exported from engine.ts. Imported by both engine
impls. 11 integration tests in test/list-all-sources.test.ts cover
defaults, filters, JSONB round-trip, archived flag, and merge semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): write last_full_cycle_at to sources.config on per-source cycle exit

Closes codex round-1 P0-5 (write site for last_full_cycle_at was
unspecified pre-PR). runCycle's exit hook persists
{ last_full_cycle_at: '<ISO>' } to sources.config JSONB when a
successful per-source cycle completes. Autopilot's v0.38 per-source
fan-out gate reads this field next tick to decide whether to skip a
source (60-min freshness floor).

Conditions for write (all required):
  - opts.sourceId is set — legacy callers without sourceId skip the
    write (autopilot will keep working today via fallback path)
  - engine is non-null — no-DB path skips
  - status is 'ok' / 'clean' / 'partial' — failed/skipped cycles do NOT
    mark a source as fresh (next cycle will redo work)
  - dryRun is false — writes are out of scope

Best-effort: write failure logs a warning but does NOT change the
CycleReport status. The cycle already succeeded by the time we get
here; the cost of missing a stale write is one redundant cycle next
tick, not data loss.

5 PGLite integration tests cover all four gate conditions plus the
"timestamp advances on each successful run" property.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(jobs): autopilot-cycle handler honors source_id + pull + archive recheck

Threads the v0.38 per-source dispatch payload through the autopilot-cycle
handler at src/commands/jobs.ts:1146. Closes three codex round-1 findings:

- P0-2 / P1-B: validates job.data.source_id at handler entry via the
  canonical source-id.ts isValidSourceId boolean check. Malformed
  source_id from a queue replay dead-letters with a clear error
  instead of reaching cycle code.

- P1-2: job.data.pull explicit boolean overrides the legacy hardcoded
  `true`, so per-source dispatch for local-only sources can pass
  pull: false (no git network round-trip for sources without remote_url).
  Missing/undefined preserves the legacy true for back-compat with cron/
  launchd callers that don't know about the new field.

- P1-5: archived-source recheck happens BEFORE runCycle is invoked
  (cheap SELECT archived FROM sources WHERE id = $1). If the source was
  archived between fan-out and worker claim, handler returns
  { status: 'skipped', reason: 'source_archived' } cleanly — no lock
  acquired, no phases run, no last_full_cycle_at touched. Same skip
  shape for source_not_found (deleted between dispatch and claim).

7 PGLite integration tests cover all five paths (legacy / valid /
not-found / archived / malformed-source_id / non-string source_id /
pull: false override).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): per-source fan-out dispatch (the headline parallelism win)

The headline change of the v0.38 federated-sync wave. Replaces autopilot's
single-job-per-tick dispatch with per-source fan-out so a 5-source
federated brain refreshes in ~5min wall-clock instead of ~25min sequential.

src/commands/autopilot-fanout.ts (new) — pure-function dispatch helper:
- resolveFanoutMax(engine): PGLite=1 (codex P1-3 — preserves single-writer
  invariant), Postgres=4, operator override via autopilot.fanout_max_per_tick
- readLastFullCycleAt(src): JSONB→Date with NULL/unparseable safety
- isSourceStale(src, now?, floorMin?): 60-min default freshness floor
- selectSourcesForDispatch(sources, fanoutMax): stale-only + oldest-first
  + alphabetical tiebreaker (deterministic for tests)
- dispatchPerSource(engine, queue, opts): the orchestrator

src/commands/autopilot.ts (modified): the existing shouldFullCycle branch
calls dispatchPerSource. Behavior preserved:
- Healthy + recent (60min floor) → sleep (unchanged)
- Targeted-plan path → unchanged (uses computeRecommendations)
- Full-cycle path → NOW fans out per-source rather than ONE job for default

Per-source dispatch shape:
- Idempotency key: `autopilot-cycle:<source_id>:<slot>` — two ticks for
  the same source within one slot coalesce; different sources never collide
- pull: !!source.config.remote_url — remotes pull, local-only don't
- maxWaiting: 1 per submit — backpressure when worker can't drain
- Per-submit try/catch (codex E1 F1) — one source's failure doesn't
  abort the tick; surfaces as fanout_submit_failed event

Fallback path: empty `sources` table (pre-v0.18 brain or fresh install
before `gbrain sources add`) falls back to the legacy single autopilot-
cycle job with no source_id, preserving today's single-source behavior.

JSON event stream extended:
- `dispatched` event gains source_id + mode='per_source' fields
- new `fanout_summary` event per tick with dispatched/skipped_fresh/
  skipped_cap arrays so operators can see what the tick did
- new `fanout_cap_reached` event when sources overflow the cap

Caveat (intentional, codex r1 P0-1 scope): per-source LOCKS let two
cycles RUN concurrently, but several phases (embed, orphans, purge,
resolve_symbol_edges, grade_takes, calibration_profile) still walk the
brain globally inside each cycle. PHASE_SCOPE taxonomy from the prior
commit documents this. Genuine per-phase per-source isolation is the
deferred Phase 2 follow-up.

27 unit tests in test/autopilot-fanout.test.ts pin every branch (stale
gate, cap behavior, idempotency keys, legacy fallback, per-submit error
isolation, oldest-first sort, alphabetical tiebreaker).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(doctor): cycle_freshness check — sibling to sync_freshness

New check_cycle_freshness sibling to checkSyncFreshness. Where
sync_freshness reads sources.last_sync_at (one phase), this check
reads sources.config->>'last_full_cycle_at' — the canonical
"this whole cycle completed" timestamp the v0.38 runCycle exit hook
writes and the v0.38 autopilot fan-out gate reads.

Operator sees exactly what autopilot sees when deciding to skip a
source. Default thresholds tighter than sync_freshness (6h warn /
24h fail vs 24h/72h) because full-cycle staleness compounds: sync
stale → extract stale → embed stale → search returns stale results.

Env overrides:
- GBRAIN_CYCLE_FRESHNESS_WARN_HOURS (default 6)
- GBRAIN_CYCLE_FRESHNESS_FAIL_HOURS (default 24)

Edge cases covered (9 PGLite integration tests):
- empty (no federated sources) → ok
- last_full_cycle_at present + fresh → ok
- last_full_cycle_at present + warn window → warn
- last_full_cycle_at present + fail window → fail
- last_full_cycle_at NULL (never cycled) → fail
- mixed severity → highest wins
- future timestamp (clock skew) → warn
- unparseable timestamp → warn
- local_path NULL sources filtered (codex P1-4 parity)

Failure messages embed source.id so the printed fix command
`gbrain dream --source <id>` matches what the user copy-pastes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: fill gaps surfaced by post-Phase-2 audit (resolver silent-fallback + PGLite ordering)

Two test gaps identified by the test-coverage audit after Phase 1 + 2 shipped:

1. src/core/source-resolver.ts — the migration to isValidSourceId for
   silent-fallback tiers (dotfile read, brain_default config) had no
   direct test. Pre-PR these used inline regex; post-PR they use the
   canonical isValidSourceId. Codex P1-F intent (silent fallback for
   invalid input on tiers 3+5, throw on tiers 1+2) deserved an explicit
   test.

   test/source-resolver-silent-fallback.test.ts (12 cases):
     - tier 3: valid dotfile honored; underscore/whitespace/uppercase
       silently falls through to next tier
     - tier 5: valid brain_default honored; underscore + 33+ char silently
       falls through
     - tier 1: valid explicit --source returns; underscore/whitespace
       THROWS (contract distinction)
     - tier 2: valid env GBRAIN_SOURCE returns; underscore THROWS

2. src/core/cycle.ts — the PGLite file+DB ordering invariant (codex r2
   P0-C + P0-D) was implemented in Phase 1 (T5) but had no test pinning
   the ordering / cleanup / per-source DB lock ID semantics.

   test/cycle-pglite-lock-ordering.test.ts (6 cases):
     - global file lock acquired during PGLite cycle
     - cycle for source A then B serializes (file lock held in turn)
     - DB-lock acquire failure releases file lock cleanly (no stranded state)
     - engine=null path still uses file lock
     - DB lock row uses per-source ID (gbrain-cycle:<source>) not legacy
     - consecutive cycles can re-acquire both locks (release-on-exit works)

Plus .context/PHASE_3_ASSESSMENT.md (gitignored) documenting why each
Phase 3 DRY refactor item from the original plan is deferred: each item
on closer inspection is either a premature abstraction or was explicitly
rejected by the original author (BaseCyclePhase header comment).

18 new tests; 0 fails. typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: autopilot wiring static guard + Postgres parity e2e for fan-out

Closes the remaining test gaps identified by the post-Phase-2 audit:

test/autopilot-fanout-wiring.test.ts (5 cases) — static-shape regression
for autopilot.ts ↔ dispatchPerSource. The fan-out helper itself has 27
unit tests; this file pins the WIRING in autopilot.ts:
  - imports dispatchPerSource + resolveFanoutMax
  - calls dispatchPerSource inside the shouldFullCycle branch (not the
    targeted-plan path)
  - updates lastFullCycleAt after dispatch
  - does NOT regress to the pre-PR single-job dispatch (regex-grep guard
    against the legacy `autopilot-cycle:${slot}` idempotency-key shape
    reappearing in autopilot.ts)

Same canonical static-shape pattern as test/autopilot-supervisor-wiring.test.ts.

test/e2e/list-all-sources-postgres.test.ts (10 cases) — Postgres parity
for engine.listAllSources + updateSourceConfig. The PGLite path has
unit-level coverage; the Postgres path has separate impls (sql.json
serialization, sql.count semantics) that could drift. Specifically pins:
  - returns rows, filters archived/localPath correctly
  - JSONB config parses to object (autopilot reads last_full_cycle_at)
  - default source sorts first
  - updateSourceConfig: not-found returns false, patch merges, same-key
    overwrites, idempotent on repeat
  - jsonb_typeof regression: round-trip stores real JSONB object, NOT
    a JSON-encoded string (feedback_postgres_jsonb_double_encode class)

test/e2e/autopilot-fanout-postgres.test.ts (6 cases) — end-to-end
integration on Postgres:
  - 3 sources fan out as 3 distinct jobs with per-source idempotency keys
  - re-dispatch within same slot dedupes (idempotency-key coalesce)
  - last_full_cycle_at < 60min ago sources are skipped by gate
  - end-to-end: updateSourceConfig → listAllSources → selectSourcesForDispatch
    correctly classifies fresh sources
  - fan-out cap honored (5 sources, fanoutMax=2 → 2 dispatched)
  - empty federated brain falls back to legacy single-job dispatch

21 new test cases. Brings the v0.38 wave coverage to 132 unit + 16 e2e.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(autopilot): drop maxWaiting from per-source submit (E2E found silent coalesce)

The Postgres E2E for fan-out surfaced two real bugs that the unit-stub
tests + PGLite tests couldn't catch:

1) **CRITICAL** — dispatchPerSource passed `maxWaiting: 1` to every
   per-source queue.add. maxWaiting is per-(name, queue) — since all
   per-source jobs share `name='autopilot-cycle'`, the second + third +
   Nth source's submit silently coalesced into the FIRST source's
   waiting job. Net result on a real worker: 1 job processed per tick,
   not N. The entire fan-out feature was a silent no-op past the first
   source.

   Per-source idempotency_key (`autopilot-cycle:<source_id>:<slot>`)
   already handles "two ticks for same source within slot" dedup, which
   is the only thing maxWaiting was buying us. Dropping it fixes fan-out
   without losing dedup.

   New unit-stub regression test asserts maxWaiting is NOT in the per-
   source submit opts so a future refactor that re-adds it gets caught
   in 100x faster CI (test/autopilot-fanout.test.ts).

2) **postgres-engine.ts:updateSourceConfig** — initial impl used
   sql.json() correctly but my mid-debug rewrite to executeRaw +
   positional `$1::jsonb` produced JSONB STRING shape (not OBJECT)
   because postgres-js double-encodes JS string params in unsafe mode.
   `||` between JSONB object + JSONB string yields a JSONB ARRAY,
   wiping every existing config key on update.

   Same latent bug class exists at src/commands/sources.ts:482 (gbrain
   sources federate/unfederate path); flagged for follow-up but
   out-of-scope here.

   Reverted to sql.json() inside the template tag (verified via direct
   psql round-trip: jsonb_typeof = 'object'). Updated the e2e seed
   helper to use sql.json() too — the executeRaw + JSON.stringify
   pattern was producing string-shape JSONB at SEED time which made
   the failure cascade harder to debug.

Coverage adds:
- test/e2e/list-all-sources-postgres.test.ts: 11 cases pin Postgres
  parity for listAllSources + updateSourceConfig including jsonb_typeof
  round-trip
- test/e2e/autopilot-fanout-postgres.test.ts: 6 cases end-to-end
  including 3-source fan-out producing 3 distinct rows, idempotency
  coalesce within slot, cap honored, legacy fallback path
- test/autopilot-fanout.test.ts: +1 regression guard on maxWaiting

This is the kind of bug that justifies the user's "fill test gaps then
run E2E" mandate. The unit tests + PGLite parity tests + typecheck all
passed cleanly; only the real-Postgres E2E found the coalesce.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): update multi-source-bug-class validateSourceId expectations for strict regex

The v0.32.8 test pinned the OLD permissive ^[a-z0-9_-]+$ behavior including
the underscored case 'jarvis_memory'. The v0.38 wave (this PR's E2 + codex
P1-D) tightens validateSourceId to the strict kebab regex shared with
sources-ops. 'jarvis_memory' now lives in the rejected set, not the
allowed set.

Updated the test to assert the new contract:
- Replaced 'jarvis_memory' allowed case with 'jarvis-memory' (kebab)
- Added 'a' (single-char) to allowed cases
- Added 'jarvis_memory', 'snake_case', '-leading', 'trailing-', and a
  33-char string to the rejected cases — the v0.38 strict-regex additions

Comment explains the contract shift so future readers don't see the test
as flapping intent.

Found by running the main E2E suite — the test file is in the canonical
e2e set and would have failed CI otherwise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.39.3.0: per-source autopilot fan-out + cycle lock primitive + phase taxonomy

VERSION + CHANGELOG bump for the parallel-federated-sync wave (14 commits).

Headline change: federated brains refresh all sources in parallel via
per-source autopilot dispatch instead of one source per 5-min tick.
Five-source brain wall-clock: ~25min → ~5min.

Test infra adjustments for the v0.38 test-isolation lint that landed via
upstream master merge:
  - test/source-resolver-silent-fallback.test.ts now uses withEnv() for
    GBRAIN_SOURCE mutations (was direct process.env mutation)
  - test/cycle-pglite-lock-ordering.test.ts → .serial.test.ts (the
    file-wide GBRAIN_HOME setup needs quarantine from the parallel pool)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): bump cycle-abort handler-source-grep window 2000→6000

CI on c835f93f failed on 1 of 2005 tests: the v0.20.5
"autopilot-cycle handler contract" regression guard at
test/cycle-abort.test.ts:99 does a source-grep over the first 2000
chars after `worker.register('autopilot-cycle'` to assert `signal:
job.signal` appears.

The v0.38 wave (in c835f93f) added source_id validation + archive
recheck + pull-flag threading at the top of that handler, pushing the
runCycle({signal: job.signal}) call from ~chars 800 to ~chars 2100.
The slice cut off before reaching it, so the assertion failed even
though the code still propagates the signal correctly (line 1213).

Fix: bump the window to 6000 chars and add a comment explaining why.
The guard's intent is unchanged ("handler passes job.signal to
runCycle"); the window just needs to be wide enough to span any
reasonable handler. Also added an existence check on `handlerStart`
so a future refactor that renames the register call surfaces a
clearer error than `undefined.toContain`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(merge): add schema-suggest to PHASE_SCOPE taxonomy

The v0.39.1.0 master merge added a new 'schema-suggest' cycle phase.
PHASE_SCOPE is a TS Record<CyclePhase, PhaseScope> so the compiler
required an entry. Classified as 'source' — the phase accepts
sourceId and operates per-source via runSuggest().

* chore(version): renumber v0.39.3.0 → v0.39.2.0

Renumber to claim the next available slot after v0.39.1.0 (schema packs)
landed on master. No code changes.

* fix(test): bump PHASE_SCOPE count 16→17 for schema-suggest

The v0.39.1.0 master merge added the 17th cycle phase ('schema-suggest').
The previous commit added it to PHASE_SCOPE; this commit updates the
count-pin regression test to match.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 21:04:30 -07:00
Garry Tan
5d9dc4393e v0.22.10 fix: autopilot-cycle handler forwards job.data.phases to runCycle (#521) 
* fix: autopilot-cycle handler forwards job.data.phases to runCycle

The autopilot-cycle handler always ran ALL_PHASES regardless of job data.
This caused production stalls when the embed phase had a large backlog
(17K+ stale chunks) that exceeded the 30-minute job timeout. Every 5-min
cycle would start, hit the embed wall, stall, and get force-killed —
creating an infinite stall loop that kept the queue perpetually unhealthy.

The fix validates job.data.phases against ALL_PHASES (preventing injection)
and forwards the selected phases to runCycle(). Callers can now submit
fast cycles (lint+backlinks+sync+extract) on a 5-min cron and run embed
separately with a longer timeout during off-peak hours.

If phases is omitted, not an array, or filters to empty, behavior is
unchanged (all phases run).

Tests: 4 new cases covering phase restriction, invalid name filtering,
empty array fallback, and non-array type safety.

* test: widen autopilot-cycle handler-block window for phases-passthrough

The regression guard sliced the first 500 chars after `worker.register('autopilot-cycle'`
and asserted `signal: job.signal` was present. The phase-validation block added in
787ec7de pushed the signal arg past that boundary, so CI test shard 3 failed even
though the handler still propagates the signal correctly. Bump the window to 2000.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.22.10)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: sync release notes for v0.22.10

Note autopilot-cycle phases passthrough fix on the src/commands/jobs.ts
key-files annotation so future readers know the handler honors
job.data.phases (validated against ALL_PHASES) as of v0.22.10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt for v0.22.10 CLAUDE.md update

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-29 22:13:05 -07:00
Garry Tan
e2961c04bd v0.22.1 autopilot fix wave — 5 prod hotfixes (#417, #403, #406, #363, #409) (#447) 
* fix: propagate AbortSignal to runCycle + worker force-eviction safety net

Root cause: autopilot-cycle handler called runCycle() without passing
the job's AbortSignal. When the per-job timeout fired abort(), runCycle
never checked it and kept grinding through extract (54,605 pages).
The executeJob promise never resolved, inFlight never decremented, and
the worker thought it was at capacity forever — 98 jobs piled up waiting
with 0 active while a live worker sat idle.

Three-layer fix:

1. CycleOpts.signal: new optional AbortSignal field. runCycle checks it
   between every phase via checkAborted(). A timed-out cycle now bails
   after the current phase completes instead of running all 6 phases.

2. autopilot-cycle handler: passes job.signal to runCycle so the abort
   actually propagates.

3. Worker safety net: 30s after the abort fires, if the handler still
   hasn't resolved, force-evict from inFlight and mark as dead in DB.
   This is the last-resort escape hatch for any handler that ignores
   AbortSignal — the worker resumes claiming new jobs instead of
   wedging forever.

Incident: 2026-04-24, 98 waiting / 0 active / worker alive but idle.
143 existing minions tests pass unchanged.

* test: abort signal propagation + worker recovery regression tests

16 new tests across 3 files covering the 2026-04-24 worker wedge:

test/minions.test.ts (6 new, 149 total):
  - handler receiving abort signal exits cleanly
  - handler ignoring abort still gets signal delivered
  - worker claims new jobs after timeout (no wedge) ← key regression
  - checkAborted pattern: undefined/non-aborted/aborted signals

test/cycle-abort.test.ts (7 new):
  - CycleOpts.signal type contract
  - runCycle accepts signal without error
  - runCycle bails on pre-aborted signal
  - runCycle bails mid-flight when signal fires between phases
  - Source-level guard: jobs.ts passes job.signal to runCycle
  - Source-level guard: worker.ts has force-eviction safety net
  - Source-level guard: cycle.ts has checkAborted between all 6 phases

test/e2e/worker-abort-recovery.test.ts (3 new):
  - worker recovers from timed-out handler and processes next job
  - concurrency=2 processes parallel jobs during timeout
  - multiple sequential timeouts don't permanently wedge worker

All 159 tests pass.

* perf: incremental extract — only process slugs that sync touched

The autopilot-cycle runs every 5 min. Its extract phase was doing a full
filesystem walk of ALL markdown files (54K+) — twice (links + timeline).
On a brain this size, extract alone exceeded the 600s job timeout,
producing zero useful writes.

Fix: sync already returns pagesAffected (the slugs it added/modified).
Pipe that list through to extract. When provided, extract reads ONLY
those files instead of walking the entire brain directory.

- Add ExtractOpts.slugs for targeted extraction
- Add extractForSlugs() — single-pass links + timeline for specific slugs
- cycle.ts: capture sync's pagesAffected, pass to runPhaseExtract
- If sync didn't run or failed, extract falls back to full walk (safe)
- If pagesAffected is empty (nothing changed), extract returns instantly

Expected improvement: 54K file reads → ~10-50 per cycle. The full walk
is still available via CLI `gbrain extract` and on first-run.

* fix: connection resilience for minion supervisor + worker

Three fixes for the minion supervisor dying silently when PgBouncer rotates:

1. PostgresEngine: executeRaw retries once on connection-class errors
   (ECONNREFUSED, password auth failed, connection terminated, etc.)
   by tearing down the poisoned pool and creating a fresh one via
   reconnect(). Prevents cascading failures when Supabase bounces.

2. Supervisor: tracks consecutive health check failures. After 3 in a
   row, emits health_warn with reason=db_connection_degraded and attempts
   engine.reconnect() if available. Resets counter on success.

3. Supervisor: worker_exited events now include likely_cause field:
   SIGKILL → oom_or_external_kill, SIGTERM → graceful_shutdown,
   code=1 → runtime_error. Makes it trivial to distinguish OOM kills
   from connection deaths in logs.

Tests: 23 new tests covering connection error detection, reconnect
guard against concurrent reconnects, retry-once-not-infinite-loop,
health failure tracking, and exit classification.

* fix(db): set session timeouts on every connection to kill orphan backends

Prevents the failure mode from #361: a single autopilot UPDATE on
minion_jobs can leave a pooler backend in state='active'/ClientRead
for 24h+, holding a RowExclusiveLock that blocks every subsequent
ALTER TABLE minion_jobs. The stuck backend never times out on its
own because Supabase Micro has no default idle_in_transaction_session_timeout
and autovacuum can't reap sessions that hold active locks.

Fix: deliver statement_timeout + idle_in_transaction_session_timeout
as startup parameters via postgres.js's `connection` option, applied
automatically on every new backend connection. Works correctly on
both session-mode and transaction-mode PgBouncer poolers (startup
params persist for the backend's lifetime, unlike SET commands
which transaction-mode PgBouncer strips between transactions).

Defaults chosen conservatively so they don't interfere with bulk
work like multi-minute embed passes or CREATE INDEX on large pages
tables:
  - statement_timeout: '5min'
  - idle_in_transaction_session_timeout: '2min'

Each overridable per-GUC via env var (GBRAIN_STATEMENT_TIMEOUT,
GBRAIN_IDLE_TX_TIMEOUT). Set any to '0' or 'off' to disable.

client_connection_check_interval is the specific GUC that would
kill the observed state='active'/ClientRead case, but it's
Postgres 14+ and some managed poolers reject unknown startup
parameters. Made it opt-in only via GBRAIN_CLIENT_CHECK_INTERVAL
for users who know their Postgres supports it.

Applied in both the module-level singleton connect (src/core/db.ts)
and the per-engine-instance pool used by `gbrain jobs work`
(src/core/postgres-engine.ts) via a shared resolveSessionTimeouts()
helper.

Tests: 5 new cases in migrate.test.ts covering defaults, env
overrides, '0'/'off' disable, and multi-GUC disable. 39/39 pass
(34 pre-existing + 5 new).

Closes #361.

Co-Authored-By: orendi84 <orendigergo@gmail.com>

* fix(embed): server-side staleness filter for embed --stale (v0.20.5)

embed --stale walked listPages + per-page getChunks (incl. vector(1536)
embedding column) on every call, then client-side-filtered for chunks
where embedding was missing. On a 1.5K-page brain at 100% coverage, ~76 MB
pulled per call, all discarded. With autopilot firing every 5-10 min plus
a 2h cron, this hit Supabase's 5 GB free-tier ceiling at 102 GB used
(2058% over) twice in one week.

Two new BrainEngine methods replace the page walk with a SQL-side filter:
- countStaleChunks(): single SELECT count(*) WHERE embedding IS NULL.
  Pre-flight short-circuit; ~50 bytes wire when 0 stale.
- listStaleChunks(): slug + chunk_index + chunk_text + chunk_source +
  model + token_count for stale rows only. Excludes the (NULL) embedding
  column. Bounded by LIMIT 100000 mirroring listPages.

embedAll forks: staleOnly=true takes the new SQL-side path
(embedAllStale); staleOnly=false (--all) keeps existing behavior verbatim.

embedAllStale preserves non-stale chunks on partially-stale pages: it
re-fetches existing chunks per stale slug and merges (embedding=undefined
for non-stale → COALESCE preserves existing). Without the merge, the
upsertChunks != ALL filter would delete non-stale chunks. Re-fetch cost
is bounded by stale slug count; the autopilot common case (0 stale)
never reaches this path.

Predicate uses `embedding IS NULL`, not `embedded_at IS NULL`. The bulk-
import path could leave embedded_at populated while embedding was NULL
(see upsertChunks consistency fix below), so `embedding IS NULL` is the
truth source for "this chunk needs an embedding".

Also fixes the upsertChunks consistency bug in both engines: when
chunk_text changes and no new embedding is supplied, embedding correctly
clears to NULL but embedded_at kept its old timestamp. New behavior
resets BOTH columns together, keeping write-time honesty.

Wire-cost impact (measured against current behavior on a 1.5K-page brain):
- 0 stale chunks (autopilot common case): ~76 MB → ~50 bytes (~1.5M× reduction)
- 100 stale across 10 pages: ~76 MB → ~150 KB (~500× reduction)
- 8K stale across 1.5K pages (cold start): ~76 MB → ~12 MB (~6× reduction)

Tests: 4 new in test/embed.test.ts (zero-stale short-circuit; N-stale-
across-M-pages with non-stale preservation; --stale dry-run; --all path
byte-identical). Existing --stale tests updated for the new mock surface.

Migration impact: none. embedded_at and embedding columns have been on
content_chunks since schema inception.

Co-Authored-By: atrevino47 <atbuster47@gmail.com>

* chore(wave): post-merge tightening — drop executeRaw retry (D3) + gate noExtract (F2)

- Drop #406's per-call executeRaw retry wrapper. The regex idempotence
  boundary is unsound (writable CTEs, side-effecting SELECTs). Recovery
  now happens at the supervisor level via 3-strikes-then-reconnect.
- Update db.ts: setSessionDefaults becomes a back-compat no-op.
  resolveSessionTimeouts (from #363) is the source of truth, sending
  GUCs as startup parameters that survive PgBouncer transaction mode.
  Bumped idle_in_transaction default from 2min to 5min to match v0.21.0
  posture.
- Gate noExtract in cycle's runPhaseSync on whether extract phase is
  scheduled. Avoids silently dropping extraction when the user runs
  `gbrain dream --phase sync` (Codex F2).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(db): rephrase docstring to avoid false-positive in test source-grep

The migrate.test.ts structural check counts `SET idle_in_transaction_session_timeout`
matches in source. The literal string in this docstring was tripping it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: backfill regression guards for #417, D3, F2 (Step 5)

15 new test cases across 3 files, ~250 LOC, all PGLite/in-memory:

test/extract-incremental.test.ts (NEW, 8 cases for #417):
- slugs: [] returns immediately (early-return)
- slugs: undefined falls through to full-walk
- slugs: [a, b] reads only those files
- Slug whose file no longer exists is silently skipped
- Mode filter (links) skips timeline extraction
- dryRun: true does not invoke addLinksBatch / addTimelineEntriesBatch
- BATCH_SIZE flush — >100 candidate links exercise mid-iteration flush
- Full-slug-set resolution — link to file outside changed set still resolves

test/core/cycle.test.ts (4 new cases for #417 + Codex F2):
- cycle threads sync.pagesAffected into extract phase as the slugs argument
- extract phase falls back to full walk when sync was skipped
- F2 guard: full cycle (sync + extract) sets noExtract=true on sync
- F2 guard: phases:[sync] only sets noExtract=false (no silent extract drop)

test/connection-resilience.test.ts (3 new cases for D3):
- PostgresEngine.executeRaw is a single-statement passthrough (no try/catch)
- PostgresEngine.reconnect() still exists for supervisor-driven recovery
- Supervisor still has the 3-strikes-then-reconnect path

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(wave): v0.21.1 release notes + 3 follow-up TODOs + CLAUDE.md updates

CHANGELOG.md: segment-aware entry per CEO-review D1 — 'For everyone'
section (#417 incremental extract, #403 cycle abort) leads, 'For Postgres /
Supabase users' section (#406, #363, #409) follows. Production proof
point as a sidebar, not the lead.

TODOS.md: 3 follow-up items per Eng-review D6:
  1. Caller-opt-in retry for executeRaw (D3 follow-up)
  2. Replace walkMarkdownFiles with engine.getAllSlugs() (F1 follow-up)
  3. err.code-based connection-error matching (B1 follow-up)

CLAUDE.md: 6 file-reference updates for the wave's behavioral additions
(postgres-engine, db, cycle, worker, supervisor, embed, extract).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): bump version 0.21.1 → 0.22.1 + document version locations

User-explicit version override on /ship: ship as v0.22.1 (MINOR jump from
master's 0.21.0) instead of the v0.21.1 PATCH the wave originally targeted.
The wave bundles 5 production fixes which is meaningful enough to clear a
MINOR version, even though the API surface is additive.

Files updated to 0.22.1:
- VERSION (single source of truth)
- package.json (Bun/npm version)
- CHANGELOG.md (release header + "To take advantage of v0.22.1" block)
- TODOS.md (3 follow-up TODOs reference the version that filed them)
- CLAUDE.md (Key Files annotations cite the release that introduced behavior)

Also adds a "Version locations" section to CLAUDE.md documenting all five
required files plus the auto-derived (bun.lock, llms-full.txt) and
historical (skills/migrations/v*.md, src/commands/migrations/v*.ts,
test/migrations-v*.test.ts) categories. Future /ship runs and the
auto-update agent now have a canonical list of where versions live.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): unbreak CI typecheck — annotate signal as AbortSignal | undefined

CI's `bun run typecheck` step was failing with TS2339 at
test/minions.test.ts:2026 — `const signal = undefined` narrows to literal
`undefined`, which has no `.aborted` property, so `signal?.aborted`
doesn't compile.

Fix uses `as AbortSignal | undefined` to preserve the union type. A
plain type annotation gets narrowed back via control-flow analysis; the
`as` cast doesn't. Runtime behavior is unchanged — the optional-chain
still short-circuits as intended.

Verified: bunx tsc --noEmit → exit 0; the 3 checkAborted cases still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(doctor): forward-progress override for stale minions partials

The minions_migration check reads ~/.gbrain/migrations/completed.jsonl
and flags any version that has a `partial` entry without a matching
`complete`. Long-lived installs accumulate partial records from
historical stopgap runs (notably v0.11.0). Without time decay or
forward-progress detection, the FAIL flag fires forever once any
partial lands, even on installs that have been running clean at
v0.22+ for months.

Concrete failure: test/e2e/mechanical.test.ts "gbrain doctor exits 0
on healthy DB" was flaking on dev machines whose ~/.gbrain/ carried
v0.11.0 partials from earlier in the day. The fresh test DB had
nothing wrong with it; doctor was just reading host filesystem state
that bled in via $HOME.

Fix: a partial vX.Y.Z is treated as stale (not stuck) if any vA.B.C
where A.B.C >= X.Y.Z has a `complete` entry anywhere in the file.
The reasoning: if a newer migration successfully landed, the install
has clearly moved past the older partial. compareVersions() from
src/commands/migrations/index.ts handles the semver compare.

Cases preserved:
- v0.10 complete + v0.11 partial → still FAILs (older complete doesn't
  supersede newer partial)
- v0.16 partial alone → still FAILs (no override exists)
- Fresh install (no completed.jsonl) → no warning
- Real partial-then-complete-same-version → no warning

Cases now fixed:
- v0.16 complete + v0.11 partial → no FAIL (forward progress made;
  the v0.11 record is stale)

Two regression tests in test/doctor-minions-check.test.ts cover both
directions of the override (when it fires, when it doesn't).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(docs): regenerate llms-full.txt after CLAUDE.md updates

CI's build-llms regen-drift guard caught that llms-full.txt was stale
relative to CLAUDE.md after the wave's documentation commits (the
"Version locations" section + 6 file-reference annotations for the
wave's behavioral additions).

CLAUDE.md notes that llms-full.txt is auto-derived — bumped via
'bun run build:llms' when CLAUDE.md's file-references change. This
commit catches up.

llms.txt is unchanged; the curated index doesn't pull from CLAUDE.md's
file-reference body. Only llms-full.txt (the inlined single-fetch
bundle) needed regeneration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: orendi84 <orendigergo@gmail.com>
Co-authored-by: atrevino47 <atbuster47@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 15:49:48 -07:00