v0.42.20.0 fix: reliability wave — PGLite capture lock-pin + Postgres reconnect race + search embed-hang (#1762 #1745 #1775) (#1810)

* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762)

New src/core/background-work.ts registry (Map<name,drainer>, ordered drain,
awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and
eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths
(op-dispatch finally + handleCliOnly finally) drain the registry before
engine.disconnect() so a PGLite db.close() can't race in-flight work into the
re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path
converts process.exit(1) to exitCode+return so the finally still drains.

* fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775)

withDefaultTimeout composes a per-touchpoint default deadline (chat 300s,
embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the
SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch
embed) — covers native-anthropic + retries — plus per-request multimodal fetch.
embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS.

* fix(postgres): module-mode reconnect preserves the shared singleton (#1745)

reconnect() branches on connection style. Module-singleton engines re-establish
idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager
read pool, never db.disconnect() — so a transient blip no longer nulls the shared
sql out from under concurrent ops (which threw 'connect() has not been called').
Fail-loud on real connect failure. Instance pools keep teardown+recreate.

* fix(search): bound the query-time embed so a stall falls back to keyword (#1775)

search/query default to cheap-hybrid (embeds the query); a stalled provider made
the embed never settle, so the keyword fallback never engaged and the command
force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per
embed) covers both the cache-lookup and inner embeds via embedQueryBounded
(abortSignal + Promise.race) → existing keyword fallback engages. Also registers
the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS.

* test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775)

New: background-work registry unit, query-embed deadline unit, eval-capture
drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in
the PGLite serial test. Updated fix-wave-structural assertions to the registry
shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done.

Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout);
the residual hung-Haiku hole is closed by the facts shutdown() abort belt.

Co-Authored-By: ElliotDrel <noreply@github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms)

* chore: bump release version 0.42.11.0 -> 0.42.20.0

Rename the reliability-wave release version per request. Trio
(VERSION / package.json / CHANGELOG) reconciled; in-code version-tag
comments and test fixtures updated; llms regenerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: ElliotDrel <noreply@github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-03 08:42:25 -07:00
committed by GitHub
parent 3d2add15d9
commit ec5fed2921
20 changed files with 1095 additions and 122 deletions

View File

@@ -4781,18 +4781,48 @@ export class PostgresEngine implements BrainEngine {
}
/**
* Reconnect the engine by tearing down the current pool and creating a fresh one.
* No-ops if no saved config (module-singleton mode) or if already reconnecting.
* Reconnect the engine after a transient connection blip. Branches on
* connection style; no-ops if no saved config or if already reconnecting.
*
* v0.42.x (#1685 GAP B): records a pool-recovery audit event so the
* `pool_reap_health` doctor check can answer "reaped N times AND not
* auto-recovering." `ctx.error` (threaded by retry.ts) is classified: a
* CONNECTION_ENDED match is a true pooler reap; anything else (or no error,
* e.g. the supervisor's health-check reconnect) is `reconnect_other`. All
* audit calls are best-effort and never block the reconnect (CODEX #8).
* - MODULE-singleton engines SHARE `db.ts`'s `sql` (#1745). Calling
* `db.disconnect()` here (via `this.disconnect()`) would null it out from
* under EVERY concurrent op (other dream-cycle phases, minion-queue
* `promoteDelayed`), which then throw "connect() has not been called" in the
* disconnect→connect window. postgres.js already auto-replaces dead sockets
* inside its pool, so a transient blip recovers WITHOUT a teardown. Recover
* idempotently instead: `db.connect()` is a no-op when the singleton is alive
* (the common case) and re-establishes it only if some other path nulled it —
* never introducing a null window — then refreshes the ConnectionManager read
* pool. Scope: fixes the singleton-NULL-window bug specifically; it does NOT
* rebuild a genuinely WEDGED-but-live pool (db.connect() no-ops there) — a
* different failure mode postgres.js owns.
*
* - INSTANCE pools (worker engines, `poolSize` set) own their `_sql` — tearing
* it down and rebuilding is correct and isolated; nobody else shares it. This
* path also records a pool-recovery audit event (#1685 GAP B) so the
* `pool_reap_health` doctor check can answer "reaped N times AND not
* auto-recovering." `ctx.error` (threaded by retry.ts) is classified: a
* CONNECTION_ENDED match is a true pooler reap; anything else (or no error,
* e.g. the supervisor's health-check reconnect) is `reconnect_other`. All
* audit calls are best-effort and never block the reconnect (CODEX #8).
*/
async reconnect(ctx?: { error?: unknown }): Promise<void> {
if (!this._savedConfig || this._reconnecting) return;
if (this._connectionStyle !== 'instance') {
// Module-singleton: never tear down the shared pool. db.connect() is
// idempotent (no-op when the singleton is alive — the common #1745 path).
// FAIL-LOUD (codex): do NOT swallow a real connect failure — a swallowed
// error would make reconnect() resolve "successfully" and let the
// supervisor reset its health-failure counter / emit db_reconnected when
// the DB is actually down. A throw propagates as the real cause (matches
// the withRetry+reconnect contract and the instance path's posture).
await db.connect(this._savedConfig);
// If db.connect() RE-CREATED the singleton (another path nulled it), the
// ConnectionManager set at connect-time still points at the ended old
// pool. Refresh it. Idempotent no-op when the singleton was already alive.
this.connectionManager?.setReadPool(db.getConnection());
return;
}
this._reconnecting = true;
let isReap = false;
@@ -4808,9 +4838,8 @@ export class PostgresEngine implements BrainEngine {
} catch { /* audit is best-effort */ }
try {
// Tear down old pool (best-effort — it may already be dead)
// Instance pool: tear down old pool (best-effort — it may already be dead).
try { await this.disconnect(); } catch { /* swallow */ }
// Create fresh pool
await this.connect(this._savedConfig);
try {
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');