v0.42.20.0 fix: reliability wave — PGLite capture lock-pin + Postgres reconnect race + search embed-hang (#1762 #1745 #1775) (#1810)
* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762) New src/core/background-work.ts registry (Map<name,drainer>, ordered drain, awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths (op-dispatch finally + handleCliOnly finally) drain the registry before engine.disconnect() so a PGLite db.close() can't race in-flight work into the re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path converts process.exit(1) to exitCode+return so the finally still drains. * fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775) withDefaultTimeout composes a per-touchpoint default deadline (chat 300s, embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch embed) — covers native-anthropic + retries — plus per-request multimodal fetch. embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS. * fix(postgres): module-mode reconnect preserves the shared singleton (#1745) reconnect() branches on connection style. Module-singleton engines re-establish idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager read pool, never db.disconnect() — so a transient blip no longer nulls the shared sql out from under concurrent ops (which threw 'connect() has not been called'). Fail-loud on real connect failure. Instance pools keep teardown+recreate. * fix(search): bound the query-time embed so a stall falls back to keyword (#1775) search/query default to cheap-hybrid (embeds the query); a stalled provider made the embed never settle, so the keyword fallback never engaged and the command force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per embed) covers both the cache-lookup and inner embeds via embedQueryBounded (abortSignal + Promise.race) → existing keyword fallback engages. Also registers the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS. * test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775) New: background-work registry unit, query-embed deadline unit, eval-capture drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in the PGLite serial test. Updated fix-wave-structural assertions to the registry shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done. Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout); the residual hung-Haiku hole is closed by the facts shutdown() abort belt. Co-Authored-By: ElliotDrel <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms) * chore: bump release version 0.42.11.0 -> 0.42.20.0 Rename the reliability-wave release version per request. Trio (VERSION / package.json / CHANGELOG) reconciled; in-code version-tag comments and test fixtures updated; llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ElliotDrel <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -4781,18 +4781,48 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconnect the engine by tearing down the current pool and creating a fresh one.
|
||||
* No-ops if no saved config (module-singleton mode) or if already reconnecting.
|
||||
* Reconnect the engine after a transient connection blip. Branches on
|
||||
* connection style; no-ops if no saved config or if already reconnecting.
|
||||
*
|
||||
* v0.42.x (#1685 GAP B): records a pool-recovery audit event so the
|
||||
* `pool_reap_health` doctor check can answer "reaped N times AND not
|
||||
* auto-recovering." `ctx.error` (threaded by retry.ts) is classified: a
|
||||
* CONNECTION_ENDED match is a true pooler reap; anything else (or no error,
|
||||
* e.g. the supervisor's health-check reconnect) is `reconnect_other`. All
|
||||
* audit calls are best-effort and never block the reconnect (CODEX #8).
|
||||
* - MODULE-singleton engines SHARE `db.ts`'s `sql` (#1745). Calling
|
||||
* `db.disconnect()` here (via `this.disconnect()`) would null it out from
|
||||
* under EVERY concurrent op (other dream-cycle phases, minion-queue
|
||||
* `promoteDelayed`), which then throw "connect() has not been called" in the
|
||||
* disconnect→connect window. postgres.js already auto-replaces dead sockets
|
||||
* inside its pool, so a transient blip recovers WITHOUT a teardown. Recover
|
||||
* idempotently instead: `db.connect()` is a no-op when the singleton is alive
|
||||
* (the common case) and re-establishes it only if some other path nulled it —
|
||||
* never introducing a null window — then refreshes the ConnectionManager read
|
||||
* pool. Scope: fixes the singleton-NULL-window bug specifically; it does NOT
|
||||
* rebuild a genuinely WEDGED-but-live pool (db.connect() no-ops there) — a
|
||||
* different failure mode postgres.js owns.
|
||||
*
|
||||
* - INSTANCE pools (worker engines, `poolSize` set) own their `_sql` — tearing
|
||||
* it down and rebuilding is correct and isolated; nobody else shares it. This
|
||||
* path also records a pool-recovery audit event (#1685 GAP B) so the
|
||||
* `pool_reap_health` doctor check can answer "reaped N times AND not
|
||||
* auto-recovering." `ctx.error` (threaded by retry.ts) is classified: a
|
||||
* CONNECTION_ENDED match is a true pooler reap; anything else (or no error,
|
||||
* e.g. the supervisor's health-check reconnect) is `reconnect_other`. All
|
||||
* audit calls are best-effort and never block the reconnect (CODEX #8).
|
||||
*/
|
||||
async reconnect(ctx?: { error?: unknown }): Promise<void> {
|
||||
if (!this._savedConfig || this._reconnecting) return;
|
||||
if (this._connectionStyle !== 'instance') {
|
||||
// Module-singleton: never tear down the shared pool. db.connect() is
|
||||
// idempotent (no-op when the singleton is alive — the common #1745 path).
|
||||
// FAIL-LOUD (codex): do NOT swallow a real connect failure — a swallowed
|
||||
// error would make reconnect() resolve "successfully" and let the
|
||||
// supervisor reset its health-failure counter / emit db_reconnected when
|
||||
// the DB is actually down. A throw propagates as the real cause (matches
|
||||
// the withRetry+reconnect contract and the instance path's posture).
|
||||
await db.connect(this._savedConfig);
|
||||
// If db.connect() RE-CREATED the singleton (another path nulled it), the
|
||||
// ConnectionManager set at connect-time still points at the ended old
|
||||
// pool. Refresh it. Idempotent no-op when the singleton was already alive.
|
||||
this.connectionManager?.setReadPool(db.getConnection());
|
||||
return;
|
||||
}
|
||||
this._reconnecting = true;
|
||||
|
||||
let isReap = false;
|
||||
@@ -4808,9 +4838,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
} catch { /* audit is best-effort */ }
|
||||
|
||||
try {
|
||||
// Tear down old pool (best-effort — it may already be dead)
|
||||
// Instance pool: tear down old pool (best-effort — it may already be dead).
|
||||
try { await this.disconnect(); } catch { /* swallow */ }
|
||||
// Create fresh pool
|
||||
await this.connect(this._savedConfig);
|
||||
try {
|
||||
const { logPoolRecovery } = await import('./audit/pool-recovery-audit.ts');
|
||||
|
||||
Reference in New Issue
Block a user