v0.42.5.0 fix(minions): RSS watchdog opacity + pooler-reap self-heal + silent lens backlog + cycle lint DB-disconnect (#1678) (#1735)
* fix(minions): self-identifying RSS watchdog + cgroup-aware default + pooler-reap self-heal (#1678) Problem 1: distinct WORKER_EXIT_RSS_WATCHDOG exit code + cause-keyed supervisor breaker (bypasses the stable-run reset that hid the 400x/24h loop) + rss_watchdog audit bucket + 80% soft-warn; cgroup-aware resolveDefaultMaxRssMb replaces the flat 2048 default at every spawn site. Problem 2: CONNECTION_ENDED classified retryable; postgres-engine sql getter throws a retryable error on a reaped instance pool instead of the misleading module-singleton fallthrough; promoteDelayed reconnect-retry; claim recovers on the next poll tick (no double-claim); lock-renewal tick reconnect-once dep. * feat(cycle): surface silent extract_atoms backlog + bounded --drain + fix lint clobbering the shared DB connection (#1678) Problem 3: extract_atoms_backlog doctor check + pack_gated skip marker + shared countExtractAtomsBacklog; `gbrain dream --phase extract_atoms --drain [--window N]` single-hold bounded drain (same cycleLockIdFor, rediscover each batch, reports remaining, exits non-zero while work remains). Also fixes a real production bug found via E2E: the cycle lint phase's resolveLintContentSanity created + disconnected a module-style engine that nulled the shared db singleton mid-cycle, breaking every later phase with "connect() has not been called". Lint now reuses the caller's live engine (cycle + Minion handlers thread it; standalone CLI keeps the create-own path). * chore: bump version and changelog (v0.41.39.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#1678): pre-landing review — route transaction/withReservedConnection through the sql getter + drain treats failed count as incomplete Codex adversarial review findings: - #2: transaction(), withReservedConnection(), and one other site bypassed the v0.42.2.0 sql-getter self-heal via `this._sql || db.getConnection()`, so a reaped instance pool fell through to the module singleton there. Route all three through `this.sql` so they throw the retryable instance-pool error and recover consistently (MinionQueue.transaction hits this). - #4: `gbrain dream --drain` treated a null backlog count (query failure) as success via `remaining ?? 0`; now null exits EXIT_DRAIN_INCOMPLETE so automation never believes an unverified backlog drained. - #1 (claim orphan) + #3 (PGLite drain lock) documented as follow-ups in TODOS. * docs: document v0.42.2.0 #1678 modules + behavior in CLAUDE.md Adds Key Files entries for worker-exit-codes.ts, rss-default.ts, and extract-atoms-drain.ts, plus v0.42.2.0 annotations on worker.ts, child-worker-supervisor.ts, lock-renewal-tick.ts, and dream.ts. Regenerated llms-full.txt to match (test/build-llms.test.ts gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-version v0.42.2.0 → v0.42.5.0 across VERSION/package.json/CHANGELOG/docs/comments Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -121,6 +121,22 @@ export class PostgresEngine implements BrainEngine {
|
||||
// Instance connection (for workers) or fall back to module global (backward compat)
|
||||
get sql(): ReturnType<typeof postgres> {
|
||||
if (this._sql) return this._sql;
|
||||
// issue #1678: an instance-pool engine whose _sql went null (a mid-process
|
||||
// disconnect/reconnect, or a reaped socket) must NOT fall through to the
|
||||
// module singleton — that singleton was never connected on a worker, so
|
||||
// db.getConnection() throws the misleading "connect() has not been called".
|
||||
// Throw a tailored RETRYABLE error instead (isRetryableConnError matches
|
||||
// problem === 'No database connection'), so a caller wrapped in
|
||||
// withRetry+reconnect rebuilds this instance's pool and recovers. The
|
||||
// module / never-connected path (style 'module' or null) keeps the legacy
|
||||
// getConnection() behavior.
|
||||
if (this._connectionStyle === 'instance') {
|
||||
throw new GBrainError(
|
||||
'No database connection',
|
||||
'instance connection pool was torn down (socket reaped or mid-process disconnect)',
|
||||
'Transient — the operation reconnects and retries. If it persists, check pooler/Supavisor health.',
|
||||
);
|
||||
}
|
||||
return db.getConnection();
|
||||
}
|
||||
|
||||
@@ -793,7 +809,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
const conn = this._sql || db.getConnection();
|
||||
const conn = this.sql;
|
||||
return conn.begin(async (tx) => {
|
||||
// Create a scoped engine with tx as its connection, no shared state mutation
|
||||
const txEngine = Object.create(this) as PostgresEngine;
|
||||
@@ -804,7 +820,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
const pool = this._sql || db.getConnection();
|
||||
const pool = this.sql;
|
||||
const reserved = await pool.reserve();
|
||||
try {
|
||||
const conn: ReservedConnection = {
|
||||
@@ -4113,7 +4129,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
oldRow: number,
|
||||
newRow: Omit<TakeBatchInput, 'page_id' | 'row_num' | 'superseded_by'>,
|
||||
): Promise<{ oldRow: number; newRow: number }> {
|
||||
const conn = this._sql || db.getConnection();
|
||||
const conn = this.sql;
|
||||
return await conn.begin(async (tx) => {
|
||||
const [existing] = await tx`
|
||||
SELECT resolved_at FROM takes WHERE page_id = ${pageId} AND row_num = ${oldRow}
|
||||
|
||||
Reference in New Issue
Block a user