v0.42.24.0 fix(minions): route lock claim/renewLock through direct session pool (#1822)

* fix(minions): route lock claim/renewLock through direct session pool

The Minion lock heartbeat (claim + renewLock) ran every UPDATE through
engine.executeRaw(), which is hardcoded to the read pool. On Supabase that
is the transaction-mode pooler (6543), which recycles connections per
transaction. A lock is held for minutes, so the pooler periodically reaps
the socket mid-heartbeat -> CONNECTION_ENDED -> the lock looks expired ->
the worker force-evicts its own job and the claim loop wedges silently.

Add BrainEngine.executeRawDirect(): same contract as executeRaw, but routes
to the direct session-mode pool (5432, GBRAIN_DIRECT_DATABASE_URL) when
dual-pool is active. No-op delegation on PGLite / non-Supabase / kill-switch.
claim/renewLock now use it. Single-statement UPDATEs only, so the double-claim
guard and the renewLock no-inline-retry contract are preserved. Statements
inside an open transaction keep their tx connection (in-transaction guard keys
on peekReadPool() !== _sql); the lock hot-path never runs inside transaction().

The Postgres impl shares its cancellation plumbing with executeRaw via a
private runUnsafe helper. New test/postgres-execute-raw-direct.test.ts covers
the routing decision (dual-pool on/off x in-tx/not + abort short-circuit)
without a live Postgres; queue-lock-retry.test.ts gains a guard that claim
can never fall back to executeRaw.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* v0.42.24.0 chore: bump version and changelog

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.42.24.0

Document executeRawDirect on the BrainEngine contract and the
claim/renewLock direct-session-pool routing in KEY_FILES.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: make D3 executeRaw-no-retry guard refactor-aware

The DRY refactor in this PR extracted executeRaw/executeRawDirect's shared
cancellation plumbing into a private runUnsafe(conn, ...) helper, so the single
conn.unsafe() call moved out of executeRaw's body. The D3 guard read
executeRaw's source and asserted conn.unsafe( appeared exactly once there,
which now fails (it's zero — executeRaw delegates).

The D3 invariant (no per-call retry wrapper) is unchanged; it just spans the
delegate now. Update the guard to check both public methods delegate to
runUnsafe without reconnect/retry, and assert the exactly-once conn.unsafe +
cancel-only catch in runUnsafe. Also extends coverage to executeRawDirect so
the lock hot-path can't reintroduce a retry wrapper either.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-03 22:40:12 -07:00
committed by GitHub
parent f11d56cfca
commit f868257405
12 changed files with 279 additions and 31 deletions

View File

@@ -4879,17 +4879,25 @@ export class PostgresEngine implements BrainEngine {
}
}
async executeRaw<T = Record<string, unknown>>(
/**
* Shared body for executeRaw / executeRawDirect: run a raw statement on the
* given connection and wire AbortSignal cancellation onto the pending query.
* The ONLY difference between the two public methods is which connection they
* pick (read pool vs direct session pool), so the cancellation plumbing lives
* here in one place rather than being copy-pasted.
*
* v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's .cancel()
* on the pending query. Init nudge (3s wallclock cap) is the first consumer;
* the AbortSignal fires when the timer trips. An already-aborted signal
* short-circuits before the network round-trip.
*/
private runUnsafe<T>(
conn: ReturnType<typeof postgres>,
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
const conn = this.sql;
const pending = conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]);
// v0.41.18.0 (A20, codex #7): real cancellation via postgres.js's
// .cancel() on the pending query. Init nudge (3s wallclock cap) is the
// first consumer; the AbortSignal fires when the timer trips.
// Already-aborted signal short-circuits before the network round-trip.
if (opts?.signal) {
if (opts.signal.aborted) {
// .cancel() is fire-and-forget; the awaited query rejects with the
@@ -4905,7 +4913,7 @@ export class PostgresEngine implements BrainEngine {
try {
(pending as unknown as { cancel?: () => void }).cancel?.();
} catch {
// best-effort; the .then below settles regardless
// best-effort; the .finally below settles regardless
}
};
opts.signal.addEventListener('abort', onAbort, { once: true });
@@ -4913,7 +4921,15 @@ export class PostgresEngine implements BrainEngine {
opts.signal?.removeEventListener('abort', onAbort);
});
}
return pending as unknown as T[];
return pending as unknown as Promise<T[]>;
}
async executeRaw<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
return this.runUnsafe<T>(this.sql, sql, params, opts);
// Pre-#406 behavior: throw on any error including connection death.
// Per-call auto-retry is not safe here because executeRaw is also used
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
@@ -4924,6 +4940,34 @@ export class PostgresEngine implements BrainEngine {
// swap in a fresh pool. See db.ts setSessionDefaults / supervisor.ts.
}
/**
* Minion lock hot-path variant of executeRaw. Routes to the DIRECT
* session-mode pool (port 5432) when dual-pool is active so lock
* heartbeats survive the transaction-pooler's per-transaction connection
* recycling. See BrainEngine.executeRawDirect for the full rationale.
*
* When this engine is a transaction-scoped clone (txEngine from
* transaction()), `connectionManager` is inherited but `this.sql` is the tx
* connection; we intentionally honor the tx connection in that case by
* falling through to this.sql, because routing a statement inside an open
* transaction onto a different pool would break atomicity. The lock
* hot-path (claim/renewLock) does NOT run inside transaction(), so in
* practice this always reaches the direct pool there.
*/
async executeRawDirect<T = Record<string, unknown>>(
sql: string,
params?: unknown[],
opts?: { signal?: AbortSignal },
): Promise<T[]> {
// Inside an open transaction, _sql is the reserved tx connection (set via
// defineProperty in transaction()); never reroute off it.
const inTransaction = this._sql !== null && this.connectionManager?.peekReadPool() !== this._sql;
const conn = (!inTransaction && this.connectionManager?.isDualPoolActive())
? await this.connectionManager.ddl()
: this.sql;
return this.runUnsafe<T>(conn, sql, params, opts);
}
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
// ============================================================