* feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard
Prevents stall-induced queue blockage discovered in production (OpenClaw):
1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms
(or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB
connections where FOR UPDATE SKIP LOCKED stall detection skips them.
2. Submission backpressure (maxWaiting): caps waiting jobs per name at
submission time. Prevents autopilot-cycle flood when the queue is blocked.
3. --no-worker flag for autopilot: skips spawning the built-in worker child.
For environments where the worker lifecycle is managed externally (systemd,
Docker, OpenClaw service-manager).
4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the
worker is spawned by autopilot (which can't pass CLI flags to the child).
5. Shell job env guard with clear logging: shell handler is always registered
but throws UnrecoverableError with a clear message when
GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering.
* feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI
Addresses three production-hardening findings from the CEO + Eng + Codex
adversarial review of PR #379:
D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both
see waitingCount < max and both insert. Wrap the count+select+insert in
pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent
decisions for the SAME key while leaving different keys fully parallel.
Lock auto-releases on txn commit/rollback — no cleanup path to leak.
Also fix the missing queue-scope bug: count and select now filter on
(name, queue) not name alone, so cross-queue same-name jobs don't
suppress each other.
D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from
parseInt. `inFlight.size < NaN` is always false → worker claims nothing →
silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr
warning naming the bad value.
D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the
MinionJobInput field. Wire the flag with clamp [1, 100], mirror
`--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing.
Q1: Silent coalesce was invisible by design. New
src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week
JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce
events write one JSONL line with (queue, name, waiting_count, max_waiting,
returned_job_id, ts). Best-effort — disk-full never blocks submission.
A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case.
Forges a wedged-worker row state, invokes handleStalled + handleTimeouts
+ handleWallClockTimeouts in order, asserts only wall-clock evicts.
Mirrors the v0.14.3 `--sigkill-rescue` shape.
Tests: 23 new unit cases in test/minions.test.ts covering wall-clock
timeout (3 cases + non-interference with handleTimeouts), maxWaiting
(coalesce, clamp 0, floor, concurrent-submitter race via Promise.all,
cross-queue isolation, unset fallthrough), concurrency clamp (7 cases
incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure
audit file write.
Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook
A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite
has no multi-process worker surface). Two subchecks, both cheap (single
SELECT each, status-index-covered):
- stalled-forever: any active job with started_at > 1h. Surfaces the
worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel`
fix hints. The incident that motivated v0.19.1 ran 90+ min before the
operator noticed.
- waiting-depth: per-name waiting count exceeds threshold. Default 10,
overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a
submitter probably needs maxWaiting set.
Worker-heartbeat subcheck from the original plan dropped (D4/H4): no
minion_workers table exists, and lock_until-on-active-jobs is a lossy
proxy that can't distinguish idle-worker from dead-worker. Tracked as
follow-up B7.
A4: --no-worker peer-liveness probe in autopilot. When --no-worker is
set, every cycle runs a cheap SELECT checking for any active job whose
lock_until was refreshed in the last 2 minutes. After 3 consecutive
idle ticks, logs a loud WARNING naming the silent-wedge vector and
referencing B7 as the ground-truth follow-up. Re-arms on next live
signal so the warning doesn't spam every cycle.
A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60
lines). "My queue looks wedged — what do I run?" in order of
escalation. What each doctor subcheck means. Self-check for the
--no-worker / no-worker-running footgun.
CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer
3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health
doctor check (v0.19.1 D4), and backpressure-audit.ts.
Tests: all 143 minions + 13 doctor unit tests pass. No new test cases
required in Lane B; the doctor queue_health exercise is in the E2E
verification step (needs real PG to produce meaningful stalled-forever
rows). The --no-worker probe is exercised by the smoke case's wedge
setup in Lane A.
README: unchanged. Existing `gbrain jobs submit` examples don't show
--max-stalled, so no --max-waiting precedent to extend per A6 conditional.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md
VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change,
no new user-facing vocabulary).
CHANGELOG: new v0.19.1 entry at the top with the full release-summary
template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers
that matter" before/after table measured against the real incident,
"what this means for OpenClaw users" closer, required "To take
advantage of v0.19.1" block naming the worker-restart requirement,
itemized changes by area, and "For contributors" section closing the
loop on the stale autopilot-idempotency narrative the CEO review was
based on.
Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by
missing idempotency (autopilot already passes
`idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The
18 jobs were 18 DIFFERENT slots stacking up behind the wedged one.
`maxWaiting` still caps the pile; the incident just wasn't about
idempotency. Adversarial review caught this before ship.
SPEC.md: deleted from repo root. It was Wintermute's planning artifact
for the original PR, not a shipped spec. Design docs belong under
docs/designs/ per repo convention; leaving one at repo root set a
precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file
at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are
the durable artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip
Smoke case was setting lock_until in the past, so handleStalled's
requeue path fired before handleWallClockTimeouts had a chance to
evict. Production scenario is "lock_until still live (worker
renewing) + timeout_at disqualified" — only wall-clock matches.
Single-connection smoke can't simulate a row lock held by another
txn, so we force the equivalent outcome:
- lock_until = now() + 30s → handleStalled skips (not a stall)
- timeout_at = NULL → handleTimeouts skips (needs NOT NULL)
- started_at = now() - 10s, timeout_ms=1000 → wall-clock matches
(2 × timeout_ms = 2000ms threshold exceeded)
Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: CI failures — shell-handler tests + llms-full.txt drift
Two CI failure clusters, both pre-existing but surfaced by the v0.20.3
merge:
1) test/minions-shell.test.ts — 12 failing cases. The shell handler
throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the
production RCE guard at shell.ts:210). The unit tests exercise
handler mechanics, not the guard, but never set the env var — so
every invocation exits through the guard path instead of the code
being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll,
restore in afterAll. The env-guard IS still tested separately via
the test/minions.test.ts case added in v0.20.3 Lane A which toggles
the var itself.
2) llms-full.txt — stale against CLAUDE.md. Key-files entries for
queue.ts, doctor.ts, and the new backpressure-audit.ts updated in
v0.20.3 Lane B triggered the build-llms drift guard. Regenerated
via `bun run build:llms`; no behavior change, just the inlined-docs
bundle catching up to source.
Full test run: 2367 pass, 0 fail across 137 files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
d838d4792b