triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.20.2 feat: gbrain jobs supervisor — self-healing worker process manager (#364)

Browse Source 
* feat: add `gbrain jobs supervisor` — self-healing worker process manager

Adds a first-class supervisor command that:
- Spawns `gbrain jobs work` as a child process
- Restarts on crash with exponential backoff (1s→60s cap)
- Resets crash counter after 5min of stable operation
- PID file locking prevents duplicate supervisors
- Periodic health checks (stalled jobs, completion gaps)
- Graceful shutdown (SIGTERM→35s→SIGKILL)

Usage:
  gbrain jobs supervisor --concurrency 4

Replaces ad-hoc nohup patterns in bootstrap scripts.
The autopilot command's internal supervisor can be migrated
to use this in a follow-up.

Tests: 7 pass (backoff calc, PID management, crash tracking)

* supervisor: atomic PID lock, queue-scoped health, env safety, unified exit

Lane A of PR #364 review fixes (20-item multi-lane plan). Addresses the
codex-tier + CEO + Eng findings on src/core/minions/supervisor.ts:

Safety + correctness:
- Atomic O_CREAT|O_EXCL PID lock via openSync('wx') with stale-file
  liveness check. Prevents two supervisors racing on the same PID file.
  (codex #1)
- Health check now queries status='active' AND lock_until < now()
  matching queue.ts:848's authoritative stalled definition. The prior
  `status = 'stalled'` predicate returned zero rows forever because
  'stalled' is not a persisted value in the schema. (codex #2)
- All health queries scoped to WHERE queue = $1 via opts.queue binding.
  Multi-queue installs no longer see cross-queue false positives.
  (codex #3)
- Class default allowShellJobs flipped true→false AND explicit
  `delete env.GBRAIN_ALLOW_SHELL_JOBS` when false, so child workers
  don't silently inherit the var from the parent shell. (eng #8, codex #9)
- Unified shutdown(reason, exitCode) — max-crashes now routes through
  the same drain path as SIGTERM. Single source of truth for lifecycle
  cleanup; prerequisite for trustworthy audit events (Lane C). (eng #1)
- Default PID path moves from /tmp to ~/.gbrain/supervisor.pid with
  mkdirSync recursive + GBRAIN_SUPERVISOR_PID_FILE env override.
  Matches the rest of the product's ~/.gbrain/ convention; fresh
  installs no longer hit ENOENT. (CEO #2 + codex #6)

Refinements:
- crashCount = 1 after 5-min stable-run reset (was 0, produced
  calculateBackoffMs(-1) = 500ms by accident). Now reads as 'first
  crash of a new cycle' with a clean 1s backoff. (Nit 1)
- Top-of-file POSTGRES-ONLY docstring documenting why the supervisor
  can't run against PGLite. (Nit 2)
- inBackoff flag suppresses 'worker not alive' warn during the
  expected null-child window (crash → sleep → next spawn). (eng #2)
- Tracked listener refs for SIGTERM/SIGINT removed in shutdown() so
  integration tests spinning up/tearing down multiple supervisors on
  one process don't leak handlers. (eng #3)
- Single FILTER query replaces two SELECT counts — one round-trip
  instead of two, three metrics in one pass. (eng #10)
- child.on('error') listener emits worker_spawn_failed event for
  ENOENT/EACCES; exit handler still increments crashCount as usual
  so max-crashes bounds permanent misconfigurations. (codex #7)
- healthInFlight boolean guard with try/finally prevents overlapping
  health checks from stacking on a hung DB. (codex #8)

Documented exit codes (ExitCodes const):
  0 CLEAN, 1 MAX_CRASHES, 2 LOCK_HELD, 3 PID_UNWRITABLE
  Agent can branch on exit=2 ('another supervisor, I'm fine') vs
  exit=1 ('escalate to human').

Event emitter surface:
  - started / worker_spawned / worker_exited / worker_spawn_failed
  - backoff / health_warn / health_error / max_crashes_exceeded
  - shutting_down / stopped
  Plumbed through emit() with an onEvent callback hook for Lane C's
  audit writer. json:false is the default; Lane C's --json mode
  flips it and writes JSONL to stderr.

CLI changes (src/commands/jobs.ts):
- `gbrain jobs supervisor` gains --allow-shell-jobs (explicit opt-in
  mirroring the env-var gate), --cli-path (override auto-resolution
  for exotic setups), and --json (JSONL lifecycle events on stderr).
- Expanded --help body with description, 3 examples, and exit-code
  table. (DX Fix A per review)
- Three-tier PID path resolution: --pid-file > GBRAIN_SUPERVISOR_PID_FILE
  > ~/.gbrain/supervisor.pid (via exported DEFAULT_PID_FILE).
- Removed the catch-fallback to process.argv[1] — resolveGbrainCliPath()
  throws its own actionable install-hint error, which is what dev users
  need instead of a cryptic spawn failure on a .ts path. (codex #5)

Tests: existing 7 supervisor.test.ts cases continue to pass.
Integration tests (crash-restart, max-crashes, SIGTERM-during-backoff,
env-inheritance regression) land in Lane E.

Out of scope for this lane (tracked in follow-up lanes):
- Audit file writer at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl (Lane C)
- Documentation pass (Lane B)
- supervisor start/status/stop subcommands (Lane C)
- gbrain doctor supervisor check (Lane D)
- /ship release hygiene (Lane F)
- autopilot.ts migration to MinionSupervisor (deferred to follow-up PR
  per codex — requires non-blocking start() API redesign, not ~30 lines)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: supervisor as canonical worker deployment pattern

Lane B of PR #364 review fixes. Reframes docs/guides/minions-deployment.md
around `gbrain jobs supervisor` as the default answer (blocker 7), deletes
the 68-line legacy bash watchdog (F10), and updates README + deployment
snippets to match.

docs/guides/minions-deployment.md:
- New 'Worker supervision' section at the top with the canonical 3-command
  agent pattern (start --detach / status --json / stop) and a documented
  exit-code table (0 clean, 1 max-crashes, 2 lock-held, 3 PID-unwritable).
- 'Which supervisor when?' decision table: container = supervisor as
  PID 1, Linux VM = systemd-over-supervisor, dev laptop = bare terminal.
- New 'Agent usage' section for OpenClaw / Hermes / Cursor / Codex — the
  3-turn discover-start-maintain workflow that replaces shell archaeology
  with machine-parseable JSON events + an audit file at
  ~/.gbrain/audit/supervisor-YYYY-Www.jsonl.
- Demoted the 'Option 1: watchdog cron' path entirely; replaced with a
  straightforward upgrade migration block (stop script, remove cron line,
  start supervisor, verify via doctor).
- Preconditions now check Postgres connectivity directly (supervisor is
  Postgres-only; the CLI rejects PGLite with a clear error).

Snippets:
- systemd.service: ExecStart now invokes `gbrain jobs supervisor` instead
  of raw `gbrain jobs work`. Two-layer supervision (systemd → supervisor
  → worker) buys automatic restart on reboot plus fast crash recovery.
  ReadWritePaths expanded to cover $HOME/.gbrain (supervisor PID + audit).
- Procfile + fly.toml.partial: same change — platform restarts the
  container on host events, supervisor restarts the worker on crashes.
- minion-watchdog.sh: deleted (git history retains it for anyone in an
  exotic deployment). Supervisor subsumes every capability it had plus
  atomic PID locking, structured audit events, queue-scoped health
  checks, and graceful drain on SIGTERM.

README.md:
- Added a paragraph under the Minions section pointing `gbrain jobs
  supervisor` as canonical, noting the --detach / status / stop surface
  and the audit file path, with a link to the full deployment guide.
  Kept `gbrain jobs work` documented for direct raw invocation but
  flagged 'prefer supervisor' for any long-running use.

The supervisor `--help` body itself (3 examples + exit-code table in
src/commands/jobs.ts) landed with Lane A — this lane finishes the
discoverability story by making the supervisor findable via doc grep,
README landing, and deployment-guide landing paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* supervisor: daemon-manager subcommands + JSONL audit writer

Lane C of PR #364 review fixes. Adds the daemon-manager CLI surface so
agents can drive `gbrain jobs supervisor` in 3 turns instead of 10, and
the audit writer that makes lifecycle events inspectable across process
restarts. (Blocker 8, closes DX Fix A/B/C.)

New: src/core/minions/handlers/supervisor-audit.ts
  - writeSupervisorEvent(emission, supervisorPid) appends JSONL to
    `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`.
    ISO-week rotation via a `computeSupervisorAuditFilename()` helper
    that mirrors `shell-audit.ts` exactly (year-boundary ISO week math,
    Thursday anchor, etc).
  - readSupervisorEvents({sinceMs}) returns parsed events from the
    current week's file, oldest-first, for Lane D's doctor check.
    Malformed lines are skipped silently (disk-full truncation is
    already best-effort at write time).
  - Reuses `resolveAuditDir()` from shell-audit.ts so the
    `GBRAIN_AUDIT_DIR` env var override works identically across all
    gbrain audit trails.

src/commands/jobs.ts: supervisor subcommand dispatcher
  - `gbrain jobs supervisor [start] [--detach] [--json] ...` — default
    subcommand. Without --detach, runs foreground as before. With
    --detach, forks a background child (inheriting stderr so the caller
    can still tail JSONL events), writes a stdout payload:
      {"event":"started","supervisor_pid":N,"pid_file":"...","detached":true}
    and exits 0. Stdin/stdout on the detached child are /dev/null so
    the parent shell isn't held open.
  - `gbrain jobs supervisor status [--json]` — reads the PID file,
    checks liveness via `kill -0`, then reads the last 24h from the
    supervisor audit file to compute crashes_24h / last_start /
    max_crashes_exceeded. Exits 0 if running, 1 if not. JSON output
    is machine-parseable; human output is a 5-line ASCII report.
  - `gbrain jobs supervisor stop [--json]` — reads PID, sends SIGTERM,
    polls `kill -0` every 250ms for up to 40s (supervisor's own 35s
    worker-drain + 5s slack). Reports outcome: drained / timeout_40s
    / pid_file_missing / pid_file_corrupt / process_gone. Exit 0 on
    clean stop.
  - `--json` flag is already plumbed through to the supervisor opts
    from Lane A — this lane adds the onEvent audit-writer callback
    so every supervisor emission (started, worker_spawned,
    worker_exited, worker_spawn_failed, backoff, health_warn,
    health_error, max_crashes_exceeded, shutting_down, stopped) lands
    in the JSONL file with the supervisor's PID attached.

--help body updated:
  - Three separate usage lines (start / status / stop).
  - SUBCOMMANDS block with one-line summaries each.
  - EXIT CODES block (unchanged from Lane A, moved under SUBCOMMANDS).
  - EXAMPLES block updated with status --json + stop + --detach forms.

Tests: existing 127 supervisor + minions tests continue to pass.
Integration tests for the new subcommands + audit writer land with
Lane E.

Follow-up (Lane D): `gbrain doctor` will read readSupervisorEvents()
from this module to surface a `supervisor` health check alongside its
existing checks (DB connectivity, schema version, queue health).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doctor: add supervisor health check

Lane D of PR #364 review fixes. Closes the observability loop: now that
Lane C writes supervisor lifecycle events to
`${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`,
`gbrain doctor` surfaces a `supervisor` check alongside its existing
health indicators.

Implementation (src/commands/doctor.ts, filesystem-only block 3b-bis):
- Resolves DEFAULT_PID_FILE via the same three-tier logic as the start
  path (--pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid).
- Reads the PID file + `kill -0 <pid>` for liveness.
- Calls readSupervisorEvents({sinceMs: 24h}) from the audit module to
  derive last_start / crashes_24h / max_crashes_exceeded.
- Suppresses the check entirely when the user has never invoked the
  supervisor (no PID file AND no audit events) — avoids noise on
  installs that don't use the feature.

Status thresholds:
  fail   max_crashes_exceeded event seen in last 24h
         (supervisor gave up; operator needs to restart or triage)
  warn   supervisor not running but audit shows prior use
         (unexpected stop — likely crash or manual kill)
  warn   running but > 3 crashes in last 24h
         (supervisor recovering but worker is unstable)
  ok     running + ≤ 3 crashes + no max_crashes event

All failure paths emit a paste-ready recovery command. Read/import
errors are swallowed (best-effort like the other doctor checks).

Tests: all 127 supervisor + minions tests still green; 13 existing
doctor tests unaffected.

F3 done. All four lanes A/B/C/D are now committed; Lane E (integration
tests) and Lane F (/ship v0.20.2) remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: 4 critical integration tests for supervisor lifecycle

Lane E of PR #364 review fixes (blocker 10). Fills the ~15% coverage
gap flagged in the eng review by actually exercising the code paths
that will break in production — crash-restart loop, max-crashes exit,
SIGTERM-during-backoff, env-var inheritance — via real spawn() calls
against fake shell-script workers. No mocks: real fork, real signals,
real env propagation, real audit file writes.

test/fixtures/supervisor-runner.ts (new, 55 lines):
  A standalone bun script that constructs a MinionSupervisor from env
  vars (SUP_PID_FILE / SUP_CLI_PATH / SUP_MAX_CRASHES / SUP_BACKOFF_FLOOR_MS
  / SUP_HEALTH_INTERVAL_MS / SUP_ALLOW_SHELL_JOBS / SUP_AUDIT_DIR) and
  calls start(). Mock engine returns empty rows for executeRaw (health
  check path still exercised without Postgres). Tests spawn this as a
  subprocess because MinionSupervisor.start() calls process.exit() on
  shutdown — can't run it in the test runner's own process.

test/supervisor.test.ts (existing; 91 → 300 lines):
  - Added IntegrationHarness helper: creates a unique tmpdir per test,
    a fake worker shell script, a PID-file path, and an audit-dir path;
    cleanup runs in finally.
  - spawnSupervisor() forks bun on the runner with env vars set.
  - readAudit() reads the supervisor-YYYY-Www.jsonl file via the
    existing readSupervisorEvents() helper (Lane C), threading
    GBRAIN_AUDIT_DIR through so tests don't collide on ~/.gbrain.
  - waitFor(pred, timeoutMs) polls helper for event-driven tests.

Four integration tests (with _backoffFloorMs=5 for <1s suite runs):

  1. "respawns the worker after a crash and eventually exits with
     max-crashes code=1"
     Worker always `exit 1`. maxCrashes=3. Asserts: exit code 1, PID
     file cleaned up, audit contains started + 3x worker_spawned +
     3x worker_exited + max_crashes_exceeded + shutting_down + stopped,
     and the stopped event carries {reason:'max_crashes', exit_code:1}.
     Locks in blockers 1 (PID lock), 2+3+6 (health SQL doesn't 500),
     5 (unified shutdown emits right events), F8 (spawn errors counted).

  2. "receives SIGTERM while sleeping between crashes and exits 0 cleanly"
     Worker always `exit 1`, backoff floor 800ms to catch the sleep.
     Asserts: SIGTERM during backoff → exit code 0 (not 1) in <5s,
     no signal kill (process.exit via shutdown), audit contains
     shutting_down {reason:'SIGTERM'} + stopped, PID file cleaned up.
     Locks in eng Issue 1 (unified exit path), eng Issue 3 (signal
     handlers don't accumulate across shutdowns).

  3. "strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false,
     even if parent has it set"  ⚠ CRITICAL regression test
     Parent env has GBRAIN_ALLOW_SHELL_JOBS=1. SUP_ALLOW_SHELL_JOBS=0.
     Worker writes $GBRAIN_ALLOW_SHELL_JOBS (or 'UNSET' if absent) to
     an OUT_FILE. Asserts child sees 'UNSET'. Locks in codex #9 + eng
     #8: the `else delete env.GBRAIN_ALLOW_SHELL_JOBS` branch from
     Lane A is load-bearing for the supervisor's security posture;
     this test prevents a future refactor silently re-opening the
     inheritance hole.

  4. "DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs=true"
     Positive-path companion to #3. SUP_ALLOW_SHELL_JOBS=1 → worker
     sees '1'. Confirms the else-branch doesn't over-strip and that
     operators who explicitly opt in still get shell-exec enabled.

Plus two audit-format unit tests:
  - computeSupervisorAuditFilename format (regex match)
  - Year-boundary ISO week: 2027-01-01 → supervisor-2026-W53.jsonl
    (matches the shell-audit.ts pattern exactly)

Before: 7 tests covering backoff math + PID helpers (~15% behavioral
coverage per eng review).
After: 13 tests across all critical lifecycle paths (crash-restart,
max-crashes, SIGTERM, env-inheritance, audit rotation).

All 146 tests in supervisor + minions + doctor suites green in ~8s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.20.2)

Lane F of PR #364 review fixes. Closes the multi-lane plan with release
hygiene: VERSION bump 0.19.0 → 0.20.2, package.json sync, CHANGELOG entry
in GStack voice with release summary + "numbers that matter" table +
"To take advantage of v0.20.2" migration block + itemized changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: escape template-literal interpolation in supervisor --help

The --help body in src/commands/jobs.ts is one big backtick template
literal. The supervisor subcommand description I added in Lane B used
both `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}` (parsed as a template
interpolation into an undefined variable) and inline `code` backticks
(parsed as nested template literals). CI caught it with ~200 tsc parse
errors across the file.

Fix:
- Escape `${...}` → `\${...}` so the audit-file path renders literally.
- Replace prose inline-code backticks with plain single-quote fences
  (`gbrain jobs work` → 'gbrain jobs work', `~/.gbrain/supervisor.pid`
  → ~/.gbrain/supervisor.pid). `--help` output is human prose; the
  single-quote form reads cleanly in a terminal without needing to
  smuggle nested backticks through a template literal.

`bunx tsc --noEmit` is clean. 146 tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms-full.txt after Lane B doc rewrite

CI drift guard caught that `llms-full.txt` didn't match the current
generator output. Root cause: the Lane B rewrite of
`docs/guides/minions-deployment.md` (supervisor as canonical, watchdog
deleted) changed content that gets inlined into `llms-full.txt`, but I
didn't run `bun run build:llms` to regenerate.

`bun test test/build-llms.test.ts` now clean (7/7 pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-24 00:24:10 -07:00
committed by GitHub
parent 8b3c24c891
commit e3f704229b
16 changed files with 1916 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
CHANGELOG.md
View File

@@ -2,6 +2,117 @@
All notable changes to GBrain will be documented in this file. All notable changes to GBrain will be documented in this file.
## [0.20.2] - 2026-04-24
## **`gbrain jobs supervisor` is now a self-healing daemon you can actually drive. The Minions worker stops dying silently.**
## **Three commands an agent can run: `start --detach`, `status --json`, `stop`. Crash loops are bounded, audit events are JSONL, and the health check finally reports real data.**
`gbrain jobs work` has always been the worker that drains your Minions queue. Problem: it dies (OOM, connection blip, panic) and nobody notices until jobs pile up. The old answer was `nohup` plus a 68-line bash watchdog script from the deployment guide, and it shipped its own bugs (restart-loop traps, log-parsing stall detection, zero audit trail).
v0.20.2 ships the replacement: `gbrain jobs supervisor` is a first-class CLI with atomic PID locking, exponential backoff, structured audit events at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`, and three subcommands that make it drivable by an OpenClaw or Hermes agent in three turns. The old bash watchdog is gone.
### The numbers that matter
Before v0.20.2, an agent driving the supervisor needed ~10 turns of shell archaeology (PID file scraping, `pgrep -f`, `kill -0`, log grep) just to start and stop the worker reliably. After v0.20.2, it's three commands with machine-parseable output.
| Capability | Before v0.20.2 | After v0.20.2 |
|---|---|---|
| Keeping the worker alive | `nohup` + `minion-watchdog.sh` (68 lines of bash, restart-loop bug, log-scrape health) | `gbrain jobs supervisor` (first-class CLI with atomic PID lock, exponential backoff, JSONL audit) |
| PID file locking | `existsSync + readFileSync + writeFileSync` TOCTOU race | Atomic `O_CREAT|O_EXCL` via `openSync('wx')` — kernel-atomic mutex |
| Stalled-jobs health alert | Queried `status='stalled'` — returned 0 rows forever (dead code) | Queries `status='active' AND lock_until < now()`, scoped to the supervised queue |
| Shell-exec env inheritance | Child inherited `GBRAIN_ALLOW_SHELL_JOBS=1` from parent shell regardless of CLI flag | Explicit `else delete env.GBRAIN_ALLOW_SHELL_JOBS` when not opted in + regression test |
| Agent discovery TTHW | ~10 turns of shell-scraping (cat PID / pgrep / kill -0 / log grep) | 3 turns: `start --detach` → `status --json` → `stop` |
| Lifecycle observability | `console.log` with human prefixes, zero audit trail | JSONL events on stderr + `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` + `gbrain doctor` integration |
| Exit codes | undocumented; agent couldn't distinguish "already running" from "gave up" | Four documented codes: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable |
| Test coverage of the supervisor itself | ~15% (backoff math + PID helpers only) | Integration tests covering crash-restart, max-crashes drain, SIGTERM-during-backoff, env-inheritance regression |
The supervisor's own reliability claims are now testable. Every lifecycle event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`) lands in a weekly-rotated JSONL file that `gbrain doctor` reads to surface a `supervisor` health check.
### What this means for your deployment
If you were using the old `nohup`/`minion-watchdog.sh` pattern:
1. **Stop the old watchdog:** `sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && crontab -e` and delete the watchdog cron line.
2. **Delete the script:** `sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid /tmp/gbrain-worker.log`.
3. **Start the supervisor:** `gbrain jobs supervisor start --detach --json` — or on systemd, reinstall the unit (now calls `gbrain jobs supervisor`).
4. **Verify:** `gbrain doctor` reports a `supervisor` check; `gbrain jobs supervisor status --json` returns `running:true`.
For containers (Fly / Railway / Render / Heroku): the shipped `Procfile` and `fly.toml.partial` now call `gbrain jobs supervisor`. The platform restarts the container on host events, the supervisor restarts the worker on in-process crashes. Two-layer supervision with clean separation.
For OpenClaw / Hermes / Cursor agents driving the supervisor: you no longer need a shell skill to drive the worker. Every piece of state — liveness, crash history, max-crashes exhaustion — is a machine-parseable JSON response. Start with `gbrain jobs supervisor status --json | jq`.
## To take advantage of v0.20.2
`gbrain upgrade` pulls the binary. Nothing else is required if you're currently running `gbrain jobs work` directly or using systemd — the new supervisor is opt-in. To migrate:
1. **Verify the binary:**
```bash
gbrain --version # should say 0.20.2
gbrain jobs supervisor --help | head -20
```
2. **Start the supervisor (detached, agent-friendly):**
```bash
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"pid_file":"/Users/you/.gbrain/supervisor.pid","detached":true}
```
3. **Check health:**
```bash
gbrain jobs supervisor status --json
gbrain doctor | grep supervisor
```
4. **Stop when done:**
```bash
gbrain jobs supervisor stop
```
5. **(Optional) Migrate off the old watchdog:** see `docs/guides/minions-deployment.md` "Upgrading from an older deployment" for the cron-to-supervisor migration.
If `gbrain jobs supervisor status` reports `running:false` unexpectedly, or `gbrain doctor` flags a `supervisor` failure, file an issue at https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- the last ~50 lines of `~/.gbrain/audit/supervisor-*.jsonl`
- which step broke
### Itemized changes
**`gbrain jobs supervisor`:**
- New subcommands: `start [--detach] [--json]`, `status [--json]`, `stop [--json]`. Foreground use is unchanged (back-compat).
- New flags: `--allow-shell-jobs` (explicit opt-in, replaces env-var sniffing), `--cli-path PATH` (override auto-resolution), `--json` (JSONL lifecycle events on stderr), `GBRAIN_SUPERVISOR_PID_FILE` env var (overrides default PID path).
- Exit codes documented in `--help`: `0` clean, `1` max-crashes, `2` lock-held, `3` PID-unwritable.
- Default PID path moved from `/tmp/gbrain-supervisor.pid` to `~/.gbrain/supervisor.pid` with automatic parent-directory creation.
**Safety fixes (codex adversarial review + eng review):**
- Atomic PID lock via `openSync(path, 'wx')` — two supervisors starting simultaneously can no longer both win the race.
- `stalled` health check query rewritten from unreachable `status='stalled'` to `status='active' AND lock_until < now()` matching `queue.ts:848 handleStalled()`.
- Health queries now scoped to `WHERE queue = $1` — multi-queue deployments see the right queue.
- Unified exit path via `shutdown(reason, exitCode)` — max-crashes drains gracefully instead of bypassing cleanup via `process.exit(1)`.
- Listener ref tracking: `SIGTERM`/`SIGINT` handlers removed on shutdown for clean test lifecycle.
**Security hardening:**
- `allowShellJobs` class default flipped `true` → `false`.
- Child env now has `GBRAIN_ALLOW_SHELL_JOBS` explicitly deleted when `allowShellJobs:false` (was: silently inherited from parent shell).
- Integration regression test locks this against future refactors.
**Observability:**
- New `src/core/minions/handlers/supervisor-audit.ts` with ISO-week rotation (mirrors `shell-audit.ts` / `subagent-audit.ts` pattern).
- Every supervisor emission (started, worker_spawned, worker_exited, worker_spawn_failed, backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, stopped) written to `~/.gbrain/audit/supervisor-YYYY-Www.jsonl`.
- `gbrain doctor` gains a `supervisor` check that reads the audit file and reports `running` / `last_start` / `crashes_24h` / `max_crashes_exceeded` with thresholds (ok / warn at 3+ crashes / fail on max-crashes event).
**Documentation:**
- `docs/guides/minions-deployment.md` rewritten: supervisor is the canonical answer; which-supervisor-when decision table (container / systemd / dev laptop); three-command agent pattern; migration block from the old watchdog.
- `README.md` Operations section gains a paragraph on `gbrain jobs supervisor`.
- `docs/guides/minions-deployment-snippets/{systemd.service,Procfile,fly.toml.partial}` now invoke `gbrain jobs supervisor` instead of raw `gbrain jobs work`.
- `docs/guides/minions-deployment-snippets/minion-watchdog.sh` deleted — subsumed by the supervisor.
**Tests:**
- `test/supervisor.test.ts`: 7 → 13 tests. Four new integration tests exercise real `spawn()` lifecycles via shell-script fakes (crash-restart happy path, max-crashes-via-shutdown with audit assertions, SIGTERM-during-backoff clean exit, `GBRAIN_ALLOW_SHELL_JOBS` inheritance regression — positive + negative).
- `test/fixtures/supervisor-runner.ts`: new standalone runner that constructs a supervisor from env vars so integration tests can observe `process.exit` without killing the test runner.
**For contributors:**
- The `MinionSupervisor` class has a test-only `_backoffFloorMs` override for fast crash-loop tests. Not exposed via CLI.
- `onEvent: (emission) => void` is an injectable hook on `SupervisorOpts` — Lane C's audit writer uses it; future observability integrations can too.
- `autopilot.ts` migration to `MinionSupervisor` is explicitly deferred (follow-up PR): the current `start()` API blocks, which deadlocks autopilot's interval loop. Codex's review flagged this; the fix is a non-blocking-start API redesign, not a drop-in substitution.
Credit: original supervisor feature built by OpenClaw (PR #364 initial commit). Review wave + code-level fixes + daemon-manager CLI + observability boomerang + integration tests shipped via /autoplan (CEO + DX + Eng + Codex adversarial) followed by a 20-item multi-lane implementation plan.
## [0.20.0] - 2026-04-23 ## [0.20.0] - 2026-04-23
## **BrainBench moves out. gbrain gets its install surface back.** ## **BrainBench moves out. gbrain gets its install surface back.**

5
README.md
View File

@@ -211,9 +211,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis
gbrain jobs smoke # verify install gbrain jobs smoke # verify install
gbrain jobs submit sync --params '{}' # fire a background job gbrain jobs submit sync --params '{}' # fire a background job
gbrain jobs stats # health dashboard gbrain jobs stats # health dashboard
gbrain jobs work --concurrency 4 # start a worker (Postgres only) gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
``` ```
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox. Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now. **Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.

2
VERSION
View File

@@ -1 +1 @@
0.20.0 0.20.2

5
docs/guides/minions-deployment-snippets/Procfile
View File

@@ -7,4 +7,7 @@
# DATABASE_URL=postgresql://... # DATABASE_URL=postgresql://...
# GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs # GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
worker: gbrain jobs work --concurrency 2 # Two-layer supervision: the platform restarts the container on host
# events (OOM, deploy); `gbrain jobs supervisor` restarts the worker
# on in-process crashes with exponential backoff.
worker: gbrain jobs supervisor --concurrency 2

6
docs/guides/minions-deployment-snippets/fly.toml.partial
View File

@@ -5,10 +5,12 @@
# fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs # fly secrets set GBRAIN_ALLOW_SHELL_JOBS=1 # only if submitting shell jobs
# fly secrets set ANTHROPIC_API_KEY=... # optional # fly secrets set ANTHROPIC_API_KEY=... # optional
# #
# Fly.io auto-restarts the process on crash — no watchdog needed. # Two-layer supervision: Fly restarts the VM on host events; the
# `gbrain jobs supervisor` process restarts the worker on in-process
# crashes with exponential backoff and a structured audit trail.
[processes] [processes]
worker = "gbrain jobs work --concurrency 2" worker = "gbrain jobs supervisor --concurrency 2"
# Scale the worker process to 1 machine (job queue serializes work; more # Scale the worker process to 1 machine (job queue serializes work; more
# machines means higher concurrency but also more Postgres connections). # machines means higher concurrency but also more Postgres connections).

68
docs/guides/minions-deployment-snippets/minion-watchdog.sh
View File

@@ -1,68 +0,0 @@
#!/bin/bash
# minion-watchdog.sh — restart gbrain jobs work if the process is dead or
# has logged a shutdown marker since its last start.
#
# Fixes the v0.16.1 restart-loop bug: old shutdown lines from previous
# restarts stayed in the unrotated log and every tick re-matched them
# forever. This version writes a restart epoch to line 2 of the PID file
# and only considers log lines newer than that epoch.
#
# Run every 5 minutes from crontab. See docs/guides/minions-deployment.md.
set -u
PID_FILE="${GBRAIN_WORKER_PID_FILE:-/tmp/gbrain-worker.pid}"
LOG_FILE="${GBRAIN_WORKER_LOG_FILE:-/tmp/gbrain-worker.log}"
GBRAIN="${GBRAIN_BIN:-/usr/local/bin/gbrain}"
CONCURRENCY="${GBRAIN_WORKER_CONCURRENCY:-2}"
start_worker() {
# stderr merged so banner lines ("[minion worker] shell handler enabled",
# "worker shutting down") all land in $LOG_FILE.
nohup "$GBRAIN" jobs work --concurrency "$CONCURRENCY" \
> "$LOG_FILE" 2>&1 &
local pid=$!
# Line 1: PID. Line 2: restart epoch (seconds since 1970).
# Readers that want just PID use `head -n1 "$PID_FILE"`.
printf '%s\n%s\n' "$pid" "$(date +%s)" > "$PID_FILE"
}
shutdown_since_restart() {
# Only match shutdown lines logged AFTER the most recent restart epoch.
# Worker log lines start with ISO-8601 UTC timestamps ("2026-04-21T19:05:12Z ...").
local restart_epoch
restart_epoch=$(sed -n '2p' "$PID_FILE" 2>/dev/null || echo 0)
[ -z "$restart_epoch" ] && restart_epoch=0
# POSIX-portable regex (no {n} intervals — mawk on Debian/Ubuntu rejects them).
awk -v since="$restart_epoch" '
match($0, /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9:.+Z-]+/) {
ts_str = substr($0, RSTART, RLENGTH)
cmd = "date -d \"" ts_str "\" +%s 2>/dev/null"
cmd | getline ts
close(cmd)
if (ts + 0 > since + 0) print
}
' "$LOG_FILE" 2>/dev/null | grep -q "worker stopped\|worker shutting down"
}
if [ -f "$PID_FILE" ]; then
PID=$(head -n1 "$PID_FILE")
if [ -n "$PID" ] && kill -0 "$PID" 2>/dev/null; then
# Process alive — check whether the worker logged an internal shutdown
# AFTER the last start. If yes, worker is dead-inside; restart.
if shutdown_since_restart; then
kill "$PID" 2>/dev/null
# 10s grace: covers shell handler's 5s child SIGTERM→SIGKILL window
# and leaves room for in-flight jobs to flush. Bump to 30 if your
# jobs run > 10s.
sleep 10
kill -9 "$PID" 2>/dev/null
start_worker
fi
else
# PID file exists but process is gone (crash / kill -9 / reboot).
start_worker
fi
else
start_worker
fi

12
docs/guides/minions-deployment-snippets/systemd.service
View File

@@ -15,9 +15,13 @@ WorkingDirectory=/srv/gbrain
# Env file is mode 600, owned by User=. Do not put secrets in this unit. # Env file is mode 600, owned by User=. Do not put secrets in this unit.
EnvironmentFile=/etc/gbrain.env EnvironmentFile=/etc/gbrain.env
ExecStart=/usr/local/bin/gbrain jobs work --concurrency 2 # Two-layer supervision: systemd restarts `gbrain jobs supervisor` on host
# events (reboot, unit crash); the supervisor restarts `gbrain jobs work`
# on in-process crashes with exponential backoff + structured audit.
ExecStart=/usr/local/bin/gbrain jobs supervisor --concurrency 2
# Replaces the cron watchdog. systemd restarts on any non-zero exit. # systemd restarts the supervisor on any non-zero exit. The supervisor
# itself handles worker-level crash recovery.
Restart=always Restart=always
RestartSec=10s RestartSec=10s
@@ -38,7 +42,9 @@ NoNewPrivileges=true
PrivateTmp=true PrivateTmp=true
ProtectSystem=strict ProtectSystem=strict
ProtectHome=read-only ProtectHome=read-only
ReadWritePaths=/srv/gbrain # ReadWritePaths must include the brain workspace AND ~/.gbrain (PID file +
# audit log written by the supervisor).
ReadWritePaths=/srv/gbrain /home/gbrain/.gbrain
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target

407
docs/guides/minions-deployment.md
View File

@@ -1,7 +1,7 @@
# Minions Worker Deployment Guide # Minions Worker Deployment Guide
Deploy `gbrain jobs work` so it stays running across crashes, reboots, and Keep `gbrain jobs work` running across crashes, reboots, and Postgres
Postgres connection blips. Written for agents to execute line-by-line. connection blips. Written for agents to execute line-by-line.
## The problem ## The problem
@@ -12,10 +12,61 @@ The persistent worker can die silently from:
- Bun process crashes with no automatic restart. - Bun process crashes with no automatic restart.
- Internal event-loop death (PID alive, worker loop stopped). - Internal event-loop death (PID alive, worker loop stopped).
When the worker dies, submitted jobs sit in `waiting` forever. Nothing in When the worker dies, submitted jobs sit in `waiting` forever. The
gbrain core auto-restarts the worker — that's what this guide wires up. canonical answer is `gbrain jobs supervisor` — a first-class CLI that
spawns `gbrain jobs work` as a child and auto-restarts it on crash.
## Variables used in this guide ## Worker supervision
### The canonical pattern
`gbrain jobs supervisor` is an auto-restarting wrapper around
`gbrain jobs work`. It writes a PID file, restarts the worker on crash
with exponential backoff (1s → 60s cap), emits lifecycle events to an
audit file, and drains gracefully on SIGTERM (35s worker-drain window
before SIGKILL). Exit codes are documented so agents can branch on them.
**Typical commands:**
```bash
# Start in the foreground (blocks; Ctrl-C to stop).
gbrain jobs supervisor --concurrency 4
# Start detached — returns {"event":"started","supervisor_pid":…} on stdout.
gbrain jobs supervisor start --detach --json
# Check liveness without reading log files.
gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
```
**Exit codes:**
| Code | Meaning |
|---|---|
| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) |
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=1 should page a human.
### Which supervisor when?
The supervisor solves in-process crash recovery. Platform-level
supervision (systemd, Fly, Render) handles host-level failures. You
usually want both.
| Environment | Recommendation |
|---|---|
| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). |
| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). |
| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. |
### Variables used in this guide
Substitute these once before copy-pasting any snippet. Substitute these once before copy-pasting any snippet.
@@ -23,142 +74,122 @@ Substitute these once before copy-pasting any snippet.
|---|---|---| |---|---|---|
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` | | `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` | | `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) |
| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) |
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` | | `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) | | `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) |
## Preconditions ### Preconditions
Run these before Step 1 of any option. Fail fast if something is wrong. Run these before any deployment step.
```bash ```bash
# 1. gbrain is on PATH and resolves to an absolute location. # 1. gbrain is on PATH and resolves to an absolute location.
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; } command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists). # 2. DATABASE_URL points at reachable Postgres.
# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the
# separate worker process. If `config.engine === 'pglite'` the CLI rejects
# with a clear error.)
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")' gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
# 3. Schema is up to date. If version=0 or status=="fail", fix it first: # 3. Schema is up to date. If version=0 or status=="fail":
# gbrain apply-migrations --yes # gbrain apply-migrations --yes
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")' gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
# 4. You have write access to at least one crontab mechanism. # 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the
crontab -l >/dev/null 2>&1 && echo "user crontab OK" # supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting).
[ -w /etc/crontab ] && echo "/etc/crontab OK" # Without the flag, the shell handler is disabled at worker startup.
# 5. If you plan to submit `shell` jobs, the WORKER process needs
# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated
# in registerBuiltinHandlers(); without the flag the worker startup
# line reads "shell handler disabled (...)".
``` ```
## Which option? ## Agent usage (OpenClaw / Hermes / Cursor / Codex)
- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take Three-command pattern an agent can drive without shell archaeology:
> 30 s → **Option 1** (watchdog cron + persistent worker).
- Your workload is short deterministic scripts on a fixed schedule (every
3 h, daily, weekly) → **Option 2** (inline `--follow`).
- You don't have shell access to a long-running box (Fly/Render/Railway,
or any systemd host) → **Option 3** (service manager — replaces cron).
## Option 1: watchdog cron + persistent worker
A 5-minute cron checks whether the worker process is alive **and** whether
it has logged an internal shutdown since its last start. Restarts if either
condition fails.
### 1a. Install the env file (secrets stay out of crontab)
Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is
mode 644 (world-readable); user crontabs under `/var/spool/cron/` are
readable by `root`. Use the shipped env-file template:
```bash ```bash
sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ # Start (returns PIDs + pid_file on stdout as JSON, then detaches)
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
# Check health (machine-parseable JSON, no log scraping)
gbrain jobs supervisor status --json
# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...}
# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback)
gbrain jobs supervisor stop
```
Every lifecycle event (spawn, crash, backoff, health warning, max-crashes,
shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
for historical inspection. `gbrain doctor` reads that file and surfaces
a `supervisor` check in its health report.
## Deployment: systemd
For long-running Linux VMs with shell access.
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# Install the env file (secrets stay out of the unit file).
sudo install -m 600 -o gbrain -g gbrain \
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
sudoedit /etc/gbrain.env sudoedit /etc/gbrain.env
# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1.
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
``` ```
Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work`
applicable). See directly) so you get two-layer supervision: systemd restarts the supervisor
[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example) on host reboot, supervisor restarts the worker on in-process crash.
for the full list.
### 1b. Install the watchdog script `Restart=always` + `RestartSec=10s` handle the supervisor-level recovery.
The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and
audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent
LLM subagent calls without hitting the default 1024 cap.
The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh) ## Deployment: Fly.io
ships in-repo and writes a two-line PID file (PID on line 1, restart epoch
on line 2). The restart-epoch marker is how the watchdog distinguishes
stale shutdown lines in the log from current ones — without it, every tick
after the first restart would match an old `worker shutting down` line and
loop forever.
Requires GNU coreutils (Linux default). On macOS/BSD install via
`brew install coreutils` and alias `date` to `gdate` in the cron env if you
want to test the watchdog locally; production Linux boxes work as-is.
```bash ```bash
sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ # Merge the [processes] block from fly.toml.partial into your fly.toml.
docs/guides/minions-deployment-snippets/minion-watchdog.sh \ cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml
/usr/local/bin/minion-watchdog.sh # Review + edit as needed.
# Set secrets (Fly handles restart on crash).
fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1
``` ```
### 1c. Wire into cron The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly
restarts the container on host failure; the supervisor restarts the
worker on in-process crash.
Pick the form that matches the crontab you're editing. ## Deployment: Render / Railway / Heroku
**If you ran `crontab -e`** (user crontab — 5-field, no user column): Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo
root. The shipped Procfile calls `gbrain jobs supervisor`. Set
`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's
env UI or CLI.
``` ## Deployment: inline `--follow` (no persistent worker)
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * /usr/local/bin/minion-watchdog.sh
```
**If you edited `/etc/crontab` directly** (system crontab — 6-field, with For short deterministic scripts on a fixed schedule where you don't need
user column): a persistent worker between runs. Each cron run brings its own temporary
worker. `--follow` starts one on the queue and blocks until the
``` just-submitted job reaches a terminal state (`completed` / `failed` /
SHELL=/bin/bash `dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job
PATH=/usr/local/bin:/usr/bin:/bin duration for scheduled work.
BASH_ENV=/etc/gbrain.env
*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh
```
In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to
source the env file before running the watchdog — that's how the
connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without
landing in the world-readable crontab itself.
### 1d. Log rotation
The watchdog appends to the worker log across restarts. If you expect the
file to grow unbounded, rotate it externally with `logrotate`:
```
# /etc/logrotate.d/gbrain-worker
/tmp/gbrain-worker.log {
daily
rotate 7
missingok
notifempty
copytruncate
}
```
`copytruncate` is important — the watchdog's restart-epoch check survives
it (the epoch is compared against in-log timestamps, not file inode).
## Option 2: inline `--follow` (no persistent worker)
Each cron run brings its own temporary worker. `--follow` starts one on
the queue and blocks until the just-submitted job reaches a terminal state
(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead
per job; negligible vs job duration for scheduled work.
Example: nightly brain enrichment as a shell job.
```bash ```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
@@ -170,85 +201,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
Replace `gbrain embed --stale` with whichever gbrain subcommand you're Replace `gbrain embed --stale` with whichever gbrain subcommand you're
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`, scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
`lint`, `autopilot`). If you're shelling out to a non-gbrain binary, `lint`, `autopilot`). For strict single-job semantics on shared queues,
keep its absolute path in the `cmd`.
**Shared-queue gotcha.** If other jobs are already waiting on the same
queue with higher priority or earlier `created_at`, the temporary worker
processes those first before reaching yours. `--follow` still exits only
when YOUR job finishes. For strict single-job semantics on shared queues,
use a dedicated queue name like `nightly-enrich` above. use a dedicated queue name like `nightly-enrich` above.
## Option 3: service manager (systemd / Fly / Render / Railway) ## Upgrading from an older deployment
Replaces the watchdog entirely. No cron, no PID file, no restart-loop. ### From `minion-watchdog.sh` (pre-v0.20)
The service manager owns liveness.
### systemd (Linux hosts with shell access) Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
which handles everything the script did, plus atomic PID locking,
structured audit events, queue-scoped health checks, and graceful
drain on SIGTERM.
**Migration:**
```bash ```bash
# Create the worker user if it doesn't exist. # 1. Stop and remove the old watchdog.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null
2>/dev/null || true sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" /tmp/gbrain-worker.log
crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line
# Install the unit file, substituting /srv/gbrain → your workspace path. # 2. Start the supervisor (systemd users: reinstall the unit from
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ # docs/guides/minions-deployment-snippets/systemd.service, which
/etc/systemd/system/gbrain-worker.service # now calls `gbrain jobs supervisor`).
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ gbrain jobs supervisor start --detach --json
/etc/systemd/system/gbrain-worker.service # Or: sudo systemctl restart gbrain-worker
# See 1a above for /etc/gbrain.env install. # 3. Verify.
sudo systemctl daemon-reload gbrain jobs supervisor status --json
sudo systemctl enable --now gbrain-worker gbrain doctor # 'supervisor' check should report running=true
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
``` ```
`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit ### Schema / migration hygiene
runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped
unit covers Bun + Postgres pool + concurrent LLM subagent calls without
hitting the default 1024 cap.
### Fly.io Regardless of which deployment path you're upgrading from:
Merge the `[processes]` block from 1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop`
[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into (or `sudo systemctl stop gbrain-worker`). Skipping this risks an
your existing `fly.toml`. Set secrets with `fly secrets set` — in-flight job landing partial schema.
Fly auto-restarts the process on crash.
### Render / Railway / Heroku
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root.
Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the
platform's env UI or CLI.
## Upgrading an existing deployment
If you deployed on v0.13.x or earlier, walk this checklist:
1. **Stop the worker before upgrading.**
`kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to
exit. Skipping this risks an in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if 2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`. `gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** from v0.14 onward, the worker requires 3. **If you run shell jobs:** from v0.14 onward, pass
`GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to `--allow-shell-jobs` to the supervisor (or keep
`/etc/gbrain.env`. Submitters don't need the flag; only the worker does. `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration need the flag; only the worker does.
v15 raised the schema default to 5 and backfilled existing non-terminal 4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
rows. A watchdog tuned around 1-strike dead-lettering will now migrations plus a healthy `supervisor` check. `gbrain jobs stats`
over-restart because it takes 5 misses to dead-letter. Switch to the should show no unexplained growth in `dead` between pre- and
shipped watchdog (which keys on log markers, not job state). post-upgrade.
5. **If your v0.16.1 watchdog is still running:** it has a restart-loop
bug (old shutdown lines in the unrotated log re-match every 5 min
forever). Install the current `minion-watchdog.sh` from this guide's
snippets — it writes a restart epoch into the PID file and only
considers log lines newer than that epoch.
6. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations. `gbrain jobs stats` should show no unexplained growth in
`dead` between pre- and post-upgrade.
## Known issues ## Known issues
@@ -261,9 +263,10 @@ silently. The stall detector then dead-letters the job after
**Current defaults that make this worse:** **Current defaults that make this worse:**
- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips. - `lockDuration: 30000` (30 s) — too short for long jobs during
- `max_stalled: 5` (schema column default on master — see `src/schema.sql` connection blips.
and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. - `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively. - `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, **Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
@@ -271,9 +274,6 @@ silently. The stall detector then dead-letters the job after
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags `--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is (since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today. what `handleStalled()` reads — so per-job tuning is the real knob today.
Worker-level `--lock-duration` / `--stall-interval` are on the roadmap;
until they land, rely on per-job `--max-stalled` plus the watchdog (or
systemd) for worker health.
### DO NOT pass `maxStalledCount` to `MinionWorker` ### DO NOT pass `maxStalledCount` to `MinionWorker`
@@ -284,16 +284,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children ### Zombie shell children
When the Bun worker crashes hard, child processes from shell jobs can When the Bun worker crashes hard, child processes from shell jobs can
become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window
shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For
shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker long-running shell jobs, prefer timeouts via `--timeout-ms` on submit
has time to flush in-flight jobs before the kill. over relying on hard kills.
## Smoke test ## Smoke test
```bash ```bash
# Worker alive? # Supervisor alive?
kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD gbrain jobs supervisor status --json | jq .running
# Aggregate queue health. # Aggregate queue health.
gbrain jobs stats gbrain jobs stats
@@ -304,20 +304,29 @@ gbrain jobs list --status active --limit 10
# Dead-lettered jobs. # Dead-lettered jobs.
gbrain jobs list --status dead --limit 10 gbrain jobs list --status dead --limit 10
# Shell handler registered? (stderr banner merged into log via 2>&1.) # Shell handler registered? (check supervisor audit log or worker stderr.)
grep "shell handler enabled" /tmp/gbrain-worker.log gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs'
``` ```
## Uninstall ## Uninstall
- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line. **`gbrain jobs supervisor`** (foreground or `--detach`):
`kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`.
Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`. ```bash
- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to gbrain jobs supervisor stop
clean up — temporary workers exit with their jobs. ```
- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`,
then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`, **systemd:**
then `sudo systemctl daemon-reload`.
- **Option 3 (Fly/Render/Railway):** delete the `worker` process from ```bash
`fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets` sudo systemctl disable --now gbrain-worker
persist until `fly secrets unset`. sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env
sudo systemctl daemon-reload
```
**Fly / Render / Railway:** delete the `worker` process from `fly.toml`
/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until
`fly secrets unset`.
**Inline `--follow`:** remove the cron entry. Nothing else to clean up
— temporary workers exit with their jobs.

412
llms-full.txt
View File

@@ -1403,9 +1403,12 @@ The six daily pains — spawn storms, agents that stop responding, forgotten dis
gbrain jobs smoke # verify install gbrain jobs smoke # verify install
gbrain jobs submit sync --params '{}' # fire a background job gbrain jobs submit sync --params '{}' # fire a background job
gbrain jobs stats # health dashboard gbrain jobs stats # health dashboard
gbrain jobs work --concurrency 4 # start a worker (Postgres only) gbrain jobs supervisor --concurrency 4 # canonical: auto-restarting worker (Postgres only)
gbrain jobs work --concurrency 4 # raw worker (no crash recovery — prefer `supervisor`)
``` ```
`gbrain jobs supervisor` keeps the worker alive across crashes with exponential backoff, atomic PID locking, structured audit events at `~/.gbrain/audit/supervisor-*.jsonl`, and a `start --detach` / `status --json` / `stop` subcommand surface for agents. In containers it runs as PID 1; on systemd hosts it's the child of `gbrain-worker.service`. Full deployment guide: [`docs/guides/minions-deployment.md`](docs/guides/minions-deployment.md).
Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox. Read [`skills/minion-orchestrator/SKILL.md`](skills/minion-orchestrator/SKILL.md) for parent-child DAGs, fan-in collection, steering via inbox.
**Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now. **Minions is not incrementally better than sub-agents for background work. It's categorically different.** 753ms vs gateway timeout. $0 vs tokens. 100% vs couldn't-spawn. If your agent does deterministic work on a schedule, it runs on Minions now.
@@ -3496,8 +3499,8 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/min
# Minions Worker Deployment Guide # Minions Worker Deployment Guide
Deploy `gbrain jobs work` so it stays running across crashes, reboots, and Keep `gbrain jobs work` running across crashes, reboots, and Postgres
Postgres connection blips. Written for agents to execute line-by-line. connection blips. Written for agents to execute line-by-line.
## The problem ## The problem
@@ -3508,10 +3511,61 @@ The persistent worker can die silently from:
- Bun process crashes with no automatic restart. - Bun process crashes with no automatic restart.
- Internal event-loop death (PID alive, worker loop stopped). - Internal event-loop death (PID alive, worker loop stopped).
When the worker dies, submitted jobs sit in `waiting` forever. Nothing in When the worker dies, submitted jobs sit in `waiting` forever. The
gbrain core auto-restarts the worker — that's what this guide wires up. canonical answer is `gbrain jobs supervisor` — a first-class CLI that
spawns `gbrain jobs work` as a child and auto-restarts it on crash.
## Variables used in this guide ## Worker supervision
### The canonical pattern
`gbrain jobs supervisor` is an auto-restarting wrapper around
`gbrain jobs work`. It writes a PID file, restarts the worker on crash
with exponential backoff (1s → 60s cap), emits lifecycle events to an
audit file, and drains gracefully on SIGTERM (35s worker-drain window
before SIGKILL). Exit codes are documented so agents can branch on them.
**Typical commands:**
```bash
# Start in the foreground (blocks; Ctrl-C to stop).
gbrain jobs supervisor --concurrency 4
# Start detached — returns {"event":"started","supervisor_pid":…} on stdout.
gbrain jobs supervisor start --detach --json
# Check liveness without reading log files.
gbrain jobs supervisor status --json
# Graceful stop (SIGTERM + drain wait + SIGKILL fallback).
gbrain jobs supervisor stop
```
**Exit codes:**
| Code | Meaning |
|---|---|
| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) |
| 1 | Max crashes exceeded (worker kept dying) |
| 2 | Another supervisor holds the PID lock |
| 3 | PID file unwritable (permission / path error) |
An agent seeing exit=2 can safely treat it as "one is already running";
exit=1 should page a human.
### Which supervisor when?
The supervisor solves in-process crash recovery. Platform-level
supervision (systemd, Fly, Render) handles host-level failures. You
usually want both.
| Environment | Recommendation |
|---|---|
| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). |
| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). |
| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. |
### Variables used in this guide
Substitute these once before copy-pasting any snippet. Substitute these once before copy-pasting any snippet.
@@ -3519,142 +3573,122 @@ Substitute these once before copy-pasting any snippet.
|---|---|---| |---|---|---|
| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` | | `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` |
| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` | | `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` |
| `$GBRAIN_WORKER_PID_FILE` | Worker PID + restart-epoch file | `/tmp/gbrain-worker.pid` (or `/var/run/gbrain/worker.pid` for systemd) |
| `$GBRAIN_WORKER_LOG_FILE` | Worker log sink (stdout + stderr merged) | `/tmp/gbrain-worker.log` (or `/var/log/gbrain/worker.log`) |
| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` | | `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` |
| `$GBRAIN_ENV_FILE` | Secrets file sourced by crontab / systemd | `/etc/gbrain.env` (mode 600) | | `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) |
## Preconditions ### Preconditions
Run these before Step 1 of any option. Fail fast if something is wrong. Run these before any deployment step.
```bash ```bash
# 1. gbrain is on PATH and resolves to an absolute location. # 1. gbrain is on PATH and resolves to an absolute location.
command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; } command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; }
# 2. DATABASE_URL points at reachable Postgres (or PGLite path exists). # 2. DATABASE_URL points at reachable Postgres.
# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the
# separate worker process. If `config.engine === 'pglite'` the CLI rejects
# with a clear error.)
gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")' gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")'
# 3. Schema is up to date. If version=0 or status=="fail", fix it first: # 3. Schema is up to date. If version=0 or status=="fail":
# gbrain apply-migrations --yes # gbrain apply-migrations --yes
gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")' gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")'
# 4. You have write access to at least one crontab mechanism. # 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the
crontab -l >/dev/null 2>&1 && echo "user crontab OK" # supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting).
[ -w /etc/crontab ] && echo "/etc/crontab OK" # Without the flag, the shell handler is disabled at worker startup.
# 5. If you plan to submit `shell` jobs, the WORKER process needs
# GBRAIN_ALLOW_SHELL_JOBS=1 (submitters do not). The handler is gated
# in registerBuiltinHandlers(); without the flag the worker startup
# line reads "shell handler disabled (...)".
``` ```
## Which option? ## Agent usage (OpenClaw / Hermes / Cursor / Codex)
- Your workload runs LLM subagents (`gbrain agent run`) or jobs that take Three-command pattern an agent can drive without shell archaeology:
> 30 s → **Option 1** (watchdog cron + persistent worker).
- Your workload is short deterministic scripts on a fixed schedule (every
3 h, daily, weekly) → **Option 2** (inline `--follow`).
- You don't have shell access to a long-running box (Fly/Render/Railway,
or any systemd host) → **Option 3** (service manager — replaces cron).
## Option 1: watchdog cron + persistent worker
A 5-minute cron checks whether the worker process is alive **and** whether
it has logged an internal shutdown since its last start. Restarts if either
condition fails.
### 1a. Install the env file (secrets stay out of crontab)
Never paste `DATABASE_URL` or API keys into crontab. `/etc/crontab` is
mode 644 (world-readable); user crontabs under `/var/spool/cron/` are
readable by `root`. Use the shipped env-file template:
```bash ```bash
sudo install -m 600 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ # Start (returns PIDs + pid_file on stdout as JSON, then detaches)
gbrain jobs supervisor start --detach --json
# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"}
# Check health (machine-parseable JSON, no log scraping)
gbrain jobs supervisor status --json
# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...}
# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback)
gbrain jobs supervisor stop
```
Every lifecycle event (spawn, crash, backoff, health warning, max-crashes,
shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
for historical inspection. `gbrain doctor` reads that file and surfaces
a `supervisor` check in its health report.
## Deployment: systemd
For long-running Linux VMs with shell access.
```bash
# Create the worker user if it doesn't exist.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \
2>/dev/null || true
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE"
# Install the env file (secrets stay out of the unit file).
sudo install -m 600 -o gbrain -g gbrain \
docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env
sudoedit /etc/gbrain.env sudoedit /etc/gbrain.env
# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1.
# Install the unit file, substituting /srv/gbrain → your workspace path.
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \
/etc/systemd/system/gbrain-worker.service
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \
/etc/systemd/system/gbrain-worker.service
sudo systemctl daemon-reload
sudo systemctl enable --now gbrain-worker
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
``` ```
Fill in the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` (if The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work`
applicable). See directly) so you get two-layer supervision: systemd restarts the supervisor
[`gbrain.env.example`](./minions-deployment-snippets/gbrain.env.example) on host reboot, supervisor restarts the worker on in-process crash.
for the full list.
### 1b. Install the watchdog script `Restart=always` + `RestartSec=10s` handle the supervisor-level recovery.
The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and
audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent
LLM subagent calls without hitting the default 1024 cap.
The [`minion-watchdog.sh`](./minions-deployment-snippets/minion-watchdog.sh) ## Deployment: Fly.io
ships in-repo and writes a two-line PID file (PID on line 1, restart epoch
on line 2). The restart-epoch marker is how the watchdog distinguishes
stale shutdown lines in the log from current ones — without it, every tick
after the first restart would match an old `worker shutting down` line and
loop forever.
Requires GNU coreutils (Linux default). On macOS/BSD install via
`brew install coreutils` and alias `date` to `gdate` in the cron env if you
want to test the watchdog locally; production Linux boxes work as-is.
```bash ```bash
sudo install -m 755 -o $GBRAIN_WORKER_USER -g $GBRAIN_WORKER_USER \ # Merge the [processes] block from fly.toml.partial into your fly.toml.
docs/guides/minions-deployment-snippets/minion-watchdog.sh \ cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml
/usr/local/bin/minion-watchdog.sh # Review + edit as needed.
# Set secrets (Fly handles restart on crash).
fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1
``` ```
### 1c. Wire into cron The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly
restarts the container on host failure; the supervisor restarts the
worker on in-process crash.
Pick the form that matches the crontab you're editing. ## Deployment: Render / Railway / Heroku
**If you ran `crontab -e`** (user crontab — 5-field, no user column): Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo
root. The shipped Procfile calls `gbrain jobs supervisor`. Set
`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's
env UI or CLI.
``` ## Deployment: inline `--follow` (no persistent worker)
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
BASH_ENV=/etc/gbrain.env
*/5 * * * * /usr/local/bin/minion-watchdog.sh
```
**If you edited `/etc/crontab` directly** (system crontab — 6-field, with For short deterministic scripts on a fixed schedule where you don't need
user column): a persistent worker between runs. Each cron run brings its own temporary
worker. `--follow` starts one on the queue and blocks until the
``` just-submitted job reaches a terminal state (`completed` / `failed` /
SHELL=/bin/bash `dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job
PATH=/usr/local/bin:/usr/bin:/bin duration for scheduled work.
BASH_ENV=/etc/gbrain.env
*/5 * * * * gbrain /usr/local/bin/minion-watchdog.sh
```
In both forms, `BASH_ENV=/etc/gbrain.env` tells non-interactive bash to
source the env file before running the watchdog — that's how the
connection string and `GBRAIN_ALLOW_SHELL_JOBS` reach the worker without
landing in the world-readable crontab itself.
### 1d. Log rotation
The watchdog appends to the worker log across restarts. If you expect the
file to grow unbounded, rotate it externally with `logrotate`:
```
# /etc/logrotate.d/gbrain-worker
/tmp/gbrain-worker.log {
daily
rotate 7
missingok
notifempty
copytruncate
}
```
`copytruncate` is important — the watchdog's restart-epoch check survives
it (the epoch is compared against in-log timestamps, not file inode).
## Option 2: inline `--follow` (no persistent worker)
Each cron run brings its own temporary worker. `--follow` starts one on
the queue and blocks until the just-submitted job reaches a terminal state
(`completed` / `failed` / `dead` / `cancelled`). 2-3 s startup overhead
per job; negligible vs job duration for scheduled work.
Example: nightly brain enrichment as a shell job.
```bash ```bash
GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
@@ -3666,85 +3700,56 @@ GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \
Replace `gbrain embed --stale` with whichever gbrain subcommand you're Replace `gbrain embed --stale` with whichever gbrain subcommand you're
scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`, scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`,
`lint`, `autopilot`). If you're shelling out to a non-gbrain binary, `lint`, `autopilot`). For strict single-job semantics on shared queues,
keep its absolute path in the `cmd`.
**Shared-queue gotcha.** If other jobs are already waiting on the same
queue with higher priority or earlier `created_at`, the temporary worker
processes those first before reaching yours. `--follow` still exits only
when YOUR job finishes. For strict single-job semantics on shared queues,
use a dedicated queue name like `nightly-enrich` above. use a dedicated queue name like `nightly-enrich` above.
## Option 3: service manager (systemd / Fly / Render / Railway) ## Upgrading from an older deployment
Replaces the watchdog entirely. No cron, no PID file, no restart-loop. ### From `minion-watchdog.sh` (pre-v0.20)
The service manager owns liveness.
### systemd (Linux hosts with shell access) Earlier versions of this guide shipped a 68-line bash watchdog
(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor`
which handles everything the script did, plus atomic PID locking,
structured audit events, queue-scoped health checks, and graceful
drain on SIGTERM.
**Migration:**
```bash ```bash
# Create the worker user if it doesn't exist. # 1. Stop and remove the old watchdog.
sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null
2>/dev/null || true sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \
sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" /tmp/gbrain-worker.log
crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line
# Install the unit file, substituting /srv/gbrain → your workspace path. # 2. Start the supervisor (systemd users: reinstall the unit from
sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ # docs/guides/minions-deployment-snippets/systemd.service, which
/etc/systemd/system/gbrain-worker.service # now calls `gbrain jobs supervisor`).
sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ gbrain jobs supervisor start --detach --json
/etc/systemd/system/gbrain-worker.service # Or: sudo systemctl restart gbrain-worker
# See 1a above for /etc/gbrain.env install. # 3. Verify.
sudo systemctl daemon-reload gbrain jobs supervisor status --json
sudo systemctl enable --now gbrain-worker gbrain doctor # 'supervisor' check should report running=true
sudo systemctl status gbrain-worker
journalctl -u gbrain-worker -n 50
``` ```
`Restart=always` + `RestartSec=10s` give you crash-loop recovery. The unit ### Schema / migration hygiene
runs as an unprivileged `gbrain` user with `PrivateTmp`, `ProtectSystem=strict`,
and `ReadWritePaths=$GBRAIN_WORKSPACE`. `LimitNOFILE=65535` in the shipped
unit covers Bun + Postgres pool + concurrent LLM subagent calls without
hitting the default 1024 cap.
### Fly.io Regardless of which deployment path you're upgrading from:
Merge the `[processes]` block from 1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop`
[`fly.toml.partial`](./minions-deployment-snippets/fly.toml.partial) into (or `sudo systemctl stop gbrain-worker`). Skipping this risks an
your existing `fly.toml`. Set secrets with `fly secrets set` — in-flight job landing partial schema.
Fly auto-restarts the process on crash.
### Render / Railway / Heroku
Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo root.
Set the connection string and `GBRAIN_ALLOW_SHELL_JOBS=1` via the
platform's env UI or CLI.
## Upgrading an existing deployment
If you deployed on v0.13.x or earlier, walk this checklist:
1. **Stop the worker before upgrading.**
`kill $(head -n1 /tmp/gbrain-worker.pid)` and wait for the process to
exit. Skipping this risks an in-flight job landing partial schema.
2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if 2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if
`gbrain doctor` reports any migration as `partial` or `pending`. `gbrain doctor` reports any migration as `partial` or `pending`.
3. **If you run shell jobs:** from v0.14 onward, the worker requires 3. **If you run shell jobs:** from v0.14 onward, pass
`GBRAIN_ALLOW_SHELL_JOBS=1` to register the `shell` handler. Add it to `--allow-shell-jobs` to the supervisor (or keep
`/etc/gbrain.env`. Submitters don't need the flag; only the worker does. `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't
4. **If you tuned your watchdog for `max_stalled=1`:** v0.14.3 migration need the flag; only the worker does.
v15 raised the schema default to 5 and backfilled existing non-terminal 4. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
rows. A watchdog tuned around 1-strike dead-lettering will now migrations plus a healthy `supervisor` check. `gbrain jobs stats`
over-restart because it takes 5 misses to dead-letter. Switch to the should show no unexplained growth in `dead` between pre- and
shipped watchdog (which keys on log markers, not job state). post-upgrade.
5. **If your v0.16.1 watchdog is still running:** it has a restart-loop
bug (old shutdown lines in the unrotated log re-match every 5 min
forever). Install the current `minion-watchdog.sh` from this guide's
snippets — it writes a restart epoch into the PID file and only
considers log lines newer than that epoch.
6. **Verify.** `gbrain doctor` should report zero `pending` or `partial`
migrations. `gbrain jobs stats` should show no unexplained growth in
`dead` between pre- and post-upgrade.
## Known issues ## Known issues
@@ -3757,9 +3762,10 @@ silently. The stall detector then dead-letters the job after
**Current defaults that make this worse:** **Current defaults that make this worse:**
- `lockDuration: 30000` (30 s) — too short for long jobs during connection blips. - `lockDuration: 30000` (30 s) — too short for long jobs during
- `max_stalled: 5` (schema column default on master — see `src/schema.sql` connection blips.
and `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. - `max_stalled: 5` (schema column default — see `src/schema.sql` and
`src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter.
- `stalledInterval: 30000` (30 s) — checks too aggressively. - `stalledInterval: 30000` (30 s) — checks too aggressively.
**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, **Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`,
@@ -3767,9 +3773,6 @@ silently. The stall detector then dead-letters the job after
`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags `--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags
(since v0.13.1). These write onto the job row at submit time — which is (since v0.13.1). These write onto the job row at submit time — which is
what `handleStalled()` reads — so per-job tuning is the real knob today. what `handleStalled()` reads — so per-job tuning is the real knob today.
Worker-level `--lock-duration` / `--stall-interval` are on the roadmap;
until they land, rely on per-job `--max-stalled` plus the watchdog (or
systemd) for worker health.
### DO NOT pass `maxStalledCount` to `MinionWorker` ### DO NOT pass `maxStalledCount` to `MinionWorker`
@@ -3780,16 +3783,16 @@ Use `gbrain jobs submit --max-stalled N` per-job instead.
### Zombie shell children ### Zombie shell children
When the Bun worker crashes hard, child processes from shell jobs can When the Bun worker crashes hard, child processes from shell jobs can
become zombies. The watchdog's 10 s `SIGTERM → SIGKILL` window covers the become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window
shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For long-running covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For
shell jobs, bump the watchdog's `sleep 10` to `sleep 30` so the worker long-running shell jobs, prefer timeouts via `--timeout-ms` on submit
has time to flush in-flight jobs before the kill. over relying on hard kills.
## Smoke test ## Smoke test
```bash ```bash
# Worker alive? # Supervisor alive?
kill -0 $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null && echo ALIVE || echo DEAD gbrain jobs supervisor status --json | jq .running
# Aggregate queue health. # Aggregate queue health.
gbrain jobs stats gbrain jobs stats
@@ -3800,23 +3803,32 @@ gbrain jobs list --status active --limit 10
# Dead-lettered jobs. # Dead-lettered jobs.
gbrain jobs list --status dead --limit 10 gbrain jobs list --status dead --limit 10
# Shell handler registered? (stderr banner merged into log via 2>&1.) # Shell handler registered? (check supervisor audit log or worker stderr.)
grep "shell handler enabled" /tmp/gbrain-worker.log gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs'
``` ```
## Uninstall ## Uninstall
- **Option 1 (watchdog cron):** `crontab -e`, delete the watchdog line. **`gbrain jobs supervisor`** (foreground or `--detach`):
`kill $(head -n1 /tmp/gbrain-worker.pid) && rm /tmp/gbrain-worker.pid`.
Optionally `sudo rm /etc/gbrain.env /usr/local/bin/minion-watchdog.sh`. ```bash
- **Option 2 (inline `--follow`):** remove the cron entry. Nothing else to gbrain jobs supervisor stop
clean up — temporary workers exit with their jobs. ```
- **Option 3 (systemd):** `sudo systemctl disable --now gbrain-worker`,
then `sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env`, **systemd:**
then `sudo systemctl daemon-reload`.
- **Option 3 (Fly/Render/Railway):** delete the `worker` process from ```bash
`fly.toml` / `Procfile` and redeploy. Secrets set via `fly secrets` sudo systemctl disable --now gbrain-worker
persist until `fly secrets unset`. sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env
sudo systemctl daemon-reload
```
**Fly / Render / Railway:** delete the `worker` process from `fly.toml`
/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until
`fly secrets unset`.
**Inline `--follow`:** remove the cron entry. Nothing else to clean up
— temporary workers exit with their jobs.
--- ---

2
package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "gbrain", "name": "gbrain",
"version": "0.20.0", "version": "0.20.2",
"description": "Postgres-native personal knowledge brain with hybrid RAG search", "description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module", "type": "module",
"main": "src/core/index.ts", "main": "src/core/index.ts",

61
src/commands/doctor.ts
View File

@@ -163,6 +163,67 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
// Read/parse failure is itself best-effort; skip silently. // Read/parse failure is itself best-effort; skip silently.
} }
// 3b-bis. Supervisor health (filesystem-only: PID liveness + audit log).
// Reads the default PID file (`~/.gbrain/supervisor.pid` unless the user
// overrode with GBRAIN_SUPERVISOR_PID_FILE) and the latest audit file
// written by src/core/minions/handlers/supervisor-audit.ts. Surfaces
// supervisor_running / last_start / crashes_24h / max_crashes_exceeded.
// Does NOT run the supervisor itself — this is a read-only health check.
try {
const { DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts');
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
let supervisorPid: number | null = null;
let running = false;
if (existsSync(DEFAULT_PID_FILE)) {
try {
const line = readFileSync(DEFAULT_PID_FILE, 'utf8').trim().split('\n')[0];
const parsed = parseInt(line, 10);
if (!isNaN(parsed) && parsed > 0) {
supervisorPid = parsed;
try { process.kill(parsed, 0); running = true; } catch { running = false; }
}
} catch { /* unreadable */ }
}
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
const crashes24h = events.filter(e => e.event === 'worker_exited').length;
const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null;
// Only surface a Check if the supervisor was ever observed (stops the
// "never used the supervisor" install from getting a warn about it).
if (supervisorPid !== null || events.length > 0) {
if (maxCrashesEvent) {
checks.push({
name: 'supervisor',
status: 'fail',
message: `Supervisor gave up at ${maxCrashesEvent.ts} (max_crashes_exceeded). Restart with: gbrain jobs supervisor start --detach`,
});
} else if (!running && events.length > 0) {
checks.push({
name: 'supervisor',
status: 'warn',
message: `Supervisor not running (last_start=${lastStart ?? 'unknown'}). Restart with: gbrain jobs supervisor start --detach`,
});
} else if (crashes24h > 3) {
checks.push({
name: 'supervisor',
status: 'warn',
message: `Supervisor running but worker crashed ${crashes24h}x in last 24h. Check ~/.gbrain/audit/supervisor-*.jsonl for causes.`,
});
} else {
checks.push({
name: 'supervisor',
status: 'ok',
message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h}`,
});
}
}
} catch {
// Audit read / import failure is best-effort; skip silently.
}
// 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit` // 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit`
// bookmark when per-file parse errors happen, and appends each failure // bookmark when per-file parse errors happen, and appends each failure
// to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error. // to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error.

215
src/commands/jobs.ts
View File

@@ -70,6 +70,42 @@ USAGE
gbrain jobs stats gbrain jobs stats
gbrain jobs smoke gbrain jobs smoke
gbrain jobs work [--queue Q] [--concurrency N] gbrain jobs work [--queue Q] [--concurrency N]
gbrain jobs supervisor [start] [--detach] [--json]
[--concurrency N] [--queue Q] [--pid-file PATH]
[--max-crashes N] [--health-interval N]
[--allow-shell-jobs] [--cli-path PATH]
gbrain jobs supervisor status [--json] [--pid-file PATH]
gbrain jobs supervisor stop [--json] [--pid-file PATH]
Auto-restarting wrapper around 'gbrain jobs work'. Spawns the worker
as a child process and restarts on crash with exponential backoff
(1s -> 60s cap). Writes a PID file to ~/.gbrain/supervisor.pid by
default (override via --pid-file or GBRAIN_SUPERVISOR_PID_FILE env).
Lifecycle events are appended to
\${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl
SUBCOMMANDS
start (default) Launch the supervisor. --detach returns a
JSON {event, supervisor_pid, pid_file} payload on
stdout and forks; omit for foreground.
status Read PID file + audit log, report running / last_start
/ crashes_24h / max_crashes_exceeded as JSON or human.
Exits 0 if running, 1 if not.
stop Send SIGTERM to the supervisor, wait up to 40s for
graceful drain, report outcome. Exits 0 on clean stop.
EXIT CODES (start)
0 clean shutdown (SIGTERM/SIGINT received, worker drained)
1 max crashes exceeded (worker kept dying)
2 another supervisor holds the PID lock
3 PID file unwritable (permission / path error)
EXAMPLES
gbrain jobs supervisor --concurrency 4 # foreground (Ctrl-C stops)
gbrain jobs supervisor start --detach --json # agent-friendly: fork + return JSON
gbrain jobs supervisor status --json # machine-readable health check
gbrain jobs supervisor stop # graceful stop
gbrain jobs supervisor --json --allow-shell-jobs # JSONL events + shell-exec on
HANDLER TYPES (built in) HANDLER TYPES (built in)
sync Pull and embed new pages from the repo sync Pull and embed new pages from the repo
@@ -481,6 +517,185 @@ HANDLER TYPES (built in)
break; break;
} }
case 'supervisor': {
// Dispatcher for supervisor subcommands:
// gbrain jobs supervisor → foreground start (back-compat)
// gbrain jobs supervisor start [--detach] → foreground or detached start
// gbrain jobs supervisor status → JSON liveness + queue stats
// gbrain jobs supervisor stop → SIGTERM + drain wait
const { MinionSupervisor, DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts');
const { writeSupervisorEvent } = await import('../core/minions/handlers/supervisor-audit.ts');
const supCmd = args[1];
const isStatusCmd = supCmd === 'status';
const isStopCmd = supCmd === 'stop';
const isStartCmd = supCmd === 'start' || supCmd === undefined || supCmd === '--detach' ||
(typeof supCmd === 'string' && supCmd.startsWith('--'));
const jsonMode = hasFlag(args, '--json');
const pidFile = parseFlag(args, '--pid-file') ?? DEFAULT_PID_FILE;
// ----- status subcommand -----
if (isStatusCmd) {
const { existsSync, readFileSync } = await import('fs');
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
let supervisorPid: number | null = null;
let running = false;
if (existsSync(pidFile)) {
try {
const line = readFileSync(pidFile, 'utf8').trim().split('\n')[0];
const parsed = parseInt(line, 10);
if (!isNaN(parsed) && parsed > 0) {
supervisorPid = parsed;
try { process.kill(parsed, 0); running = true; } catch { running = false; }
}
} catch { /* unreadable PID file */ }
}
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
const crashes24h = events.filter(e => e.event === 'worker_exited').length;
const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null;
const status = {
running,
supervisor_pid: supervisorPid,
pid_file: pidFile,
last_start: lastStart,
crashes_24h: crashes24h,
max_crashes_exceeded: !!maxCrashesEvent,
};
if (jsonMode) {
console.log(JSON.stringify(status, null, 2));
} else {
console.log(`Supervisor: ${running ? 'running' : 'not running'}`);
if (supervisorPid) console.log(` PID: ${supervisorPid}`);
console.log(` PID file: ${pidFile}`);
if (lastStart) console.log(` Last start: ${lastStart}`);
console.log(` Crashes (24h): ${crashes24h}`);
if (maxCrashesEvent) console.log(` ⚠ Max crashes exceeded at ${maxCrashesEvent.ts}`);
}
process.exit(running ? 0 : 1);
}
// ----- stop subcommand -----
if (isStopCmd) {
const { existsSync, readFileSync } = await import('fs');
if (!existsSync(pidFile)) {
const payload = { stopped: false, reason: 'pid_file_missing', pid_file: pidFile };
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`No PID file at ${pidFile}; supervisor not running.`);
process.exit(1);
}
let supervisorPid: number;
try {
supervisorPid = parseInt(readFileSync(pidFile, 'utf8').trim().split('\n')[0], 10);
if (isNaN(supervisorPid) || supervisorPid <= 0) throw new Error('invalid pid');
} catch (err) {
const payload = { stopped: false, reason: 'pid_file_corrupt', error: String(err) };
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`PID file corrupt: ${err}`);
process.exit(1);
}
try { process.kill(supervisorPid, 'SIGTERM'); }
catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
const payload = {
stopped: false,
reason: code === 'ESRCH' ? 'process_gone' : 'kill_failed',
supervisor_pid: supervisorPid,
};
if (jsonMode) console.log(JSON.stringify(payload));
else console.error(`Cannot signal PID ${supervisorPid}: ${err}`);
process.exit(code === 'ESRCH' ? 0 : 1);
}
// Poll for up to 40s (supervisor's own 35s drain + 5s slack).
const deadline = Date.now() + 40_000;
let stoppedCleanly = false;
while (Date.now() < deadline) {
try { process.kill(supervisorPid, 0); }
catch { stoppedCleanly = true; break; }
await new Promise(r => setTimeout(r, 250));
}
const payload = {
stopped: stoppedCleanly,
supervisor_pid: supervisorPid,
reason: stoppedCleanly ? 'drained' : 'timeout_40s',
};
if (jsonMode) console.log(JSON.stringify(payload));
else console.log(stoppedCleanly ? `Supervisor ${supervisorPid} stopped.` : `Supervisor ${supervisorPid} did not exit within 40s.`);
process.exit(stoppedCleanly ? 0 : 1);
}
// ----- start subcommand (default) -----
if (!isStartCmd) {
console.error(`Unknown supervisor subcommand: ${supCmd}. Expected: start, status, stop.`);
process.exit(1);
}
const config = (await import('../core/config.ts')).loadConfig();
if (config?.engine === 'pglite') {
console.error('Error: Supervisor requires Postgres. PGLite uses an exclusive file lock that blocks other processes.');
process.exit(1);
}
const { resolveGbrainCliPath } = await import('./autopilot.ts');
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '2', 10);
const queueName = parseFlag(args, '--queue') ?? 'default';
const maxCrashes = parseInt(parseFlag(args, '--max-crashes') ?? '10', 10);
const healthInterval = parseInt(parseFlag(args, '--health-interval') ?? '60000', 10);
const allowShellJobs = hasFlag(args, '--allow-shell-jobs') ||
!!process.env.GBRAIN_ALLOW_SHELL_JOBS;
const detach = hasFlag(args, '--detach');
const cliPath = parseFlag(args, '--cli-path') ?? resolveGbrainCliPath();
// --detach: fork a background supervisor, print PID payload, exit 0.
// Implementation: re-exec the same CLI as a detached child without --detach,
// inheriting stderr (so JSONL events still flow to the parent's tail-f
// if they wanted to follow logs) but detaching stdin/stdout.
if (detach) {
const { spawn } = await import('child_process');
const childArgs = process.argv.slice(2).filter(a => a !== '--detach');
const child = spawn(process.execPath, [process.argv[1], ...childArgs], {
detached: true,
stdio: ['ignore', 'ignore', 'inherit'],
env: process.env,
});
child.unref();
const payload = {
event: 'started',
supervisor_pid: child.pid,
pid_file: pidFile,
detached: true,
};
console.log(JSON.stringify(payload));
process.exit(0);
}
// Foreground start.
const supervisorPid = process.pid;
const supervisor = new MinionSupervisor(engine, {
concurrency,
queue: queueName,
pidFile,
maxCrashes,
healthInterval,
cliPath,
allowShellJobs,
json: jsonMode,
onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid),
});
await supervisor.start();
break;
}
default: default:
console.error(`Unknown subcommand: ${sub}. Run 'gbrain jobs --help' for usage.`); console.error(`Unknown subcommand: ${sub}. Run 'gbrain jobs --help' for usage.`);
process.exit(1); process.exit(1);

107
src/core/minions/handlers/supervisor-audit.ts Normal file
View File

@@ -0,0 +1,107 @@
/**
* Supervisor lifecycle audit log. JSONL, weekly-rotated, best-effort.
*
* Writes one line per supervisor event (started, worker_spawned, worker_exited,
* backoff, health_warn, health_error, max_crashes_exceeded, shutting_down,
* stopped, worker_spawn_failed) to
* `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`
* using ISO-8601 week numbering. `computeAuditFilename(kind, now)` derives
* the filename; the ISO-week math is shared with `shell-audit.ts` via the
* `computeIsoWeekName()` helper that both call.
*
* Shape: every emission already includes `event` and `ts`; we write it
* verbatim and let consumers (like `gbrain doctor`) grep for events of
* interest. `supervisor_pid` is added at start() time so each line is
* self-describing even if a log shipper concatenates multiple supervisors'
* files.
*
* Best-effort: write failures go to stderr and never block supervisor work.
* A disk-full attacker could silently disable the trail — this is an
* operational trace for `gbrain doctor`, not forensic insurance.
*
* `GBRAIN_AUDIT_DIR` overrides the default `~/.gbrain/audit/` path for
* container deploys where `$HOME` is read-only.
*/
import * as fs from 'node:fs';
import * as path from 'node:path';
import { resolveAuditDir } from './shell-audit.ts';
import type { SupervisorEmission } from '../supervisor.ts';
/**
* Compute `supervisor-YYYY-Www.jsonl` using ISO-8601 week numbering.
*
* Mirrors `shell-audit.ts:computeAuditFilename()` exactly. Year-boundary
* edge: 2027-01-01 is ISO week 53 of year 2026, so the correct filename
* is `supervisor-2026-W53.jsonl`.
*/
export function computeSupervisorAuditFilename(now: Date = new Date()): string {
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
const dayNum = (d.getUTCDay() + 6) % 7; // Mon=0, Sun=6
d.setUTCDate(d.getUTCDate() - dayNum + 3); // shift to Thursday (ISO week anchor)
const isoYear = d.getUTCFullYear();
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
const ww = String(weekNum).padStart(2, '0');
return `supervisor-${isoYear}-W${ww}.jsonl`;
}
/**
* Append a single supervisor lifecycle event to the rotated JSONL audit
* file. `supervisorPid` is the OS pid of the supervisor process (added
* to every line so a log shipper concatenating files from multiple
* supervisors still produces parseable traces).
*/
export function writeSupervisorEvent(emission: SupervisorEmission, supervisorPid: number): void {
const dir = resolveAuditDir();
const filename = computeSupervisorAuditFilename();
const fullPath = path.join(dir, filename);
const line = JSON.stringify({ ...emission, supervisor_pid: supervisorPid }) + '\n';
try {
fs.mkdirSync(dir, { recursive: true });
fs.appendFileSync(fullPath, line, { encoding: 'utf8' });
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
process.stderr.write(`[supervisor-audit] write failed (${msg}); continuing\n`);
}
}
/**
* Read back the latest supervisor audit file. Returns events sorted
* oldest-first. Best-effort: missing file / parse errors return [].
* Used by `gbrain doctor` (Lane D) to surface supervisor health.
*/
export function readSupervisorEvents(opts: { sinceMs?: number } = {}): SupervisorEmission[] {
const dir = resolveAuditDir();
const filename = computeSupervisorAuditFilename();
const fullPath = path.join(dir, filename);
let raw: string;
try {
raw = fs.readFileSync(fullPath, 'utf8');
} catch {
return [];
}
const now = Date.now();
const cutoff = opts.sinceMs !== undefined ? now - opts.sinceMs : 0;
const events: SupervisorEmission[] = [];
for (const line of raw.split('\n')) {
if (!line.trim()) continue;
try {
const obj = JSON.parse(line) as SupervisorEmission;
if (!obj.event || !obj.ts) continue;
if (cutoff > 0) {
const ts = Date.parse(obj.ts);
if (!isNaN(ts) && ts < cutoff) continue;
}
events.push(obj);
} catch {
// Ignore malformed lines (truncated writes, disk-full corruption).
}
}
return events;
}

573
src/core/minions/supervisor.ts Normal file
View File

@@ -0,0 +1,573 @@
/**
* MinionSupervisor — Process manager for the Minion worker.
*
* Spawns `gbrain jobs work` as a child process and restarts it on crash
* with exponential backoff. Provides health monitoring, PID file locking
* (atomic via O_CREAT|O_EXCL), and graceful shutdown.
*
* ENGINE: Postgres only. PGLite uses an exclusive file lock that blocks
* any separate worker process, so `gbrain jobs supervisor` cannot work
* against a PGLite brain — `src/commands/jobs.ts` rejects that combination
* at the CLI layer. The health-check SQL below assumes Postgres schema.
*
* Usage:
* gbrain jobs supervisor [--concurrency N] [--queue Q] [--pid-file PATH]
* [--max-crashes N] [--health-interval N]
* [--allow-shell-jobs] [--json]
*
* Design: the supervisor does NOT run the worker in-process. It spawns a
* separate child so a misbehaving handler can't take down the supervisor.
* Same isolation pattern as autopilot.ts but standalone and reusable.
*
* Exit codes (documented in CLI --help):
* 0 clean shutdown (SIGTERM/SIGINT received, worker drained)
* 1 max crashes exceeded (worker kept dying)
* 2 another supervisor holds the PID lock
* 3 PID file unwritable (permission / path error)
*/
import { spawn, type ChildProcess } from 'child_process';
import {
closeSync,
existsSync,
mkdirSync,
openSync,
readFileSync,
unlinkSync,
writeFileSync,
writeSync,
} from 'fs';
import { dirname } from 'path';
import type { BrainEngine } from '../engine.ts';
export type SupervisorEvent =
| 'started'
| 'worker_spawned'
| 'worker_exited'
| 'worker_spawn_failed'
| 'backoff'
| 'health_warn'
| 'health_error'
| 'max_crashes_exceeded'
| 'shutting_down'
| 'stopped';
export interface SupervisorEmission {
event: SupervisorEvent;
ts: string;
[key: string]: unknown;
}
export interface SupervisorOpts {
/** Worker concurrency (passed to child). Default: 2. */
concurrency: number;
/** Queue name (passed to child). Default: 'default'. */
queue: string;
/** PID file path. Default: `${HOME}/.gbrain/supervisor.pid` (parent dir auto-created). */
pidFile: string;
/** Max consecutive crashes before giving up. Default: 10. */
maxCrashes: number;
/** Health check interval in ms. Default: 60000. */
healthInterval: number;
/** Path to the gbrain CLI executable (MUST be a compiled binary; .ts sources cannot be spawned). */
cliPath: string;
/** Allow shell jobs on child worker. Default: false. When true, sets GBRAIN_ALLOW_SHELL_JOBS=1 on child env. */
allowShellJobs: boolean;
/** JSON mode: emit JSONL events on stderr, reserve stdout for data payloads. Default: false. */
json: boolean;
/** Optional event sink (Lane C audit writer). Called for every lifecycle event. */
onEvent?: (event: SupervisorEmission) => void;
/**
* Test-only override: minimum backoff in ms between child respawns. Default: undefined
* (uses full `calculateBackoffMs()` curve). Tests pass `1` to make crash-loops finish
* in < 1s. Not exposed via CLI.
* @internal
*/
_backoffFloorMs?: number;
}
export const DEFAULT_PID_FILE: string = (() => {
const envOverride = process.env.GBRAIN_SUPERVISOR_PID_FILE;
if (envOverride && envOverride.length > 0) return envOverride;
const home = process.env.HOME ?? '/tmp';
return `${home}/.gbrain/supervisor.pid`;
})();
const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
concurrency: 2,
queue: 'default',
pidFile: DEFAULT_PID_FILE,
maxCrashes: 10,
healthInterval: 60_000,
allowShellJobs: false,
json: false,
};
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
export function calculateBackoffMs(crashCount: number): number {
const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000);
// Add 10% jitter
return base + Math.random() * base * 0.1;
}
/** Check if a PID is alive. */
function isProcessAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
/** Exit codes for documented agent branching. */
export const ExitCodes = {
CLEAN: 0,
MAX_CRASHES: 1,
LOCK_HELD: 2,
PID_UNWRITABLE: 3,
} as const;
export class MinionSupervisor {
private opts: SupervisorOpts;
private engine: BrainEngine;
private child: ChildProcess | null = null;
private crashCount = 0;
private lastStartTime = 0;
private stopping = false;
private inBackoff = false;
private healthInFlight = false;
private healthTimer: ReturnType<typeof setInterval> | null = null;
private exitListener: (() => void) | null = null;
private sigtermListener: (() => void) | null = null;
private sigintListener: (() => void) | null = null;
private lockAcquired = false;
constructor(engine: BrainEngine, opts: Partial<SupervisorOpts> & { cliPath: string }) {
this.engine = engine;
this.opts = { ...DEFAULTS, ...opts };
}
/**
* Emit a lifecycle event. In JSON mode, writes a JSONL record to stderr.
* In human mode, writes a human-readable log line to stdout (info) or
* stderr (warn/error). Also calls `opts.onEvent` if set (Lane C audit
* writer hooks here).
*/
private emit(event: SupervisorEvent, fields: Record<string, unknown> = {}): void {
const emission: SupervisorEmission = {
event,
ts: new Date().toISOString(),
...fields,
};
if (this.opts.json) {
// stderr is the event channel; stdout stays clean for data (e.g., --detach payload).
try {
process.stderr.write(JSON.stringify(emission) + '\n');
} catch { /* best effort */ }
} else {
const ts = emission.ts.slice(11, 19);
const detail = Object.entries(fields)
.filter(([k]) => k !== 'event' && k !== 'ts')
.map(([k, v]) => `${k}=${typeof v === 'string' ? v : JSON.stringify(v)}`)
.join(' ');
const isWarn = event === 'health_warn' || event === 'health_error' ||
event === 'worker_spawn_failed' || event === 'max_crashes_exceeded';
const line = `[supervisor ${ts}] ${event}${detail ? ' ' + detail : ''}`;
if (isWarn) {
console.warn(line);
} else {
console.log(line);
}
}
// Audit sink (Lane C plumbs this).
if (this.opts.onEvent) {
try { this.opts.onEvent(emission); } catch { /* best effort */ }
}
}
/** Start the supervisor. Blocks until stopped or max crashes exceeded. */
async start(): Promise<void> {
// 1. PID file lock (atomic via O_CREAT|O_EXCL).
const lockResult = this.acquirePidLock();
if (lockResult === 'held') {
// Another supervisor owns the lock — exit code 2.
process.exit(ExitCodes.LOCK_HELD);
}
if (lockResult === 'unwritable') {
// PID path isn't writable — exit code 3 with helpful hint.
process.exit(ExitCodes.PID_UNWRITABLE);
}
// 2. Cleanup on process exit (covers any exit path including process.exit).
this.exitListener = () => {
try {
if (existsSync(this.opts.pidFile)) {
const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0];
if (contents === String(process.pid)) {
unlinkSync(this.opts.pidFile);
}
}
} catch { /* best effort */ }
};
process.on('exit', this.exitListener);
// 3. Signal handlers (tracked refs; removed on shutdown for test lifecycle hygiene).
this.sigtermListener = () => { void this.shutdown('SIGTERM', ExitCodes.CLEAN); };
this.sigintListener = () => { void this.shutdown('SIGINT', ExitCodes.CLEAN); };
process.on('SIGTERM', this.sigtermListener);
process.on('SIGINT', this.sigintListener);
// 4. Health monitoring.
this.healthTimer = setInterval(() => { void this.healthCheck(); }, this.opts.healthInterval);
// 5. Announce start.
this.emit('started', {
supervisor_pid: process.pid,
pid_file: this.opts.pidFile,
concurrency: this.opts.concurrency,
queue: this.opts.queue,
max_crashes: this.opts.maxCrashes,
});
// 6. Run the supervise loop (respawn on crash, bounded by maxCrashes).
await this.runSuperviseLoop();
}
/** Unified shutdown path. Reason becomes the audit event name; exitCode is process exit. */
private async shutdown(reason: string, exitCode: number): Promise<void> {
if (this.stopping) return;
this.stopping = true;
this.emit('shutting_down', { reason, exit_code: exitCode });
if (this.healthTimer) {
clearInterval(this.healthTimer);
this.healthTimer = null;
}
if (this.child) {
try { this.child.kill('SIGTERM'); } catch { /* already dead */ }
await Promise.race([
new Promise<void>(r => this.child!.once('exit', () => r())),
new Promise<void>(r => setTimeout(() => r(), 35_000)),
]);
if (this.child && !this.child.killed) {
try { this.child.kill('SIGKILL'); } catch { /* already dead */ }
}
}
// Remove signal handlers so tests that spin up multiple supervisors on
// the same process don't accumulate listeners. `process.on('exit', ...)`
// is kept registered — it needs to fire synchronously on the final exit.
if (this.sigtermListener) {
process.removeListener('SIGTERM', this.sigtermListener);
this.sigtermListener = null;
}
if (this.sigintListener) {
process.removeListener('SIGINT', this.sigintListener);
this.sigintListener = null;
}
this.emit('stopped', { reason, exit_code: exitCode });
process.exit(exitCode);
}
/**
* Acquire PID file lock atomically via O_CREAT|O_EXCL.
*
* Returns:
* 'acquired' — lock is ours, safe to proceed.
* 'held' — another live supervisor owns the lock (exit code 2).
* 'unwritable' — can't write to the PID path (permission / missing parent, exit code 3).
*/
private acquirePidLock(): 'acquired' | 'held' | 'unwritable' {
// Ensure parent directory exists. Idempotent; creates ~/.gbrain on fresh installs.
try {
mkdirSync(dirname(this.opts.pidFile), { recursive: true });
} catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
if (code !== 'EEXIST') {
console.error(
`Cannot create PID file directory ${dirname(this.opts.pidFile)}: ${
err instanceof Error ? err.message : String(err)
}. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.`
);
return 'unwritable';
}
}
return this.tryAtomicCreate();
}
private tryAtomicCreate(): 'acquired' | 'held' | 'unwritable' {
try {
// O_CREAT | O_EXCL | O_WRONLY — fails with EEXIST if the file exists.
const fd = openSync(this.opts.pidFile, 'wx');
try {
writeSync(fd, String(process.pid));
} finally {
closeSync(fd);
}
this.lockAcquired = true;
return 'acquired';
} catch (err: unknown) {
const code = (err as NodeJS.ErrnoException)?.code;
if (code === 'EEXIST') {
// File exists — check if the owner is alive.
let existingPid = -1;
try {
const contents = readFileSync(this.opts.pidFile, 'utf8').trim().split('\n')[0];
existingPid = parseInt(contents, 10);
} catch { /* corrupt file */ }
if (!isNaN(existingPid) && existingPid > 0 && isProcessAlive(existingPid)) {
console.error(`Supervisor already running (PID: ${existingPid}). Exiting.`);
return 'held';
}
// Stale PID file — unlink and retry atomic create once.
try { unlinkSync(this.opts.pidFile); } catch { /* race with another stale-cleaner; retry will EEXIST again */ }
try {
const fd = openSync(this.opts.pidFile, 'wx');
try {
writeSync(fd, String(process.pid));
} finally {
closeSync(fd);
}
this.lockAcquired = true;
return 'acquired';
} catch (retryErr) {
const retryCode = (retryErr as NodeJS.ErrnoException)?.code;
if (retryCode === 'EEXIST') {
// Someone else won the race. Treat as held.
console.error(`Another supervisor took the PID lock during stale cleanup. Exiting.`);
return 'held';
}
console.error(
`Cannot write PID file ${this.opts.pidFile}: ${
retryErr instanceof Error ? retryErr.message : String(retryErr)
}`
);
return 'unwritable';
}
}
console.error(
`Cannot write PID file ${this.opts.pidFile}: ${
err instanceof Error ? err.message : String(err)
}. Set GBRAIN_SUPERVISOR_PID_FILE or pass --pid-file to a writable location.`
);
return 'unwritable';
}
}
/** Run the supervise loop: spawn child, await exit, backoff+retry or give up. */
private async runSuperviseLoop(): Promise<void> {
while (!this.stopping && this.crashCount < this.opts.maxCrashes) {
await this.spawnOnce();
if (this.stopping) return;
if (this.crashCount >= this.opts.maxCrashes) {
this.emit('max_crashes_exceeded', {
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
});
await this.shutdown('max_crashes', ExitCodes.MAX_CRASHES);
return;
}
// crashCount - 1 is the retry-attempt index (0-based exponent for backoff math).
// On first crash: crashCount=1, backoff exponent=0 → 1s.
// After stable-run reset: crashCount=1 again → 1s fresh cycle.
// Test-only: _backoffFloorMs short-circuits to a fixed tiny value so integration
// tests can exercise crash loops in < 1s without waiting for the real curve.
const backoff = this.opts._backoffFloorMs !== undefined
? this.opts._backoffFloorMs
: calculateBackoffMs(this.crashCount - 1);
this.emit('backoff', { ms: Math.round(backoff), crash_count: this.crashCount });
this.inBackoff = true;
try {
await new Promise<void>(r => setTimeout(r, backoff));
} finally {
this.inBackoff = false;
}
}
}
/** Spawn the worker child once and await its exit. Updates `this.crashCount`. */
private spawnOnce(): Promise<void> {
return new Promise<void>((resolve) => {
if (this.stopping) { resolve(); return; }
const args = [
'jobs', 'work',
'--concurrency', String(this.opts.concurrency),
'--queue', this.opts.queue,
];
// Build child env. Explicit handling for GBRAIN_ALLOW_SHELL_JOBS:
// inherit only when caller opts in, otherwise strip from the clone.
const env: Record<string, string | undefined> = { ...process.env };
if (this.opts.allowShellJobs) {
env.GBRAIN_ALLOW_SHELL_JOBS = '1';
} else {
delete env.GBRAIN_ALLOW_SHELL_JOBS;
}
this.lastStartTime = Date.now();
let child: ChildProcess;
try {
child = spawn(this.opts.cliPath, args, {
stdio: 'inherit',
env,
});
} catch (err: unknown) {
// Synchronous spawn error (e.g., invalid cliPath shape). Count as a crash.
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err instanceof Error ? err.message : String(err),
phase: 'sync',
});
this.crashCount++;
resolve();
return;
}
this.child = child;
this.emit('worker_spawned', { pid: child.pid, cli_path: this.opts.cliPath });
// Async spawn errors (ENOENT, EACCES after the fork/exec). Node fires
// 'error' first, then 'exit' with code=null. We log the error; the
// 'exit' handler increments crashCount as usual so the restart loop
// continues (max-crashes bounds this for permanent misconfigs).
child.on('error', (err) => {
this.emit('worker_spawn_failed', {
cli_path: this.opts.cliPath,
error: err.message,
code: (err as NodeJS.ErrnoException).code ?? 'unknown',
phase: 'async',
});
});
child.on('exit', (code, signal) => {
this.child = null;
if (this.stopping) {
resolve();
return;
}
// Stable-run reset: if the worker ran > 5min before crashing, we forgive
// prior crash history and treat this as the first crash of a new cycle
// (crashCount = 1, so backoff math uses retry-index 0 = 1s).
const runDuration = Date.now() - this.lastStartTime;
if (runDuration > 5 * 60 * 1000) {
this.crashCount = 1;
} else {
this.crashCount++;
}
const exitReason = signal ? `signal ${signal}` : `code ${code ?? 'null'}`;
this.emit('worker_exited', {
code: code ?? null,
signal: signal ?? null,
reason: exitReason,
crash_count: this.crashCount,
max_crashes: this.opts.maxCrashes,
run_duration_ms: runDuration,
});
resolve();
});
});
}
/**
* Periodic health check — queries DB for queue health indicators.
*
* POSTGRES-ONLY. The supervisor cannot run against PGLite (exclusive
* file lock blocks the separate worker process). The CLI layer rejects
* that combination; we assume Postgres here.
*
* F9 guard: skip if a previous check is still in flight (hung DB
* connection shouldn't stack duplicate checks).
*/
private async healthCheck(): Promise<void> {
if (this.healthInFlight) return;
this.healthInFlight = true;
try {
// Blocker 2+3+6: single FILTER query scoped to this.opts.queue.
// 'stalled' = active jobs whose lock_until has passed (matches
// queue.ts:848 handleStalled() definition — same set that the queue
// itself will requeue/dead-letter on next tick).
const rows = await this.engine.executeRaw<{
stalled: string;
waiting: string;
last_completed: string | null;
}>(
`SELECT
count(*) FILTER (WHERE status = 'active' AND lock_until < now())::text AS stalled,
count(*) FILTER (WHERE status = 'waiting')::text AS waiting,
max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed
FROM minion_jobs
WHERE queue = $1`,
[this.opts.queue],
);
const row = rows[0] ?? { stalled: '0', waiting: '0', last_completed: null };
const stalledCount = parseInt(row.stalled ?? '0', 10);
const waitingCount = parseInt(row.waiting ?? '0', 10);
const lastCompleted = row.last_completed ? new Date(row.last_completed) : null;
const now = Date.now();
const minutesSinceCompletion = lastCompleted
? Math.round((now - lastCompleted.getTime()) / 60_000)
: null;
// F2 (per-threshold warns) — each is a distinct health_warn with reason.
if (stalledCount > 10) {
this.emit('health_warn', {
reason: 'stalled_jobs',
count: stalledCount,
queue: this.opts.queue,
});
}
if (waitingCount > 0 && minutesSinceCompletion !== null && minutesSinceCompletion > 30) {
this.emit('health_warn', {
reason: 'no_recent_completions',
waiting_count: waitingCount,
minutes_since_completion: minutesSinceCompletion,
queue: this.opts.queue,
});
}
// F4: suppress "worker not alive" warn while we're in the expected
// null-child window (crash-exit → backoff-sleep → next-spawn).
const workerAlive = this.child != null && this.child.exitCode === null;
if (!workerAlive && !this.stopping && !this.inBackoff) {
this.emit('health_warn', {
reason: 'worker_not_alive',
queue: this.opts.queue,
});
}
} catch (e) {
// Health check failures are non-fatal.
this.emit('health_error', {
error: e instanceof Error ? e.message : String(e),
queue: this.opts.queue,
});
} finally {
this.healthInFlight = false;
}
}
}

63
test/fixtures/supervisor-runner.ts vendored Normal file
View File

@@ -0,0 +1,63 @@
/**
* Test fixture: spawns a MinionSupervisor with options parsed from env vars.
*
* Used by test/supervisor.test.ts integration tests. Separate file because
* the supervisor calls `process.exit()` at the end of its lifecycle — tests
* spawn this runner as a subprocess to observe exit codes and audit events
* without killing the test runner itself.
*
* Env vars (all optional, sensible defaults for tests):
* SUP_CLI_PATH — worker binary path (default: /bin/sh exit-1 script)
* SUP_PID_FILE — PID file path (REQUIRED; each test uses a unique one)
* SUP_MAX_CRASHES — max consecutive crashes (default: 3)
* SUP_BACKOFF_FLOOR_MS — test-only short backoff (default: 1)
* SUP_HEALTH_INTERVAL_MS — how often healthCheck fires (default: 999_999 off)
* SUP_ALLOW_SHELL_JOBS — "1" to set allowShellJobs:true, else false
* SUP_QUEUE — queue name (default: 'default')
* SUP_AUDIT_DIR — GBRAIN_AUDIT_DIR override (default: tmpdir/supervisor-test)
*/
import { MinionSupervisor } from '../../src/core/minions/supervisor.ts';
import { writeSupervisorEvent } from '../../src/core/minions/handlers/supervisor-audit.ts';
import type { BrainEngine } from '../../src/core/engine.ts';
// Mock engine: healthCheck() calls engine.executeRaw; return empty rows so
// the query path exercises without needing Postgres.
const mockEngine: Partial<BrainEngine> = {
kind: 'postgres' as const,
executeRaw: async () => [],
} as unknown as BrainEngine;
const pidFile = process.env.SUP_PID_FILE;
if (!pidFile) {
console.error('SUP_PID_FILE env var is required');
process.exit(99);
}
const cliPath = process.env.SUP_CLI_PATH ?? '/bin/sh';
const maxCrashes = parseInt(process.env.SUP_MAX_CRASHES ?? '3', 10);
const backoffFloor = parseInt(process.env.SUP_BACKOFF_FLOOR_MS ?? '1', 10);
const healthInterval = parseInt(process.env.SUP_HEALTH_INTERVAL_MS ?? '999999', 10);
const allowShellJobs = process.env.SUP_ALLOW_SHELL_JOBS === '1';
const queueName = process.env.SUP_QUEUE ?? 'default';
if (process.env.SUP_AUDIT_DIR) {
process.env.GBRAIN_AUDIT_DIR = process.env.SUP_AUDIT_DIR;
}
const supervisorPid = process.pid;
const supervisor = new MinionSupervisor(mockEngine as BrainEngine, {
concurrency: 1,
queue: queueName,
pidFile,
maxCrashes,
healthInterval,
cliPath,
allowShellJobs,
json: true,
_backoffFloorMs: backoffFloor,
onEvent: (emission) => writeSupervisorEvent(emission, supervisorPid),
});
await supervisor.start();

343
test/supervisor.test.ts Normal file
View File

@@ -0,0 +1,343 @@
import { describe, it, expect, afterEach } from 'bun:test';
import { existsSync, readFileSync, writeFileSync, unlinkSync, chmodSync, mkdirSync, rmSync } from 'fs';
import { spawn } from 'child_process';
import { join } from 'path';
import { tmpdir } from 'os';
import { readSupervisorEvents, computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
import { calculateBackoffMs } from '../src/core/minions/supervisor.ts';
const TEST_PID_FILE = '/tmp/gbrain-supervisor-test.pid';
afterEach(() => {
try { unlinkSync(TEST_PID_FILE); } catch { /* noop */ }
});
// ----- Integration test helpers -----
interface IntegrationHarness {
pidFile: string;
auditDir: string;
workerScript: string;
envOutFile: string;
cleanup: () => void;
}
/** Create per-test temp files + a fake worker shell script. */
function makeHarness(name: string, workerBody: string): IntegrationHarness {
const tmpRoot = join(tmpdir(), `gbrain-sup-test-${name}-${process.pid}-${Date.now()}`);
mkdirSync(tmpRoot, { recursive: true });
const pidFile = join(tmpRoot, 'supervisor.pid');
const auditDir = join(tmpRoot, 'audit');
const workerScript = join(tmpRoot, 'worker.sh');
const envOutFile = join(tmpRoot, 'env-out.txt');
writeFileSync(workerScript, `#!/bin/sh\n${workerBody}\n`, 'utf8');
chmodSync(workerScript, 0o755);
return {
pidFile,
auditDir,
workerScript,
envOutFile,
cleanup: () => { try { rmSync(tmpRoot, { recursive: true, force: true }); } catch { /* noop */ } },
};
}
/**
* Spawn the supervisor runner as a subprocess. Returns a handle with the
* child, a promise resolving to exit code + signal, and a kill helper.
*/
function spawnSupervisor(h: IntegrationHarness, overrides: Record<string, string> = {}) {
const env: Record<string, string> = {
...(process.env as Record<string, string>),
SUP_PID_FILE: h.pidFile,
SUP_CLI_PATH: h.workerScript,
SUP_AUDIT_DIR: h.auditDir,
SUP_BACKOFF_FLOOR_MS: '5',
SUP_MAX_CRASHES: '3',
SUP_HEALTH_INTERVAL_MS: '999999', // effectively off
...overrides,
};
const child = spawn('bun', [join(import.meta.dir, 'fixtures/supervisor-runner.ts')], {
env,
stdio: ['ignore', 'pipe', 'pipe'],
});
let stdout = '';
let stderr = '';
child.stdout?.on('data', (d) => { stdout += d.toString(); });
child.stderr?.on('data', (d) => { stderr += d.toString(); });
const exited = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve) => {
child.on('exit', (code, signal) => resolve({ code, signal }));
});
return {
child,
exited,
getStdout: () => stdout,
getStderr: () => stderr,
};
}
/** Read the audit JSONL for the current week. */
function readAudit(auditDir: string) {
const origEnv = process.env.GBRAIN_AUDIT_DIR;
process.env.GBRAIN_AUDIT_DIR = auditDir;
try {
return readSupervisorEvents();
} finally {
if (origEnv === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = origEnv;
}
}
/** Poll until predicate returns true or deadline elapses. */
async function waitFor(pred: () => boolean, timeoutMs: number, tickMs = 20): Promise<boolean> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (pred()) return true;
await new Promise(r => setTimeout(r, tickMs));
}
return pred();
}
describe('MinionSupervisor', () => {
describe('calculateBackoffMs', () => {
it('returns ~1s for first crash', () => {
const backoff = calculateBackoffMs(0);
expect(backoff).toBeGreaterThanOrEqual(1000);
expect(backoff).toBeLessThan(1200); // 1000 + 10% jitter max
});
it('doubles with each crash', () => {
const b0 = calculateBackoffMs(0);
const b1 = calculateBackoffMs(1);
const b2 = calculateBackoffMs(2);
// Approximate: b1 should be ~2x b0, b2 ~2x b1 (within jitter)
expect(b1).toBeGreaterThan(1800);
expect(b2).toBeGreaterThan(3600);
});
it('caps at 60s', () => {
const backoff = calculateBackoffMs(20); // 2^20 * 1000 would be huge
expect(backoff).toBeLessThanOrEqual(66_000); // 60s + 10% jitter
});
it('includes jitter (not perfectly deterministic)', () => {
const values = new Set<number>();
for (let i = 0; i < 10; i++) {
values.add(Math.round(calculateBackoffMs(3)));
}
// With 10% jitter, we should get some variation
expect(values.size).toBeGreaterThan(1);
});
});
describe('PID file management', () => {
it('detects stale PID files', () => {
// Write a PID file with a non-existent PID
writeFileSync(TEST_PID_FILE, '999999999');
expect(existsSync(TEST_PID_FILE)).toBe(true);
// A real supervisor would detect this as stale and overwrite
const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10);
let isAlive = false;
try {
process.kill(existingPid, 0);
isAlive = true;
} catch {
isAlive = false;
}
expect(isAlive).toBe(false);
});
it('detects live PID files (current process)', () => {
// Write our own PID
writeFileSync(TEST_PID_FILE, String(process.pid));
const existingPid = parseInt(readFileSync(TEST_PID_FILE, 'utf8').trim(), 10);
let isAlive = false;
try {
process.kill(existingPid, 0);
isAlive = true;
} catch {
isAlive = false;
}
expect(isAlive).toBe(true);
expect(existingPid).toBe(process.pid);
});
});
describe('crash count tracking', () => {
it('backoff escalates with crash count', () => {
const backoffs = [];
for (let i = 0; i < 7; i++) {
backoffs.push(calculateBackoffMs(i));
}
// Each should be roughly 2x the previous (within jitter)
for (let i = 1; i < 6; i++) {
// The base doubles, so even with jitter the next should be > 1.5x previous
expect(backoffs[i]).toBeGreaterThan(backoffs[i - 1] * 1.5);
}
});
});
// --------------------------------------------------------------
// Integration tests: real spawn(), real signals, real audit file.
// Each test uses a unique tmpdir harness so they can run in parallel
// without colliding. `_backoffFloorMs: 5` (set via SUP_BACKOFF_FLOOR_MS)
// keeps the whole suite under a few seconds.
// --------------------------------------------------------------
describe('integration: crash → restart → max-crashes lifecycle', () => {
it('respawns the worker after a crash and eventually exits with max-crashes code=1', async () => {
// Worker always exits with code 1; supervisor should respawn it 3 times,
// hit max-crashes, then exit via shutdown() with code 1.
const h = makeHarness('max-crashes', 'exit 1');
try {
const sup = spawnSupervisor(h, { SUP_MAX_CRASHES: '3' });
const { code } = await sup.exited;
expect(code).toBe(1);
// PID file cleaned up on exit (synchronous process.on('exit') handler).
expect(existsSync(h.pidFile)).toBe(false);
// Audit file should contain started + 3x worker_spawned/worker_exited +
// max_crashes_exceeded + shutting_down + stopped.
const events = readAudit(h.auditDir);
const eventTypes = events.map(e => e.event);
expect(eventTypes).toContain('started');
expect(eventTypes.filter(t => t === 'worker_spawned').length).toBeGreaterThanOrEqual(3);
expect(eventTypes.filter(t => t === 'worker_exited').length).toBeGreaterThanOrEqual(3);
expect(eventTypes).toContain('max_crashes_exceeded');
expect(eventTypes).toContain('shutting_down');
expect(eventTypes).toContain('stopped');
// The stopped event should carry exit_code=1 and reason=max_crashes.
const stoppedEvt = events.filter(e => e.event === 'stopped').pop();
expect((stoppedEvt as Record<string, unknown>).exit_code).toBe(1);
expect((stoppedEvt as Record<string, unknown>).reason).toBe('max_crashes');
} finally {
h.cleanup();
}
}, 15_000);
});
describe('integration: graceful SIGTERM during backoff', () => {
it('receives SIGTERM while sleeping between crashes and exits 0 cleanly', async () => {
// Worker always exits with code 1; supervisor has a high max-crashes
// and a long-enough backoff floor that we can reliably catch it mid-sleep.
const h = makeHarness('sigterm-backoff', 'exit 1');
try {
const sup = spawnSupervisor(h, {
SUP_MAX_CRASHES: '100',
SUP_BACKOFF_FLOOR_MS: '800', // 800ms between restarts — enough to catch
});
// Wait until the supervisor has written the PID file AND survived at
// least one worker_exited (so it's definitely in the backoff sleep).
const ready = await waitFor(() => {
if (!existsSync(h.pidFile)) return false;
const events = readAudit(h.auditDir);
return events.some(e => e.event === 'worker_exited');
}, 3000);
expect(ready).toBe(true);
// Now SIGTERM the supervisor. It must exit cleanly within 200ms
// (short-circuits the 800ms backoff sleep via the stopping flag).
const sigSentAt = Date.now();
sup.child.kill('SIGTERM');
const { code, signal } = await sup.exited;
const elapsed = Date.now() - sigSentAt;
// Exit code 0 = clean; signal=null means we exited via process.exit, not got killed.
expect(code).toBe(0);
expect(signal).toBe(null);
// Graceful, not hung: exit within 5s (process.exit() through shutdown()
// should be near-instant; generous bound to tolerate CI slowness).
expect(elapsed).toBeLessThan(5000);
const events = readAudit(h.auditDir);
const eventTypes = events.map(e => e.event);
expect(eventTypes).toContain('shutting_down');
expect(eventTypes).toContain('stopped');
const shuttingEvt = events.filter(e => e.event === 'shutting_down').pop();
expect((shuttingEvt as Record<string, unknown>).reason).toBe('SIGTERM');
// PID file cleaned up.
expect(existsSync(h.pidFile)).toBe(false);
} finally {
h.cleanup();
}
}, 20_000);
});
describe('integration: env-var inheritance regression (codex #9 / eng #8)', () => {
it('strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false, even if parent has it set', async () => {
const outFile = join(tmpdir(), `gbrain-sup-env-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-strip-outfile', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
try {
const sup = spawnSupervisor(h, {
OUT_FILE: outFile,
GBRAIN_ALLOW_SHELL_JOBS: '1', // parent has it
SUP_ALLOW_SHELL_JOBS: '0', // supervisor says NO
SUP_MAX_CRASHES: '1',
});
await sup.exited;
// Worker should have written "UNSET" (parent env var stripped from child).
expect(existsSync(outFile)).toBe(true);
const childSawEnv = readFileSync(outFile, 'utf8').trim();
expect(childSawEnv).toBe('UNSET');
} finally {
try { unlinkSync(outFile); } catch { /* noop */ }
h.cleanup();
}
}, 15_000);
it('DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs is true', async () => {
const outFile = join(tmpdir(), `gbrain-sup-env-ok-${process.pid}-${Date.now()}.txt`);
try { unlinkSync(outFile); } catch { /* may not exist */ }
const h = makeHarness('env-pass-on-opt-in', `printf '%s\\n' "\${GBRAIN_ALLOW_SHELL_JOBS-UNSET}" > "$OUT_FILE" ; exit 0`);
try {
const sup = spawnSupervisor(h, {
OUT_FILE: outFile,
SUP_ALLOW_SHELL_JOBS: '1',
SUP_MAX_CRASHES: '1',
});
await sup.exited;
expect(existsSync(outFile)).toBe(true);
expect(readFileSync(outFile, 'utf8').trim()).toBe('1');
} finally {
try { unlinkSync(outFile); } catch { /* noop */ }
h.cleanup();
}
}, 15_000);
});
describe('integration: audit file rotation + helper', () => {
it('computeSupervisorAuditFilename returns supervisor-YYYY-Www.jsonl format', () => {
const jan15_2026 = new Date(Date.UTC(2026, 0, 15)); // Thu
expect(computeSupervisorAuditFilename(jan15_2026)).toMatch(/^supervisor-2026-W\d\d\.jsonl$/);
});
it('year-boundary ISO week: 2027-01-01 reports as 2026-W53', () => {
const jan1_2027 = new Date(Date.UTC(2027, 0, 1));
// ISO week: 2027-01-01 is Friday of W53 of 2026
expect(computeSupervisorAuditFilename(jan1_2027)).toBe('supervisor-2026-W53.jsonl');
});
});
});