---
feature_pitch:
  headline: Bare workers now self-monitor and fail-stop into your PM's restart loop
  body: |
    Bare `gbrain jobs work` now ships with the same health protection the
    supervisor already had: DB liveness probes (with per-probe timeout so a
    hung connection can't wedge the monitor), stall detection filtered by
    registered handler names, and an RSS watchdog default of 2048 MB.

    When the worker detects it's wedged (stuck pgbouncer connection, hung
    event loop, stalled job claim), it emits `'unhealthy'` and the CLI calls
    `process.exit(1)`. This is **fail-stop**: it requires an external process
    manager (systemd, Docker `restart: always`, launchd `KeepAlive`, cron
    watchdog) to bring the worker back. Without one, the process exits and
    stays dead — that's a regression from pre-v0.22.14 self-healing.

    Pre-v0.22.14 behavior: bare workers had ZERO health monitoring. A wedged
    worker stayed alive doing nothing while jobs piled up in `waiting` and
    your PM's `pgrep` check happily reported green.

    If you're using `gbrain jobs supervisor`, you're already protected — the
    supervisor handles spawn-on-crash itself. The fail-stop concern only
    applies to direct `gbrain jobs work` invocations.
---

# v0.22.14 — Bare-worker self-health-monitoring

## ⚠️ Pre-flight: confirm you have a process supervisor

If you run `gbrain jobs work` directly (NOT under `gbrain jobs supervisor`),
verify your process manager is configured to restart the worker on exit
BEFORE upgrading:

| Manager | What to check |
|---|---|
| systemd | `Restart=always` (or `Restart=on-failure`) in the `.service` unit |
| Docker | `restart: always` / `restart: unless-stopped` in compose, OR `--restart` flag |
| launchd (macOS) | `<key>KeepAlive</key><true/>` in the plist |
| cron watchdog | Cron entry that re-spawns when `pgrep -f "gbrain jobs work"` is empty |
| supervisord | `autorestart=true` |

**If your bare worker has no restart loop, the v0.22.14 fail-stop behavior
will leave you with a dead worker after the first DB blip.** Either add a
restart policy OR switch to `gbrain jobs supervisor` (which spawns its own
child + restarts on crash internally).

## What ships

- DB liveness probes inside `gbrain jobs work` (60s interval, 3 strikes → exit)
- Stall detection (5min warn / 10min exit when waiting jobs accumulate but
  in-flight is empty)
- `--max-rss` defaults to 2048 MB for bare workers (matches supervisor default;
  was 0 = disabled)
- New `MinionWorkerOpts.{healthCheckInterval, stallWarnAfterMs,
  stallExitAfterMs, dbFailExitAfter, dbProbeTimeoutMs}` for tuning (5 fields)
- `MinionWorker` now extends `EventEmitter`; emits `'unhealthy'` event with
  a structured reason payload. **No-listener fallback**: if the caller does
  not subscribe to `'unhealthy'`, the worker calls `process.exit(1)` itself
  to preserve the pre-refactor fail-stop behavior. The CLI subscribes; direct
  API consumers without a listener inherit the fail-stop default. Inline
  paths (`jobs submit --follow`, `jobs smoke`) explicitly pass
  `healthCheckInterval: 0` to disable the timer entirely so the fallback
  cannot trip on a one-shot run.
- `GBRAIN_SUPERVISED=1` env var (set by supervisor on its child) causes the
  child worker to skip its own health timer (no double-monitoring)
- Constructor validation: throws if `stallExitAfterMs <= stallWarnAfterMs`
  (the contract requires strict greater-than)

## ⚠️ Breaking change you may need to act on

**`gbrain jobs work` now defaults `--max-rss 2048` (MB).** Pre-v0.22.14, bare
workers had no RSS limit. If you ran bare workers with intentionally large
embed/import working sets, your worker may now get killed by the watchdog after
upgrade.

**You need to act if any of the following is true:**

- You run `gbrain jobs work` directly under a process manager (systemd, Docker,
  cron, supervisord, your own bash wrapper)
- Your worker handles large embed batches, multi-GB import jobs, or
  memory-intensive subagent tasks that legitimately exceed 2 GB RSS
- You rely on bare-worker resident memory exceeding 2 GB

**You do NOT need to act if:**

- You run `gbrain jobs supervisor` (the supervisor already defaulted to
  `--max-rss 2048` and has reconnect-then-fail logic on top)
- Your typical worker stays well under 2 GB RSS

## What the agent should do post-upgrade

### 1. Run the orchestrator (mechanical migration)

```bash
gbrain apply-migrations --yes
```

This is a no-op for v0.22.14 — there's no schema migration. The mechanical work
is in the binary; the user-side decision is the breaking-change handling below.

### 2. Detect bare-worker deployments

If the user runs OpenClaw, Hermes, or any agent fork that uses `gbrain jobs
work` directly (NOT under `gbrain jobs supervisor`), they need to know. Check:

```bash
# Look for bare worker invocations in cron, systemd, Docker, launchctl, etc.
grep -r "gbrain jobs work" ~/.config/systemd/ /etc/systemd/ 2>/dev/null | head
crontab -l 2>/dev/null | grep "gbrain jobs work" | head
launchctl list 2>/dev/null | grep gbrain | head
ps -ef | grep "gbrain jobs work" | grep -v grep | grep -v supervisor
```

If you find bare-worker invocations, surface them to the user.

### 3. Ask about RSS profile

Use AskUserQuestion to confirm whether the user's bare workers need >2 GB RSS:

> "v0.22.14 changes `gbrain jobs work` to default `--max-rss 2048` (MB).
> Pre-v0.22.14 bare workers had no limit. If your worker handles large embed
> batches or multi-GB imports, the watchdog may now kill it. Do you want
> to keep the new 2 GB default, raise the limit, or opt out entirely?"

Options:
- **A) Keep 2 GB default (recommended for most)** — protects against memory
  leaks; restarts on overflow; matches supervisor behavior.
- **B) Raise to N GB (specify N)** — pass `--max-rss <N*1024>` to the worker
  invocation.
- **C) Opt out** — pass `--max-rss 0`.

### 4. Apply the user's choice

For each bare-worker invocation, edit the unit/cron/launchctl/script to add
the chosen `--max-rss` flag.

**systemd (~/.config/systemd/user/gbrain-worker.service):**

```ini
ExecStart=/usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
# Or to opt out: --max-rss 0
```

Then `systemctl --user daemon-reload && systemctl --user restart gbrain-worker`.

**cron (`crontab -e`):**

```cron
@reboot /usr/local/bin/gbrain jobs work --queue default --concurrency 3 --max-rss 4096
```

**Docker compose:**

```yaml
command: ["gbrain", "jobs", "work", "--queue", "default", "--concurrency", "3", "--max-rss", "4096"]
```

**launchctl (~/Library/LaunchAgents/com.user.gbrain-worker.plist):**

```xml
<key>ProgramArguments</key>
<array>
  <string>/usr/local/bin/gbrain</string>
  <string>jobs</string>
  <string>work</string>
  <string>--max-rss</string>
  <string>4096</string>
</array>
```

Then `launchctl unload ... && launchctl load ...`.

### 5. (Optional) Tune health-check thresholds

The new opts default to sensible values (60s probe interval, 5min warn / 10min
exit, 3 DB failures). If you have specific SLAs, you can pass `--health-interval
<ms>` to adjust the probe cadence. Stall thresholds are not yet CLI-exposed
(only the API; CLI flags coming in a follow-up).

To disable self-monitoring entirely (e.g. you have your own external health
checker):

```bash
gbrain jobs work --health-interval 0 --max-rss 0
```

### 6. Verify

```bash
gbrain jobs stats              # queue should be flowing normally
gbrain doctor --json | jq '.'  # no critical warnings
ps -o rss= -p $(pgrep -f "gbrain jobs work") | awk '{print $1/1024 " MB"}'
```

Worker startup log line should now show health-check status:

```
Minion worker started (queue: default, concurrency: 3, watchdog: 2048MB, health-check: 60s)
```

If running under supervisor, you'll see the watchdog but NOT the `health-check:
60s` segment (because `GBRAIN_SUPERVISED=1` skips the child's self-monitor).

### 7. If anything fails

Open an issue at https://github.com/garrytan/gbrain/issues with:
- Output of `gbrain doctor`
- Contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- Your bare-worker invocation (systemd unit / cron line / Dockerfile snippet)
- Which step broke
