5fd9cd2644
* feat(minions): add protected-name constant + ctx.shutdownSignal
Introduce PROTECTED_JOB_NAMES ('shell') in a side-effect-free core module
so queue.ts can check it without importing from handlers/. MinionJobContext
gains shutdownSignal (distinct from signal) — handlers that need to run
SIGTERM-triggered cleanup subscribe to both; most handlers ignore shutdown
and run through the worker's 30s cleanup race to natural completion.
* fix(minions): MinionQueue.add gains trusted 4th arg + trim-normalized guard
Adds allowProtectedSubmit opt-in as a separate 4th parameter (NOT folded into
opts) so callers spreading user-provided opts ({...userOpts}) can't accidentally
carry the trust flag. PROTECTED_JOB_NAMES check runs on the trimmed name BEFORE
insert, closing the queue.add(' shell ', ...) whitespace bypass that would have
evaded a has(name) check.
* fix(minions): worker calls failJob on abort + wires ctx.shutdownSignal
Pre-v0.13.0 worker returned silently when ctx.signal.aborted fired, leaving
jobs in 'active' until stall sweep. Handlers using cooperative cancel had
no deterministic status flip — timeout/cancel/lock-loss all looked the same
from downstream callers (gbrain jobs get, --follow loops).
Fix: derive abort reason from abort.signal.reason ('timeout' | 'cancel' |
'lock-lost' | 'shutdown') and call failJob with 'aborted: <reason>' text.
failJob is idempotent via token+status match, so no-op when another path
already flipped status (handleTimeouts, cancelJob, stall).
Also: new shutdownAbort (instance-level AbortController) fires on process
SIGTERM/SIGINT and propagates to every handler's ctx.shutdownSignal.
Shell handler listens to both signals and runs SIGTERM→5s→SIGKILL on its
child on either; other handlers only listen to ctx.signal so deploy
restarts don't cancel them mid-flight.
* feat(minions): add shell job handler + submission audit log
New 'shell' job type spawns arbitrary commands under the Minions worker.
Deterministic cron scripts (API fetch, token refresh, scrape+write) can
move off the LLM gateway — zero Opus tokens per fire.
Handler contract:
- cmd or argv (exactly one required). cmd spawns via /bin/sh -c (absolute
path, not 'sh', to block PATH-override shell substitution). argv spawns
direct with no shell.
- cwd required, must be absolute. Operator-trust boundary.
- env defaults to SHELL_ENV_ALLOWLIST ({PATH, HOME, USER, LANG, TZ,
NODE_ENV}) picked from process.env, with caller overrides merged on top.
Prevents accidental $OPENAI_API_KEY interpolation into scripts.
- stdout/stderr retained as UTF-8-safe tails (64KB/16KB) via
string_decoder.StringDecoder. Prepends [truncated N bytes] marker.
- Abort (either ctx.signal or ctx.shutdownSignal) fires SIGTERM → 5s grace
→ SIGKILL on child. Timer NOT .unref'd so worker's 30s race waits for
the child to actually die.
shell-audit.ts writes a JSONL line per submission to
~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl (ISO-week rotated, override via
GBRAIN_AUDIT_DIR). argv logged as JSON array (not space-joined, which would
flatten args with spaces). Never logs env values. Best-effort writes:
failures log to stderr but don't block submission.
* feat(jobs): submit_job MCP guard + CLI --timeout-ms + starvation warning
submit_job operation gains timeout_ms param (was missing — couldn't plumb
the existing MinionJobInput field through from either CLI or MCP). When
ctx.remote=true and name is in PROTECTED_JOB_NAMES, throws
OperationError('permission_denied'). Combined with the queue.add trusted
guard, MCP callers can never submit shell jobs even if the env flag is on.
CLI submit: new --timeout-ms N flag. Passes {allowProtectedSubmit:true}
as the 4th arg to queue.add only when the submitted name is protected
(not blanket-set for every job). Prints a starvation-warning block to
stderr when a shell job is submitted without --follow, pointing at both
--follow and 'gbrain jobs work' remediation. Fires for every shell submit
regardless of the submitter's env — the submitter env is a weak proxy for
the worker env.
Worker handler registration: conditional on GBRAIN_ALLOW_SHELL_JOBS=1.
Default: off. 'gbrain jobs submit --help' now lists handler types with a
pointer to docs/guides/minions-shell-jobs.md for shell.
* test(minions): 40 unit + 4 E2E cases for shell handler
Unit (test/minions-shell.test.ts):
- Protected names: trim-normalized, case-sensitive, whitespace bypass defense
- MinionQueue.add: trusted opt-in, whitespace bypass, non-protected untouched
- Handler validation: cmd|argv exclusive, cwd required/absolute, env strings
- Spawn: cmd/argv happy paths, non-zero exit, ENOENT, result shape
- Env allowlist: leaked-secret blocked, PATH inherited, caller override
- Abort: ctx.signal, ctx.shutdownSignal, pre-aborted signal
- Audit: ISO-week year boundary (2027-01-01 → W53 2026), mid-year W52/W53,
GBRAIN_AUDIT_DIR override, argv as JSON array, env never logged, EACCES
non-blocking
- Output truncation: 100KB → last 64KB with [truncated N bytes] marker
E2E (test/e2e/minions-shell.test.ts):
- Full lifecycle: submit → worker claim → spawn → complete
- MinionQueue.add without trusted arg throws (including whitespace bypass)
- submit_job with ctx.remote=true rejects shell (MCP guard)
- submit_job with ctx.remote=false allows shell (CLI path)
* chore: bump version and changelog (v0.13.0)
Move gateway crons to Minions. Zero LLM tokens per cron fire.
Worker abort path finally marks aborted jobs dead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: reframe v0.13.0 copy for OpenClaw operators (not Wintermute-specific)
gbrain is an open-source product for any OpenClaw/Hermes operator, not
Garry's personal Wintermute deployment. Rewords the v0.13.0 CHANGELOG
entry, the minions-shell-jobs guide, and the deferred TODOS entries to
speak to "your OpenClaw" / "OpenClaw operators" instead.
Replaces /data/wintermute cwd examples with the canonical
/data/.openclaw/workspace path. Pre-existing Wintermute references in
older CHANGELOG entries (v0.11/v0.10.3) left unchanged.
* feat(migrations): add v0.13.0 adoption playbook for shell jobs
Adding the migration file the CEO review originally scoped out. Without
it, operators upgrade to v0.13.0 and the capability ships but adoption
doesn't happen — the 60% gateway CPU reduction only lands if someone
actually rewrites their crontab.
skills/migrations/v0.13.0.md is the instruction manual the host agent
reads on gbrain upgrade:
- Enable worker: GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work (Postgres)
or per-tick --follow (PGLite)
- Audit cron manifest: classify LLM-requiring vs deterministic
- Propose per-cron rewrites with diffs, approved one at a time
- Env allowlist guidance for scripts that need API keys
- Verification playbook: run one fire, compare pre/post, only then
approve the next batch
- Starvation sanity-check runbook item
Iron rules: never auto-rewrite the operator's crontab (host-specific
code per CLAUDE.md). LLM-requiring crons stay on the gateway. Ambiguous
cases ask the operator.
No mechanical orchestrator ships with this migration — every rewrite
is operator judgment. A future gbrain crontab-to-minions helper is
tracked in TODOS.md as P1.
* docs: sync UPGRADING + SKILLPACK with v0.13.0 shell jobs
UPGRADING_DOWNSTREAM_AGENTS.md: append v0.13.0 section per the file's
convention (each release appends). No skill edits required, feature is
off-by-default, optional adoption via skills/migrations/v0.13.0.md.
Lists typical LLM-vs-deterministic classifications so operators know
which of their crons are candidates for migration.
GBRAIN_SKILLPACK.md: add shell-jobs guide row to the cron/Minions guide
table so it's discoverable alongside existing Cron via Minions, Plugin
Handlers, and Minions fix guides.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7.8 KiB
7.8 KiB
GBrain Skillpack: Reference Architecture for AI Agents
This is a reference architecture for how a production AI agent uses gbrain as its knowledge backbone. Based on patterns from a real deployment with 14,700+ brain files, 40+ skills, and 20+ cron jobs running continuously.
The memex vision, realized. Vannevar Bush imagined a device where an individual stores everything, mechanized so it may be consulted with exceeding speed. GBrain is that device, except the memex builds itself. The agent detects entities, enriches pages, creates cross-references, and maintains compiled truth automatically.
Each section below is a standalone guide. Click through to the full content.
Core Patterns
The foundational read-write loop and data model.
| Guide | What It Covers |
|---|---|
| The Brain-Agent Loop | The read-write cycle that makes the brain compound over time |
| Entity Detection | Run it on every message. Capture original thinking + entity mentions |
| The Originals Folder | Capturing WHAT YOU THINK, not just what you found |
| Brain-First Lookup | Check the brain before calling any external API |
| Compiled Truth + Timeline | Above the line: current synthesis. Below: append-only evidence |
| Source Attribution | Every fact needs a citation. Format and hierarchy |
Data Pipelines
Getting data in and keeping it current.
| Guide | What It Covers |
|---|---|
| Enrichment Pipeline | 7-step protocol, tier system (Tier 1/2/3 by importance) |
| Meeting Ingestion | Always pull complete transcript, propagate to all entity pages |
| Content & Media Ingestion | YouTube, social media bundles, PDFs/documents |
| Diligence Ingestion | Data room materials: pitch decks, financial models, cap tables |
| Deterministic Collectors | Code for data, LLMs for judgment. The collector pattern |
| Idea Capture & Originals | Depth test, originality distribution, deep cross-linking |
| Getting Data In | Integration recipes: voice, email, X, calendar |
Operations
Running a production brain.
| Guide | What It Covers |
|---|---|
| Reference Cron Schedule | 20+ recurring jobs, quiet hours, dream cycle |
| Cron via Minions | Why scheduled work runs as Minion jobs, not agentTurn. Auto-applied by v0.11.0 migration for built-in handlers; host-specific handlers use the plugin contract below. |
| Plugin Handlers | Registering host-specific Minion handlers via code (no data-file exec surface). |
| Minions fix | Repairing a half-migrated v0.11.0 install. |
| Shell jobs (v0.14.0+) | Move deterministic crons (API fetch, token refresh, scrape+write) off the LLM gateway. Zero tokens per fire, ~60% gateway headroom. Follow skills/migrations/v0.14.0.md for the adoption playbook. |
| Quiet Hours & Timezone | Hold notifications during sleep, timezone-aware delivery |
| Executive Assistant Pattern | Email triage, meeting prep, scheduling |
| Operational Disciplines | Signal detection, brain-first, sync-after-write, heartbeat, dream cycle |
| Skill Development Cycle | 5-step cycle: concept, prototype, evaluate, codify, cron |
Subagent routing (v0.11.0+): agents that dispatch background work should route through
skills/conventions/subagent-routing.md — it reads ~/.gbrain/preferences.json#minion_mode
and branches between native subagents and Minion jobs. The v0.11.0 migration auto-injects
a marker into AGENTS.md pointing at this convention.
Cron routing (v0.11.0+): scheduled work goes through Minions, not OpenClaw's agentTurn.
See skills/conventions/cron-via-minions.md for the rewrite pattern. The v0.11.0 migration
auto-rewrites entries whose handler is a gbrain builtin; host-specific handlers (e.g.
ea-inbox-sweep) need a code-level registration per docs/guides/plugin-handlers.md.
Architecture
How to structure your system.
| Guide | What It Covers |
|---|---|
| Two-Repo Architecture | Agent repo vs brain repo, boundary rules, decision tree |
| Sub-Agent Model Routing | Which model for which task, signal detector pattern, cost optimization |
| The Three Search Modes | Keyword, hybrid, direct. When to use each |
| Brain vs Agent Memory | 3 layers: GBrain (world knowledge), agent memory, session |
Integrations
Wiring up your life.
| Guide | What It Covers |
|---|---|
| Credential Gateway | ClawVisor / Hermes for Gmail, Calendar, Contacts |
| Meeting & Call Webhooks | Circleback transcripts + Quo/OpenPhone SMS/calls |
| Voice-to-Brain | Phone calls + WebRTC browser calls create brain pages. 25 production patterns: identity separation, bid system, conversation timing, proactive advisor, prompt compression, caller routing, dynamic VAD, real-time logging, belt-and-suspenders post-call |
| Email-to-Brain | Gmail messages flow into entity pages via deterministic collector |
| X-to-Brain | Twitter monitoring with deletion detection + engagement velocity |
| Calendar-to-Brain | Google Calendar events become searchable daily brain pages |
| Meeting Sync | Circleback transcripts auto-import with attendee propagation |
Administration
Keeping it running and up to date.
| Guide | What It Covers |
|---|---|
| Upgrades & Auto-Update | check-update, agent notifications, migration files |
| Live Sync | Keep the index current: cron, --watch, webhook approaches |
Appendix: GBrain CLI Quick Reference
| Command | Purpose |
|---|---|
gbrain search "term" |
Keyword search across all brain pages |
gbrain query "question" |
Hybrid search (vector + keyword + RRF) |
gbrain get <slug> |
Read a specific brain page by slug |
gbrain sync |
Sync local markdown repo to gbrain index |
gbrain import <path> |
Import files into the brain |
gbrain embed --stale |
Re-embed pages with stale or missing embeddings |
gbrain integrations |
Manage integration recipes (senses + reflexes) |
gbrain stats |
Show brain statistics (page count, last sync, etc.) |
gbrain doctor |
Diagnose brain health issues |
gbrain check-update |
Check for new versions and integration recipes |
Run gbrain --help for the full command reference.
Architecture & Philosophy
- Infrastructure Layer — Import pipeline, chunking, embedding, search
- Thin Harness, Fat Skills — Architecture philosophy
- Markdown Skills as Recipes — Why markdown is code and your agent is a package manager
- Homebrew for Personal AI — The 10-star vision
- Recommended Schema — Directory structure for your brain repo
- Verification Runbook — End-to-end installation verification