gbrain

Author	SHA1	Message	Date
Garry Tan	7ea92d602c	v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241 ) * feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean, never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's --ff-only contract is unchanged. * feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules) hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed brain-commit-push.sh (one shared push-retry template), repo-scoped credential with existing-helper reuse, push-probe verify, active-resolver-file rules with taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction via redactSecretsInText. * feat(sources): harden/pull/unharden commands + auto-harden on add --url sources harden/pull/unharden subcommands; --pat-file/--no-harden on add; auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite). * test(durability): unit + integration coverage for brain-repo durability git helpers, core harden/unharden, hook+helper E2E (real background push), cron generators. 41 tests across 4 files. * chore: bump version and changelog (v0.42.48.0) Brain-repo git durability: auto-harden a brain's working tree (local auto-push hook, committed commit-push helper, always-on agent rules, DB-free pull cron, repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sources): route harden exit code through setCliExitVerdict A raw process.exitCode write is zeroed by the owned-verdict flush-exit (#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports needs-attention to cron/automation. * docs: document brain-repo durability (KEY_FILES + multi-source guide) KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe, detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md: add a Durability (auto-harden) how-to covering sources harden/pull/unharden, --pat-file, the guarantees, and the security posture. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 06:41:37 -07:00
Garry Tan	a81f7e05e8	v0.42.43.0 feat(context): push-based context (#2095 ) + teardown-exit hardening (#2084 ) (#2175 ) * fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084) Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the bound raced past — keep Bun's event loop alive, so every `gbrain query` paid a flat 10s tax exiting via the hard-deadline force-exit banner. Three changes, one contract: - flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the command is not a daemon, exit deliberately — after stdout AND stderr drain (writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the regression class; pinned by a 256KB real-pipe subprocess test. - drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites (op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation x3, ze-switch, dream, read-only timeout path). Drains the background-work registry, then disconnect (best-effort), bounded by the 10s unref'd hard-deadline — which is now armed around the TEARDOWN window only, not before the op handler (the old placement would have force-killed any op slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites previously skipped the drain entirely and had no hang timer at all. - Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing paths (status, friction, claw-test, smoke-test, eval cross-modal / takes-quality replay / conversation-parser / whoknows-thin, status-thin) become process.exitCode + return so they flow through the drains and the flush-exit. Pre-engine usage/parse/refusal exits stay as-is. BrainRegistry.disconnectAll deliberately unchanged: zero production callers in src/, per-engine disconnects already bounded, and the kernel reclaims sockets on exit (src/core/timeout.ts doctrine). DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084) Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs verified only against one production deployment — CI had no transaction-mode pooler and could never see the class. Now it can: - docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting postgres-1, mirroring the production split-pool topology (direct :5432 + pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext password in the userlist) + IGNORE_STARTUP_PARAMETERS for the statement_timeout/idle_in_transaction_session_timeout startup params gbrain's client sets (the Supabase pooler whitelists the same). - test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs), then spawns the real CLI against the POOLED url and asserts: exit 0, stdout intact (the #1959 truncation class), and NO "did not return within 10000ms — force-exiting" banner (pre-#2084 it printed on 100% of query-shaped ops on this topology). Class bound, not exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL. - scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL + GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases. Verified live: both tests green against pgbouncer 1.25.2 in transaction mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095) One row per page the brain volunteers (op / reflex / watch channels). "Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at (the existing bumpLastRetrievedAt write-back is the open/cite signal), so there is no second tracking path. session_id/turn are nullable caller-supplied attribution; rationale is a deterministic template string, never raw conversation text. - Migration v116 (idempotent) + mirrors in src/schema.sql + src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also folds in pre-existing comment-only drift from the v114 links edits). - src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE multi-row parameterized INSERT — never per-row awaited round-trips) + purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains). - Dream cycle purge phase prunes stale events alongside op_checkpoints / brainstorm checkpoints / batch-retry audit files. - RLS on Postgres comes from the v35 auto_rls_on_create_table event trigger (the same mechanism that covered v110 page_aliases and v115 op_checkpoint_paths); the volunteer Postgres e2e pins it. - No ::jsonb anywhere; no bootstrap probe needed (nothing references the table pre-creation; writers guard with try/catch). Tests: v116 shape + columns + indexes + live insert/purge round-trip on PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095) - entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing per-turn extractor across the last N turns (oldest→newest), merges by the normalizeAlias form with occurrence/newest-turn/user-mention metadata, and orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES cap drops stale assistant chatter, not the entity the user just named. Closes the filed assistant-introduced-entities recall TODO; true pronoun coreference (never-named antecedents) stays out of scope. - retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence + matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6) lives next to the arm definitions so identity and score can't drift. Arm-2 provenance is classified in JS (codex D8 — the combined OR can't report which predicate matched). Federated sourceIds[] scope (alias arm loops per source; arm 2 uses source_id = ANY — no engine-interface change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for windowing): the legacy title-whole-word rule would suppress every entity merely MENTIONED in a prior window turn, breaking the feature by construction — slugs only enter context when a pointer/page was actually surfaced. Default stays 'slug-and-title' for the window=1 legacy path. - volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF, unprefixed → one user turn), volunteerContext (zero-LLM: extract → resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate → cap 3/5; deterministic rationale strings, never raw conversation text), and volunteerUsageStats (per-arm/channel precision from the last_retrieved_at join, labeled approximate — 5-min throttle false negatives, unrelated-read false positives; codex D9). Tests: 35 green across volunteer-context (window parsing, pronoun follow-up via assistant-introduced entity, confidence gating, slug-only suppression, takes-fence privacy, multi-source scope, caps, stats join math) + retrieval-reflex back-compat + resolve-ipc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095) New read-scope op on the contract surface (CLI `gbrain volunteer-context` with stdin → window, MCP tool for free): takes a rolling conversation window, returns confidence-gated page pointers with rationales + synopses. `window` is optional-unless-stats (validated in the handler, codex D9); `stats: true` returns the volunteered-vs-used precision summary, labeled APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both bias the join). Source scope threads through sourceScopeOpts — federated grants narrow the volunteer to the granted sources. Event logging is fire-and-forget through a new `volunteer-events` background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked dangling promise set + bounded drain + snapshot-drop on timeout so a long-lived process never accumulates ghosts). ONE batched INSERT per call, drained on every exit path by the commit-1 drain hoist; failure never fails the op (pinned by an injected failing-engine test). cli formatResult renders both shapes (pointer lines with confidence/arm/ rationale; the stats summary with per-arm precision). Tests: op contract surface, window-required validation, sink round-trip with session_id/turn attribution, failing-engine fail-open, federated grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e proving the op + sink + stats join AND that context_volunteer_events has RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(context): reflex consumes the rolling window + ambient-channel logging (#2095) The default-on retrieval reflex now extracts entities from the last N turns (retrieval_reflex_window_turns, default 4; env GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy current-turn-only behavior exactly). assemble() passes the recent user/assistant turns (hard cap 12); the reflex slices to the configured window. Assistant-introduced entities and "what did she invest in?" follow-ups whose antecedent was NAMED in the window now surface pointers — the issue's "zero agent-initiated queries" success criterion on the ambient path. Under windowing, suppression switches to slug-only (codex D7): the legacy title-whole-word rule would suppress every entity merely MENTIONED in a prior window turn, breaking the feature by construction. Slugs only enter prior context when a pointer/page was actually surfaced, so already-surfaced pages still suppress. The suppression mode flows through all three resolver rungs (host opts, serve IPC request, direct Postgres). Ambient-channel feedback (codex D11): the server-side resolver paths (serve IPC + direct Postgres) log volunteered pointers with channel: 'reflex' through the drained volunteer-events sink, so `gbrain volunteer-context --stats` measures the default-on path where most volunteering happens. Host-injected resolvers (no gbrain engine) can't log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the pointer cap are unchanged. Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only vs already-surfaced suppression; throwing resolver stays fail-open (16 green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): gbrain watch — push transport over stdin (#2095) The issue's headline: the brain volunteers pages as the conversation flows, instead of waiting to be asked. `some-transcript-feed \| gbrain watch` reads turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed lines are user turns), keeps a rolling window (--window-turns, default 4), and streams confidence-gated pointers with rationales to stdout (--json for JSONL). Session dedupe rides the core's slug-only suppression — a slug is volunteered at most once per session. Events log on channel 'watch' with session_id + turn through the drained sink. Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF — so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1 placeholder): when main() resolves the work is over, the CLI_ONLY finally drains volunteer events via drainThenDisconnect, and the entrypoint flush-exit ends the process. Keeping it in the daemon set would have made the piped EOF path hang on lingering sockets — the exact #2084 class. SIGINT closes the stream and flows through the same drain path instead of killing mid-write. Per-turn resolution failures are fail-open (the stream never dies on a transient DB error). Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) + THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP op) + main --help entry. Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling window via assistant-introduced entity, session dedupe, --json shape with turn attribution, channel-watch event rows, --min-confidence gate, CRLF/ blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch` on a fresh PGLite brain exits 0 at EOF with no force-exit banner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave - docs/architecture/KEY_FILES.md (current-state): context entries gain the window extractor, arm provenance/confidence, suppression modes, volunteer + volunteer-events modules; background-work entry now lists FIVE sinks and the drainThenDisconnect owner-disconnect contract; new entries for src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts. - docs/guides/push-context.md (new): the three channels (reflex/op/watch), the confidence model, CLI usage, config keys, and the approximate-stats caveat. Linked from CLAUDE.md's reference map. - CLAUDE.md: ops line mentions volunteer_context + the guide link; bun run build:llms regenerated in the same commit (freshness test green). - TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor check, structured messages[] param); the #1981 entity-detection TODO narrowed (window extraction covered assistant-introduced entities + named-antecedent follow-ups); the drain-hoist P3 marked DONE by the #2084 wave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(e2e): truncate context_volunteer_events in setupDB (#2095) The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows persisted across e2e runs on a reused database and poisoned count/stats assertions in volunteer-context-postgres on the second run. No FK to pages (slug join), so position before pages is for hygiene only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084) Caught by the full unit suite: `gbrain apply-migrations` on PGLite started exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM backend's proc_exit status into process.exitCode (initdb at create-time, the postmaster at close-time — `exitCode=status` in pglite's dist), and the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around create/close (a guarded attempt verified this). The pre-#2084 success path never read process.exitCode, so the pollution was invisible; the new deliberate flush-exit propagated it faithfully. Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in cli-force-exit.ts — every gbrain-owned exit-code assignment routes through the setter (still mirrored to process.exitCode for outside readers), and both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect hard-deadline backstop) read the getter. Swept all assignment sites: cli.ts (op error, friction, claw-test, smoke-test, eval runners, status, import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot. Also updates the v0.42.20 structural pins to the drainThenDisconnect shape (ordering invariant asserted INSIDE the helper + >=8 helper call sites, superseding the two-inline-pairs assertion). Verified: apply-migrations spawn test green; `init --migrate-only` exits 0; an errored op still exits 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: re-pin the teardown-arming invariant at its post-#2084 home Master's v0.42.41.0 triage wave and the #2084 wave fixed the same pre-armed-timer bug independently; the merge keeps #2084's shape (arming inside the shared drainThenDisconnect helper, covering all 8 exit paths). The structural pin now asserts the same invariant — no pre-try arming; gated, unref'd, before-drain, cleared — at the helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: coverage for ambient reflex-channel logging + watch window/cap flags Ship coverage audit (85%, gate PASS) named five gaps; the two substantive cheap ones close here: the codex-D11 logChannel='reflex' path now has a behavioral pin (events land on channel 'reflex' through the drained sink; no logChannel → no events), and gbrain watch's --window-turns / --max-pages flags are exercised (turn-1 attribution under window=1; cap to one page). Remaining flagged-not-blocking: the wallclock-timeout branch (untestable without >10s real-clock flake — same rationale as the arming pin), formatResult's volunteer case (module-private), and the cycle purge wiring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT formatResult exported for tests (same import-safety contract as cliAliases); test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message, and approximate stats summary. test/watch-command.test.ts gains a real subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream, assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under the actual signal, not just the shared exit path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin The merged-state suite caught it: doctor --fast --json reported FAIL but exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes (doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit silently zeroes — getCliExitCode() deliberately never reads ambient process.exitCode (the PGLite-Emscripten pollution defense), so any setter that bypasses setCliExitCode reports success on failure. Swept both sites and added the structural class pin: a test greps src/ for raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that introduces one fails loudly instead of lying about exit codes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v0.42.43.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: quarantine the watch SIGINT subprocess test to the serial lane The parallel unit shards flake on concurrent CLI subprocess spawns (failed at 7ms in-suite, green solo) — same isolation rationale as apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v0.42.43.0 Post-ship doc verification against the release diff (#2095 push-based context + #2084 superset hardening), with a cross-model doc review: - push-context.md: version tag corrected to v0.42.43.0; per-call knobs now cover prior_context/days and watch's flag surface accurately; feedback-log writes described as best-effort; synopsis fence-strip described as unconditional. - CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context release reference corrected to v0.42.43.0. - KEY_FILES.md: ci-local entry rewritten to current topology (4-shard parallel default, four Postgres services, transaction-mode PgBouncer + GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts dropped from the selector entry. - TESTING.md: inventory entries for the new #2084 structural pins (cli-exit-verdict-pin, cli-pipe-truncation), the push-context test suite (volunteer-context, watch-command, watch-sigint.serial, cli-format-volunteer), migrate v117 coverage, and the two new E2E files (pgbouncer-teardown env gating, volunteer-context-postgres RLS pin); check:all row corrected (not a superset of verify). - AGENTS.md + RELEASING.md: ci:local descriptions updated to the sharded + pooler topology. - CHANGELOG (wording only, entry preserved): "retrieved" instead of "opened" for the used-signal, pooler scoped to the local CI gate, feedback log labeled best-effort. - llms-config.ts: index the new push-context guide; bundles regenerated (build:llms) and freshness test green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(test): correct the v116 reference — the table shipped as migration v117 * fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095) Five specialist reviewers (testing/maintainability/security/performance/ data-migration) on the reconciled diff; every finding applied: Performance: the alias arm now resolves all granted sources CONCURRENTLY (a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources cross-region, inside the reflex's 1.5s budget); watch's session dedupe is O(1) Set membership instead of a monotonically growing priorContext string (O(T²) over a long-lived session); getWindowTurns iterates from the tail (per-turn cost no longer grows with session length); the resolver's provenance maps fold into the existing candidate pass. Security: volunteer_context clamps caller-supplied attribution at the trust boundary — session_id capped at 256 chars (a read-scoped token could bank ~1MiB TEXT per request, retained 90 days), turn logged only when a safe integer (a non-integer threw inside the batched INSERT and silently dropped the whole batch). The privacy comments now state precisely what rationale may contain (the matched entity's surface form — which by construction resolved to an existing alias/title/slug — never free conversation text). Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from volunteer.ts and shared by watch/cli (the two surfaces can no longer drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly site for all three channels; watch's window default now honors the same retrieval_reflex_window_turns config knob the reflex reads; the stale pre-v116 comments swept to pre-v117. Testing: the two flake-class CRITICALs fixed (pipe test asserts the backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test waits on watch's new machine-readable ready line instead of a fixed 15s sleep — 2.5s and deterministic now); new coverage for the sink's timeout branch + ghost-reference drop, watch per-turn fail-open, untrusted knob clamps (min_confidence/max_pages/days), window-cap ordering (newest user mention survives), serve-IPC suppression passthrough + channel=reflex logging, windowTurnCount edge semantics, and structural pins for the sink registration + cycle purge wiring. The exit-verdict pin's grep is now operator/whitespace-tolerant. Deferred with TODOs: resolver index shapes for the per-turn query; batched first-prune after a long dream-cycle gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp Four red-team findings on the #2095 push-context surface: - RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's cap, so a recurring already-pushed entity burned cap slots every turn and starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside the pointer loop BEFORE the confidence gate and the cap. - RT3 honest stats: reflex-channel event logging moved from inside the resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires only after the response write succeeds, and buildReflexAddition logs only after the per-turn timeout admits the block. A block the client's 250ms budget abandoned was never injected and no longer counts as volunteered. (logChannel resolver opt removed; logDeliveredReflexPointers is the seam.) - RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo can't reintroduce the re-scan-everything-per-turn cost class. - RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP, push-context guide, TODO to route watch via serve IPC); host-resolver suppression contract at ResolveEntitiesFn (TODO for a capability gate). Tests: starvation guard (watch + volunteerContext unit), window clamp floor + ceiling, delivery-side logging (helper writes channel=reflex through the drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims Three CI-only check failures, two root causes: 1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when loadConfig() returned null (no config file AND no DATABASE_URL — a clean CI shard with no brain). loadConfig drops its env→config mapping in that case, so the documented escape hatch silently died and the window fell back to 4 → windowed extraction widened when the test set window=1 → prior-turn entity leaked. Fixed: read the env var directly in windowTurnCount, mirroring reflexEnabled's direct process.env read. This is a real product bug, not just a test artifact — any config-less host using the env hatch was affected. Regression test pins it. 2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway and never reset it. The legacy-embedding preload only restores the OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign config survives into the next file — and a file's beforeAll runs BEFORE the preload's restoring beforeEach, so federation-health built a vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new test files reshuffled the deterministic file→shard assignment, exposing this latent ordering bug. Hardened both victims to establish the gateway state they assert (sync-cost-preview resets to the unconfigured fallback; federation-health pins legacy 1536 before initSchema) so they're order-independent. Verified against a simulated leaker run before them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(context): use withEnv() in the window env-hatch test (test-isolation guard) The regression test added in 82cc7fff mutated process.env directly, which check:test-isolation (R1) forbids — use the withEnv() helper that restores on exit, same as the rest of this file. Behavior identical; guard green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-14 09:32:58 -07:00
Garry Tan	5a06af5a57	v0.42.32.0 fix(sync): coerce non-string frontmatter titles + bounded auto-skip failure ledger (#1939 ) (#1956 ) * fix(import): coerce non-string frontmatter title/slug/type (#1939) YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number; the old `(frontmatter.X as string)` cast was a compile-time lie, so downstream `.toLowerCase()` threw and (via the importer failure gate) could wedge sync indefinitely. parseMarkdown now coerces via coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the pure assessContentSanity self-protects against a non-string title. * feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939) New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe, multi-source, concurrent bounded auto-skip valve. A file that fails N consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it can't freeze all indexing forever, while fresh failures still fail-closed and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed. - (source_id, path) keying — failures never merge across sources - success clears a path so attempts are truly consecutive - advance-before-ack ordering (a crash can't mark a file skipped while wedged) - shared applySyncFailureGate used by BOTH the incremental and full-sync gates - legacy-row normalization + duplicate collapse on load - cross-process lock + atomic temp-rename, age-based stale-lock break sync.ts re-exports the ledger for existing callers; import.ts records source-scoped and defers the bookmark to the gate under managedBookmark. * fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939) Local buildChecks and remote doctorReportRemote now both route through decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL consistently (oldest-open age > fail cadence, or large unresolved count), auto-skipped pages stay visible (WARN, not hidden), and the acknowledged/acknowledged_at field-split that caused drift is gone. The remote surface stays subprocess-free (file read + Date.parse only). * chore(test): add trailing newline to e5-lease-cap-ab baseline fixture * fix(sync): address adversarial review findings on the failure ledger (#1939) - #1: a parse-failed file that is later deleted/renamed-away no longer leaves a permanent open ledger row. Removed paths (filtered.deleted, renamed-from, and the "gone from disk" forward-delete skip branch) are treated as resolved so the ledger self-heals instead of aging doctor to a stuck FAIL. - #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures only — auto_skipped rows already advanced the bookmark, so they stay WARN-visible regardless of count, matching the state-machine contract. * chore: bump version and changelog (v0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sync-failure ledger + auto-skip valve for v0.42.30.0 KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate, GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger, re-exported), doctor.ts (sync_failures severity via shared rule on both surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark). live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.32.0 (queue collision) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-07 19:22:33 -07:00
Garry Tan	805814451e	v0.42.26.0 docs(supabase): update connection-string setup to new UI + Transaction pooler (#1848 ) (#1875 ) * docs(supabase): update connection-string setup to new UI + Transaction pooler Supabase moved the connection string under "Connect" in the top nav and now shows three options (Direct, Transaction pooler, Session pooler). Update the tutorial, gbrain init prompts, the setup skill, the verify runbook, and the live-sync guide to recommend the Transaction pooler (port 6543) — which gbrain is tuned for (prepared statements disabled, DDL/locks routed to a derived direct connection). Document the IPv4 footgun: the derived direct connection is IPv6-only, so on IPv4-only hosts reads work but sync silently skips pages. Tutorial 7c now leads with the free fix (GBRAIN_DIRECT_DATABASE_URL -> Session pooler, port 5432) and keeps the IPv4 add-on as the paid alternative. Removes stale "transaction mode breaks sync (.begin() is not a function)" warnings and the port-6543 "Session pooler" mislabels. Extends PR #1848 by @FilipHarald. * chore: bump version and changelog (v0.42.26.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-04 21:34:43 -07:00
Garry Tan	f11d56cfca	v0.42.23.0 feat(jobs): --nice scheduling-priority flag for jobs work/supervisor (#1815 ) (#1820 ) * feat(jobs): niceness core, worker registry, shared supervisor-pid reader OS scheduling-priority primitives for issue #1815: - niceness.ts: parseNiceValue (whole-string), applyNiceness (re-reads effective in success AND failure paths), getEffectiveNiceness, formatNice. - worker-registry.ts: live workers self-register pid + requested/effective nice under gbrainPath('workers'); readWorkers prunes ESRCH (keeps EPERM) with a pid-reuse start-time guard. - supervisor-pid.ts: readSupervisorPid extracted from the copy-pasted PID-file + liveness block. * feat(jobs): --nice flag for jobs work/supervisor + doctor niceness check Wires the --nice <n> flag (and GBRAIN_NICE env) through the CLI (issue #1815): - jobs work: applies niceness + registers the worker; cleanup on finally and process.on('exit'). - jobs supervisor: applies in the foreground-start path only (after the --detach fork), passes the apply result into MinionSupervisor. - supervisor.ts: nice opts, extracted testable buildWorkerArgs (appends --nice), emits niceness on started/worker_spawned audit events. - jobs stats / supervisor status: surface effective worker + supervisor nice. - doctor: separate supervisor_niceness check (warns on requested != effective) so it can't clobber the supervisor crash-check precedence; registered in doctor-categories. * test(jobs): cover niceness, worker registry, supervisor-pid, build args Unit tests for issue #1815: parseNiceValue rejects 3.5/10abc that parseInt would accept; applyNiceness re-reads effective on EPERM; registry ESRCH/EPERM + pid-reuse guard + brain-isolated path; readSupervisorPid states; parseNiceFlag flag>env precedence; buildWorkerArgs --nice propagation. * chore: bump version and changelog (v0.42.23.0) --nice flag for jobs work/supervisor (issue #1815). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document --nice flag for jobs work/supervisor (v0.42.23.0) - minions-deployment.md: niceness tuning section (full concurrency, low priority). - KEY_FILES.md: entries for niceness.ts, worker-registry.ts, supervisor-pid.ts; supervisor.ts entry notes buildWorkerArgs + nice opts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): add enrich_thin to dream cycle EXPECTED_PHASES The enrich_thin cycle phase (src/core/cycle.ts ALL_PHASES, between conversation_facts_backfill and skillopt) shipped without updating the e2e phase-order expectation, so dream-cycle-phase-order-pglite failed on master. Sync the expected list to the real ALL_PHASES order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): align sync-lock-recovery with the shipped --break-lock --all contract v0.41.13.0 intentionally dropped the "--break-lock + --all is refused" guard so cron can self-heal every source in one call (sync.ts runBreakLock iterates sources under --all). The e2e test still asserted the old exit-1 refusal and failed on master. Assert the current contract: the combination is accepted and takes the iterate / no-active-sources path (exit 0, no refusal message). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): de-flake ingestion-roundtrip chokidar first-drop race The native fsevents watcher occasionally missed a freshly written file, timing out the 15s waitFor (~1/3 on master under load). Three fixes: - inject a polling chokidar watcher via the source's _watchFactory seam (usePolling, 20ms interval) so detection never depends on fsevents timing; - drop deterministic fixtures BEFORE start so the initial scan (ignoreInitial:false) emits them, keeping live-watch coverage only where it's robust; - poll for the dedup hit instead of a fixed 600ms sleep. 15/15 green under stress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): make hermetic-PGLite serve tests actually hermetic connect-bearer and serve-stdio-roundtrip init a PGLite brain and spawn serve, but passed {...process.env} through — leaking an ambient DATABASE_URL / GBRAIN_DATABASE_URL into the subprocess, which then came up on Postgres and failed the `engine: pglite` assertion. Strip both DB vars from the spawned env so the tests are deterministic whether or not the shell/CI has a DB URL set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): type the hermetic-PGLite env so tsc passes The DATABASE_URL/GBRAIN_DATABASE_URL strip used `delete` on a narrowly-typed env literal (tsc-only failure; bun test doesn't typecheck). Annotate connect-bearer's env as Record<string,string\|undefined> and build serve-stdio's as a concrete Record<string,string> (StdioClientTransport.env rejects undefined). Runtime behavior unchanged (7/7 + 3/3 green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: quarantine worker-registry to .serial (R1 env-mutation isolation) worker-registry.test.ts sets process.env.GBRAIN_HOME per-test so gbrainPath resolves to a temp dir, then lazy-imports the module — a process-global mutation the parallel isolation lint (rule R1) forbids. Rename to worker-registry.serial.test.ts: it runs in the serial pass (own bun process, max-concurrency=1) where env mutation is safe, and the lint skips .serial files. No logic change (6/6 green); fixes the failing `verify` CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 15:24:00 -07:00
Garry Tan	f4959348c2	v0.42.22.0 fix(minions): supervisor progress watchdog + worker DB self-defense — alive-but-wedged worker self-heals (#1801 ) (#1824 ) * fix(minions): supervisor progress watchdog + worker DB self-defense under supervision (#1801) Alive-but-wedged worker (dead DB pool, process still up) now self-heals in minutes instead of a silent 15h halt. - supervisor: progress watchdog restarts a child that makes no forward progress on claimable work (name+queue-scoped, active_healthy/due-delayed aware, startup-grace + loop-budget bounded); runtime handler-name derivation. - child-worker-supervisor: killChild gates on liveness not .killed (also fixes the existing shutdown SIGKILL no-op); restartCurrentChild kills the captured child ref; intentional restart doesn't count toward max_crashes. - worker: DB-liveness probe runs under supervision (db_dead self-exit), stall detection stays supervised-off. - doctor: standalone per-queue wedged_queue check + state->status fix in the remote queue_health check. - jobs/queue: queue-scoped getStats wedge fields + jobs stats WEDGED line. * fix(minions): wedge_restart_loop one-shot + supervised-probe comment + jobs-stats threshold (review) Pre-landing adversarial review findings: - wedge_restart_loop warn now fires once per exhausted window via a re-arming flag, not every health tick (was flooding the audit log for the full window). - Correct the stale GBRAIN_SUPERVISED comment: the DB probe runs under supervision now; only stall detection is skipped. - jobs stats WEDGED line reads GBRAIN_WEDGED_QUEUE_WARN_MINUTES so it agrees with the doctor wedged_queue threshold. * chore: bump version and changelog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: queue-ops runbook + KEY_FILES for the #1801 wedge watchdog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 14:14:13 -07:00
Garry Tan	a57d98b813	v0.42.12.0 feat: self-upgrading gbrain — invocation-riding update check + opt-in auto-upgrade (#1798 ) * feat(self-upgrade): decision/cache/snooze foundation + atomic binary self-update Pure decideSelfUpgrade (invocation + autopilot channels), atomic untrusted cache + escalating snooze + shared marker grammar (forged-marker rejection), semver helpers, and real darwin-arm64/linux-x64 binary self-update (download -> fsync -> smoke -> atomic rename; failure leaves old binary intact). Tests incl. real-HTTP-server swap E2E. * feat(self-upgrade): check-update cache/markers, self-upgrade command, CLI heartbeat hook check-update gains gstack-style cache/snooze/markers + refreshUpdateCache + exported fetchLatestRelease. New 'gbrain self-upgrade' command. cli.ts emits the update marker on every invocation (cache-read-only hot path, detached single-flight refresh, skip-set + recursion guard + NODE_ENV=test gate). * feat(self-upgrade): autopilot silent channel, doctor check, runPostUpgrade setup, config + identity marker autopilot opt-in silent channel (auto+quiet+idle, swap-only+breadcrumb+exit-relaunch) + installSystemd Restart=always + migrateSystemdUnitToRestartAlways. doctor self_upgrade_health. runPostUpgrade applySelfUpgradeSetup (one-time consent + systemd rewrite). init defaults mode=notify. config self_upgrade plane + KNOWN_CONFIG_KEYS. get_brain_identity carries update marker. * docs(self-upgrade): gbrain-upgrade agent skill, RESOLVER/manifest, auto-update doc reversal, HEARTBEAT New skills/gbrain-upgrade agent flow (mirror gstack-upgrade) wired into RESOLVER + manifest. upgrades-auto-update.md reversed to document opt-in auto + conservative gates. HEARTBEAT self-upgrade --check-only line. llms-full regenerated. * chore: bump version and changelog (v0.42.12.0) Self-upgrading gbrain: invocation-riding update marker + opt-in autopilot silent channel + real atomic binary self-update. Mirrors gstack's mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(self-upgrade): write just-upgraded-from breadcrumb + clear stale cache after upgrade Codex ship-review P3: the CLI startup hook reads just-upgraded-from to print the one-time JUST_UPGRADED confirmation, but nothing wrote it — dead path. runUpgrade now writes the breadcrumb (covers full + --swap-only) and clears the update-check cache + snooze so a now-applied 'upgrade available' marker stops nudging. * feat(self-upgrade): surface what's-new in notify + wire agent integration (AGENTS.md, HEARTBEAT) + e2e - self-upgrade --check-only --json now includes changelog_diff + release_url (export fetchChangelog); the gbrain-upgrade skill shows 3-5 what's-new bullets before the 4-option prompt instead of just version numbers. - setup injects a self-upgrade marker protocol into AGENTS.md so interactive agents (Claude Code, Codex) act on the UPGRADE_AVAILABLE stderr marker — the piece that makes notify actually fire for them. - HEARTBEAT daily beat routes through the gbrain-upgrade skill (OpenClaw/Hermes cron cadence); auto-mode daemons ride the autopilot tick. - e2e: real subprocess invocation proves the marker fires (notify emits; off/snooze/up-to-date silent; JUST_UPGRADED fires+clears; --quiet suppresses). Serial test: --check-only surfaces the changelog. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 06:20:03 -07:00
Garry Tan	d4211f4176	v0.42.11.0 feat(skillopt): held-out eval gate, honest receipts, ENFORCE + ablation opts (#1759 ) * feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/*/.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite\|--supabase\|--url] Create brain (PGLite default, no server) migrate --to <supabase\|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>\|--all\|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in\|out\|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links\|timeline\|all> Extract links/timeline (idempotent) [--source fs\|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check\|fix> [dir] Find/fix missing back-links across brain lint <dir\|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet\|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save\|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save\|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function\|class\|method\|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show\|get\|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: vX.Y.Z (#NNN): clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the 284c50a4 band-aid: re-inlines docs/what-schemas-unlock.md now that the restructure freed ~530KB of bundle headroom (llms-full.txt 740KB -> 225KB). verify 30/30 green (incl. new check:doc-history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate verbose release process to docs/RELEASING.md The highest-/ship-risk commit (isolated so it can revert alone). Moves the verbose release + contributor procedure out of CLAUDE.md, keeping every ship-critical IRON RULE inline so /ship + /document-release (which read CLAUDE.md) cannot regress. Moved to docs/RELEASING.md: pre-ship test requirements; the CHANGELOG-branch-scoped + CHANGELOG voice + release-summary template; the 'To take advantage of vX' block spec; version migrations + migration-is-canonical; schema state tracking; GitHub Actions SHA maintenance; PR-descriptions-cover-the-branch; community-PR-wave; checking-out-PRs-from-garrytan-agents. Kept INLINE in CLAUDE.md (ship-critical IRON RULES — do NOT move): - the Version-locations table (5-file sync) + the 3-line consistency audit - Conductor branch=workspace - Post-ship /document-release (MANDATORY) - Privacy + Responsible-disclosure rules (Privacy also anchors the check-privacy allowlist — the only place allowed to name the fork) - PR-title-version-first - never-hand-roll-ship (Skill routing) Plus a new ## Releasing pointer ('Before any ship, read docs/RELEASING.md in full') and a resolver row. CLAUDE.md 61KB -> 39KB (592KB -> 39KB overall, 93% cut; ~9k tokens auto-loaded vs ~147k). CLAUDE.md size-gate tightened 90KB -> 60KB. The content-contract tests pin that the inline IRON RULES (MAJOR.MINOR.PATCH.MICRO, document-release, hand-roll ship) did NOT move out. The moved ranges carry no banned fork name, so RELEASING.md needs no privacy allowlist entry. verify 30/30; bundle 225KB -> 204KB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(changelog): note CLAUDE.md restructure in v0.42.9.0 The CLAUDE.md thin-resolver restructure (592KB → 39KB) rides in this release; record it under the existing v0.42.9.0 For-contributors section. No version bump — v0.42.9.0 is unreleased and already allocated to this PR. * fix(ci): ci-cache-hash re-admit matched a literal \t, a no-op on GNU grep The policy-doc re-admit (75992b77) put `\t` inline in the ALLOW patterns passed to `grep -E`. BSD grep (macOS local) treats `\t` as a tab so it worked locally; GNU grep (Ubuntu CI) treats it as literal `t`, so nothing re-admitted and docs/TESTING.md / docs/RELEASING.md stayed deny-listed — the two policy-doc tests failed on CI shard 6 (1097 pass / 2 fail). Build ALLOW_RE with `printf '\t(%s)'` so the tab is a real byte, identical in construction to DENY_RE (line 117), which the CI log shows matches correctly on GNU grep. End-to-end: editing docs/TESTING.md now flips the hash; a normal docs/.md add still does not (deny stays scoped). fix(skillopt): feed the scorer's success criteria to the optimizer Surfaced by the SkillOpt real-LLM eval (Track B). The reflect step was shown only a pass/fail score and the agent transcript — never WHAT the benchmark judge rewards. On a skill judged by structure (e.g. "must include a Confidence: line") the optimizer proposed plausible-but-off edits ("close with a synthesis") that never satisfied the literal check; every candidate scored 0 on D_sel, the validation gate rejected them all, and the skill text never changed (optimized === baseline === 0). Fix: render each benchmark Judge (rule checks / llm rubric / qrels) into plain-English criteria via new exported describeJudge / describeJudges, and thread them into the reflect prompt (a SUCCESS CRITERIA block) for both the loop reflect calls and the one-shot-rewrite path. The orchestrator computes the distinct criteria across train+sel+test once. The optimizer system prompt now instructs it to satisfy the criteria through genuine content, never empty keywords — reward-hacking stays defended by the independent held-out gate (cat32 confirms the gate catches a keyword-stuffing hack). End-to-end this took a deficient skill from 0.00 to 1.00 on a held-out set it never trained on. Pinned by test/skillopt/reflect.test.ts (describeJudge per kind, describeJudges dedup, criteria present/absent in the prompt). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-03 05:50:21 -07:00
Garry Tan	eefe8b5741	v0.42.1.0 feat: gbrain skillopt — self-evolving skills (closes #1481 ) (#1563 ) * feat(skillopt): foundation modules — types, lr-schedule, benchmark, score, audit, lock * feat(skillopt): edit primitives — apply-edits (D5+D9), rejected-buffer LRU, version-store (D8 history-intent-first) * feat(skillopt): rollout (D2 gateway.toolLoop + D13 read-only allowlist), reflect (D7 two calls), validate-gate (D12 median+epsilon, D4 parallel), preflight (D3), bundled-skill-gate (D16) * feat(skillopt): orchestrator (D6 slow-update, D10 ASCII diagrams, D11 caching), checkpoint, bootstrap (D15 sentinel), CLI dispatch + help * feat(skillopt): cycle phase (F1 dream-loop wiring), PROTECTED_JOB_NAMES + MCP op (F6 admin scope + allowlist) + Minion handler (F7 --background) * feat(skillopt): full cathedral — --all batch (F4), --target-models fleet (F5), write-capture (F10), held-out scaffold (F11), adversarial suite 41 cases (F2), E2E PGLite (F3), meta-skill bundle (T7), reflect+judge evals (F8+F9), docs (T10) * chore: bump version to v0.42.0.0 (MINOR — significant new feature) * fix(skillopt): wire trajectories from forward gate to reflect + fix parseEditsResponse parser misuse Two related v0.42.0.0 bugs that conspired to make `runSkillOpt` structurally unable to accept any candidate edit. Either alone would have killed self-evolution; together they made the loop a no-op for every input. Bug 1 (orchestrator gap): `runOptimizationLoop` in orchestrator.ts called `runReflect({successes: [], failures: []})` with hardcoded empty arrays. The forward gate's `scoredRollouts` were computed then voided. `runReflect` short-circuits both modes when their batches are empty, so the optimizer was never asked to propose an edit. Every step hit the no_edits_applied branch. Fix: add `scoredRollouts: ScoredRollout[]` to `GateResult` and `runsPerTask?: number` to `ValidateGateOpts`. Forward pass uses `runsPerTask: 1`; orchestrator partitions returned rollouts by `score >= 0.5` and threads real successes + failures into `runReflect`. Bug 2 (parser misuse): `parseEditsResponse` in reflect.ts routed every optimizer response through `parseJudgeJson` first. `parseJudgeJson` looks for a `score` key (it's a judge-output parser, not an edits parser) and returns null for any JSON without one — including the well-formed `{"edits": [...]}` the optimizer is contractually required to emit. The function then early- returned `[]` and the actual `tryExtractEdits` path on the next line was unreachable dead code. Fix: drop the wrong-typed guard. `parseEditsResponse` now calls `tryExtractEdits` directly. Export it so `reflect.test.ts` can pin the contract independently of the chat transport. Why this slipped through 152 prior skillopt tests: zero unit coverage of `parseEditsResponse` or `runReflect`. The existing E2E `all-reject` case asserted no_improvement (which was true for the wrong reason — empty edits, not gate rejection). Both bugs were structurally invisible to the existing test surface. New coverage: - `test/skillopt/reflect.test.ts` (15 cases): - 8 `parseEditsResponse` cases including the IRON-RULE regression pin for the v0.42.0.1 fix (`{"edits": [...]}` JSON must survive the parser). - 7 `runReflect` D7 contract cases: both modes fire, empty-batch skips, additive token usage, one-mode-throws-other-still-works, rejected-buffer flows into anti-bias prompt. - Documents the trailing-comma limitation as an explicit out-of-scope pin (so a future tightening of `tryExtractEdits` lights this test up intentionally). - `test/e2e/skillopt-loop.serial.test.ts` (7 cases): - HAPPY PATH: stubbed `gateway.chat` acts as both target agent (emits sections based on skill content) and optimizer (proposes a real add-Citations edit). Drives `runSkillOpt` end-to-end against PGLite. Asserts outcome=accepted, SKILL.md mutated with new section, frontmatter preserved (D5), history has one committed row, best.md mirrors disk, delta > epsilon, receipt fields populated. - 5 broken cases (each isolates a distinct orchestrator-visible failure): 1. Below-baseline regression: optimizer proposes a destructive edit; gate rejects with reason=below_baseline; SKILL.md unchanged; rejected-buffer captures the bad edit for anti-bias context. 2. Malformed reflect JSON: orchestrator degrades gracefully to no_improvement without crashing. 3. Anchor-not-found: applyEditBatch rejects all; sel gate skipped; rejected-buffer captures with reason=apply_failed. 4. Budget exhausted mid-step: outcome=aborted, no pending rows survive. 5. Converged-skill re-run: starting from already-perfect skill → no_improvement (no thrash on a well-tuned starting point). - IDEMPOTENT RE-RUN: drive runSkillOpt twice in sequence. Run 1 accepts. Run 2 sees improved baseline, no failures, returns no_improvement. SKILL.md byte-identical to post-run-1; history still has exactly 1 committed row. Proves stability at the fixed point. All hermetic (no DATABASE_URL, no API keys). PGLite in-memory engine, tempdir SKILL.md + benchmark, stubbed gateway.chat via `__setChatTransportForTests`. `.serial.test.ts` because the stub installs module state and the loop walks shared disk state across epochs. Test counts after fix: 174 skillopt-surface tests pass (149 pre-existing unit + 15 new reflect unit + 3 existing E2E + 7 new E2E). Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cycle): align ALL_PHASES skillopt position with actual dispatch order v0.42.0.0 added skillopt to ALL_PHASES right after `patterns` (line 127), but the dispatch block in runCycle (line ~1912) actually runs skillopt between `conversation_facts_backfill` and `embed`. The two were inconsistent, and the serial test `report.phases.map(p => p.phase)).toEqual(ALL_PHASES)` was failing on master because of it. A second pre-existing failure: the two phase-count assertions in `test/core/cycle.serial.test.ts` still said `toBe(20)` even though ALL_PHASES grew to 21 when skillopt was added. The author bumped the array but forgot the test. Two fixes, one commit: 1. Move `'skillopt'` in ALL_PHASES from after `patterns` to between `conversation_facts_backfill` and `embed`, matching where runCycle actually dispatches it. Runtime behavior is unchanged — only the declaration order moves. Updated the surrounding comment to call out the position invariant and reference the test that pins it. 2. Update both `toBe(20)` assertions in cycle.serial.test.ts to `toBe(21)` with a v0.42.0.0 history line in the running comments. Why declaration follows runtime (not the other way around): the comment intent ("Runs AFTER patterns — graph-fresh") is still satisfied because "after the entire main graph-mutating cluster" is strictly fresher than "right after patterns". No design intent is lost. Test result: cycle.serial.test.ts is now 28/28 (was 27/28 on master + my prior commit). Skillopt suite still 174/174. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): bump PHASE_SCOPE assertion to 21 + fix skill-optimizer Anti-Patterns case Two CI failures pre-existing on this branch since the v0.42.0.0 skillopt cathedral landed; master is green because skillopt didn't exist there yet. 1. test/phase-scope-coverage.test.ts asserted ALL_PHASES.length === 20. skillopt is the 21st phase. Bumped to 21 with v0.42.0.0 history line in the comment chain. Sibling fix to the cycle.serial.test.ts bump in commit 08ad2468. 2. skills/skill-optimizer/SKILL.md had `## Anti-patterns` (lowercase p). skills-conformance.test.ts asserts `## Anti-Patterns` (capital P) as the required section header. Single-character rename. Local: 174 skillopt-surface tests + 6 phase-scope tests + 249 skills- conformance tests all green. Typecheck clean. Remaining CI delta: 5 put_page facts backstop failures in shard 10 that reproduce only on Linux CI, not locally even with empty env / cleared HOME / max-concurrency=1. The error surface is `r.isError === true` with no further detail captured in the bun:test output. Pushing these 2 fixes first to narrow the CI signal; will instrument if the 5 persist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(e2e): align dream-cycle-phase-order + onboard-full-flow with v0.41/v0.42 reality Two stale E2E assertion files surfaced by a full local E2E run against real Postgres (the gbrain-test-pg container on port 5434). Neither file is in the CI E2E job (CI only runs mechanical.test.ts + mcp.test.ts + skills.test.ts + zeroentropy-live.test.ts), so the drift has been latent. 1. `test/e2e/dream-cycle-phase-order-pglite.test.ts` EXPECTED_PHASES was missing 4 phases that landed in master since the list was last revised: - extract_atoms (v0.41 T9 — atom extraction, after extract_facts) - synthesize_concepts (v0.41 T9 — concept synthesis, after patterns) - conversation_facts_backfill (v0.41.11.0, after calibration_profile) - skillopt (v0.42.0.0 — self-evolving skills, between conversation_facts_backfill and embed) Updated to 21 entries in the actual runtime dispatch order (matches ALL_PHASES exactly). 5/5 tests in the file pass after. 2. `test/e2e/onboard-full-flow.test.ts` `runAllOnboardChecks` shape test asserted exactly 4 checks; v0.42's type-unification cathedral (PR #1542, T13-T15) added 3 more (`pack_upgrade_available`, `type_proliferation`, `dangling_aliases`) for a total of 7. And `empty brain returns 0 remediations` regressed because `pack_upgrade_available` can emit a manual_only remediation on brains where gbrain-base@1.x is active and gbrain-base-v2 is registered as a successor. Tightened that assertion to `total <= 1` AND kept a per-check guard asserting takes_count remediations stay 0 (the original test's load-bearing claim — A12 two-gate consent). 13/13 tests in the file pass after. Honest scope: 4 other E2E files still fail locally after this commit (cycle.test.ts, dream.test.ts, phantom-redirect.test.ts, sync-lock-recovery.test.ts), each for a distinct pre-existing master bug unrelated to v0.42 skillopt work: - cycle.test.ts (5 fails): PostgresEngine.getConfig falls back to db.getConnection() singleton via the `get sql()` getter when no poolSize is set; the new conversation_facts_backfill phase chain hits this fallback even though the test's setupDB() connects both the singleton AND the engine. Race condition between the test's singleton lifecycle and the phase's getConfig call. Deeper fix needed in PostgresEngine.getConfig (use this._sql directly with explicit fallback only on user-driven CLI paths). - dream.test.ts (1 fail): expects "concepts/testing" slug to appear in dream cycle output, gets empty array. Related to v0.42 concept type-unification semantics. - phantom-redirect.test.ts (2 fails): concurrent-sync race + postgres-js text-string embedding survival. Master-level data-path bug; would need its own fix wave. - sync-lock-recovery.test.ts (1 fail): `gbrain sync --break-lock --all` exits 0 but test expects 1 with a shell-loop hint. CLI behavior changed in a master commit; need to either restore the refusal behavior or update the assertion. None of these 4 block CI (E2E job doesn't run them). Filed as a TODOS.md entry for a follow-up wave; the 2 in this commit are the ones that mirror v0.42 work landing. Local: 130/136 E2E files green, 927/940 tests pass (was 925/940 before these fixes; the 2 files this commit fixes added 7 newly- passing tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): quarantine query-cache-knobs-hash.test.ts to serial runner CI shard 10 (commit 4d721077) failed 5 tests in the `SemanticQueryCache cross-mode isolation (CDX-4 hotfix)` describe block, all ~7-34ms each, all expecting writes/reads to round-trip through one shared PGLite engine + a `beforeEach DELETE FROM query_cache`. Passes 9/9 locally; fails 5/9 on Linux CI under bun's default in-file max-concurrency=4. Classic intra-file concurrency race shape: test A's `beforeEach` clears the table → test A's `store` writes a row → test B's `beforeEach` (concurrent with A's `store`) clears the table → test A's follow-up COUNT query returns 0. Same root cause that quarantined `embed-stale.test.ts`, `brain-allowlist.test.ts`, and `schema-pack-find-pack-successors.test.ts` to the serial runner in prior fix waves (documented in v0.41.22.0 CI fix wave). Fix: rename to `query-cache-knobs-hash.serial.test.ts` so the v0.26.7 serial-tests runner picks it up at `max-concurrency=1`. Tests still exercise the actual cache logic — no test deleted, no production code changed. The describe block's `beforeAll` engine + `beforeEach` TRUNCATE pattern works correctly at serial concurrency. Local: 12/12 in this file + 52/52 in the serial runner. Production SemanticQueryCache code is untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(heavy): frontmatter_scan_wallclock — opt into --no-embedding so CI runners work Heavy tests workflow run 26542447602 (commit 483a5577) failed on the first heavy script: [fm_wallclock] FAIL: gbrain init exited non-zero No embedding provider configured. Set one of: OPENAI_API_KEY / ZEROENTROPY_API_KEY / VOYAGE_API_KEY Or defer setup: gbrain init --pglite --no-embedding The v0.37 D9 hard-require landed in init.ts: `gbrain init --pglite` now refuses to proceed without an embedding provider configured. The heavy-tests GitHub workflow doesn't pipe any embedding API keys (deliberate — the heavy tests measure ops shape, not LLM behavior), so every CI invocation now blocks at step 2 of this script. The script's whole purpose is measuring `gbrain doctor`'s frontmatter-scan wallclock — it never embeds, never calls `gbrain embed`, never queries vectors. The right fix is to opt out of the provider requirement via the same `--no-embedding` flag init.ts already exposes for this exact "deferred setup" case. Verified locally: TMP=$(mktemp -d); GBRAIN_HOME="$TMP" \ bun run src/cli.ts init --pglite --yes --no-embedding # exit 0, brain initialized. No production code change. One-line + comment in the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(heavy): sync_lock_regression — pass --no-embed so CI runs measure lock contention, not key absence Heavy tests workflow run 26542545802 (commit 7962d312, after the previous fm_wallclock fix) failed at the next heavy script in the chain: [sync_lock_regression] outcomes: winners=0 losers=0 unknown=4 [sync_lock_regression] FAIL: expected 1 winner, got 0 [sync_lock_regression] FAIL: expected 3 lock-busy losers, got 0 Each of the 4 parallel `gbrain sync` invocations failed for the same reason — none of them ever even got to the lock-acquire step: Embedding model "zeroentropyai:zembed-1" requires ZEROENTROPY_API_KEY. Re-run with --no-embed to import-only and embed later once the key is set. The CI runner doesn't pipe any embedding-provider API keys (deliberate — heavy tests measure ops shape, not LLM behavior), and sync now hard-fails when its embed step can't reach a configured provider. This script measures the writer-lock race shape — `gbrain-sync` row in `gbrain_cycle_locks`, exactly-one-winner semantics, N-1 fail-fast losers with "Another sync is in progress", zero leaked rows post-run. It never needed embeddings; the original write predates the hard-require landing. Fix: pass `--no-embed` to the sync invocation. Same kind of fix as fm_wallclock (commit 7962d312) but on the sync side rather than init. No production code touched. One-line change in the bash script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(heavy): sync_lock_regression — register source via psql + use --repo + tolerate doctor warns Heavy tests run 26542638471 (commit 60145eee, after the --no-embed fix) failed at the same script but at a downstream step: > Source "default" has no local_path. Run: gbrain sources add default --path <path> Three independent bugs in the script that all surfaced at once after v0.41's source-registry landed: 1. `gbrain config set sync.repo_path` is the legacy way; sync now reads `sources.local_path` first. Replaced with an upsert into the sources table via psql: INSERT INTO sources (id, name, local_path) VALUES ('default', 'default', $BRAIN_DIR) ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path Kept the legacy `config set sync.repo_path` line too as belt-and-suspenders for any downstream caller that still reads it. 2. `gbrain sync --dir <path>` is silently ignored; sync's CLI parser recognizes `--repo`, not `--dir`. Switched to `--repo`. 3. `bun run src/cli.ts doctor --json` at the top (used to apply migrations as a side effect) exits non-zero whenever ANY check warns — including the new "no embedding provider configured" warning on a fresh CI runner. The script's `set -e` aborted at line 53 before reaching any of the sync invocations. Added `\|\| true` since the migration runs regardless of doctor's exit verdict. Verified locally — `DATABASE_URL=... bash tests/heavy/sync_lock_regression.sh` output: [sync 1] rc= (lock-busy: 'Another sync is in progress') [sync 2] rc=0 (winner) [sync 3] rc= (lock-busy: 'Another sync is in progress') [sync 4] rc= (lock-busy: 'Another sync is in progress') outcomes: winners=1 losers=3 unknown=0 post-run gbrain_cycle_locks(gbrain-sync) row count: 0 OK — 1 winner, 3 lock-busy losers, no leaked lock rows. Production code untouched. All three fixes are in the bash script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(skillopt): hands-on tutorial for auto-improving a skill + discoverability There was no tutorial for skillopt — only a reference guide (docs/guides/skillopt.md) that opens at --bootstrap-from-routing and assumes you already understand benchmarks, and an agent-facing SKILL.md. README had ZERO skillopt mention. The one thing a user must hand-author (the benchmark JSONL) was taught nowhere with a worked example. New: docs/tutorials/improving-skills-with-skillopt.md — Diataxis tutorial (learning-oriented), copy-pasteable end to end: 1. mental model in two sentences (SKILL.md is the trainable param, the agent is frozen) 2. write your first benchmark from scratch — a complete 15-task rule-judge starter you paste and run, with the full check-op table (contains/regex/section_present/max_chars/min_citations/tool_called/ tool_not_called) 3. --dry-run cost preview (and that it exits 2 by convention, not failure) 4. real run + reading accepted(0)/no_improvement(1)/aborted(2) with the actual stderr output shape 5. where output lands (best.md, versions/, history.json, rejected.json, audit jsonl) 6. accept/reject — bundled vs user skills, --no-mutate vs --allow-mutate-bundled 7. iterate by sharpening the benchmark The load-bearing fix the tutorial makes that the reference guide got wrong: the DEFAULT --split 4:1:5 needs ~50 tasks before it runs (sel = N/10, floor 5). A first-time author writing 10-15 tasks hits `D_sel has N task(s) (need >=5)` and bounces. The tutorial ships 15 tasks + `--split 1:1:1` (clean 5/5/5) so the copy-paste path actually works. Verified against the real loadBenchmark + splitBench: the exact shipped block parses 15 unique tasks and splits 5/5/5 with sel>=5; the system's own error message confirms "need ~50 total for 4:1:5". Discoverability (Diataxis cross-linking): - README.md tutorials section: new entry (was zero skillopt mention) - docs/tutorials/README.md: added under ## Shipped - docs/guides/skillopt.md: "New to this? Start with the tutorial" callout Every claim devex-verified against source: exit-code map from skillopt.ts (accepted:0/no_improvement:1/aborted:2/errored:2), stderr format from skillopt.ts:286-292, check ops from score.ts, output paths from SKILL.md, split math from benchmark.ts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: regenerate llms-full.txt after skillopt tutorial + README edit Refreshes the inlined doc bundle so the committed llms-full.txt matches fresh `bun run build:llms` output (test/build-llms.test.ts drift guard). Picks up the README tutorials-section edit from c39dbdb1. The new tutorial file itself isn't curated into scripts/llms-config.ts (the bundle curates a fixed doc set, not every tutorial) — this is purely the README delta. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): stop embed-preflight leaking gateway config into facts-backstop shard CI shard 10 failed 5 `put_page facts backstop` tests with: [embed(openai:text-embedding-3-small)] Incorrect API key provided: sk-test (captured by the diagnostic stderr added in a prior commit). Root cause is a cross-file module-state leak, not a logic bug: - `embed-preflight.test.ts` calls `configureGateway({env:{OPENAI_API_KEY: 'sk-test'}})` to drive credential-validation scenarios. It resets the gateway `beforeEach` but never AFTER its last test, so it leaves the gateway configured with `sk-test`. - bun runs every file in a shard inside ONE process. The residual config bleeds into the next file. When `facts-backstop-gating.test.ts` lands in the same shard, its put_page calls see `isAvailable('embedding') === true` (the key is present, just invalid), so put_page attempts a real embed and 401s before the backstop gating even runs. - It's intermittent across master merges because shard bin-packing changes which files co-locate. (It "resolved" after the v107 merge earlier for exactly this reason, then came back.) R1/R2 test-isolation lint doesn't catch this — it's `configureGateway` module state, not `process.env` or `mock.module`. Two fixes, both using the gateway's own `resetGateway()` seam (no process.env, R-compliant): 1. embed-preflight.test.ts — `afterAll(() => resetGateway())` so the leaker cleans up after the whole file. Primary fix; also protects any OTHER shard-mate that reads gateway state. 2. facts-backstop-gating.test.ts — `beforeEach(() => resetGateway())` so the suite is deterministic regardless of ambient gateway config. Defense in depth: isAvailable('embedding') is now reliably false → put_page uses noEmbed → the import never embeds → only the backstop gating (the suite's actual subject) is exercised. Verified: running leaker+victim in one process (the shard repro) goes 16/16; full shard 10 goes 1208/1208 (was 5 fail in CI). Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(skillopt): make benchmark authoring an agent job, not a human chore The prior tutorial taught a human to hand-write a 15-task benchmark — but nobody does that. The real workflow is: user says "make skill X better," the AGENT authors the benchmark and runs the optimizer. The agent-facing dispatcher didn't actually cover that. Gap found: skill-optimizer/SKILL.md documented exactly one authoring path, `--bootstrap-from-routing`, which (a) requires a pre-existing routing-eval.jsonl (bootstrap-benchmark.ts:57-63 refuses without it) and (b) generates tasks from ROUTING fixtures — which test dispatch ("does this phrasing pick this skill"), not output quality. So an agent told to improve a skill with no benchmark had no documented way to author a quality benchmark; it'd have to reinvent the JSONL format the human tutorial teaches. Two fixes: 1. skills/skill-optimizer/SKILL.md — new "Authoring the benchmark yourself (the common case)" section: read the target SKILL.md, generate ~15 realistic tasks, attach rule judges (contains/max_chars/min_citations/ section_present/regex/tool_called), write the JSONL, run with `--split 1:1:1` (the default 4:1:5 needs ~50 tasks). Decision-tree row "New skill, no benchmark" now says "Author one" instead of pointing at bootstrap-from-routing; the bootstrap row is reframed as a head-start that only applies when routing fixtures exist and notes routing tasks test dispatch, not quality. 2. docs/tutorials/improving-skills-with-skillopt.md — new "The easiest path: ask your agent" section up top. Tells humans to just tell their agent "improve my X skill — write a benchmark first," and frames the manual walkthrough as "read this when you want to understand or hand-curate what the agent is doing." Verified: conformance 249/0, resolver 99/0, build-llms drift guard 7/0, cross-link resolves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(skillopt): --bootstrap-from-skill starter benchmark generator Generate a quality benchmark from a skill's SKILL.md directly, no routing-eval.jsonl required. One LLM call emits JSONL tasks (each with rule judges) that the agent reviews + strengthens before optimizing. - runBootstrapFromSkill: JSONL output parsed line-by-line with skip-bad-line salvage (a truncated final line drops, the rest survive); a task is kept only when >=2 valid rule checks survive; provider errors propagate instead of collapsing to bootstrap_empty. - --bootstrap-tasks N (default 15, cap 50); maxTokens scales with the count. - Extracted assertBenchmarkAbsent + readSkillBodyOrThrow shared with the routing bootstrap; hardened runBootstrap's routing-eval parse to skip malformed lines. - CLI: --bootstrap-from-skill short-circuit + 6-way mutual exclusion; parseFlags exported for unit tests. The benchmark-not-found hint + --help now point here. - The generator's REVIEW line prints the paste-ready `--bootstrap-reviewed --split 1:1:1` next command (the default 4:1:5 split refuses a 15-task starter at D_sel >= 5). - 20 hermetic cases incl. round-trip into loadBenchmark + splitBench(1:1:1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(skillopt): make --bootstrap-from-skill the primary no-benchmark path The agent runs --bootstrap-from-skill, strengthens the generated judges (they are weak drafts), deletes the sentinel, then runs --bootstrap-reviewed --split 1:1:1. Freehand authoring is demoted to the fallback for the rare skill the generator can't draft well. Updates the Iron Law, decision tree, and anti-patterns to cover both bootstrap modes and the 15-task / --split 1:1:1 gotcha. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(release): v0.42.1.0 --bootstrap-from-skill VERSION + package.json -> 0.42.1.0, CHANGELOG entry, CLAUDE.md skillopt annotation, regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface --bootstrap-from-skill in README + skillopt reference - docs/guides/skillopt.md: 30-second pitch leads with --bootstrap-from-skill; flag table adds --bootstrap-from-skill + --bootstrap-tasks rows. - README.md: skillopt tutorial pointer mentions generating a starter benchmark. - Regenerated llms-full.txt (README is in the bundle). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): bump FULL_SIZE_BUDGET 700KB→750KB for legitimate CLAUDE.md growth The skillopt wave annotations + merged v0.41.34-36 master releases pushed llms-full.txt to 700,423 bytes — 423 over the 700KB cap — failing the build-llms size-budget test on CI shard 6. CLAUDE.md is ~540KB (77% of the bundle) and is the whole point of the one-fetch artifact, so it stays inlined; the budget tracks its per-release growth. 750KB still fits 200k+ context models. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-31 08:20:25 -07:00
Garry Tan	374deff579	v0.41.7.0 feat: compact list-format resolver + 300-skill scaling tutorial (#1407 ) * feat(check-resolvable): parseResolverEntries accepts compact list format Add the second parser branch alongside the existing markdown-table branch so RESOLVER.md and AGENTS.md can use the OpenClaw-native list shape: - skill-name: trigger1 \| trigger2 \| trigger3 - skill-name: trigger1 \| trigger2 Constraints: - Skill names must be kebab-lowercase ([a-z][a-z0-9-]+). Bold names starting with an uppercase letter (e.g. Note, Convention) are deliberately skipped so prose bullets in real-world AGENTS.md files don't get mis-parsed as fake skill rows. - skillPath is always derived as skills/<name>/SKILL.md. An optional arrow suffix (Unicode -> or ASCII ->) is stripped from the trigger string but NOT honored as a path. Downstream consumers (routing-eval.ts skillSlugFromPath, the manifest check at line 367) assume the convention. For non-conventional paths, use the table format. - Multiple triggers fan out to one entry per trigger. checkResolvable dedupes by skillPath downstream, so the reachability count counts each skill once regardless of trigger fan-out. The parser body is restructured to an if/else-if shape so the existing 'continue' on non-table rows no longer short-circuits the list branch. Unit tests cover 11 new cases: bold + plain name shapes, multi-trigger fan-out, Unicode and ASCII path-suffix strip, ellipsis filter, empty pipe segments, mixed-shape files, section tracking, and two D4 regression cases (prose-bullet rejection + convention-violation silent-skip). Closes #1370 — credit @garrytan-agents for the original PR that flagged the parser gap. * test(check-resolvable): integration fixtures + regression suite for compact format Two fixtures pin the v0.41.7.0 parser fix at the integration layer: test/fixtures/openclaw-compact-resolver/ List-format only RESOLVER.md with 10 fictional skills (gift-advisor, flight-tracker, email-triage, etc.), each with valid frontmatter triggers. A trailing 'Notes' section embeds 4 prose bullets (- Note:, - Convention:, - TODO:, - Important:) that pin the D4 kebab-lowercase regex tighten: if the regex ever regresses to permissive [\w-]+, those prose bullets would surface as orphan_trigger warnings and the test fails loudly. test/fixtures/openclaw-mixed-merge/ Tests the v0.31.7 D-CX-14 multi-resolver merge: workspace-root AGENTS.md (compact list, 3 skills) + skills/RESOLVER.md (table format, 5 skills). The merge dedups by skillPath and counts each skill once. The regression test (test/check-resolvable-openclaw-compact.test.ts) runs 8 assertions across both fixtures: 1. unreachable === 0 on the compact fixture (the 'pre-v0.41.7.0 reported 238 FAILs on a 306-skill OpenClaw, post-fix 0' headline). 2. zero error-severity issues; report.ok === true. 3. zero mece_gap warnings (every stub ships valid triggers). 4. zero orphan_trigger warnings for the 4 prose-bullet names — D4 regex regression guard at integration level. 5. zero missing_file warnings. 6. mixed-merge: total_skills === 8 (5 table + 3 list), all reachable. 7. mixed-merge: errors.length === 0; report.ok === true. 8. mixed-merge: each expected skill from BOTH shapes is non-unreachable (catches the bug where one shape silently swallows the other via dedup-by-skillPath). * docs(guides): scaling-skills.md walkthrough for 300-skill agents Three-tier architecture for agents that have outgrown the always-loaded skill manifest: Tier A — always loaded (~35 skills, in the system prompt every turn) Tier B — resolver-routed (~85 skills, looked up via RESOLVER.md/AGENTS.md only when no Tier A match) Tier C — dormant (~180 skills, on disk but not injected into the prompt) Real numbers from Garry's 306-skill OpenClaw: 25K tokens of skill descriptions per turn collapsed to 4K tokens (~21K tokens freed per turn) with zero capability loss. The compact list-format resolver (v0.41.7.0) is the parser-level enabler for this pattern. The guide covers: - The scaling wall (when the always-loaded manifest stops working) - The three tiers + per-turn token math - What the resolver actually does (routing-table-but-cheaper pattern) - The compact list format (kebab-lowercase contract, optional path suffix, mixed-shape support) - The 'gbrain doctor' / 'gbrain check-resolvable --strict' safety net - Implementation walkthrough (audit → tier → disable → resolver → doctor) - The scaling curve (50 → 100 → 200 → 300 → 1000, no ceiling) Voice + privacy cleanup applied per CLAUDE.md rules: - Wintermute → 'Garry's OpenClaw' / 'your OpenClaw' - Unicode em dashes stripped; ASCII '--' preserved in command flags - Made-up 'check_resolvable' invocation replaced with real 'gbrain doctor' and 'gbrain check-resolvable --json'/'--strict' - Blog-style 'Previous in this series' footer dropped Wiring: - scripts/llms-config.ts registers the new guide in the curated array so 'bun run build:llms' picks it up. docs/UPGRADING_ DOWNSTREAM_AGENTS.md excluded from the inlined bundle to stay under the 600KB FULL_SIZE_BUDGET after adding the new content. - docs/tutorials/README.md gains a one-line entry pointing at the guide under Related documentation. - llms.txt + llms-full.txt regenerated. * chore: bump version and changelog (v0.41.7.0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: update CLAUDE.md for v0.41.7.0 compact-format resolver Annotate the src/core/check-resolvable.ts entry with the v0.41.7.0 parseResolverEntries compact list-format support: kebab-lowercase name gate (closes the prose-bullet false-positive class), path-suffix strip contract (skillPath always derived as skills/<name>/SKILL.md so routing-eval and the manifest check don't drift), multi-trigger fan-out plus checkResolvable downstream dedupe, the 238 FAILs to 0 OpenClaw headline, the two integration fixtures pinning the regression, and the docs/guides/scaling-skills.md pointer for the tutorial context. Regenerate llms-full.txt to match (CLAUDE.md edit chaser, per the CLAUDE.md own rule about test/build-llms.test.ts catching drift). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-25 13:58:26 -07:00
Garry Tan	e227965024	v0.36.5.0 feat: secure DATABASE_URL access for shell jobs (inherit: ["database_url"]) (#1192 ) * v0.36.5.0 feat: secure DATABASE_URL access for shell jobs (inherit: ["database_url"]) Replaces PR #1137's plaintext-config / plaintext-env workarounds with code. Shell-job params gain `inherit: ["database_url"]`, validated pre-enqueue in both the CLI (`gbrain jobs submit`) and `submit_job` MCP op handler. Worker resolves the value from its own loadConfig() at child-spawn time; the persisted `minion_jobs.data` row stores only the name. Plain `env: { GBRAIN_DATABASE_URL: ... }` / `env: { DATABASE_URL: ... }` / `env: { GBRAIN_DIRECT_DATABASE_URL: ... }` are rejected pre-enqueue with a paste-ready hint pointing at `inherit:`. Codex pre-landing review caught two bypasses + one missing shadow name: - H1: cmd/argv inline-secret regex scan (cmd:"GBRAIN_DATABASE_URL=... gbrain sync" was a clean bypass — fixed) - H3: GBRAIN_DIRECT_DATABASE_URL added to shadowKeys - H2: honest docs about output-side leakage (stdout_tail/stderr_tail can still carry the value if the script prints it; that's the script author's responsibility, not gbrain's) Also: gbrain doctor learns home_dir_in_worktree (warns when ~/.gbrain lives inside a git worktree); ~/.gbrain/.gitignore retroactive via saveConfig + post-upgrade. New canonical guide: docs/guides/agent-to-gbrain.md (two-domain framing for downstream agent authors: MCP ops via OAuth vs localOnly admin ops via shell-job inherit:). Closes #1137. Tests: +53 new (21 validator + 12 inherit-record + 6 ensureGitignore + 5 doctor + 2 PGLite E2E + 7 codex-driven H1/H3 cases). Credit: @wintermute filed PR #1137 which made the env-stripping gap visible enough to fix in code. Thank you. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 redesign: free-form inherit:, drop closed enum User feedback: "agent spawning minions should have agency to do what it wants with secrets and pass only the ones that it needs. don't be a security nazi please." Replaces the closed INHERITABLE enum (database_url only) with three small helpers in shell-inherit.ts: - INHERIT_NAME_RE: snake_case shape guard. Rejects __proto__, leading underscore, uppercase, path-traversal. Prototype-pollution defense. - deriveEnvKey(name): config-key → child-env-key. Uppercase by default with one override: database_url → GBRAIN_DATABASE_URL. - resolveInheritValue(cfg, name): value lookup with Object.hasOwn. inherit: now accepts any snake_case config-key the worker has. Agent picks what it needs per-job (database_url, anthropic_api_key, voyage_api_key, or any custom field). Validator does NOT police WHICH keys — single-uid trust model treats agent as peer of worker. Drops the v0.36.5.0-RC rules that were paternalistic for the actual threat model: - closed-enum check - env-shadow rejection - cmd/argv inline-secret scan Keeps the parts that defend real problems: - pre-enqueue validation (closes the persistence-before-throw window) - snake_case regex (prototype-pollution + audit-log readability) - fail-fast on missing config value (UX guardrail, not security) Tests: shell-validate (existing rules + new free-form + prototype-pollution defense + T1 regression guard) and shell-inherit (regex matrix, deriveEnvKey per-name, resolveInheritValue with hasOwn defense). E2E case now exercises inherit:["anthropic_api_key"] to prove genuinely free-form. Docs and CHANGELOG rewritten to reflect the open design + the design-arc story (closed → cut → free-form). Migration file too. 7653 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 add: redact_secrets opt-in for stdout/stderr scrubbing Honest defense for the documented output-side leakage. When a script prints an inherited secret, the value lands plaintext in result.stdout_tail / result.stderr_tail / error_text. v0.36.5.0 adds: - `redact_secrets: true` ShellJobParams field - `--redact-secrets` CLI convenience flag on `gbrain jobs submit shell` - shell-redact.ts: pure `redactSecretsInText(text, secrets)` helper (string-mode replaceAll; regex metachars in values stay literal) - Handler post-processes both tails before throw/return, so the persisted row carries `<REDACTED:name>` tokens instead of values Only inherit-resolved values are scrubbed. env: values are not (those are the agent's "fine in the row" channel by design). Heuristic — defeats accidental `echo "$GBRAIN_DATABASE_URL"`, not adversarial encode-then-print. Default false for back-compat. Tests: - test/minions-shell-redact.test.ts (9 cases): pure-function behavior, regex-metachar safety, multi-secret independent redaction, substring overlap, empty-input/map edge cases - test/minions-shell-validate.test.ts: +4 cases for redact_secrets shape - test/e2e/minions-shell-pglite.test.ts: +2 cases proving redact_secrets: true scrubs persisted row AND redact_secrets:false preserves plaintext (back-compat regression guard) Docs + CHANGELOG + migration file + CLAUDE.md updated. 7667 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-19 13:12:40 -07:00
Garry Tan	03947665e4	v0.36.0.0 feat(skillpack): scaffold + reference + harvest (retire managed-block install) (#1130 ) * feat(skillpack): extract copyArtifacts shared helper (T1) Pure file-copy primitive for scaffold (gbrain→host) and harvest (host→gbrain). Atomic-refusal contract: symlink-reject + canonical-path containment validate every item before any write. Used by both directions of the v0.33 loop. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): scaffold subcommand + SKILL.md frontmatter sources (T2) New scaffold.ts replaces the managed-block installer. One-time additive copy into the user's repo via copyArtifacts; refuses to overwrite existing files (user owns them). Partial-state policy: copies missing paired sources even when the skill dir already exists. bundle.ts extended with loadSkillSources + enumerateScaffoldEntries — paired source files declared in each SKILL.md's frontmatter sources: array, not in openclaw.plugin.json. Single source of truth, co-located with the skill. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): reference command + apply-clean-hunks (T4 + T15) reference is the read-only diff lens with an agent-readable framing line. Pure-JS unified-diff producer + parser + applier (no patch(1) dependency). Two-way merge with documented limitation: without scaffold-time base tracking, applied hunks align everything to gbrain. The agent dry-runs reference first, then decides. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): migrate-fence + scrub-legacy-fence-rows (T5 + T16) migrate-fence is the one-shot transition from the pre-v0.36 managed-block model. Strips begin/end markers and the cumulative-slugs receipt comment; preserves fence rows verbatim as user-owned routing during the transition to frontmatter discovery. Receipt-then-row fallback (F-CDX-8) covers stale/missing receipts. scrub-legacy-fence-rows is the opt-in cleanup after migrate-fence. Two-condition gate: removes a row only when skills/<slug>/ exists AND that skill's frontmatter declares non-empty triggers (proof frontmatter discovery covers it). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): harvest + privacy linter (T6 + T7) The inverse loop: lift a proven skill from a host repo (~/git/wintermute, etc.) back into gbrain so other clients can scaffold it. --from <host-repo-root> is symmetric with scaffold's --workspace. Security: symlink rejection + canonical-path containment (mirrors validateUploadPath). Privacy: default-on linter scans harvested files against ~/.gbrain/harvest-private-patterns.txt plus built-in defaults (Wintermute, email, Slack channel patterns). Any match rolls back the copy and exits non-zero. --no-lint bypasses for the editorial workflow after a manual scrub. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(repo-root): cwd_walk_up tier for non-OpenClaw hosts (T9 + D3) autoDetectSkillsDir now walks up from cwd looking for any skills/ directory, ahead of the implicit ~/.openclaw/workspace fallback. cd ~/git/wintermute && gbrain skillpack scaffold ... finds wintermute automatically without requiring a RESOLVER.md/AGENTS.md to exist yet. R5 regression preserved: $OPENCLAW_WORKSPACE still wins when explicitly set. +5 test cases in test/repo-root.test.ts pin the new tier order and the R5 guard. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): rewrite CLI dispatch, drop install + uninstall (T3 + T10) skillpack.ts dispatcher rewritten for the v0.36 contract: scaffold, reference (+ --apply-clean-hunks), migrate-fence, scrub-legacy-fence-rows, harvest, plus the existing list / diff / check. install and uninstall are gone — both exit non-zero with a hint pointing at scaffold / migrate-fence. Clean break, no deprecated alias. skillpack-check gains --strict for CI gating. When invoked as the subcommand `gbrain skillpack check`, default is informational (exit 0 even with drift); --strict opts back into the cron-friendly exit-1-on-issues behavior. Top-level gbrain skillpack-check preserves its existing exit semantics for backwards compat. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skills): skillpack-harvest editorial workflow + resolver wiring (T8) The companion editorial skill for the gbrain skillpack harvest CLI. Walks the genericization checklist (scrub fork names, generalize triggers, lift fork- specific conventions to references) before the CLI runs. Routing-eval fixtures use paraphrased intents to avoid the intent_copies_trigger lint. Wires the new slug into openclaw.plugin.json#skills, skills/manifest.json, and skills/RESOLVER.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(skillpack): 9-case real-subprocess E2E flow (T11) Spawns gbrain as a subprocess against tempdir workspaces. Covers: scaffold first-run + re-run no-op, reference diff + --apply-clean-hunks, migrate-fence, scrub-legacy-fence-rows, harvest privacy-lint catch + --no-lint bypass, and the install removed-error path. No DATABASE_URL needed — skillpack is filesystem-only. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: docs + VERSION + CHANGELOG for v0.36.0.0 (T13 + T14) Skillpacks as scaffolding, not amber. v0.36 retires the managed-block install model. Six new subcommands replace install + uninstall: scaffold, reference (with --apply-clean-hunks), migrate-fence, scrub-legacy-fence-rows, harvest, plus the existing list / diff / check (check gains --strict for CI gating). Routing comes from each skill's frontmatter triggers — gbrain does not touch your RESOLVER.md or AGENTS.md. Companion editorial skill skillpack-harvest drives the genericization checklist; default-on privacy linter catches Wintermute / email / Slack references before they leak into gbrain core. New docs guide at docs/guides/skillpacks-as-scaffolding.md walks the model and the migration path for pre-v0.36 installs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): privacy checks — allow-list harvest-lint tests, scrub user-facing fork-name references CI's check-privacy.sh and check-test-real-names.sh both flagged the literal fork name across the v0.36 skillpack diff. Two failure modes, two fixes: 1. Meta-rule-enforcement files added to both allow-lists. The harvest privacy linter's whole job is to catch the banned literal leaking into gbrain; its source has the regex pattern, its tests verify the linter fires by feeding it the banned string, and the skill markdown documents the substitution policy. Same exception status as check-privacy.sh and check-proposal-pii.sh themselves. Files allow-listed: - src/core/skillpack/harvest-lint.ts - test/skillpack-harvest-lint.test.ts - test/skillpack-harvest.test.ts - test/e2e/skillpack-flow.test.ts - skills/skillpack-harvest/SKILL.md 2. User-facing references swapped for canonical phrasing per CLAUDE.md's responsible-disclosure rule. README + new docs guide + 4 src docstrings + 1 test now say 'your OpenClaw' / 'host agent repo' / 'agentRepo' var name. Behavior unchanged — only documentation strings touched. Verify gate (the script CI runs) passes locally: EXIT=0. Tests still pass: 60/60 across the affected files. llms-full.txt regenerated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): update check-resolvable-cli expectation for cwd_walk_up tier Sister fix to the test/repo-root.test.ts update in commit a31418e3. The new v0.33 cwd_walk_up tier fires before repo_root when running from inside the gbrain repo — same skills/ dir matched, different source label. Behavior unchanged; the legacy repo_root tier is now functionally subsumed (kept in the type union for back-compat). CI shard 3 failure: test/check-resolvable-cli.test.ts:171. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): pin clock in sync_freshness boundary tests (CI flake) The 24h and 72h exact-boundary tests scheduled last_sync_at relative to Date.now() at construction time, then let the check call Date.now() again internally. CI scheduler jitter between the two reads pushed ageMs past the strict > thresholds by microseconds, dropping the 72h-boundary case into the fail branch instead of warn. Fix: add an optional `opts.now` test seam to checkSyncFreshness. The two boundary tests now capture t0 once and pass it both to the timestamp constructor and to the check, making ageMs deterministically equal to the boundary. The non-boundary tests (4d, 30h, 2h, etc.) don't need pinning — they're comfortably away from the > comparison. CI shard 1 flake: test/doctor.test.ts:479. Locally 48/48 doctor tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): agent-onboarding readme + next-action hints on every CLI surface (DX review) DX audit of the v0.36 scaffold model surfaced one structural gap and four output gaps. When scaffolded files land on a downstream agent's disk, the agent had no agent-facing manifest telling it what to do — no routing contract, no upgrade flow, no two-way merge warning at the right surface. Fixes: 1. New shared dep: skills/_AGENT_README.md. Lands on every scaffold + migrate-fence alongside the existing _brain-filing-rules.md and _output-rules.md. Short, agent-readable contract: walk .SKILL.md frontmatter triggers: for routing, gbrain is reference not law on upgrade, no managed-block fence anymore, two-way merge has known limitations. Single source of truth for the agent operating contract. 2. scaffold stdout* prints a next-action hint pointing at the readme (with absolute path) and the reference --all upgrade-sweep command. 3. reference stdout adds per-category decision policy: - missing → scaffold again - differs → was edit intentional? keep it. Accidental? patch by hand or apply-clean-hunks after reading the two-way warning. 4. reference --apply-clean-hunks prints the two-way merge WARNING BEFORE the apply (to stderr, survives stdout redirect). Spells out that gbrain has no scaffold-time base and local edits in differing sections WILL be aligned to gbrain. Skipped in --json mode for machine consumers. On conflicts, prints how to inspect and patch. 5. migrate-fence stdout tells the agent its routing model just changed (fence gone, walk frontmatter now) and points at scrub-legacy-fence-rows as the eventual cleanup. References the new _AGENT_README for fresh-install agents. Smoke verified end-to-end: 16 files land (was 15, +1 for _AGENT_README), hint prints with absolute path, readme lands on disk. Tests + verify gate pass clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(skillpack): upgrade-time reference sweep + reference --since version filter (DX deferred items) Closes the last two DX gaps from the v0.36 audit: 1. Post-upgrade reference sweep. New `postUpgradeReferenceSweep` helper called at the end of `gbrain post-upgrade`. After migrations apply, auto-runs `reference --all` against the detected host workspace and prints a one-line-per-skill summary of drift. Five gates: GBRAIN_SKIP_REFERENCE_SWEEP env-var bypass, no detected workspace (silent), workspace IS gbrain repo (dev-mode silent), zero drift (silent), and pure-missing skills the host never scaffolded are filtered out as noise. All errors swallowed — never blocks post-upgrade. Helper accepts test-seam opts (gbrainRoot, targetWorkspace) for unit testability. 2. `reference --all --since <version>`. Filters the sweep to skills whose source actually changed in gbrain between <version> and HEAD, using a new `changedSlugsSinceVersion` helper in bundle.ts. Pure-JS git wrapper (spawnSync), no deps. Accepts bare '0.X.Y.Z' or 'v0.X.Y.Z' or commit SHA. Falls back loudly to full sweep when git can't resolve the ref (tarball install, missing tag). Test coverage added — total +32 new test cases: UNIT (15 cases): - test/skillpack-changed-since-version.test.ts (9 cases): git-aware filter against a fixture git repo. Covers null on non-repo, null on bad tag, empty array on no changes, single + multi-slug drift (deduped + sorted), bare + v-prefix version forms, non- skills/ path filtering, SHA-prefix ref form. - test/upgrade-reference-sweep.test.ts (6 cases): gate logic. Covers env-var bypass, zero drift, empty-host suppression, drift-detected output shape, dev-mode workspace==gbrain guard, error-swallowing contract. E2E (8 new cases in test/e2e/skillpack-flow.test.ts): - 10: scaffold lands skills/_AGENT_README.md - 11: scaffold stdout prints the Next: hint - 12: scaffold re-run (skipped-existing) suppresses the hint - 13: reference stdout prints per-category decision policy - 14: --apply-clean-hunks WARNING on stderr, not stdout - 15: --apply-clean-hunks --json suppresses the WARNING (bug fix surfaced here: code originally printed unconditionally, now gated on !json) - 16: migrate-fence stdout points at the new routing model - 17: --since with a bad tag falls back to full sweep with warn Local sweep: 579/579 pass across 18 affected test files, verify gate EXIT=0, llms regenerated. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(README): zero-base rewrite — 921 → 422 lines, refreshed catalog, MECE structure The README had drifted into a changelog dumping ground. Four 'New in vX.Y' paragraphs competed for the lead, 16 version tags scattered through headings, the production-numbers hook (17,888 pages, 4,383 people) was six months stale, and skills were described in three places (Skills section, Commands section, inline marketing prose). Zero-based rewrite: Refreshed catalog (surveyed live brain + live agent fork, broad strokes per CLAUDE.md privacy rules): - ~100K total brain items (was 17,888 in the old README — 6x stale) - ~16K people (was 4,383) - ~5K companies (was 723) - ~8K concepts, ~4K originals, ~3.5K daily notes - ~31K media (30K tweets, 179 books, papers/films/games/interviews) - 108 cron jobs running (was 21) - 273 skills in the live agent fork (35 bundled + 238 user-built) Structure — MECE, single source of truth per concept: 1. Hook + at-a-glance table (refreshed numbers) 2. Install (3 paths, terse) 3. What it does (5 capability areas — replaces 12 scattered sections) 4. Skills (categorized one-liners — 35 lines, was ~200) 5. How it works (one coherent flow — replaces 4 overlapping sections: Architecture, Knowledge Model, Knowledge Graph, Search, Why It Works) 6. Commands (terse cheatsheet — every command, one line each) 7. Docs (link map — points to docs/ for the heavy stuff) 8. Origin / Contributing / License Cut entirely (moved or deleted): - 4 'New in vX.Y' leads (→ CHANGELOG.md is the changelog) - 16 (vX.Y) version tags in section headings - Minions stats subsection (subsumed into hook + 'durable background work') - Voice section (was 12 lines of brand prose) - Engine Architecture detail (→ docs/architecture/) - File Storage section (→ docs/guides/storage-tiering.md) - Per-skill marketing prose (one-liner per skill in the table) The README is no longer the changelog. Future releases append to CHANGELOG.md; the README only changes when a structural capability does. llms-full.txt regenerated. Privacy check + verify gate pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(README): fix line-start '+' rendering bug + lead with eval evidence Two fixes in one: 1. Markdown bug fix. The OAuth 2.1 paragraph had `+ PKCE,` on a line start (column 1), which GitHub-flavored markdown interprets as a list marker — the line break before it broke the paragraph and rendered as an orphan first line followed by a bullet. Rewrote the OAuth 2.1 capabilities as inline-comma-separated, escaped the `+` semantics. Swept the whole file for the same bug class — no other instances. 2. Maximum-sell mode for evals. Surveyed every published benchmark in both this repo and ~/git/gbrain-evals. Strongest evidence pulled to the top: - 97.60% R@5 on the public LongMemEval _s (500 questions). No LLM in the retrieval loop. $0.50 per 1000 queries. Beats MemPalace raw by a point on the same dataset, beats every academic dense retriever (Stella, Contriever, BM25). Mastra/Supermemory measure a different metric (QA accuracy with LLM judge) — flagged honestly. - +31.4 points P@5 from the self-wiring knowledge graph on BrainBench v0.20.0 (240-page rich-prose corpus, 145 relational gold queries). Separable, measured, load-bearing. Zero retrieval regression across seven releases (v0.16 → v0.20). New '## Benchmarks' section after Install: - Public benchmark table with cross-system comparison - In-house BrainBench scorecard with per-adapter Δ vs gbrain - Source-swamp resistance result (93.3% top-1 vs 80% grep-only) - Skill/prompt compression: 25KB → 13KB AGENTS.md, +13-17pp accuracy across Opus 4.7 / Sonnet 4.6 / Haiku 4.5 - 'Run your own evals' subsection with copy-pasteable commands for every eval surface (longmemeval, cross-modal, eval capture/replay, BrainBench) Tightened the lead's cost-comparison claim to what's defensible per the underlying eval doc (MemPal LLM-rerank $0.001/q vs gbrain $0.0005/q; dropped the overstated '6x' I'd written initially). Privacy + verify gate + build-llms test all pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(README): integrate the eval story into the lead, move jargon into 'Receipts on the evals' Previous lead dumped metric acronyms (R@5, P@5, P@5 deltas, MemPalace, Stella, Contriever, BM25) before the reader knew what gbrain does. A 'somewhat technical' reader hits the wall of jargon and bounces. Rewritten: Lead (jargon-free, 3 paragraphs) — describes the value in plain English, with two anchor numbers: - 'right answer in top 5 results 97.6% of the time' (not 'R@5 97.60%') - 'roughly 4x more relevant than plain vector RAG' (not '+31.4 pts P@5') - 'better than every comparable system that doesn't pay for a language- model call on every retrieval' (the load-bearing honest framing, without naming the competitors mid-hook) - ends with '[Receipts on the evals →]' linking down '## Benchmarks' renamed '## Receipts on the evals' with a glossary at the top defining R@5, P@5, and 'no LLM in the loop' in one line each. Then the full tables: LongMemEval cross-system (with the metric-mismatch flag for Mastra/Supermemory), in-house BrainBench scorecard, source-swamp resistance, and prompt compression. The competitor names + metrics stay here where readers who want the receipts can find them, with the glossary so the acronyms don't tax cold readers. Net: lead reads as 'here's what it does and the proof' instead of 'here are the benchmark numbers, figure out what they mean.' Comparison facts unchanged. Privacy + verify gate + build-llms test all pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(README): name LongMemEval explicitly + first-person voice in lead Two specific edits from user feedback: 1. 'the standard public benchmark for AI memory systems' → 'LongMemEval' (linked to the HuggingFace dataset). The benchmark has a name; use it. 2. 'Built by the President and CEO of Y Combinator to run his own AI agents' (passive third-person) → 'I'm the President and CEO of Y Combinator, and I use this 16 hours a day' (active first-person). Carried the voice change through the rest of the README — the downstream 'Garry's personal agent' line and the Origin section's 'Garry Tan needed... he'd ever drafted... so he built one' all flip to first person ('my personal agent', 'I needed', 'I'd ever drafted', 'so I built one'). The README is now consistently first-person from the author's voice instead of a hagiographic third-person framing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(README): add Multi-player and company brains section Three deployment patterns documented: 1. Single GBrain server + thin MCP clients (recommended). Tailscale private networking, OAuth scope, source-scoped clients, exhaustive what-clients-can/cannot-do lists. 2. Local PGLite + GStack for per-worktree code search. 3. Federated repos (advanced) — multiple servers indexing the same brain repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(README): tighten install path + add tech-orientation block + visceral query example Self-eval as a cold reader surfaced four gaps blocking a 10/10 first read: 1. Lead never says WHAT it is technically — CLI? service? cloud? local? Added a "What it is, technically" block right after the hook: open-source MIT, Bun CLI + MCP server, local-first, data stays on disk, MCP-native. 2. Install path optimized for committed users not evaluators. The old "recommended" path (deploy OpenClaw on Render, 8GB RAM) blocked anyone trying gbrain for the first time. Reordered into 3 paths by commitment: 60-second standalone CLI first, MCP for Claude Code / Cursor second, full agentic install third. 3. No example output showing what success looks like. Added a real sample `gbrain query` invocation with the hybrid-search result format so a reader can feel the experience before they install. 4. Privacy / data-locality unaddressed in lead. Now stated up front: embedding calls only hit external APIs if you configure them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-18 12:53:01 -07:00
garrytan-agents	9c2dc4cd54	v0.26.8 feat(migration): v35 auto-RLS event trigger — new tables always secure (#612 ) * feat(migration): v35 auto-RLS event trigger — new tables always secure Postgres event trigger that fires on every CREATE TABLE and auto-enables Row Level Security. Prevents the face_detections bug: tables created outside gbrain migrations (Baku, manual SQL, other apps sharing the same Supabase project) were silently unprotected until gbrain doctor caught it. This is the Supabase-recommended approach — no dashboard toggle exists. Migration v35 (auto_rls_event_trigger): - CREATE FUNCTION auto_enable_rls() — event trigger handler - CREATE EVENT TRIGGER auto_rls_on_create_table — fires on ddl_command_end - PGLite: no-op (no RLS engine, no event triggers) Tests (3 cases): - Event trigger exists after migration - New table automatically gets RLS enabled - auto_enable_rls function exists Closes the gap identified in production on 2026-05-04 when face_detections was found without RLS. * feat(migration): v35 — drop FORCE, public-only, bundle backfill, cover CTAS+SELECT INTO Apply the corrections surfaced by /plan-eng-review + /codex consult against the original PR #612. The trigger now matches v24/v29/schema.sql posture (ENABLE only, no FORCE), scopes to the public schema, and covers all three table-creation syntaxes Postgres reports. Bundles a one-time backfill of every existing public.* table without RLS, honoring doctor.ts's GBRAIN:RLS_EXEMPT regex and quoting identifiers via format('%I.%I'). Drops the EXCEPTION wrap inside the trigger so per-table failures abort the offending CREATE TABLE (loud rollback) rather than producing a silent permissive default. Drops the hand-rolled privilege pre-check — the runner already fails loud on permission errors and gates the version bump. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment before upgrading or the backfill will flip them on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-04 17:31:32 -07:00
Garry Tan	d838d4792b	feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency (#379 ) * feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard Prevents stall-induced queue blockage discovered in production (OpenClaw): 1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms (or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB connections where FOR UPDATE SKIP LOCKED stall detection skips them. 2. Submission backpressure (maxWaiting): caps waiting jobs per name at submission time. Prevents autopilot-cycle flood when the queue is blocked. 3. --no-worker flag for autopilot: skips spawning the built-in worker child. For environments where the worker lifecycle is managed externally (systemd, Docker, OpenClaw service-manager). 4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the worker is spawned by autopilot (which can't pass CLI flags to the child). 5. Shell job env guard with clear logging: shell handler is always registered but throws UnrecoverableError with a clear message when GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering. * feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI Addresses three production-hardening findings from the CEO + Eng + Codex adversarial review of PR #379: D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both see waitingCount < max and both insert. Wrap the count+select+insert in pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent decisions for the SAME key while leaving different keys fully parallel. Lock auto-releases on txn commit/rollback — no cleanup path to leak. Also fix the missing queue-scope bug: count and select now filter on (name, queue) not name alone, so cross-queue same-name jobs don't suppress each other. D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from parseInt. `inFlight.size < NaN` is always false → worker claims nothing → silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr warning naming the bad value. D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the MinionJobInput field. Wire the flag with clamp [1, 100], mirror `--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing. Q1: Silent coalesce was invisible by design. New src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce events write one JSONL line with (queue, name, waiting_count, max_waiting, returned_job_id, ts). Best-effort — disk-full never blocks submission. A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case. Forges a wedged-worker row state, invokes handleStalled + handleTimeouts + handleWallClockTimeouts in order, asserts only wall-clock evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape. Tests: 23 new unit cases in test/minions.test.ts covering wall-clock timeout (3 cases + non-interference with handleTimeouts), maxWaiting (coalesce, clamp 0, floor, concurrent-submitter race via Promise.all, cross-queue isolation, unset fallthrough), concurrency clamp (7 cases incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure audit file write. Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite has no multi-process worker surface). Two subchecks, both cheap (single SELECT each, status-index-covered): - stalled-forever: any active job with started_at > 1h. Surfaces the worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel` fix hints. The incident that motivated v0.19.1 ran 90+ min before the operator noticed. - waiting-depth: per-name waiting count exceeds threshold. Default 10, overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a submitter probably needs maxWaiting set. Worker-heartbeat subcheck from the original plan dropped (D4/H4): no minion_workers table exists, and lock_until-on-active-jobs is a lossy proxy that can't distinguish idle-worker from dead-worker. Tracked as follow-up B7. A4: --no-worker peer-liveness probe in autopilot. When --no-worker is set, every cycle runs a cheap SELECT checking for any active job whose lock_until was refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud WARNING naming the silent-wedge vector and referencing B7 as the ground-truth follow-up. Re-arms on next live signal so the warning doesn't spam every cycle. A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60 lines). "My queue looks wedged — what do I run?" in order of escalation. What each doctor subcheck means. Self-check for the --no-worker / no-worker-running footgun. CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer 3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health doctor check (v0.19.1 D4), and backpressure-audit.ts. Tests: all 143 minions + 13 doctor unit tests pass. No new test cases required in Lane B; the doctor queue_health exercise is in the E2E verification step (needs real PG to produce meaningful stalled-forever rows). The --no-worker probe is exercised by the smoke case's wedge setup in Lane A. README: unchanged. Existing `gbrain jobs submit` examples don't show --max-stalled, so no --max-waiting precedent to extend per A6 conditional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change, no new user-facing vocabulary). CHANGELOG: new v0.19.1 entry at the top with the full release-summary template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers that matter" before/after table measured against the real incident, "what this means for OpenClaw users" closer, required "To take advantage of v0.19.1" block naming the worker-restart requirement, itemized changes by area, and "For contributors" section closing the loop on the stale autopilot-idempotency narrative the CEO review was based on. Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by missing idempotency (autopilot already passes `idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The 18 jobs were 18 DIFFERENT slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before ship. SPEC.md: deleted from repo root. It was Wintermute's planning artifact for the original PR, not a shipped spec. Design docs belong under docs/designs/ per repo convention; leaving one at repo root set a precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are the durable artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip Smoke case was setting lock_until in the past, so handleStalled's requeue path fired before handleWallClockTimeouts had a chance to evict. Production scenario is "lock_until still live (worker renewing) + timeout_at disqualified" — only wall-clock matches. Single-connection smoke can't simulate a row lock held by another txn, so we force the equivalent outcome: - lock_until = now() + 30s → handleStalled skips (not a stall) - timeout_at = NULL → handleTimeouts skips (needs NOT NULL) - started_at = now() - 10s, timeout_ms=1000 → wall-clock matches (2 × timeout_ms = 2000ms threshold exceeded) Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: CI failures — shell-handler tests + llms-full.txt drift Two CI failure clusters, both pre-existing but surfaced by the v0.20.3 merge: 1) test/minions-shell.test.ts — 12 failing cases. The shell handler throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the production RCE guard at shell.ts:210). The unit tests exercise handler mechanics, not the guard, but never set the env var — so every invocation exits through the guard path instead of the code being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll, restore in afterAll. The env-guard IS still tested separately via the test/minions.test.ts case added in v0.20.3 Lane A which toggles the var itself. 2) llms-full.txt — stale against CLAUDE.md. Key-files entries for queue.ts, doctor.ts, and the new backpressure-audit.ts updated in v0.20.3 Lane B triggered the build-llms drift guard. Regenerated via `bun run build:llms`; no behavior change, just the inlined-docs bundle catching up to source. Full test run: 2367 pass, 0 fail across 137 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 01:09:28 -07:00
Garry Tan	e3f704229b	v0.20.2 feat: gbrain jobs supervisor — self-healing worker process manager (#364 ) * feat: add `gbrain jobs supervisor` — self-healing worker process manager Adds a first-class supervisor command that: - Spawns `gbrain jobs work` as a child process - Restarts on crash with exponential backoff (1s→60s cap) - Resets crash counter after 5min of stable operation - PID file locking prevents duplicate supervisors - Periodic health checks (stalled jobs, completion gaps) - Graceful shutdown (SIGTERM→35s→SIGKILL) Usage: gbrain jobs supervisor --concurrency 4 Replaces ad-hoc nohup patterns in bootstrap scripts. The autopilot command's internal supervisor can be migrated to use this in a follow-up. Tests: 7 pass (backoff calc, PID management, crash tracking) * supervisor: atomic PID lock, queue-scoped health, env safety, unified exit Lane A of PR #364 review fixes (20-item multi-lane plan). Addresses the codex-tier + CEO + Eng findings on src/core/minions/supervisor.ts: Safety + correctness: - Atomic O_CREAT\|O_EXCL PID lock via openSync('wx') with stale-file liveness check. Prevents two supervisors racing on the same PID file. (codex #1) - Health check now queries status='active' AND lock_until < now() matching queue.ts:848's authoritative stalled definition. The prior `status = 'stalled'` predicate returned zero rows forever because 'stalled' is not a persisted value in the schema. (codex #2) - All health queries scoped to WHERE queue = $1 via opts.queue binding. Multi-queue installs no longer see cross-queue false positives. (codex #3) - Class default allowShellJobs flipped true→false AND explicit `delete env.GBRAIN_ALLOW_SHELL_JOBS` when false, so child workers don't silently inherit the var from the parent shell. (eng #8, codex #9) - Unified shutdown(reason, exitCode) — max-crashes now routes through the same drain path as SIGTERM. Single source of truth for lifecycle cleanup; prerequisite for trustworthy audit events (Lane C). (eng #1) - Default PID path moves from /tmp to ~/.gbrain/supervisor.pid with mkdirSync recursive + GBRAIN_SUPERVISOR_PID_FILE env override. Matches the rest of the product's ~/.gbrain/ convention; fresh installs no longer hit ENOENT. (CEO #2 + codex #6) Refinements: - crashCount = 1 after 5-min stable-run reset (was 0, produced calculateBackoffMs(-1) = 500ms by accident). Now reads as 'first crash of a new cycle' with a clean 1s backoff. (Nit 1) - Top-of-file POSTGRES-ONLY docstring documenting why the supervisor can't run against PGLite. (Nit 2) - inBackoff flag suppresses 'worker not alive' warn during the expected null-child window (crash → sleep → next spawn). (eng #2) - Tracked listener refs for SIGTERM/SIGINT removed in shutdown() so integration tests spinning up/tearing down multiple supervisors on one process don't leak handlers. (eng #3) - Single FILTER query replaces two SELECT counts — one round-trip instead of two, three metrics in one pass. (eng #10) - child.on('error') listener emits worker_spawn_failed event for ENOENT/EACCES; exit handler still increments crashCount as usual so max-crashes bounds permanent misconfigurations. (codex #7) - healthInFlight boolean guard with try/finally prevents overlapping health checks from stacking on a hung DB. (codex #8) Documented exit codes (ExitCodes const): 0 CLEAN, 1 MAX_CRASHES, 2 LOCK_HELD, 3 PID_UNWRITABLE Agent can branch on exit=2 ('another supervisor, I'm fine') vs exit=1 ('escalate to human'). Event emitter surface: - started / worker_spawned / worker_exited / worker_spawn_failed - backoff / health_warn / health_error / max_crashes_exceeded - shutting_down / stopped Plumbed through emit() with an onEvent callback hook for Lane C's audit writer. json:false is the default; Lane C's --json mode flips it and writes JSONL to stderr. CLI changes (src/commands/jobs.ts): - `gbrain jobs supervisor` gains --allow-shell-jobs (explicit opt-in mirroring the env-var gate), --cli-path (override auto-resolution for exotic setups), and --json (JSONL lifecycle events on stderr). - Expanded --help body with description, 3 examples, and exit-code table. (DX Fix A per review) - Three-tier PID path resolution: --pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid (via exported DEFAULT_PID_FILE). - Removed the catch-fallback to process.argv[1] — resolveGbrainCliPath() throws its own actionable install-hint error, which is what dev users need instead of a cryptic spawn failure on a .ts path. (codex #5) Tests: existing 7 supervisor.test.ts cases continue to pass. Integration tests (crash-restart, max-crashes, SIGTERM-during-backoff, env-inheritance regression) land in Lane E. Out of scope for this lane (tracked in follow-up lanes): - Audit file writer at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl (Lane C) - Documentation pass (Lane B) - supervisor start/status/stop subcommands (Lane C) - gbrain doctor supervisor check (Lane D) - /ship release hygiene (Lane F) - autopilot.ts migration to MinionSupervisor (deferred to follow-up PR per codex — requires non-blocking start() API redesign, not ~30 lines) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: supervisor as canonical worker deployment pattern Lane B of PR #364 review fixes. Reframes docs/guides/minions-deployment.md around `gbrain jobs supervisor` as the default answer (blocker 7), deletes the 68-line legacy bash watchdog (F10), and updates README + deployment snippets to match. docs/guides/minions-deployment.md: - New 'Worker supervision' section at the top with the canonical 3-command agent pattern (start --detach / status --json / stop) and a documented exit-code table (0 clean, 1 max-crashes, 2 lock-held, 3 PID-unwritable). - 'Which supervisor when?' decision table: container = supervisor as PID 1, Linux VM = systemd-over-supervisor, dev laptop = bare terminal. - New 'Agent usage' section for OpenClaw / Hermes / Cursor / Codex — the 3-turn discover-start-maintain workflow that replaces shell archaeology with machine-parseable JSON events + an audit file at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl. - Demoted the 'Option 1: watchdog cron' path entirely; replaced with a straightforward upgrade migration block (stop script, remove cron line, start supervisor, verify via doctor). - Preconditions now check Postgres connectivity directly (supervisor is Postgres-only; the CLI rejects PGLite with a clear error). Snippets: - systemd.service: ExecStart now invokes `gbrain jobs supervisor` instead of raw `gbrain jobs work`. Two-layer supervision (systemd → supervisor → worker) buys automatic restart on reboot plus fast crash recovery. ReadWritePaths expanded to cover $HOME/.gbrain (supervisor PID + audit). - Procfile + fly.toml.partial: same change — platform restarts the container on host events, supervisor restarts the worker on crashes. - minion-watchdog.sh: deleted (git history retains it for anyone in an exotic deployment). Supervisor subsumes every capability it had plus atomic PID locking, structured audit events, queue-scoped health checks, and graceful drain on SIGTERM. README.md: - Added a paragraph under the Minions section pointing `gbrain jobs supervisor` as canonical, noting the --detach / status / stop surface and the audit file path, with a link to the full deployment guide. Kept `gbrain jobs work` documented for direct raw invocation but flagged 'prefer supervisor' for any long-running use. The supervisor `--help` body itself (3 examples + exit-code table in src/commands/jobs.ts) landed with Lane A — this lane finishes the discoverability story by making the supervisor findable via doc grep, README landing, and deployment-guide landing paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * supervisor: daemon-manager subcommands + JSONL audit writer Lane C of PR #364 review fixes. Adds the daemon-manager CLI surface so agents can drive `gbrain jobs supervisor` in 3 turns instead of 10, and the audit writer that makes lifecycle events inspectable across process restarts. (Blocker 8, closes DX Fix A/B/C.) New: src/core/minions/handlers/supervisor-audit.ts - writeSupervisorEvent(emission, supervisorPid) appends JSONL to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`. ISO-week rotation via a `computeSupervisorAuditFilename()` helper that mirrors `shell-audit.ts` exactly (year-boundary ISO week math, Thursday anchor, etc). - readSupervisorEvents({sinceMs}) returns parsed events from the current week's file, oldest-first, for Lane D's doctor check. Malformed lines are skipped silently (disk-full truncation is already best-effort at write time). - Reuses `resolveAuditDir()` from shell-audit.ts so the `GBRAIN_AUDIT_DIR` env var override works identically across all gbrain audit trails. src/commands/jobs.ts: supervisor subcommand dispatcher - `gbrain jobs supervisor [start] [--detach] [--json] ...` — default subcommand. Without --detach, runs foreground as before. With --detach, forks a background child (inheriting stderr so the caller can still tail JSONL events), writes a stdout payload: {"event":"started","supervisor_pid":N,"pid_file":"...","detached":true} and exits 0. Stdin/stdout on the detached child are /dev/null so the parent shell isn't held open. - `gbrain jobs supervisor status [--json]` — reads the PID file, checks liveness via `kill -0`, then reads the last 24h from the supervisor audit file to compute crashes_24h / last_start / max_crashes_exceeded. Exits 0 if running, 1 if not. JSON output is machine-parseable; human output is a 5-line ASCII report. - `gbrain jobs supervisor stop [--json]` — reads PID, sends SIGTERM, polls `kill -0` every 250ms for up to 40s (supervisor's own 35s worker-drain + 5s slack). Reports outcome: drained / timeout_40s / pid_file_missing / pid_file_corrupt / process_gone. Exit 0 on clean stop. - `--json` flag is already plumbed through to the supervisor opts from Lane A — this lane adds the onEvent audit-writer callback so every supervisor emission (started, worker_spawned, worker_exited, worker_spawn_failed, backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, stopped) lands in the JSONL file with the supervisor's PID attached. --help body updated: - Three separate usage lines (start / status / stop). - SUBCOMMANDS block with one-line summaries each. - EXIT CODES block (unchanged from Lane A, moved under SUBCOMMANDS). - EXAMPLES block updated with status --json + stop + --detach forms. Tests: existing 127 supervisor + minions tests continue to pass. Integration tests for the new subcommands + audit writer land with Lane E. Follow-up (Lane D): `gbrain doctor` will read readSupervisorEvents() from this module to surface a `supervisor` health check alongside its existing checks (DB connectivity, schema version, queue health). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doctor: add supervisor health check Lane D of PR #364 review fixes. Closes the observability loop: now that Lane C writes supervisor lifecycle events to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`, `gbrain doctor` surfaces a `supervisor` check alongside its existing health indicators. Implementation (src/commands/doctor.ts, filesystem-only block 3b-bis): - Resolves DEFAULT_PID_FILE via the same three-tier logic as the start path (--pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid). - Reads the PID file + `kill -0 <pid>` for liveness. - Calls readSupervisorEvents({sinceMs: 24h}) from the audit module to derive last_start / crashes_24h / max_crashes_exceeded. - Suppresses the check entirely when the user has never invoked the supervisor (no PID file AND no audit events) — avoids noise on installs that don't use the feature. Status thresholds: fail max_crashes_exceeded event seen in last 24h (supervisor gave up; operator needs to restart or triage) warn supervisor not running but audit shows prior use (unexpected stop — likely crash or manual kill) warn running but > 3 crashes in last 24h (supervisor recovering but worker is unstable) ok running + ≤ 3 crashes + no max_crashes event All failure paths emit a paste-ready recovery command. Read/import errors are swallowed (best-effort like the other doctor checks). Tests: all 127 supervisor + minions tests still green; 13 existing doctor tests unaffected. F3 done. All four lanes A/B/C/D are now committed; Lane E (integration tests) and Lane F (/ship v0.20.2) remain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: 4 critical integration tests for supervisor lifecycle Lane E of PR #364 review fixes (blocker 10). Fills the ~15% coverage gap flagged in the eng review by actually exercising the code paths that will break in production — crash-restart loop, max-crashes exit, SIGTERM-during-backoff, env-var inheritance — via real spawn() calls against fake shell-script workers. No mocks: real fork, real signals, real env propagation, real audit file writes. test/fixtures/supervisor-runner.ts (new, 55 lines): A standalone bun script that constructs a MinionSupervisor from env vars (SUP_PID_FILE / SUP_CLI_PATH / SUP_MAX_CRASHES / SUP_BACKOFF_FLOOR_MS / SUP_HEALTH_INTERVAL_MS / SUP_ALLOW_SHELL_JOBS / SUP_AUDIT_DIR) and calls start(). Mock engine returns empty rows for executeRaw (health check path still exercised without Postgres). Tests spawn this as a subprocess because MinionSupervisor.start() calls process.exit() on shutdown — can't run it in the test runner's own process. test/supervisor.test.ts (existing; 91 → 300 lines): - Added IntegrationHarness helper: creates a unique tmpdir per test, a fake worker shell script, a PID-file path, and an audit-dir path; cleanup runs in finally. - spawnSupervisor() forks bun on the runner with env vars set. - readAudit() reads the supervisor-YYYY-Www.jsonl file via the existing readSupervisorEvents() helper (Lane C), threading GBRAIN_AUDIT_DIR through so tests don't collide on ~/.gbrain. - waitFor(pred, timeoutMs) polls helper for event-driven tests. Four integration tests (with _backoffFloorMs=5 for <1s suite runs): 1. "respawns the worker after a crash and eventually exits with max-crashes code=1" Worker always `exit 1`. maxCrashes=3. Asserts: exit code 1, PID file cleaned up, audit contains started + 3x worker_spawned + 3x worker_exited + max_crashes_exceeded + shutting_down + stopped, and the stopped event carries {reason:'max_crashes', exit_code:1}. Locks in blockers 1 (PID lock), 2+3+6 (health SQL doesn't 500), 5 (unified shutdown emits right events), F8 (spawn errors counted). 2. "receives SIGTERM while sleeping between crashes and exits 0 cleanly" Worker always `exit 1`, backoff floor 800ms to catch the sleep. Asserts: SIGTERM during backoff → exit code 0 (not 1) in <5s, no signal kill (process.exit via shutdown), audit contains shutting_down {reason:'SIGTERM'} + stopped, PID file cleaned up. Locks in eng Issue 1 (unified exit path), eng Issue 3 (signal handlers don't accumulate across shutdowns). 3. "strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false, even if parent has it set" ⚠ CRITICAL regression test Parent env has GBRAIN_ALLOW_SHELL_JOBS=1. SUP_ALLOW_SHELL_JOBS=0. Worker writes $GBRAIN_ALLOW_SHELL_JOBS (or 'UNSET' if absent) to an OUT_FILE. Asserts child sees 'UNSET'. Locks in codex #9 + eng #8: the `else delete env.GBRAIN_ALLOW_SHELL_JOBS` branch from Lane A is load-bearing for the supervisor's security posture; this test prevents a future refactor silently re-opening the inheritance hole. 4. "DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs=true" Positive-path companion to #3. SUP_ALLOW_SHELL_JOBS=1 → worker sees '1'. Confirms the else-branch doesn't over-strip and that operators who explicitly opt in still get shell-exec enabled. Plus two audit-format unit tests: - computeSupervisorAuditFilename format (regex match) - Year-boundary ISO week: 2027-01-01 → supervisor-2026-W53.jsonl (matches the shell-audit.ts pattern exactly) Before: 7 tests covering backoff math + PID helpers (~15% behavioral coverage per eng review). After: 13 tests across all critical lifecycle paths (crash-restart, max-crashes, SIGTERM, env-inheritance, audit rotation). All 146 tests in supervisor + minions + doctor suites green in ~8s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.20.2) Lane F of PR #364 review fixes. Closes the multi-lane plan with release hygiene: VERSION bump 0.19.0 → 0.20.2, package.json sync, CHANGELOG entry in GStack voice with release summary + "numbers that matter" table + "To take advantage of v0.20.2" migration block + itemized changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: escape template-literal interpolation in supervisor --help The --help body in src/commands/jobs.ts is one big backtick template literal. The supervisor subcommand description I added in Lane B used both `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}` (parsed as a template interpolation into an undefined variable) and inline `code` backticks (parsed as nested template literals). CI caught it with ~200 tsc parse errors across the file. Fix: - Escape `${...}` → `\${...}` so the audit-file path renders literally. - Replace prose inline-code backticks with plain single-quote fences (`gbrain jobs work` → 'gbrain jobs work', `~/.gbrain/supervisor.pid` → ~/.gbrain/supervisor.pid). `--help` output is human prose; the single-quote form reads cleanly in a terminal without needing to smuggle nested backticks through a template literal. `bunx tsc --noEmit` is clean. 146 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms-full.txt after Lane B doc rewrite CI drift guard caught that `llms-full.txt` didn't match the current generator output. Root cause: the Lane B rewrite of `docs/guides/minions-deployment.md` (supervisor as canonical, watchdog deleted) changed content that gets inlined into `llms-full.txt`, but I didn't run `bun run build:llms` to regenerate. `bun test test/build-llms.test.ts` now clean (7/7 pass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 00:24:10 -07:00
Garry Tan	275158137a	fix: v0.18.1 — RLS hardening + schema backfill (supersedes #336 ) (#343 ) * fix(doctor): check ALL public tables for RLS, not just gbrain's own The RLS check was hardcoded to only verify 10 gbrain-managed tables: pages, content_chunks, links, tags, raw_data, page_versions, timeline_entries, ingest_log, config, files. Any other table in the public schema (created by the application, extensions, or manually) was invisible to the check. This allowed 12 tables to exist without RLS for months — publicly readable by anyone with the Supabase anon key. Changes: - Query ALL tables in public schema, not a hardcoded list - Upgrade severity from 'warn' to 'fail' — missing RLS is a security issue, not a suggestion - Include table count in success message for visibility - Include remediation SQL in failure message Supabase exposes the public schema via PostgREST. Any table without RLS is readable/writable by the anon key by default. * fix(schema): enable RLS on 10 gbrain-managed public tables The base schema and prior migrations shipped 10 public tables without Row Level Security enabled: access_tokens, mcp_request_log, minion_inbox, minion_attachments, subagent_messages, subagent_tool_executions, subagent_rate_leases, gbrain_cycle_locks, budget_ledger, budget_reservations. Supabase exposes the public schema via PostgREST, so tables without RLS are readable and writable by anyone holding the anon key. access_tokens and the subagent conversation history tables carry the most sensitive data in the set. Fix: add the missing ENABLE RLS statements to src/schema.sql (inside the existing BYPASSRLS-gated DO block, so dev sessions without bypass don't get locked out). Add a new schema migration v17 rls_backfill_missing_tables that does the same on existing brains. budget_ledger and budget_reservations were previously migration-only (v12); promoted to the base schema so fresh installs pick up RLS from the standard gate. Regenerated src/core/schema-embedded.ts. * fix(doctor): widen RLS check to all public tables, add GBRAIN:RLS_EXEMPT escape hatch The RLS check was hardcoded to 10 gbrain-managed tables; any other table in the public schema (plugin-created, user-created, extension- created) was invisible to the check. Widen the scan to every pg_tables row in the public schema. Upgrade severity warn to fail. Missing RLS is a security issue, not a suggestion. gbrain doctor now exits 1 when any public table lacks RLS. Cron and CI wrappers that call gbrain doctor should be aware of the exit-code flip. Add an explicit escape hatch for tables that should stay readable by the anon key on purpose (analytics, public materialized views, plugin tables). The doctor reads pg_description for each non-RLS table and treats a comment matching GBRAIN:RLS_EXEMPT reason=<why> as an intentional exemption. Doctor enumerates exempt tables by name on every successful run so they never go invisible. There is no gbrain rls-exempt CLI subcommand by design. The escape hatch is deliberately painful: operators drop to psql and type the justification as raw SQL. Comment lives in pg_description, survives pg_dump, shows up in schema diffs, and appears in shell history. PGLite is now explicitly skipped with an ok status (embedded and single-user, no PostgREST exposure). Previously hit the db.getConnection() throw-catch path and surfaced a misleading warn. Remediation SQL now quotes identifiers (ALTER TABLE "public"."<name>" ...) so it works on tables with hyphens, reserved words, or mixed case. See docs/guides/rls-and-you.md for the full user-facing guide. * test: coverage for RLS hardening (doctor + migration + e2e) Four layers of guard for the v0.18 RLS changes: test/doctor.test.ts: source-grep structural regression guards on the doctor RLS block — absence of the old tablename IN filter, presence of status=fail on the gap branch, quoted-identifier remediation SQL, PGLite skip wrapper, GBRAIN:RLS_EXEMPT parsing with required reason=. Fast, no DB needed. Mirrors the statement_timeout regression pattern in test/postgres-engine.test.ts. test/migrate.test.ts: structural guard for migration v17. Asserts the migration exists with the expected name, all 10 ALTER TABLE statements are present, BYPASSRLS gating is in place, and LATEST_VERSION has caught up. test/e2e/mechanical.test.ts: rewrote the E2E RLS Verification block. The old hardcoded-allowlist query is replaced with an every-public-table-has-RLS assertion. Four new CLI-spawn cases verify real end-to-end behavior: (a) no-RLS public table makes gbrain doctor --json return status=fail with ALTER TABLE in the message and exit code 1, (b) a GBRAIN:RLS_EXEMPT comment with a valid reason makes doctor report the table as explicitly exempt and keep status=ok, (c) a GBRAIN:RLS_EXEMPT prefix without a reason= segment still fails doctor, (d) an unrelated comment on a no-RLS table still fails doctor. All helpers use try/finally with unique-per-run suffixes (gbrain_rls_..._<pid>_<timestamp>) so assertion failures don't pollute subsequent tests. * docs: one-page guide for RLS and GBRAIN:RLS_EXEMPT escape hatch Covers why RLS matters on Supabase (PostgREST exposes the public schema to the anon key), what to do when gbrain doctor fails, the exact SQL template for an intentional exemption, how to audit exemptions later, and how the check behaves on PGLite vs self-hosted Postgres. Emphasizes that the escape hatch is deliberately painful on purpose: there is no gbrain rls-exempt CLI subcommand and no config-file allowlist. The operator drops to psql and writes the justification in SQL, which makes the action visible in shell history, pg_dump, schema diffs, and doctor output on every run. Referenced from gbrain doctor's failure message when any public table lacks RLS. * chore: bump version and changelog (v0.18.0) Reconciles VERSION and package.json (were drifting: 0.17.0 vs 0.16.4). Runtime gbrain --version reads from package.json via src/version.ts, so prior ships were reporting 0.16.4. Both now land on 0.18.0. Minor bump (not patch) because gbrain doctor's exit code semantics change: missing RLS on a public table was warn+exit-0, is now fail+exit-1. Any external cron, CI, or skillpack-check wrapper around gbrain doctor needs to be aware. skillpack-check.ts itself is unaffected (uses --fast, skips DB checks). CHANGELOG entry follows the release-summary format from CLAUDE.md: headline, lead paragraph, numbers-that-matter table, what-this- means-for-your-workflow, To take advantage of v0.18.0 block with remediation SQL + exemption format, itemized changes. Also sweeps a stale @Wintermute reference in the 0.17.0 entry to "Garry's OpenClaw" per the CLAUDE.md privacy rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.18.1): address codex review (orchestrator wiring + fail-closed + identifier escape) Four fixes from `/codex` review of the merged diff: 1. HIGH — wire migration v24 into the `gbrain apply-migrations` upgrade path. Without an orchestrator entry, `gbrain upgrade`'s post-upgrade step runs `apply-migrations --yes`, which walks the registry in `src/commands/migrations/index.ts`. The registry stopped at v0_18_0, so v24 never fired on upgrade (connectEngine and doctor do not call initSchema). New `v0_18_1.ts` orchestrator mirrors v0.18.0's Phase A: shells out to `gbrain init --migrate-only`, which triggers initSchema → runMigrations → v24 applies. Registered in the migrations array. 2. HIGH — fail loudly when v24 runs under a non-BYPASSRLS role instead of RAISE WARNING-then-silently-bumping-version. The runner at migrate.ts:773 unconditionally calls `setConfig('version', String(m.version))` when a migration completes without throwing, so a WARNING-and-continue path would permanently lock the backfill out: schema_version=24 on the next run means `m.version > current` is false and v24 is skipped forever, even after the role gets BYPASSRLS. Changed `RAISE WARNING` → `RAISE EXCEPTION` so the transaction aborts, schema_version stays at 23, and a subsequent initSchema retries cleanly after the role is fixed. Test asserts the SQL uses EXCEPTION and does not use WARNING. 3. MEDIUM — escape double-quote characters in the remediation SQL output. doctor.ts was building `ALTER TABLE "public"."${n}"` with `n` un-escaped, so a pathological table name containing a literal `"` would break out of the quoted identifier and produce invalid copy-paste SQL. Double the `"` before interpolating, matching Postgres quoted-identifier escaping rules. Extremely rare in practice, cheap to get right. 4. LOW — CHANGELOG cleanup: corrected the upgrade-behavior claim (v24 runs via `apply-migrations --yes` through the new orchestrator, not during `gbrain doctor`) and split the "tables with RLS" row into two metrics (21 base-schema tables + 2 migration-only budget_* tables = 23 managed total, all covered). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add v0.18.1 to apply-migrations skippedFuture expectations CI-only failure: test/apply-migrations.test.ts hardcodes the orchestrator-migration version list in two `skippedFuture` expectations. The v0.18.1 orchestrator I added in the prior commit pushed the list to 8 entries. Both assertions now include 0.18.1 at the tail. Caught by the gbrain CI run on the merged branch — locally the rest of the unit suite (dream/orphans) is flaky due to unrelated PGLite parallelism, but `bun test test/apply-migrations.test.ts` now passes 18/18. CI should follow. * docs: scrub v0.18.1 CHANGELOG — remove specific-table attack surface Responsible-disclosure pass on the public-facing release notes. The prior CHANGELOG entry enumerated which gbrain-managed public tables had shipped without RLS and highlighted the most sensitive ones by name. That gives anyone reading the CHANGELOG a directed probe list for unpatched Supabase installs before operators have had a chance to run `gbrain upgrade`. Rewritten to describe the change at a functional level (what doctor does now, what the upgrade path does, what the escape hatch is) without naming the specific tables or quantifying the gap. The actual SQL remains in the binary — anyone reverse-engineering can find it there — but we shouldn't put it on the release page with a banner. User-facing content kept intact: the "To take advantage of" block, the upgrade commands, the exemption SQL template, the breaking exit-code note. * docs(CLAUDE.md): add responsible-disclosure rule for release notes Prior incident on this branch: the original v0.18.1 CHANGELOG entry enumerated the specific public tables that had shipped without RLS, quantified the exposure duration, and highlighted the most sensitive ones by name. Garry caught it. Scrubbed in ecd06a0. This directive codifies the rule so future sessions (or other agents working in this repo) don't repeat the mistake: - Describe security fixes functionally, not by attack surface. - Public artifacts (CHANGELOG, README, docs/, PR titles/bodies, commit messages, release pages) get the functional description. - Private artifacts (plan files under ~/.claude/plans/ or ~/.gstack/projects/) keep the detailed before/after tables. - Source code will disclose the specifics to reverse engineers anyway — that's intrinsic. The concern is the broadcast-channel asymmetry of a release page. Also added a corresponding feedback memory at ~/.claude/projects/.../feedback_responsible_disclosure.md so the rule carries across sessions and other projects, not just gbrain. Placed right after the existing privacy rule (scrub real names) since they share the same "public artifact hygiene" posture. * chore: regenerate llms.txt + llms-full.txt (CLAUDE.md drift) Adding the responsible-disclosure rule to CLAUDE.md in ffe340d diverged the committed llms-full.txt from the generator output. The build-llms drift-guard test caught it in CI. Regenerated. * fix(v24): guard budget_ledger + budget_reservations with IF EXISTS Garry flagged: migration v24 fires `ALTER TABLE budget_ledger ENABLE ROW LEVEL SECURITY` unconditionally. budget_ledger and budget_reservations are migration-only (v12) — not in schema.sql, not re-created on every initSchema. In the normal flow v12 runs before v24 so they exist, but two edge cases break that assumption: 1. An operator manually dropped them (budget data is regenerable from resolver call logs, so `DROP TABLE` is a reasonable cleanup move). 2. A brain was somehow running an old gbrain that lacked v12, and is only catching up now. Bare ALTER hits 42P01 (relation does not exist), aborts the transaction, and leaves schema_version at 23. On next initSchema, v24 retries and hits the same error — stuck in a loop. Fix: wrap each of the two budget ALTERs in IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' AND table_name = '<tbl>') THEN ... END IF; The other 8 tables are not guarded. schema.sql creates them idempotently on every initSchema run before migrations fire, so they are guaranteed to exist by the time v24 runs. Adding guards there would be unnecessary and make the SQL noisier. Also simplified the DECLARE/BEGIN structure: moved the non-BYPASSRLS early-exit to the top so the happy path reads cleanly without the outer IF. Tests: - test/migrate.test.ts: new assertion that both budget_* ALTERs are wrapped in information_schema.tables IF EXISTS blocks; BYPASSRLS gate assertion relaxed to match either phrasing. - Manual e2e: fresh Postgres init (v0→v24), then DROP TABLE budget_ledger + budget_reservations, reset version=23, re-run init. v24 applied cleanly, version advanced to 24, budget_* stayed dropped. Without the guard this would have errored out. * test(e2e): v24 self-heals when budget_* tables are missing Behavioral e2e proof for the IF EXISTS guard added in 2fc7780. Scenario: 1. Fresh Postgres init to v24 (setupDB in beforeAll). 2. DROP TABLE budget_ledger + budget_reservations. 3. Roll config.version back to '23'. 4. CLI-spawn `gbrain init --non-interactive` to re-trigger initSchema. 5. Assert: exit 0, no 42P01 in stderr, version advances to 24, budget_* stay dropped (since v12 doesn't re-run at current=23 > v12=12). Without the guard, step 4 hits 42P01 (relation does not exist), aborts the transaction, leaves version at 23, and the next initSchema re-runs v24 forever — an infinite retry loop. This test catches any future regression that strips the guard. Cleanup (finally block) restores budget_* with the exact migration v12 schema so downstream tests that reference these tables see the original shape. Version is restored from the pre-test snapshot. Runs with the rest of the E2E: RLS Verification block. 78/78 in test/e2e/mechanical.test.ts with the addition. --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-23 07:17:40 -07:00
Garry Tan	90c5d93fce	feat: v0.18.0 — multi-source brains (one DB, many repos, federation + dotfile resolution) (#337 ) * feat(v0.17.0 step 1/9): sources primitive — additive-only multi-source foundation Lane A of the multi-repo plan. Installs the sources table and seeds a 'default' row that inherits sync.repo_path/last_commit from existing config. This is the bisectable foundation every later step builds on; the breaking schema changes (composite UNIQUE, files FK rewrite, resolution_type, ingest_log.source_id) land with their paired code rewrites in Steps 2/4/5/7 so no single commit breaks the engine. - migration v16 (sources_table_additive) + v0_17_0 orchestrator skeleton - sort-by-version guard in runMigrations (array insertion order can never cause a later migration to skip a lower one again) - default source seeded with config '{"federated": true}' so pre-v0.17 brains keep single-namespace search semantics after upgrade - orchestrator phase B detects absence of file_migration_ledger and no-ops until Step 7 lands it - 8 new structural tests in test/migrate.test.ts (shape, idempotency, scope-guard that nothing else was smuggled into v16) - apply-migrations tests include v0.17.0 in the registered list Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 2/9): pages.source_id + composite UNIQUE (Lane B) Migration v17 adds pages.source_id with DEFAULT 'default' and swaps the global UNIQUE(slug) for composite UNIQUE(source_id, slug). Ships atomically with the engine's ON CONFLICT rewrite so the constraint swap and the code that writes under it land in the same commit — no window where the engine sees one shape and the schema has another. Minimum-surface engine change: only putPage's ON CONFLICT target needs re-targeting. Other slug-based queries work unchanged because single- source brains (the only brain shape pre-Step-5) have exactly one source 'default', so slug remains effectively unique within it. Step 5+ will surface an explicit sourceId param on putPage for cross-source sync. - migration v17 (pages_source_id_composite_unique) in src/core/migrate.ts - pages.source_id + composite UNIQUE added to schema.sql + pglite-schema.ts for fresh installs - ON CONFLICT (slug) → ON CONFLICT (source_id, slug) in both pglite-engine and postgres-engine putPage - DEFAULT 'default' closes the Codex-flagged race where an INSERT between ADD COLUMN and SET NOT NULL could leave source_id NULL - 5 new v17 structural tests (29 pass / 0 fail in migrate.test.ts) - Full suite: 1979 pass / 3 fail (same as baseline — no regressions) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 6/9): sources CLI + source-resolver (Lane C) Adds the CLI surface for multi-source management. Users can now register, list, rename, federate/unfederate, and attach-to-directory a source. The source-resolver is the shared 6-priority helper that Steps 4/5 will use when they start surfacing an explicit --source flag on sync/extract/query. Commands: gbrain sources add <id> --path <p> [--name <n>] [--federated\|--no-federated] gbrain sources list [--json] gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage] gbrain sources rename <id> <new-name> gbrain sources default <id> gbrain sources attach <id> — writes .gbrain-source in CWD gbrain sources detach gbrain sources federate <id> / unfederate <id> Resolution priority (source-resolver.ts) — highest first: 1. --source flag 2. GBRAIN_SOURCE env 3. .gbrain-source dotfile walk-up 4. longest-prefix match on registered local_path (Codex #2 fix) 5. sources.default config 6. fallback 'default' - add: validates id format (kebab-case alnum, 1-32), rejects overlapping paths (eng review §4 finding 4.1), supports federated default opt-in - remove: guards against --yes omission + refuses to remove 'default', supports --dry-run, reports cascade page count - attach/detach: matches kubectl/terraform context-pinning semantics - Throws on overlap rather than process.exit() so the CLI error wrapper reports it consistently (also makes unit testing clean) 28 new tests across sources.test.ts (dispatcher + validation + overlap guard) and source-resolver.test.ts (full 6-priority coverage including longest-prefix). Full suite: 2012 pass / 3 fail (pre-existing PGLite infra timeouts). NOT in scope for Step 6 (deferred): - import-from-github (SSRF + clone integration) - prune (retention/TTL, lands v0.18) - MCP tool-defs regen for source-scoping on read ops (Step 5) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.17.0 step 8/9): getting-started guide + migration skill + citation rule Step 8 (Lane F) documents what Steps 1+2+6 have shipped and sets up the agent-facing rules for multi-source. New files: - skills/migrations/v0.17.0.md — migration skill read by host agents after `gbrain apply-migrations`. Covers the v16+v17 chain, what's in v0.17.0 vs what lands later (v0.17.1 ACL, v0.18 sessions), and the new sources CLI surface. Cites docs/guides/multi-source-brains.md as the recipe. - docs/guides/multi-source-brains.md — getting-started for end users. Three canonical scenarios (unified wiki+gstack / purpose-separated yc-media+garrys-list / mixed), full resolution priority, federation flag semantics, command reference, and citation format. skills/brain-ops/SKILL.md — new "Cross-source citation format" section mandating `[source-id:slug]` when the brain has multiple sources. Matches the contract the /plan-devex-review DX review pinned down (DX Finding 5: surface source_id in every page payload + citation contract). Key must be sources.id (immutable), never sources.name. No behavior change — this is pure documentation for what already exists in the binary. 144 skills conformance tests still pass. NOT in this commit (deferred to later steps): - docs/guides/repo-architecture.md rewrite (lands with the full v0.17.0 PR description + release notes) - skills/_brain-filing-rules.md "which source to file into" guidance (lands with Step 5 when sync surfaces --source) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 5/9): sync --source <id> routes through sources table (Lane D) Adds the --source flag to `gbrain sync`. When set, sync reads local_path + last_commit from the matching sources(id) row instead of the global sync.repo_path / sync.last_commit config keys, and writes last_commit + last_sync_at back to the same row. Backward compat: --source omitted = pre-v0.17 behavior exactly, global config path unchanged. - SyncOpts.sourceId threaded through performSync + performFullSync - readSyncAnchor/writeSyncAnchor helpers centralize the sources-vs-config branch so every read/write goes through one decision point. Makes Step 5's later per-source sync-failures tracking a one-file change. - --source resolved via src/core/source-resolver.ts (Step 6), so any command that shell-exposes resolveSourceId gets env var + dotfile walk-up + longest-prefix for free. - Error message for missing source local_path is actionable: Source "gstack" has no local_path. Run: gbrain sources add gstack --path <path> - last_sync_at auto-updates on every last_commit advance so `gbrain sources list` shows real recency. No regression: 2012 pass / 3 fail (same as baseline). NOT in this commit (deferred per plan): - Per-source failure tracking (~/.gbrain/sources/<id>/sync-failures.jsonl) - runImport source-awareness (import.ts path — Step 5 continuation) - Partial-success semantics when walking N sources — single-source flow today, multi-walk lands when the top-level `gbrain sync` without --source starts iterating all sources. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 4/9): qualified [[source:slug]] + links.resolution_type (Lane B) Adds source-pinned wikilink syntax and records the resolution kind on each edge so `gbrain extract --refresh-unqualified` (future) can re-resolve bare references when the source topology changes. Wikilink syntax extension: [[concepts/ai]] — unqualified; resolves via local-first fallback [[wiki:concepts/ai]] — qualified; target pinned to sources.id='wiki' [[gstack:projects/foo\|Display]] — qualified + display name The qualified regex runs first and masks matched spans so the unqualified pass can't double-emit. Source id format enforced to match the sources CLI validation: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])? Schema: - migration v18 adds links.resolution_type TEXT with CHECK constraint ('qualified'\|'unqualified' or NULL for legacy/manual/frontmatter edges) - schema.sql + pglite-schema.ts updated for fresh installs EntityRef type: - sourceId is OPTIONAL (only set on qualified wikilinks). Markdown [Name](path) and unqualified wikilinks omit it so strict toEqual tests pre-v0.17 keep working (69 existing tests still pass). Tests: - 5 new qualified-wikilink extraction tests + 1 migration v18 structural assertion. 75 tests in test/link-extraction.test.ts (up from 69). - Full suite: 2018 pass / 3 fail (pre-existing PGLite infra timeouts). NOT in this commit (deferred to Step 3 / Step 5 continuation): - Writing resolution_type to the DB (addLink / addLinksBatch don't carry the field yet — that's the plumb-through that lands with Step 3 when search/dedup also needs source-aware result keys). - `gbrain extract --refresh-unqualified` re-resolver. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 3/9): source-aware search dedup composite keys (Lane B) Search dedup now keys on (source_id, slug) instead of slug alone. Pre- v0.17 would collapse two same-slug pages in different sources into one, destroying cross-source recall. Codex outside-voice review flagged this as regression-critical — this commit ships the fix plus tests that lock the invariant in. Dedup pipeline (src/core/search/dedup.ts): - pageKey(r) helper — one canonical composite-key derivation. Falls back to source_id='default' for pre-v0.17 rows so single-source brains behave identically to before. - Layer 1 (dedupBySource): group-by composite key. - Layer 4 (capPerPage): count-by composite key. - guaranteeCompiledTruth: swap scoped to matching (source_id, slug), so wiki:topics/ai can't accidentally pull gstack:topics/ai's compiled_truth chunk. SearchResult type gains optional source_id — populated by SQL JOINs in both engines, falls through as 'default' for legacy callers. Engine SQL: - pglite-engine.ts + postgres-engine.ts: search SELECTs add p.source_id - rowToSearchResult (utils.ts): maps row.source_id → result.source_id when present. Shape stays backward compatible (field optional). Tests — 4 new in test/dedup.test.ts: - same-slug-different-source does NOT collapse (the critical regression guard Codex called out) - same-slug-same-source DOES still collapse (no over-correction) - missing source_id falls back to 'default' for pre-v0.17 compat - compiled_truth guarantee scopes to composite key (Codex second pass caught this specific path would leak otherwise) Full suite: 2022 pass / 3 fail (3 pre-existing PGLite infra timeouts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.17.0 step 7/9): file_migration_ledger + phase-B storage backfill (Lane E) Adds files.source_id + files.page_id + the file_migration_ledger state machine that drives storage object rewrites. Each per-file transition is its own transaction so crash-point recovery is a ledger read, not a filesystem inspection. Codex second-pass review flagged that "skip if already has source prefix" was an unsafe heuristic — the ledger replaces it with explicit state tracking. Schema: - migration v19 (files_source_id_page_id_ledger): handler-only (PGLite has no files table; Postgres-only gate). ADDs source_id + page_id to files, backfills page_id from page_slug scoped to source_id='default', creates file_migration_ledger with PK on file_id (Codex: not storage_path_old — two sources can share an old path during migration). - schema.sql updated for fresh Postgres installs; file_migration_ledger gets RLS alongside other tables. Runtime: - src/commands/migrations/v0_17_0-storage-backfill.ts: drives the ledger state machine pending → copy_done → db_updated → complete. Idempotent per row: re-running resumes from whichever state crashed. Old objects preserved (no delete) so operators can verify the soak window before a future cleanup release. - phase B in v0_17_0.ts orchestrator: wires the storage backend (Supabase/S3/local) through createStorage, runs runStorageBackfill, reports per-state counts + first-three error details. Tests — 13 new in test/storage-backfill.test.ts: - pending → copy_done → db_updated → complete happy path - 3 crash-point recovery tests (resume from copy_done, resume from db_updated, failed rows don't auto-retry) - already-complete rows are skipped with zero side effects - idempotent re-upload (exists-check skips redundant upload) - dry-run mode (no storage, reports counts without mutating) Plus 5 new migrate.test.ts assertions for v19 structure (handler- only, PGLite gate, source_id + page_id + ledger DDL, default-source backfill scope, state machine values). Full suite: 2035 pass / 3 fail (3 pre-existing PGLite infra timeouts). NOT in this commit (explicitly deferred): - DROP old page_slug column — kept for backward compat until operators have time to verify page_id everywhere. - DROP old UNIQUE(storage_path) in favor of UNIQUE(source_id, storage_path) — same reason, deferred to later cleanup. - Actual cleanup phase that deletes old objects post-soak. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.17.0 step 9/9): full multi-source PGLite integration suite (Lane G) End-to-end exercise of every v0.17.0 surface against real PGLite (in-memory, fast — no DATABASE_URL needed). The migration chain v2→v19 runs start-to-finish and the test asserts each Step's invariants hold together. 16 new integration tests across 7 describes: 1. Migration-installed state: - sources('default') exists with federated=true config - pages.source_id column has DEFAULT 'default' - composite UNIQUE (source_id, slug) is installed 2. Default-source write path: - putPage without explicit source → source_id='default' via schema default clause (no engine API change needed for single-source brains) 3. Composite UNIQUE regression guards (Codex-flagged): - Same slug in two different sources coexists - Third insert with same (source_id, slug) hits the UNIQUE constraint 4. sources CLI round-trip: - federate / unfederate flips config.federated - rename changes display, id stays immutable 5. Source resolution priority (integration): - Explicit flag > env var > fallback to default - Unregistered explicit source errors with actionable message 6. Cascade semantics: - sources remove cascades to pages; default source untouched 7. links.resolution_type (Step 4): - Qualified/unqualified values accepted - CHECK constraint rejects invalid values All 16 tests pass. Full suite: 2042 pass / 4 fail (4 pre-existing PGLite beforeEach timeouts in test/wait-for-completion, test/extract-fs, test/e2e/search-quality, test/e2e/graph-quality — count fluctuated 3-5 on baseline from variance alone). Total new tests across Steps 1-9: ~85 unit + integration tests (sources, source-resolver, migrate v16/v17/v18/v19 structural, link-extraction qualified wikilinks, dedup regression-critical, storage-backfill state machine + crash recovery, full multi-source PGLite integration). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump to v0.18.0 + CHANGELOG entry (multi-source brains) One-viewport release summary + itemized changes covering all 9 steps of the multi-source primitive. Notes the v0.17 → v0.18 version bump rationale (master shipped gbrain dream as v0.17 while this branch was in flight). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): v0_18_0 orchestrator TS narrow + mechanical test ON CONFLICT Two CI failures on PR #337: 1. tsc TS2367 at src/commands/migrations/v0_18_0.ts:190 — after the early-return on `a.status === 'failed'` (line 179), TypeScript narrows `a.status` to `'skipped' \| 'complete'`, so the subsequent `a.status === 'failed' ? 'failed' :` branch was dead code and refused to compile. Dropped the redundant check. 2. E2E `file_list LIMIT enforcement` at test/e2e/mechanical.test.ts:636 — the test pre-seeded a pages row with `ON CONFLICT (slug) DO NOTHING` but v21 swapped the global UNIQUE for `UNIQUE (source_id, slug)`, so Postgres rejects with "no unique or exclusion constraint matching". Updated the conflict target to the composite key. Tier-1 E2E had only this one failing test; everything else passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): v0.18.0 multi-source against real Postgres (v20-v23 schema + cascade + sync) Closes the three biggest confidence gaps the author flagged in the self-audit of PR #337: 1. No real Postgres E2E — PGLite has no files table, so v23's files.source_id + files.page_id rewrite + file_migration_ledger seed was NEVER executed against the real DB. This file covers it. 2. `gbrain sync --source <id>` had zero direct tests. Now has two: one that asserts performSync({sourceId}) reads local_path from the sources row (not the global config), one that asserts no-sourceId falls back to the global sync.repo_path. 3. Cascade delete coverage — previously verified only pages count after source removal. Now verifies pages + content_chunks + timeline_entries + links + files ALL cascade-delete when a source is removed. 6 describes, 16 tests total: - Schema shape (fresh install): 6 tests confirming sources('default'), pages.source_id NOT NULL with DEFAULT, composite UNIQUE pages (source_id, slug) replaces global UNIQUE(slug), links.resolution_type column + CHECK, files.source_id + page_id columns, file_migration_ledger table + status CHECK. - Composite UNIQUE semantics: 3 tests confirming same-slug in two sources coexists (Codex-critical regression guard), duplicate (source_id, slug) hits the UNIQUE, putPage targets default source by schema DEFAULT. - Cascade delete: 1 test building a fully populated source (2 pages, chunks, timeline, links, files) then removing it + asserting every dependent row is gone. - Sync routing: 2 tests confirming performSync({sourceId}) reads per-source local_path vs global config. - Sources surface: 3 tests for federate/unfederate flipping + rename preserving id. - Storage backfill: 1 end-to-end test seeding ledger + running runStorageBackfill against a stub StorageBackend, asserting pending → complete transition and files.storage_path rewrite. Gated by DATABASE_URL per CLAUDE.md E2E lifecycle. Each describe's beforeAll defensively DELETEs non-default sources + file_migration_ledger rows so reruns are hermetic (sources isn't in helpers.ALL_TABLES). Verified: 16/16 pass on first run AND second run (residual-state fix holds). Full E2E suite still green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): TS2352 in multi-source E2E — cast postgres.js RowList via unknown tsc rejects the direct `(rows as { column_name: string }[]).map(...)` cast because postgres.js RowList rows have an iterable-row shape that doesn't overlap with the plain-object target. Standard fix: cast via `unknown` first so the narrowing is explicit. Verified: `bunx tsc --noEmit` clean (ignoring the pre-existing baseUrl deprecation warning). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.18.0): addLinksBatch + addTimelineEntriesBatch source-aware JOINs Batch APIs JOINed on pages.slug globally, so two pages sharing the same slug across sources would silently fan out — addLinksBatch(['a->b']) in a brain with 'a' in both 'default' and 'alt' wrote 2 edges instead of 1. Same bug on addTimelineEntriesBatch. Fix: - LinkBatchInput + TimelineBatchInput gain optional source_id fields (from_source_id, to_source_id, origin_source_id for links; source_id for timeline). All default to 'default' so existing callers are backward-compatible on single-source brains. - pglite-engine + postgres-engine batch JOINs now composite-key on (slug, source_id). Postgres adds 3 more unnest arrays for links + 1 for timeline — still one bind per column, no 65535-param cap risk. - LEFT JOIN for origin pages also source-qualified so frontmatter- provenance edges don't cross-pollinate across sources. Regression coverage: - test/pglite-engine.test.ts: 5 new tests covering default-path isolation, explicit alt-source writes, and cross-source edges. - test/e2e/multi-source.test.ts: 4 new tests against real Postgres so postgres-js's unnest() bind path is exercised (structurally different from PGLite's). Gap #4 from the PR self-audit — latent bug, not previously reachable because every existing caller wrote to the default source only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-22 16:24:23 -07:00
Garry Tan	418d955fd3	docs: v0.16.1 — minions worker deployment guide (from #287 ) (#317 ) * docs: v0.16.1 — minions worker deployment guide (from #287) New docs/guides/minions-deployment.md covering persistent worker deploy patterns (watchdog cron, inline --follow for cron-only workloads) plus the sharp edges of running gbrain jobs work against Supabase in production. Addresses a real gap: existing minions docs (minions-fix.md, minions-shell-jobs.md) cover schema repair and shell-job security, not deploy patterns. With v0.16.0's durable agent runtime, the persistent worker is now load-bearing for subagent + subagent_aggregator handlers too, so a supervised deploy story matters. Pre-landing accuracy pass corrected five factual bugs against current source: - max_stalled column default (5, not 1 or 3) - stalled-jobs smoke-test query (active, not waiting) - watchdog SIGTERM-to-SIGKILL grace (10s minimum, not 2s) - cron env pattern (crontab env lines, not source ~/.bashrc) - --follow exit semantics (blocks until submitted job is terminal, not until queue is empty) Docs-only. No code changed. Zero migration required. Contributed by a downstream agent fork via #287. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: credit Wintermute correctly in v0.16.1 CHANGELOG Wintermute is gbrain's own OpenClaw instance running in production, not a community contributor. The original CHANGELOG framing ("community contributor @wintermute") understated the funnier truth: the agent built on top of the project wrote the deploy guide for the project after hitting its sharp edges in production. Dogfooding with extra steps. Co-Authored-By: Wintermute (OpenClaw) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rewrite minions deployment guide for agent line-by-line execution Fixes 12 findings from reading v0.16.1 guide as-an-agent would: Real bugs: - Crontab syntax wrong for user crontabs (6-field format dumped into `crontab -e` got "bad minute" or parsed `user` as the command). Now two labeled blocks: 5-field for `crontab -e`, 6-field for `/etc/crontab`. - Watchdog restart loop (old shutdown lines in unrotated log re-matched every 5 min forever). New `minion-watchdog.sh` writes 2-line PID file (PID + restart epoch) and only considers log lines newer than the epoch. Regex rewritten explicit (mawk rejects `{n}` intervals). - Credentials in world-readable /etc/crontab. Secrets move to /etc/gbrain.env (mode 600), referenced via BASH_ENV in crontab. Structural: - Preconditions block (5 fail-fast checks). - "Which option?" decision tree. - Template variable table (6 vars documented). - Upgrade section (v0.13.x -> v0.16.2 checklist). - Option 3: systemd.service + Procfile + fly.toml.partial snippets. - Uninstall section. - `--follow` example uses `gbrain embed --stale` (a real command) instead of the fictional `gbrain enrich`. - Dead-end "Proposed CLI flags (not yet implemented)" replaced with a "Tune per-job today" callout pointing at flags that exist. - Known Issues rewritten as imperatives. Also wires `docs/guides/minions-deployment.md` into `scripts/llms-config.ts` under the Configuration section so remote agents fetching llms.txt / llms-full.txt see the guide by name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.16.2) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync v0.16.2 CHANGELOG with the actual --follow example in the guide The shipped docs/guides/minions-deployment.md uses `gbrain embed --stale` (a real command) but the v0.16.2 CHANGELOG entry still referenced `gbrain enrich --brain $GBRAIN_WORKSPACE` (the older draft). Bring the CHANGELOG in line with what actually shipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-22 00:01:08 -07:00
Garry Tan	0e9f8814a5	feat: v0.16.0 — durable agent runtime (gbrain agent + subagent handler + plugin loader) (#258 ) * refactor(mcp): extract buildToolDefs helper for subagent tool registry reuse The inline operations.map(...) block in src/mcp/server.ts became the only source of truth for agent-facing tool definitions. Extract into a reusable exported helper so the v0.15 subagent tool registry can call it with a filtered OPERATIONS subset instead of duplicating the shape. Byte-for-byte equivalence regression pinned in test/mcp-tool-defs.test.ts — legacy inline mapping kept verbatim inside the test so any future drift between the new helper and the pre-extraction MCP schema fails loudly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(operations): subagent-aware OperationContext + put_page namespace Adds three optional fields to OperationContext: - jobId?: number — the currently running Minion job id - subagentId?: number — the owning subagent job id for tool-dispatched calls - viaSubagent?: boolean — FAIL-CLOSED flag for agent-path gating put_page now enforces a namespace rule when invoked on the subagent tool dispatch path (viaSubagent=true): writes MUST target `wiki/agents/<subagentId>/...`. Anchored, slash-boundary enforced so a collision like `wiki/agents/12evil/...` can't impersonate subagent 12. The check runs BEFORE the dry-run short-circuit so preview calls surface the same rejection. Fail-closed: a missing subagentId with viaSubagent=true rejects every slug rather than letting a dispatcher bug open a hole. Existing callers unaffected — all three fields are optional and the legacy put_page behavior is unchanged when viaSubagent is undefined/false. 12 regression + namespace tests pin: - local CLI writes (viaSubagent unset) accept arbitrary slugs - MCP writes (remote=true, viaSubagent unset) accept arbitrary slugs - subagent-path: anchored prefix accepted, wrong id rejected, prefix- collision defeated, leading-slash rejected, bare-prefix rejected, fail-closed on missing/NaN subagentId, permission_denied code emitted Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(schema): v0.15.0 subagent runtime tables + migration orchestrator Adds three new tables for the durable LLM agent runtime: subagent_messages — Anthropic message-block persistence. Parallel tool_use blocks in one assistant message live in content_blocks JSONB, not across rows (fixes the (job_id, turn_idx, role) misdesign codex caught in v0.13 drafting). subagent_tool_executions — Two-phase tool ledger. INSERT pending before execute, UPDATE complete/failed after. Replay re-runs pending rows only if the tool is idempotent (v1 ships only idempotent tools so this is preventive). subagent_rate_leases — Lease-based concurrency cap for outbound providers (e.g. anthropic:messages). Stale leases auto-prune on next acquire so crashed workers can't strand capacity. All DDL uses CREATE TABLE/INDEX IF NOT EXISTS — order-independent vs PR #244's initSchema() reorder, and idempotent across fresh-install + upgrade paths. Shipped in both src/schema.sql (Postgres) and src/core/pglite-schema.ts (PGLite); schema-embedded.ts regenerated. Migration orchestrator v0_15_0.ts (phases: schema → verify → record). v0_14_0.ts is a no-op stub so the registry's version sequence stays gapless (v0.14.0 shipped shell-jobs — code change, no DB migration). 10 unit tests for registry wiring, ordering, dry-run phase behavior, and schema-embedded table presence. test/apply-migrations.test.ts updated for the two new registry entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): emit child_done on every terminal + max_stalled per-job + terminal set fix Three correctness fixes the v0.15 subagent aggregator spine depends on: 1. child_done emission on ALL terminal transitions, not just success. - completeJob already emitted on success — now also tags outcome='complete'. - failJob newly emits on terminal 'failed' or 'dead' (outcome='failed'\|'dead', error=<text>), BEFORE the parent-terminal UPDATE so the EXISTS guard on the inbox INSERT doesn't skip it on fail_parent paths (codex catch). - cancelJob now emits outcome='cancelled' per descendant with a parent. - handleTimeouts now emits outcome='timeout' per timed-out child. ChildDoneMessage gains optional { outcome, error } — backwards compatible (legacy writers omitted them; consumers treat absent outcome as 'complete'). 2. Parent-resolution terminal set now includes 'failed'. Pre-v0.15 the `NOT EXISTS (... status NOT IN ('completed','dead','cancelled'))` guard treated a failed child as still-pending, stranding aggregator parents that chose on_child_fail='continue' or 'ignore' in waiting-children forever. Expanded to {completed, failed, dead, cancelled} everywhere parent resolution reads child status (completeJob inline, failJob remove_dep + continue, cancelJob sweep, handleTimeouts sweep, and the resolveParent method itself). 3. MinionJobInput.max_stalled threads through MinionQueue.add() on INSERT. Column exists with default 1 — that is "first stall → dead", which defeats crash recovery for long-running handlers. Subagent children will set max_stalled: 3 to survive mid-run worker kills. Second-submitter under an idempotency-key hit does NOT mutate the existing row (codex-flagged footgun — first-submit options are load-bearing state). 13 unit tests pin: emission on each of completeJob/failJob/cancelJob/ handleTimeouts, insertion order on fail_parent, terminal-set expansion with continue policy, max_stalled default + override + idempotency behavior. E2E tier 1 (Postgres) passes 141 tests unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): rate-leases + waitForCompletion infra for v0.15 subagent Two infrastructure modules the subagent handler spine depends on: rate-leases.ts — lease-based concurrency cap for outbound providers (anthropic:messages, openai:, etc.). Counter-based limiters leak capacity on worker crash; leases are owner-tagged rows with expires_at that auto-prune on the next acquire. Two-phase: txn-scoped pg_advisory_xact_lock guards the check-then-insert so concurrent acquires can't both win the "last slot". renewLeaseWithBackoff retries 3x (250/500/1000ms) for mid- call DB blips — on persistent failure the LLM-loop caller aborts with a renewable error so the worker re-claims and the rate invariant is preserved. Owner FK cascades clean up leases on job deletion. wait-for-completion.ts — poll-until-terminal helper for CLI callers. Minions' NOTIFY is worker-side only; `gbrain agent run --follow` polls getJob() until status is {completed, failed, dead, cancelled}. TimeoutError carries jobId + elapsedMs and does NOT cancel the job — the user can inspect via `gbrain jobs get <id>` later. Supports AbortSignal for Ctrl-C without throwing. Default pollMs is 1000 on Postgres, 250 on PGLite (inline CLI has no network RTT). 21 unit tests cover: single/multi acquire under cap, rejection past cap, release frees slot, different keys are independent, stale prune, cascade on owner delete, renew bumps expires_at, renew on missing is false, backoff path success + pruned short-circuit. waitForCompletion: fast-path terminal, transitions mid-wait (completed/failed/cancelled), TimeoutError shape, abort-signal early exit, non-existent job error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> feat(minions): subagent ToolDef types + brain-tool registry (v0.15) Types first so the handler has a stable contract: - SubagentHandlerData / AggregatorHandlerData — the two job.data shapes - ToolCtx (engine, jobId, remote, signal) + ToolDef (name, description, input_schema, idempotent, execute) — Anthropic-envelope, distinct from the MCP McpToolDef extraction landed earlier - ContentBlock discriminated union for subagent_messages.content_blocks - SubagentStopReason + SubagentResult emitted on terminal completion brain-allowlist.ts derives one ToolDef per allow-listed OPERATION. Reuses the ParamDef → JSONSchema shape from the MCP extraction in a local helper (Anthropic's input_schema field diverges from MCP's inputSchema by a character). The 11-name allow-list is read-safe + put_page — every destructive / filesystem / identity-mutating op stays off by default. put_page gets a namespace-wrapped tool schema: `slug` pattern = anchored `^wiki/agents/<subagentId>/.+`. The server-side check in put_page op (shipped in prior commit) is still the authoritative gate — the schema just helps the model write correct slugs first-try. `subagentId` is plumbed into the ToolCtx so the viaSubagent=true fail-closed path lights up on every tool-dispatched put_page. filterAllowedTools narrows a registry by subagent_def's allowed_tools frontmatter field. Rejects unknown names at load time (no silent drop — typos in a skills/subagents/.md would otherwise ship to prod with a tool silently missing). 18 tests pin: every allowlist name exists in OPERATIONS (catches upstream rename), Anthropic name regex, put_page namespace pattern per-subagent, execute() routes through the op handler with viaSubagent=true, out-of- namespace put_page throws permission_denied, filter passes prefixed + unprefixed names, rejects unknowns, deduplicates. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> feat(minions): subagent-audit JSONL + transcript renderer Two small plumbing pieces the v0.15 subagent handler + `gbrain agent logs` depend on: subagent-audit.ts — JSONL-rotated audit log mirroring the shell-audit pattern. Two event flavors: submission (one line per job submit) and heartbeat (one line per turn boundary — llm_call_started / completed / tool_called / tool_result / tool_failed). Heartbeats fix the "--follow on a long Anthropic call shows nothing for 30 seconds" problem codex flagged. Never logs prompts or tool inputs (PII risk — subagent input_vars may carry user-supplied free text); DOES log tokens, ms_elapsed, tool_name, first 200 chars of error text. Rotates weekly via ISO week. `readSubagent AuditForJob` is the readback path for `gbrain agent logs` — scans the current + prior week file so job boundaries across weeks still resolve. `GBRAIN_AUDIT_DIR` overrides the default ~/.gbrain/audit/ for container deploys. transcript.ts — renders subagent_messages + subagent_tool_executions to markdown. Message order is authoritative; tool rows splice under their owning assistant tool_use by tool_use_id. Handles text, tool_use (with pending / complete / failed execution rows), tool_result (skipped if we already rendered the owning tool_use — avoids double-printing), and unknown block types (fenced JSON dump for diagnostics). Output is UTF-8-safe truncated at maxOutputBytes. 21 unit tests: ISO week filename rotation (incl. 2027-01-01 → W53-2026 boundary), submission + heartbeat write shapes, 200-char error cap, best- effort write failure doesn't throw, readback filters by job_id and sinceIso. Transcript: empty input, ordering, token line, tool_use + complete/failed/pending execution rendering, truncation, unknown-block diagnostic dump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): subagent LLM-loop handler with crash-resumable replay The main event: runs one Anthropic Messages API conversation with tool use, persists every turn + tool execution, and resumes cleanly after a worker kill anywhere in the loop. Design points that carry the v0.15 guarantees: 1. Two-phase tool persistence. INSERT status='pending' before dispatch, UPDATE to 'complete' or 'failed' after. subagent_messages rows are the canonical conversation; subagent_tool_executions rows are the canonical "did this tool run + what did it return". Either DB commit is atomic, so replay has a single source of truth. 2. Replay reconciliation. If the last persisted message is an assistant with tool_use blocks AND no following synthesized user message, we crashed mid-dispatch. On resume, finish those tools first (respecting idempotent flag for 'pending' rows), synthesize the user turn, and THEN call the LLM again. Non-idempotent pending rows abort the job with a clear error — v0.15 ships only idempotent tools so this is preventive. 3. Rate lease around every LLM call. acquireLease before, releaseLease after (both success and error paths). acquired=false throws RateLeaseUnavailableError — the worker treats it as a renewable error and re-claims later, so a temporary capacity cap doesn't fail the job terminally. 4. Anthropic prompt caching. system block gets cache_control=ephemeral; the LAST tool def gets it too (Anthropic caches everything up to and including the marked block). ~10x cost reduction on multi-turn agents per the plan. 5. Dual-signal abort. AbortSignal.any merges ctx.signal (timeout / lock loss / cancel) with ctx.shutdownSignal (worker SIGTERM). Both feed the Anthropic call's AbortSignal; mid-turn abort bails before the next LLM call with whatever turns are already persisted. Node ≥ 20 has AbortSignal.any; older runtimes get a manual-merge polyfill. 6. Injectable Anthropic client. The real SDK implements MessagesClient structurally; tests inject a FakeMessagesClient that scripts responses. 12 unit tests pin: no-tool happy path, single tool_use complete, tool throws → failed row + loop continues, unknown tool name rejection, max_turns cap, crash-then-resume with partial state, replay skips already- complete tool execs without re-invoking execute, non-idempotent pending rejects on resume, lease acquire + release roundtrip, RateLeaseUnavailable under cap-full, missing prompt validation, allowed_tools unknown-name. NOT in v0.15: refusal detection (stop_reason + content shape), stop_reason =max_tokens partial recovery, mid-call lease renewal with backoff loop. All three are documented as P2 items in the plan file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): subagent_aggregator handler with mixed-outcome rendering Claims AFTER all subagent children resolve — by then Lane 1B's queue changes have posted one child_done message per terminal transition into this job's inbox (complete / failed / dead / cancelled / timeout). The aggregator reads those, builds a deterministic markdown summary, and returns it as the handler result. Not an LLM call in v0.15 — output is reproducible concatenation so fan-out runs stay comparable. v0.16+ can add an LLM synthesis pass behind an opt-in flag. Contract: - empty children_ids → `(no children)` marker - missing child_done (shouldn't happen under v0.15 invariants but possible if a terminal-state path slipped past Lane 1B) → counted as failed with "no child_done message observed" error - non-complete outcomes: result is null in the output so no payload leaks alongside a failure label - children appear in the order children_ids was supplied - custom aggregate_prompt_template replaces the markdown header 13 unit tests cover: empty input, all-success, mixed outcomes, result suppression on failure, missing child_done handling, order preservation, custom template, progress + log emission, stringified JSONB payload parsing, non-child_done inbox filtering, legacy-writer outcome fallback, and internal helper edges. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): GBRAIN_PLUGIN_PATH loader + plugin-authors guide (v0.15) Plumbing that makes Wintermute (and future downstream agents) day-1 usable on v0.15. Host repos drop a `gbrain.plugin.json` + `subagents/` directory somewhere, set GBRAIN_PLUGIN_PATH (colon-separated like \$PATH), and their custom subagent defs load at worker startup. Path policy is strict: absolute paths only. Relative, ~-prefixed, and URL-style (https://, file://) all rejected with warnings — the user controls where plugins live. Non-existent paths and files (not dirs) are warned and skipped so a typo doesn't crash worker startup. Collision policy: left-wins. If two plugins ship a subagent with the same name, the first one in GBRAIN_PLUGIN_PATH keeps it and the other gets a warning naming both sources. Deterministic + debuggable. Trust policy: plugins ship subagent defs ONLY. Cannot declare new tools, cannot extend the brain allow-list, cannot override safety flags. The subagent def's `allowed_tools:` frontmatter MUST subset the derived registry — validation happens at load time (worker startup), not at dispatch time, so a typo in a skill gives a loud startup error instead of silently "tool never fires at 3am." Manifest `plugin_version: "gbrain-plugin-v1"` locks the contract. Unknown versions rejected. `subagents` field escape attempts (`../../../etc` etc) rejected. gray-matter handles the markdown frontmatter parse — subagent defs don't conform to the page schema, so we don't use parseMarkdown. docs/guides/plugin-authors.md is the Wintermute-facing walkthrough. Covers the minimum viable plugin shape, the three policies, the frontmatter fields, known caveats (audit JSONL is local-only, tool calls always run remote=true, put_page is namespace-scoped). 22 unit tests pin path rejection, missing/invalid manifest, unsupported version, escape-attempt, basename fallback for missing frontmatter.name, allowed_tools round-trip, unknown-tool rejection with validAgentToolNames, empty env, multi-path, collision warning with left-wins, trimmed paths, manifest-rejection as warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): gbrain agent run + logs + worker registration (v0.15 Lane 4H) Three integration seams wired: src/commands/agent.ts — \`gbrain agent run\`. Submits subagent jobs (or a fan-out of N + aggregator) under the trusted-submit flag so the PROTECTED_JOB_NAMES guard doesn't reject. Fan-out path creates the aggregator first (so children can reference its id as parent), submits each child with on_child_fail='continue' (required by Lane 1B's terminal- set + child_done machinery), then jsonb_set's the aggregator's children_ids. Short-circuits a 1-entry manifest to a single subagent with no aggregator. Follow mode runs agent-logs streaming + waitFor Completion in parallel and exits on terminal status; detach prints the job id and exits. Ctrl-C is handled as detach, not cancel — the job keeps running, consistent with durability invariants. src/commands/agent-logs.ts — \`gbrain agent logs\`. Merges ~/.gbrain/audit/ subagent-jobs-.jsonl (heartbeats + submissions) with subagent_messages (persisted conversation) in one chronological stream. --follow polls at 1s and exits when the job hits terminal. --since accepts ISO-8601 OR relative shorthand (5m / 1h / 2d). Writes transcript tail (full message + tool tree) only for terminal jobs, so mid-run --follow doesn't spam a half-rendered transcript. src/commands/jobs.ts registerBuiltinHandlers — matches the shell-handler opt-in shape. GBRAIN_ALLOW_LLM_JOBS=1 registers the subagent + subagent_aggregator handlers, then loads plugins from GBRAIN_PLUGIN_PATH with validAgentToolNames pulled from BRAIN_TOOL_ALLOWLIST. Every plugin warning + loaded-plugin line prints to stderr, mirroring the openclaw- seam startup convention. src/core/minions/protected-names.ts — subagent + subagent_aggregator join the protected set. MCP submit_job returns permission_denied; only trusted-CLI callers (with allowProtectedSubmit) can insert these rows. src/cli.ts — adds 'agent' to CLI_ONLY + dispatches it like 'jobs'. Test fallout: subagent-handler.test.ts + subagent-transcript.test.ts helpers now submit under allowProtectedSubmit (they insert rows named 'subagent' directly against the queue). 23 new tests in agent-cli.test.ts cover: flag parsing (including --detach implies !follow, --tools comma split, -- terminator, unknown flag throw), --since parse (ISO, relative 5m/2h/1d, unparseable error), protected-name guard for all three names, trusted-submit gate, and a fan-out integration check that verifies the aggregator + children shape after --fanout-manifest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> test(e2e): rename max_children test's spawned jobs off the protected 'subagent' name The spawn-storm test submitted 50 literal-string 'subagent' children to exercise the max_children row-lock serialization. In v0.15 'subagent' is a PROTECTED_JOB_NAME (CLI-only; trusted submit required), so the old literal submission now throws before reaching the row-lock check. The test is about max_children semantics, not the v0.15 subagent runtime specifically — rename the child name to 'child_worker' so the test exercises the exact same queue.add path without tripping the new guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(ship): v0.15.0 — VERSION, CHANGELOG, README, upgrading-agents, CLAUDE.md Bumps VERSION → 0.15.0 and package.json → 0.15.0 (resolves the pre-existing drift — on master, VERSION=0.14.0 but package.json=0.13.1; src/version.ts reads package.json, so this is what the binary prints now). CHANGELOG lands the release-summary entry in the GStack voice + the full itemized change list (11 new modules, 3 new tables, queue correctness fixes, trust-model additions, 159 new unit tests). Voice rules respected — no em dashes, no AI vocabulary, real file names + real numbers. README gets a "Durable agents: `gbrain agent` (v0.15)" section next to the Minions block, with the three canonical CLI shapes (single run, fanout-manifest, logs --follow) and a pointer to plugin-authors.md. docs/UPGRADING_DOWNSTREAM_AGENTS.md gets a full v0.15.0 section covering the four adoption steps downstream agents (Wintermute and similar) need: (1) worker opt-in via GBRAIN_ALLOW_LLM_JOBS, (2) moving custom subagent defs to a plugin repo, (3) replacing ephemeral subagent runs with durable `gbrain agent run`, (4) the put_page namespace rule for agent-driven writes. CLAUDE.md updated with concise per-file descriptions for every new module: the handler, aggregator, audit, rate-leases, wait-for-completion, transcript, plugin-loader, brain-allowlist, tool-defs extraction, agent CLI + logs CLI, and the registerBuiltinHandlers wiring for subagent handlers + plugin-loader. Verified: binary builds (940 modules, 89ms compile), prints `gbrain 0.15.0`, `gbrain agent --help` shows the new subcommand shape. 170 new tests pass (full v0.15 surface). Full unit suite passes bar one parallel-load flake on a pre-existing E2E (graph-quality, passes in isolation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): drop GBRAIN_ALLOW_LLM_JOBS flag — subagent handlers always-on The env flag was ceremony. Shell jobs need the flag because they execute arbitrary CLI commands (RCE surface). Subagent jobs don't — they call the Anthropic API with whatever ANTHROPIC_API_KEY is in env, so the key is already the cost gate (no key → SDK fails on the first turn). And who-can-submit is already protected by PROTECTED_JOB_NAMES + TrustedSubmitOpts: MCP callers get permission_denied; only `gbrain agent run` with allowProtectedSubmit can insert subagent / subagent_aggregator rows. The flag added nothing the existing guards didn't already give us. registerBuiltinHandlers now always registers subagent + subagent_aggregator and loads GBRAIN_PLUGIN_PATH plugins. Worker startup prints: [minion worker] subagent handlers enabled instead of the conditional enabled/disabled pair. Plugin discovery runs unconditionally — empty PATH is a no-op. README, CHANGELOG, docs/UPGRADING_DOWNSTREAM_AGENTS, CLAUDE.md, agent CLI help text, and subagent handler docstring all updated to drop the flag reference. Shell handler's GBRAIN_ALLOW_SHELL_JOBS gate is untouched — separate concern (RCE, not billing). Full suite: 1859 pass, 0 fail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: scrub private agent-fork name from all public artifacts Enforces the rule added to CLAUDE.md (privacy section): never say `Wintermute` in any CHANGELOG, README, doc, PR, or commit message. Reader-facing copy says `your OpenClaw` (the term covers every downstream OpenClaw deployment — Wintermute, Hermes, AlphaClaw — in one umbrella the reader already recognizes). First-person / origin-story copy says `Garry's OpenClaw` (honest that this is the production deployment driving the feature, without exposing the private agent's name). Swept across: CHANGELOG.md (v0.15 entry + 4 historical mentions) README.md TODOS.md docs/UPGRADING_DOWNSTREAM_AGENTS.md docs/guides/plugin-authors.md (including example plugin names) docs/guides/plugin-handlers.md docs/guides/minions-fix.md docs/designs/KNOWLEDGE_RUNTIME.md (27 refs, mostly analytical) docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md skills/migrations/v0.11.0.md skills/skillpack-check/SKILL.md scripts/skillify-check.ts src/commands/doctor.ts src/commands/migrations/v0_15_0.ts src/commands/skillpack-check.ts src/core/enrichment/completeness.ts src/core/minions/plugin-loader.ts src/core/operations.ts src/core/output/scaffold.ts Intentionally kept (these mentions define/test the rule itself): CLAUDE.md — the privacy rule section necessarily uses the literal name to define the restriction and examples test/plugin-loader.test.ts — fixture name in a plugin-loading test; renaming risks breaking assertion logic test/integrations.test.ts — the word appears in a privacy-regex test that explicitly enforces name redaction test/doctor-minions-check.test.ts — a comment referencing the rule CEO plan artifact at ~/.gstack/projects/… — private, not distributed Binary builds (941 modules), 198/198 relevant tests pass, `gbrain --version` prints `0.15.0`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: gitignore bun --compile artifacts with a glob, not specific hashes Each `bun build --compile` emits a fresh hash-named `.-.bun-build` file in cwd. The prior entries listed two specific hashes that were already stale, so every build after those created a new untracked file requiring manual cleanup. Replace the two stale entries with `.bun-build` so any current or future compile artifact is ignored automatically. Verified: ran `bun build --compile`, got two new `.-.bun-build` files, `git status` stays clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> chore(ship): rename v0.15.0 → v0.16.0 gbrain master is at 0.14.2. Other 0.15.x PRs may land before/after this one — we bump the minor (new capability) and lock to 0.16.0 so ordering with concurrent work doesn't matter. Touches: - VERSION: 0.15.0 → 0.16.0 - package.json: 0.15.0 → 0.16.0 - Rename src/commands/migrations/v0_15_0.ts → v0_16_0.ts (+ all version strings inside + import in index.ts registry) - Rename test/migrations-v0_15_0.test.ts → migrations-v0_16_0.test.ts - test/apply-migrations.test.ts: skippedFuture lists now reference '0.16.0' - test/put-page-namespace.test.ts + test/mcp-tool-defs.test.ts: Lane comment refs updated - src/schema.sql + src/core/pglite-schema.ts: "v0.15.0" section comment updated; src/core/schema-embedded.ts regenerated - CHANGELOG.md: top entry renamed to [0.16.0]; inline v0_15_0 / v0.15.0 refs swept - docs/UPGRADING_DOWNSTREAM_AGENTS.md: section heading v0.15.0 → v0.16.0 Verified: `gbrain --version` prints 0.16.0, migration registry / buildPlan / put_page / mcp-tool-defs / handlers tests all green (49/49). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: reframe v0.16 durability headline around OpenClaw crashes "Laptop closed mid-run" framing implied a consumer workflow. Real pain is OpenClaw subagents dying daily on worker kill, memory blip, or timeout. Headline + README copy match the body now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms-full.txt after README copy change Regen drift guard caught the README edit from 83beec4. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 21:14:17 -07:00
Garry Tan	5fd9cd2644	feat: shell job type + worker abort-path fix (v0.13.0) (#217 ) * feat(minions): add protected-name constant + ctx.shutdownSignal Introduce PROTECTED_JOB_NAMES ('shell') in a side-effect-free core module so queue.ts can check it without importing from handlers/. MinionJobContext gains shutdownSignal (distinct from signal) — handlers that need to run SIGTERM-triggered cleanup subscribe to both; most handlers ignore shutdown and run through the worker's 30s cleanup race to natural completion. * fix(minions): MinionQueue.add gains trusted 4th arg + trim-normalized guard Adds allowProtectedSubmit opt-in as a separate 4th parameter (NOT folded into opts) so callers spreading user-provided opts ({...userOpts}) can't accidentally carry the trust flag. PROTECTED_JOB_NAMES check runs on the trimmed name BEFORE insert, closing the queue.add(' shell ', ...) whitespace bypass that would have evaded a has(name) check. * fix(minions): worker calls failJob on abort + wires ctx.shutdownSignal Pre-v0.13.0 worker returned silently when ctx.signal.aborted fired, leaving jobs in 'active' until stall sweep. Handlers using cooperative cancel had no deterministic status flip — timeout/cancel/lock-loss all looked the same from downstream callers (gbrain jobs get, --follow loops). Fix: derive abort reason from abort.signal.reason ('timeout' \| 'cancel' \| 'lock-lost' \| 'shutdown') and call failJob with 'aborted: <reason>' text. failJob is idempotent via token+status match, so no-op when another path already flipped status (handleTimeouts, cancelJob, stall). Also: new shutdownAbort (instance-level AbortController) fires on process SIGTERM/SIGINT and propagates to every handler's ctx.shutdownSignal. Shell handler listens to both signals and runs SIGTERM→5s→SIGKILL on its child on either; other handlers only listen to ctx.signal so deploy restarts don't cancel them mid-flight. * feat(minions): add shell job handler + submission audit log New 'shell' job type spawns arbitrary commands under the Minions worker. Deterministic cron scripts (API fetch, token refresh, scrape+write) can move off the LLM gateway — zero Opus tokens per fire. Handler contract: - cmd or argv (exactly one required). cmd spawns via /bin/sh -c (absolute path, not 'sh', to block PATH-override shell substitution). argv spawns direct with no shell. - cwd required, must be absolute. Operator-trust boundary. - env defaults to SHELL_ENV_ALLOWLIST ({PATH, HOME, USER, LANG, TZ, NODE_ENV}) picked from process.env, with caller overrides merged on top. Prevents accidental $OPENAI_API_KEY interpolation into scripts. - stdout/stderr retained as UTF-8-safe tails (64KB/16KB) via string_decoder.StringDecoder. Prepends [truncated N bytes] marker. - Abort (either ctx.signal or ctx.shutdownSignal) fires SIGTERM → 5s grace → SIGKILL on child. Timer NOT .unref'd so worker's 30s race waits for the child to actually die. shell-audit.ts writes a JSONL line per submission to ~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl (ISO-week rotated, override via GBRAIN_AUDIT_DIR). argv logged as JSON array (not space-joined, which would flatten args with spaces). Never logs env values. Best-effort writes: failures log to stderr but don't block submission. * feat(jobs): submit_job MCP guard + CLI --timeout-ms + starvation warning submit_job operation gains timeout_ms param (was missing — couldn't plumb the existing MinionJobInput field through from either CLI or MCP). When ctx.remote=true and name is in PROTECTED_JOB_NAMES, throws OperationError('permission_denied'). Combined with the queue.add trusted guard, MCP callers can never submit shell jobs even if the env flag is on. CLI submit: new --timeout-ms N flag. Passes {allowProtectedSubmit:true} as the 4th arg to queue.add only when the submitted name is protected (not blanket-set for every job). Prints a starvation-warning block to stderr when a shell job is submitted without --follow, pointing at both --follow and 'gbrain jobs work' remediation. Fires for every shell submit regardless of the submitter's env — the submitter env is a weak proxy for the worker env. Worker handler registration: conditional on GBRAIN_ALLOW_SHELL_JOBS=1. Default: off. 'gbrain jobs submit --help' now lists handler types with a pointer to docs/guides/minions-shell-jobs.md for shell. * test(minions): 40 unit + 4 E2E cases for shell handler Unit (test/minions-shell.test.ts): - Protected names: trim-normalized, case-sensitive, whitespace bypass defense - MinionQueue.add: trusted opt-in, whitespace bypass, non-protected untouched - Handler validation: cmd\|argv exclusive, cwd required/absolute, env strings - Spawn: cmd/argv happy paths, non-zero exit, ENOENT, result shape - Env allowlist: leaked-secret blocked, PATH inherited, caller override - Abort: ctx.signal, ctx.shutdownSignal, pre-aborted signal - Audit: ISO-week year boundary (2027-01-01 → W53 2026), mid-year W52/W53, GBRAIN_AUDIT_DIR override, argv as JSON array, env never logged, EACCES non-blocking - Output truncation: 100KB → last 64KB with [truncated N bytes] marker E2E (test/e2e/minions-shell.test.ts): - Full lifecycle: submit → worker claim → spawn → complete - MinionQueue.add without trusted arg throws (including whitespace bypass) - submit_job with ctx.remote=true rejects shell (MCP guard) - submit_job with ctx.remote=false allows shell (CLI path) * chore: bump version and changelog (v0.13.0) Move gateway crons to Minions. Zero LLM tokens per cron fire. Worker abort path finally marks aborted jobs dead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: reframe v0.13.0 copy for OpenClaw operators (not Wintermute-specific) gbrain is an open-source product for any OpenClaw/Hermes operator, not Garry's personal Wintermute deployment. Rewords the v0.13.0 CHANGELOG entry, the minions-shell-jobs guide, and the deferred TODOS entries to speak to "your OpenClaw" / "OpenClaw operators" instead. Replaces /data/wintermute cwd examples with the canonical /data/.openclaw/workspace path. Pre-existing Wintermute references in older CHANGELOG entries (v0.11/v0.10.3) left unchanged. * feat(migrations): add v0.13.0 adoption playbook for shell jobs Adding the migration file the CEO review originally scoped out. Without it, operators upgrade to v0.13.0 and the capability ships but adoption doesn't happen — the 60% gateway CPU reduction only lands if someone actually rewrites their crontab. skills/migrations/v0.13.0.md is the instruction manual the host agent reads on gbrain upgrade: - Enable worker: GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work (Postgres) or per-tick --follow (PGLite) - Audit cron manifest: classify LLM-requiring vs deterministic - Propose per-cron rewrites with diffs, approved one at a time - Env allowlist guidance for scripts that need API keys - Verification playbook: run one fire, compare pre/post, only then approve the next batch - Starvation sanity-check runbook item Iron rules: never auto-rewrite the operator's crontab (host-specific code per CLAUDE.md). LLM-requiring crons stay on the gateway. Ambiguous cases ask the operator. No mechanical orchestrator ships with this migration — every rewrite is operator judgment. A future gbrain crontab-to-minions helper is tracked in TODOS.md as P1. * docs: sync UPGRADING + SKILLPACK with v0.13.0 shell jobs UPGRADING_DOWNSTREAM_AGENTS.md: append v0.13.0 section per the file's convention (each release appends). No skill edits required, feature is off-by-default, optional adoption via skills/migrations/v0.13.0.md. Lists typical LLM-vs-deterministic classifications so operators know which of their crons are candidates for migration. GBRAIN_SKILLPACK.md: add shell-jobs guide row to the cron/Minions guide table so it's discoverable alongside existing Cron via Minions, Plugin Handlers, and Minions fix guides. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 10:54:31 +08:00
Garry Tan	d8613366a5	Minions v7 + v0.11.1 canonical migration + skillify (#130 ) * feat: add minion_jobs schema, migration v5, and executeRaw to BrainEngine Foundation for the Minions job queue system. Adds: - minion_jobs table (20 columns) with CHECK constraints, partial indexes, and RLS. Inspired by BullMQ's job model, adapted for Postgres. - Migration v5 creates the table for existing databases. - executeRaw<T>() method on BrainEngine interface for raw SQL access, needed by the Minions module for claim queries (FOR UPDATE SKIP LOCKED), token-fenced writes, and atomic stall detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions job queue — queue, worker, backoff, types BullMQ-inspired Postgres-native job queue built into GBrain. No Redis. No external dependencies. Postgres transactions replace Lua scripts. - MinionQueue: submit, claim (FOR UPDATE SKIP LOCKED), complete/fail (token-fenced), atomic stall detection (CTE), delayed promotion, parent-child resolution, prune, stats - MinionWorker: handler registry, lock renewal, graceful SIGTERM, exponential backoff with jitter, UnrecoverableError bypass - MinionJobContext: updateProgress(), log(), isActive() for handlers - 8-state machine: waiting/active/completed/failed/delayed/dead/ cancelled/waiting-children Patterns stolen from: BullMQ (lock tokens, stall detection, flows), Sidekiq (dead set, backoff formula), Inngest (checkpoint/resume). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 43 tests for Minions job queue Full coverage of the Minions module against PGLite in-memory: - Queue CRUD (9): submit, get, list, remove, cancel, retry, duplicate - State machine (6): waiting→active→completed/failed, retry→delayed→waiting - Backoff (4): exponential, fixed, jitter range, attempts_made=0 edge - Stall detection (3): detect stalled, counter increment, max→dead - Dependencies (5): parent waits, fail_parent, continue, remove_dep, orphan - Worker lifecycle (5): register, start-without-handlers, claim+execute, non-Error throws, UnrecoverableError bypass - Lock management (3): renewal, token mismatch, claim sets lock fields - Claim mechanics (4): empty queue, priority ordering, name filtering, delayed promotion timing - Cancel & retry (2): cancel active, retry dead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions CLI commands and MCP operations Wire Minions into the GBrain CLI and MCP layer: CLI (gbrain jobs): submit <name> [--params JSON] [--follow] [--dry-run] list [--status S] [--queue Q] [--limit N] get <id> — detailed view with attempt history cancel/retry/delete <id> prune [--older-than 30d] stats — job health dashboard work [--queue Q] [--concurrency N] — Postgres-only worker daemon 6 MCP operations (contract-first, auto-exposed via MCP server): submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress Built-in handlers: sync, embed, lint, import. --follow runs inline. Worker daemon blocked on PGLite (exclusive file lock). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for Minions job queue CLAUDE.md: added Minions files to key files, updated operation count (36), BrainEngine method count (38), test file count (45), added jobs CLI commands. CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall detection, worker daemon). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions v2 — agent orchestration primitives (pause/resume, inbox, tokens, replay) Adds the foundation for Minions as universal agent orchestration infrastructure. GBrain's Postgres-native job queue now supports durable, observable, steerable background agents. The OpenClaw plugin (separate repo) will consume these via library import, not MCP, for zero-latency local integration. ## New capabilities - Concurrent worker — Promise pool replaces sequential loop. Per-job AbortController for cooperative cancellation. Graceful shutdown waits for all in-flight jobs via Promise.allSettled. - Pause/resume — pauseJob clears the lock and fires AbortSignal on active jobs. Handlers check ctx.signal.aborted and exit cleanly. resumeJob returns paused jobs to waiting. Catch block skips failJob when signal.aborted. - Inbox (separate table) — minion_inbox table for sidechannel messages. sendMessage with sender validation (parent job or admin). readInbox is token-fenced and marks read_at atomically. Separate table avoids row bloat from rewriting JSONB on every send. - Token accounting — tokens_input/tokens_output/tokens_cache_read columns. updateTokens accumulates; completeJob rolls child tokens up to parent. USD cost computed at read time (no cost_usd column — pricing too volatile). - Job replay — replayJob clones a terminal job with optional data overrides. New job, fresh attempts, no parent link. ## Handler contract additions MinionJobContext now provides: - `signal: AbortSignal` — cooperative cancellation - `updateTokens(tokens)` — accumulate token usage - `readInbox()` — check for sidechannel messages - `log()` — now accepts string or TranscriptEntry ## MCP operations added pause_job, resume_job, replay_job, send_job_message — all auto-generate CLI commands and MCP server endpoints. ## Library exports package.json exports map adds ./minions and ./engine-factory paths so plugins can `import { MinionQueue } from 'gbrain/minions'` for direct library use. ## Instruction layer (the teaching) - skills/minion-orchestrator/SKILL.md — when/how to use Minions, decision matrix, lifecycle management, anti-patterns - skills/conventions/subagent-routing.md — cross-cutting rule: all background work goes through Minions - RESOLVER.md — trigger entries for agent orchestration - manifest.json — registered ## Schema migration v6 Additive: 3 token columns, paused status, minion_inbox table with unread index. Full Postgres + PGLite support. No backfill needed. ## Tests 65 tests (was 43): pause/resume (5), inbox (6), tokens (4), replay (4), concurrent worker context (3), plus all existing coverage. ## What's NOT in this commit Deferred to follow-up PRs: - LISTEN/NOTIFY subscribe (needs real Postgres E2E) - Resource governor (depends on concurrent worker stress testing) - Routing eval harness (needs API keys + benchmark data) - OpenClaw plugin (separate @gbrain/openclaw-minions-plugin repo) See docs/designs/MINIONS_AGENT_ORCHESTRATION.md for full CEO-approved design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): migration v7 — agent_parity_layer schema Adds columns on minion_jobs (depth, max_children, timeout_ms, timeout_at, remove_on_complete, remove_on_fail, idempotency_key) plus the new minion_attachments table. Three partial indexes for bounded scans: idx_minion_jobs_timeout, idx_minion_jobs_parent_status, and uniq_minion_jobs_idempotency. Check constraints enforce non-negative depth and positive child cap / timeout. Additive migration — existing installs pick it up via ensureSchema on next use. No user action required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): extend types for v7 parity layer Extends MinionJob with depth/max_children/timeout_ms/timeout_at/ remove_on_complete/remove_on_fail/idempotency_key. Extends MinionJobInput with the same options plus max_spawn_depth override. Adds MinionQueueOpts (maxSpawnDepth default 5, maxAttachmentBytes default 5 MiB). Adds AttachmentInput/Attachment shapes and ChildDoneMessage in the InboxMessage union. rowToMinionJob updated to pick up the new columns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): attachments validator New module validateAttachment() gates every attachment write. Rejects empty filenames, path traversal (.., /, \), null bytes, oversized content (5 MiB default, per-queue override), invalid base64, and implausible content_type headers. Returns normalized { filename, content_type, content (Buffer), sha256, size } on success. The DB also enforces UNIQUE (job_id, filename) as defense-in-depth for concurrent addAttachment races — JS-only checks are not sufficient. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): queue v7 — depth, child cap, timeouts, cascade, idempotency, child_done Wraps completeJob and failJob in engine.transaction() so parent hook invocations (resolveParent, failParent, removeChildDependency) fold into the same transaction as the child update. A process crash between child and parent can't strand the parent in waiting-children anymore. Adds v7 behaviors: - Depth tracking. add() computes depth = parent.depth + 1 and rejects past maxSpawnDepth (default 5). - Per-parent child cap. add() takes SELECT ... FOR UPDATE on the parent, counts non-terminal children, rejects when count >= max_children. NULL max_children = no cap. - Per-job wall-clock timeout. claim() populates timeout_at when timeout_ms is set. New handleTimeouts() dead-letters expired rows with error_text='timeout exceeded'. Terminal — no retry. - Cascade cancel. cancelJob() walks descendants via recursive CTE with depth-100 runaway cap. Returns the root row. Re-parented descendants (parent_job_id NULL) are naturally excluded. - Idempotency. add() uses INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING; falls back to SELECT when RETURNING is empty. Same key always yields the same job id. - child_done inbox. completeJob inserts {type:'child_done', child_id, job_name, result} into the parent's inbox in the same transaction as the token rollup, guarded by EXISTS so terminal/deleted parents skip without FK violation. New readChildCompletions(parent_id, lock_token, since?) helper; token-fenced like readInbox. - removeOnComplete / removeOnFail. Deletes the row after the parent hook fires, so parent policy sees consistent state. - Attachment methods. addAttachment validates via validateAttachment then INSERTs; UNIQUE (job_id, filename) backs the JS dup check. listAttachments, getAttachment, deleteAttachment round out the API. Fixes pre-existing inverted status bug: add() now puts children in waiting/delayed (not waiting-children) and atomically flips the parent to waiting-children in the same transaction. Tests no longer need manual UPDATE workarounds. Two correctness fixes: - Sibling completion race. Under READ COMMITTED, two grandchildren completing concurrently each saw the other as still-active in the pre-commit snapshot and neither flipped the parent. Fixed by taking SELECT ... FOR UPDATE on the parent row at the start of completeJob and failJob transactions, serializing siblings on the parent lock. - JSONB double-encode. postgres.js conn.unsafe(sql, params) auto- JSON-encodes parameters. Calling JSON.stringify(obj) first stored a JSON string literal (jsonb_typeof=string) and broke payload->>'key' queries silently. Removed JSON.stringify from three call sites (child_done inbox post, updateProgress, sendMessage). PGLite tolerated both forms so unit tests missed it — real-PG E2E caught it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): worker — timeout safety net + handleTimeouts tick Worker tick now calls handleStalled() first, then handleTimeouts() — stall requeue wins over timeout dead-letter when both could fire in the same cycle. handleTimeouts() guards on lock_until > now() so stalled jobs take the retryable path. launchJob schedules a per-job setTimeout(timeout_ms) that fires ctx.signal as a best-effort handler interrupt. The timer is always cleared in .finally so process exit isn't delayed by a dangling timer. Handlers that respect AbortSignal stop cleanly; handlers that ignore it still get dead-lettered by the DB-side handleTimeouts. Removed post-completeJob and post-failJob parent-hook calls from the worker — those are now inside the queue method transactions. Worker becomes simpler and crash-safer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(minions): 33 new unit tests for v7 parity layer Covers depth cap, per-parent child cap, timeout dead-letter, cascade cancel (including the re-parent edge case), removeOnComplete / removeOnFail, idempotency (single + concurrent), child_done inbox (posted in txn + survives child removeOnComplete + since cursor), attachment validation (oversize, path traversal, null byte, duplicates, base64), AbortSignal firing on pause mid-handler, catch-block skipping failJob when aborted, worker in-flight bookkeeping, token-rollup guard when parent already terminal, and setTimeout safety-net cleanup. Existing tests updated to remove the inverted-status manual UPDATE workarounds that the add() fix made obsolete. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(e2e): Minions v7 concurrency + OpenClaw resilience coverage minions-concurrency.test.ts spins two MinionWorker instances against the test Postgres, submits 20 jobs, and asserts zero double-claims (every job runs exactly once). This is the only test that actually proves FOR UPDATE SKIP LOCKED under real concurrency — PGLite runs on a single connection and can't exercise the race. minions-resilience.test.ts covers the six OpenClaw daily pains: 1. Spawn storm caps enforce under concurrent submit. 2. Agent stall → handleStalled() requeues; handleTimeouts() skips (lock_until guard). 3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade cancel stops grandchildren mid-flight. 5. Deep tree fan-in (parent → 3 children → 2 grandchildren each) completes with the full inbox chain. 6. Parent crash/recovery resumes from persisted state. helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and minion_jobs (FK dependents first) so E2E teardown doesn't leak rows between runs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: release v0.11.0 — Minions v7 agent orchestration primitives Bumps VERSION / package.json to 0.11.0. Adds CHANGELOG entry covering depth tracking, max_children, per-job timeouts, cascade cancel, idempotency keys, child_done inbox, removeOnComplete/Fail, attachments, migration v7, plus the two correctness fixes (sibling completion race and JSONB double-encode). TODOS.md captures the four v7 follow-ups: per-queue rate limiting, repeat/cron scheduler, worker event emitter, and waitForChildren convenience helpers. 1066 unit + 105 E2E = 1171 tests passing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): unify JSONB inserts, tighten nullish coalescing Three non-blocker cleanups from post-ship review of v0.11.0: - queue.ts add() and completeJob(): pre-stringifying with JSON.stringify while other sites pass raw objects with $n::jsonb casts. postgres.js double-encodes if you stringify first — works on PGLite (text→JSONB auto-cast), fails silently on real PG. Unify on raw object + explicit $n::jsonb cast. - queue.ts readChildCompletions: since clause used sent_at > $2 relying on PG's implicit text→TIMESTAMPTZ coercion. Explicit $2::timestamptz is safer and clearer. - types.ts rowToMinionJob: parent_job_id used \|\| which coerces 0 to null. Harmless today (SERIAL IDs start at 1) but ?? is semantically correct. All 110 unit tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): updateProgress missed $1::jsonb cast in unification Residual from c502b7e — updateProgress was the only remaining JSONB write without the explicit ::jsonb cast. Not broken (implicit cast works) but breaks the convention the prior commit unified everywhere else. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * doc: Minions v7 skill count + jobs subcommands (26 skills) README: bump skill count 25 → 26, add minion-orchestrator row, add `gbrain jobs` command family block so v0.11.0's headline feature is actually discoverable from the top-level commands reference. CLAUDE.md: unit test count 48 → 49 (minions.test.ts expanded), skill count 25 → 26, add minion-orchestrator to Key files + skills categorization, expand MinionQueue one-liner to cover v7 primitives (depth/child-cap, timeouts, idempotency, child_done inbox, removeOnComplete/Fail). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: Minions adoption UX — smoke test + migration + pain-triggered routing Teach OpenClaw when to reach for Minions vs native subagents. Ship three pieces so upgrading from v0.10.x actually lands for real users: - `gbrain jobs smoke` — one-command health check that submits a `noop` job, runs a worker, verifies completion, and prints engine-aware guidance (PGLite installs get the "daemon needs Postgres, use --follow" note). Fails loud if schema's below v7 so the user knows to `gbrain init`. - `skills/migrations/v0.11.0.md` — post-upgrade migration file the auto-update agent reads. Six steps: apply schema, run smoke, ask user via AskUserQuestion which mode they want (always / pain_triggered / off), write to `~/.gbrain/preferences.json`, sanity-check handlers, mark done. Completeness scores on each option so the recommendation is explicit. - `skills/conventions/subagent-routing.md` rewritten — was a "MUST use Minions for ALL background work" mandate, now reads preferences.json on every routing decision and branches on three modes. Mode B (pain_triggered) is the default: keep subagents until gateway drops state, parallel > 3, runtime > 5min, or user expresses frustration. Then pitch the switch in-session with a specific script. Rename pass: "Minions v7" → "Minions" in README (JOBS block), TODOS.md (P1 section header + depends-on), CHANGELOG.md v0.11.0 entry. v7 stays as the internal schema version in code/migration contexts. The product name is just Minions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * doc(readme): promote Minions — 6 OpenClaw pains + how each is fixed The one-line mention in the skills table wasn't doing the work. Added a dedicated section between "How It Works" and "Getting Data In" that leads with the six multi-agent failures every OpenClaw user hits daily (spawn storms, hung handlers, forgotten dispatches, unstructured debugging, gateway crashes, runaway grandchildren) and maps each pain to the specific Minions primitive that fixes it. Includes the smoke test command, the adoption default (pain_triggered), and a pointer to skills/minion-orchestrator for the full patterns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(bench): add harness for Minions vs OpenClaw subagent dispatch Shared harness (openclawDispatch + minionsHandler) using matching claude-haiku-4-5 calls on both sides so the delta measures queue+ dispatch overhead on top of identical LLM work. Includes statsFromResults (p50/p95/p99) and formatStats helpers. Uses `openclaw agent --local` embedded mode; does not test gateway multi-agent fan-out (documented in the harness header). * test(bench): durability under SIGKILL — Minions vs OpenClaw --local Headline bench for the claim: when the orchestrator dies mid-dispatch, Minions rescues via PG state + stall detection; OpenClaw --local loses in-flight work outright. Minions side: seed 10 active+expired-lock rows (exact state a SIGKILLed worker leaves) then run a rescue worker. Expect 10/10 completed. OpenClaw side: spawn 10 `openclaw agent --local` in parallel, SIGKILL each at 500ms, count pre-kill delivered output. Expect 0/10 — no persistence layer, nothing to recover. Budget: ~$0 (Minions handlers sleep 10ms; OC calls die at 500ms so partial LLM billing is negligible). * test(bench): per-dispatch throughput — Minions vs OpenClaw --local 20 serial dispatches each side, identical claude-haiku-4-5 call with the same trivial prompt. p50/p95/p99 reported via statsFromResults. Serial (not parallel) so the per-dispatch cost is measured honestly and LLM token spend stays bounded (~$0.08 total). Minions: one queue, one worker, one concurrency. Submit → poll to completion before next submit. OpenClaw: N sequential `openclaw agent --local` spawns. * test(bench): fan-out — Minions 10-wide concurrency vs 10 parallel OC spawns Parent dispatches 10 children, waits for all to return. Minions uses worker concurrency=10 sharing one warm process; OpenClaw parallel `openclaw agent --local` spawns, each boots its own runtime. 3 runs × 10 children per run. Reports ok count and wall time per run plus summary. Honest caveat documented: does not test OC gateway multi-agent fan-out — that needs a custom WS client and LLM-backed parent agent. This measures what users script today. Budget: ~$0.12 LLM spend. * test(bench): memory — 10 in-flight subagents, single-proc vs 10-proc cost Measures resident memory for keeping 10 subagents in flight. Minions: one worker process, concurrency=10 with handlers that park on a promise — sample RSS of the test process via process.memoryUsage(). OpenClaw: 10 parallel `openclaw agent --local` processes, sum their RSS via `ps -o rss=`. Handlers are cheap sleeps, no LLM — we want harness memory, not LLM client state. Budget: $0. * test(bench): fan-out — don't gate on OC success rate, report numbers Initial run showed OC parallel `--local` at 10-wide hits 40% failure rate (17/30 across 3 runs). That's the finding, not a test bug — process startup stampede + LLM rate limits. Bench now prints error samples and reports the numbers instead of gating. Minions side still gates at 90% (30/30 observed in practice). * doc(benchmarks): Minions vs OpenClaw --local subagent dispatch Real numbers on four claims: durability, throughput, fan-out, memory. Same claude-haiku-4-5 call on both sides so the delta is queue+dispatch+ process cost on top of identical LLM work. Headline: Minions rescues 10/10 from a SIGKILLed worker in 458ms while OpenClaw --local loses all 10; ~10× faster per dispatch (778ms p50 vs 8086ms p50); ~21× faster at 10-wide fan-out AND 100% reliable vs OC's 43% failure rate; 2 MB vs 814 MB to keep 10 subagents in flight. Honest caveats section covers what this doesn't test (OC gateway multi-agent, load tests, other models). Fully reproducible via test/e2e/bench-vs-openclaw/. * doc(readme): inject Minions vs OpenClaw bench numbers Headline deltas now in the Minions section: 10/10 vs 0/10 on crash, ~10× faster per dispatch, ~21× faster fan-out at 10-wide with 0% failure vs 43%, ~400× less memory. Links to the full bench doc. Prose first said Minions "fixes all six pains." Now it shows the numbers that prove it. * bench: production Wintermute benchmark — Minions 753ms vs sub-agent timeout Real deployment: 45K-page brain on Render+Supabase. Task: pull 99 tweets, write brain page, commit, sync. Minions: 753ms, $0. Sub-agent: gateway timeout (>10s, couldn't even spawn under production load). Also: 19,240 tweets backfilled across 36 months in 15 min at $0. Sub-agents would cost $1.08 and fail 40% of spawns. * bench: tweet ingestion — Minions 719ms vs OpenClaw 12.5s (17×) Production benchmark with runnable test code: - test/e2e/bench-vs-openclaw/tweet-ingest.bench.ts (reusable) - docs/benchmarks/2026-04-18-tweet-ingestion.md (publishable) Task: pull 100 tweets from X API, write brain page, commit, sync. Minions: 719ms mean, $0, 100% success. OpenClaw: 12,480ms mean, $0.03/run, 60% success (gateway timeouts). At scale: 36-month backfill, 19K tweets, 15 min, $0 vs est. $1.08. * doc(benchmarks): Wintermute production data point for Minions vs OpenClaw Adds a production-environment data point to the Minions README section: one month of tweet ingest on Wintermute (Render + Supabase + 45K-page brain) ran end-to-end in 753ms for \$0.00 via Minions, while the equivalent sessions_spawn hit the 10s gateway timeout and produced nothing. Full methodology + logs in docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(core): preferences.ts + cli-util.ts — foundations for v0.11.1 Adds two foundational modules that apply-migrations (Lane A-4), the v0.11.0 orchestrator (Lane C-1), and the stopgap script (Lane C-4) all depend on. - src/core/preferences.ts: atomic-write ~/.gbrain/preferences.json (mktemp + rename, 0o600, forward-compatible for unknown keys) with validateMinionMode, loadPreferences, savePreferences. Plus appendCompletedMigration + loadCompletedMigrations for the ~/.gbrain/migrations/completed.jsonl log (tolerates malformed lines). Uses process.env.HOME \|\| homedir() so $HOME overrides work in CI and tests; Bun's os.homedir() caches the initial value and ignores later mutations. - src/core/cli-util.ts: promptLine(prompt) helper, extracted from src/commands/init.ts:212-224. Shared so init, apply-migrations, and the v0.11.0 orchestrator's mode prompt don't each reinvent it. test/preferences.test.ts: 21 unit tests covering load/save atomicity, 0o600 perms, forward-compat for unknown keys, minion_mode validation, completed.jsonl JSONL append idempotence, auto-ts population, malformed- line tolerance in loadCompletedMigrations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(init): add --migrate-only flag (schema-only, no saveConfig) Context: v0.11.0 migration orchestrators need a safe way to re-apply the schema against an existing brain without risking a config flip. Today running bare `gbrain init` with no flags defaults to PGLite and calls saveConfig, which would silently overwrite an existing Postgres database_url — caught by Codex in the v0.11.1 plan review as a show-stopper data-loss bug. The new --migrate-only path: - loadConfig() reads the existing config (does NOT call saveConfig) - errors out with a clear "run gbrain init first" if no config exists - connects via the already-configured engine, calls engine.initSchema(), disconnects - --json emits structured success/error payloads Everything downstream in the v0.11.1 migration chain (apply-migrations, the stopgap bash script, the package.json postinstall hook) will invoke this flag rather than bare gbrain init. test/init-migrate-only.test.ts: 4 tests covering the no-config error path, --json error payload shape, happy-path with a PGLite fixture (verifies config.json content is byte-identical after the call — the real invariant), and idempotent rerun. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(migrations): TS registry replaces filesystem migration scan Context: Codex flagged that bun build --compile produces a self-contained binary, and the existing findMigrationsDir() in upgrade.ts:145 walks skills/migrations/v.md on disk — which fails on a compiled install because the markdown files aren't bundled. The plan's fix is a TS registry: migrations are code, imported directly, visible to both source installs and compiled binaries. - src/commands/migrations/types.ts: shared Migration, OrchestratorOpts, OrchestratorResult types. - src/commands/migrations/index.ts: exports the migrations[] array, getMigration(version), and compareVersions() (semver comparator). The feature_pitch data that lived in the MD file frontmatter now lives here as a code constant on each Migration, so runPostUpgrade's post-upgrade pitch printer can consume it without a filesystem read. - src/commands/migrations/v0_11_0.ts: stub orchestrator + pitch. The full phase implementation lands in Lane C-1; for now the stub throws a clear "not yet implemented" so apply-migrations --list (Lane A-4) can still enumerate the migration. test/migrations-registry.test.ts: 9 tests covering ascending-semver ordering, feature_pitch shape invariants, getMigration lookup, and compareVersions edge cases (equal / newer / older / single-digit across major bumps). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> feat(cli): gbrain apply-migrations — migration runner CLI Reads ~/.gbrain/migrations/completed.jsonl, diffs against the TS migration registry, runs pending orchestrators. Resumes status:"partial" entries (the stopgap bash script writes these so v0.11.1 apply-migrations can pick up where it left off). Idempotent: rerunning when up-to-date exits 0. Flags: --list Show applied + partial + pending + future. --dry-run Print the plan; take no action. --yes / --non-interactive Skip prompts (used by runPostUpgrade + postinstall). --mode <a\|p\|o> Preset minion_mode (bypasses the Phase C TTY prompt). --migration vX.Y.Z Force-run one specific version. --host-dir <path> Include $PWD in host-file walk (default is $HOME/.claude + $HOME/.openclaw only). --no-autopilot-install Skip Phase F. Diff rule (Codex H9): apply when no status:"complete" entry exists AND migration.version ≤ installed VERSION. Previously proposed rule was "version > currentVersion", which would SKIP v0.11.0 when running v0.11.1; regression test in apply-migrations.test.ts pins the correct semantics. Registered in src/cli.ts CLI_ONLY Set; dispatched before connectEngine so each phase owns its own engine/subprocess lifecycle (no double-connect when the orchestrator shells out to init --migrate-only or jobs smoke). test/apply-migrations.test.ts: 18 unit tests covering parseArgs for every flag, indexCompleted/statusForVersion correctness (including stopgap-then- complete transition), and buildPlan's four buckets (applied / partial / pending / skippedFuture) with the Codex H9 regression pinned. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(upgrade): runPostUpgrade tail-calls apply-migrations; postinstall hook Closes the v0.11.0 mega-bug: migration skills never fired on upgrade. `runPostUpgrade` now does two things: 1. Cosmetic: prints feature_pitch headlines for migrations newer than the prior binary. Uses the TS registry (Codex K) instead of walking skills/migrations/.md on disk — compiled binaries see the same list source installs do. 2. Mechanical: invokes apply-migrations --yes --non-interactive in the same process so Phase F (autopilot install) doesn't hit a subprocess timeout wall. Catches + surfaces errors without failing the upgrade. Also: - Drops the early-return on missing upgrade-state.json (Codex H8). runPostUpgrade now runs apply-migrations unconditionally; it's cheap when nothing is pending. This repairs every broken-v0.11.0 install on their next upgrade attempt. - Bumps the `gbrain post-upgrade` subprocess timeout in runUpgrade from 30s → 300s (Codex H7). A v0.11.0→v0.11.1 migration that has to schema-init + smoke + prefs + host-rewrite + launchd-install exceeds 30s trivially. - Removes now-dead findMigrationsDir + extractFeaturePitch helpers and their filesystem-reading imports (readdirSync, resolve). - src/cli.ts post-upgrade dispatch now awaits the async runPostUpgrade. apply-migrations (Lane A-4): - First-install guard: loadConfig() check at the top. No brain configured = exit silently for --yes / --non-interactive (postinstall stays quiet on fresh `bun add gbrain`); explicit message on --list / --dry-run. package.json: - New `postinstall` script: gbrain --version >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive 2>/dev/null \|\| true. The --version sanity check guards against a half-written binary (Codex review criticism). \|\| true prevents `bun update gbrain` failure mid-upgrade. Manual smoke verified: fresh $HOME with no config → apply-migrations --yes silently exits 0; --dry-run prints the one-liner "No brain configured... Nothing to migrate." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> refactor(commands): extract library-level Core functions that throw not exit Codex architecture finding #5: reusing CLI entry-point functions as Minions handler bodies is wrong. If a Minion invokes runExtract / runEmbed / runBacklinks / runLint and the handler hits a process.exit(1), the ENTIRE WORKER process dies — killing every other in-flight job. Handlers need library-level APIs that throw, and the CLI stays a thin wrapper that catches + exits. Per-command shape: - runXxxCore(opts): throws on validation errors, returns structured result. Handler-safe. - runXxx(args): arg parser; calls Core; catches; process.exit(1) on thrown errors. CLI-safe. Shipped: - runExtractCore({ mode, dir, dryRun?, jsonMode? }) → ExtractResult - runEmbedCore({ slug? \| slugs? \| all? \| stale? }) → void - runBacklinksCore({ action, dir, dryRun? }) → BacklinksResult - runLintCore({ target, fix?, dryRun? }) → LintResult sync.ts is already correct — performSync throws; runSync wraps. No change. import.ts deferred to v0.12.0 (its one process.exit fires only on a missing dir arg; handlers always pass a dir, so worker-kill risk is zero in practice). Noted in the plan's Out-of-scope. Smoke verified: all four Core functions throw on invalid mode / missing dir / not-found target instead of exiting the process. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(jobs): Tier 1 handlers + autopilot-cycle (the killer handler) registerBuiltinHandlers now handlers every operation autopilot needs to dispatch via Minions + the single autopilot-cycle handler the autopilot loop actually submits each interval. Existing handlers (sync, embed, lint) rewired to call library-level Core functions directly instead of the CLI wrappers. CLI wrappers call process.exit(1) on validation errors; if a worker claimed a badly-formed job, the WORKER PROCESS would die — killing every in-flight job. Cores throw, so one bad job fails one job. New handlers: - extract → runExtractCore (mode: links\|timeline\|all, dir) - backlinks → runBacklinksCore (action: check\|fix, dir) - autopilot-cycle → THE killer handler. Runs sync → extract → embed → backlinks inline. Each step wrapped in try/catch; returns { partial: true, failed_steps: [...] } when any step fails. Does NOT throw on partial failure — that would trigger Minion retry, and an intermittent extract bug would block every future cycle. Replaces the 4-job parent-child DAG proposed in early plan drafts (Codex H3/H4: parent/child is NOT a depends_on primitive in Minions). import.ts handler still uses the CLI wrapper (runImport) — import's one process.exit fires only on a missing dir arg and the handler always passes a dir; Core extraction deferred to v0.12.0 when Tier 2 refactors happen. registerBuiltinHandlers promoted from private to exported for testability. test/handlers.test.ts: 4 tests. Asserts every expected handler name registers. Asserts autopilot-cycle against a nonexistent repo returns { partial: true, failed_steps: ['sync', 'extract', 'backlinks'] } — does NOT throw. Asserts autopilot-cycle against an empty (but real) git repo returns a result with a steps map, never throws. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(autopilot): Minions dispatch + worker spawn supervisor + async shutdown Autopilot now dispatches each cycle as a single `autopilot-cycle` Minion job (with idempotency_key on the cycle slot) instead of running steps inline. A forked `gbrain jobs work` child drains the queue durably, supervised by autopilot. The user runs ONE install step (`gbrain autopilot --install`) and gets sync + extract + embed + backlinks + durable job processing, with no separate worker daemon to manage. Mode selection: - minion_mode=always OR pain_triggered (default), engine=postgres → Minions dispatch. Spawn child, submit autopilot-cycle each interval. - minion_mode=off, OR engine=pglite, OR `--inline` flag → run steps inline in-process, same as pre-v0.11.1. PGLite has an exclusive file lock that blocks a second worker process, so the inline path is the only path that works there. Worker supervision: - spawn(resolveGbrainCliPath(), ['jobs', 'work'], { stdio: 'inherit' }). stdio:'inherit' avoids pipe-buffer blocking (Codex architecture #2). - On worker exit: 10s backoff + restart. Crash counter caps at 5 → autopilot stops with a clear error. - resolveGbrainCliPath() prefers argv[1] (cli.ts / /gbrain), then process.execPath (compiled binary suffix check), then `which gbrain` (installed to $PATH). NEVER blindly uses process.execPath, which on source installs is the Bun runtime, not `gbrain` (Codex architecture #1). Shutdown: - Async SIGTERM/SIGINT handler: sends SIGTERM to worker, awaits its exit for up to 35s (the worker's own drain is 30s; we add buffer for signal-delivery latency), then SIGKILL if still alive. - Drops the old `process.on('exit')` lock-cleanup handler — its callback runs synchronously and can't wait for the worker drain. Lock file cleanup moved inside the async shutdown. Lock-file mtime refresh every cycle (Codex C) so a long-lived autopilot doesn't get declared "stale" by the next cron-fired invocation after 10 minutes. Inline fallback path calls the new Core fns (runExtractCore, runEmbedCore) instead of the CLI wrappers. That way a bad arg from inside the loop can't process.exit() the autopilot itself (matches Codex #5). test/autopilot-resolve-cli.test.ts: 3 tests covering argv[1]-as-gbrain, argv[1]-as-cli.ts, and graceful error when no path resolves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(autopilot): env-aware install + OpenClaw bootstrap injection Expand installDaemon from 2 targets (macOS launchd, Linux crontab) to 4: - macos → launchd plist (unchanged) - linux-systemd → ~/.config/systemd/user/gbrain-autopilot.service with Restart=on-failure, RestartSec=30, and an is-system-running probe to confirm the user bus actually works (Codex architecture #7 hardened — the naive /run/systemd/system existence check was a false-positive magnet) - ephemeral-container → detects RENDER / RAILWAY_ENVIRONMENT / FLY_APP_NAME / /.dockerenv. Crontab is unreliable here (wiped on deploy), so we write ~/.gbrain/start-autopilot.sh and tell the user to source it from their agent's bootstrap - linux-cron → existing crontab path (unchanged) detectInstallTarget() + --target flag for explicit override. Also: - --inject-bootstrap / --no-inject control OpenClaw ensure-services.sh auto-injection. Default is ON when OpenClaw is detected (OPENCLAW_HOME env var, openclaw.json in CWD or $HOME, or an ensure-services.sh found). Injection adds ONE line with a `# gbrain:autopilot v0.11.0` marker and writes .bak.<ISO-timestamp> before touching the file. Idempotent — the marker check prevents double injection. uninstallDaemon mirrors all four targets. A user can now run `gbrain autopilot --uninstall` after moving hosts (macOS laptop → Linux server) and the uninstall will find + remove every artifact. writeWrapperScript now uses resolveGbrainCliPath() instead of blindly baking process.execPath into the wrapper script — on source installs that path is the Bun runtime, not gbrain (Codex architecture #1 fix propagated to the install path too). test/autopilot-install.test.ts: 4 tests covering detectInstallTarget's platform + env-var branches. Deeper E2E coverage (systemd unit file contents, ephemeral start-script contents + exec bit, OpenClaw marker injection + .bak) lives in Task 14's E2E fixture test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(migrations): v0.11.0 orchestrator — phases A through G, full implementation Replaces the stub from commit de027ce. The orchestrator runs all seven phases of the v0.11.0 Minions adoption migration idempotently, resumable from any prior status:"partial" run (the stopgap bash script writes those). Phases: A. Schema — `gbrain init --migrate-only` (NEVER bare `gbrain init`, which defaults to PGLite and clobbers existing configs — Codex H1 show-stopper). B. Smoke — `gbrain jobs smoke`. Abort loudly on non-zero. C. Mode — --mode flag wins. Preserved from prefs on resume. Non-TTY or --yes defaults pain_triggered with explicit print. Interactive: numbered 1/2/3 menu via shared promptLine. D. Prefs — savePreferences({minion_mode, set_at, set_in_version}). E. Host — AGENTS.md marker injection + cron manifest rewrites. For cron entries whose skill matches a gbrain builtin (sync/embed/lint/import/extract/backlinks/autopilot-cycle) rewrites kind:agentTurn → kind:shell with a gbrain jobs submit command. PGLite branch keeps --follow (inline execution, the only path that works without a worker daemon); Postgres branch drops --follow + adds --idempotency-key ${handler}:${slot} so long cron jobs don't stack up (same Codex fix as the autopilot-cycle dispatch). For non-builtin handlers (host-specific, like ea-inbox-sweep, frameio-scan, x-dm-triage) emits a structured TODO row to ~/.gbrain/migrations/pending-host-work.jsonl so the host agent can walk through plugin-contract work per skills/migrations/v0.11.0.md. F. Install — `gbrain autopilot --install --yes`. Best-effort (failure doesn't abort; user can run manually). G. Record — append to completed.jsonl. status:"complete" unless pending_host_work > 0, in which case status:"partial" + apply_migrations_pending: true. Safety guards (Codex code-quality tension #3: strict-skip, no rollback): - Scope: $HOME/.claude + $HOME/.openclaw only by default. --host-dir must be explicit to include $PWD or any other path. - Symlink escape: SKIP if the resolved target leaves the scoped root. - >1 MB files: SKIP with warning. - Permission denied: SKIP with warning; other files continue. - Malformed JSON manifest: SKIP with parse error logged; continue. - mtime re-check right before write: bail the file if changed between read + write; other files continue. - Every edit writes a .bak.<ISO-timestamp> sibling first (second- precision so two same-day runs don't collide). - Idempotency: `_gbrain_migrated_by: "v0.11.0"` JSON property marker on each rewritten cron entry (JSON can't have comments — Codex G); AGENTS.md marker `<!-- gbrain:subagent-routing v0.11.0 -->`. - TODO dedupe: JSONL appends deduped by (handler, manifest_path) so reruns don't grow the file. Post-run summary: when pending_host_work > 0, prints a one-liner pointing the user at the JSONL path + the v0.11.0 skill file. The skill (Lane C-3 / C-4) is the host-agent instruction manual. test/migrations-v0_11_0.test.ts: 18 tests covering: - AGENTS.md injection: happy path, .bak creation, idempotent rerun, --dry-run no-op, symlink-escape SKIP, >1MB SKIP. - Cron rewrite: builtin handlers rewrite to shell+gbrain jobs submit, non-builtins emit JSONL TODOs without touching the manifest, mixed manifests get both treatments in one pass, idempotent rerun, TODO dedupe, malformed JSON SKIP, no-entries-array SKIP, --dry-run no-op. - findAgentsMdFiles + findCronManifests: scoped walk to $HOME/.claude + $HOME/.openclaw, --host-dir opt-in for $PWD. - BUILTIN_HANDLERS frozen at the canonical 7 names. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(skill): port skillify from Wintermute, pair with check-resolvable Skillify is the "meta skill": turn any raw feature or script into a properly-skilled, tested, resolvable, evaled unit of agent-visible capability. Proven in production on Wintermute; paired with gbrain's existing `check-resolvable` it becomes a user-controllable equivalent of Hermes' auto-skill-creation — you decide when and what, the tooling keeps the checklist honest. Shipped: - skills/skillify/SKILL.md — ported from ~/git/wintermute/workspace/ skills/skillify/SKILL.md. Genericized: * /data/.openclaw/workspace → \${PROJECT_ROOT} (runtime-detected). * services/voice-agent/__tests__/ → test/ (detected from repo). * Manual `grep skills/... AGENTS.md` replaced with a reference to `gbrain check-resolvable`, which does reachability + MECE + DRY + gap detection properly instead of grep-matching a path string. - scripts/skillify-check.ts — ported from ~/git/wintermute/workspace/scripts/skillify-check.mjs. Preserves the --recent flag and --json output shape. Detects project root via package.json walkup; detects test dir (test/ → __tests__/ → tests/ → spec/). Runs the 10-item checklist per target and exits non-zero if any required item is missing. - test/skillify-check.test.ts — 4 CLI tests: happy-path against publish.ts (known-skilled), --json shape + schema, --recent smoke, bogus-target exit code. - skills/RESOLVER.md — adds the trigger row ("Skillify this", "is this a skill?", "make this proper") → skills/skillify/SKILL.md. - skills/manifest.json — adds the skillify entry so the conformance test passes. Why the pair: * Hermes auto-creates skills in the background. Fine until you don't know what the agent shipped — checklists decay silently. * gbrain ships the same capability as two user-controlled tools: /skillify builds the checklist, gbrain check-resolvable validates reachability + MECE + DRY across the whole skill tree. * Human keeps judgment. Tooling keeps the checklist honest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.11.1): cron-via-minions convention, plugin-handlers guide, minions-fix, skill updates New reference docs: - skills/conventions/cron-via-minions.md — the rewrite convention for cron manifests. Shows the Postgres (fire-and-forget + idempotency- key) vs PGLite (--follow inline) branch; explains why builtin-only auto-rewrite is safe + how host-specific handlers get the plugin contract. - docs/guides/plugin-handlers.md — the plugin contract for host- specific Minion handlers. Code-level registration via import + worker.register(), not a data file (Codex D: handlers.json was an RCE surface). Concrete TypeScript skeleton + handler contract (ctx.data, ctx.signal, ctx.inbox) + full migration flow from TODO JSONL to a rewritten cron entry. - docs/guides/minions-fix.md — user-facing troubleshooting for half-migrated v0.11.0 installs. Paste-one-liner for the stopgap, gbrain apply-migrations path for v0.11.1+, verification commands, failure-mode recipes. Rewrites + updates: - skills/migrations/v0.11.0.md — body restored as the host-agent instruction manual. Audience is the host agent reading ~/.gbrain/migrations/pending-host-work.jsonl after the CLI orchestrator has done the mechanical phases. Walks each TODO type through the 10-item skillify checklist (plugin contract, ship bootstrap, unit tests, integration tests, LLM evals, resolver trigger, trigger eval, E2E smoke, brain filing, check-resolvable). Reverses the earlier "delete the body" decision (1B) because the body serves a different audience now — host-agent, not CLI documentation. - skills/cron-scheduler/SKILL.md — Phase 4 ("Register with host scheduler") now references cron-via-minions + plugin-handlers. - skills/maintain/SKILL.md — new "Fix a half-migrated install" section with the apply-migrations recipe. - skills/setup/SKILL.md — new Phase C.5 "One-step autopilot + Minions install (v0.11.1+)" explaining the four install targets + the OpenClaw auto-injection default. - docs/GBRAIN_SKILLPACK.md — Operations section adds the three new guides + the subagent-routing and cron-routing SKILLPACK notes (v0.11.0+). All 167 related tests (conformance + resolver + skillify-check + v0_11_0 orchestrator) stay green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.11.1): stopgap script + CLAUDE.md directive + README + CHANGELOG + version bump scripts/fix-v0.11.0.sh — the paste-command for broken-v0.11.0 installs. Released on the v0.11.1 tag so: curl -fsSL https://raw.githubusercontent.com/garrytan/gbrain/v0.11.1/scripts/fix-v0.11.0.sh \| bash always works (master branch could be renamed). 8 steps: schema apply, smoke, mode prompt (non-TTY defaults pain_triggered), atomic write of preferences.json (0o600), append completed.jsonl with status:"partial" and apply_migrations_pending:true so the v0.11.1 apply-migrations run resumes correctly (does NOT poison the permanent migration path — Codex H2 avoidance), AGENTS.md + cron/jobs.json detection with guidance printed as text only (never auto-edits from a curl-piped script), and a closing line telling the user to run `gbrain autopilot --install` as the one-stop finisher. CLAUDE.md — new "Migration is canonical, not advisory" section pinning the design principle. Any host-repo change (AGENTS.md, cron manifests, launchctl units) is GBrain's responsibility via the migration; the exception is host-specific handler registration, which goes via the code-level plugin contract in docs/guides/plugin-handlers.md. README.md — new sections: - "v0.11.0 migration didn't fire on your upgrade?" with both repair paths (v0.11.1 binary and pre-v0.11.1 stopgap). - "Skillify + check-resolvable: user-controllable auto-skill-creation" explaining why the user-controlled pair beats Hermes-style auto generation. Includes the scripts/skillify-check.ts invocation. CHANGELOG.md — v0.11.1 entry (per CLAUDE.md voice: lead with what the user can now do that they couldn't before; frame as benefits, not files changed). Covers: mega-bug fix + apply-migrations + postinstall + stopgap, autopilot-supervises-worker + single-install-step + env-aware targets, Core fn extraction so handlers don't kill workers, skillify + check-resolvable pair, host-agnostic plugin contract replacing handlers.json (RCE concern), gbrain init --migrate-only, TS migration registry + H8/H9 diff-rule fixes, CLAUDE.md directive. All Codex hard blockers (H1, H3/H4, H5, H6, H7, H8, H9, K) + architecture issues (#1/#2/#4/#5/#7) resolved. package.json — version bump 0.11.0 → 0.11.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): migration-flow E2E against live Postgres + Bun env quirk fix Ships test/e2e/migration-flow.test.ts — the end-to-end integration test for the v0.11.0 orchestrator. Spins up against a live Postgres (gated on DATABASE_URL per CLAUDE.md lifecycle) and exercises four scenarios: - Fresh install: schema apply (Phase A via `gbrain init --migrate-only`) → smoke (Phase B) → mode resolution (C) → prefs (D) → host rewrite (E, empty fixture) → record (G). Asserts preferences.json exists with 0o600, completed.jsonl has a v0.11.0 entry, autopilot install was skipped per --no-autopilot-install. - Idempotent rerun: second orchestrator invocation on a completed install doesn't blow up; mode stays stable. - Host rewrite mixed manifest: 4-entry cron/jobs.json with 2 gbrain- builtin handlers (sync, embed) + 2 non-builtin (ea-inbox-sweep, morning-briefing). Asserts builtins rewrite to `gbrain jobs submit` kind:shell, non-builtins are LEFT on kind:agentTurn, and 2 JSONL TODOs are emitted with correct shape. AGENTS.md gets the marker injected. Status is "partial" because pending-host-work > 0. - Resumable: stopgap writes a partial completed.jsonl row first; orchestrator re-runs successfully against it and appends a new post-orchestrator entry. 1 partial + 1 complete = 2 rows total. Critical fix surfaced by the E2E: src/commands/migrations/v0_11_0.ts's three execSync calls (gbrain init --migrate-only, gbrain jobs smoke, gbrain autopilot --install) now explicitly pass `env: process.env`. Bun's execSync default does NOT propagate post-start `process.env.PATH` mutations to subprocesses — only the initial PATH snapshot. Without the explicit env, any user-side env tweak (e.g. setting GBRAIN_DATABASE_URL in a script before calling the orchestrator) would be invisible to the orchestrator's subprocesses. This is also the reason the E2E needs a PATH shim installed at module-load time to expose the `gbrain` command. test/init-migrate-only.test.ts: subprocess env now strips DATABASE_URL and GBRAIN_DATABASE_URL. The "no config" error-path tests need loadConfig() to return null, which it won't if the env-var fallback at src/core/config.ts:30 fires. Before this fix, running the unit tests with DATABASE_URL set (e.g. during an E2E run) caused false failures because `gbrain init --migrate-only` saw the env var and succeeded. Full test totals with live Postgres: 1265 pass, 0 fail, 3497 expect calls, 67 files, ~95s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump VERSION file to 0.11.1 Commit 5c4cf1d bumped package.json version to 0.11.1 but missed the root VERSION file. src/version.ts reads from package.json so `gbrain --version` prints 0.11.1 correctly, but any tool or script that reads the VERSION file directly (like /ship's idempotency check) saw the stale 0.11.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.11.1): doctor self-heal check + skillpack-check command for cron health reports Closes the discoverability hole from the v0.11.0 mega-bug: once a user is on v0.11.1 (or later), every `gbrain doctor` invocation immediately surfaces a half-migrated state, and `gbrain skillpack-check` gives host agents (Wintermute's morning-briefing, any OpenClaw cron) a single exit-coded JSON pipe to check from their own skills. gbrain doctor — two new checks: 1. Filesystem-only (fires on every `doctor` invocation, even --fast): if `~/.gbrain/migrations/completed.jsonl` has any status:"partial" entry with no matching status:"complete" for the same version, print `MINIONS HALF-INSTALLED (partial migration: vX.Y.Z). Run: gbrain apply-migrations --yes`. Typical cause is the stopgap wrote a partial record but nobody ran `apply-migrations` afterward. 2. DB-path: if schema version is v7+ (Minions present) AND `~/.gbrain/preferences.json` is missing, print the same banner. Catches installs that never ran the stopgap or apply-migrations at all — the classic v0.11.0 "upgrade landed, migration never fired" state. Both checks status:"fail" so doctor exits non-zero when either fires. Test `test/doctor-minions-check.test.ts` pins the five branches (partial present → FAIL, partial+complete → quiet, no-jsonl → quiet, multiple versions named correctly, human-readable banner contains the exact "MINIONS HALF-INSTALLED" phrase Wintermute's cron can grep for). gbrain skillpack-check — new command + skill: - `src/commands/skillpack-check.ts` wraps `doctor --fast --json` + `apply-migrations --list` into one JSON report with `{healthy, summary, actions[], doctor, migrations}`. Exit 0 on healthy, 1 on action-needed, 2 on determine-failure. `--quiet` flag for cron pipes that want exit-code-only behavior. - `actions[]` is the remediation list. Doctor messages of the form `... Run: <cmd>` get their command extracted (regex fixed to match the full remainder of the line, not just the first word). Pending or partial migrations push `gbrain apply-migrations --yes` to the front of actions[]. - `gbrainSpawn()` helper resolves the gbrain invocation correctly on compiled binary installs (`argv[1] = /usr/local/bin/gbrain`) AND source installs (`argv[1] = src/cli.ts`, prefix with `bun run`). Same Codex #1 fix pattern as autopilot's resolveGbrainCliPath. - `skills/skillpack-check/SKILL.md` teaches agents when to run it, what to do with the output, and anti-patterns (don't run without --quiet in a cron that emails; don't ignore exit 2). - Registered in skills/RESOLVER.md and skills/manifest.json. Test `test/skillpack-check.test.ts` (5 tests) covers healthy fresh install, half-migrated exit-1 with apply-migrations in actions[], --quiet suppresses stdout in both states, --help prints usage, summary includes top action when multiple are present. 1192 unit tests pass (+15 new). The 38 failing tests are all DATABASE_URL E2Es — same pre-existing pattern, unchanged by this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doc(v0.11.1): reframe README + minions-fix — v0.11.0 was never released v0.11.0 was cut but never released publicly. v0.11.1 is the first public Minions ship, and fixes the upgrade-migration mega-bug so it self-heals on every future `gbrain upgrade` + `bun update gbrain`. The README was wrongly framing the fix as a retrospective for v0.11.0 users — none exist, so remove it. README changes: - Delete the "v0.11.0 migration didn't fire on your upgrade?" section. Replace with "Health check and self-heal": the `gbrain doctor`, `gbrain skillpack-check --quiet`, and `gbrain skillpack-check \| jq` recipes that ship in v0.11.1. Still links to docs/guides/minions-fix.md for deeper troubleshooting. - Promote the production benchmark to top billing. The previous section led with the lab benchmark (same LLM, localhost) and buried the production data point as a single follow-up sentence. Real deployment numbers are the stronger signal: * 753ms vs >10s gateway timeout (sub-agent couldn't even spawn) * $0.00 vs ~$0.03 per run * 100% vs 0% success rate under 19-cron production load * 36-month tweet backfill: 19,240 tweets, ~15 min, $0.00 Lab numbers stay (separate table, labeled "controlled environment") so readers can see both layers. - Add the "The routing rule" closer: Deterministic → Minions, Judgment → Sub-agents. This is the clearest framing in the production benchmark doc and belongs in the README so readers leave with the right mental model. `minion_mode: pain_triggered` automates it. docs/guides/minions-fix.md rewrite: - Reframe as: v0.11.0 never released, v0.11.1 is the first ship, `gbrain apply-migrations --yes` is canonical. Stopgap stays documented for pre-v0.11.1 branch builds (e.g. Wintermute's minions-jobs checkout before v0.11.1 tags). - Add the detection + verification commands (doctor + skillpack-check) at the top. - Cross-reference skills/skillpack-check/SKILL.md as the agent-facing health-check pattern. Zero lingering "v0.11.0 released" references in README or minions-fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): remove "schema v7+ no prefs → FAIL" check (too aggressive) CI failure in Tier 1 Mechanical E2E: (fail) E2E: Doctor Command > gbrain doctor exits 0 on healthy DB Root cause: the doctor half-migration detection added two checks. The second check (`schema v7+ AND ~/.gbrain/preferences.json missing → minions_config FAIL`) was too aggressive. It treated a valid fresh- install state as broken. `gbrain init` against Postgres applies schema v7 but doesn't write preferences.json — that's the migration orchestrator's Phase D, which only runs via `apply-migrations`. Between `init` finishing and the user running `apply-migrations`, the install is legitimately in a "schema-applied, no prefs" state. Doctor was exiting 1 on this valid state, breaking the pre-existing CI test that init's + docters a healthy DB. Fix: drop the check. The filesystem check (step 3 — partial-completed without a matching complete) is sufficient signal for genuine half- migration. Added a regression test pinning the exact CI scenario: no completed.jsonl present, no preferences.json, doctor must not fail any minions_* check. Also removes the now-unused `preferencesPaths` import. Verified against live Postgres: CI-equivalent `gbrain doctor` + `gbrain doctor --json` both pass. Full suite: 1281/1281 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doc(readme): Minions section — lead with the story, compress the rest The previous section opened with "six daily pains" as a numbered list before the hook, buried the production numbers halfway down, and had a table explaining how each pain gets fixed. Fine for a spec doc; wrong for a README that needs to land the impact fast. Rewrite: - Lead with "your sub-agents won't drop work anymore" — the reason a reader is here. - Production numbers promoted, framed as a story: "Here's my personal OpenClaw deployment: one Render container, Supabase Postgres holding a 45,000-page brain, 19 cron jobs firing on schedule, the X Enterprise API on the wire..." Gives the reader the setup before the punchline. - The routing rule (deterministic → Minions, judgment → sub-agents) survives unchanged. It's the clearest framing in the whole section. - Lose the "how each pain gets fixed" table. Compress the six pains + their fixes into one paragraph that names the primitives by name (max_children, timeout_ms, child_done inbox, cascade cancel, idempotency keys, attachment validation). Readers who want depth click through to skills/minion-orchestrator/SKILL.md. - Close with "not incrementally better — categorically different" and the three headline numbers. - Drop the separate Lab Numbers table; the production numbers are stronger and the lab data is one click away via the link. Lines: 75 → 42. Same signal, less scroll. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doc: scrub X Enterprise API + @garrytan references from user-facing docs User feedback: shouldn't name the specific enterprise-tier API product or the account in the README or benchmark docs. Genericize: - "X Enterprise API on the wire" → drop entirely; the 19-cron load story carries the setup without naming the vendor - "X Enterprise API ($50K/mo firehose)" → "external API" - "@garrytan tweets" → "my social posts" - "Pull ~100 @garrytan tweets" → "Pull ~100 of my social posts" - "X Enterprise API (full-archive)" env var comment → "external API bearer token" Scope: - README.md — the Minions production story line + scaling callout - docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md - docs/benchmarks/2026-04-18-tweet-ingestion.md Plain "X API" references in the tweet-ingestion methodology stay — those describe which public HTTP endpoint was called, not the enterprise-tier product. Benchmark doc filenames (tweet-ingestion.md) stay to preserve inbound links; content is genericized. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doc(readme): Skillify section — match Minions energy, land the category shift The previous section was competent but undersold what skillify actually is. Rewrite matches the Minions section's shape: lead with the hook, tell the story, land the punchline. Key changes: - Title: "your skills tree stops being a black box." Names the thing skillify actually solves. - Open with the problem: Hermes auto-creates skills as a background behavior. Six months later you have an opaque pile nobody's read or tested. Make the liability concrete. - Promote the 10 items by name (SKILL.md + script + unit tests + integration tests + LLM evals + resolver trigger + trigger eval + E2E + brain filing + check-resolvable audit). Showing the list makes the scope of the unlock visible. - New subsection "Why this is the right answer for OpenClaw" names the debugging-the-black-box pain directly. Skillify makes the tree legible: when something breaks, you know which layer (contract, test, eval, trigger, or route) to inspect. When anything goes stale, check-resolvable flags it. - Close with "compounding quality instead of compounding entropy" + "not a nice-to-have. It's the piece that makes the skills tree survive six months." - Expand the code block to include `gbrain check-resolvable` (the other half of the pair) so readers see the whole workflow. Length goes from 17 to 34 lines — still shorter than Minions, still one section. Worth the space because this is a category shift for how agent skills get built, not a feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: root <root@localhost>	2026-04-18 16:57:38 +08:00
Garry Tan	ce15062694	feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39 ) * docs: break SKILLPACK into 17 individual guides The 1,281-line SKILLPACK monolith is now 17 individually linkable guides in docs/guides/, organized by category: core patterns, data pipelines, operations, search, and administration. GBRAIN_SKILLPACK.md becomes a structured index with categorized tables linking to each guide. The URL stays stable for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add integration guides, architecture docs, and ethos New documentation directories: - docs/integrations/ — "Getting Data In" landing page, credential gateway, meeting webhooks. Includes recipe format documentation. - docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search) - docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide - docs/designs/ — "Homebrew for Personal AI" 10-star vision document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain integrations command + voice-to-brain recipe New CLI command: gbrain integrations (list/show/status/doctor/stats/test) - Standalone command, no database connection needed - Uses gray-matter directly for recipe parsing (not parseMarkdown) - --json flag on every subcommand for agent-parseable output - Bare command shows senses/reflexes dashboard - Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl First recipe: recipes/twilio-voice-brain.md - Phone calls create brain pages via Twilio + OpenAI Realtime - Opinionated defaults: caller screening, brain-first lookup, quiet hours - Outbound call smoke test (GBrain calls the user to prove it works) - Validate-as-you-go credential testing - Twilio signature validation for webhook security Migration file for v0.7.0 with agent-readable changelog. 13 unit tests covering parseRecipe, CLI routing, and recipe validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Getting Data In to README, update CLAUDE.md and manifest README: voice calls in intro bullet list, new "Getting Data In" section with integration table (voice, email, X, calendar) and recipe philosophy. CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/, docs/integrations/, docs/architecture/, docs/ethos/). manifest.json: bump to v0.7.0, add recipes_dir field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.7.0 CHANGELOG, TODOS, VERSION bump CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain, gbrain integrations command, SKILLPACK breakout, and new documentation. TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL, community recipe submission, always-on deployment recipes). VERSION + package.json: bump to 0.7.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite voice recipe with agent instructions and verified links Major improvements to recipes/twilio-voice-brain.md: - Agent preamble: explains WHY sequential execution matters (each step depends on the previous), defines 4 stop points where the agent MUST pause and verify, tells agent to never say "something went wrong" but instead explain the exact error and fix - User actions are now specific: exact URLs for every credential (Twilio console, OpenAI API keys page, ngrok dashboard), what buttons to click, what fields to copy, common failure modes - All URLs verified via web search against current 2026 documentation: Twilio SID/token at twilio.com/console, OpenAI keys at platform.openai.com/api-keys, ngrok token at dashboard.ngrok.com/get-started/your-authtoken - Cost estimate corrected: OpenAI Realtime is $0.06/min input + $0.24/min output (was understated), total ~$20-22/mo for 100 min - Validate-as-you-go: each credential tested immediately with exact curl commands, failure messages explain what went wrong and how to fix - Smoke test flow: tells user exactly what to say, verifies ALL three outputs (messaging notification + brain page + search result) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add "Homebrew for Personal AI" essay (markdown is code) New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution corollary to "Thin Harness, Fat Skills." Argues that markdown skill files are simultaneously documentation, specification, package, and source code. The agent is the package manager. The git repo is the app store. Referenced from SKILLPACK index and CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite agent instructions as command language, promote skills The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide. Every step is an imperative command with exact verification criteria and explicit stop-on-failure behavior. No FYI, no suggestions, just rails. Key changes: - 11-step setup with STOP points after each step - Exact user instructions for Supabase connection string (what to click, what NOT to give the agent, what the string looks like) - "Verify: run X. You must see Y. If not: Z" after every step - Skills table now links to both skill files AND guide docs - Integration recipes table simplified (no "coming soon" placeholders) - Docs section reorganized: for agents / for humans / reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 4 codex findings + add email-to-brain recipe Codex review found 4 issues, all fixed: 1. getStatus() returned "configured" if ANY secret was set (e.g. just OPENAI_API_KEY). Now requires ALL required secrets before marking configured. Prevents false "configured" status and spurious doctor runs. 2. Twilio health check hit unauthenticated endpoint (always 401). Now uses authenticated curl with SID:token, matching the setup validation. 3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md. 4. Compiled binary can't find recipes/ via import.meta.dir. Added GBRAIN_RECIPES_DIR env var override + global bun install path fallback. Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern with ClawVisor credential gateway, validate-as-you-go, agent instructions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add email, X, calendar, and meeting sync recipes Four new integration recipes extracted from production wintermute patterns: - recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector pattern (code pulls emails with baked-in links, agent does judgment), noise filtering, signature detection, digest generation - recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search, deletion detection (diffs previous run, verifies 404), engagement velocity tracking, rate limit awareness - recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical backfill (years of data), daily markdown files with attendees + locations, attendee enrichment for brain pages - recipes/meeting-sync.md: Circleback API, transcript import with speaker labels, attendee detection + filtering, entity propagation to people/ company pages, action item extraction, idempotent by source_id All recipes follow the same format: agent preamble with sequential execution rules, validate-as-you-go credentials, exact URLs for API key setup, stop-on-failure verification, and heartbeat logging. Updated README, SKILLPACK index, and integrations landing page with all 5 recipes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes Both recipes now offer two auth options: - Option A: ClawVisor (recommended, handles OAuth + token refresh) - Option B: Google OAuth2 directly (no extra service, you manage tokens) Option B includes step-by-step instructions for Google Cloud Console: exact URLs, which buttons to click, which scopes to add, how to enable the API, and the OAuth flow for token exchange. This removes ClawVisor as a hard dependency for getting started. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation guides with pseudocode and test suggestions Every recipe now includes an "Implementation Guide" section with: - Production-tested pseudocode the agent can follow to build each collector - Edge cases and failure modes discovered in real deployment - Non-obvious implementation details (why the 48h staleness heuristic, why Gmail links need authuser, why SSE responses need double-parsing) - Test suggestions: what the agent should verify after setup email-to-brain: noise filtering algorithm, signature detection patterns, Gmail link generation (authuser is critical), sent-mail dedup x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness, API verification), engagement velocity thresholds (50 min for 2x, 100 absolute jump), atomic writes, stdout contract, rate limit handling calendar-to-brain: smart chunking (monthly for sparse years, weekly for dense), attendee filtering (rooms, groups, distros), merge-with-existing (only replace ## Calendar section), date/time parsing edge cases meeting-sync: SSE double-JSON parsing, idempotency double-check (grep + filename), auto-tagging from meeting names, git commit after sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: 6 new guides from production patterns (wintermute extraction) New guides extracted and generalized from production deployment: - repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge). Strict boundary rules, decision tree, hard rule: never write knowledge to the agent repo. - sub-agent-routing.md: Model routing table by task type. Signal detector pattern (spawn Sonnet on every message). Research pipeline pattern (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization. - skill-development.md: 5-step cycle (concept, prototype, evaluate, codify, cron). MECE discipline (no overlapping skills). Quality bar checklist. "If you ask twice, it should already be a skill." - idea-capture.md: Originality distribution rating (0-100 across 4 populations). Depth test ("could someone unfamiliar understand WHY?"). Deep cross-linking mandate. Notability filtering. - quiet-hours.md: Hold notifications 11pm-8am local time. Held messages directory pattern. Timezone-aware delivery. Morning briefing pickup. - diligence-ingestion.md: 9-step pipeline for data room materials. Detection patterns (PDF filenames, spreadsheet tabs, user language). Index.md template with bull/bear case. Company page enrichment. All PII scrubbed. Patterns generalized for any user. SKILLPACK index updated with 6 new entries. CLAUDE.md references added. All 37 SKILLPACK links verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: upgrade all guides to operational playbooks with pseudocode Every guide now follows the playbook structure: - Goal: one sentence, what this achieves - What the User Gets: without this / with this - Implementation: pseudocode with actual gbrain commands - Tricky Spots: production-tested gotchas - How to Verify: test steps the agent runs after setup Guides upgraded (15 files): - brain-agent-loop: on_message() loop with read/write/sync pseudocode - brain-first-lookup: 4-step lookup cascade with exact commands - brain-vs-memory: routing algorithm for 3 knowledge layers - compiled-truth: page structure + rewrite vs append rules - content-media: 3 ingest patterns (YouTube, social, PDFs) - cron-schedule: full schedule table + dream cycle pseudocode - enrichment-pipeline: 7-step protocol with tier classification - entity-detection: spawn pattern + detection prompt + notability filter - executive-assistant: 3 workflow algorithms (triage, prep, post-inbox) - meeting-ingestion: 6-step transcript-to-brain flow - operational-disciplines: 5 executable discipline blocks - originals-folder: detection + exact-phrasing capture + cross-linking - search-modes: decision tree for keyword vs hybrid vs direct - source-attribution: citation format + hierarchy + conflict resolution - Plus Goal/What User Gets headers on 6 newer guides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add WebRTC to voice recipe + ngrok Hobby setup guide Voice recipe updates: - Added WebRTC endpoint (POST /session, GET /call, POST /tool) for browser-based calling with RNNoise noise suppression - WebRTC pseudocode with the 4 non-obvious gotchas from production (voice under audio.output.voice, no turn_detection, no session.update on connect, trigger greeting via data channel) - Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier - Fixed domain means URLs never change, Twilio never breaks New guide: docs/mcp/NGROK_SETUP.md - How to set up ngrok Hobby for both MCP and voice agent - Fixed domain setup, watchdog pattern, AI client configuration - Claude Desktop requires Settings > Integrations (not JSON config) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency graph + ngrok-tunnel + credential-gateway recipes Recipes now have real dependencies via the `requires` field: - voice-to-brain requires ngrok-tunnel (needs public URL for Twilio) - email-to-brain requires credential-gateway (needs Gmail access) - calendar-to-brain requires credential-gateway (needs Calendar access) - x-to-brain and meeting-sync are standalone (direct API keys) Two new infrastructure recipes: - ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby ($8/mo) for a domain that never changes. Includes watchdog pattern. - credential-gateway: secure Google service access via ClawVisor (recommended) or direct OAuth2. One setup, all Google recipes use it. Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure, not MCP-specific. README and integrations landing page show dependency chains. When agent installs voice-to-brain, it sets up ngrok-tunnel first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add infra category, fix dashboard alignment, show dependencies DX audit found two bugs in gbrain integrations dashboard: 1. Column alignment broken — IDs > 18 chars ran into descriptions with no space. Fixed: pad to 22 chars. 2. ngrok-tunnel and credential-gateway showed as SENSES but they're infrastructure. Added 'infra' category. Dashboard now shows three sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES. 3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)" Also added 'requires' field to JSON output for agent consumption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add frontier model requirement disclaimer to README GBrain's markdown-is-code approach requires models capable of interpreting intent and implementing from architecture descriptions. Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models will struggle with the recipe format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add PGLite → Supabase upgrade path to README Clarify the database progression: start with PGLite (Postgres as WASM, zero infrastructure, pgvector built in, nothing to install). Graduate to Supabase or self-hosted Postgres when you need connection pooling, concurrency, and remote MCP access from Claude Desktop, Cowork, ChatGPT, Perplexity Computer, or any MCP-compatible agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: revert PGLite mention (coming in next branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify) Every guide now has exactly these sections in this order: - ## Goal (one sentence) - ## What the User Gets (without this / with this) - ## Implementation (pseudocode with gbrain commands) - ## Tricky Spots (3-5 numbered gotchas) - ## How to Verify (3-5 numbered test steps) 11 guides restructured from non-standard headings: - deterministic-collectors, live-sync, upgrades-auto-update (full rewrites) - entity-detection, diligence-ingestion, idea-capture, quiet-hours, repo-architecture, skill-development, sub-agent-routing (restructured) 23/23 guides now pass consistency audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: restructure README around the #1 blocker (getting data in) The README was leading with Postgres and database architecture. Most users are stuck at step zero: "I have an agent but it doesn't know anything about my life." New structure: 1. The Problem — your agent doesn't know your life 2. Getting Data In — integration recipes, front and center 3. The Compounding Thesis — why this matters 4. How this happened — credibility, origin story 5. When you need Postgres — scale, not starting point Postgres is de-emphasized from a full section to two paragraphs: "You don't need Postgres to start" and "When you need Postgres" (1,000+ files, remote MCP access, multiple AI clients). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move Install to top of README, remove duplicate section Install now appears right after Getting Data In (line 38), not buried at line 295. The user sees: Problem → Getting Data In → Install. Removed the duplicate Install section (262 lines) that was lower in the README. The agent instructions block, CLI quickstart, and all content is now in the single Install section near the top. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move agent install block to first thing in README "Start here: paste this into your agent" is now the first section, right after the one-line pitch. No scrolling, no context, no preamble. User opens the README, sees the paste block, copies it into OpenClaw or Hermes, and the agent takes over. Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: compress install block from 11 steps to 5 The agent install block was 102 lines and 11 steps. Now it's 40 lines and 5 steps. Same coverage, half the text. Changes: - Merged "prove keyword search" + "embed" + "prove hybrid search" into one SEARCH step (the user doesn't care about the intermediate) - Merged skillpack, sync, auto-update, integrations, verification into one GO LIVE step with sub-items (post-install polish, not install) - Shortened database instructions (one line instead of 5 sub-steps) - Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now just "Do not skip steps. Verify each step.") The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: gitignore all .env files, not just specific ones CSO audit found .gitignore covered .env.testing and .env.production but not bare .env. A user creating .env with database credentials could accidentally commit it. Fix: .env and .env.* are now gitignored. .env..example files are explicitly un-ignored so templates remain tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> security: scrub PII from essay and recipe examples - 510-MY-GARRY phone mnemonic → "Your Phone Number" - "Garry → Authenticated Mode" → "Owner → Authenticated Mode" - "Telegram" → "secure channel" in auth example - @garrytan → @yourhandle in X recipe example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-10 23:39:06 -10:00

22 Commits