triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
81 Commits 2 Branches 0 Tags
b5fa3d044ae25d08f23068c1e2a00ec77405ccce
Commit Graph

9 Commits

Author SHA1 Message Date
Garry Tan
d8613366a5 Minions v7 + v0.11.1 canonical migration + skillify (#130) 
* feat: add minion_jobs schema, migration v5, and executeRaw to BrainEngine

Foundation for the Minions job queue system. Adds:
- minion_jobs table (20 columns) with CHECK constraints, partial indexes,
  and RLS. Inspired by BullMQ's job model, adapted for Postgres.
- Migration v5 creates the table for existing databases.
- executeRaw<T>() method on BrainEngine interface for raw SQL access,
  needed by the Minions module for claim queries (FOR UPDATE SKIP LOCKED),
  token-fenced writes, and atomic stall detection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: Minions job queue — queue, worker, backoff, types

BullMQ-inspired Postgres-native job queue built into GBrain. No Redis.
No external dependencies. Postgres transactions replace Lua scripts.

- MinionQueue: submit, claim (FOR UPDATE SKIP LOCKED), complete/fail
  (token-fenced), atomic stall detection (CTE), delayed promotion,
  parent-child resolution, prune, stats
- MinionWorker: handler registry, lock renewal, graceful SIGTERM,
  exponential backoff with jitter, UnrecoverableError bypass
- MinionJobContext: updateProgress(), log(), isActive() for handlers
- 8-state machine: waiting/active/completed/failed/delayed/dead/
  cancelled/waiting-children

Patterns stolen from: BullMQ (lock tokens, stall detection, flows),
Sidekiq (dead set, backoff formula), Inngest (checkpoint/resume).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: 43 tests for Minions job queue

Full coverage of the Minions module against PGLite in-memory:
- Queue CRUD (9): submit, get, list, remove, cancel, retry, duplicate
- State machine (6): waiting→active→completed/failed, retry→delayed→waiting
- Backoff (4): exponential, fixed, jitter range, attempts_made=0 edge
- Stall detection (3): detect stalled, counter increment, max→dead
- Dependencies (5): parent waits, fail_parent, continue, remove_dep, orphan
- Worker lifecycle (5): register, start-without-handlers, claim+execute,
  non-Error throws, UnrecoverableError bypass
- Lock management (3): renewal, token mismatch, claim sets lock fields
- Claim mechanics (4): empty queue, priority ordering, name filtering,
  delayed promotion timing
- Cancel & retry (2): cancel active, retry dead

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: Minions CLI commands and MCP operations

Wire Minions into the GBrain CLI and MCP layer:

CLI (gbrain jobs):
  submit <name> [--params JSON] [--follow] [--dry-run]
  list [--status S] [--queue Q] [--limit N]
  get <id> — detailed view with attempt history
  cancel/retry/delete <id>
  prune [--older-than 30d]
  stats — job health dashboard
  work [--queue Q] [--concurrency N] — Postgres-only worker daemon

6 MCP operations (contract-first, auto-exposed via MCP server):
  submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress

Built-in handlers: sync, embed, lint, import. --follow runs inline.
Worker daemon blocked on PGLite (exclusive file lock).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for Minions job queue

CLAUDE.md: added Minions files to key files, updated operation count (36),
BrainEngine method count (38), test file count (45), added jobs CLI commands.
CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall
detection, worker daemon).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: Minions v2 — agent orchestration primitives (pause/resume, inbox, tokens, replay)

Adds the foundation for Minions as universal agent orchestration infrastructure.
GBrain's Postgres-native job queue now supports durable, observable, steerable
background agents. The OpenClaw plugin (separate repo) will consume these via
library import, not MCP, for zero-latency local integration.

## New capabilities

- **Concurrent worker** — Promise pool replaces sequential loop. Per-job
  AbortController for cooperative cancellation. Graceful shutdown waits for
  all in-flight jobs via Promise.allSettled.
- **Pause/resume** — pauseJob clears the lock and fires AbortSignal on active
  jobs. Handlers check ctx.signal.aborted and exit cleanly. resumeJob returns
  paused jobs to waiting. Catch block skips failJob when signal.aborted.
- **Inbox (separate table)** — minion_inbox table for sidechannel messages.
  sendMessage with sender validation (parent job or admin). readInbox is
  token-fenced and marks read_at atomically. Separate table avoids row bloat
  from rewriting JSONB on every send.
- **Token accounting** — tokens_input/tokens_output/tokens_cache_read columns.
  updateTokens accumulates; completeJob rolls child tokens up to parent.
  USD cost computed at read time (no cost_usd column — pricing too volatile).
- **Job replay** — replayJob clones a terminal job with optional data overrides.
  New job, fresh attempts, no parent link.

## Handler contract additions

MinionJobContext now provides:
- `signal: AbortSignal` — cooperative cancellation
- `updateTokens(tokens)` — accumulate token usage
- `readInbox()` — check for sidechannel messages
- `log()` — now accepts string or TranscriptEntry

## MCP operations added

pause_job, resume_job, replay_job, send_job_message — all auto-generate CLI
commands and MCP server endpoints.

## Library exports

package.json exports map adds ./minions and ./engine-factory paths so plugins
can `import { MinionQueue } from 'gbrain/minions'` for direct library use.

## Instruction layer (the teaching)

- skills/minion-orchestrator/SKILL.md — when/how to use Minions, decision
  matrix, lifecycle management, anti-patterns
- skills/conventions/subagent-routing.md — cross-cutting rule: all background
  work goes through Minions
- RESOLVER.md — trigger entries for agent orchestration
- manifest.json — registered

## Schema migration v6

Additive: 3 token columns, paused status, minion_inbox table with unread index.
Full Postgres + PGLite support. No backfill needed.

## Tests

65 tests (was 43): pause/resume (5), inbox (6), tokens (4), replay (4),
concurrent worker context (3), plus all existing coverage.

## What's NOT in this commit

Deferred to follow-up PRs:
- LISTEN/NOTIFY subscribe (needs real Postgres E2E)
- Resource governor (depends on concurrent worker stress testing)
- Routing eval harness (needs API keys + benchmark data)
- OpenClaw plugin (separate @gbrain/openclaw-minions-plugin repo)

See docs/designs/MINIONS_AGENT_ORCHESTRATION.md for full CEO-approved design.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(minions): migration v7 — agent_parity_layer schema

Adds columns on minion_jobs (depth, max_children, timeout_ms, timeout_at,
remove_on_complete, remove_on_fail, idempotency_key) plus the new
minion_attachments table. Three partial indexes for bounded scans:
idx_minion_jobs_timeout, idx_minion_jobs_parent_status, and
uniq_minion_jobs_idempotency. Check constraints enforce non-negative depth
and positive child cap / timeout.

Additive migration — existing installs pick it up via ensureSchema on next
use. No user action required.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(minions): extend types for v7 parity layer

Extends MinionJob with depth/max_children/timeout_ms/timeout_at/
remove_on_complete/remove_on_fail/idempotency_key. Extends MinionJobInput
with the same options plus max_spawn_depth override. Adds MinionQueueOpts
(maxSpawnDepth default 5, maxAttachmentBytes default 5 MiB). Adds
AttachmentInput/Attachment shapes and ChildDoneMessage in the InboxMessage
union. rowToMinionJob updated to pick up the new columns.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(minions): attachments validator

New module validateAttachment() gates every attachment write. Rejects empty
filenames, path traversal (.., /, \), null bytes, oversized content (5 MiB
default, per-queue override), invalid base64, and implausible content_type
headers. Returns normalized { filename, content_type, content (Buffer),
sha256, size } on success.

The DB also enforces UNIQUE (job_id, filename) as defense-in-depth for
concurrent addAttachment races — JS-only checks are not sufficient.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(minions): queue v7 — depth, child cap, timeouts, cascade, idempotency, child_done

Wraps completeJob and failJob in engine.transaction() so parent hook
invocations (resolveParent, failParent, removeChildDependency) fold into
the same transaction as the child update. A process crash between child
and parent can't strand the parent in waiting-children anymore.

Adds v7 behaviors:
- Depth tracking. add() computes depth = parent.depth + 1 and rejects
  past maxSpawnDepth (default 5).
- Per-parent child cap. add() takes SELECT ... FOR UPDATE on the parent,
  counts non-terminal children, rejects when count >= max_children.
  NULL max_children = no cap.
- Per-job wall-clock timeout. claim() populates timeout_at when
  timeout_ms is set. New handleTimeouts() dead-letters expired rows with
  error_text='timeout exceeded'. Terminal — no retry.
- Cascade cancel. cancelJob() walks descendants via recursive CTE with
  depth-100 runaway cap. Returns the root row. Re-parented descendants
  (parent_job_id NULL) are naturally excluded.
- Idempotency. add() uses INSERT ... ON CONFLICT (idempotency_key) DO
  NOTHING RETURNING; falls back to SELECT when RETURNING is empty. Same
  key always yields the same job id.
- child_done inbox. completeJob inserts {type:'child_done', child_id,
  job_name, result} into the parent's inbox in the same transaction as
  the token rollup, guarded by EXISTS so terminal/deleted parents skip
  without FK violation. New readChildCompletions(parent_id, lock_token,
  since?) helper; token-fenced like readInbox.
- removeOnComplete / removeOnFail. Deletes the row after the parent hook
  fires, so parent policy sees consistent state.
- Attachment methods. addAttachment validates via validateAttachment
  then INSERTs; UNIQUE (job_id, filename) backs the JS dup check.
  listAttachments, getAttachment, deleteAttachment round out the API.

Fixes pre-existing inverted status bug: add() now puts children in
waiting/delayed (not waiting-children) and atomically flips the parent
to waiting-children in the same transaction. Tests no longer need
manual UPDATE workarounds.

Two correctness fixes:
- Sibling completion race. Under READ COMMITTED, two grandchildren
  completing concurrently each saw the other as still-active in the
  pre-commit snapshot and neither flipped the parent. Fixed by taking
  SELECT ... FOR UPDATE on the parent row at the start of completeJob
  and failJob transactions, serializing siblings on the parent lock.
- JSONB double-encode. postgres.js conn.unsafe(sql, params) auto-
  JSON-encodes parameters. Calling JSON.stringify(obj) first stored a
  JSON string literal (jsonb_typeof=string) and broke payload->>'key'
  queries silently. Removed JSON.stringify from three call sites
  (child_done inbox post, updateProgress, sendMessage). PGLite tolerated
  both forms so unit tests missed it — real-PG E2E caught it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(minions): worker — timeout safety net + handleTimeouts tick

Worker tick now calls handleStalled() first, then handleTimeouts() — stall
requeue wins over timeout dead-letter when both could fire in the same
cycle. handleTimeouts() guards on lock_until > now() so stalled jobs take
the retryable path.

launchJob schedules a per-job setTimeout(timeout_ms) that fires ctx.signal
as a best-effort handler interrupt. The timer is always cleared in .finally
so process exit isn't delayed by a dangling timer. Handlers that respect
AbortSignal stop cleanly; handlers that ignore it still get dead-lettered
by the DB-side handleTimeouts.

Removed post-completeJob and post-failJob parent-hook calls from the worker
— those are now inside the queue method transactions. Worker becomes
simpler and crash-safer.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(minions): 33 new unit tests for v7 parity layer

Covers depth cap, per-parent child cap, timeout dead-letter, cascade
cancel (including the re-parent edge case), removeOnComplete /
removeOnFail, idempotency (single + concurrent), child_done inbox
(posted in txn + survives child removeOnComplete + since cursor),
attachment validation (oversize, path traversal, null byte, duplicates,
base64), AbortSignal firing on pause mid-handler, catch-block skipping
failJob when aborted, worker in-flight bookkeeping, token-rollup guard
when parent already terminal, and setTimeout safety-net cleanup.

Existing tests updated to remove the inverted-status manual UPDATE
workarounds that the add() fix made obsolete.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(e2e): Minions v7 concurrency + OpenClaw resilience coverage

minions-concurrency.test.ts spins two MinionWorker instances against the
test Postgres, submits 20 jobs, and asserts zero double-claims (every job
runs exactly once). This is the only test that actually proves FOR UPDATE
SKIP LOCKED under real concurrency — PGLite runs on a single connection
and can't exercise the race.

minions-resilience.test.ts covers the six OpenClaw daily pains:
1. Spawn storm caps enforce under concurrent submit. 2. Agent stall →
handleStalled() requeues; handleTimeouts() skips (lock_until guard).
3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade
cancel stops grandchildren mid-flight. 5. Deep tree fan-in
(parent → 3 children → 2 grandchildren each) completes with the full
inbox chain. 6. Parent crash/recovery resumes from persisted state.

helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and
minion_jobs (FK dependents first) so E2E teardown doesn't leak rows
between runs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: release v0.11.0 — Minions v7 agent orchestration primitives

Bumps VERSION / package.json to 0.11.0. Adds CHANGELOG entry covering
depth tracking, max_children, per-job timeouts, cascade cancel,
idempotency keys, child_done inbox, removeOnComplete/Fail, attachments,
migration v7, plus the two correctness fixes (sibling completion race
and JSONB double-encode).

TODOS.md captures the four v7 follow-ups: per-queue rate limiting,
repeat/cron scheduler, worker event emitter, and waitForChildren
convenience helpers.

1066 unit + 105 E2E = 1171 tests passing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(minions): unify JSONB inserts, tighten nullish coalescing

Three non-blocker cleanups from post-ship review of v0.11.0:

- queue.ts add() and completeJob(): pre-stringifying with JSON.stringify
  while other sites pass raw objects with $n::jsonb casts. postgres.js
  double-encodes if you stringify first — works on PGLite (text→JSONB
  auto-cast), fails silently on real PG. Unify on raw object + explicit
  $n::jsonb cast.
- queue.ts readChildCompletions: since clause used sent_at > $2 relying
  on PG's implicit text→TIMESTAMPTZ coercion. Explicit $2::timestamptz
  is safer and clearer.
- types.ts rowToMinionJob: parent_job_id used || which coerces 0 to null.
  Harmless today (SERIAL IDs start at 1) but ?? is semantically correct.

All 110 unit tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(minions): updateProgress missed $1::jsonb cast in unification

Residual from c502b7e — updateProgress was the only remaining JSONB write
without the explicit ::jsonb cast. Not broken (implicit cast works) but
breaks the convention the prior commit unified everywhere else.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* doc: Minions v7 skill count + jobs subcommands (26 skills)

README: bump skill count 25 → 26, add minion-orchestrator row, add
`gbrain jobs` command family block so v0.11.0's headline feature is
actually discoverable from the top-level commands reference.

CLAUDE.md: unit test count 48 → 49 (minions.test.ts expanded), skill
count 25 → 26, add minion-orchestrator to Key files + skills categorization,
expand MinionQueue one-liner to cover v7 primitives (depth/child-cap,
timeouts, idempotency, child_done inbox, removeOnComplete/Fail).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: Minions adoption UX — smoke test + migration + pain-triggered routing

Teach OpenClaw when to reach for Minions vs native subagents. Ship three
pieces so upgrading from v0.10.x actually lands for real users:

- `gbrain jobs smoke` — one-command health check that submits a `noop` job,
  runs a worker, verifies completion, and prints engine-aware guidance
  (PGLite installs get the "daemon needs Postgres, use --follow" note).
  Fails loud if schema's below v7 so the user knows to `gbrain init`.

- `skills/migrations/v0.11.0.md` — post-upgrade migration file the
  auto-update agent reads. Six steps: apply schema, run smoke, ask user
  via AskUserQuestion which mode they want (always / pain_triggered / off),
  write to `~/.gbrain/preferences.json`, sanity-check handlers, mark done.
  Completeness scores on each option so the recommendation is explicit.

- `skills/conventions/subagent-routing.md` rewritten — was a "MUST use
  Minions for ALL background work" mandate, now reads preferences.json
  on every routing decision and branches on three modes. Mode B
  (pain_triggered) is the default: keep subagents until gateway drops
  state, parallel > 3, runtime > 5min, or user expresses frustration.
  Then pitch the switch in-session with a specific script.

Rename pass: "Minions v7" → "Minions" in README (JOBS block), TODOS.md
(P1 section header + depends-on), CHANGELOG.md v0.11.0 entry. v7 stays
as the internal schema version in code/migration contexts. The product
name is just Minions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* doc(readme): promote Minions — 6 OpenClaw pains + how each is fixed

The one-line mention in the skills table wasn't doing the work. Added a
dedicated section between "How It Works" and "Getting Data In" that leads
with the six multi-agent failures every OpenClaw user hits daily (spawn
storms, hung handlers, forgotten dispatches, unstructured debugging,
gateway crashes, runaway grandchildren) and maps each pain to the
specific Minions primitive that fixes it.

Includes the smoke test command, the adoption default (pain_triggered),
and a pointer to skills/minion-orchestrator for the full patterns.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(bench): add harness for Minions vs OpenClaw subagent dispatch

Shared harness (openclawDispatch + minionsHandler) using matching
claude-haiku-4-5 calls on both sides so the delta measures queue+
dispatch overhead on top of identical LLM work. Includes
statsFromResults (p50/p95/p99) and formatStats helpers. Uses
`openclaw agent --local` embedded mode; does not test gateway
multi-agent fan-out (documented in the harness header).

* test(bench): durability under SIGKILL — Minions vs OpenClaw --local

Headline bench for the claim: when the orchestrator dies mid-dispatch,
Minions rescues via PG state + stall detection; OpenClaw --local loses
in-flight work outright.

Minions side: seed 10 active+expired-lock rows (exact state a SIGKILLed
worker leaves) then run a rescue worker. Expect 10/10 completed.
OpenClaw side: spawn 10 `openclaw agent --local` in parallel, SIGKILL
each at 500ms, count pre-kill delivered output. Expect 0/10 — no
persistence layer, nothing to recover.

Budget: ~$0 (Minions handlers sleep 10ms; OC calls die at 500ms so
partial LLM billing is negligible).

* test(bench): per-dispatch throughput — Minions vs OpenClaw --local

20 serial dispatches each side, identical claude-haiku-4-5 call with the
same trivial prompt. p50/p95/p99 reported via statsFromResults. Serial
(not parallel) so the per-dispatch cost is measured honestly and LLM
token spend stays bounded (~$0.08 total).

Minions: one queue, one worker, one concurrency. Submit → poll to
completion before next submit. OpenClaw: N sequential
`openclaw agent --local` spawns.

* test(bench): fan-out — Minions 10-wide concurrency vs 10 parallel OC spawns

Parent dispatches 10 children, waits for all to return. Minions uses
worker concurrency=10 sharing one warm process; OpenClaw parallel
`openclaw agent --local` spawns, each boots its own runtime.

3 runs × 10 children per run. Reports ok count and wall time per run
plus summary. Honest caveat documented: does not test OC gateway
multi-agent fan-out — that needs a custom WS client and LLM-backed
parent agent. This measures what users script today.

Budget: ~$0.12 LLM spend.

* test(bench): memory — 10 in-flight subagents, single-proc vs 10-proc cost

Measures resident memory for keeping 10 subagents in flight. Minions:
one worker process, concurrency=10 with handlers that park on a
promise — sample RSS of the test process via process.memoryUsage().
OpenClaw: 10 parallel `openclaw agent --local` processes, sum their
RSS via `ps -o rss=`.

Handlers are cheap sleeps, no LLM — we want harness memory, not LLM
client state. Budget: $0.

* test(bench): fan-out — don't gate on OC success rate, report numbers

Initial run showed OC parallel `--local` at 10-wide hits 40% failure
rate (17/30 across 3 runs). That's the finding, not a test bug —
process startup stampede + LLM rate limits. Bench now prints error
samples and reports the numbers instead of gating.

Minions side still gates at 90% (30/30 observed in practice).

* doc(benchmarks): Minions vs OpenClaw --local subagent dispatch

Real numbers on four claims: durability, throughput, fan-out, memory.
Same claude-haiku-4-5 call on both sides so the delta is queue+dispatch+
process cost on top of identical LLM work.

Headline: Minions rescues 10/10 from a SIGKILLed worker in 458ms while
OpenClaw --local loses all 10; ~10× faster per dispatch (778ms p50 vs
8086ms p50); ~21× faster at 10-wide fan-out AND 100% reliable vs OC's
43% failure rate; 2 MB vs 814 MB to keep 10 subagents in flight.

Honest caveats section covers what this doesn't test (OC gateway
multi-agent, load tests, other models). Fully reproducible via
test/e2e/bench-vs-openclaw/.

* doc(readme): inject Minions vs OpenClaw bench numbers

Headline deltas now in the Minions section: 10/10 vs 0/10 on crash,
~10× faster per dispatch, ~21× faster fan-out at 10-wide with 0%
failure vs 43%, ~400× less memory. Links to the full bench doc.

Prose first said Minions "fixes all six pains." Now it shows the
numbers that prove it.

* bench: production Wintermute benchmark — Minions 753ms vs sub-agent timeout

Real deployment: 45K-page brain on Render+Supabase. Task: pull 99 tweets,
write brain page, commit, sync. Minions: 753ms, $0. Sub-agent: gateway
timeout (>10s, couldn't even spawn under production load).

Also: 19,240 tweets backfilled across 36 months in 15 min at $0.
Sub-agents would cost $1.08 and fail 40% of spawns.

* bench: tweet ingestion — Minions 719ms vs OpenClaw 12.5s (17×)

Production benchmark with runnable test code:
- test/e2e/bench-vs-openclaw/tweet-ingest.bench.ts (reusable)
- docs/benchmarks/2026-04-18-tweet-ingestion.md (publishable)

Task: pull 100 tweets from X API, write brain page, commit, sync.
Minions: 719ms mean, $0, 100% success.
OpenClaw: 12,480ms mean, $0.03/run, 60% success (gateway timeouts).
At scale: 36-month backfill, 19K tweets, 15 min, $0 vs est. $1.08.

* doc(benchmarks): Wintermute production data point for Minions vs OpenClaw

Adds a production-environment data point to the Minions README section:
one month of tweet ingest on Wintermute (Render + Supabase + 45K-page brain)
ran end-to-end in 753ms for \$0.00 via Minions, while the equivalent
sessions_spawn hit the 10s gateway timeout and produced nothing.

Full methodology + logs in docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(core): preferences.ts + cli-util.ts — foundations for v0.11.1

Adds two foundational modules that apply-migrations (Lane A-4), the
v0.11.0 orchestrator (Lane C-1), and the stopgap script (Lane C-4) all
depend on.

- src/core/preferences.ts: atomic-write ~/.gbrain/preferences.json
  (mktemp + rename, 0o600, forward-compatible for unknown keys) with
  validateMinionMode, loadPreferences, savePreferences. Plus
  appendCompletedMigration + loadCompletedMigrations for the
  ~/.gbrain/migrations/completed.jsonl log (tolerates malformed lines).
  Uses process.env.HOME || homedir() so $HOME overrides work in CI and
  tests; Bun's os.homedir() caches the initial value and ignores later
  mutations.
- src/core/cli-util.ts: promptLine(prompt) helper, extracted from
  src/commands/init.ts:212-224. Shared so init, apply-migrations, and
  the v0.11.0 orchestrator's mode prompt don't each reinvent it.

test/preferences.test.ts: 21 unit tests covering load/save atomicity,
0o600 perms, forward-compat for unknown keys, minion_mode validation,
completed.jsonl JSONL append idempotence, auto-ts population, malformed-
line tolerance in loadCompletedMigrations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(init): add --migrate-only flag (schema-only, no saveConfig)

Context: v0.11.0 migration orchestrators need a safe way to re-apply the
schema against an existing brain without risking a config flip. Today
running bare `gbrain init` with no flags defaults to PGLite and calls
saveConfig, which would silently overwrite an existing Postgres
database_url — caught by Codex in the v0.11.1 plan review as a
show-stopper data-loss bug.

The new --migrate-only path:
  - loadConfig() reads the existing config (does NOT call saveConfig)
  - errors out with a clear "run gbrain init first" if no config exists
  - connects via the already-configured engine, calls engine.initSchema(),
    disconnects
  - --json emits structured success/error payloads

Everything downstream in the v0.11.1 migration chain (apply-migrations,
the stopgap bash script, the package.json postinstall hook) will invoke
this flag rather than bare gbrain init.

test/init-migrate-only.test.ts: 4 tests covering the no-config error
path, --json error payload shape, happy-path with a PGLite fixture
(verifies config.json content is byte-identical after the call — the
real invariant), and idempotent rerun.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(migrations): TS registry replaces filesystem migration scan

Context: Codex flagged that bun build --compile produces a self-contained
binary, and the existing findMigrationsDir() in upgrade.ts:145 walks
skills/migrations/v*.md on disk — which fails on a compiled install
because the markdown files aren't bundled. The plan's fix is a TS
registry: migrations are code, imported directly, visible to both source
installs and compiled binaries.

- src/commands/migrations/types.ts: shared Migration, OrchestratorOpts,
  OrchestratorResult types.
- src/commands/migrations/index.ts: exports the migrations[] array,
  getMigration(version), and compareVersions() (semver comparator).
  The feature_pitch data that lived in the MD file frontmatter now
  lives here as a code constant on each Migration, so runPostUpgrade's
  post-upgrade pitch printer can consume it without a filesystem read.
- src/commands/migrations/v0_11_0.ts: stub orchestrator + pitch. The
  full phase implementation lands in Lane C-1; for now the stub throws
  a clear "not yet implemented" so apply-migrations --list (Lane A-4)
  can still enumerate the migration.

test/migrations-registry.test.ts: 9 tests covering ascending-semver
ordering, feature_pitch shape invariants, getMigration lookup, and
compareVersions edge cases (equal / newer / older / single-digit
across major bumps).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): gbrain apply-migrations — migration runner CLI

Reads ~/.gbrain/migrations/completed.jsonl, diffs against the TS migration
registry, runs pending orchestrators. Resumes status:"partial" entries
(the stopgap bash script writes these so v0.11.1 apply-migrations can
pick up where it left off). Idempotent: rerunning when up-to-date exits 0.

Flags:
  --list                    Show applied + partial + pending + future.
  --dry-run                 Print the plan; take no action.
  --yes / --non-interactive Skip prompts (used by runPostUpgrade + postinstall).
  --mode <a|p|o>            Preset minion_mode (bypasses the Phase C TTY prompt).
  --migration vX.Y.Z        Force-run one specific version.
  --host-dir <path>         Include $PWD in host-file walk (default is
                            $HOME/.claude + $HOME/.openclaw only).
  --no-autopilot-install    Skip Phase F.

Diff rule (Codex H9): apply when no status:"complete" entry exists AND
migration.version ≤ installed VERSION. Previously proposed rule was
"version > currentVersion", which would SKIP v0.11.0 when running v0.11.1;
regression test in apply-migrations.test.ts pins the correct semantics.

Registered in src/cli.ts CLI_ONLY Set; dispatched before connectEngine so
each phase owns its own engine/subprocess lifecycle (no double-connect
when the orchestrator shells out to init --migrate-only or jobs smoke).

test/apply-migrations.test.ts: 18 unit tests covering parseArgs for every
flag, indexCompleted/statusForVersion correctness (including stopgap-then-
complete transition), and buildPlan's four buckets (applied / partial /
pending / skippedFuture) with the Codex H9 regression pinned.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(upgrade): runPostUpgrade tail-calls apply-migrations; postinstall hook

Closes the v0.11.0 mega-bug: migration skills never fired on upgrade.
`runPostUpgrade` now does two things:

  1. Cosmetic: prints feature_pitch headlines for migrations newer than
     the prior binary. Uses the TS registry (Codex K) instead of walking
     skills/migrations/*.md on disk — compiled binaries see the same list
     source installs do.
  2. Mechanical: invokes apply-migrations --yes --non-interactive in the
     same process so Phase F (autopilot install) doesn't hit a subprocess
     timeout wall. Catches + surfaces errors without failing the upgrade.

Also:
  - Drops the early-return on missing upgrade-state.json (Codex H8).
    runPostUpgrade now runs apply-migrations unconditionally; it's cheap
    when nothing is pending. This repairs every broken-v0.11.0 install on
    their next upgrade attempt.
  - Bumps the `gbrain post-upgrade` subprocess timeout in runUpgrade from
    30s → 300s (Codex H7). A v0.11.0→v0.11.1 migration that has to
    schema-init + smoke + prefs + host-rewrite + launchd-install exceeds
    30s trivially.
  - Removes now-dead findMigrationsDir + extractFeaturePitch helpers and
    their filesystem-reading imports (readdirSync, resolve).
  - src/cli.ts post-upgrade dispatch now awaits the async runPostUpgrade.

apply-migrations (Lane A-4):
  - First-install guard: loadConfig() check at the top. No brain
    configured = exit silently for --yes / --non-interactive (postinstall
    stays quiet on fresh `bun add gbrain`); explicit message on --list /
    --dry-run.

package.json:
  - New `postinstall` script: gbrain --version >/dev/null 2>&1 && gbrain
    apply-migrations --yes --non-interactive 2>/dev/null || true. The
    --version sanity check guards against a half-written binary (Codex
    review criticism). || true prevents `bun update gbrain` failure
    mid-upgrade.

Manual smoke verified: fresh $HOME with no config → apply-migrations
--yes silently exits 0; --dry-run prints the one-liner "No brain
configured... Nothing to migrate."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(commands): extract library-level Core functions that throw not exit

Codex architecture finding #5: reusing CLI entry-point functions as Minions
handler bodies is wrong. If a Minion invokes runExtract / runEmbed /
runBacklinks / runLint and the handler hits a process.exit(1), the ENTIRE
WORKER process dies — killing every other in-flight job. Handlers need
library-level APIs that throw, and the CLI stays a thin wrapper that
catches + exits.

Per-command shape:
  - runXxxCore(opts): throws on validation errors, returns structured
    result. Handler-safe.
  - runXxx(args): arg parser; calls Core; catches; process.exit(1) on
    thrown errors. CLI-safe.

Shipped:
  - runExtractCore({ mode, dir, dryRun?, jsonMode? }) → ExtractResult
  - runEmbedCore({ slug? | slugs? | all? | stale? }) → void
  - runBacklinksCore({ action, dir, dryRun? }) → BacklinksResult
  - runLintCore({ target, fix?, dryRun? }) → LintResult

sync.ts is already correct — performSync throws; runSync wraps. No change.

import.ts deferred to v0.12.0 (its one process.exit fires only on a
missing dir arg; handlers always pass a dir, so worker-kill risk is
zero in practice). Noted in the plan's Out-of-scope.

Smoke verified: all four Core functions throw on invalid mode / missing
dir / not-found target instead of exiting the process.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(jobs): Tier 1 handlers + autopilot-cycle (the killer handler)

registerBuiltinHandlers now handlers every operation autopilot needs to
dispatch via Minions + the single autopilot-cycle handler the autopilot
loop actually submits each interval.

Existing handlers (sync, embed, lint) rewired to call library-level Core
functions directly instead of the CLI wrappers. CLI wrappers call
process.exit(1) on validation errors; if a worker claimed a badly-formed
job, the WORKER PROCESS would die — killing every in-flight job. Cores
throw, so one bad job fails one job.

New handlers:
  - extract  → runExtractCore (mode: links|timeline|all, dir)
  - backlinks → runBacklinksCore (action: check|fix, dir)
  - autopilot-cycle → THE killer handler. Runs sync → extract → embed →
    backlinks inline. Each step wrapped in try/catch; returns
    { partial: true, failed_steps: [...] } when any step fails. Does NOT
    throw on partial failure — that would trigger Minion retry, and an
    intermittent extract bug would block every future cycle. Replaces
    the 4-job parent-child DAG proposed in early plan drafts (Codex
    H3/H4: parent/child is NOT a depends_on primitive in Minions).

import.ts handler still uses the CLI wrapper (runImport) — import's one
process.exit fires only on a missing dir arg and the handler always
passes a dir; Core extraction deferred to v0.12.0 when Tier 2 refactors
happen.

registerBuiltinHandlers promoted from private to exported for testability.

test/handlers.test.ts: 4 tests. Asserts every expected handler name
registers. Asserts autopilot-cycle against a nonexistent repo returns
{ partial: true, failed_steps: ['sync', 'extract', 'backlinks'] } — does
NOT throw. Asserts autopilot-cycle against an empty (but real) git repo
returns a result with a steps map, never throws.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): Minions dispatch + worker spawn supervisor + async shutdown

Autopilot now dispatches each cycle as a single `autopilot-cycle` Minion
job (with idempotency_key on the cycle slot) instead of running steps
inline. A forked `gbrain jobs work` child drains the queue durably,
supervised by autopilot. The user runs ONE install step
(`gbrain autopilot --install`) and gets sync + extract + embed + backlinks
+ durable job processing, with no separate worker daemon to manage.

Mode selection:
  - minion_mode=always OR pain_triggered (default), engine=postgres →
    Minions dispatch. Spawn child, submit autopilot-cycle each interval.
  - minion_mode=off, OR engine=pglite, OR `--inline` flag → run steps
    inline in-process, same as pre-v0.11.1. PGLite has an exclusive file
    lock that blocks a second worker process, so the inline path is the
    only path that works there.

Worker supervision:
  - spawn(resolveGbrainCliPath(), ['jobs', 'work'], { stdio: 'inherit' }).
    stdio:'inherit' avoids pipe-buffer blocking (Codex architecture #2).
  - On worker exit: 10s backoff + restart. Crash counter caps at 5 →
    autopilot stops with a clear error.
  - resolveGbrainCliPath() prefers argv[1] (cli.ts / /gbrain), then
    process.execPath (compiled binary suffix check), then `which gbrain`
    (installed to $PATH). NEVER blindly uses process.execPath, which on
    source installs is the Bun runtime, not `gbrain` (Codex architecture
    #1).

Shutdown:
  - Async SIGTERM/SIGINT handler: sends SIGTERM to worker, awaits its
    exit for up to 35s (the worker's own drain is 30s; we add buffer for
    signal-delivery latency), then SIGKILL if still alive.
  - Drops the old `process.on('exit')` lock-cleanup handler — its
    callback runs synchronously and can't wait for the worker drain.
    Lock file cleanup moved inside the async shutdown.

Lock-file mtime refresh every cycle (Codex C) so a long-lived autopilot
doesn't get declared "stale" by the next cron-fired invocation after 10
minutes.

Inline fallback path calls the new Core fns (runExtractCore, runEmbedCore)
instead of the CLI wrappers. That way a bad arg from inside the loop
can't process.exit() the autopilot itself (matches Codex #5).

test/autopilot-resolve-cli.test.ts: 3 tests covering argv[1]-as-gbrain,
argv[1]-as-cli.ts, and graceful error when no path resolves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(autopilot): env-aware install + OpenClaw bootstrap injection

Expand installDaemon from 2 targets (macOS launchd, Linux crontab) to 4:

  - macos              → launchd plist (unchanged)
  - linux-systemd      → ~/.config/systemd/user/gbrain-autopilot.service
                         with Restart=on-failure, RestartSec=30, and an
                         is-system-running probe to confirm the user bus
                         actually works (Codex architecture #7 hardened —
                         the naive /run/systemd/system existence check was
                         a false-positive magnet)
  - ephemeral-container → detects RENDER / RAILWAY_ENVIRONMENT /
                          FLY_APP_NAME / /.dockerenv. Crontab is unreliable
                          here (wiped on deploy), so we write
                          ~/.gbrain/start-autopilot.sh and tell the user
                          to source it from their agent's bootstrap
  - linux-cron         → existing crontab path (unchanged)

detectInstallTarget() + --target flag for explicit override. Also:
  - --inject-bootstrap / --no-inject control OpenClaw ensure-services.sh
    auto-injection. Default is ON when OpenClaw is detected (OPENCLAW_HOME
    env var, openclaw.json in CWD or $HOME, or an ensure-services.sh
    found). Injection adds ONE line with a `# gbrain:autopilot v0.11.0`
    marker and writes .bak.<ISO-timestamp> before touching the file.
    Idempotent — the marker check prevents double injection.

uninstallDaemon mirrors all four targets. A user can now run
`gbrain autopilot --uninstall` after moving hosts (macOS laptop → Linux
server) and the uninstall will find + remove every artifact.

writeWrapperScript now uses resolveGbrainCliPath() instead of blindly
baking process.execPath into the wrapper script — on source installs
that path is the Bun runtime, not gbrain (Codex architecture #1 fix
propagated to the install path too).

test/autopilot-install.test.ts: 4 tests covering detectInstallTarget's
platform + env-var branches. Deeper E2E coverage (systemd unit file
contents, ephemeral start-script contents + exec bit, OpenClaw marker
injection + .bak) lives in Task 14's E2E fixture test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(migrations): v0.11.0 orchestrator — phases A through G, full implementation

Replaces the stub from commit de027ce. The orchestrator runs all seven
phases of the v0.11.0 Minions adoption migration idempotently, resumable
from any prior status:"partial" run (the stopgap bash script writes
those).

Phases:
  A. Schema  — `gbrain init --migrate-only` (NEVER bare `gbrain init`,
               which defaults to PGLite and clobbers existing configs —
               Codex H1 show-stopper).
  B. Smoke   — `gbrain jobs smoke`. Abort loudly on non-zero.
  C. Mode    — --mode flag wins. Preserved from prefs on resume. Non-TTY
               or --yes defaults pain_triggered with explicit print.
               Interactive: numbered 1/2/3 menu via shared promptLine.
  D. Prefs   — savePreferences({minion_mode, set_at, set_in_version}).
  E. Host    — AGENTS.md marker injection + cron manifest rewrites. For
               cron entries whose skill matches a gbrain builtin
               (sync/embed/lint/import/extract/backlinks/autopilot-cycle)
               rewrites kind:agentTurn → kind:shell with a
               gbrain jobs submit command. PGLite branch keeps --follow
               (inline execution, the only path that works without a
               worker daemon); Postgres branch drops --follow + adds
               --idempotency-key ${handler}:${slot} so long cron jobs
               don't stack up (same Codex fix as the autopilot-cycle
               dispatch). For non-builtin handlers (host-specific, like
               ea-inbox-sweep, frameio-scan, x-dm-triage) emits a
               structured TODO row to
               ~/.gbrain/migrations/pending-host-work.jsonl so the host
               agent can walk through plugin-contract work per
               skills/migrations/v0.11.0.md.
  F. Install — `gbrain autopilot --install --yes`. Best-effort (failure
               doesn't abort; user can run manually).
  G. Record  — append to completed.jsonl. status:"complete" unless
               pending_host_work > 0, in which case status:"partial" +
               apply_migrations_pending: true.

Safety guards (Codex code-quality tension #3: strict-skip, no rollback):
  - Scope: $HOME/.claude + $HOME/.openclaw only by default. --host-dir
    must be explicit to include $PWD or any other path.
  - Symlink escape: SKIP if the resolved target leaves the scoped root.
  - >1 MB files: SKIP with warning.
  - Permission denied: SKIP with warning; other files continue.
  - Malformed JSON manifest: SKIP with parse error logged; continue.
  - mtime re-check right before write: bail the file if changed between
    read + write; other files continue.
  - Every edit writes a .bak.<ISO-timestamp> sibling first (second-
    precision so two same-day runs don't collide).
  - Idempotency: `_gbrain_migrated_by: "v0.11.0"` JSON property marker
    on each rewritten cron entry (JSON can't have comments — Codex G);
    AGENTS.md marker `<!-- gbrain:subagent-routing v0.11.0 -->`.
  - TODO dedupe: JSONL appends deduped by (handler, manifest_path) so
    reruns don't grow the file.

Post-run summary: when pending_host_work > 0, prints a one-liner
pointing the user at the JSONL path + the v0.11.0 skill file. The skill
(Lane C-3 / C-4) is the host-agent instruction manual.

test/migrations-v0_11_0.test.ts: 18 tests covering:
  - AGENTS.md injection: happy path, .bak creation, idempotent rerun,
    --dry-run no-op, symlink-escape SKIP, >1MB SKIP.
  - Cron rewrite: builtin handlers rewrite to shell+gbrain jobs submit,
    non-builtins emit JSONL TODOs without touching the manifest, mixed
    manifests get both treatments in one pass, idempotent rerun, TODO
    dedupe, malformed JSON SKIP, no-entries-array SKIP, --dry-run no-op.
  - findAgentsMdFiles + findCronManifests: scoped walk to $HOME/.claude +
    $HOME/.openclaw, --host-dir opt-in for $PWD.
  - BUILTIN_HANDLERS frozen at the canonical 7 names.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(skill): port skillify from Wintermute, pair with check-resolvable

Skillify is the "meta skill": turn any raw feature or script into a
properly-skilled, tested, resolvable, evaled unit of agent-visible
capability. Proven in production on Wintermute; paired with gbrain's
existing `check-resolvable` it becomes a user-controllable equivalent of
Hermes' auto-skill-creation — you decide when and what, the tooling
keeps the checklist honest.

Shipped:
  - skills/skillify/SKILL.md — ported from ~/git/wintermute/workspace/
    skills/skillify/SKILL.md. Genericized:
      * /data/.openclaw/workspace → \${PROJECT_ROOT} (runtime-detected).
      * services/voice-agent/__tests__/ → test/ (detected from repo).
      * Manual `grep skills/... AGENTS.md` replaced with a reference to
        `gbrain check-resolvable`, which does reachability + MECE + DRY
        + gap detection properly instead of grep-matching a path string.
  - scripts/skillify-check.ts — ported from
    ~/git/wintermute/workspace/scripts/skillify-check.mjs. Preserves the
    --recent flag and --json output shape. Detects project root via
    package.json walkup; detects test dir (test/ → __tests__/ → tests/
    → spec/). Runs the 10-item checklist per target and exits non-zero
    if any required item is missing.
  - test/skillify-check.test.ts — 4 CLI tests: happy-path against
    publish.ts (known-skilled), --json shape + schema, --recent smoke,
    bogus-target exit code.
  - skills/RESOLVER.md — adds the trigger row ("Skillify this", "is
    this a skill?", "make this proper") → skills/skillify/SKILL.md.
  - skills/manifest.json — adds the skillify entry so the conformance
    test passes.

Why the pair:
  * Hermes auto-creates skills in the background. Fine until you don't
    know what the agent shipped — checklists decay silently.
  * gbrain ships the same capability as two user-controlled tools:
    /skillify builds the checklist, gbrain check-resolvable validates
    reachability + MECE + DRY across the whole skill tree.
  * Human keeps judgment. Tooling keeps the checklist honest.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.11.1): cron-via-minions convention, plugin-handlers guide, minions-fix, skill updates

New reference docs:
  - skills/conventions/cron-via-minions.md — the rewrite convention for
    cron manifests. Shows the Postgres (fire-and-forget + idempotency-
    key) vs PGLite (--follow inline) branch; explains why builtin-only
    auto-rewrite is safe + how host-specific handlers get the plugin
    contract.
  - docs/guides/plugin-handlers.md — the plugin contract for host-
    specific Minion handlers. Code-level registration via import +
    worker.register(), not a data file (Codex D: handlers.json was an
    RCE surface). Concrete TypeScript skeleton + handler contract
    (ctx.data, ctx.signal, ctx.inbox) + full migration flow from TODO
    JSONL to a rewritten cron entry.
  - docs/guides/minions-fix.md — user-facing troubleshooting for
    half-migrated v0.11.0 installs. Paste-one-liner for the stopgap,
    gbrain apply-migrations path for v0.11.1+, verification commands,
    failure-mode recipes.

Rewrites + updates:
  - skills/migrations/v0.11.0.md — body restored as the host-agent
    instruction manual. Audience is the host agent reading
    ~/.gbrain/migrations/pending-host-work.jsonl after the CLI
    orchestrator has done the mechanical phases. Walks each TODO type
    through the 10-item skillify checklist (plugin contract, ship
    bootstrap, unit tests, integration tests, LLM evals, resolver
    trigger, trigger eval, E2E smoke, brain filing, check-resolvable).
    Reverses the earlier "delete the body" decision (1B) because the
    body serves a different audience now — host-agent, not CLI
    documentation.
  - skills/cron-scheduler/SKILL.md — Phase 4 ("Register with host
    scheduler") now references cron-via-minions + plugin-handlers.
  - skills/maintain/SKILL.md — new "Fix a half-migrated install"
    section with the apply-migrations recipe.
  - skills/setup/SKILL.md — new Phase C.5 "One-step autopilot +
    Minions install (v0.11.1+)" explaining the four install targets
    + the OpenClaw auto-injection default.
  - docs/GBRAIN_SKILLPACK.md — Operations section adds the three new
    guides + the subagent-routing and cron-routing SKILLPACK notes
    (v0.11.0+).

All 167 related tests (conformance + resolver + skillify-check + v0_11_0
orchestrator) stay green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.11.1): stopgap script + CLAUDE.md directive + README + CHANGELOG + version bump

scripts/fix-v0.11.0.sh — the paste-command for broken-v0.11.0 installs.
Released on the v0.11.1 tag so:
  curl -fsSL https://raw.githubusercontent.com/garrytan/gbrain/v0.11.1/scripts/fix-v0.11.0.sh | bash
always works (master branch could be renamed). 8 steps: schema apply,
smoke, mode prompt (non-TTY defaults pain_triggered), atomic write of
preferences.json (0o600), append completed.jsonl with status:"partial"
and apply_migrations_pending:true so the v0.11.1 apply-migrations run
resumes correctly (does NOT poison the permanent migration path —
Codex H2 avoidance), AGENTS.md + cron/jobs.json detection with guidance
printed as text only (never auto-edits from a curl-piped script), and a
closing line telling the user to run `gbrain autopilot --install` as the
one-stop finisher.

CLAUDE.md — new "Migration is canonical, not advisory" section pinning
the design principle. Any host-repo change (AGENTS.md, cron manifests,
launchctl units) is GBrain's responsibility via the migration; the
exception is host-specific handler registration, which goes via the
code-level plugin contract in docs/guides/plugin-handlers.md.

README.md — new sections:
  - "v0.11.0 migration didn't fire on your upgrade?" with both repair
    paths (v0.11.1 binary and pre-v0.11.1 stopgap).
  - "Skillify + check-resolvable: user-controllable auto-skill-creation"
    explaining why the user-controlled pair beats Hermes-style auto
    generation. Includes the scripts/skillify-check.ts invocation.

CHANGELOG.md — v0.11.1 entry (per CLAUDE.md voice: lead with what the
user can now do that they couldn't before; frame as benefits, not files
changed). Covers: mega-bug fix + apply-migrations + postinstall +
stopgap, autopilot-supervises-worker + single-install-step + env-aware
targets, Core fn extraction so handlers don't kill workers, skillify +
check-resolvable pair, host-agnostic plugin contract replacing
handlers.json (RCE concern), gbrain init --migrate-only, TS migration
registry + H8/H9 diff-rule fixes, CLAUDE.md directive. All Codex hard
blockers (H1, H3/H4, H5, H6, H7, H8, H9, K) + architecture issues
(#1/#2/#4/#5/#7) resolved.

package.json — version bump 0.11.0 → 0.11.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): migration-flow E2E against live Postgres + Bun env quirk fix

Ships test/e2e/migration-flow.test.ts — the end-to-end integration test
for the v0.11.0 orchestrator. Spins up against a live Postgres (gated
on DATABASE_URL per CLAUDE.md lifecycle) and exercises four scenarios:

  - Fresh install: schema apply (Phase A via `gbrain init --migrate-only`)
    → smoke (Phase B) → mode resolution (C) → prefs (D) → host rewrite
    (E, empty fixture) → record (G). Asserts preferences.json exists with
    0o600, completed.jsonl has a v0.11.0 entry, autopilot install was
    skipped per --no-autopilot-install.
  - Idempotent rerun: second orchestrator invocation on a completed
    install doesn't blow up; mode stays stable.
  - Host rewrite mixed manifest: 4-entry cron/jobs.json with 2 gbrain-
    builtin handlers (sync, embed) + 2 non-builtin (ea-inbox-sweep,
    morning-briefing). Asserts builtins rewrite to `gbrain jobs submit`
    kind:shell, non-builtins are LEFT on kind:agentTurn, and 2 JSONL
    TODOs are emitted with correct shape. AGENTS.md gets the marker
    injected. Status is "partial" because pending-host-work > 0.
  - Resumable: stopgap writes a partial completed.jsonl row first;
    orchestrator re-runs successfully against it and appends a new
    post-orchestrator entry. 1 partial + 1 complete = 2 rows total.

Critical fix surfaced by the E2E: src/commands/migrations/v0_11_0.ts's
three execSync calls (gbrain init --migrate-only, gbrain jobs smoke,
gbrain autopilot --install) now explicitly pass `env: process.env`.
Bun's execSync default does NOT propagate post-start `process.env.PATH`
mutations to subprocesses — only the initial PATH snapshot. Without the
explicit env, any user-side env tweak (e.g. setting GBRAIN_DATABASE_URL
in a script before calling the orchestrator) would be invisible to the
orchestrator's subprocesses. This is also the reason the E2E needs a
PATH shim installed at module-load time to expose the `gbrain` command.

test/init-migrate-only.test.ts: subprocess env now strips DATABASE_URL
and GBRAIN_DATABASE_URL. The "no config" error-path tests need
loadConfig() to return null, which it won't if the env-var fallback at
src/core/config.ts:30 fires. Before this fix, running the unit tests
with DATABASE_URL set (e.g. during an E2E run) caused false failures
because `gbrain init --migrate-only` saw the env var and succeeded.

Full test totals with live Postgres: 1265 pass, 0 fail, 3497 expect
calls, 67 files, ~95s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump VERSION file to 0.11.1

Commit 5c4cf1d bumped package.json version to 0.11.1 but missed the
root VERSION file. src/version.ts reads from package.json so
`gbrain --version` prints 0.11.1 correctly, but any tool or script
that reads the VERSION file directly (like /ship's idempotency check)
saw the stale 0.11.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.11.1): doctor self-heal check + skillpack-check command for cron health reports

Closes the discoverability hole from the v0.11.0 mega-bug: once a user is
on v0.11.1 (or later), every `gbrain doctor` invocation immediately
surfaces a half-migrated state, and `gbrain skillpack-check` gives host
agents (Wintermute's morning-briefing, any OpenClaw cron) a single
exit-coded JSON pipe to check from their own skills.

gbrain doctor — two new checks:
  1. Filesystem-only (fires on every `doctor` invocation, even --fast):
     if `~/.gbrain/migrations/completed.jsonl` has any status:"partial"
     entry with no matching status:"complete" for the same version, print
     `MINIONS HALF-INSTALLED (partial migration: vX.Y.Z). Run: gbrain
     apply-migrations --yes`. Typical cause is the stopgap wrote a
     partial record but nobody ran `apply-migrations` afterward.
  2. DB-path: if schema version is v7+ (Minions present) AND
     `~/.gbrain/preferences.json` is missing, print the same banner.
     Catches installs that never ran the stopgap or apply-migrations at
     all — the classic v0.11.0 "upgrade landed, migration never fired"
     state.

Both checks status:"fail" so doctor exits non-zero when either fires.
Test `test/doctor-minions-check.test.ts` pins the five branches
(partial present → FAIL, partial+complete → quiet, no-jsonl → quiet,
multiple versions named correctly, human-readable banner contains the
exact "MINIONS HALF-INSTALLED" phrase Wintermute's cron can grep for).

gbrain skillpack-check — new command + skill:
  - `src/commands/skillpack-check.ts` wraps `doctor --fast --json` +
    `apply-migrations --list` into one JSON report with `{healthy,
    summary, actions[], doctor, migrations}`. Exit 0 on healthy, 1 on
    action-needed, 2 on determine-failure. `--quiet` flag for cron
    pipes that want exit-code-only behavior.
  - `actions[]` is the remediation list. Doctor messages of the form
    `... Run: <cmd>` get their command extracted (regex fixed to match
    the full remainder of the line, not just the first word). Pending
    or partial migrations push `gbrain apply-migrations --yes` to the
    front of actions[].
  - `gbrainSpawn()` helper resolves the gbrain invocation correctly on
    compiled binary installs (`argv[1] = /usr/local/bin/gbrain`) AND
    source installs (`argv[1] = src/cli.ts`, prefix with `bun run`).
    Same Codex #1 fix pattern as autopilot's resolveGbrainCliPath.
  - `skills/skillpack-check/SKILL.md` teaches agents when to run it,
    what to do with the output, and anti-patterns (don't run without
    --quiet in a cron that emails; don't ignore exit 2).
  - Registered in skills/RESOLVER.md and skills/manifest.json.

Test `test/skillpack-check.test.ts` (5 tests) covers healthy fresh
install, half-migrated exit-1 with apply-migrations in actions[],
--quiet suppresses stdout in both states, --help prints usage, summary
includes top action when multiple are present.

1192 unit tests pass (+15 new). The 38 failing tests are all
DATABASE_URL E2Es — same pre-existing pattern, unchanged by this
commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doc(v0.11.1): reframe README + minions-fix — v0.11.0 was never released

v0.11.0 was cut but never released publicly. v0.11.1 is the first
public Minions ship, and fixes the upgrade-migration mega-bug so it
self-heals on every future `gbrain upgrade` + `bun update gbrain`.
The README was wrongly framing the fix as a retrospective for v0.11.0
users — none exist, so remove it.

README changes:
  - Delete the "v0.11.0 migration didn't fire on your upgrade?" section.
    Replace with "Health check and self-heal": the `gbrain doctor`,
    `gbrain skillpack-check --quiet`, and `gbrain skillpack-check | jq`
    recipes that ship in v0.11.1. Still links to docs/guides/minions-fix.md
    for deeper troubleshooting.
  - Promote the production benchmark to top billing. The previous section
    led with the lab benchmark (same LLM, localhost) and buried the
    production data point as a single follow-up sentence. Real deployment
    numbers are the stronger signal:
      * 753ms vs >10s gateway timeout (sub-agent couldn't even spawn)
      * $0.00 vs ~$0.03 per run
      * 100% vs 0% success rate under 19-cron production load
      * 36-month tweet backfill: 19,240 tweets, ~15 min, $0.00
    Lab numbers stay (separate table, labeled "controlled environment")
    so readers can see both layers.
  - Add the "The routing rule" closer: Deterministic → Minions, Judgment
    → Sub-agents. This is the clearest framing in the production
    benchmark doc and belongs in the README so readers leave with the
    right mental model. `minion_mode: pain_triggered` automates it.

docs/guides/minions-fix.md rewrite:
  - Reframe as: v0.11.0 never released, v0.11.1 is the first ship,
    `gbrain apply-migrations --yes` is canonical. Stopgap stays
    documented for pre-v0.11.1 branch builds (e.g. Wintermute's
    minions-jobs checkout before v0.11.1 tags).
  - Add the detection + verification commands (doctor + skillpack-check)
    at the top.
  - Cross-reference skills/skillpack-check/SKILL.md as the agent-facing
    health-check pattern.

Zero lingering "v0.11.0 released" references in README or minions-fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(doctor): remove "schema v7+ no prefs → FAIL" check (too aggressive)

CI failure in Tier 1 Mechanical E2E:
  (fail) E2E: Doctor Command > gbrain doctor exits 0 on healthy DB

Root cause: the doctor half-migration detection added two checks. The
second check (`schema v7+ AND ~/.gbrain/preferences.json missing →
minions_config FAIL`) was too aggressive. It treated a valid fresh-
install state as broken.

`gbrain init` against Postgres applies schema v7 but doesn't write
preferences.json — that's the migration orchestrator's Phase D, which
only runs via `apply-migrations`. Between `init` finishing and the user
running `apply-migrations`, the install is legitimately in a
"schema-applied, no prefs" state. Doctor was exiting 1 on this valid
state, breaking the pre-existing CI test that init's + docters a
healthy DB.

Fix: drop the check. The filesystem check (step 3 — partial-completed
without a matching complete) is sufficient signal for genuine half-
migration. Added a regression test pinning the exact CI scenario: no
completed.jsonl present, no preferences.json, doctor must not fail any
minions_* check.

Also removes the now-unused `preferencesPaths` import.

Verified against live Postgres: CI-equivalent `gbrain doctor` + `gbrain
doctor --json` both pass. Full suite: 1281/1281 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doc(readme): Minions section — lead with the story, compress the rest

The previous section opened with "six daily pains" as a numbered list
before the hook, buried the production numbers halfway down, and had
a table explaining how each pain gets fixed. Fine for a spec doc;
wrong for a README that needs to land the impact fast.

Rewrite:
  - Lead with "your sub-agents won't drop work anymore" — the reason
    a reader is here.
  - Production numbers promoted, framed as a story: "Here's my
    personal OpenClaw deployment: one Render container, Supabase
    Postgres holding a 45,000-page brain, 19 cron jobs firing on
    schedule, the X Enterprise API on the wire..." Gives the reader
    the setup before the punchline.
  - The routing rule (deterministic → Minions, judgment → sub-agents)
    survives unchanged. It's the clearest framing in the whole section.
  - Lose the "how each pain gets fixed" table. Compress the six pains
    + their fixes into one paragraph that names the primitives by
    name (max_children, timeout_ms, child_done inbox, cascade cancel,
    idempotency keys, attachment validation). Readers who want depth
    click through to skills/minion-orchestrator/SKILL.md.
  - Close with "not incrementally better — categorically different"
    and the three headline numbers.
  - Drop the separate Lab Numbers table; the production numbers are
    stronger and the lab data is one click away via the link.

Lines: 75 → 42. Same signal, less scroll.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doc: scrub X Enterprise API + @garrytan references from user-facing docs

User feedback: shouldn't name the specific enterprise-tier API product
or the account in the README or benchmark docs. Genericize:

  - "X Enterprise API on the wire" → drop entirely; the 19-cron load
    story carries the setup without naming the vendor
  - "X Enterprise API ($50K/mo firehose)" → "external API"
  - "@garrytan tweets" → "my social posts"
  - "Pull ~100 @garrytan tweets" → "Pull ~100 of my social posts"
  - "X Enterprise API (full-archive)" env var comment → "external API
    bearer token"

Scope:
  - README.md — the Minions production story line + scaling callout
  - docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md
  - docs/benchmarks/2026-04-18-tweet-ingestion.md

Plain "X API" references in the tweet-ingestion methodology stay —
those describe which public HTTP endpoint was called, not the
enterprise-tier product. Benchmark doc filenames (tweet-ingestion.md)
stay to preserve inbound links; content is genericized.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* doc(readme): Skillify section — match Minions energy, land the category shift

The previous section was competent but undersold what skillify actually
is. Rewrite matches the Minions section's shape: lead with the hook,
tell the story, land the punchline.

Key changes:
  - Title: "your skills tree stops being a black box." Names the thing
    skillify actually solves.
  - Open with the problem: Hermes auto-creates skills as a background
    behavior. Six months later you have an opaque pile nobody's read
    or tested. Make the liability concrete.
  - Promote the 10 items by name (SKILL.md + script + unit tests +
    integration tests + LLM evals + resolver trigger + trigger eval +
    E2E + brain filing + check-resolvable audit). Showing the list
    makes the scope of the unlock visible.
  - New subsection "Why this is the right answer for OpenClaw" names
    the debugging-the-black-box pain directly. Skillify makes the tree
    legible: when something breaks, you know which layer (contract,
    test, eval, trigger, or route) to inspect. When anything goes
    stale, check-resolvable flags it.
  - Close with "compounding quality instead of compounding entropy" +
    "not a nice-to-have. It's the piece that makes the skills tree
    survive six months."
  - Expand the code block to include `gbrain check-resolvable` (the
    other half of the pair) so readers see the whole workflow.

Length goes from 17 to 34 lines — still shorter than Minions, still
one section. Worth the space because this is a category shift for
how agent skills get built, not a feature.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: root <root@localhost>
 2026-04-18 16:57:38 +08:00
Garry Tan
e5a9f0126a feat: GStackBrain — 16 new skills, resolver, conventions, identity layer (v0.10.0) (#120) 
* feat: migrate 8 existing skills to conformance format

Add YAML frontmatter (name, version, description, triggers, tools, mutating),
Contract, Anti-Patterns, and Output Format sections to all existing skills.
Rename Workflow to Phases. Ingest becomes thin router delegating to specialized
ingestion skills (Phase 2).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add RESOLVER.md, conventions directory, and output rules

RESOLVER.md is the skill dispatcher modeled on Wintermute's AGENTS.md.
Categorized routing table: Always-on, Brain ops, Ingestion, Thinking,
Operational, Setup, Identity. Conventions directory extracts cross-cutting
rules (quality, brain-first lookup, model routing, test-before-bulk).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add skills conformance and resolver validation tests

skills-conformance.test.ts validates every skill has YAML frontmatter with
required fields, Contract, Anti-Patterns, and Output Format sections, and
manifest.json coverage. resolver.test.ts validates routing table categories,
skill path existence, and manifest-to-resolver coverage. 50 new tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add 9 brain skills from Wintermute (Phase 2)

Generalized from Wintermute's battle-tested skills:
- signal-detector: always-on idea+entity capture on every message
- brain-ops: brain-first lookup, read-enrich-write loop, source attribution
- idea-ingest: links/articles/tweets with author people page mandatory
- media-ingest: video/audio/PDF/book with entity extraction (absorbs video/youtube/book)
- meeting-ingestion: transcripts with attendee enrichment chaining
- citation-fixer: audit and fix citation formatting
- repo-architecture: filing rules by primary subject
- skill-creator: create skills with conformance standard + MECE check
- daily-task-manager: task lifecycle with priority levels

All Garry-specific references generalized. Core workflows preserved.
Updated RESOLVER.md and manifest.json.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add operational infrastructure + identity layer (Phase 3)

Operational skills:
- daily-task-prep: morning prep with calendar context and open threads
- cross-modal-review: quality gate via second model with refusal routing
- cron-scheduler: schedule staggering, quiet hours, wake-up override, idempotency
- reports: timestamped reports with keyword routing
- testing: skill validation framework (conformance checks)
- soul-audit: 6-phase interview generating SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md
- webhook-transforms: external events to brain signals with dead-letter queue

Identity layer:
- SOUL.md template (agent identity, generated by soul-audit)
- USER.md template (user profile, generated by soul-audit)
- ACCESS_POLICY.md template (4-tier access control)
- HEARTBEAT.md template (operational cadence)
- cross-modal.yaml convention (review pairs, refusal routing chain)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md with 24 skills, RESOLVER.md, conventions, templates

GBrain is now a GStack mod for agent platforms. Updated architecture description,
key files listing (16 new skill files, RESOLVER.md, conventions, templates), skills
section (24 skills organized by resolver categories), and testing section (new
conformance and resolver tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add GStack detection + mod status to gbrain init (Phase 4)

After brain initialization, gbrain init now reports:
- Number of skills loaded (from manifest.json)
- GStack detection (checks known host paths, uses gstack-global-discover if available)
- GStack install instructions if not found
- Resolver and soul-audit pointers

Also adds installDefaultTemplates() for SOUL.md/USER.md/ACCESS_POLICY.md/HEARTBEAT.md
deployment, and detectGStack() using gstack-global-discover with fallback to known paths
(DRY: doesn't reimplement GStack's host detection logic).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: v0.10.0 release documentation

- CHANGELOG: 24 skills, signal detector, RESOLVER.md, soul-audit, access control,
  conventions, conformance standard, GStack detection in init
- README: updated skill section with 24 skills, resolver, conventions
- TODOS: added runtime MCP access control (P1)
- VERSION: 0.9.2 → 0.10.0
- package.json + manifest.json version bumped

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add skill table to CHANGELOG v0.10.0

16-row table detailing every new skill, what it does, and why it matters.
Written to sell the upgrade, not document the implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore package.json version after merge conflict resolution

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: zero-based README rewrite for GStackBrain v0.10.0

Lead with GStack mod identity. 24 skills table organized by category.
Install block references RESOLVER.md and soul-audit. GBrain+GStack
relationship explained. Removed redundancy (733 -> 406 lines).
All essential content preserved: install, recipes, architecture,
search, commands, engines, voice, knowledge model.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: extract install block to INSTALL_FOR_AGENTS.md, simplify README

The 30-line copy-paste install block becomes one line:
"Retrieve and follow INSTALL_FOR_AGENTS.md"

Benefits: agent always gets latest instructions (no stale copy-paste),
README stays clean, install details live where agents read them.

README now leads with what GBrain does ("gives your agent a brain")
instead of GStack relationship. Removed "requires frontier model" note.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 3 bugs in init.ts from merge conflict resolution

1. llstatSync typo (merge corruption) → lstatSync
2. __dirname undefined in ESM module → fileURLToPath polyfill
3. require('fs') in ESM → use imported readFileSync

All three would crash gbrain init at runtime. Caught by /review.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add checkResolvable shared core function for resolver validation

Shared function at src/core/check-resolvable.ts validates that all skills
are reachable from RESOLVER.md, detects MECE overlaps (with whitelist for
always-on/router skills), finds gaps in frontmatter triggers, and scans
for DRY violations. Returns structured ResolvableIssue objects with
machine-parseable fix objects alongside human-readable action strings.

Three call sites: bun test, gbrain doctor, skill-creator skill.

Cleans up test/resolver.test.ts: removes stale 9-line skip list, imports
from production check-resolvable.ts instead of reimplementing parsing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: expand doctor with resolver validation, filesystem-first architecture

Doctor now runs filesystem checks (resolver health, skill conformance) before
connecting to DB. New --fast flag skips DB checks. Falls back to filesystem-only
when DB is unavailable. Adds schema_version: 2 to JSON output, composite health
score (0-100), and structured issues array with action strings for agent parsing.

Resolver health check calls checkResolvable() and surfaces actionable fix
instructions. Link integrity check uses engine.getHealth() dead_links count.

CLI routing split: doctor dispatched before connectEngine() so filesystem
checks always run. Fixes Codex-identified blocker where doctor required DB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add adaptive load-aware throttling and fail-improve loop

backoff.ts: System load checking (CPU via os.loadavg, memory via os.freemem),
exponential backoff with 20-attempt max guard, active hours multiplier (2x
slower during waking hours), concurrent process limit (max 2). Windows-safe:
defaults to "proceed" when os.loadavg returns zeros.

fail-improve.ts: Deterministic-first, LLM-fallback pattern with JSONL failure
logging. Cascade failure handling: when both paths fail, throws LLM error and
logs both. Log rotation at 1000 entries. Call count tracking for deterministic
hit rate metrics. Auto-generates test cases from successful LLM fallbacks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add transcription service and enrichment-as-a-service

transcription.ts: Groq Whisper (default) with OpenAI fallback. Files >25MB
segmented via ffmpeg. Provider auto-detection from env vars. Clear error
messages for missing API keys and unsupported formats.

enrichment-service.ts: Global enrichment service callable from any ingest
pathway. Entity slug generation (people/jane-doe, companies/acme-corp),
mention counting via searchKeyword, tier auto-escalation (Tier 3→2→1 based
on mention frequency and source diversity), batch enrichment with backoff
throttling, regex-based entity extraction from text.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add data-research skill with recipe system, extraction, dedup, tracker

New skill: data-research — one parameterized pipeline for any email-to-
structured-data workflow (investor updates, donations, company metrics).
7-phase pipeline: define recipe, search, classify, extract (with extraction
integrity rule), archive, deduplicate, update tracker.

data-research.ts: Recipe validation, MRR/ARR/runway/headcount regex
extraction (battle-tested patterns), dedup with configurable tolerance,
markdown tracker parsing/appending, quarterly/monthly date windowing,
6-phase HTML email stripping with 500KB ReDoS cap.

Registers data-research in manifest.json (25th skill) and RESOLVER.md.
Fixes backoff test robustness for high-load systems.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.10.0 infrastructure additions

CLAUDE.md: added 6 new core files (check-resolvable, backoff, fail-improve,
transcription, enrichment-service, data-research), 6 new test files, updated
skill count to 25, test file count to 34.

README.md: updated skill count to 25, added data-research to skills table.

CHANGELOG.md: added Infrastructure section documenting resolver validation,
doctor expansion, adaptive throttling, fail-improve loop, voice transcription,
enrichment service, and data-research skill.

TODOS.md: anonymized personal references.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: doctor.ts use ES module imports, harden backoff test

Replace require('fs') with ES module import in doctor.ts for consistency
with the rest of the file. Backoff test made resilient to parallel test
execution leaking module-level state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: README rewrite with production brain stats, sample output, new infrastructure

Lead with the flex: 17,888 pages, 4,383 people, 723 companies, 526 meeting
transcripts built in 12 days. Show sample query output so readers see what
they'll get. Document self-improving infrastructure (tier auto-escalation,
fail-improve loop, doctor trajectory). Add data-research recipes to Getting
Data In. Update commands section with doctor --fix, transcribe, research
init/list. Fix stale "24" references to "25".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: README lead with YC President origin and production agent deployments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: README lead with skill philosophy and link to Thin Harness Fat Skills

Skills section now explains: skill files are code, they encode entire
workflows, they call deterministic TypeScript for the parts that shouldn't
be LLM judgment. Links to the tweet and the architecture essay.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: link GStack repo, add 70K stars and 30K daily users

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: remove meeting transcript count from README (sensitive)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: README lead with YC President origin and production agent deployments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: rename political-donations recipe to expense-tracker (sensitivity)

Renamed the built-in data-research recipe from political-donations to
expense-tracker across README, CHANGELOG, SKILL.md, and reports routing.
Same extraction patterns (amounts, dates, recipients), neutral framing.
Also renamed social-radar keyword route to social-mentions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-14 19:41:34 -10:00
Garry Tan
baf3517868 feat: v0.9.0 -- smart file storage, publish, production-grade skills (#62) 
* feat: battle-tested skill patterns from production deployment

Backport production-learned brain-operations patterns:
- Iron Law of Back-Linking (mandatory bidirectional linking)
- Brain filing rules (file by primary subject, not format)
- Enrichment protocol (7-step pipeline, 3-tier system, person/company templates)
- Media ingest workflows (articles, videos, podcasts, PDFs, screenshots)
- Citation requirements (mandatory [Source: ...] on every fact)
- Test Before Bulk operating principle
- Voice recipe: unicode crash fix, PII scrub, identity-first prompt, DIY STT+LLM+TTS
- X-to-Brain recipe: image OCR, Filtered Stream, tweet rating rubric, cron stagger

* chore: bump version and changelog (v0.8.1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add _brain-filing-rules.md to CLAUDE.md key files

* feat: smart file upload with TUS resumable and .redirect.yaml pointers

- Supabase Storage auto-selects upload method by file size:
  < 100 MB standard POST, >= 100 MB TUS resumable (6 MB chunks + retry)
- Signed URL generation for private bucket access (1-hour expiry)
- New `upload-raw` command with size routing: small text stays in git,
  large/media files go to cloud with .redirect.yaml pointer
- New `signed-url` command for generating access links
- File resolver supports both .redirect.yaml (v0.9+) and .redirect (legacy)
- Redirect format upgraded: 10 fields with full metadata
- All migration commands (mirror, redirect, restore, clean) handle both formats

* feat: skills reference actual gbrain file commands

- Filing rules document upload-raw, signed-url, and .redirect.yaml format
- Ingest skill uses gbrain files upload-raw for raw source preservation
- Maintain skill adds file storage health checks
- Setup skill adds storage configuration phase with migration guidance
- Voice recipe uses upload-raw for call audio storage
- Migration v0.9.0 with complete storage setup instructions

* chore: bump version and changelog (v0.9.0)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: gbrain publish -- shareable HTML with password protection

First code+skill pair: deterministic code does the work (strip private data,
encrypt with AES-256-GCM, generate self-contained HTML), the skill tells the
agent when and how to use it. 34 new tests.

See: https://x.com/garrytan/status/2042925773300908103

* feat: backlinks check/fix, page lint, and report commands

Three new deterministic tools (zero LLM calls):

- gbrain backlinks check/fix -- scans brain for entity mentions without
  back-links, creates them. Enforces the Iron Law from the skills.
- gbrain lint [--fix] -- catches LLM preambles, code fence wrapping,
  placeholder dates, missing frontmatter, broken citations, empty sections.
  --fix auto-strips fixable artifacts.
- gbrain report --type <name> -- saves timestamped reports to
  brain/reports/{type}/YYYY-MM-DD-HHMM.md for audit trails.

33 new tests (409 total, 0 fail).

* feat: v0.9.0 migration tells agents to swap scripts for built-in commands

Migration file now:
- Lists all 5 new deterministic commands with usage examples
- Includes a script-to-command replacement table (old -> new)
- Tells the agent to find custom script references in AGENTS.md,
  skills, and cron jobs and replace with gbrain commands
- Adds recommended cron jobs for daily backlink fix + weekly lint
- References the Thin Harness, Fat Skills thread

* fix: CLI routing bugs found during DX review

- Fixed subArgs reference error in handleCliOnly (used wrong variable name)
- Renamed gbrain backlinks check/fix to gbrain check-backlinks to avoid
  conflict with existing backlinks operation (per-page incoming links)
- Added TOOLS section to --help output showing publish, check-backlinks,
  lint, report
- Added upload-raw and signed-url to FILES section in --help
- Updated all docs/migration references to use check-backlinks

* fix: security hardening from adversarial review

- XSS: sanitize marked.parse() output (strip script/iframe/on* attrs)
- Path traversal: validate report --type against [a-z0-9-] pattern
- TUS: HEAD request before retry to get server's actual offset (TUS spec)
- Pointer: upload-raw now includes pointer content in JSON output
- Symlinks: use lstatSync in all walkers to prevent directory escape

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-11 21:46:07 -10:00
Garry Tan
91ced664b6 feat: Voice v0.8.0 + feature discovery + Edge Function removal (#55) 
* chore: remove Supabase Edge Function MCP deployment

The Edge Function never worked reliably. All MCP traffic goes through
self-hosted server + ngrok tunnel. Removes deploy-remote.sh, edge-entry.ts,
supabase/functions/, .env.production.example, and CHATGPT.md (OAuth not
implemented).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite MCP docs for self-hosted + ngrok deployment

All per-client guides updated from Edge Function URLs to self-hosted
server + ngrok tunnel pattern. DEPLOY.md rewritten with local vs remote
paths. ALTERNATIVES.md now shows self-hosted as primary, with ngrok,
Tailscale, and Fly.io/Railway comparison.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: voice recipe v0.8.0 — 25 production patterns from real deployment

Identity separation, pre-computed bid system, conversation timing fix,
proactive advisor mode, radical prompt compression, OpenAI Realtime
Prompting Guide structure, auth-before-speech, brain escalation, stuck
watchdog, never-hang-up rule, thinking sounds, fallback TwiML, tool set
architecture, trusted user auth, caller routing, dynamic VAD, on-screen
debug UI, live moment capture, belt-and-suspenders post-call, mandatory
3-step post-call, WebRTC parity, dual API events, report-aware query
routing. WebRTC pseudocode updated with native FormData and 6 gotchas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: post-upgrade feature discovery framework

upgrade.ts captures old version before upgrading, then execs
gbrain post-upgrade (new binary) to read migration files and print
feature pitches. Migration files get YAML frontmatter with feature_pitch
field (headline, description, recipe, tiers). CLI prints excited builder
tone post-upgrade. v0.8.0 migration offers voice setup with environment
detection (server vs local) and 3-tier progressive disclosure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add Voice section to README with WebRTC screenshot + tweet link

Her out of the box: voice-to-brain with 25 production patterns. WebRTC
client screenshot embedded. Remote MCP section rewritten for self-hosted
+ ngrok. Setup block genericized.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add recipe validation tests + genericize personal refs

5 new integration tests: secrets completeness, semver version, requires
resolution, all-recipes-parse, no-personal-references. Test fixture
genericized. CLAUDE.md/TODOS.md/SKILLPACK updated for v0.8.0. build:edge
script removed from package.json.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.8.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-11 10:52:30 -10:00
Garry Tan
ce15062694 feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39) 
* docs: break SKILLPACK into 17 individual guides

The 1,281-line SKILLPACK monolith is now 17 individually linkable guides
in docs/guides/, organized by category: core patterns, data pipelines,
operations, search, and administration.

GBRAIN_SKILLPACK.md becomes a structured index with categorized tables
linking to each guide. The URL stays stable for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add integration guides, architecture docs, and ethos

New documentation directories:
- docs/integrations/ — "Getting Data In" landing page, credential gateway,
  meeting webhooks. Includes recipe format documentation.
- docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search)
- docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide
- docs/designs/ — "Homebrew for Personal AI" 10-star vision document

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add gbrain integrations command + voice-to-brain recipe

New CLI command: gbrain integrations (list/show/status/doctor/stats/test)
- Standalone command, no database connection needed
- Uses gray-matter directly for recipe parsing (not parseMarkdown)
- --json flag on every subcommand for agent-parseable output
- Bare command shows senses/reflexes dashboard
- Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl

First recipe: recipes/twilio-voice-brain.md
- Phone calls create brain pages via Twilio + OpenAI Realtime
- Opinionated defaults: caller screening, brain-first lookup, quiet hours
- Outbound call smoke test (GBrain calls the user to prove it works)
- Validate-as-you-go credential testing
- Twilio signature validation for webhook security

Migration file for v0.7.0 with agent-readable changelog.
13 unit tests covering parseRecipe, CLI routing, and recipe validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Getting Data In to README, update CLAUDE.md and manifest

README: voice calls in intro bullet list, new "Getting Data In" section
with integration table (voice, email, X, calendar) and recipe philosophy.

CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/,
docs/integrations/, docs/architecture/, docs/ethos/).

manifest.json: bump to v0.7.0, add recipes_dir field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: v0.7.0 CHANGELOG, TODOS, VERSION bump

CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain,
gbrain integrations command, SKILLPACK breakout, and new documentation.

TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL,
community recipe submission, always-on deployment recipes).

VERSION + package.json: bump to 0.7.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite voice recipe with agent instructions and verified links

Major improvements to recipes/twilio-voice-brain.md:

- Agent preamble: explains WHY sequential execution matters (each step
  depends on the previous), defines 4 stop points where the agent MUST
  pause and verify, tells agent to never say "something went wrong"
  but instead explain the exact error and fix

- User actions are now specific: exact URLs for every credential
  (Twilio console, OpenAI API keys page, ngrok dashboard), what
  buttons to click, what fields to copy, common failure modes

- All URLs verified via web search against current 2026 documentation:
  Twilio SID/token at twilio.com/console, OpenAI keys at
  platform.openai.com/api-keys, ngrok token at
  dashboard.ngrok.com/get-started/your-authtoken

- Cost estimate corrected: OpenAI Realtime is $0.06/min input +
  $0.24/min output (was understated), total ~$20-22/mo for 100 min

- Validate-as-you-go: each credential tested immediately with exact
  curl commands, failure messages explain what went wrong and how to fix

- Smoke test flow: tells user exactly what to say, verifies ALL
  three outputs (messaging notification + brain page + search result)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add "Homebrew for Personal AI" essay (markdown is code)

New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution
corollary to "Thin Harness, Fat Skills." Argues that markdown skill files
are simultaneously documentation, specification, package, and source code.
The agent is the package manager. The git repo is the app store.

Referenced from SKILLPACK index and CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite agent instructions as command language, promote skills

The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide.
Every step is an imperative command with exact verification criteria and
explicit stop-on-failure behavior. No FYI, no suggestions, just rails.

Key changes:
- 11-step setup with STOP points after each step
- Exact user instructions for Supabase connection string (what to click,
  what NOT to give the agent, what the string looks like)
- "Verify: run X. You must see Y. If not: Z" after every step
- Skills table now links to both skill files AND guide docs
- Integration recipes table simplified (no "coming soon" placeholders)
- Docs section reorganized: for agents / for humans / reference

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 4 codex findings + add email-to-brain recipe

Codex review found 4 issues, all fixed:

1. getStatus() returned "configured" if ANY secret was set (e.g. just
   OPENAI_API_KEY). Now requires ALL required secrets before marking
   configured. Prevents false "configured" status and spurious doctor runs.

2. Twilio health check hit unauthenticated endpoint (always 401). Now
   uses authenticated curl with SID:token, matching the setup validation.

3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after
   SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md.

4. Compiled binary can't find recipes/ via import.meta.dir. Added
   GBRAIN_RECIPES_DIR env var override + global bun install path fallback.

Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern
with ClawVisor credential gateway, validate-as-you-go, agent instructions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add email, X, calendar, and meeting sync recipes

Four new integration recipes extracted from production wintermute patterns:

- recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector
  pattern (code pulls emails with baked-in links, agent does judgment),
  noise filtering, signature detection, digest generation

- recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search,
  deletion detection (diffs previous run, verifies 404), engagement
  velocity tracking, rate limit awareness

- recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical
  backfill (years of data), daily markdown files with attendees + locations,
  attendee enrichment for brain pages

- recipes/meeting-sync.md: Circleback API, transcript import with speaker
  labels, attendee detection + filtering, entity propagation to people/
  company pages, action item extraction, idempotent by source_id

All recipes follow the same format: agent preamble with sequential execution
rules, validate-as-you-go credentials, exact URLs for API key setup,
stop-on-failure verification, and heartbeat logging.

Updated README, SKILLPACK index, and integrations landing page with all 5 recipes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes

Both recipes now offer two auth options:
- Option A: ClawVisor (recommended, handles OAuth + token refresh)
- Option B: Google OAuth2 directly (no extra service, you manage tokens)

Option B includes step-by-step instructions for Google Cloud Console:
exact URLs, which buttons to click, which scopes to add, how to enable
the API, and the OAuth flow for token exchange.

This removes ClawVisor as a hard dependency for getting started.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add implementation guides with pseudocode and test suggestions

Every recipe now includes an "Implementation Guide" section with:

- Production-tested pseudocode the agent can follow to build each collector
- Edge cases and failure modes discovered in real deployment
- Non-obvious implementation details (why the 48h staleness heuristic,
  why Gmail links need authuser, why SSE responses need double-parsing)
- Test suggestions: what the agent should verify after setup

email-to-brain: noise filtering algorithm, signature detection patterns,
  Gmail link generation (authuser is critical), sent-mail dedup

x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness,
  API verification), engagement velocity thresholds (50 min for 2x, 100
  absolute jump), atomic writes, stdout contract, rate limit handling

calendar-to-brain: smart chunking (monthly for sparse years, weekly for
  dense), attendee filtering (rooms, groups, distros), merge-with-existing
  (only replace ## Calendar section), date/time parsing edge cases

meeting-sync: SSE double-JSON parsing, idempotency double-check (grep +
  filename), auto-tagging from meeting names, git commit after sync

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: 6 new guides from production patterns (wintermute extraction)

New guides extracted and generalized from production deployment:

- repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge).
  Strict boundary rules, decision tree, hard rule: never write knowledge
  to the agent repo.

- sub-agent-routing.md: Model routing table by task type. Signal detector
  pattern (spawn Sonnet on every message). Research pipeline pattern
  (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization.

- skill-development.md: 5-step cycle (concept, prototype, evaluate, codify,
  cron). MECE discipline (no overlapping skills). Quality bar checklist.
  "If you ask twice, it should already be a skill."

- idea-capture.md: Originality distribution rating (0-100 across 4
  populations). Depth test ("could someone unfamiliar understand WHY?").
  Deep cross-linking mandate. Notability filtering.

- quiet-hours.md: Hold notifications 11pm-8am local time. Held messages
  directory pattern. Timezone-aware delivery. Morning briefing pickup.

- diligence-ingestion.md: 9-step pipeline for data room materials. Detection
  patterns (PDF filenames, spreadsheet tabs, user language). Index.md
  template with bull/bear case. Company page enrichment.

All PII scrubbed. Patterns generalized for any user.
SKILLPACK index updated with 6 new entries. CLAUDE.md references added.
All 37 SKILLPACK links verified.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: upgrade all guides to operational playbooks with pseudocode

Every guide now follows the playbook structure:
- Goal: one sentence, what this achieves
- What the User Gets: without this / with this
- Implementation: pseudocode with actual gbrain commands
- Tricky Spots: production-tested gotchas
- How to Verify: test steps the agent runs after setup

Guides upgraded (15 files):
- brain-agent-loop: on_message() loop with read/write/sync pseudocode
- brain-first-lookup: 4-step lookup cascade with exact commands
- brain-vs-memory: routing algorithm for 3 knowledge layers
- compiled-truth: page structure + rewrite vs append rules
- content-media: 3 ingest patterns (YouTube, social, PDFs)
- cron-schedule: full schedule table + dream cycle pseudocode
- enrichment-pipeline: 7-step protocol with tier classification
- entity-detection: spawn pattern + detection prompt + notability filter
- executive-assistant: 3 workflow algorithms (triage, prep, post-inbox)
- meeting-ingestion: 6-step transcript-to-brain flow
- operational-disciplines: 5 executable discipline blocks
- originals-folder: detection + exact-phrasing capture + cross-linking
- search-modes: decision tree for keyword vs hybrid vs direct
- source-attribution: citation format + hierarchy + conflict resolution
- Plus Goal/What User Gets headers on 6 newer guides

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add WebRTC to voice recipe + ngrok Hobby setup guide

Voice recipe updates:
- Added WebRTC endpoint (POST /session, GET /call, POST /tool) for
  browser-based calling with RNNoise noise suppression
- WebRTC pseudocode with the 4 non-obvious gotchas from production
  (voice under audio.output.voice, no turn_detection, no session.update
  on connect, trigger greeting via data channel)
- Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier
- Fixed domain means URLs never change, Twilio never breaks

New guide: docs/mcp/NGROK_SETUP.md
- How to set up ngrok Hobby for both MCP and voice agent
- Fixed domain setup, watchdog pattern, AI client configuration
- Claude Desktop requires Settings > Integrations (not JSON config)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add dependency graph + ngrok-tunnel + credential-gateway recipes

Recipes now have real dependencies via the `requires` field:
- voice-to-brain requires ngrok-tunnel (needs public URL for Twilio)
- email-to-brain requires credential-gateway (needs Gmail access)
- calendar-to-brain requires credential-gateway (needs Calendar access)
- x-to-brain and meeting-sync are standalone (direct API keys)

Two new infrastructure recipes:
- ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby
  ($8/mo) for a domain that never changes. Includes watchdog pattern.
- credential-gateway: secure Google service access via ClawVisor
  (recommended) or direct OAuth2. One setup, all Google recipes use it.

Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure,
not MCP-specific.

README and integrations landing page show dependency chains.
When agent installs voice-to-brain, it sets up ngrok-tunnel first.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add infra category, fix dashboard alignment, show dependencies

DX audit found two bugs in gbrain integrations dashboard:

1. Column alignment broken — IDs > 18 chars ran into descriptions
   with no space. Fixed: pad to 22 chars.

2. ngrok-tunnel and credential-gateway showed as SENSES but they're
   infrastructure. Added 'infra' category. Dashboard now shows three
   sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES.

3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)"

Also added 'requires' field to JSON output for agent consumption.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add frontier model requirement disclaimer to README

GBrain's markdown-is-code approach requires models capable of
interpreting intent and implementing from architecture descriptions.
Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models
will struggle with the recipe format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add PGLite → Supabase upgrade path to README

Clarify the database progression: start with PGLite (Postgres as WASM,
zero infrastructure, pgvector built in, nothing to install). Graduate
to Supabase or self-hosted Postgres when you need connection pooling,
concurrency, and remote MCP access from Claude Desktop, Cowork,
ChatGPT, Perplexity Computer, or any MCP-compatible agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: revert PGLite mention (coming in next branch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify)

Every guide now has exactly these sections in this order:
- ## Goal (one sentence)
- ## What the User Gets (without this / with this)
- ## Implementation (pseudocode with gbrain commands)
- ## Tricky Spots (3-5 numbered gotchas)
- ## How to Verify (3-5 numbered test steps)

11 guides restructured from non-standard headings:
- deterministic-collectors, live-sync, upgrades-auto-update (full rewrites)
- entity-detection, diligence-ingestion, idea-capture, quiet-hours,
  repo-architecture, skill-development, sub-agent-routing (restructured)

23/23 guides now pass consistency audit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: restructure README around the #1 blocker (getting data in)

The README was leading with Postgres and database architecture. Most
users are stuck at step zero: "I have an agent but it doesn't know
anything about my life."

New structure:
1. The Problem — your agent doesn't know your life
2. Getting Data In — integration recipes, front and center
3. The Compounding Thesis — why this matters
4. How this happened — credibility, origin story
5. When you need Postgres — scale, not starting point

Postgres is de-emphasized from a full section to two paragraphs:
"You don't need Postgres to start" and "When you need Postgres"
(1,000+ files, remote MCP access, multiple AI clients).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move Install to top of README, remove duplicate section

Install now appears right after Getting Data In (line 38), not buried
at line 295. The user sees: Problem → Getting Data In → Install.

Removed the duplicate Install section (262 lines) that was lower in
the README. The agent instructions block, CLI quickstart, and all
content is now in the single Install section near the top.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move agent install block to first thing in README

"Start here: paste this into your agent" is now the first section,
right after the one-line pitch. No scrolling, no context, no preamble.
User opens the README, sees the paste block, copies it into OpenClaw
or Hermes, and the agent takes over.

Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: compress install block from 11 steps to 5

The agent install block was 102 lines and 11 steps. Now it's 40 lines
and 5 steps. Same coverage, half the text.

Changes:
- Merged "prove keyword search" + "embed" + "prove hybrid search"
  into one SEARCH step (the user doesn't care about the intermediate)
- Merged skillpack, sync, auto-update, integrations, verification
  into one GO LIVE step with sub-items (post-install polish, not install)
- Shortened database instructions (one line instead of 5 sub-steps)
- Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now
  just "Do not skip steps. Verify each step.")

The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: gitignore all .env files, not just specific ones

CSO audit found .gitignore covered .env.testing and .env.production
but not bare .env. A user creating .env with database credentials
could accidentally commit it.

Fix: .env and .env.* are now gitignored. .env.*.example files are
explicitly un-ignored so templates remain tracked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: scrub PII from essay and recipe examples

- 510-MY-GARRY phone mnemonic → "Your Phone Number"
- "Garry → Authenticated Mode" → "Owner → Authenticated Mode"
- "Telegram" → "secure channel" in auth example
- @garrytan → @yourhandle in X recipe example

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 23:39:06 -10:00
Garry Tan
912a321cfa GBrain v0.4.0 — production agent documentation + reference architecture (#10) 
* fix: widen validateSlug to accept any filename characters

Git is the system of record. Slugs are lowercased repo-relative paths.
The restrictive regex rejected spaces, parens, and special chars, blocking
5,861 Apple Notes files from importing. Now only rejects empty slugs,
path traversal (..), and leading slash.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: enable RLS on all tables with BYPASSRLS safety check

Without RLS, the Supabase anon key gives full read access to the DB.
Enable RLS on all 10 tables with no policies — the postgres role
(used by gbrain via pooler) has BYPASSRLS and is unaffected. Only
enables if the current role actually has BYPASSRLS privilege to
avoid locking ourselves out on non-Supabase setups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: import resilience — 5MB limit, error suppression, structured progress

Raise MAX_FILE_SIZE from 1MB to 5MB for Apple Notes with attachments.
Track error patterns and suppress after 5 identical errors to prevent
5,861 identical warnings from killing the agent process. Replace \r
progress bar with structured log lines (rate, ETA) for agent parsing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: init detects IPv6-only Supabase URLs, adds pgvector check

Detect db.*.supabase.co direct URLs and warn about IPv6 failure.
On ECONNREFUSED/ETIMEDOUT to Supabase, suggest the Session pooler
connection string with exact dashboard click path. Check for pgvector
extension after connecting and fail with clear instructions if missing.
Update wizard hints to show pooler URL format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add pre-ship requirement for E2E tests

E2E tests against real Postgres+pgvector must pass before /ship or
/review. Adds the requirement to CLAUDE.md so all agents enforce it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: parallel import with per-worker engine instances

Refactor PostgresEngine to support instance-level DB connections instead
of only the module-global singleton. Each worker gets its own connection
with poolSize:2 (vs 10 for the main engine), so 8 workers = 16 connections.

Add --workers N flag to gbrain import. Workers pull from a shared queue
and use independent engine instances — no transaction context corruption.

The bottleneck is network round-trips to Supabase (one per page upsert).
Parallel workers cut import time proportionally.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: automatic schema migration runner

Migrations are embedded as string constants in migrate.ts (survives
Bun --compile). Each migration runs in a transaction for clean rollback
on failure. Runs automatically on initSchema() — no manual step needed
when a user updates the gbrain binary against an older DB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: pluggable storage backend (S3 + Supabase Storage + local)

Add StorageBackend interface with three implementations:
- S3Storage: works with AWS S3, Cloudflare R2, MinIO (any S3-compatible)
- SupabaseStorage: uses Supabase Storage REST API with service role key
- LocalStorage: filesystem-based, for testing

Add file-resolver.ts with fallback chain: local file → .redirect
breadcrumb → .supabase marker → storage backend. Supports the
three-stage migration (mirror → redirect → clean).

Add yaml-lite.ts for parsing marker and breadcrumb files without
adding a YAML dependency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: gbrain doctor command — health checks with --json output

Checks: connection, pgvector extension, RLS on all tables, schema
version, embedding coverage. Outputs structured JSON with --json flag
for agent parsing. Exit code 0 if healthy, 1 if issues found.

Agents should run gbrain doctor --json when any command fails.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite setup skill + README for agent-first DX

Setup skill: add Why Supabase, step-by-step project creation, explicit
agent instructions (nohup for large imports, doctor on failure, don't
ask for anon key), available init flags, file migration offer after
first import. Remove ClawHub references.

README: simplify to single OpenClaw install path, remove ClawHub, fix
squatted npm name to github:garrytan/gbrain, add Supabase settings
note about Session pooler.

Add Apple Notes test fixtures with spaces and parens in filenames for
E2E testing of the slug fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add RLS verification, schema health, and nohup hints to maintain skill

Maintenance skill now checks RLS status and schema version as part of
periodic health checks. Adds nohup pattern for large embedding refreshes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: import resume checkpoint + Supabase smart URL parsing

Import resume: saves checkpoint every 100 files to ~/.gbrain/import-checkpoint.json.
On restart with same directory and file count, skips already-processed files.
Use --fresh to ignore checkpoint and start over. Cleared on successful completion.

Supabase admin: extractProjectRef() parses any Supabase URL format (dashboard,
direct, pooler, project URL) to extract the project ref. discoverPoolerUrl()
uses the Management API to find the correct pooler connection string (including
the exact region prefix). checkRls() verifies RLS status via the API.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add 56 unit tests for all new code

8 new test files covering every feature added in this branch:
- slug-validation.test.ts: spaces, parens, unicode, path traversal (10 tests)
- yaml-lite.test.ts: parse + stringify, marker/redirect formats (9 tests)
- supabase-admin.test.ts: extractProjectRef for 4 URL formats (7 tests)
- migrate.test.ts: version export, runMigrations callable (2 tests)
- storage.test.ts: LocalStorage CRUD + createStorage factory (14 tests)
- file-resolver.test.ts: fallback chain, redirect, marker parsing (6 tests)
- import-resume.test.ts: checkpoint save/load/resume/fresh (6 tests)
- doctor.test.ts: module export, CLI registration (3 tests)

Total: 184 pass, 0 fail (up from 128).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: bulk chunk INSERT + E2E tests for all new features

Bulk INSERT: upsertChunks now builds a multi-row VALUES query instead
of inserting chunks one-by-one. Reduces DB round-trips by ~50x per page.

E2E tests added to mechanical.test.ts:
- Slug with special chars: import Apple Notes fixtures with spaces/parens,
  verify search finds them, verify idempotency
- RLS verification: check pg_tables.rowsecurity on all tables, verify
  current user has BYPASSRLS
- Doctor command: verify exit 0 on healthy DB, --json produces valid JSON
  with check structure
- Parallel import: --workers 2 produces same page count as sequential

Unit tests added:
- setup-branching.test.ts: IPv6 detection, defaultWorkers auto-tuning,
  smart URL parsing across all Supabase URL formats

Fixtures added:
- large/big-file.md (2.1MB) for testing raised file size limit
- apple-notes/ fixtures already existed

Total: 200 pass, 0 fail (up from 184).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: --json on init/import, file migration CLI, lifecycle tests

--json flag: init and import now support --json for structured output.
Agents get parseable JSON instead of human-readable text.

File migration CLI: implement mirror, unmirror, redirect, restore,
clean, and status subcommands for the three-stage file migration
lifecycle (local → mirrored → redirected → cloud-only).

File migration tests: full lifecycle test covering every transition
in the state machine (LOCAL → MIRROR → UNMIRROR → REDIRECT → RESTORE
→ CLEAN), including edge cases and file resolver at each stage.

Bulk chunk INSERT: upsertChunks now builds multi-row parameterized
VALUES query, reducing round-trips per page from ~50 to 1.

Total: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: thorough E2E tests for parallel import concurrency

Replace the weak single-comparison parallel import test with 7 tests:
- Sequential baseline: capture page count, chunk count, and all slugs
- --workers 2: verify page count matches sequential
- Chunk count matches (no duplicates from concurrent writes)
- Page slugs match exactly
- No duplicate pages (SQL GROUP BY HAVING count > 1)
- No duplicate chunks (SQL GROUP BY page_id, chunk_index)
- --workers 4: also works correctly
- Re-import with workers is idempotent

These tests catch the exact bug Codex found (db.ts singleton causing
concurrent transaction corruption) by verifying data integrity after
parallel writes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add batch embedding queue as P1 TODO

Deferred during eng review (per-worker embedding is good enough for now).
Revisit after profiling real imports to confirm embedding is the bottleneck.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: E2E test failures — fixture counts, arg parsing, doctor exit code

Fix fixture count assertions: 13 → 16 pages (added apple-notes + large file),
companies 2 → 3 (ohmygreen), concepts 3 → 5 (notes, big-file).

Fix --workers arg parsing: the worker count value (e.g. "2") was being
picked up as the directory arg. Skip flag values when finding the dir.

Fix doctor exit code: warnings (like missing embeddings) should exit 0,
only actual failures exit 1. E2E tests import with --no-embed, so
embeddings are always WARN.

Fix E2E CLI tests: add initCli() before doctor and parallel import
tests so ~/.gbrain/config.json exists for the subprocess.

All E2E tests pass: 63 pass, 0 fail.
All unit tests pass: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.4.0

New CHANGELOG entry for all post-0.3.0 features (doctor, storage backends,
parallel import, resume checkpoints, RLS, schema migrations, --json output).
Version bumped 0.3.0 → 0.4.0 across all manifests.

CLAUDE.md: test count 9→19, skill count 8→7, added key files.
CONTRIBUTING.md: fixture count 13→16, added missing source files.
README.md: added gbrain doctor to commands, fixed stale welcome PRs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add GBRAIN_SKILLPACK.md reference architecture

Production agent patterns from a real deployment with 14,700+ brain files.
Covers: entity detection on every message, brain-first lookup protocol,
7-step enrichment pipeline with tiered API spend, compiled truth + timeline,
source attribution with mandatory citations, meeting ingestion with entity
propagation, cron schedule with quiet hours and travel-aware timezone,
YouTube/media ingestion via Diarize.io, integration guides for ClawVisor,
Circleback webhooks, and Quo/OpenPhone SMS. Opens with the Vannevar Bush
memex framing and the originals folder for capturing intellectual capital.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite README opener with memex pitch and production architecture

Replace code-first opener with mimetic-desire pitch: Vannevar Bush memex
tagline, production brain numbers (10K+ files, 3K+ people, 13 years of
calendar), "ask it anything" examples, compounding thesis.

New sections: The Compounding Thesis (read-write loop), Architecture
(three-column diagram), What a Production Agent Looks Like (SKILLPACK
reference), How gbrain fits with OpenClaw (three-layer complement).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update skills with brain-first lookup, entity detection, heartbeat

setup: Phase D rewritten with brain-first lookup protocol (gbrain search
→ query → get → grep fallback), sync-after-write rule, memory_search
complement table.

query: token-budget awareness (chunks not full pages), source precedence
hierarchy (user > compiled truth > timeline > external).

ingest: entity detection on every message (scan, check brain, create or
enrich, commit and sync).

maintain: heartbeat integration (doctor, embed --stale, sync verification,
stale compiled truth detection).

briefing: gbrain-native context loading (search attendees before meetings,
search sender before email, daily deal/meeting/commitment queries).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add OpenClaw positioning to README opener

Make it clear up top that GBrain is built for OpenClaw agents and
works with any OpenClaw deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: credit Karpathy's Knowledge LLM vision, add origin story

GBrain started as Karpathy's LLM wiki idea built for real. Worked great
until the brain hit thousands of files and grep fell apart. GBrain is the
search layer that had to exist once the brain outgrew grep.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 10:17:13 -07:00
Garry Tan
a86f995883 feat: GBrain v0.3.0 — contract-first architecture + ClawHub plugin (#7) 
* feat: contract-first operations.ts with OperationError, dry_run, importFromContent

30 shared operations as single source of truth for CLI and MCP.
- OperationError with typed error codes (page_not_found, invalid_params, etc.)
- dry_run support on all mutating operations
- importFromContent split from importFile with transaction wrapping
- Idempotency hash now includes ALL fields (title, type, frontmatter, tags)
- Config env var fallback: GBRAIN_DATABASE_URL > DATABASE_URL > config file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: rewrite MCP server + CLI + tools-json from operations

server.ts: 233 -> ~80 lines. Tool definitions and dispatch generated from operations[].
cli.ts: shared operations auto-registered, CLI-only commands kept as manual dispatch.
tools-json: generated FROM operations[], eliminating the third contract surface.
Parity test verifies structural contract between operations, CLI, and MCP.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: delete 12 command files migrated to operations.ts

Handler logic for get, put, delete, list, search, query, health, stats,
tags, link, timeline, and version now lives in operations.ts.
Kept: init, upgrade, import, export, files, embed, sync, serve, call, config.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: init --non-interactive, upgrade verification, schema migration

- gbrain init --non-interactive --url <url> for plugin mode (no TTY required)
- Post-upgrade version verification in gbrain upgrade
- Drop storage_url from files table (storage_path is the only identifier)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: tool-agnostic skills + new setup skill

All 7 skills rewritten with intent-based language instead of CLI commands.
Works with both CLI and MCP plugin contexts.
New setup skill replaces install: auto-provision Supabase via CLI,
AGENTS.md injection, target TTHW < 2 min.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: ClawHub bundle plugin, CI workflows, v0.3.0

- openclaw.plugin.json with configSchema, MCP server config, skill listing
- GitHub Actions: test on push/PR, multi-platform release (macOS arm64 + Linux x64)
- Version bump 0.3.0, CHANGELOG, README ClawHub section, CLAUDE.md updated

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: idempotency hash mismatch + MCP dry_run passthrough

importFromContent now passes its all-fields hash through putPage via
content_hash on PageInput, so the stored hash matches the computed hash.
Previously the skip-if-unchanged check never fired because the hash
formulas differed.

MCP server now passes dry_run from tool params to OperationContext.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.3.0.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: schema loader handles PL/pgSQL $$ blocks

Delete the semicolon-based SQL splitter in db.ts which broke on
PL/pgSQL trigger functions containing semicolons inside $$ delimiter
blocks. Use single conn.unsafe(schemaSql) call instead — the postgres
driver handles multi-statement SQL natively. schema.sql already uses
IF NOT EXISTS / CREATE OR REPLACE for idempotency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: E2E test infrastructure + realistic brain fixtures

Add test infrastructure for running E2E tests against real
Postgres+pgvector. Includes:
- test/e2e/helpers.ts: DB lifecycle, fixture import, timing, diagnostics
- 13 fixture files as a miniature realistic brain (people, companies,
  deals, meetings, concepts, projects, sources) following the
  compiled truth + timeline format from GBRAIN_RECOMMENDED_SCHEMA.md
- docker-compose.test.yml: local pgvector convenience (port 5433)
- .env.testing.example: template for test credentials
- package.json: add test:e2e script

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: E2E test suites + CI workflow

Tier 1 (mechanical.test.ts): 14 test suites covering all operations
against real Postgres — page CRUD, search with quality scoring, links,
tags, timeline, versions, admin, chunks, resolution, ingest log, raw
data, files, idempotency stress, setup journey (full CLI flow), init
edge cases, schema idempotency, schema diff guard, performance baselines.

Tier 1 (mcp.test.ts): MCP protocol test — spawns server, sends JSON-RPC,
verifies tools/list matches operations count.

Tier 2 (skills.test.ts): OpenClaw skill tests — ingest, query, health.
Skips gracefully when dependencies missing.

CI (.github/workflows/e2e.yml): Tier 1 on every PR (pgvector service),
Tier 2 nightly/manual with API key secrets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: E2E test fixes + traverseGraph jsonb cast

- Fix traverseGraph query: cast json_agg to jsonb_agg so SELECT DISTINCT works
- Fix put_page tests to use importFromContent with noEmbed (no OpenAI key in Tier 1)
- Fix get_health assertion (page_count not total_pages)
- Fix raw_data test to handle JSONB string/object return
- Simplify MCP test to verify tool generation directly
- Add timeouts to CLI subprocess tests
- Use port 5434 for docker-compose (5433 often in use)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update all project docs for E2E test suite

- CLAUDE.md: updated test count (9 unit + 3 E2E), added E2E test
  instructions, fixed skill count to 8
- CONTRIBUTING.md: updated project structure with test/e2e/, added E2E
  test instructions, rewrote "Adding a new command" to reflect
  contract-first architecture (add to operations.ts, done)
- README.md: fixed table count (10 not 9), added recommended schema doc
  to Docs section, added E2E instructions to Contributing section
- CHANGELOG.md: added E2E test suite, docker-compose, schema loader fix,
  and traverseGraph jsonb fix to v0.3.0 entry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-08 23:26:11 -10:00
Garry Tan
ecebd5552a feat: GBrain v0.2.0 — incremental sync, file storage, install skill (#2) 
* refactor: extract importFile from import.ts + add tag reconciliation

Shared single-file import function used by both import and sync.
Adds tag reconciliation (removes stale tags on reimport), >1MB file
skip, and import->sync checkpoint continuity (writes git HEAD to
config table after import so sync picks up seamlessly).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add sync pure functions, updateSlug engine method, and sync tests

- buildSyncManifest: parses git diff --name-status -M output
- isSyncable: filters to .md pages, excludes hidden/ops/.raw/skip-list
- pathToSlug: converts file paths to page slugs with optional prefix
- updateSlug: renames page slug in-place (preserves page_id, chunks, embeddings)
- rewriteLinks: stub for v0.2 (FKs use page_id, already correct)
- 20 new tests, all passing (39 total across 3 files)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add gbrain sync command with CLI, MCP, and watch mode

18-step sync protocol: read config, git pull, ancestry validation,
git diff --name-status -M for net changes, isSyncable filter, process
deletes/renames/adds/modifies via importFile, batch optimization,
sync state checkpoint in Postgres config table. Watch mode with
polling and consecutive error counter. MCP sync_brain tool returns
structured SyncResult. Stale page deletion for un-syncable files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add files table, gbrain files commands, and config show redaction

- files table: page_slug FK with ON DELETE SET NULL + ON UPDATE CASCADE,
  storage_path, storage_url, mime_type, content_hash for dedup
- gbrain files list/upload/sync/verify commands for Supabase Storage
- gbrain config show redacts postgresql:// passwords and secret keys
- CLI help updated with FILES section

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add install skill for GBrain onboarding

6-phase install workflow: environment discovery, Supabase setup (magic
path via CLI OAuth or fallback 2-copy-paste), init + import, ongoing
sync cron, optional file migration with mandatory verification, and
agent teaching (AGENTS.md rules). Every error gets what + why + fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.2.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add v0.2 features to README (sync, files, install skill)

README.md: added sync command to IMPORT/EXPORT section, added FILES
section with 4 commands, added files table to schema diagram, added
install skill to skills table, updated MCP tools count from 20 to 21
(sync_brain added).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: OpenClaw DX improvements (skill count, upgrade docs, config show help)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: consolidate version to single source of truth

Create src/version.ts that reads from package.json via static import
(safe for bun compiled binaries). Update mcp/server.ts from hardcoded
'0.1.0' to use shared VERSION. Bump skills/manifest.json to 0.2.0.

* fix: upgrade detection order, npm→bun naming, clawhub false positives

Reorder detection: node_modules first, binary second, clawhub last.
Rename 'npm' install method to 'bun'. Use 'clawhub --version' instead
of 'which clawhub' to avoid false positives from dangling symlinks.
Add 120s timeout to execSync calls to prevent hanging. Add --help flag.

* feat: per-command --help, unknown command check before DB connection

Add COMMAND_HELP map covering all 28 commands. Check --help before
init/upgrade dispatch and before connectEngine() so help works without
a database. Use COMMAND_HELP keys as known-command set to catch unknown
commands before wasting a DB round-trip.

* docs: standardize npm references to bun, add Upgrade section to README

Fix init.ts: npx→bunx, npm→bun for supabase CLI guidance.
Fix README: npm install→bun add for standalone CLI install.
Add ## Upgrade section to README with all three install methods.
Update install skill Upgrading section to list bun, ClawHub, and binary.

* test: full coverage audit — CLI dispatch, upgrade detection, config, edge cases

New test files:
- test/cli.test.ts: COMMAND_HELP ↔ switch consistency, version from
  package.json, per-command --help, unknown command handling, global help
- test/upgrade.test.ts: detection order verification, npm→bun naming,
  clawhub --version (not which), timeout presence
- test/config.test.ts: redactUrl for postgresql URLs, edge cases

Extended existing tests:
- test/sync.test.ts: empty string pathToSlug, uppercase .MD rejection,
  deeply nested files, multiple renames, unknown status codes
- test/markdown.test.ts: multiple --- separators, missing frontmatter,
  no frontmatter at all, empty string, type inference from paths

Tests: 39 → 83 (+44 new). All pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: 100% coverage — import-file mock engine, files utils, chunker edge cases

New test files:
- test/import-file.test.ts (9 tests): mock BrainEngine to test importFile
  without DB — MAX_FILE_SIZE skip, content_hash dedup, tag reconciliation
  (remove stale + add new), compiled_truth/timeline chunking, noEmbed flag,
  sequential chunk_index
- test/files.test.ts (22 tests): getMimeType for all extensions + uppercase
  + unknown + no-extension, fileHash consistency + different content + empty,
  collectFiles pattern (skip .md, skip hidden dirs, recurse, sorted output)

Extended:
- test/chunkers/recursive.test.ts (+6 tests): single newline splits,
  word-only text, clause delimiters, lossless preservation, default options,
  mixed delimiter hierarchy

Tests: 83 → 118 (+35 new). All pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 16:50:15 -07:00
Garry Tan
b22cbd349a feat: GBrain v0.1.0 — Postgres-native personal knowledge brain (#1) 
* chore: add CLAUDE.md with project context and gstack skill routing rules

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: initialize project with Bun + TypeScript

package.json with dependencies (postgres, pgvector, openai, anthropic,
MCP SDK, gray-matter). TypeScript config targeting ESNext with bundler
module resolution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add foundation layer — engine interface, Postgres engine, schema

BrainEngine pluggable interface with full PostgresEngine: CRUD, search
(keyword + vector), links, tags, timeline, versions, stats, health,
ingest log, config. Trigger-based tsvector spanning pages +
timeline_entries. Markdown parser with frontmatter, compiled_truth /
timeline splitting, and round-trip serialization. 19 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add 3-tier chunking and embedding service

Recursive delimiter-aware chunker (5-level hierarchy, 300-word chunks,
50-word overlap). Semantic chunker with Savitzky-Golay boundary detection
and recursive fallback. LLM-guided chunker via Claude Haiku with sliding
window topic detection. OpenAI embedding service with batch support,
exponential backoff, and rate limit handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add hybrid search with RRF fusion, expansion, and 4-layer dedup

Hybrid search merges vector (pgvector HNSW) + keyword (tsvector) via
Reciprocal Rank Fusion. Multi-query expansion via Claude Haiku generates
2 alternative phrasings. 4-layer dedup pipeline: by source, cosine
similarity, type diversity (60% cap), per-page cap.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add GBRAIN_V0 spec, pluggable engine architecture, SQLite engine plan

GBRAIN_V0.md: full product spec with architecture decisions, CLI commands,
schema, search architecture, chunking strategies, first-time experience,
and future plans. ENGINES.md: pluggable engine interface, capability matrix,
how to add new backends. SQLITE_ENGINE.md: complete SQLite implementation
plan with schema, FTS5 setup, vector search options, and contributor guide.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add CLI with all commands

Full CLI dispatcher with 25+ commands: init (Supabase wizard), get, put,
delete, list, search, query (hybrid RRF), import (bulk with progress bar),
export (round-trip), embed, stats, health, tag/untag/tags, link/unlink/
backlinks/graph, timeline/timeline-add, history/revert, config, upgrade,
serve, call. Smart slug resolution on reads. Version snapshots on updates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add MCP stdio server with all brain tools

20 MCP tools mirroring CLI operations: get/put/delete/list pages,
search (keyword), query (hybrid RRF + expansion), tags, links with
graph traversal, timeline, stats, health, version history, and revert.
Auto-chunks and embeds on put_page. CLI and MCP share the same engine.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add 6 skill files and ClawHub manifest

Fat markdown skills for AI agents: ingest (meetings/docs/articles with
timeline merge), query (3-layer search + synthesis + citations), maintain
(health checks, stale detection, orphan audit), enrich (external API
enrichment), briefing (daily briefing compilation), migrate (universal
migration from Obsidian/Notion/Logseq/markdown/CSV/JSON/Roam).
ClawHub manifest for skill distribution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add README, CONTRIBUTING, update CLAUDE.md test references

README with quickstart, commands, architecture, library usage, MCP setup,
and links to design docs. CONTRIBUTING with setup, project structure,
and guides for adding commands and engines. CLAUDE.md updated to reference
actual test files instead of planned-but-unwritten import test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address adversarial review findings — 5 critical/high fixes

- revertToVersion: add page_id check to prevent cross-page data corruption
- traverseGraph: use UNION instead of UNION ALL for cycle safety
- embedAll: preserve all chunks when embedding stale subset only
- embedding: throw on retry exhaustion instead of returning zero vectors
- putPage: validate slugs to prevent path traversal on export

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.1.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: expand README with schema, install, search architecture, and motivation

Why it exists, how search works (with ASCII diagram), full database schema
with all 9 tables and index details, chunking strategies explained, storage
estimates, setup wizard walkthrough, knowledge model with example page,
library usage with more examples, expanded skills table.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: add MIT license (Copyright 2026 Garry Tan)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add OpenClaw install flow as primary option in README

OpenClaw users just say "install gbrain" and the orchestrator handles
everything: package install, Supabase setup wizard, skill registration.
Shows the conversational interface for querying, ingesting, and briefings.
ClawHub and standalone CLI paths follow as alternatives.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add prerequisites and explicit OpenClaw install instructions

Prerequisites table listing Supabase, OpenAI, and Anthropic dependencies
with links. Environment variable setup. Explicit step-by-step prompt for
OpenClaw users showing exactly what to tell the orchestrator. Note that
search degrades gracefully without API keys (keyword-only without OpenAI,
no expansion without Anthropic).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: scrub named references, add PG essay demo section to README

Replace all Pedro/Brex/Jensen Huang/River AI examples with Paul Graham
essay examples using the kindling corpus. Add "Try it" section to README
showing the power of hybrid search on PG essays in 90 seconds. Update
test fixtures to use concept pages instead of person pages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-05 12:48:10 -07:00