* fix(sync): honor --dry-run in full-sync path + expose embedded count
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
The full-sync path (performFullSync) previously called runImport() even
when opts.dryRun was true, silently writing to the DB and advancing
sync.last_commit. `gbrain sync --dry-run` on a fresh brain (or with
--full) would mutate state without warning.
Fix:
- performFullSync now early-returns a `dry_run` SyncResult when
opts.dryRun is set. Walks the repo via collectMarkdownFiles +
isSyncable to count what WOULD be imported. No writes, no git
state advance.
- SyncResult gains an `embedded: number` field (required). Tracks
pages re-embedded during the sync's auto-embed step. Existing
return sites set 0; the synced + first_sync paths set real counts
(best-estimate until commit 2 sharpens runEmbedCore's return type).
- first_sync path now returns real added + chunksCreated counts
from runImport instead of hardcoded zeros.
- printSyncResult shows embedded count in human output.
Tests (test/sync.test.ts, new `performSync dry-run never writes`
block, PGLite + temp git repo, no DATABASE_URL required):
- first-sync --dry-run: no pages, no sync.last_commit
- incremental --dry-run after real sync: bookmark unchanged
- --full --dry-run: no reimport, bookmark unchanged
- SyncResult.embedded is a number
Codex outside-voice caught this. Would have shipped silent DB writes
on dry-run for anyone using `gbrain sync --dry-run --full`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(embed): add dry-run mode + return EmbedResult with counts
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
runEmbedCore previously returned Promise<void> and had no dry-run mode.
That made it impossible for runCycle to (a) report accurate embedded
counts or (b) honor --dry-run without also skipping the entire embed
phase (which would have required runCycle to know embed's internal
semantics — a layering violation).
Changes:
- EmbedOpts gains `dryRun?: boolean`. When set, embedPage and
embedAll enumerate stale chunks (or would-be-created chunks for
unchunked pages, via local chunkText without engine.upsertChunks)
but never call embedBatch and never write to the engine.
- runEmbedCore: Promise<void> -> Promise<EmbedResult>. Result shape:
{ embedded, skipped, would_embed, total_chunks, pages_processed,
dryRun }.
embedded = chunks newly embedded (0 in dryRun).
would_embed = chunks that WOULD be embedded (0 in non-dryRun).
skipped = chunks with pre-existing embeddings.
- runEmbed CLI wrapper honors --dry-run flag and returns the result
through. `gbrain embed --stale --dry-run` is now a safe preview.
- Callers ignoring the return value (sync auto-embed, autopilot
inline fallback, jobs.ts handlers, CLI) keep compiling — the new
return type is additive for `await` callers.
Tests (test/embed.test.ts, new `runEmbedCore --dry-run` block, uses
the existing mock.module embedBatch pattern, no API key required):
- dry-run --all: zero embedBatch calls, zero upsertChunks calls,
would_embed matches stale chunk total
- dry-run --stale correctly splits stale vs already-embedded counts
- dry-run --slugs on a single page tallies per-chunk counts
- non-dry-run regression guard: embedded count matches across
concurrent workers
Codex outside-voice flagged the Promise<void> return as a blocker for
accurate CycleReport.totals.pages_embedded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(orphans): engine-injected queries, drop db.getConnection() global
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
findOrphans + queryOrphanPages previously reached into the postgres-js
singleton via db.getConnection(), which (a) didn't compose with
runCycle's explicit-engine contract and (b) was wrong for PGLite test
fixtures and for any caller not using the default global connection.
Codex outside-voice flagged this as a blocker.
Changes:
- BrainEngine interface gains findOrphanPages() — returns pages with
no inbound links via the same NOT EXISTS anti-join. Implemented on
both postgres-engine (sql tag) and pglite-engine (db.query).
- findOrphans signature: findOrphans(engine, { includePseudo }).
Engine is required. Uses engine.findOrphanPages() and
engine.getStats().page_count instead of raw SQL + global counts.
- queryOrphanPages signature: queryOrphanPages(engine). Delegates to
engine.findOrphanPages().
- src/commands/orphans.ts drops the `import * as db` — no more
global-state coupling.
- Callers updated: src/core/operations.ts find_orphans handler now
passes ctx.engine through; runOrphans CLI entry uses its engine arg.
- No signature change needed in cli.ts (it was already passing engine
via CLI_ONLY dispatch).
Tests (test/orphans.test.ts, new `findOrphans (engine-injected)`
describe block, PGLite in-memory, no DATABASE_URL required):
- links correctly scope orphans (alice links to bob -> bob not
an orphan; alice is)
- includePseudo:true surfaces _atlas-style pages
- queryOrphanPages delegates to passed engine
- empty brain returns {orphans: [], total_pages: 0} without crashing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): add runCycle primitive in src/core/cycle.ts
The brain maintenance cycle as a single function. Six phases in
semantically-driven order (fix files → sync → extract → embed →
report orphans). Pure composition of existing library calls — no
execSync, no subprocess anti-patterns, no regex-parsed output.
┌───────────────────────────────────────────────────┐
│ runCycle(engine, opts) → CycleReport │
│ Phase 1: lint --fix (fs writes) │
│ Phase 2: backlinks --fix (fs writes) │
│ Phase 3: sync (DB picks up 1+2) │
│ Phase 4: extract (DB picks up links) │
│ Phase 5: embed --stale (DB writes) │
│ Phase 6: orphans (DB read, report) │
└───────────────────────────────────────────────────┘
Why the commit-4 primitive:
- CEO + Eng + Codex reviews all converged on "extract one cycle
function, wire both dream and autopilot through it." Two CLIs,
one definition of what the brain does overnight.
- Phase order was wrong in PR #309's original dream.ts (sync
before lint+backlinks lost the "fix files, then index them"
semantic).
- This commit is the bisectable foundation; commit 5 (dream)
and commit 6 (autopilot+jobs) just call into it.
Coordination — the codex-flagged blocker:
Session-scoped pg_try_advisory_lock does not survive PgBouncer
transaction pooling (the v0.15.4 fix made pooled connections the
default). Replaced with a DB lock table (gbrain_cycle_locks) that
works through every pooler:
- Acquire: INSERT ... ON CONFLICT DO UPDATE ... WHERE ttl < NOW()
- Refresh: UPDATE ttl_expires_at between phases via hook
- Release: DELETE in finally{}
- TTL: 30 min; crashed holders auto-release
PGLite / engine=null path uses a file lock at ~/.gbrain/cycle.lock
with PID liveness check. kill(pid, 0) with EPERM treated as alive
(so init/launchd-pid holders aren't mis-classified as stale).
Lock-skip: only phases that mutate state (lint, backlinks, sync,
extract, embed) trigger lock acquisition. orphans is read-only.
Single-phase --phase orphans runs never block on a held lock.
Engine-null mode preserved: filesystem phases run, DB phases skip
with {status:'skipped', reason:'no_database'}. Matches current
dream's capability that would have been lost if runCycle required
a connected engine.
Contract details:
- CycleReport has schema_version:"1" (stable, additive) so agents
consuming --json can rely on the shape
- status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'.
'clean' = ran successfully with zero activity; agents trivially
detect a healthy brain.
- PhaseResult.error: { class, code, message, hint?, docs_url? }
(Stripe-API-tier structured failure info) when status='fail'
- yieldBetweenPhases hook: awaited between EVERY phase and before
return, runs even after phase failure, exceptions logged but
non-fatal. Required so the Minions autopilot-cycle handler can
renew its job lock between phases (prevents the v0.14 stall-death
regression codex flagged).
- git pull explicit: opts.pull defaults to false (cron-safe).
Autopilot daemon callers opt in if user configured it.
- extract phase doesn't have a dry-run mode in the underlying
library function, so runCycle honestly skips extract when
dryRun=true (status:'skipped', reason:'no_dry_run_support').
Schema migration v16: gbrain_cycle_locks table + idx_cycle_locks_ttl.
Also appended to src/schema.sql and src/core/pglite-schema.ts for
fresh installs. schema-embedded.ts regenerated via build:schema.
Tests (test/core/cycle.test.ts, PGLite in-memory + mocked library
functions, no DATABASE_URL required):
- dryRun × phases matrix: dryRun:true reaches lint/backlinks/sync/
embed; extract is honestly skipped
- Phase selection: default runs all 6 in order; --phase lint runs
only lint; --phase orphans runs only orphans
- Lock semantics: acquire + release on mutating phases, skip
entirely for read-only selections
- cycle_already_running: seeded live-holder lock → status:skipped,
zero phase runs; TTL-expired holder → auto-claimed
- Engine null: filesystem phases run, DB phases skip
- File lock (engine=null) blocks when PID 1 holds lock with fresh
mtime — exercises the PID liveness branch including EPERM
- Status derivation: 'ok' vs 'clean' vs 'partial' vs 'skipped'
- yieldBetweenPhases called N times, hook exceptions non-fatal
Next: commit 5 rewrites dream.ts as a thin CLI alias over runCycle,
commit 6 migrates autopilot daemon + jobs.ts handler to delegate to
runCycle too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dream): add gbrain dream CLI as a thin alias over runCycle
`gbrain dream` is the README brand-promise command: "the agent runs
while I sleep, the dream cycle ... I wake up and the brain is smarter."
Cron-friendly, JSON-reportable, phase-selectable. Same maintenance
cycle as `gbrain autopilot`, just scheduled differently — both
converge on runCycle (added in commit 4) so there's one source of
truth for what happens overnight.
Contract:
gbrain dream # full 6-phase cycle
gbrain dream --dry-run # preview, no writes
gbrain dream --json # CycleReport JSON (agent-readable)
gbrain dream --phase <name> # single-phase run
gbrain dream --pull # git pull before syncing
gbrain dream --dir /path/to/brain # explicit brain location
Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log
Behavior details:
- Brain-dir resolution: requires explicit --dir OR sync.repo_path
in engine config. No more walk-up-cwd-for-.git footgun that
PR #309's original dream.ts had (would lint unrelated git repos).
- engine=null mode preserved via cli.ts's try/catch around
connectEngine — filesystem phases (lint, backlinks) still run
without a DB, DB phases report skipped/no_database in the output.
- status=clean prints "Brain is healthy. N phase(s) checked in Ns."
status=skipped prints the reason (cycle_already_running, etc.).
Partial/failed prints the phase-by-phase detail.
- Exit code 1 when status=failed (cron spots real problems).
'partial' is not a failure — warnings shouldn't page you.
- --help text cross-references `autopilot --install` for users
who want continuous maintenance as a daemon.
CLI registration (src/cli.ts):
- 'dream' added to CLI_ONLY
- handleCliOnly has a pre-engine branch mirroring doctor's pattern:
try connectEngine() → ok path; catch → runDream(null, args) so
filesystem phases still run when DB is down
- Help text updated with one-line dream entry and autopilot cross-ref
Tests (test/dream.test.ts, real PGLite + real library calls, no mocks
to avoid `mock.module` leakage across test files):
- brainDir resolution: explicit --dir wins, engine config fallback,
missing + nonexistent errors
- phase selection: --phase lint|orphans produces single-phase report
- phase validation: --phase garbage exits 1
- output: --json parses as CycleReport with schema_version:"1"
- human output mentions "Brain is healthy" on clean status
- dry-run: cycle runs but DB stays untouched
- exit code: clean/ok/partial do not call process.exit
Also (test/core/cycle.test.ts): refactored to use beforeAll/afterAll
with one shared PGLite engine per describe + truncateCycleLocks
between tests. Cuts test time from ~11s to ~4s; avoids the 15-migration
penalty per test that was causing parallel-suite timeout flakes.
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.17.0 — autopilot + jobs delegate to runCycle (unifies the cycle)
Autopilot daemon (`--inline` path) and Minions `autopilot-cycle`
handler both now delegate to `runCycle` (introduced in commit 4).
Three callers, one cycle definition:
1. `gbrain dream` — one-shot cron cycle
2. `gbrain autopilot` daemon inline path — scheduled cycles
3. `autopilot-cycle` Minions handler — durable queue with retry
All three share:
- Same 6 phases in same order (lint → backlinks → sync → extract →
embed → orphans)
- Same DB lock table coordination (`gbrain_cycle_locks`)
- Same yieldBetweenPhases discipline (prevents v0.14 stall-death)
- Same structured CycleReport output
Autopilot inline path gains lint + orphan sweep that the old path
skipped. Minions autopilot-cycle handler also gains lint + orphans.
Users who run `gbrain autopilot --install` see 6-phase reports in
`gbrain jobs get <id>` starting on next interval. No config change
required.
Changes:
- `src/commands/autopilot.ts`: inline fallback path (~20 lines)
replaces the ~22-line sync+extract+embed sequence with a single
runCycle call. Uses pull:true (matches pre-v0.17 autopilot
behavior). Uses setImmediate yield hook. Status/failure reporting
derives from CycleReport.status. `--help` cross-references `gbrain
dream` for one-shot use.
- `src/commands/jobs.ts:579` (`autopilot-cycle` handler): replaces
the 4-step try/catch sequence with a runCycle call. Returns
`{ partial, status, report }` so `gbrain jobs get <id>` shows the
full structured CycleReport. Preserves partial-failure semantic
(one phase failing does NOT throw; next cycle still runs).
yieldBetweenPhases yields the event loop between phases for the
worker's lock-renewal timer.
Release scaffolding:
- VERSION: 0.16.0 → 0.17.0
- CHANGELOG.md: v0.17.0 entry in GStack voice — headline, numbers
table, "what this means" paragraph, "To take advantage" block
per CLAUDE.md post-ship rules. Itemized changes below the fold.
Credit to @Wintermute for the original PR #309 thesis.
- skills/migrations/v0.17.0.md: documents what changed for
upgrading users. No mechanical action required — schema migration
v16 (cycle locks table) + handler delegation both apply
automatically. Includes opt-out paths for users who don't want
their daemon modifying files (use `dream --phase orphans` in cron
and skip autopilot-install, or other explicit configs).
- CLAUDE.md: new entries for `src/core/cycle.ts` and
`src/commands/dream.ts` with contract details.
Tests: no new test file needed for this commit — the cycle primitive
is extensively tested in test/core/cycle.test.ts (18 cases), dream
in test/dream.test.ts (11), and autopilot's delegation is mechanical
(calls runCycle with specific opts). The handler contract is covered
implicitly: if runCycle returns a CycleReport, the handler wraps it
in `{ partial, status, report }` — nothing else to assert.
Verified:
- `bun test test/autopilot-install.test.ts test/autopilot-resolve-cli.test.ts test/core/cycle.test.ts test/dream.test.ts` → 37 pass, 0 fail
Completes the v0.17.0 feature: 6 bisectable commits on one branch
(garrytan/v0.17-dream-cycle), ready to push as one PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): add runCycle + dream E2E coverage against real Postgres
Gap from the v0.17 commit series: PR #321 shipped unit-level tests
for runCycle (test/core/cycle.test.ts) and dream (test/dream.test.ts)
but no E2E coverage that exercises the real Postgres paths. Filling
that in before merge.
test/e2e/cycle.test.ts (6 cases):
- schema migration v16 created gbrain_cycle_locks + index
- dry-run full cycle: zero DB writes + lock table empty after
- live cycle: pages + chunks materialize, sync.last_commit set
- concurrent cycle blocked by lock → status:'skipped'
- TTL-expired lock auto-claimed (crashed-holder recovery)
- --phase orphans skips lock entirely (read-only optimization)
test/e2e/dream.test.ts (3 cases):
- dream --dry-run --json emits valid CycleReport + DB stays empty
- dream (no --dry-run) syncs pages into real DB
- dream --phase orphans doesn't touch the cycle-lock table
Both files mock embedBatch via mock.module so the embed phase never
calls OpenAI even when the full 6-phase cycle runs (zero API cost,
zero flakiness from network calls).
Verified locally:
- `docker run pgvector/pgvector:pg16` on port 5434
- `DATABASE_URL=... bun test test/e2e/cycle.test.ts test/e2e/dream.test.ts` → 9 pass, 0 fail
- Full E2E suite (`bun run test:e2e`): 16 files, 150 tests, 0 fail
- Container torn down after: `docker stop + rm gbrain-test-pg`
Per CLAUDE.md E2E test DB lifecycle. These tests skip gracefully when
DATABASE_URL isn't set (via hasDatabase() helper + describe.skip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
123 lines
5.1 KiB
TypeScript
123 lines
5.1 KiB
TypeScript
/**
|
|
* Tests for registerBuiltinHandlers in src/commands/jobs.ts.
|
|
*
|
|
* Covers:
|
|
* - Every expected handler name is registered.
|
|
* - autopilot-cycle handler returns { partial, status, report } (v0.17
|
|
* runCycle-backed shape) when any step fails — does NOT throw itself
|
|
* (critical invariant: an intermittent phase failure must not cause
|
|
* the Minion to retry and block every future cycle).
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll, mock } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { MinionWorker } from '../src/core/minions/worker.ts';
|
|
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
let worker: MinionWorker;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
worker = new MinionWorker(engine, { queue: 'test' });
|
|
await registerBuiltinHandlers(worker, engine);
|
|
}, 30_000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
describe('registerBuiltinHandlers', () => {
|
|
test('registers all built-in handler names', () => {
|
|
const names = worker.registeredNames;
|
|
// Existing handlers from pre-v0.11.1
|
|
expect(names).toContain('sync');
|
|
expect(names).toContain('embed');
|
|
expect(names).toContain('lint');
|
|
expect(names).toContain('import');
|
|
// New in v0.11.1 (Tier 1 + autopilot-cycle)
|
|
expect(names).toContain('extract');
|
|
expect(names).toContain('backlinks');
|
|
expect(names).toContain('autopilot-cycle');
|
|
});
|
|
|
|
test('total handler count includes all 7 names', () => {
|
|
expect(worker.registeredNames.length).toBeGreaterThanOrEqual(7);
|
|
});
|
|
});
|
|
|
|
describe('autopilot-cycle handler — partial failure does NOT throw', () => {
|
|
test('phase failure returns partial:true + structured report, no throw', async () => {
|
|
// Call the handler directly with a job pointing at a nonexistent repo.
|
|
// Filesystem-dependent phases (lint, backlinks, sync) all fail because
|
|
// the dir / .git repo isn't there. DB-dependent phases (extract,
|
|
// embed, orphans) run fine against the in-memory test engine.
|
|
//
|
|
// CRITICAL INVARIANT: the handler must return successfully even when
|
|
// phases fail. Throwing would cause the Minion to retry, blocking
|
|
// every future cycle on an intermittent bug. v0.17 moves this
|
|
// guarantee into runCycle itself (per-phase try/catch in cycle.ts).
|
|
const handler = (worker as any).handlers.get('autopilot-cycle');
|
|
expect(handler).toBeDefined();
|
|
|
|
const result = await handler({
|
|
data: { repoPath: '/definitely-does-not-exist-for-autopilot-test' },
|
|
signal: { aborted: false } as any,
|
|
job: { id: 1, name: 'autopilot-cycle' } as any,
|
|
});
|
|
|
|
expect(result).toBeDefined();
|
|
expect((result as any).partial).toBe(true);
|
|
// v0.17 shape: { partial, status, report }. The report's phases array
|
|
// replaces the old failed_steps list.
|
|
expect(['partial', 'failed']).toContain((result as any).status);
|
|
const report = (result as any).report;
|
|
expect(report).toBeDefined();
|
|
expect(report.schema_version).toBe('1');
|
|
expect(Array.isArray(report.phases)).toBe(true);
|
|
// The filesystem-dependent phases should have failed on a missing dir.
|
|
const failedPhases = report.phases
|
|
.filter((p: any) => p.status === 'fail')
|
|
.map((p: any) => p.phase);
|
|
expect(failedPhases).toContain('lint');
|
|
expect(failedPhases).toContain('backlinks');
|
|
expect(failedPhases).toContain('sync');
|
|
});
|
|
|
|
test('all phases succeed → result has structured report (smoke)', async () => {
|
|
// Smoke: invoke against a real (if empty) git repo. If every phase
|
|
// completes (or gracefully skips), the handler returns a result
|
|
// object with the full runCycle report. Some phases may still warn
|
|
// (empty repo has nothing to lint/sync) — the invariant is that the
|
|
// handler never throws.
|
|
const fs = await import('fs');
|
|
const { execSync } = await import('child_process');
|
|
const { tmpdir } = await import('os');
|
|
const { join } = await import('path');
|
|
const dir = fs.mkdtempSync(join(tmpdir(), 'gbrain-autopilot-cycle-'));
|
|
try {
|
|
execSync('git init', { cwd: dir, stdio: 'pipe' });
|
|
execSync('git config user.email test@example.com', { cwd: dir, stdio: 'pipe' });
|
|
execSync('git config user.name Test', { cwd: dir, stdio: 'pipe' });
|
|
execSync('git commit --allow-empty -m init', { cwd: dir, stdio: 'pipe' });
|
|
|
|
const handler = (worker as any).handlers.get('autopilot-cycle');
|
|
const result = await handler({
|
|
data: { repoPath: dir },
|
|
signal: { aborted: false } as any,
|
|
job: { id: 2, name: 'autopilot-cycle' } as any,
|
|
});
|
|
// The handler MUST return a result object, never throw, regardless
|
|
// of individual phase outcomes.
|
|
expect(result).toBeDefined();
|
|
expect(typeof (result as any).partial).toBe('boolean');
|
|
expect('report' in (result as any)).toBe(true);
|
|
expect((result as any).report.schema_version).toBe('1');
|
|
} finally {
|
|
fs.rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
}, 30_000);
|
|
});
|