triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: 8 root-cause fixes from /investigate (v0.14.2) (#259)

Browse Source 
* fix: 8 root-cause fixes from /investigate wave

Consolidated bundle of bug fixes from /investigate on the 8 deferred bugs.
Each fix was designed to go at the structural gap, not the symptom. Codex
verified 20 load-bearing claims on the plan; 12 triggered plan revisions.

Bug 2  — GBRAIN_POOL_SIZE env knob + init finally blocks (no auto-detect).
         Covers both the singleton pool (db.ts) and instance pool (import.ts:140).
Bug 3  — Centralize migration ledger writes in apply-migrations runner.
         Removed appendCompletedMigration from v0_11_0, v0_12_0, v0_12_2,
         v0_13_0, v0_13_1. Added 3-partial wedge cap + --force-retry reset.
         'complete wins' preserved; no partial can regress a completed migration.
Bug 5  — v0.14.0 migration registered. src/commands/migrations/v0_14_0.ts
         ships Phase A (ALTER minion_jobs.max_stalled SET DEFAULT 3) + Phase B
         (pending-host-work ping for shell-jobs adoption).
Bug 6/10 — jsonb_agg(DISTINCT ...) in legacy traverseGraph (both engines).
         Presentation-level dedup; schema still preserves provenance rows.
Bug 7  — doctor --fast reads DB URL source via getDbUrlSource() in config.ts.
         Precise message: 'Skipping DB checks (--fast mode, URL present from env)'
         replaces the misleading 'No database configured'.
Bug 8  — max_stalled default bumped 1→3 in schema-embedded.ts, pglite-schema.ts,
         schema.sql (new installs). v0_14_0 Phase A ALTER for existing installs.
         autopilot-cycle handler yields to event loop between phases so the
         worker's lock-renewal timer fires on huge brains. (Deep AbortSignal
         threading through runEmbedCore/runExtractCore/runBacklinksCore/performSync
         deferred to v0.15 queue polish.)
Bug 9  — Gate sync.last_commit on no-failures across all three sync paths
         (incremental, full via runImport, gbrain import git continuity).
         recordSyncFailures() helper + ~/.gbrain/sync-failures.jsonl with
         dedup key path+commit+error-hash. New flags: --skip-failed (ack) +
         --retry-failed (re-attempt). Doctor surfaces unacknowledged failures.
Bug 11 — brain_score breakdown fields on BrainHealth (embed_coverage_score,
         link_density_score, timeline_coverage_score, no_orphans_score,
         no_dead_links_score); sum equals brain_score by construction.
         dead_links now on the type (resolves featuresTeaserForDoctor drift).
         orphan_pages kept as 'islanded' (no inbound AND no outbound) and
         docs updated to match — explicit semantic instead of doc drift.

New tests: test/traverse-graph-dedup.test.ts, test/sync-failures.test.ts,
test/brain-score-breakdown.test.ts, test/migration-resume.test.ts,
test/migrations-v0_14_0.test.ts. Extended: migrate, doctor, apply-migrations.

All 1696 unit tests pass locally. postgres-jsonb E2E regression unchanged
(none of these touch the JSONB write surface).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: v0.14.2 CHANGELOG + CLAUDE.md; align migration-flow E2E with runner-owned ledger

CHANGELOG: v0.14.2 entry in the standard release-summary format
(two-line headline + lead + numbers table + "what this means" +
"To take advantage of v0.14.2" self-repair block + itemized
changes grouped by reliability / observability / graph correctness /
new migration / tests / deferred-to-v0.15).

CLAUDE.md: new "Key commands added in v0.14.2" section covers
--skip-failed, --retry-failed, --force-retry, GBRAIN_POOL_SIZE env,
and the new doctor checks (sync_failures, brain_score breakdown).
Migration orchestrator docs updated to describe v0_14_0.ts + the
runner-owned ledger contract from Bug 3.

test/e2e/migration-flow.test.ts: three assertions updated to match
the Bug 3 contract — orchestrators no longer append to completed.jsonl
directly, so direct-orchestrator E2E calls leave the ledger empty.
Preferences assertions remain (that's still the orchestrator's side
of the contract). Runner's ledger write is covered by the unit suite
(test/apply-migrations.test.ts + test/migration-resume.test.ts).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-20 23:14:38 +08:00
committed by GitHub
parent ebfbd5e6f7
commit b5fa3d044a
37 changed files with 1804 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
CHANGELOG.md
View File

@@ -2,6 +2,82 @@
All notable changes to GBrain will be documented in this file.
## [0.14.2] - 2026-04-20
## **Eight deferred bugs, root-cause fixes, one clean wave.**
## **Sync stops losing files. Migrations stop retrying forever. Pooler users get a knob.**
Eight bugs were previously scoped out of a PR after Codex review caught wrong root causes and unimplementable architectures. v0.14.2 takes each back to the actual code and fixes the structural gap. `/plan-eng-review` + `/codex consult` verified every load-bearing claim before a single line of code ran (20 findings, 12 triggered plan revisions before implementation).
The practical wins for a busy brain: `gbrain sync` no longer silently loses files with unquoted-colon YAML titles across any of the three sync paths. `gbrain upgrade` can't get stuck in an infinite retry loop on a wedged migration (3-partial cap + `--force-retry` escape hatch). Supabase pooler users have `GBRAIN_POOL_SIZE` to throttle without touching schemas. `gbrain doctor --fast` tells you WHY it's skipping DB checks instead of lying about no database being configured. `brain_score` gets a breakdown so 79/100 tells you which component is costing you the 21 points.
### The numbers that matter
Measured on this branch's diff against origin/master:
| Metric | BEFORE v0.14.2 | AFTER v0.14.2 | Δ |
|---------------------------------------------------|---------------------|-----------------------------|-------------------------|
| Sync paths that silently drop files on YAML break | 3 of 3 | 0 of 3 | **no more silent loss** |
| Wedged-migration retry loops | infinite | 3-partial cap + `--force-retry` | bounded |
| Pool-size knob for Supabase pooler | none | `GBRAIN_POOL_SIZE` env | **first-class knob** |
| `doctor --fast` messages | 1 catch-all | 3 source-specific | honest signal |
| `brain_score` observability | one number | 5-field breakdown (sum == total) | diagnosable |
| Duplicate edges in `gbrain graph` output | leaked per-origin | deduped at presentation | schema preserved |
| `minion_jobs.max_stalled` default | 1 (dead-letter on first stall) | 3 | autopilot survives long embed runs |
| New + extended unit tests | 1696 | **1743 (+47 + 119 new assertions)** | +47 |
| Root-cause fixes vs symptom patches | 0 | **8 / 8** | structural |
### What this means for you
Your agent's feedback loops tighten. When sync blocks, doctor surfaces the exact file with the YAML problem and the commit where it showed up. When a migration gets stuck, there's a cap and a clear escape. When you're on Supabase's transaction pooler and `gbrain upgrade` spawns subprocesses, set `GBRAIN_POOL_SIZE=2` and stop MaxClients crashes. Run `gbrain doctor` and the `brain_score` breakdown points at what to fix first: embed coverage, link density, timeline coverage, orphans, or dead links.
## To take advantage of v0.14.2
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
1. **Run the orchestrator manually:**
```bash
gbrain apply-migrations --yes
```
2. **Supabase pooler users (port 6543) now have a knob.** If you hit MaxClients during upgrades, set `GBRAIN_POOL_SIZE=2` (or lower) in your environment before running `gbrain upgrade`.
3. **Check sync health after the upgrade:**
```bash
gbrain doctor
```
If it warns about `sync_failures`, the paths and errors are in `~/.gbrain/sync-failures.jsonl`. Fix the offending YAML frontmatter and re-run `gbrain sync`, or use `gbrain sync --skip-failed` to acknowledge known-broken files and advance past them.
4. **Wedged migrations:** If `doctor` ever flags a version with 3 consecutive partials, run `gbrain apply-migrations --force-retry vX.Y.Z` to reset the state machine, then `gbrain apply-migrations --yes` to re-attempt.
5. **If any step fails or the numbers look wrong,** file an issue: https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- which step broke
### Itemized changes
#### Reliability
- **Bug 2: `GBRAIN_POOL_SIZE` env knob** (`src/core/db.ts`, `src/commands/import.ts`). Honored by both the singleton pool and the parallel-import worker pool. Defaults to 10; lower for Supabase transaction pooler. `initPostgres` / `initPGLite` now wrap lifecycle in `try { ... } finally { await engine.disconnect() }`.
- **Bug 3: Migration ledger centralization + wedge cap** (`src/commands/apply-migrations.ts`, `src/core/preferences.ts`). Runner owns all ledger writes. 3 consecutive partials = wedged, skipped with a loud message. New `--force-retry <version>` flag writes a `'retry'` marker without faking success. `complete` status never regresses. `appendCompletedMigration` is idempotent on double-complete.
- **Bug 8: `max_stalled` default 1 → 3** (`src/core/schema-embedded.ts`, `src/core/pglite-schema.ts`, `src/schema.sql`). First lock-lost tick no longer dead-letters. `v0_14_0` Phase A ALTERs existing installs. `autopilot-cycle` handler yields to the event loop between phases so the worker's lock-renewal timer fires.
- **Bug 9: Sync gate + acknowledge mechanism** (`src/commands/sync.ts`, `src/commands/import.ts`, `src/core/sync.ts`). All 3 sync paths (incremental, full via `runImport`, `gbrain import` git continuity) gate `sync.last_commit` on no-failures. Failures append to `~/.gbrain/sync-failures.jsonl` with dedup key. New `gbrain sync --skip-failed` + `--retry-failed` flags. Doctor surfaces unacknowledged failures.
#### Observability
- **Bug 7: `doctor --fast` source-aware messages** (`src/core/config.ts`, `src/cli.ts`, `src/commands/doctor.ts`). New `getDbUrlSource()` returns `'env:GBRAIN_DATABASE_URL' | 'env:DATABASE_URL' | 'config-file' | null`. Doctor emits `Skipping DB checks (--fast mode, URL present from env:GBRAIN_DATABASE_URL)` when applicable.
- **Bug 11: `brain_score` breakdown + metric clarity** (`src/core/types.ts`, both engines' `getHealth()`). Added `embed_coverage_score`, `link_density_score`, `timeline_coverage_score`, `no_orphans_score`, `no_dead_links_score`. Sum equals `brain_score` by construction. `dead_links` now on `BrainHealth` (resolves a pre-existing `featuresTeaserForDoctor` drift). `orphan_pages` docs clarified — it's "islanded" (no inbound AND no outbound), not the stricter "zero inbound" graph definition.
#### Graph correctness
- **Bug 6/10: `jsonb_agg(DISTINCT ...)` in legacy `traverseGraph`** (`src/core/postgres-engine.ts`, `src/core/pglite-engine.ts`). Presentation-level dedup only — the schema continues to preserve per-`origin_page_id` / per-`link_source` provenance rows. Fixes duplicate edges like `works_at → companies/brex` appearing twice in `gbrain graph`.
#### New migration
- **Bug 5: `v0_14_0` migration registered** (`src/commands/migrations/v0_14_0.ts`). Phase A: `ALTER minion_jobs.max_stalled SET DEFAULT 3` (idempotent). Phase B: emits `pending-host-work.jsonl` entry pointing at `skills/migrations/v0.14.0.md` for shell-jobs adoption. Registered in `src/commands/migrations/index.ts`. `package.json` bumped to 0.14.2 (0.14.0 and 0.14.1 were taken by upstream during this branch's work).
#### Tests
- New: `test/traverse-graph-dedup.test.ts`, `test/sync-failures.test.ts`, `test/brain-score-breakdown.test.ts`, `test/migration-resume.test.ts`, `test/migrations-v0_14_0.test.ts`.
- Extended: `test/migrate.test.ts` (`resolvePoolSize`), `test/doctor.test.ts` (`dbSource`), `test/apply-migrations.test.ts` (`skippedFuture` includes `0.14.0`).
- E2E updated: `test/e2e/migration-flow.test.ts` assertions aligned with the new runner-owned-ledger contract (orchestrator no longer writes completed.jsonl directly).
#### Deferred to v0.15
- Deep `AbortSignal` threading through `runEmbedCore` / `runExtractCore` / `runBacklinksCore` / `performSync`. Between-phase yield addresses the Bug 8 lock-renewal root cause; mid-phase cancellation on huge brains belongs in the queue-polish PR.
- `failJobFromSweeper` for `handleTimeouts` / `handleStalled`. Current direct `status='dead'` writes kept.
## [0.14.1] - 2026-04-20
## **`gbrain doctor` stops crying wolf on DRY, and now repairs the real ones.**

9
CLAUDE.md
View File

@@ -65,7 +65,7 @@ strict behavior when unset.
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). All orchestrators are idempotent and resumable from `partial` status.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3, pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run]`: health checks. v0.12.3 adds two reliability detection checks: `jsonb_integrity` (scans pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata for `jsonb_typeof='string'` rows left over from v0.12.0) and `markdown_body_completeness` (flags pages whose compiled_truth is <30% of raw source when raw has multiple H2/H3 boundaries). Fix hints point at `gbrain repair-jsonb` and `gbrain sync --force`. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing.
@@ -145,6 +145,13 @@ Key commands added in v0.12.3:
- `gbrain orphans [--json] [--count] [--include-pseudo]` — surface pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. The natural consumer of the v0.12.0 knowledge graph layer: once edges are captured, find the gaps.
- `gbrain doctor` gains two new reliability detection checks: `jsonb_integrity` (v0.12.0 Postgres double-encode damage) and `markdown_body_completeness` (pages truncated by the old splitBody bug). Detection only; fix hints point at `gbrain repair-jsonb` and `gbrain sync --force`.
Key commands added in v0.14.2:
- `gbrain sync --skip-failed` — acknowledge the current set of failed-parse files recorded in `~/.gbrain/sync-failures.jsonl` so the sync bookmark advances past them. Doctor's `sync_failures` check shows previously-skipped as "all acknowledged" instead of warning.
- `gbrain sync --retry-failed` — re-walk the unacknowledged failures and re-attempt parsing. If the files now succeed, they clear from the set and the bookmark advances naturally.
- `gbrain apply-migrations --force-retry <version>` — reset a wedged migration (3 consecutive partials with no completion) by appending a `'retry'` marker. Next `apply-migrations --yes` treats the version as fresh. `complete` status never regresses to `partial` either before or after a retry marker.
- `GBRAIN_POOL_SIZE` env var — honored by both the singleton pool (`src/core/db.ts`) and the parallel-import worker pool (`src/commands/import.ts`). Default is 10; lower to 2 for Supabase transaction pooler to avoid MaxClients crashes during `gbrain upgrade` subprocess spawns. Read at call time via `resolvePoolSize()`.
- `gbrain doctor` gains two new checks: `sync_failures` (surfaces unacknowledged parse failures with exact paths + fix hints) and `brain_score` (renders the 5-component breakdown when score < 100: embed coverage / 35, link density / 25, timeline coverage / 15, orphans / 15, dead links / 10 — sum equals total).
## Testing
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run

2
VERSION
View File

@@ -1 +1 @@
0.14.1
0.14.2

2
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.14.1",
"version": "0.14.2",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",

7
src/cli.ts
View File

@@ -332,8 +332,11 @@ async function handleCliOnly(command: string, args: string[]) {
// Doctor runs filesystem checks first (no DB needed), then DB checks.
// --fast skips DB checks entirely.
const { runDoctor } = await import('./commands/doctor.ts');
const { getDbUrlSource } = await import('./core/config.ts');
if (args.includes('--fast')) {
await runDoctor(null, args);
// Pass the DB URL source so doctor can tell "no config at all" from
// "user chose --fast while config is present".
await runDoctor(null, args, getDbUrlSource());
} else {
try {
const eng = await connectEngine();
@@ -341,7 +344,7 @@ async function handleCliOnly(command: string, args: string[]) {
await eng.disconnect();
} catch {
// DB unavailable — still run filesystem checks
await runDoctor(null, args);
await runDoctor(null, args, getDbUrlSource());
}
}
return;

117
src/commands/apply-migrations.ts
View File

@@ -14,9 +14,12 @@
import { VERSION } from '../version.ts';
import { loadConfig } from '../core/config.ts';
import { loadCompletedMigrations, type CompletedMigrationEntry } from '../core/preferences.ts';
import { loadCompletedMigrations, appendCompletedMigration, type CompletedMigrationEntry } from '../core/preferences.ts';
import { migrations, compareVersions, type Migration, type OrchestratorOpts } from './migrations/index.ts';
/** Bug 3 — max consecutive partials before we wedge a migration. */
const MAX_CONSECUTIVE_PARTIALS = 3;
interface ApplyMigrationsArgs {
list: boolean;
dryRun: boolean;
@@ -26,6 +29,8 @@ interface ApplyMigrationsArgs {
specificMigration?: string;
hostDir?: string;
noAutopilotInstall: boolean;
/** Bug 3 — explicit reset for a wedged migration. Writes a 'retry' marker. */
forceRetry?: string;
help: boolean;
}
@@ -49,6 +54,7 @@ function parseArgs(args: string[]): ApplyMigrationsArgs {
specificMigration: val('--migration'),
hostDir: val('--host-dir'),
noAutopilotInstall: has('--no-autopilot-install'),
forceRetry: val('--force-retry'),
help: has('--help') || has('-h'),
};
}
@@ -63,6 +69,10 @@ Usage:
gbrain apply-migrations --list Show applied + pending migrations.
gbrain apply-migrations --migration vX.Y.Z
Force-run a specific migration by version.
gbrain apply-migrations --force-retry vX.Y.Z
Clear a wedged migration (3+ consecutive
partials). Writes a 'retry' marker so the
next run treats it as fresh.
Flags:
--mode <always|pain_triggered|off> Set minion_mode without prompting.
@@ -94,14 +104,38 @@ function indexCompleted(entries: CompletedMigrationEntry[]): CompletedIndex {
: { byVersion: new Map() };
}
/** Returns the resolved status for a migration based on its entries. */
/**
* Returns the resolved status for a migration based on its entries.
*
* Semantics (Bug 3 — keep "complete wins" safety):
* - If any entry is `complete`, the version is complete. Terminal state.
* - Otherwise, if the latest entry is `retry`, the version is pending
* (user requested a fresh attempt).
* - Otherwise, if any entry is `partial`, the version is partial.
* - Otherwise, pending.
*
* `complete` never regresses. A later accidental `partial` append cannot
* undo a completed migration.
*/
function statusForVersion(
version: string,
idx: CompletedIndex,
): 'complete' | 'partial' | 'pending' {
): 'complete' | 'partial' | 'pending' | 'wedged' {
const entries = idx.byVersion.get(version) ?? [];
if (entries.length === 0) return 'pending';
if (entries.some(e => e.status === 'complete')) return 'complete';
const latest = entries[entries.length - 1];
if (latest.status === 'retry') return 'pending';
// Bug 3 attempt cap — count consecutive partials from the end (stopping
// at any 'retry' or 'complete'). If we hit MAX_CONSECUTIVE_PARTIALS,
// the migration is wedged and needs explicit --force-retry to try again.
let consecutive = 0;
for (let i = entries.length - 1; i >= 0; i--) {
const e = entries[i];
if (e.status === 'partial') consecutive++;
else break;
}
if (consecutive >= MAX_CONSECUTIVE_PARTIALS) return 'wedged';
if (entries.some(e => e.status === 'partial')) return 'partial';
return 'pending';
}
@@ -111,6 +145,7 @@ interface Plan {
partial: Migration[];
pending: Migration[];
skippedFuture: Migration[];
wedged: Migration[];
}
/**
@@ -127,7 +162,7 @@ interface Plan {
* skip v0.11.0 when running v0.11.1. Compare against completed.jsonl.
*/
function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: string): Plan {
const plan: Plan = { applied: [], partial: [], pending: [], skippedFuture: [] };
const plan: Plan = { applied: [], partial: [], pending: [], skippedFuture: [], wedged: [] };
for (const m of migrations) {
if (filterVersion && m.version !== filterVersion) continue;
if (compareVersions(m.version, installed) > 0) {
@@ -137,6 +172,7 @@ function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: strin
const status = statusForVersion(m.version, idx);
if (status === 'complete') plan.applied.push(m);
else if (status === 'partial') plan.partial.push(m);
else if (status === 'wedged') plan.wedged.push(m);
else plan.pending.push(m);
}
return plan;
@@ -149,6 +185,7 @@ function printList(plan: Plan, installed: string): void {
const rows: Array<{ status: string; m: Migration }> = [
...plan.applied.map(m => ({ status: 'applied', m })),
...plan.partial.map(m => ({ status: 'partial', m })),
...plan.wedged.map(m => ({ status: 'wedged', m })),
...plan.pending.map(m => ({ status: 'pending', m })),
...plan.skippedFuture.map(m => ({ status: 'future', m })),
];
@@ -227,10 +264,37 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
return;
}
// Bug 3 — --force-retry: write an explicit reset marker for a wedged
// migration, then return. User re-runs `gbrain apply-migrations --yes`
// to actually re-attempt.
if (cli.forceRetry) {
const target = migrations.find(m => m.version === cli.forceRetry);
if (!target) {
console.error(`No migration registered with version "${cli.forceRetry}". Run \`gbrain apply-migrations --list\`.`);
process.exit(2);
}
appendCompletedMigration({ version: cli.forceRetry, status: 'retry' });
console.log(`Wrote 'retry' marker for v${cli.forceRetry}. Run \`gbrain apply-migrations --yes\` to re-attempt.`);
return;
}
const completed = loadCompletedMigrations();
const idx = indexCompleted(completed);
const plan = buildPlan(idx, installed, cli.specificMigration);
// Bug 3 — surface wedged migrations as a loud, actionable error.
if (plan.wedged.length > 0) {
for (const m of plan.wedged) {
console.error(
`\nMigration v${m.version} is WEDGED (${MAX_CONSECUTIVE_PARTIALS}+ consecutive partials with no completion). ` +
`Check ~/.gbrain/upgrade-errors.jsonl for the last failure reasons, fix the underlying issue, then run:\n` +
` gbrain apply-migrations --force-retry ${m.version}\n` +
`Then re-run \`gbrain apply-migrations --yes\`.`,
);
}
// Don't exit — applied/partial/pending are still worth reporting and running.
}
if (cli.specificMigration && plan.applied.length + plan.partial.length + plan.pending.length + plan.skippedFuture.length === 0) {
console.error(`No migration registered with version "${cli.specificMigration}". Run \`gbrain apply-migrations --list\` to see registered versions.`);
process.exit(2);
@@ -248,6 +312,11 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
// Run each orchestrator in registry order. An orchestrator failure aborts
// the rest of the chain; fixing the failure and re-running picks up where
// we left off (per-phase idempotency markers + resume from "partial").
//
// Bug 3 — the RUNNER owns the ledger write now. Orchestrators return their
// result; we persist it here with a canonical shape. If the write fails,
// surface the error and DO NOT proceed to the next migration (a silent
// ledger drop was the root cause of the original infinite-retry symptom).
let failed = false;
for (const m of toRun) {
console.log(`\n=== Applying migration v${m.version}: ${m.featurePitch.headline} ===`);
@@ -255,9 +324,45 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
const result = await m.orchestrator(orchestratorOptsFrom(cli));
if (result.status === 'failed') {
console.error(`Migration v${m.version} reported status=failed.`);
// Record the attempt as 'partial' (not 'complete') so the cap counts
// it. Don't let a failed orchestrator look like it never ran.
try {
appendCompletedMigration({
version: m.version,
status: 'partial',
phases: result.phases,
files_rewritten: result.files_rewritten,
autopilot_installed: result.autopilot_installed,
install_target: result.install_target,
apply_migrations_pending: result.pending_host_work ? result.pending_host_work > 0 : undefined,
});
} catch (e) {
console.error(`Also: could not persist failure record: ${e instanceof Error ? e.message : String(e)}`);
}
failed = true;
break;
}
// Persist the terminal outcome. appendCompletedMigration no-ops when
// the last entry for this version is already 'complete' (idempotency
// guard), so repeated clean runs don't spam the ledger.
try {
appendCompletedMigration({
version: m.version,
status: result.status, // 'complete' | 'partial'
phases: result.phases,
files_rewritten: result.files_rewritten,
autopilot_installed: result.autopilot_installed,
install_target: result.install_target,
apply_migrations_pending: result.pending_host_work ? result.pending_host_work > 0 : undefined,
});
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Failed to persist ledger entry for v${m.version}: ${msg}. Stopping to prevent silent drift.`);
failed = true;
break;
}
if (result.status === 'partial') {
console.log(`Migration v${m.version} finished as PARTIAL. Re-run \`gbrain apply-migrations --yes\` after resolving any pending host-work items.`);
} else {
@@ -266,6 +371,10 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Migration v${m.version} threw: ${msg}`);
// Same partial-on-throw treatment so the cap counts runaway failures.
try {
appendCompletedMigration({ version: m.version, status: 'partial' });
} catch { /* swallow ledger-write failure on throw path */ }
failed = true;
break;
}

75
src/commands/doctor.ts
View File

@@ -4,6 +4,7 @@ import { LATEST_VERSION } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
import { loadCompletedMigrations } from '../core/preferences.ts';
import type { DbUrlSource } from '../core/config.ts';
import { join } from 'path';
import { existsSync, readFileSync, readdirSync } from 'fs';
@@ -18,8 +19,13 @@ export interface Check {
* Run doctor with filesystem-first, DB-second architecture.
* Filesystem checks (resolver, conformance) run without engine.
* DB checks run only if engine is provided.
*
* `dbSource` is passed only from the `--fast` and DB-unavailable paths in
* cli.ts so we can emit a precise "why no DB check" message. When null, the
* user has no DB configured anywhere; otherwise the caller chose --fast or
* we failed to connect despite a configured URL.
*/
export async function runDoctor(engine: BrainEngine | null, args: string[]) {
export async function runDoctor(engine: BrainEngine | null, args: string[], dbSource?: DbUrlSource) {
const jsonOutput = args.includes('--json');
const fastMode = args.includes('--fast');
const doFix = args.includes('--fix');
@@ -136,11 +142,54 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
// Read/parse failure is itself best-effort; skip silently.
}
// 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit`
// bookmark when per-file parse errors happen, and appends each failure
// to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error.
// Without this doctor check, users see "sync blocked" and have no
// surface showing which files to fix.
try {
const { unacknowledgedSyncFailures, loadSyncFailures } = await import('../core/sync.ts');
const unacked = unacknowledgedSyncFailures();
const all = loadSyncFailures();
if (unacked.length > 0) {
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
checks.push({
name: 'sync_failures',
status: 'warn',
message:
`${unacked.length} unacknowledged sync failure(s). ${preview}` +
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
});
} else if (all.length > 0) {
// Acknowledged-only: informational, not a warning.
checks.push({
name: 'sync_failures',
status: 'ok',
message: `${all.length} historical sync failure(s), all acknowledged.`,
});
}
} catch {
// Best-effort. A broken JSONL should not stop doctor.
}
// --- DB checks (skip if --fast or no engine) ---
if (fastMode || !engine) {
if (!engine) {
checks.push({ name: 'connection', status: 'warn', message: 'No database configured (filesystem checks only)' });
// Pick the precise message. When dbSource is provided, we know
// whether a URL exists (env or config-file) — the caller simply
// skipped the connection. When null, there really is no config
// anywhere.
let msg: string;
if (fastMode && dbSource) {
msg = `Skipping DB checks (--fast mode, URL present from ${dbSource})`;
} else if (!fastMode && dbSource) {
msg = `Could not connect to configured DB (URL from ${dbSource}); filesystem checks only`;
} else {
msg = 'No database configured (filesystem checks only). Set GBRAIN_DATABASE_URL or run `gbrain init`.';
}
checks.push({ name: 'connection', status: 'warn', message: msg });
}
const earlyFail1 = outputResults(checks, jsonOutput);
process.exit(earlyFail1 ? 1 : 0);
@@ -229,7 +278,6 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
}
// 8. Graph health (link + timeline coverage on entity pages).
// dead_links removed in v0.10.1: ON DELETE CASCADE on link FKs makes it always 0.
try {
const health = await engine.getHealth();
const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0);
@@ -243,6 +291,27 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%. Run: gbrain link-extract && gbrain timeline-extract`,
});
}
// Bug 11 — brain_score breakdown. When the total is < 100, show which
// components contributed the deficit so users know what to fix.
// Uses distinct *_score field names (not overloading link_coverage /
// timeline_coverage, which are entity-scoped).
if (health.brain_score < 100) {
const parts = [
`embed ${health.embed_coverage_score}/35`,
`links ${health.link_density_score}/25`,
`timeline ${health.timeline_coverage_score}/15`,
`orphans ${health.no_orphans_score}/15`,
`dead-links ${health.no_dead_links_score}/10`,
];
checks.push({
name: 'brain_score',
status: health.brain_score >= 70 ? 'ok' : 'warn',
message: `Brain score ${health.brain_score}/100 (${parts.join(', ')})`,
});
} else {
checks.push({ name: 'brain_score', status: 'ok', message: `Brain score 100/100` });
}
} catch {
checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
}

58
src/commands/import.ts
View File

@@ -17,7 +17,16 @@ function defaultWorkers(): number {
return Math.min(byPool, byCpu, byMem);
}
export async function runImport(engine: BrainEngine, args: string[]) {
/** Bug 9 — surface per-file failures so callers (performFullSync) can gate state advances. */
export interface RunImportResult {
imported: number;
skipped: number;
errors: number;
chunksCreated: number;
failures: Array<{ path: string; error: string }>;
}
export async function runImport(engine: BrainEngine, args: string[], opts: { commit?: string } = {}): Promise<RunImportResult> {
const noEmbed = args.includes('--no-embed');
const fresh = args.includes('--fresh');
const jsonOutput = args.includes('--json');
@@ -69,6 +78,7 @@ export async function runImport(engine: BrainEngine, args: string[]) {
let chunksCreated = 0;
const importedSlugs: string[] = [];
const errorCounts: Record<string, number> = {};
const failures: Array<{ path: string; error: string }> = []; // Bug 9
const startTime = Date.now();
function logProgress() {
@@ -91,6 +101,8 @@ export async function runImport(engine: BrainEngine, args: string[]) {
skipped++;
if (result.error && result.error !== 'unchanged') {
console.error(` Skipped ${relativePath}: ${result.error}`);
// Bug 9 — non-"unchanged" skips carry a real error reason.
failures.push({ path: relativePath, error: result.error });
}
}
} catch (e: unknown) {
@@ -104,6 +116,7 @@ export async function runImport(engine: BrainEngine, args: string[]) {
}
errors++;
skipped++;
failures.push({ path: relativePath, error: msg });
}
processed++;
if (processed % 100 === 0 || processed === files.length) {
@@ -135,10 +148,15 @@ export async function runImport(engine: BrainEngine, args: string[]) {
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerEngines = await Promise.all(
Array.from({ length: actualWorkers }, async () => {
const eng = new PostgresEngine();
await eng.connect({ database_url: config!.database_url!, poolSize: 2 });
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
return eng;
})
);
@@ -198,17 +216,41 @@ export async function runImport(engine: BrainEngine, args: string[]) {
summary: `Imported ${imported} pages, ${skipped} skipped, ${chunksCreated} chunks`,
});
// Import → sync continuity: write sync checkpoint if this is a git repo
// Import → sync continuity: write sync checkpoint if this is a git repo.
// Bug 9 — gate last_commit on "no failures" so import doesn't silently
// stomp on the sync bookmark when parsing broke. We still write
// last_run + repo_path either way (those are progress indicators).
let gitHead: string | null = null;
try {
if (existsSync(join(dir, '.git'))) {
const head = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf-8' }).trim();
await engine.setConfig('sync.last_commit', head);
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', dir);
gitHead = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf-8' }).trim();
}
} catch {
// Not a git repo or git not available, skip checkpoint
// Not a git repo or git not available
}
if (gitHead) {
// Record failures into the central JSONL so doctor can surface them.
// Use gitHead as the commit so a later sync can tell "same broken
// state as last time" from "new broken state."
if (failures.length > 0) {
const { recordSyncFailures } = await import('../core/sync.ts');
recordSyncFailures(failures, gitHead);
}
if (failures.length === 0) {
await engine.setConfig('sync.last_commit', gitHead);
} else {
console.error(
`\nImport completed with ${failures.length} failure(s). ` +
`sync.last_commit NOT advanced — re-run 'gbrain sync' to retry, or ` +
`'gbrain sync --skip-failed' to acknowledge and move past them.`,
);
}
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', dir);
}
return { imported, skipped, errors, chunksCreated, failures };
}
export function collectMarkdownFiles(dir: string): string[] {

156
src/commands/init.ts
View File

@@ -107,36 +107,39 @@ async function initPGLite(opts: { jsonOutput: boolean; apiKey: string | null; cu
console.log(`Setting up local brain with PGLite (no server needed)...`);
const engine = await createEngine({ engine: 'pglite' });
await engine.connect({ database_path: dbPath, engine: 'pglite' });
await engine.initSchema();
try {
await engine.connect({ database_path: dbPath, engine: 'pglite' });
await engine.initSchema();
const config: GBrainConfig = {
engine: 'pglite',
database_path: dbPath,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
const config: GBrainConfig = {
engine: 'pglite',
database_path: dbPath,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
const stats = await engine.getStats();
await engine.disconnect();
const stats = await engine.getStats();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
} else {
console.log(`\nBrain ready at ${dbPath}`);
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
} else {
console.log('Next: gbrain import <dir>');
console.log(`\nBrain ready at ${dbPath}`);
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
} else {
console.log('Next: gbrain import <dir>');
}
console.log('');
console.log('When you outgrow local: gbrain migrate --to supabase');
reportModStatus();
}
console.log('');
console.log('When you outgrow local: gbrain migrate --to supabase');
reportModStatus();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}
@@ -157,64 +160,67 @@ async function initPostgres(opts: { databaseUrl: string; jsonOutput: boolean; ap
console.log('Connecting to database...');
const engine = await createEngine({ engine: 'postgres' });
try {
await engine.connect({ database_url: databaseUrl });
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Session pooler connection string instead (port 6543).');
}
throw e;
}
// Check and auto-create pgvector extension
try {
const conn = (engine as any).sql || (await import('../core/db.ts')).getConnection();
const ext = await conn`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
if (ext.length === 0) {
console.log('pgvector extension not found. Attempting to create...');
try {
await conn`CREATE EXTENSION IF NOT EXISTS vector`;
console.log('pgvector extension created successfully.');
} catch {
console.error('Could not auto-create pgvector extension. Run manually in SQL Editor:');
console.error(' CREATE EXTENSION vector;');
await engine.disconnect();
process.exit(1);
try {
await engine.connect({ database_url: databaseUrl });
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Session pooler connection string instead (port 6543).');
}
throw e;
}
} catch {
// Non-fatal
}
console.log('Running schema migration...');
await engine.initSchema();
// Check and auto-create pgvector extension
try {
const conn = (engine as any).sql || (await import('../core/db.ts')).getConnection();
const ext = await conn`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
if (ext.length === 0) {
console.log('pgvector extension not found. Attempting to create...');
try {
await conn`CREATE EXTENSION IF NOT EXISTS vector`;
console.log('pgvector extension created successfully.');
} catch {
console.error('Could not auto-create pgvector extension. Run manually in SQL Editor:');
console.error(' CREATE EXTENSION vector;');
// Throw so the outer finally runs engine.disconnect() before we die.
throw new Error('pgvector extension missing');
}
}
} catch {
// Non-fatal
}
const config: GBrainConfig = {
engine: 'postgres',
database_url: databaseUrl,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
console.log('Running schema migration...');
await engine.initSchema();
const stats = await engine.getStats();
await engine.disconnect();
const config: GBrainConfig = {
engine: 'postgres',
database_url: databaseUrl,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
} else {
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
const stats = await engine.getStats();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
} else {
console.log('Next: gbrain import <dir>');
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
} else {
console.log('Next: gbrain import <dir>');
}
reportModStatus();
}
reportModStatus();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}

16
src/commands/jobs.ts
View File

@@ -511,14 +511,30 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const steps: Record<string, unknown> = {};
const failed: string[] = [];
// Bug 8 — Between phases, yield to the event loop. The worker's lock
// renewal runs on a timer (src/core/minions/worker.ts); without a
// periodic yield, long CPU-bound phases starve the renewal callback
// and the job gets killed by the stalled-sweeper. A single
// `await new Promise(r => setImmediate(r))` gives the timer a chance
// to fire. The per-phase body is async+await already, so each phase
// internally yields on its own I/O boundaries — this is a belt for
// the gap between phases.
//
// Follow-up (deferred to v0.15): thread ctx.signal / ctx.shutdownSignal
// through each core fn so mid-phase cancellation works on huge brains.
const yieldToLoop = () => new Promise<void>(r => setImmediate(r));
try { steps.sync = await performSync(engine, { repoPath, noEmbed: true }); }
catch (e) { steps.sync = { error: e instanceof Error ? e.message : String(e) }; failed.push('sync'); }
await yieldToLoop();
try { steps.extract = await runExtractCore(engine, { mode: 'all', dir: repoPath }); }
catch (e) { steps.extract = { error: e instanceof Error ? e.message : String(e) }; failed.push('extract'); }
await yieldToLoop();
try { await runEmbedCore(engine, { stale: true }); steps.embed = { embedded: true }; }
catch (e) { steps.embed = { error: e instanceof Error ? e.message : String(e) }; failed.push('embed'); }
await yieldToLoop();
try { steps.backlinks = await runBacklinksCore({ action: 'fix', dir: repoPath }); }
catch (e) { steps.backlinks = { error: e instanceof Error ? e.message : String(e) }; failed.push('backlinks'); }

2
src/commands/migrations/index.ts
View File

@@ -16,6 +16,7 @@ import { v0_12_0 } from './v0_12_0.ts';
import { v0_12_2 } from './v0_12_2.ts';
import { v0_13_0 } from './v0_13_0.ts';
import { v0_13_1 } from './v0_13_1.ts';
import { v0_14_0 } from './v0_14_0.ts';
export const migrations: Migration[] = [
v0_11_0,
@@ -23,6 +24,7 @@ export const migrations: Migration[] = [
v0_12_2,
v0_13_0,
v0_13_1,
v0_14_0,
];
/** Look up a migration by exact version string. */

22
src/commands/migrations/v0_11_0.ts
View File

@@ -24,7 +24,8 @@ import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lst
import { join, resolve, dirname } from 'path';
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { savePreferences, loadPreferences, appendCompletedMigration } from '../../core/preferences.ts';
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
import { promptLine } from '../../core/cli-util.ts';
import { VERSION } from '../../version.ts';
@@ -441,22 +442,11 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const f = phaseFInstall(opts);
phases.push(f);
// Phase G: record in completed.jsonl. Status depends on whether any
// host work remains pending AND whether the install phase succeeded.
// Bug 3 — Phase G (record in completed.jsonl) moved to the runner. The
// runner in apply-migrations.ts persists the result after orchestrator
// returns, so we just decide the status here.
const status: 'complete' | 'partial' = (pending_host_work > 0) ? 'partial' : 'complete';
if (!opts.dryRun) {
appendCompletedMigration({
version: '0.11.0',
status,
mode,
files_rewritten,
autopilot_installed: f.status === 'complete',
install_target: undefined, // install target is decided inside autopilot --install
...(status === 'partial' ? { apply_migrations_pending: true } : {}),
});
}
phases.push({ name: 'record', status: opts.dryRun ? 'skipped' : 'complete', detail: `status=${status}` });
phases.push({ name: 'record', status: opts.dryRun ? 'skipped' : 'complete', detail: `status=${status} (ledger write in runner)` });
// Post-run: print pending-host-work summary if anything needs host action.
if (pending_host_work > 0) {

10
src/commands/migrations/v0_12_0.ts
View File

@@ -32,7 +32,7 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
@@ -225,13 +225,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.12.0', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.12.0',
status,

10
src/commands/migrations/v0_12_2.ts
View File

@@ -22,7 +22,7 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
@@ -104,13 +104,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.12.2', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.12.2',
status,

11
src/commands/migrations/v0_13_0.ts
View File

@@ -27,7 +27,8 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts). The
// orchestrator returns its result and the runner persists it.
// ── Phase A — Schema ────────────────────────────────────────
//
@@ -136,13 +137,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.13.0', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.13.0',
status,

21
src/commands/migrations/v0_13_1.ts
View File

@@ -42,7 +42,7 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
const ROLLBACK_DIR = join(homedir(), '.gbrain', 'migrations');
const ROLLBACK_FILE = join(ROLLBACK_DIR, 'v0_13_1-rollback.jsonl');
@@ -233,24 +233,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
if (!opts.dryRun && status === 'complete') {
try {
appendCompletedMigration({
version: '0.13.1',
completed_at: new Date().toISOString(),
status: 'complete',
phases: phases.map(p => ({ name: p.name, status: p.status })),
files_rewritten: filesRewritten,
});
} catch (e) {
// Recording failure is non-fatal; migration still ran.
phases.push({
name: 'record',
status: 'failed',
detail: e instanceof Error ? e.message : String(e),
});
}
}
// Bug 3 — ledger write lives in the runner now.
return {
version: '0.13.1',

180
src/commands/migrations/v0_14_0.ts Normal file
View File

@@ -0,0 +1,180 @@
/**
* v0.14.0 migration — shell-jobs adoption + autopilot cooperative fix.
*
* Ships two phases:
*
* A. Schema: `ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 3`.
* New installs already get the bumped default from schema-embedded.ts +
* pglite-schema.ts. This ALTER is for existing brains where the table
* was created under v0.13.x (default 1). Idempotent — running twice is
* a no-op because the default is a table-level attribute, not per-row.
* Existing rows keep their stored max_stalled value; only rows created
* after the ALTER pick up the new default.
*
* B. Pending-host-work ping: emit one entry to
* ~/.gbrain/migrations/pending-host-work.jsonl so the host agent knows
* to read skills/migrations/v0.14.0.md (shell-jobs adoption, autopilot
* cooperative handler wiring, GBRAIN_POOL_SIZE doc). Idempotent — the
* write checks for an existing entry before appending.
*
* Ledger writes live in the runner (Bug 3). This orchestrator returns its
* result; apply-migrations.ts persists.
*/
import { existsSync, readFileSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Resolve HOME at CALL time, not module-load time — Bun caches os.homedir()
// and ignores later HOME mutations, which breaks test isolation and scripted
// installs. Match the preferences.ts pattern.
function resolveHome(): string { return process.env.HOME || homedir(); }
function pendingHostWorkDir(): string { return join(resolveHome(), '.gbrain', 'migrations'); }
function pendingHostWorkPath(): string { return join(pendingHostWorkDir(), 'pending-host-work.jsonl'); }
// ---------------------------------------------------------------------------
// Phase A — schema: bump minion_jobs.max_stalled default 1 → 3
// ---------------------------------------------------------------------------
async function phaseASchema(opts: OrchestratorOpts): Promise<{ result: OrchestratorPhaseResult; engine: BrainEngine | null }> {
if (opts.dryRun) {
return { result: { name: 'schema', status: 'skipped', detail: 'dry-run' }, engine: null };
}
try {
const config = loadConfig();
if (!config) {
return {
result: { name: 'schema', status: 'skipped', detail: 'no brain configured (run gbrain init first)' },
engine: null,
};
}
const engine = await createEngine(toEngineConfig(config));
await engine.connect(toEngineConfig(config));
try {
// Both Postgres and PGLite accept this ALTER. Idempotent at the
// table level — setting the default to 3 twice is fine.
await engine.executeRaw('ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 3');
} catch (e) {
// If minion_jobs doesn't exist yet (brand new install), the schema
// file already has the new default, so this is moot. Skip instead of
// fail.
const msg = e instanceof Error ? e.message : String(e);
if (/does not exist|no such table|relation .* does not exist/i.test(msg)) {
return {
result: { name: 'schema', status: 'skipped', detail: 'minion_jobs not yet created (fresh install)' },
engine,
};
}
throw e;
}
return { result: { name: 'schema', status: 'complete' }, engine };
} catch (e) {
return {
result: { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) },
engine: null,
};
}
}
// ---------------------------------------------------------------------------
// Phase B — emit pending-host-work entry for the v0.14.0 skill
// ---------------------------------------------------------------------------
interface PendingHostWorkEntry {
migration: string;
ts: string;
skill: string;
reason: string;
}
function existingEntryForVersion(version: string): boolean {
const p = pendingHostWorkPath();
if (!existsSync(p)) return false;
try {
const raw = readFileSync(p, 'utf-8');
for (const line of raw.split('\n')) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
const obj = JSON.parse(trimmed) as PendingHostWorkEntry;
if (obj.migration === version) return true;
} catch { /* skip malformed */ }
}
} catch { /* read error */ }
return false;
}
function phaseBHostWork(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) {
return { name: 'host-work', status: 'skipped', detail: 'dry-run' };
}
try {
if (existingEntryForVersion('0.14.0')) {
return { name: 'host-work', status: 'skipped', detail: 'already recorded' };
}
mkdirSync(pendingHostWorkDir(), { recursive: true });
const entry: PendingHostWorkEntry = {
migration: '0.14.0',
ts: new Date().toISOString(),
skill: 'skills/migrations/v0.14.0.md',
reason: 'shell-jobs adoption + autopilot cooperative wiring',
};
appendFileSync(pendingHostWorkPath(), JSON.stringify(entry) + '\n');
return { name: 'host-work', status: 'complete', detail: pendingHostWorkPath() };
} catch (e) {
return { name: 'host-work', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
}
}
// ---------------------------------------------------------------------------
// Orchestrator
// ---------------------------------------------------------------------------
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
const phases: OrchestratorPhaseResult[] = [];
const { result: schemaRes, engine } = await phaseASchema(opts);
phases.push(schemaRes);
try {
const hostRes = phaseBHostWork(opts);
phases.push(hostRes);
} finally {
if (engine) {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
return {
version: '0.14.0',
status,
phases,
pending_host_work: phases.some(p => p.name === 'host-work' && p.status === 'complete') ? 1 : 0,
};
}
// ---------------------------------------------------------------------------
// Export
// ---------------------------------------------------------------------------
export const v0_14_0: Migration = {
version: '0.14.0',
featurePitch: {
headline: 'Shell jobs + autopilot cooperative handler + max_stalled default bump.',
description:
'v0.14.0 unlocks `shell` as a Minion job type (gated by GBRAIN_ALLOW_SHELL_JOBS=1 ' +
'on the worker). The autopilot-cycle handler now yields to the event loop ' +
'between phases so lock renewal fires on huge brains. The minion_jobs.max_stalled ' +
'default is bumped 1→3 so one lock-lost tick no longer dead-letters a job. ' +
'Host-specific skill doc: skills/migrations/v0.14.0.md.',
},
orchestrator,
};

109
src/commands/sync.ts
View File

@@ -3,11 +3,18 @@ import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { buildSyncManifest, isSyncable, pathToSlug } from '../core/sync.ts';
import {
buildSyncManifest,
isSyncable,
pathToSlug,
recordSyncFailures,
unacknowledgedSyncFailures,
acknowledgeSyncFailures,
} from '../core/sync.ts';
import type { SyncManifest } from '../core/sync.ts';
export interface SyncResult {
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run';
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
fromCommit: string | null;
toCommit: string;
added: number;
@@ -16,6 +23,7 @@ export interface SyncResult {
renamed: number;
chunksCreated: number;
pagesAffected: string[];
failedFiles?: number; // count of parse failures (Bug 9)
}
export interface SyncOpts {
@@ -25,6 +33,10 @@ export interface SyncOpts {
noPull?: boolean;
noEmbed?: boolean;
noExtract?: boolean;
/** Bug 9 — acknowledge + skip past current failure set (CLI --skip-failed). */
skipFailed?: boolean;
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
retryFailed?: boolean;
}
function git(repoPath: string, ...args: string[]): string {
@@ -213,6 +225,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// ep_poll whenever the diff crosses the old > 10 threshold that used to
// trigger the outer wrap. Per-file atomicity is also the right granularity:
// one file's failure should not roll back the others' successful imports.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
for (const path of [...filtered.added, ...filtered.modified]) {
const filePath = join(repoPath, path);
if (!existsSync(filePath)) continue;
@@ -221,15 +234,55 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
} else if (result.status === 'skipped' && (result as any).error) {
// importFile returned a non-throw skip with a reason
failedFiles.push({ path, error: String((result as any).error) });
}
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
console.error(` Warning: skipped ${path}: ${msg}`);
failedFiles.push({ path, error: msg });
}
}
const elapsed = Date.now() - start;
// Bug 9 — gate the sync bookmark on success. If any per-file parse
// failed, record it to ~/.gbrain/sync-failures.jsonl and DO NOT advance
// sync.last_commit. The next sync re-walks the same diff and re-attempts
// the failed files. Escape hatches: --skip-failed acknowledges the
// current set, --retry-failed re-parses before running the normal sync.
if (failedFiles.length > 0) {
recordSyncFailures(failedFiles, headCommit);
if (!opts.skipFailed) {
console.error(
`\nSync blocked: ${failedFiles.length} file(s) failed to parse. ` +
`Fix the YAML frontmatter in the files above and re-run, or use ` +
`'gbrain sync --skip-failed' to acknowledge and move on.`,
);
// Update last_run + repo_path (progress on infra) but NOT last_commit.
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', repoPath);
return {
status: 'blocked_by_failures',
fromCommit: lastCommit,
toCommit: headCommit,
added: filtered.added.length,
modified: filtered.modified.length,
deleted: filtered.deleted.length,
renamed: filtered.renamed.length,
chunksCreated,
pagesAffected,
failedFiles: failedFiles.length,
};
}
// --skip-failed: acknowledge the now-recorded set and proceed.
const acked = acknowledgeSyncFailures();
if (acked > 0) {
console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
}
}
// Update sync state AFTER all changes succeed
await engine.setConfig('sync.last_commit', headCommit);
await engine.setConfig('sync.last_run', new Date().toISOString());
@@ -288,7 +341,34 @@ async function performFullSync(
const { runImport } = await import('./import.ts');
const importArgs = [repoPath];
if (opts.noEmbed) importArgs.push('--no-embed');
await runImport(engine, importArgs);
const result = await runImport(engine, importArgs, { commit: headCommit });
// Bug 9 — gate the full-sync bookmark on success. runImport already
// writes its own sync.last_commit conditionally (import.ts), but
// performFullSync is called on first-sync + force-full paths where
// the sync module owns the last_commit write. Respect the same gate.
if (result.failures.length > 0) {
recordSyncFailures(result.failures, headCommit);
if (!opts.skipFailed) {
console.error(
`\nFull sync blocked: ${result.failures.length} file(s) failed. ` +
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
);
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', repoPath);
return {
status: 'blocked_by_failures',
fromCommit: null,
toCommit: headCommit,
added: 0, modified: 0, deleted: 0, renamed: 0,
chunksCreated: result.chunksCreated,
pagesAffected: [],
failedFiles: result.failures.length,
};
}
const acked = acknowledgeSyncFailures();
if (acked > 0) console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
}
// Persist sync state so next sync is incremental (C1 fix: was missing)
await engine.setConfig('sync.last_commit', headCommit);
@@ -322,8 +402,24 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const full = args.includes('--full');
const noPull = args.includes('--no-pull');
const noEmbed = args.includes('--no-embed');
const skipFailed = args.includes('--skip-failed');
const retryFailed = args.includes('--retry-failed');
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed };
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed };
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
// happens inside the regular incremental/full loop because once the commit
// pointer is behind the failures, the diff naturally revisits them.
if (retryFailed) {
const failures = unacknowledgedSyncFailures();
if (failures.length === 0) {
console.log('No unacknowledged sync failures to retry.');
} else {
console.log(`Retrying ${failures.length} previously-failed file(s)...`);
// Don't acknowledge them yet — they must succeed to clear.
}
}
if (!watch) {
const result = await performSync(engine, opts);
@@ -371,5 +467,10 @@ function printSyncResult(result: SyncResult) {
break;
case 'dry_run':
break; // already printed in performSync
case 'blocked_by_failures':
console.log(`Sync BLOCKED at ${result.toCommit.slice(0, 8)}: ${result.failedFiles ?? 0} file(s) failed to parse.`);
console.log(` See ~/.gbrain/sync-failures.jsonl for details, or run 'gbrain doctor'.`);
console.log(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`);
break;
}
}

38
src/core/config.ts
View File

@@ -1,8 +1,24 @@
import { readFileSync, writeFileSync, mkdirSync, chmodSync } from 'fs';
import { readFileSync, writeFileSync, mkdirSync, chmodSync, existsSync } from 'fs';
import { join } from 'path';
import { homedir } from 'os';
import type { EngineConfig } from './types.ts';
/**
* Where is the active DB URL coming from? Pure introspection, no connection
* attempt. Used by `gbrain doctor --fast` so the user gets a precise message
* instead of the misleading "No database configured" when GBRAIN_DATABASE_URL
* (or DATABASE_URL) is actually set.
*
* Precedence matches loadConfig(): env vars win over config-file URL. Returns
* null only when NO source provides a URL at all.
*/
export type DbUrlSource =
| 'env:GBRAIN_DATABASE_URL'
| 'env:DATABASE_URL'
| 'config-file'
| 'config-file-path' // PGLite: config file present, no URL but database_path set
| null;
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in serverless/bundled environments)
function getConfigDir() { return join(homedir(), '.gbrain'); }
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
@@ -70,3 +86,23 @@ export function configDir(): string {
export function configPath(): string {
return join(configDir(), 'config.json');
}
/**
* Introspect where the active DB URL would come from if we tried to connect.
* Never throws, never connects. Env vars take precedence (matches loadConfig).
*/
export function getDbUrlSource(): DbUrlSource {
if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL';
if (process.env.DATABASE_URL) return 'env:DATABASE_URL';
if (!existsSync(configPath())) return null;
try {
const raw = readFileSync(configPath(), 'utf-8');
const parsed = JSON.parse(raw) as Partial<GBrainConfig>;
if (parsed.database_url) return 'config-file';
if (parsed.database_path) return 'config-file-path';
return null;
} catch {
// Config file exists but is unreadable/malformed — treat as null source.
return null;
}
}

20
src/core/db.ts
View File

@@ -5,6 +5,24 @@ import { SCHEMA_SQL } from './schema-embedded.ts';
let sql: ReturnType<typeof postgres> | null = null;
let connectedUrl: string | null = null;
/**
* Default pool size for Postgres connections. Users on the Supabase transaction
* pooler (port 6543) or any multi-tenant pooler can lower this to avoid
* MaxClients errors when `gbrain upgrade` spawns subprocesses that each open
* their own pool. Set `GBRAIN_POOL_SIZE=2` (or similar) before the command.
*/
const DEFAULT_POOL_SIZE_FALLBACK = 10;
export function resolvePoolSize(explicit?: number): number {
if (typeof explicit === 'number' && explicit > 0) return explicit;
const raw = process.env.GBRAIN_POOL_SIZE;
if (raw) {
const parsed = parseInt(raw, 10);
if (Number.isFinite(parsed) && parsed > 0) return parsed;
}
return DEFAULT_POOL_SIZE_FALLBACK;
}
export function getConnection(): ReturnType<typeof postgres> {
if (!sql) {
throw new GBrainError(
@@ -36,7 +54,7 @@ export async function connect(config: EngineConfig): Promise<void> {
try {
sql = postgres(url, {
max: 10,
max: resolvePoolSize(),
idle_timeout: 20,
connect_timeout: 10,
types: {

27
src/core/pglite-engine.ts
View File

@@ -481,7 +481,12 @@ export class PGLiteEngine implements BrainEngine {
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
-- edges that originate from different provenance (markdown body
-- vs frontmatter vs auto-extracted). Presentation-only dedup;
-- the links table still preserves every provenance row. See
-- plan Bug 6/10.
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
FROM links l2
JOIN pages p3 ON p3.id = l2.to_page_id
WHERE l2.from_page_id = g.id),
@@ -850,6 +855,8 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 — orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
@@ -890,10 +897,14 @@ export class PGLiteEngine implements BrainEngine {
const timelineCoverageDensity = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
const brainScore = pageCount === 0 ? 0 : Math.round(
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverageDensity * 0.15 +
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
);
// Bug 11 — per-component points. Sum equals brainScore by construction
// so `doctor` can render a breakdown that adds up to the total.
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageDensity * 15);
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
return {
page_count: pageCount,
@@ -902,12 +913,18 @@ export class PGLiteEngine implements BrainEngine {
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,
dead_links: deadLinks,
link_coverage: Number(r.link_coverage),
timeline_coverage: Number(r.timeline_coverage),
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
slug: c.slug,
link_count: Number(c.link_count),
})),
embed_coverage_score: embedCoverageScore,
link_density_score: linkDensityScore,
timeline_coverage_score: timelineCoverageScore,
no_orphans_score: noOrphansScore,
no_dead_links_score: noDeadLinksScore,
};
}

2
src/core/pglite-schema.ts
View File

@@ -185,7 +185,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,

44
src/core/postgres-engine.ts
View File

@@ -31,11 +31,14 @@ export class PostgresEngine implements BrainEngine {
// Lifecycle
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
if (config.poolSize) {
// Instance-level connection for worker isolation
// Instance-level connection for worker isolation. resolvePoolSize lets
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
// env var is a user escape hatch, so it wins.
const url = config.database_url;
if (!url) throw new GBrainError('No database URL', 'database_url is missing', 'Provide --url');
const size = Math.min(config.poolSize, db.resolvePoolSize(config.poolSize));
this._sql = postgres(url, {
max: config.poolSize,
max: size,
idle_timeout: 20,
connect_timeout: 10,
types: { bigint: postgres.BigInt },
@@ -540,7 +543,14 @@ export class PostgresEngine implements BrainEngine {
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
-- edges that originate from different provenance (markdown body
-- vs frontmatter vs auto-extracted). The underlying links table
-- preserves every row with its origin_page_id / link_source —
-- the dedup is presentation-only for the legacy traverseGraph
-- aggregation. traversePaths has its own in-memory dedup at a
-- different layer. See plan Bug 6/10.
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
FROM links l2
JOIN pages p3 ON p3.id = l2.to_page_id
WHERE l2.from_page_id = g.id),
@@ -893,9 +903,12 @@ export class PostgresEngine implements BrainEngine {
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// dead_links omitted (always 0 under ON DELETE CASCADE on link FKs).
// orphan_pages now matches PGLite definition: no inbound links (regardless of outbound).
// stale_pages aligned to PGLite definition (page updated_at < latest timeline entry).
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
@@ -943,13 +956,16 @@ export class PostgresEngine implements BrainEngine {
// brain_score: 0-100 weighted average
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
const timelineCoverage = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
const brainScore = pageCount === 0 ? 0 : Math.round(
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverage * 0.15 +
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
);
// Per-component points. Sum equals brainScore by construction.
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageWhole * 15);
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
return {
page_count: pageCount,
@@ -958,12 +974,18 @@ export class PostgresEngine implements BrainEngine {
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
dead_links: deadLinks,
link_coverage: Number(h.link_coverage),
timeline_coverage: Number(h.timeline_coverage),
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
slug: c.slug,
link_count: Number(c.link_count),
})),
embed_coverage_score: embedCoverageScore,
link_density_score: linkDensityScore,
timeline_coverage_score: timelineCoverageScore,
no_orphans_score: noOrphansScore,
no_dead_links_score: noDeadLinksScore,
};
}

29
src/core/preferences.ts
View File

@@ -33,12 +33,23 @@ export interface Preferences {
export interface CompletedMigrationEntry {
version: string;
ts?: string;
status: 'complete' | 'partial';
/**
* - `complete` — orchestrator finished cleanly. Terminal state; future
* runs no-op this version unless `retry` is appended.
* - `partial` — orchestrator ran but reported missed phases; re-run is
* expected. Attempt cap (3 consecutive partials without a `complete`
* or `retry` between them) triggers the "wedged" skip in the runner.
* - `retry` — explicit reset marker written by `--force-retry`.
* Clears a wedge without faking success; the next upgrade treats the
* version as fresh again.
*/
status: 'complete' | 'partial' | 'retry';
mode?: MinionMode;
files_rewritten?: number;
autopilot_installed?: boolean;
install_target?: string;
apply_migrations_pending?: boolean;
phases?: Array<{ name: string; status: string; detail?: string }>;
[key: string]: unknown;
}
@@ -103,8 +114,20 @@ export function savePreferences(prefs: Preferences): void {
*/
export function appendCompletedMigration(entry: CompletedMigrationEntry): void {
if (!entry.version) throw new Error('appendCompletedMigration: version required');
if (entry.status !== 'complete' && entry.status !== 'partial') {
throw new Error(`appendCompletedMigration: status must be 'complete' or 'partial', got "${entry.status}"`);
if (entry.status !== 'complete' && entry.status !== 'partial' && entry.status !== 'retry') {
throw new Error(`appendCompletedMigration: status must be 'complete', 'partial', or 'retry', got "${entry.status}"`);
}
// Bug 3 — idempotency guard. If the most recent existing entry for this
// version is already 'complete' and we're about to write another
// 'complete', skip. This protects against accidental double-writes
// during the Bug 3 runner-owned-ledger transition (old orchestrator
// code paths and new runner path shouldn't both write).
if (entry.status === 'complete') {
const existing = loadCompletedMigrations();
const prior = existing.filter(e => e.version === entry.version);
if (prior.length > 0 && prior[prior.length - 1].status === 'complete') {
return; // no-op — already terminal
}
}
const full: CompletedMigrationEntry = {
ts: new Date().toISOString(),

2
src/core/schema-embedded.ts
View File

@@ -280,7 +280,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,

124
src/core/sync.ts
View File

@@ -133,3 +133,127 @@ export function pathToSlug(filePath: string, repoPrefix?: string): string {
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
return slug.toLowerCase();
}
// ─────────────────────────────────────────────────────────────────
// Sync failure tracking — Bug 9
// ─────────────────────────────────────────────────────────────────
//
// When a sync run catches a per-file parse error (YAML with unquoted
// colons, malformed frontmatter, etc.), we record it here instead of just
// logging and moving on. Three goals:
// 1. Gate the sync.last_commit bookmark advance in all three sync paths
// (incremental, full/runImport, `gbrain import` git continuity).
// 2. Give users a visible record of what failed, with the commit hash
// they can use to re-attempt after fixing the source file.
// 3. Let `gbrain sync --skip-failed` acknowledge a known-bad set so
// repos with many broken files aren't permanently stuck.
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
import { join as _joinPath } from 'path';
import { homedir as _homedir } from 'os';
import { createHash as _createHash } from 'crypto';
export interface SyncFailure {
path: string;
error: string;
commit: string;
line?: number;
ts: string;
acknowledged?: boolean;
acknowledged_at?: string;
}
function _failuresDir(): string {
return _joinPath(_homedir(), '.gbrain');
}
export function syncFailuresPath(): string {
return _joinPath(_failuresDir(), 'sync-failures.jsonl');
}
function _hashError(msg: string): string {
return _createHash('sha256').update(msg).digest('hex').slice(0, 12);
}
function _dedupKey(f: { path: string; commit: string; error: string }): string {
return `${f.path}|${f.commit}|${_hashError(f.error)}`;
}
/**
* Read the failures JSONL, skipping malformed lines with a warning to stderr.
* Returns empty array if the file doesn't exist.
*/
export function loadSyncFailures(): SyncFailure[] {
const path = syncFailuresPath();
if (!_existsSync(path)) return [];
const raw = _readFileSync(path, 'utf-8');
const out: SyncFailure[] = [];
for (const line of raw.split('\n')) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
out.push(JSON.parse(trimmed) as SyncFailure);
} catch {
console.warn(`[sync-failures] skipping malformed line: ${trimmed.slice(0, 120)}`);
}
}
return out;
}
/**
* Append failure entries to the JSONL. Dedups by (path, commit, error-hash) —
* the same file failing with the same error on the same commit writes ONCE
* to the log, not once per sync run.
*/
export function recordSyncFailures(
failures: Array<{ path: string; error: string; line?: number }>,
commit: string,
): void {
if (failures.length === 0) return;
const existing = loadSyncFailures();
const seen = new Set(existing.map(f => _dedupKey(f)));
_mkdirSync(_failuresDir(), { recursive: true });
const now = new Date().toISOString();
for (const f of failures) {
const entry: SyncFailure = {
path: f.path,
error: f.error,
commit,
line: f.line,
ts: now,
};
if (seen.has(_dedupKey(entry))) continue;
_appendFileSync(syncFailuresPath(), JSON.stringify(entry) + '\n');
seen.add(_dedupKey(entry));
}
}
/**
* Mark all unacknowledged failures as acknowledged. Used by
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
*
* We do not delete — acknowledged entries stay as historical record so
* doctor can still show them under a "previously skipped" bucket.
*/
export function acknowledgeSyncFailures(): number {
const entries = loadSyncFailures();
if (entries.length === 0) return 0;
const now = new Date().toISOString();
let changed = 0;
const updated = entries.map(e => {
if (e.acknowledged) return e;
changed++;
return { ...e, acknowledged: true, acknowledged_at: now };
});
if (changed === 0) return 0;
_mkdirSync(_failuresDir(), { recursive: true });
const fd = require('fs').writeFileSync;
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
return changed;
}
/** Return only unacknowledged failures. */
export function unacknowledgedSyncFailures(): SyncFailure[] {
return loadSyncFailures().filter(f => !f.acknowledged);
}

33
src/core/types.ts
View File

@@ -181,17 +181,46 @@ export interface BrainHealth {
page_count: number;
embed_coverage: number;
stale_pages: number;
/** Pages with zero inbound links. Definition aligned across PGLite and Postgres. */
/**
* Islanded pages — zero inbound AND zero outbound links. A hub page
* that has references out but no back-references is NOT an orphan under
* this definition (it's working as intended as an index). The metric
* aims at "pages I forgot to connect to anything", not the stricter
* graph-theory "no inbound" definition. Both engines share this
* semantics after Bug 11 doc-drift fix.
*/
orphan_pages: number;
missing_embeddings: number;
/** Composite quality score (0-10). Computed from coverage, staleness, orphans. */
/**
* Composite quality score, 0-100. Weighted sum of five components: embed
* coverage, link density, timeline coverage, orphan avoidance, dead-link
* avoidance. See the per-component *_score fields below for breakdown.
*/
brain_score: number;
/**
* Number of links whose to_page_id no longer resolves to a page. Under
* `ON DELETE CASCADE` this is always 0, but malformed data or direct SQL
* DELETEs can produce dangling references.
*/
dead_links: number;
/** Fraction of entity pages (person/company) with >= 1 inbound link. */
link_coverage: number;
/** Fraction of entity pages (person/company) with >= 1 structured timeline entry. */
timeline_coverage: number;
/** Top 5 entities by total link count (in + out). */
most_connected: Array<{ slug: string; link_count: number }>;
/**
* Per-component contribution to brain_score. Sum equals brain_score by
* construction. Displayed by `gbrain doctor` when brain_score < 100.
* Field names are distinct from the entity-scoped link_coverage /
* timeline_coverage above to avoid semantic collision (these reflect
* whole-brain measures used in the score formula).
*/
embed_coverage_score: number; // 0-35
link_density_score: number; // 0-25
timeline_coverage_score: number; // 0-15
no_orphans_score: number; // 0-15
no_dead_links_score: number; // 0-10
}
// Ingest log

2
src/schema.sql
View File

@@ -276,7 +276,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,

13
test/apply-migrations.test.ts
View File

@@ -104,8 +104,9 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
expect(plan.pending.map(m => m.version)).toContain('0.11.0');
// Future migrations (registered but newer than installed VERSION) land in
// skippedFuture until the binary catches up. v0.13.0 = frontmatter graph
// (master), v0.13.1 = Knowledge Runtime grandfather (this branch).
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1']);
// (master), v0.13.1 = Knowledge Runtime grandfather, v0.14.0 = shell
// jobs + autopilot cooperative.
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0']);
});
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
@@ -141,10 +142,10 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
const idx = indexCompleted([]);
const plan = buildPlan(idx, '0.12.0');
expect(plan.pending.map(m => m.version)).toContain('0.11.0');
// v0.12.2, v0.13.0, and v0.13.1 were added later; installed=0.12.0 means
// they belong in skippedFuture, not pending. v0.11.0 and v0.12.0 stay
// pending despite being ≤ installed — that is the H9 invariant.
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1']);
// v0.12.2, v0.13.0, v0.13.1, and v0.14.0 were added later; installed=0.12.0
// means they belong in skippedFuture, not pending. v0.11.0 and v0.12.0
// stay pending despite being ≤ installed — that is the H9 invariant.
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0']);
});
test('--migration filter narrows to one version', () => {

140
test/brain-score-breakdown.test.ts Normal file
View File

@@ -0,0 +1,140 @@
/**
* Bug 11 — brain_score needs a breakdown + orphan_pages metric is wrong.
*
* Assertions:
* 1. getHealth() returns the new *_score breakdown fields.
* 2. Breakdown fields sum to brain_score by construction.
* 3. orphan_pages counts pages with zero INBOUND links, regardless of
* whether they have outbound links (was: required both).
* 4. BrainHealth type now carries dead_links.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
for (const t of ['links', 'content_chunks', 'timeline_entries', 'raw_data', 'tags', 'page_versions', 'ingest_log', 'pages']) {
await (engine as any).db.exec(`DELETE FROM ${t}`);
}
});
describe('Bug 11 — brain_score breakdown sums to total', () => {
test('empty brain returns zero score with all breakdown fields present', async () => {
const h = await engine.getHealth();
expect(h.brain_score).toBe(0);
expect(h.embed_coverage_score).toBe(0);
expect(h.link_density_score).toBe(0);
expect(h.timeline_coverage_score).toBe(0);
expect(h.no_orphans_score).toBe(0);
expect(h.no_dead_links_score).toBe(0);
// dead_links is now on the type.
expect(h.dead_links).toBe(0);
});
test('breakdown fields always sum to brain_score', async () => {
// Seed a small graph — some pages, some links, some embeds.
for (const slug of ['a', 'b', 'c']) {
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `content of ${slug}`, frontmatter: {} });
}
const h = await engine.getHealth();
const sum =
h.embed_coverage_score +
h.link_density_score +
h.timeline_coverage_score +
h.no_orphans_score +
h.no_dead_links_score;
expect(sum).toBe(h.brain_score);
});
test('brain_score caps at 100', async () => {
const h = await engine.getHealth();
expect(h.brain_score).toBeGreaterThanOrEqual(0);
expect(h.brain_score).toBeLessThanOrEqual(100);
});
});
describe('Bug 11 — orphan_pages is "no inbound links"', () => {
test('a page with outbound-only links is NOT an orphan', async () => {
// Hub page: links out to three others, but nothing links back to it.
// Previous (buggy) behavior: hub counted as orphan because it had no
// inbound links (correct) AND the old query also required no outbound.
await engine.putPage('hub', { type: 'note', title: 'Hub', compiled_truth: 'index', frontmatter: {} });
await engine.putPage('leaf1', { type: 'note', title: 'L1', compiled_truth: 'x', frontmatter: {} });
await engine.putPage('leaf2', { type: 'note', title: 'L2', compiled_truth: 'y', frontmatter: {} });
await engine.putPage('leaf3', { type: 'note', title: 'L3', compiled_truth: 'z', frontmatter: {} });
const hubId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='hub'`)).rows[0].id;
for (const target of ['leaf1', 'leaf2', 'leaf3']) {
const tid = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug=$1`, [target])).rows[0].id;
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
[hubId, tid],
);
}
const h = await engine.getHealth();
// hub has outbound, no inbound → NOT orphan (under the fixed definition).
// leaf1/2/3 have inbound from hub → NOT orphan.
// So orphan_pages should be 0.
expect(h.orphan_pages).toBe(0);
});
test('a page with no links at all IS an orphan', async () => {
await engine.putPage('loner', { type: 'note', title: 'Loner', compiled_truth: 'alone', frontmatter: {} });
const h = await engine.getHealth();
expect(h.orphan_pages).toBe(1);
});
test('a page with inbound links only is NOT an orphan', async () => {
await engine.putPage('sink', { type: 'note', title: 'Sink', compiled_truth: 'target', frontmatter: {} });
await engine.putPage('source', { type: 'note', title: 'Source', compiled_truth: 'origin', frontmatter: {} });
const sinkId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='sink'`)).rows[0].id;
const srcId = (await (engine as any).db.query(`SELECT id FROM pages WHERE slug='source'`)).rows[0].id;
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, 'mentions')`,
[srcId, sinkId],
);
const h = await engine.getHealth();
// sink has 1 inbound (from source) → not orphan.
// source has no inbound (but has outbound) → not orphan under new definition.
expect(h.orphan_pages).toBe(0);
});
});
describe('Bug 11 — doctor renders brain_score breakdown', () => {
test('doctor source contains brain_score breakdown rendering', async () => {
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
expect(source).toContain('brain_score');
expect(source).toContain('embed_coverage_score');
expect(source).toContain('link_density_score');
expect(source).toContain('no_orphans_score');
expect(source).toContain('no_dead_links_score');
});
});
describe('Bug 11 — BrainHealth type shape', () => {
test('type includes dead_links + breakdown scores', async () => {
const typesSource = await Bun.file(new URL('../src/core/types.ts', import.meta.url)).text();
expect(typesSource).toContain('dead_links: number');
expect(typesSource).toContain('embed_coverage_score: number');
expect(typesSource).toContain('link_density_score: number');
expect(typesSource).toContain('timeline_coverage_score: number');
expect(typesSource).toContain('no_orphans_score: number');
expect(typesSource).toContain('no_dead_links_score: number');
// The stale "(0-10)" comment must be corrected to 0-100.
expect(typesSource).toContain('0-100');
});
});

38
test/doctor.test.ts
View File

@@ -36,9 +36,41 @@ describe('doctor command', () => {
test('runDoctor accepts null engine for filesystem-only mode', async () => {
const { runDoctor } = await import('../src/commands/doctor.ts');
// runDoctor should accept null engine — it runs filesystem checks only
// We can't call it directly (it calls process.exit), but we verify the signature
expect(runDoctor.length).toBe(2); // engine, args
// runDoctor should accept null engine — it runs filesystem checks only.
// Signature is (engine, args, dbSource?) — third param is optional and
// used by --fast to distinguish "no config" from "user skipped DB check".
// Function.length counts required params only (JS ignores ?-marked).
expect(runDoctor.length).toBeGreaterThanOrEqual(2);
expect(runDoctor.length).toBeLessThanOrEqual(3);
});
// Bug 7 — --fast should differentiate "no config anywhere" from "user
// chose --fast with GBRAIN_DATABASE_URL / config-file URL present".
test('getDbUrlSource reflects GBRAIN_DATABASE_URL env var', async () => {
const { getDbUrlSource } = await import('../src/core/config.ts');
const orig = process.env.GBRAIN_DATABASE_URL;
const origAlt = process.env.DATABASE_URL;
try {
process.env.GBRAIN_DATABASE_URL = 'postgresql://test@localhost/x';
expect(getDbUrlSource()).toBe('env:GBRAIN_DATABASE_URL');
delete process.env.GBRAIN_DATABASE_URL;
process.env.DATABASE_URL = 'postgresql://test@localhost/x';
expect(getDbUrlSource()).toBe('env:DATABASE_URL');
} finally {
if (orig === undefined) delete process.env.GBRAIN_DATABASE_URL;
else process.env.GBRAIN_DATABASE_URL = orig;
if (origAlt === undefined) delete process.env.DATABASE_URL;
else process.env.DATABASE_URL = origAlt;
}
});
test('doctor --fast emits source-specific message when URL present', async () => {
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
// The source-aware message must reference the variable name so users
// know where their URL is coming from.
expect(source).toContain('Skipping DB checks (--fast mode, URL present from');
// The null-source fallback must still mention both config + env paths.
expect(source).toContain('GBRAIN_DATABASE_URL');
});
// v0.12.2 reliability wave — doctor detects JSONB double-encode + truncated

34
test/e2e/migration-flow.test.ts
View File

@@ -121,13 +121,14 @@ describeE2E('E2E: v0.11.0 orchestrator against live Postgres', () => {
expect(prefs.set_at).toBeTruthy();
expect(prefs.set_in_version).toBeTruthy();
// Phase G: completed.jsonl has one entry for v0.11.0.
// Bug 3 (v0.14.2) — orchestrator no longer writes completed.jsonl.
// The runner (apply-migrations.ts) persists the result after the
// orchestrator returns. A direct orchestrator call in E2E leaves the
// ledger empty; the runner path is tested separately in
// test/apply-migrations.test.ts + test/migration-resume.test.ts.
const completed = loadCompletedMigrations();
expect(completed.length).toBeGreaterThanOrEqual(1);
const v0110Entries = completed.filter(e => e.version === '0.11.0');
expect(v0110Entries.length).toBe(1);
expect(['complete', 'partial']).toContain(v0110Entries[0].status!);
expect(v0110Entries[0].mode).toBe('pain_triggered');
expect(v0110Entries.length).toBe(0);
// Phase F is skipped per COMMON_OPTS — autopilot should NOT have been
// installed on this host.
@@ -142,15 +143,13 @@ describeE2E('E2E: v0.11.0 orchestrator against live Postgres', () => {
const second = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(second.status);
// completed.jsonl accumulates entries per run (each run appends one).
// The runtime semantics for resume are governed by the diff rule in
// apply-migrations; here we just assert the orchestrator itself doesn't
// blow up or produce different results on a second run.
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
expect(v0110.length).toBeGreaterThanOrEqual(2);
// Preferences should be stable (same mode, unchanged content).
// Bug 3 (v0.14.2) — orchestrator does not write completed.jsonl, so
// repeated direct invocations don't accumulate ledger entries. Assert
// the preferences state stays stable (the real idempotency signal for
// this orchestrator is "running again doesn't corrupt preferences").
expect(loadPreferences().minion_mode).toBe('pain_triggered');
const completed = loadCompletedMigrations();
expect(completed.filter(e => e.version === '0.11.0').length).toBe(0);
}, 90_000);
test('host rewrite: builtin handlers auto-rewritten, non-builtins queued as JSONL TODOs', async () => {
@@ -233,15 +232,16 @@ describeE2E('E2E: v0.11.0 orchestrator against live Postgres', () => {
// Orchestrator re-running on a partial → should succeed (schema apply
// and smoke are idempotent; prefs are preserved from the partial
// record; host-rewrite runs its safe-skip pass; completed appends a
// new status:"complete" row).
// record; host-rewrite runs its safe-skip pass). Per Bug 3 (v0.14.2),
// the orchestrator itself doesn't append to completed.jsonl — the
// runner does. The stopgap's partial entry stays unchanged here.
const result = await v0_11_0.orchestrator(COMMON_OPTS);
expect(['complete', 'partial']).toContain(result.status);
const completed = loadCompletedMigrations();
const v0110 = completed.filter(e => e.version === '0.11.0');
// 1 partial (stopgap) + 1 post-orchestrator entry.
expect(v0110.length).toBe(2);
// Just the stopgap partial — orchestrator doesn't add its own entry.
expect(v0110.length).toBe(1);
expect(v0110[0].status).toBe('partial');
expect(v0110[0].source).toBe('fix-v0.11.0.sh');
}, 90_000);

46
test/migrate.test.ts
View File

@@ -201,3 +201,49 @@ describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K d
expect(helperIdx.length).toBe(0);
});
});
// ─────────────────────────────────────────────────────────────────
// resolvePoolSize — GBRAIN_POOL_SIZE env override
// ─────────────────────────────────────────────────────────────────
//
// Guards the Bug 2 fix: users on constrained poolers (Supabase port 6543)
// must be able to cap the pool size via GBRAIN_POOL_SIZE. The default
// (10) is unchanged when the env var is unset.
describe('resolvePoolSize — env var + explicit override', () => {
const { resolvePoolSize } = require('../src/core/db.ts');
const original = process.env.GBRAIN_POOL_SIZE;
afterAll(() => {
if (original === undefined) delete process.env.GBRAIN_POOL_SIZE;
else process.env.GBRAIN_POOL_SIZE = original;
});
test('returns 10 default when unset and no explicit override', () => {
delete process.env.GBRAIN_POOL_SIZE;
expect(resolvePoolSize()).toBe(10);
});
test('reads GBRAIN_POOL_SIZE as an integer', () => {
process.env.GBRAIN_POOL_SIZE = '2';
expect(resolvePoolSize()).toBe(2);
process.env.GBRAIN_POOL_SIZE = '5';
expect(resolvePoolSize()).toBe(5);
});
test('ignores invalid GBRAIN_POOL_SIZE values', () => {
process.env.GBRAIN_POOL_SIZE = 'not-a-number';
expect(resolvePoolSize()).toBe(10);
process.env.GBRAIN_POOL_SIZE = '0';
expect(resolvePoolSize()).toBe(10);
process.env.GBRAIN_POOL_SIZE = '-1';
expect(resolvePoolSize()).toBe(10);
});
test('explicit argument wins over env + default', () => {
delete process.env.GBRAIN_POOL_SIZE;
expect(resolvePoolSize(3)).toBe(3);
process.env.GBRAIN_POOL_SIZE = '7';
expect(resolvePoolSize(3)).toBe(3);
});
});

170
test/migration-resume.test.ts Normal file
View File

@@ -0,0 +1,170 @@
/**
* Bug 3 regression — migration resume semantics.
*
* Covers:
* - statusForVersion prefers 'complete' over 'partial' (never regresses).
* - Three consecutive 'partial' entries flip a migration to 'wedged'.
* - 'retry' marker resets the counter; next run treats it as fresh.
* - appendCompletedMigration no-ops on double 'complete' (idempotency).
*
* Infrastructure: point HOME at a tmpdir so the ledger writes don't
* stomp the real ~/.gbrain/migrations/completed.jsonl.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, readFileSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
let tmpHome: string;
const originalHome = process.env.HOME;
beforeEach(() => {
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-migration-resume-'));
process.env.HOME = tmpHome;
});
afterEach(() => {
if (originalHome) process.env.HOME = originalHome;
else delete process.env.HOME;
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
});
describe('Bug 3 — statusForVersion semantics', () => {
test("complete wins over partial regardless of order", async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'complete' },
{ version: '0.13.0', status: 'partial' },
] as any);
expect(__testing.statusForVersion('0.13.0', idx)).toBe('complete');
const idx2 = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'complete' },
] as any);
expect(__testing.statusForVersion('0.13.0', idx2)).toBe('complete');
});
test('two consecutive partials stay at partial', async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
] as any);
expect(__testing.statusForVersion('0.13.0', idx)).toBe('partial');
});
test('three consecutive partials flip to wedged', async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
] as any);
expect(__testing.statusForVersion('0.13.0', idx)).toBe('wedged');
});
test("retry marker resets the counter", async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'retry' },
] as any);
// After 'retry', the version is pending (fresh start).
expect(__testing.statusForVersion('0.13.0', idx)).toBe('pending');
});
test('complete after wedge is still complete (terminal)', async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'retry' },
{ version: '0.13.0', status: 'complete' },
] as any);
expect(__testing.statusForVersion('0.13.0', idx)).toBe('complete');
});
});
describe('Bug 3 — appendCompletedMigration idempotency', () => {
test('writing complete when last entry is already complete is a no-op', async () => {
const { appendCompletedMigration, loadCompletedMigrations } = await import('../src/core/preferences.ts');
appendCompletedMigration({ version: '9.9.9', status: 'complete' });
const first = loadCompletedMigrations().filter(e => e.version === '9.9.9');
expect(first.length).toBe(1);
appendCompletedMigration({ version: '9.9.9', status: 'complete' });
const second = loadCompletedMigrations().filter(e => e.version === '9.9.9');
expect(second.length).toBe(1);
});
test('partial always appends (needed for attempt-cap counter)', async () => {
const { appendCompletedMigration, loadCompletedMigrations } = await import('../src/core/preferences.ts');
appendCompletedMigration({ version: '9.9.9', status: 'partial' });
appendCompletedMigration({ version: '9.9.9', status: 'partial' });
const entries = loadCompletedMigrations().filter(e => e.version === '9.9.9');
expect(entries.length).toBe(2);
});
test("'retry' status is accepted", async () => {
const { appendCompletedMigration, loadCompletedMigrations } = await import('../src/core/preferences.ts');
appendCompletedMigration({ version: '9.9.9', status: 'retry' } as any);
const entries = loadCompletedMigrations().filter(e => e.version === '9.9.9');
expect(entries.length).toBe(1);
expect(entries[0].status).toBe('retry');
});
});
describe('Bug 3 — orchestrator no longer writes the ledger directly', () => {
test('v0_13_0 does not import appendCompletedMigration', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_13_0.ts', import.meta.url)).text();
expect(source).not.toContain('import { appendCompletedMigration }');
});
test('v0_13_1 does not import appendCompletedMigration', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_13_1.ts', import.meta.url)).text();
expect(source).not.toContain('import { appendCompletedMigration }');
});
test('v0_12_0 does not import appendCompletedMigration', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_12_0.ts', import.meta.url)).text();
expect(source).not.toContain('import { appendCompletedMigration }');
});
test('v0_12_2 does not import appendCompletedMigration', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_12_2.ts', import.meta.url)).text();
expect(source).not.toContain('import { appendCompletedMigration }');
});
test('v0_11_0 does not import appendCompletedMigration', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_11_0.ts', import.meta.url)).text();
// Import statement should not reference appendCompletedMigration; the
// old call site is replaced with a comment.
expect(source).not.toMatch(/import .*appendCompletedMigration.*from/);
});
test('apply-migrations.ts runner writes the ledger', async () => {
const source = await Bun.file(new URL('../src/commands/apply-migrations.ts', import.meta.url)).text();
expect(source).toContain("import { loadCompletedMigrations, appendCompletedMigration");
expect(source).toContain("appendCompletedMigration({");
expect(source).toContain("'retry'");
expect(source).toContain('--force-retry');
expect(source).toContain('MAX_CONSECUTIVE_PARTIALS');
});
});
describe('Bug 3 — buildPlan surfaces wedged migrations', () => {
test('wedged bucket exists in the plan', async () => {
const { __testing } = await import('../src/commands/apply-migrations.ts');
const idx = __testing.indexCompleted([
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
{ version: '0.13.0', status: 'partial' },
] as any);
const plan = __testing.buildPlan(idx, '0.13.0', '0.13.0'); // filter to just this version
expect(plan.wedged.length).toBe(1);
expect(plan.wedged[0].version).toBe('0.13.0');
expect(plan.pending.length).toBe(0);
expect(plan.partial.length).toBe(0);
});
});

103
test/migrations-v0_14_0.test.ts Normal file
View File

@@ -0,0 +1,103 @@
/**
* Bug 5 + Bug 8 — v0_14_0 orchestrator regression.
*
* The migration ships:
* - Phase A (schema): ALTER minion_jobs.max_stalled SET DEFAULT 3
* - Phase B (host-work): append skill-ping entry to
* ~/.gbrain/migrations/pending-host-work.jsonl
*
* Both phases are idempotent — re-running the migration is a no-op after
* the first successful pass.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, readFileSync, existsSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
let tmpHome: string;
const originalHome = process.env.HOME;
beforeEach(() => {
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-v0_14_0-'));
process.env.HOME = tmpHome;
});
afterEach(() => {
if (originalHome) process.env.HOME = originalHome;
else delete process.env.HOME;
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
});
describe('Bug 5 + Bug 8 — v0_14_0 module shape', () => {
test('v0_14_0 is registered in migrations/index.ts', async () => {
const { migrations } = await import('../src/commands/migrations/index.ts');
const m = migrations.find(x => x.version === '0.14.0');
expect(m).toBeDefined();
expect(m!.featurePitch.headline).toBeTruthy();
});
test('v0_14_0 does NOT write the ledger directly', async () => {
const source = await Bun.file(new URL('../src/commands/migrations/v0_14_0.ts', import.meta.url)).text();
expect(source).not.toContain('appendCompletedMigration');
});
test('orchestrator returns complete when phase A is skipped (no config)', async () => {
const { v0_14_0 } = await import('../src/commands/migrations/v0_14_0.ts');
// No loadConfig() backing → phaseASchema reports skipped (no brain).
// Phase B still emits the host-work ping.
const result = await v0_14_0.orchestrator({
yes: true,
dryRun: false,
noAutopilotInstall: true,
});
expect(['complete', 'partial']).toContain(result.status);
expect(result.version).toBe('0.14.0');
const hostWork = result.phases.find(p => p.name === 'host-work');
expect(hostWork).toBeDefined();
});
});
describe('Bug 5 — Phase B host-work entry dedup', () => {
test('first run writes the entry, second run is a skip', async () => {
const { v0_14_0 } = await import('../src/commands/migrations/v0_14_0.ts');
const first = await v0_14_0.orchestrator({ yes: true, dryRun: false, noAutopilotInstall: true });
const hostPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(hostPath)).toBe(true);
const beforeLines = readFileSync(hostPath, 'utf-8').split('\n').filter(l => l.trim()).length;
expect(beforeLines).toBe(1);
// Second run — Phase B should skip, not duplicate.
await v0_14_0.orchestrator({ yes: true, dryRun: false, noAutopilotInstall: true });
const afterLines = readFileSync(hostPath, 'utf-8').split('\n').filter(l => l.trim()).length;
expect(afterLines).toBe(1);
const entry = JSON.parse(readFileSync(hostPath, 'utf-8').split('\n')[0]);
expect(entry.migration).toBe('0.14.0');
expect(entry.skill).toBe('skills/migrations/v0.14.0.md');
});
test('dry-run writes nothing', async () => {
const { v0_14_0 } = await import('../src/commands/migrations/v0_14_0.ts');
await v0_14_0.orchestrator({ yes: true, dryRun: true, noAutopilotInstall: true });
const hostPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
expect(existsSync(hostPath)).toBe(false);
});
});
describe('Bug 8 — max_stalled default bumped in schema files', () => {
test('schema-embedded.ts has max_stalled DEFAULT 3', async () => {
const source = await Bun.file(new URL('../src/core/schema-embedded.ts', import.meta.url)).text();
expect(source).toContain('max_stalled INTEGER NOT NULL DEFAULT 3');
});
test('pglite-schema.ts has max_stalled DEFAULT 3', async () => {
const source = await Bun.file(new URL('../src/core/pglite-schema.ts', import.meta.url)).text();
expect(source).toContain('max_stalled INTEGER NOT NULL DEFAULT 3');
});
test('schema.sql has max_stalled DEFAULT 3', async () => {
const source = await Bun.file(new URL('../src/schema.sql', import.meta.url)).text();
expect(source).toContain('max_stalled INTEGER NOT NULL DEFAULT 3');
});
});

160
test/sync-failures.test.ts Normal file
View File

@@ -0,0 +1,160 @@
/**
* Bug 9 regression — sync silently drops files with broken YAML.
*
* Before the fix, sync.ts caught per-file parse errors, printed a warning,
* and still advanced sync.last_commit. The failed file was never retried
* because it was behind the bookmark. Silent data loss.
*
* After the fix:
* - failures append to ~/.gbrain/sync-failures.jsonl (with dedup)
* - incremental + full-sync + import git-continuity paths gate the
* sync.last_commit advance on "no failures"
* - `gbrain sync --skip-failed` acknowledges the current set
* - `gbrain doctor` surfaces unacknowledged failures
*
* This suite exercises the helper + the dedup behavior. The full CLI
* round-trip is covered by E2E tests.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, rmSync, readFileSync, existsSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
// Point HOME at a tmpdir so we don't stomp the real ~/.gbrain/sync-failures.jsonl
let tmpHome: string;
const originalHome = process.env.HOME;
beforeEach(async () => {
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-sync-failures-'));
process.env.HOME = tmpHome;
// Belt-and-suspenders: explicitly clear the jsonl at the resolved path.
const { syncFailuresPath } = await import('../src/core/sync.ts');
try { rmSync(syncFailuresPath(), { force: true }); } catch { /* none */ }
});
afterEach(() => {
if (originalHome) process.env.HOME = originalHome;
else delete process.env.HOME;
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
});
describe('Bug 9 — sync-failures JSONL helpers', () => {
test('recordSyncFailures appends one line per failure with dedup', async () => {
const { recordSyncFailures, loadSyncFailures, syncFailuresPath } = await import('../src/core/sync.ts');
recordSyncFailures([
{ path: 'people/alice.md', error: 'YAML: unexpected colon in title' },
{ path: 'notes/broken.md', error: 'YAML: duplicated key' },
], 'abc123def456');
expect(existsSync(syncFailuresPath())).toBe(true);
const entries = loadSyncFailures();
expect(entries.length).toBe(2);
expect(entries[0].path).toBe('people/alice.md');
expect(entries[0].commit).toBe('abc123def456');
expect(entries[0].acknowledged).toBeUndefined();
// Same failure on same commit should NOT re-append.
recordSyncFailures([
{ path: 'people/alice.md', error: 'YAML: unexpected colon in title' },
], 'abc123def456');
expect(loadSyncFailures().length).toBe(2);
// Different commit → new entry.
recordSyncFailures([
{ path: 'people/alice.md', error: 'YAML: unexpected colon in title' },
], 'zzz999');
expect(loadSyncFailures().length).toBe(3);
});
test('acknowledgeSyncFailures marks unacked entries, leaves acked alone', async () => {
const { recordSyncFailures, acknowledgeSyncFailures, loadSyncFailures } = await import('../src/core/sync.ts');
recordSyncFailures([
{ path: 'a.md', error: 'err1' },
{ path: 'b.md', error: 'err2' },
], 'commit1');
const n = acknowledgeSyncFailures();
expect(n).toBe(2);
const after = loadSyncFailures();
expect(after.every(e => e.acknowledged === true)).toBe(true);
expect(after.every(e => typeof e.acknowledged_at === 'string')).toBe(true);
// Second ack: nothing new to mark.
expect(acknowledgeSyncFailures()).toBe(0);
// Adding a fresh failure then ack: only the new one flips.
recordSyncFailures([{ path: 'c.md', error: 'err3' }], 'commit2');
expect(acknowledgeSyncFailures()).toBe(1);
expect(loadSyncFailures().length).toBe(3);
expect(loadSyncFailures().every(e => e.acknowledged === true)).toBe(true);
});
test('unacknowledgedSyncFailures filters correctly', async () => {
const { recordSyncFailures, acknowledgeSyncFailures, unacknowledgedSyncFailures } = await import('../src/core/sync.ts');
recordSyncFailures([{ path: 'a.md', error: 'err1' }], 'c1');
acknowledgeSyncFailures();
recordSyncFailures([{ path: 'b.md', error: 'err2' }], 'c2');
const unacked = unacknowledgedSyncFailures();
expect(unacked.length).toBe(1);
expect(unacked[0].path).toBe('b.md');
});
test('loadSyncFailures returns [] when file is missing', async () => {
const { loadSyncFailures } = await import('../src/core/sync.ts');
expect(loadSyncFailures()).toEqual([]);
});
test('loadSyncFailures tolerates malformed lines', async () => {
const { loadSyncFailures, syncFailuresPath, recordSyncFailures } = await import('../src/core/sync.ts');
// Seed one valid entry.
recordSyncFailures([{ path: 'a.md', error: 'err1' }], 'c1');
// Append garbage.
writeFileSync(syncFailuresPath(), readFileSync(syncFailuresPath(), 'utf-8') + 'NOT-JSON\n', { flag: 'w' });
const out = loadSyncFailures();
expect(out.length).toBe(1);
expect(out[0].path).toBe('a.md');
});
});
describe('Bug 9 — doctor surfaces sync failures', () => {
test('doctor source contains sync_failures check', async () => {
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
expect(source).toContain('sync_failures');
expect(source).toContain('unacknowledgedSyncFailures');
expect(source).toContain("'gbrain sync --skip-failed'");
});
});
describe('Bug 9 — sync.ts CLI flag wiring', () => {
test('runSync parses --skip-failed and --retry-failed flags', async () => {
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
expect(source).toContain("args.includes('--skip-failed')");
expect(source).toContain("args.includes('--retry-failed')");
expect(source).toContain('skipFailed');
expect(source).toContain('retryFailed');
});
test('performSync gates sync.last_commit on failedFiles.length', async () => {
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
// The gate exists and references the failure set.
expect(source).toContain('failedFiles.length > 0');
expect(source).toContain('blocked_by_failures');
});
test('performFullSync gates on result.failures from runImport', async () => {
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
expect(source).toContain('result.failures.length > 0');
});
test('runImport returns RunImportResult with failures list', async () => {
const source = await Bun.file(new URL('../src/commands/import.ts', import.meta.url)).text();
expect(source).toContain('RunImportResult');
expect(source).toContain('failures: Array<{ path: string; error: string }>');
expect(source).toContain('recordSyncFailures');
});
});

106
test/traverse-graph-dedup.test.ts Normal file
View File

@@ -0,0 +1,106 @@
/**
* Bug 6/10 regression — legacy traverseGraph jsonb_agg duplicate edges.
*
* The links table deliberately allows multiple rows with the same
* (from_page_id, to_page_id, link_type) when origin_page_id or link_source
* differ. That's how markdown-body edges and frontmatter edges coexist for
* the same pair. The duplicates should NOT surface in the legacy
* traverseGraph() aggregated output — dedup is presentation-only in the
* jsonb_agg step. This test seeds two such rows and asserts the aggregation
* collapses them. It also asserts the underlying `links` table still has
* both rows (provenance preserved).
*
* Runs against PGLite (unit, always). The postgres-engine path uses the
* same SQL; an E2E test covers Postgres.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
for (const t of ['links', 'pages']) {
await (engine as any).db.exec(`DELETE FROM ${t}`);
}
});
describe('Bug 6/10 — traverseGraph jsonb_agg DISTINCT', () => {
test('collapses two provenance rows for the same (from,to,type) edge', async () => {
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', frontmatter: {} });
await engine.putPage('companies/acme', { type: 'company', title: 'Acme', compiled_truth: '', frontmatter: {} });
const alice = await (engine as any).db.query(`SELECT id FROM pages WHERE slug = 'people/alice'`);
const acme = await (engine as any).db.query(`SELECT id FROM pages WHERE slug = 'companies/acme'`);
const fromId = alice.rows[0].id as string;
const toId = acme.rows[0].id as string;
// Two rows, same (from, to, type), different provenance:
// row 1 from markdown body (origin_page_id = from page itself, link_source 'markdown')
// row 2 from frontmatter (origin_page_id = null, link_source 'frontmatter')
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, origin_page_id, link_source)
VALUES ($1, $2, 'works_at', $1, 'markdown')`,
[fromId, toId],
);
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, origin_page_id, link_source)
VALUES ($1, $2, 'works_at', NULL, 'frontmatter')`,
[fromId, toId],
);
// Provenance preserved at the table level.
const rawCount = await (engine as any).db.query(
`SELECT count(*)::int as n FROM links WHERE from_page_id = $1 AND to_page_id = $2 AND link_type = 'works_at'`,
[fromId, toId],
);
expect(rawCount.rows[0].n).toBe(2);
// Aggregated output dedups.
const nodes = await engine.traverseGraph('people/alice', 2);
const alicedNode = nodes.find(n => n.slug === 'people/alice');
expect(alicedNode).toBeDefined();
const worksAtEdges = alicedNode!.links.filter(
l => l.to_slug === 'companies/acme' && l.link_type === 'works_at',
);
expect(worksAtEdges.length).toBe(1);
});
test('keeps genuinely distinct link types even between same nodes', async () => {
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: '', frontmatter: {} });
await engine.putPage('companies/widget', { type: 'company', title: 'Widget', compiled_truth: '', frontmatter: {} });
const bob = await (engine as any).db.query(`SELECT id FROM pages WHERE slug = 'people/bob'`);
const widget = await (engine as any).db.query(`SELECT id FROM pages WHERE slug = 'companies/widget'`);
const fromId = bob.rows[0].id as string;
const toId = widget.rows[0].id as string;
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, origin_page_id, link_source)
VALUES ($1, $2, 'works_at', $1, 'markdown')`,
[fromId, toId],
);
await (engine as any).db.query(
`INSERT INTO links (from_page_id, to_page_id, link_type, origin_page_id, link_source)
VALUES ($1, $2, 'founded', $1, 'markdown')`,
[fromId, toId],
);
const nodes = await engine.traverseGraph('people/bob', 2);
const bobNode = nodes.find(n => n.slug === 'people/bob');
const edges = bobNode!.links.filter(l => l.to_slug === 'companies/widget');
const types = edges.map(l => l.link_type).sort();
expect(types).toEqual(['founded', 'works_at']);
});
});