v0.38.2.0 fix(doctor): bounded frontmatter scan + partial-state surfacing (supersedes #1287) (#1297)
* fix(frontmatter): prune vendor dirs at descent + bounded wall-clock with partial-state surfacing Two production-grade fixes for the v0.38.2.0 wave (supersedes PR #1287). Root cause Fix 1 (the bug that hung gbrain doctor on 216K-page brains): both brain-writer.ts:walkDir and frontmatter.ts:collectFiles recursed into every subdirectory without calling pruneDir, the canonical descent-time pruner used by sync/extract/transcript-discovery since v0.35.5.0. On brains that double as code workspaces, the walkers stat'd hundreds of thousands of entries under node_modules / .git / .obsidian / *.raw / ops that isSyncable filtered out at the leaf — paying the IO cost for nothing. Wiring pruneDir at descent (with the v0.37.7.0 #1169 submodule-gitfile check) eliminates the bulk of the wall-clock pain. Fix 2 (codex outside-voice C1): AbortSignal.timeout cannot interrupt the synchronous walker — readdirSync / lstatSync / readFileSync block the event loop, so timer callbacks never fire mid-walk. The load-bearing wall-clock bound is now a deadline check inside scanOneSource's visit callback (Date.now() > opts.deadline). AbortSignal still works at source boundaries. Shape changes (codex C2 + C4): - ScanOpts: + deadline?: number, + dbPageCountForSource hook, + visitDir test seam - PerSourceReport: + status: 'scanned' | 'partial' | 'skipped', + files_scanned, + db_page_count - AuditReport: + partial: boolean, + aborted_at_source: string | null - ok = grandTotal === 0 && !partial (a clean prefix from a timed-out scan no longer falsely reports clean) walkDir + collectFiles now exported with an optional visitDir callback for the regression suite. Production callers don't pass it. Tests: - test/brain-writer-walk-prune.test.ts (new, 12 cases): visitDir-based descent-time pruning assertions for both walkers. Pins the property output-based tests can't catch (isSyncable rejects vendor files at the leaf — so a test checking only output passes under the original bug). - test/brain-writer-partial-scan.test.ts (new, 5 cases): deadline + partial state + ok-after-abort + numerator/denominator coverage. Uses deadline, NOT AbortSignal, since codex C1 proved abort can't interrupt sync. - test/brain-writer.test.ts: existing "abort mid-scan" test refit to the new partial-state contract (per_source has 'skipped' entries instead of being empty — gives doctor visibility into which sources weren't checked). - test/migrations-v0_22_4.test.ts: AuditReport fixture extended with the new required fields. Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): wire deadline + partial-state into frontmatter_integrity check Adopts the v0.38.2.0 ScanBrainSources surface in doctor's frontmatter_integrity check. - AbortSignal.timeout(fmTimeoutMs) for between-source bound. - deadline = Date.now() + fmTimeoutMs (the load-bearing mid-walk bound — codex C1 caught that AbortSignal alone can't fire inside the sync walker). - GBRAIN_DOCTOR_FM_TIMEOUT_MS env override (default 30000ms; invalid values fall back to default rather than crash). - Per-source DB denominator via SELECT COUNT(*) FROM pages WHERE source_id = $1 AND deleted_at IS NULL (codex C3: deleted_at filter so soft-deleted pages don't inflate the count). - Honest partial-render: "PARTIAL — scanned ~N files (source has ~M pages in DB), K issue(s) so far" instead of "scanned ~N of M pages" (codex C3 — the two populations are overlapping but not identical sets). - "NOT SCANNED (timeout — run gbrain frontmatter validate <id>)" per skipped source so the user knows which sources didn't get checked. - Catch block simplified to "unexpected error only" (codex D4 — the AbortError special case from PR #1287 was unreachable in a sync walker). Tests: test/doctor-frontmatter-partial.test.ts (new, 11 cases) — structural source-grep pins on every load-bearing render string plus the simplified- catch contract. Behavioral coverage is deferred to the heavy script (tests/heavy/frontmatter_scan_wallclock.sh, T6) because runDoctor calls process.exit unconditionally and can't be driven from bun:test directly; refactoring runDoctor to return rather than exit is a separate TODO. Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.38.2.0 release notes, Phase 2 design sketch, heavy wall-clock smoke - CHANGELOG.md: ELI10-lead-first release entry per CLAUDE.md voice rules. Names the user-visible behavior change, the per-source partial render, the performance numbers table, the "things to watch" caveats. Credits @garrytan-agents for PR #1287's diagnosis. - VERSION + package.json: 0.37.11.0 -> 0.38.2.0. - docs/architecture/frontmatter-scan-incremental.md: Phase 2 design sketch for DB-backed scan state. Schema, migration shape, writer paths (sync-side UPSERT + incremental scan + autopilot cycle phase), doctor reader, sequencing concerns, two-phase rollout plan. Starting point for the follow-up PR — sub-second steady-state doctor needs incremental state, but the schema migration carries its own contract surface (forward-reference bootstrap, schema-drift E2E, PGLite-vs-Postgres parity) that deserves its own focused PR. - tests/heavy/frontmatter_scan_wallclock.sh (new, manual / nightly per tests/heavy/README.md): seeds a synthetic 60K-file brain (10K real + 50K under node_modules/) and asserts gbrain doctor completes in <15s with frontmatter_integrity: ok. Codex C7 caught that the original plan's 1500-file budget was too small to be a meaningful guard — at that scale the test passes BEFORE AND AFTER the fix, proving nothing. 60K is the minimum that catches the descent-into-vendor-trees regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: adversarial review followups — CLI hint, deadline-vs-await race, between-source breadcrumb Codex adversarial review caught 4 real bugs in the v0.38.2.0 wave. All four fixed before ship. #1 (user-facing): `gbrain frontmatter validate` takes a filesystem PATH, not a source id. Pre-fix the NOT SCANNED hint pointed users at `gbrain frontmatter validate src-a` — which would fail with "no such directory", breaking the very remediation this PR ships to give them. Fix: render `src.source_path` instead. #2 (correctness): between sources, `await dbPageCountForSource(src.id)` ran unchecked. A slow query could blow past the deadline, then scanOneSource was still called and returned `status='partial'` with `files_scanned=0` — misleading ("partial scan" when actually zero files were scanned). Fix: add a post-await deadline re-check; mark source + remainder as 'skipped' if the budget already burned. #3 (UX): when the outer-loop deadline check fired BETWEEN sources, `aborted_at_source` stayed null and the doctor message said "PARTIAL SCAN" with no source name. Fix: stamp `aborted_at_source` with the source we were about to start. #4 (correctness): the COUNT query had no per-call deadline. A wedged Postgres pool could make a single COUNT hang past the budget and defeat the wall-clock guarantee. Fix: Promise.race against the remaining deadline; on timeout, resolve null and the post-await re-check (#2) marks the source skipped. Tests: 3 new regression cases in brain-writer-partial-scan.test.ts pinning the fixed contracts (skipped-vs-partial under slow COUNT, hanging COUNT within deadline, aborted_at_source before any source starts). 8648 pass / 0 fail across the full suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(README): refresh production-brain stats — 8.2x pages, 5.6x people, 7.4x companies Pre-update line (months stale): "17,888 pages, 4,383 people, 723 companies, 21 cron jobs running autonomously, built in 12 days." Fresh counts from ~/git/brain (the wintermute production brain): - pages: 17,888 → 146,646 (8.2x) - people: 4,383 → 24,585 (5.6x) - companies: 723 → 5,339 (7.4x) - cron jobs running: 21 → 66 (113 total, 66 enabled per ~/git/wintermute/workspace/ops/cron-snapshot.json) Dropped "built in 12 days" — at 146K pages the initial-velocity claim is stale narrative that no longer matches the current scale story. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
67
CHANGELOG.md
67
CHANGELOG.md
@@ -2,6 +2,72 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.38.2.0] - 2026-05-22
|
||||
|
||||
**`gbrain doctor` no longer hangs on big brains, and gives you real signal when it has to give up.**
|
||||
|
||||
If you've got a sizeable brain (tens of thousands of pages and up, especially in a repo that doubles as a code workspace), `gbrain doctor` used to feel like it locked up. The frontmatter check would sit there for a minute or more, and any cron monitor wrapping it would time out and report the whole health report as failed. The actual problem turned out to be embarrassing: the scanner was crawling into `node_modules/`, `.git/`, `.obsidian/`, and other vendor directories on every doctor run, paying the cost of touching hundreds of thousands of files it was always going to throw away. This release stops doing that, AND adds a real wall-clock budget so even pathologically large sources get a useful answer instead of a hang.
|
||||
|
||||
When the budget does run out, doctor now tells you exactly which sources finished, which one got partway, and which ones never got checked, with a paste-ready remediation command per source. No more black-box "scan timed out."
|
||||
|
||||
### How to upgrade
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
# That's it. No manual steps. Doctor is faster on the next run.
|
||||
```
|
||||
|
||||
### What you'd see in a concrete example
|
||||
|
||||
A brain with 3 federated sources where source-b is huge:
|
||||
|
||||
```
|
||||
$ gbrain doctor
|
||||
...
|
||||
frontmatter_integrity: warn
|
||||
216 frontmatter issue(s) (PARTIAL SCAN — timeout after 30s).
|
||||
src-a: 14 (NESTED_QUOTES=14);
|
||||
src-b: PARTIAL — scanned ~42000 files (source has ~200000 pages in DB), 202 issue(s) so far, NESTED_QUOTES=202;
|
||||
src-c: NOT SCANNED (timeout — run `gbrain frontmatter validate src-c`).
|
||||
Raise GBRAIN_DOCTOR_FM_TIMEOUT_MS or run `gbrain frontmatter validate <source>` directly.
|
||||
```
|
||||
|
||||
You get the breakdown per source instead of a single opaque warn. If you want a bigger budget for that one cron, `export GBRAIN_DOCTOR_FM_TIMEOUT_MS=60000` (default is 30 seconds).
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
| Brain shape | Pre-v0.38.2.0 | Post-v0.38.2.0 |
|
||||
|---|---|---|
|
||||
| 10K real pages, no vendor dirs | seconds | seconds (no regression) |
|
||||
| 10K real pages + node_modules with 50K files | hangs past 30s | seconds (descent prunes vendor trees) |
|
||||
| 200K+ real pages, single source | hangs forever | partial result inside budget, per-source breakdown |
|
||||
| Cron wrapped in `timeout 60s` | exit 124, full health report fails | exit 0, `frontmatter_integrity: warn` with actionable detail |
|
||||
|
||||
### Things to watch
|
||||
|
||||
- **`gbrain frontmatter validate <source>` is also faster.** The same fix applies to that command's walker (there were two walkers with the same bug; one PR closed both).
|
||||
- **`AbortSignal.timeout` is the lesser bound, not the load-bearing one.** The hard wall-clock guarantee comes from a deadline check inside the file-by-file scan loop. (Pre-merge a Codex review caught that the timer-based abort can't interrupt synchronous filesystem calls; the deadline check is what actually fires mid-walk.)
|
||||
- **Steady-state cost is still O(N) in real pages.** This release delivers bounded wall-clock and honest partial-state signal, NOT sub-second steady-state doctor. Driving the steady state to constant-cost needs DB-backed scan state — designed in `docs/architecture/frontmatter-scan-incremental.md`, deferred to a follow-up PR because schema migrations carry their own contract surface.
|
||||
|
||||
### Supersedes PR #1287
|
||||
|
||||
Community PR #1287 (by @garrytan-agents) diagnosed the hang correctly and proposed a 10-line `AbortSignal.timeout` band-aid. We took the bug seriously enough to find a deeper root cause: the walker was descending into vendor directories on every tick. PR #1287 closes after this release lands. Thanks to the agent for the diagnosis — the timeout plumbing it added is part of the safety net that ships here.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/brain-writer.ts`: `walkDir` now consults `pruneDir(name, parentDir)` at descent time, the canonical pruner used by sync/extract/transcript-discovery since v0.35.5.0. Skips `node_modules`, `.git`, `.obsidian`, `*.raw`, `ops`, all dot-prefix dirs, and git submodule dirs (`.git` as FILE per v0.37.7.0 #1169). `ScanOpts` gains `deadline?: number` (epoch ms) checked per file inside `scanOneSource` — the load-bearing wall-clock bound. `PerSourceReport` gains `status: 'scanned' | 'partial' | 'skipped'`, `files_scanned: number` (numerator), and `db_page_count?: number | null` (denominator from the doctor-supplied SQL hook). `AuditReport` gains `partial: boolean` and `aborted_at_source: string | null`. `ok` is now `grandTotal === 0 && !partial` — a clean prefix from a timed-out scan no longer falsely reports clean. `walkDir` exported with an optional `visitDir` test callback for the regression suite (production callers don't pass it).
|
||||
- `src/commands/frontmatter.ts`: `collectFiles` (the second walker — drives `gbrain frontmatter validate`) gets the same `pruneDir` wiring + optional `visitDir` callback. Now `export`ed. Without this fix, doctor's own remediation hint pointed users at a command that hung the same way.
|
||||
- `src/commands/doctor.ts`: `frontmatter_integrity` adopts the deadline + AbortSignal.timeout pair (configurable via `GBRAIN_DOCTOR_FM_TIMEOUT_MS`, default 30000ms). Per-source DB denominator via `SELECT COUNT(*) FROM pages WHERE source_id = $1 AND deleted_at IS NULL`. Partial render distinguishes `'scanned' | 'partial' | 'skipped'` with honest "scanned ~N files (source has ~M pages in DB)" wording — the DB COUNT and the on-disk scan are overlapping but not identical sets, and the message reflects that. Catch block simplified to "unexpected error only" (no AbortError special case; the abort path returns cleanly via partial state, not via throw).
|
||||
- Tests: `test/brain-writer-walk-prune.test.ts` (12 cases, REGRESSION suite for both walkers — uses the new `visitDir` callback for direct descent-time observability, not leaf-output assertions which would pass under the original bug). `test/brain-writer-partial-scan.test.ts` (5 cases for deadline + partial state + ok-after-abort + numerator/denominator). `test/doctor-frontmatter-partial.test.ts` (11 structural source-grep cases pinning the load-bearing render strings).
|
||||
- `tests/heavy/frontmatter_scan_wallclock.sh` (new, manual / nightly per `tests/heavy/README.md`): seeds a synthetic 60K-file brain (10K real + 50K under node_modules) and asserts `gbrain doctor` completes in <15s with `frontmatter_integrity: ok`. Catches the regression at a scale where the original bug actually shows.
|
||||
- `docs/architecture/frontmatter-scan-incremental.md`: Phase 2 design sketch for DB-backed scan state — schema migration, sync-side writes, autopilot cycle phase, doctor reader. Captured so the follow-up PR has a starting point.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The walker test surface changes shape: `walkDir` (brain-writer.ts) and `collectFiles` (frontmatter.ts) are now exported with an optional `visitDir` callback. Tests should use this for descent-time pruning assertions; leaf-output tests can pass under the original bug since `isSyncable` filters at the leaf.
|
||||
- `scanBrainSources` now returns `partial` + `aborted_at_source` on `AuditReport` and `status` + `files_scanned` (+ optional `db_page_count`) per `PerSourceReport`. Any test that constructs `AuditReport` literals needs to include the new required fields. (One pre-existing test was updated as part of this PR.)
|
||||
- `runDoctor` still calls `process.exit` at the end — behavioral tests against it can't run via the unit-test runner. That refactor stays a TODO; the unit-test layer covers `scanBrainSources` + doctor's render shape via source-grep, and the heavy script covers end-to-end against a subprocess.
|
||||
|
||||
## [0.38.1.0] - 2026-05-21
|
||||
|
||||
**Your `gbrain agent run` loop can now run on any provider with native tool calling — not just Anthropic.** OpenAI, Google Gemini, OpenRouter, openai-compatible servers (Ollama, LiteLLM, vLLM, llama-server) all work. Pick the cheapest model that does the job for your agent, or stay on Anthropic if you want the prompt-cache cost savings on long loops.
|
||||
@@ -260,6 +326,7 @@ These do not block the v0.38 release: the substrate is shipped and queryable; so
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
|
||||
warns about a partial migration:
|
||||
|
||||
## [0.37.11.0] - 2026-05-21
|
||||
|
||||
**Fresh `gbrain init --pglite` works out of the box now.**
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Your AI agent is smart but forgetful. GBrain gives it a brain.
|
||||
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
|
||||
200
docs/architecture/frontmatter-scan-incremental.md
Normal file
200
docs/architecture/frontmatter-scan-incremental.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# Frontmatter scan: DB-backed incremental state (Phase 2 design sketch)
|
||||
|
||||
**Status:** Designed, not built. Captured here as the starting point for the
|
||||
follow-up PR after v0.38.2.0.
|
||||
|
||||
## Why this exists
|
||||
|
||||
v0.38.2.0 fixed the load-bearing bug class that caused `gbrain doctor` to
|
||||
hang on large brains: the disk walker descended into `node_modules/`, `.git/`,
|
||||
and other vendor trees on every tick. After that fix doctor completes in
|
||||
seconds on most brains, and bounded wall-clock (default 30s, with honest
|
||||
partial-state surfacing) on any brain.
|
||||
|
||||
But the steady-state cost of `frontmatter_integrity` is still O(N) in real
|
||||
syncable pages: every doctor tick re-walks the filesystem and re-parses
|
||||
every `.md` file. For users with 200K+ pages the steady-state cost is in
|
||||
the seconds even after Fix 1. For sub-second steady-state doctor (the
|
||||
right shape for cron-monitored health checks), the scan needs to become
|
||||
incremental.
|
||||
|
||||
This document captures the Phase 2 design before the follow-up PR starts,
|
||||
so the implementer doesn't have to re-derive it.
|
||||
|
||||
## Goal
|
||||
|
||||
Doctor's `frontmatter_integrity` check completes in O(1) SQL queries
|
||||
regardless of brain size, with the same per-source breakdown and partial-
|
||||
state semantics as v0.38.2.0's bounded-walk approach. Incremental refresh
|
||||
runs as a sync-side write + an autopilot cycle phase, so the steady-state
|
||||
work is amortized across the workflow that already touches each file.
|
||||
|
||||
## Schema
|
||||
|
||||
New table:
|
||||
|
||||
```sql
|
||||
CREATE TABLE frontmatter_scan_state (
|
||||
source_id TEXT NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
path TEXT NOT NULL, -- relative to source.local_path
|
||||
mtime_ms BIGINT NOT NULL,
|
||||
content_hash TEXT NOT NULL, -- sha256 of file content at scan time
|
||||
codes JSONB NOT NULL DEFAULT '[]'::jsonb, -- ParseValidationCode[]
|
||||
last_scanned_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (source_id, path)
|
||||
);
|
||||
|
||||
CREATE INDEX frontmatter_scan_state_has_issues_idx
|
||||
ON frontmatter_scan_state (source_id)
|
||||
WHERE codes != '[]'::jsonb;
|
||||
```
|
||||
|
||||
Why these columns:
|
||||
- `mtime_ms` + `content_hash`: incremental check picks one. mtime is faster
|
||||
(no read); content_hash is the truth (defeats touch-without-change cases).
|
||||
The incremental walker uses mtime as a fast gate and content_hash as the
|
||||
fallback when mtime suggests change.
|
||||
- `codes` JSONB: per-row error code list, NULL/`[]` means clean. Doctor
|
||||
aggregates with `jsonb_array_length(codes) > 0`.
|
||||
- Partial index on `WHERE codes != '[]'::jsonb`: doctor's aggregate query
|
||||
only walks rows with issues, which is a small fraction of pages.
|
||||
|
||||
This follows the canonical `applyForwardReferenceBootstrap` pattern in
|
||||
`src/core/pglite-engine.ts` (and `postgres-engine.ts`) — the new column /
|
||||
table additions go into the bootstrap probe set per CLAUDE.md so old brains
|
||||
walking forward through the schema chain don't wedge on the table not
|
||||
existing.
|
||||
|
||||
## Migration shape
|
||||
|
||||
```ts
|
||||
// src/core/migrate.ts — append after the v80 entry
|
||||
const migrations = [
|
||||
// ...existing v1-v80...
|
||||
{
|
||||
version: 81,
|
||||
name: 'frontmatter_scan_state',
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS frontmatter_scan_state (...);
|
||||
CREATE INDEX IF NOT EXISTS frontmatter_scan_state_has_issues_idx ...;
|
||||
`,
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
Plus the forward-reference probe entries in both engine bootstraps. Plus
|
||||
the `REQUIRED_BOOTSTRAP_COVERAGE` extension in
|
||||
`test/schema-bootstrap-coverage.test.ts`.
|
||||
|
||||
## Writers
|
||||
|
||||
Two paths write rows:
|
||||
|
||||
1. **Sync-side write** (canonical). `src/core/sync.ts:performSync` already
|
||||
parses every file it touches. After the existing `parseMarkdown` call,
|
||||
`UPSERT` into `frontmatter_scan_state` with the file's path / mtime /
|
||||
content_hash / codes. Cost: one row per file synced. Zero extra parse
|
||||
work — the parse already happened.
|
||||
|
||||
2. **Incremental scan** (`gbrain frontmatter scan --incremental`). Walks
|
||||
the disk via `walkBrainTree`, for each file checks `mtime > last_scanned_at`
|
||||
OR `content_hash != stored`, only re-parses changed files. Most ticks:
|
||||
zero work after the first full backfill. Also exposed as an autopilot
|
||||
cycle phase (`frontmatter_scan`) so it runs alongside the other periodic
|
||||
maintenance phases.
|
||||
|
||||
The incremental walker handles two cases sync misses:
|
||||
- Files edited outside sync (user opens an editor, saves, never `git
|
||||
commit`s).
|
||||
- Sources whose `local_path` isn't a git repo (sync only sees git-touched
|
||||
files).
|
||||
|
||||
## Doctor reader
|
||||
|
||||
```ts
|
||||
// src/commands/doctor.ts:frontmatter_integrity (Phase 2 shape)
|
||||
const rows = await engine.executeRaw<{ source_id: string; issues: number }>(
|
||||
`SELECT source_id, count(*) FILTER (WHERE jsonb_array_length(codes) > 0)::int AS issues
|
||||
FROM frontmatter_scan_state
|
||||
GROUP BY source_id`,
|
||||
);
|
||||
```
|
||||
|
||||
One SQL query, constant time regardless of brain size. The partial-state
|
||||
surfacing from v0.38.2.0 stays — when `frontmatter_scan_state` is stale
|
||||
(no rows for a registered source, or `last_scanned_at` >24h old for any
|
||||
source), doctor warns about freshness rather than reporting potentially-
|
||||
stale data as authoritative.
|
||||
|
||||
## Sequencing concerns
|
||||
|
||||
1. **First-ever scan.** A fresh upgrade has no rows in
|
||||
`frontmatter_scan_state`. Two options:
|
||||
- Lazy: doctor reports "no scan state yet; run `gbrain frontmatter scan
|
||||
--incremental` once" (operator-driven).
|
||||
- Eager: the migration that creates the table also enqueues an autopilot
|
||||
cycle job to do the first full scan.
|
||||
|
||||
Recommendation: lazy, with a clear hint. The autopilot path is heavier
|
||||
surface (must add the new `frontmatter_scan` phase to the existing
|
||||
cycle.ts machinery + the doctor-routed background job system).
|
||||
|
||||
2. **Source archival / deletion.** `frontmatter_scan_state` has `ON DELETE
|
||||
CASCADE` on `sources(id)`, so soft-delete + 72h TTL + purge already
|
||||
clean it up. No additional logic needed.
|
||||
|
||||
3. **Path renames inside a source.** Sync would `DELETE` the old row by
|
||||
path (via a periodic reconcile step) and `INSERT` the new row. Without
|
||||
that step, the table accumulates stale path rows. Either:
|
||||
- A reconcile step in the incremental scanner: any path-row not seen
|
||||
during the walk gets deleted.
|
||||
- Or: doctor reports "N stale rows in frontmatter_scan_state" as a
|
||||
freshness signal, with `gbrain frontmatter scan --reconcile` as the
|
||||
remediation.
|
||||
|
||||
## Cost estimate
|
||||
|
||||
- One UPSERT per file synced. Negligible vs the parse + DB write that sync
|
||||
already does.
|
||||
- Incremental refresh runtime: dominated by mtime stats. ~ms per 1000 files
|
||||
on SSD.
|
||||
- Doctor read: one indexed SQL query. Sub-100ms on any brain size.
|
||||
|
||||
## What this design deliberately does NOT do
|
||||
|
||||
- **Replace v0.38.2.0's bounded-walk safety net.** Phase 2 makes the
|
||||
steady-state cheap, but the disk walker (with its deadline check) stays
|
||||
as the source-of-truth fallback for sources whose scan state is missing
|
||||
or stale. Belt-and-suspenders.
|
||||
- **Introduce a separate frontmatter validation rule set.** Reuses
|
||||
`parseMarkdown(..., {validate: true})` and the existing
|
||||
`ParseValidationCode` enum. Single source of truth.
|
||||
- **Add a new background daemon.** Wires into the existing
|
||||
`autopilot-cycle` Minion handler as a new phase, alongside sync /
|
||||
extract / embed / etc.
|
||||
|
||||
## Open questions for the implementer
|
||||
|
||||
1. **Path normalization.** `pages.source_path` and the disk walker's
|
||||
relative path computation are similar but not identical (slashes,
|
||||
leading `./`, etc.). The incremental scanner needs to match what sync
|
||||
stores so UPSERTs key correctly. Audit before writing.
|
||||
2. **Soft-delete interaction.** A page that gets soft-deleted in the DB
|
||||
(v0.26.5) still has a file on disk. Should the incremental scan
|
||||
continue to track its frontmatter state? Probably yes (so a future
|
||||
`restore_page` doesn't surprise with stale frontmatter), but worth
|
||||
confirming with the soft-delete owner.
|
||||
3. **Two-phase rollout.** Land the table + writes first, let it backfill
|
||||
for a release cycle, then switch the doctor reader. Avoids the
|
||||
"Phase 2 ships but the table is empty" case where doctor regresses to
|
||||
reporting "no scan state."
|
||||
|
||||
## TODO file entry
|
||||
|
||||
```
|
||||
- [ ] Implement Phase 2: DB-backed frontmatter scan state.
|
||||
Design lives at docs/architecture/frontmatter-scan-incremental.md.
|
||||
Schema migration v81 + sync-side UPSERT + incremental scan command
|
||||
+ autopilot cycle phase + doctor reader. Two-phase rollout: ship
|
||||
table + writes first; flip the reader one release later.
|
||||
```
|
||||
@@ -2439,7 +2439,7 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/README.md
|
||||
|
||||
Your AI agent is smart but forgetful. GBrain gives it a brain.
|
||||
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
|
||||
Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain behind his OpenClaw and Hermes deployments: **146,646 pages, 24,585 people, 5,339 companies**, 66 cron jobs running autonomously. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up smarter than when you went to bed.
|
||||
|
||||
The brain wires itself. Every page write extracts entity references and creates typed links (`attended`, `works_at`, `invested_in`, `founded`, `advises`) with zero LLM calls. Hybrid search. Self-wiring knowledge graph. Structured timeline. Backlink-boosted ranking. Ask "who works at Acme AI?" or "what did Bob invest in this quarter?" and get answers vector search alone can't reach. Benchmarked side-by-side: gbrain lands **P@5 49.1%, R@5 97.9%** on a 240-page Opus-generated rich-prose corpus, beating its graph-disabled variant by **+31.4 points P@5** and ripgrep-BM25 + vector-only RAG by a similar margin. Full BrainBench scorecards live in the sibling [gbrain-evals](https://github.com/garrytan/gbrain-evals) repo.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.38.1.0",
|
||||
"version": "0.38.2.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -3094,18 +3094,65 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4).
|
||||
// 11a. Frontmatter integrity (v0.22.4, hardened in v0.38.2.0).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
// file. Reports per-source counts grouped by error code. The fix path is
|
||||
// `gbrain frontmatter validate <source-path> --fix`, which writes .bak
|
||||
// backups so it works for both git and non-git brain repos.
|
||||
//
|
||||
// v0.38.2.0 wave (this PR supersedes PR #1287):
|
||||
// - `pruneDir` now applies at descent inside brain-writer.ts:walkDir so
|
||||
// the scan no longer recurses into node_modules / .git / .obsidian /
|
||||
// *.raw / ops. That alone takes the 216K-page user from "hangs
|
||||
// forever" to "completes in seconds" on the typical brain.
|
||||
// - `deadline` (per-file Date.now() check inside the sync loop) is the
|
||||
// load-bearing wall-clock bound. AbortSignal.timeout (kept for
|
||||
// between-source aborts) cannot interrupt sync readdirSync /
|
||||
// readFileSync — codex outside-voice C1 caught the original plan's
|
||||
// assumption that it could.
|
||||
// - Partial-result surfacing: per-source status ('scanned' | 'partial' |
|
||||
// 'skipped'), files_scanned numerator, and an honest "scanned ~N files
|
||||
// (source has ~M pages in DB)" message when the deadline fires. The
|
||||
// `partial` and `aborted_at_source` fields on AuditReport feed the
|
||||
// JSON consumer.
|
||||
// - Configurable via GBRAIN_DOCTOR_FM_TIMEOUT_MS (default 30000ms).
|
||||
progress.heartbeat('frontmatter_integrity');
|
||||
const fmHb = startHeartbeat(progress, 'scanning frontmatter…');
|
||||
const fmTimeoutMs = (() => {
|
||||
const raw = process.env.GBRAIN_DOCTOR_FM_TIMEOUT_MS;
|
||||
const n = raw ? parseInt(raw, 10) : NaN;
|
||||
return Number.isFinite(n) && n > 0 ? n : 30000;
|
||||
})();
|
||||
try {
|
||||
const { scanBrainSources } = await import('../core/brain-writer.ts');
|
||||
const report = await scanBrainSources(engine);
|
||||
if (report.total === 0) {
|
||||
const fmDeadline = Date.now() + fmTimeoutMs;
|
||||
const fmAbort = AbortSignal.timeout(fmTimeoutMs);
|
||||
// Per-source DB denominator. Coarse — DB pages and on-disk syncable
|
||||
// files are overlapping but not identical (unsynced disk files,
|
||||
// soft-deleted DB rows, auto-generated pages). Wording in the partial
|
||||
// message makes the mismatch honest. Failure of the COUNT degrades to
|
||||
// null and the message falls back to bare numerator.
|
||||
const dbPageCountForSource = async (sourceId: string): Promise<number | null> => {
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT COUNT(*)::text AS n FROM pages WHERE source_id = $1 AND deleted_at IS NULL`,
|
||||
[sourceId],
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
const parsed = parseInt(rows[0].n, 10);
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
const report = await scanBrainSources(engine, {
|
||||
signal: fmAbort,
|
||||
deadline: fmDeadline,
|
||||
dbPageCountForSource,
|
||||
});
|
||||
|
||||
if (report.total === 0 && !report.partial) {
|
||||
const sources = report.per_source.length;
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
@@ -3115,23 +3162,55 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
: `${sources} source(s) clean — no frontmatter issues`,
|
||||
});
|
||||
} else {
|
||||
// Build per-source breakdown that distinguishes scanned / partial /
|
||||
// skipped so the user can tell which sources weren't checked.
|
||||
const sourceMessages: string[] = [];
|
||||
for (const src of report.per_source) {
|
||||
if (src.total === 0) continue;
|
||||
if (src.status === 'skipped') {
|
||||
// Codex adversarial #1: `gbrain frontmatter validate` takes a
|
||||
// filesystem PATH, not a source id. Pre-fix the hint pointed users
|
||||
// at a command that would fail with "no such directory" — breaking
|
||||
// the very remediation path this PR ships to give them.
|
||||
sourceMessages.push(
|
||||
`${src.source_id}: NOT SCANNED (timeout — run \`gbrain frontmatter validate ${src.source_path}\`)`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (src.status === 'partial') {
|
||||
const denom = src.db_page_count != null ? ` (source has ~${src.db_page_count} pages in DB)` : '';
|
||||
const codes = src.total > 0
|
||||
? `, ${Object.entries(src.errors_by_code).map(([k, v]) => `${k}=${v}`).join(', ')}`
|
||||
: '';
|
||||
sourceMessages.push(
|
||||
`${src.source_id}: PARTIAL — scanned ~${src.files_scanned} files${denom}, ${src.total} issue(s) so far${codes}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
// status === 'scanned'
|
||||
if (src.total === 0) continue; // clean source — don't clutter the message
|
||||
const codes = Object.entries(src.errors_by_code)
|
||||
.map(([k, v]) => `${k}=${v}`)
|
||||
.join(', ');
|
||||
sourceMessages.push(`${src.source_id}: ${src.total} (${codes})`);
|
||||
}
|
||||
const fixHint = report.partial
|
||||
? `Raise GBRAIN_DOCTOR_FM_TIMEOUT_MS or run \`gbrain frontmatter validate <source>\` directly. Fix issues: \`gbrain frontmatter validate <source> --fix\``
|
||||
: `Fix: gbrain frontmatter validate <source-path> --fix`;
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${report.total} frontmatter issue(s) across ${sourceMessages.length} source(s). ` +
|
||||
`${sourceMessages.join('; ')}. Fix: gbrain frontmatter validate <source-path> --fix`,
|
||||
`${report.total} frontmatter issue(s)` +
|
||||
(report.partial ? ` (PARTIAL SCAN — timeout after ${fmTimeoutMs / 1000}s)` : '') +
|
||||
`. ${sourceMessages.join('; ')}. ${fixHint}`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
// Codex outside-voice D4: the abort path returns cleanly via partial
|
||||
// state — this catch is purely for unexpected errors (FS permission,
|
||||
// OOM, disk full, etc.). Pre-v0.38.2.0 (PR #1287) had an unreachable
|
||||
// abort-classifier branch here; removed because timer-based aborts
|
||||
// in a sync walker can't surface as a thrown error anyway.
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
|
||||
@@ -29,7 +29,7 @@ import {
|
||||
type AuditReport,
|
||||
type AuditFix,
|
||||
} from '../core/brain-writer.ts';
|
||||
import { isSyncable, slugifyPath } from '../core/sync.ts';
|
||||
import { isSyncable, pruneDir, slugifyPath } from '../core/sync.ts';
|
||||
|
||||
export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
@@ -245,13 +245,35 @@ async function runValidate(rest: string[]): Promise<void> {
|
||||
process.exitCode = totalErrors > 0 && !flags.fix ? 1 : 0;
|
||||
}
|
||||
|
||||
function collectFiles(target: string): string[] {
|
||||
/**
|
||||
* Recursively collect every syncable `.md` file under `target`.
|
||||
*
|
||||
* Uses the canonical `pruneDir(name, parentDir)` gate (sync.ts:258) to
|
||||
* skip vendor / hidden / generated subtrees at descent time. Pre-v0.38.2.0
|
||||
* this walker descended into every subtree and let `isSyncable` filter at
|
||||
* the leaf — paying the IO cost of stat'ing every entry under node_modules,
|
||||
* .git, .obsidian, etc. That was the second instance of the v0.38.2.0 hang
|
||||
* class (the first being brain-writer.ts:walkDir). Codex outside-voice
|
||||
* caught it during plan-eng-review — fixing only walkDir would have left
|
||||
* `gbrain frontmatter validate` (doctor's own remediation hint) hanging
|
||||
* users in the same way.
|
||||
*
|
||||
* Optional `visitDir(dir)` is the test-observability hook: fired once per
|
||||
* directory the walker descends into (post-pruneDir). Production callers
|
||||
* don't pass it; the regression suite uses it to assert descent-time
|
||||
* pruning directly.
|
||||
*/
|
||||
export function collectFiles(
|
||||
target: string,
|
||||
visitDir?: (dirPath: string) => void,
|
||||
): string[] {
|
||||
const st = lstatSync(target);
|
||||
if (st.isFile()) {
|
||||
return [target];
|
||||
}
|
||||
const out: string[] = [];
|
||||
const stack = [target];
|
||||
if (visitDir) visitDir(target);
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
@@ -270,6 +292,10 @@ function collectFiles(target: string): string[] {
|
||||
}
|
||||
if (entryStat.isSymbolicLink()) continue;
|
||||
if (entryStat.isDirectory()) {
|
||||
// Descent-time prune — the actual fix for the second walker bug
|
||||
// class (codex outside-voice C5).
|
||||
if (!pruneDir(name, dir)) continue;
|
||||
if (visitDir) visitDir(full);
|
||||
stack.push(full);
|
||||
} else if (entryStat.isFile()) {
|
||||
const rel = relative(target, full);
|
||||
|
||||
@@ -27,7 +27,7 @@ import {
|
||||
type ParseValidationCode,
|
||||
type ParseValidationError,
|
||||
} from './markdown.ts';
|
||||
import { isSyncable, slugifyPath } from './sync.ts';
|
||||
import { isSyncable, pruneDir, slugifyPath } from './sync.ts';
|
||||
|
||||
export type { ParseValidationCode };
|
||||
|
||||
@@ -43,6 +43,18 @@ export interface PerSourceReport {
|
||||
errors_by_code: Partial<Record<ParseValidationCode, number>>;
|
||||
sample: { path: string; codes: ParseValidationCode[] }[];
|
||||
ignoredMissingOpen: number;
|
||||
/** Did this source finish the walk, get interrupted, or never start?
|
||||
* 'scanned' = full walk completed.
|
||||
* 'partial' = deadline/abort fired mid-walk; counts reflect prefix only.
|
||||
* 'skipped' = source was never visited (outer loop broke before reaching it). */
|
||||
status: 'scanned' | 'partial' | 'skipped';
|
||||
/** Count of .md files actually parsed (numerator for "scanned ~N of M" doctor message).
|
||||
* Distinct from `total` (which counts ERRORS) and `ignoredMissingOpen`. */
|
||||
files_scanned: number;
|
||||
/** DB-side denominator for the partial-state message. Coarse — DB pages and
|
||||
* on-disk syncable files are overlapping but not identical sets. NULL when the
|
||||
* COUNT query failed or wasn't issued. */
|
||||
db_page_count?: number | null;
|
||||
}
|
||||
|
||||
export interface AuditReport {
|
||||
@@ -52,6 +64,12 @@ export interface AuditReport {
|
||||
per_source: PerSourceReport[];
|
||||
scanned_at: string;
|
||||
ignored_missing_open?: number;
|
||||
/** True when any source got `status: 'partial'` or `'skipped'`. Doctor uses
|
||||
* this to render the warn message and to ensure `ok` is false even when the
|
||||
* scanned prefix happened to be clean (codex C2 fix). */
|
||||
partial: boolean;
|
||||
/** Source id where deadline/abort fired mid-walk, or null when no partial. */
|
||||
aborted_at_source: string | null;
|
||||
}
|
||||
|
||||
const SAMPLE_PER_SOURCE = 20;
|
||||
@@ -365,7 +383,29 @@ export interface ScanOpts {
|
||||
* YAML frontmatter. */
|
||||
strictMissingOpen?: boolean;
|
||||
onProgress?: ProgressReporter;
|
||||
/** Outer-loop abort: checked at source boundaries in scanBrainSources.
|
||||
* Does NOT interrupt the synchronous file-walk inside a single source —
|
||||
* use `deadline` for that. */
|
||||
signal?: AbortSignal;
|
||||
/** Hard wall-clock bound for the synchronous walk. When set, scanOneSource
|
||||
* checks `Date.now() > deadline` before parsing each file and returns
|
||||
* partial state when exceeded.
|
||||
*
|
||||
* This is the load-bearing mid-walk interruption mechanism. AbortSignal.timeout
|
||||
* cannot interrupt sync readdirSync/lstatSync/readFileSync (event loop blocked),
|
||||
* so a deadline epoch-ms check is the only way to actually bound wall-clock
|
||||
* inside the visit loop. Worst-case overshoot: one file's parse time. */
|
||||
deadline?: number;
|
||||
/** Async per-source DB hook: doctor uses this to fetch a coarse denominator
|
||||
* (`SELECT COUNT(*) FROM pages WHERE source_id = $1 AND deleted_at IS NULL`)
|
||||
* for the "scanned ~N of M" partial message. Returns null on query failure;
|
||||
* scanBrainSources stamps the result on PerSourceReport.db_page_count.
|
||||
* Optional — when omitted, db_page_count stays undefined. */
|
||||
dbPageCountForSource?: (sourceId: string) => Promise<number | null>;
|
||||
/** Test seam — fired by walkDir once per directory it descends into
|
||||
* (post-pruneDir). Production callers don't pass this; the regression
|
||||
* suite uses it to assert descent-time pruning directly. */
|
||||
visitDir?: (dirPath: string) => void;
|
||||
}
|
||||
|
||||
export async function scanBrainSources(
|
||||
@@ -377,9 +417,41 @@ export async function scanBrainSources(
|
||||
const perSource: PerSourceReport[] = [];
|
||||
let grandTotal = 0;
|
||||
let ignoredMissingOpen = 0;
|
||||
let abortedAtSource: string | null = null;
|
||||
|
||||
for (const src of sources) {
|
||||
if (opts.signal?.aborted) break;
|
||||
// Helper: mark sources from index i onward as 'skipped'. Used at every
|
||||
// between-source abort point (top of loop AND after the COUNT await).
|
||||
const markRemainingSkipped = (startIdx: number) => {
|
||||
for (let j = startIdx; j < sources.length; j++) {
|
||||
const skipped = sources[j];
|
||||
if (!skipped.local_path) continue;
|
||||
perSource.push({
|
||||
source_id: skipped.id,
|
||||
source_path: skipped.local_path,
|
||||
total: 0,
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
ignoredMissingOpen: 0,
|
||||
status: 'skipped',
|
||||
files_scanned: 0,
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
for (let i = 0; i < sources.length; i++) {
|
||||
const src = sources[i];
|
||||
// Between-source abort check: AbortSignal works here (no sync I/O blocking
|
||||
// the event loop at this boundary). For mid-walk interruption use deadline.
|
||||
if (opts.signal?.aborted || (opts.deadline && Date.now() > opts.deadline)) {
|
||||
// Codex adversarial review #3: when deadline fires BETWEEN sources,
|
||||
// also stamp aborted_at_source with the source we were about to start.
|
||||
// Pre-fix, the doctor message said "PARTIAL SCAN" with no source name.
|
||||
if (abortedAtSource === null && src.local_path) {
|
||||
abortedAtSource = src.id;
|
||||
}
|
||||
markRemainingSkipped(i);
|
||||
break;
|
||||
}
|
||||
if (!src.local_path) continue;
|
||||
if (!existsSync(src.local_path)) {
|
||||
// Source registered but path is missing on disk; surface as a zero-row
|
||||
@@ -391,10 +463,53 @@ export async function scanBrainSources(
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
ignoredMissingOpen: 0,
|
||||
status: 'scanned',
|
||||
files_scanned: 0,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Best-effort denominator fetch — degrades gracefully on query failure.
|
||||
// Codex adversarial #4: also race against the deadline. A wedged Postgres
|
||||
// pool can make this await hang past the budget. Without the race, we'd
|
||||
// wait indefinitely AND defeat the wall-clock guarantee.
|
||||
let dbPageCount: number | null = null;
|
||||
if (opts.dbPageCountForSource) {
|
||||
try {
|
||||
if (opts.deadline) {
|
||||
const remainingMs = opts.deadline - Date.now();
|
||||
if (remainingMs <= 0) {
|
||||
dbPageCount = null;
|
||||
} else {
|
||||
// Race COUNT against the deadline so a hung query can't eat the budget.
|
||||
dbPageCount = await Promise.race([
|
||||
opts.dbPageCountForSource(src.id),
|
||||
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs)),
|
||||
]);
|
||||
}
|
||||
} else {
|
||||
dbPageCount = await opts.dbPageCountForSource(src.id);
|
||||
}
|
||||
} catch {
|
||||
dbPageCount = null;
|
||||
}
|
||||
}
|
||||
|
||||
// Codex adversarial #2: re-check deadline AFTER the COUNT await. If the
|
||||
// await ate the budget, we must NOT call scanOneSource — it would return
|
||||
// status='partial' with files_scanned=0, which is misleading ("partial
|
||||
// scan" when actually nothing was scanned). Mark this source + remainder
|
||||
// as 'skipped' so the doctor message is honest.
|
||||
if (opts.signal?.aborted || (opts.deadline && Date.now() > opts.deadline)) {
|
||||
if (abortedAtSource === null) {
|
||||
abortedAtSource = src.id;
|
||||
}
|
||||
markRemainingSkipped(i);
|
||||
break;
|
||||
}
|
||||
|
||||
const report = scanOneSource(src.id, src.local_path, opts);
|
||||
report.db_page_count = dbPageCount;
|
||||
perSource.push(report);
|
||||
grandTotal += report.total;
|
||||
ignoredMissingOpen += report.ignoredMissingOpen;
|
||||
@@ -402,15 +517,25 @@ export async function scanBrainSources(
|
||||
const k = code as ParseValidationCode;
|
||||
totals[k] = (totals[k] ?? 0) + (n as number);
|
||||
}
|
||||
if (report.status === 'partial' && abortedAtSource === null) {
|
||||
abortedAtSource = src.id;
|
||||
}
|
||||
}
|
||||
|
||||
const hasPartialOrSkipped = perSource.some(r => r.status === 'partial' || r.status === 'skipped');
|
||||
|
||||
return {
|
||||
ok: grandTotal === 0,
|
||||
// Partial scans can never be 'ok' even when the scanned prefix is clean
|
||||
// (codex outside-voice C2 — a clean prefix doesn't speak for the
|
||||
// unscanned suffix).
|
||||
ok: grandTotal === 0 && !hasPartialOrSkipped,
|
||||
total: grandTotal,
|
||||
errors_by_code: totals,
|
||||
per_source: perSource,
|
||||
scanned_at: new Date().toISOString(),
|
||||
ignored_missing_open: ignoredMissingOpen || undefined,
|
||||
partial: hasPartialOrSkipped,
|
||||
aborted_at_source: abortedAtSource,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -425,9 +550,22 @@ function scanOneSource(
|
||||
let scanned = 0;
|
||||
let total = 0;
|
||||
let ignoredMissingOpen = 0;
|
||||
let interrupted = false;
|
||||
|
||||
walkDir(rootResolved, (absPath) => {
|
||||
if (opts.signal?.aborted) return false;
|
||||
// Per-file deadline + abort gate. Deadline is the load-bearing
|
||||
// wall-clock bound (sync I/O blocks the event loop so timer-based
|
||||
// AbortSignal.timeout can't fire mid-walk — codex C1).
|
||||
if (opts.deadline && Date.now() > opts.deadline) {
|
||||
interrupted = true;
|
||||
return false;
|
||||
}
|
||||
if (opts.signal?.aborted) {
|
||||
interrupted = true;
|
||||
return false;
|
||||
}
|
||||
// visitDir is consulted from walkDir directly (passed below). The
|
||||
// per-file visit closure doesn't need it.
|
||||
const relPath = relative(rootResolved, absPath);
|
||||
if (!isSyncable(relPath, { strategy: 'markdown' })) return true;
|
||||
scanned++;
|
||||
@@ -460,7 +598,7 @@ function scanOneSource(
|
||||
opts.onProgress.tick(50);
|
||||
}
|
||||
return true;
|
||||
});
|
||||
}, opts.visitDir);
|
||||
|
||||
if (opts.onProgress) {
|
||||
opts.onProgress.heartbeat(`scanned ${scanned} pages in ${sourceId}`);
|
||||
@@ -473,16 +611,49 @@ function scanOneSource(
|
||||
errors_by_code: errorsByCode,
|
||||
sample,
|
||||
ignoredMissingOpen,
|
||||
status: interrupted ? 'partial' : 'scanned',
|
||||
files_scanned: scanned,
|
||||
};
|
||||
}
|
||||
|
||||
/** Recursive directory walker with symlink-loop protection (via lstat).
|
||||
* Calls `visit` for each regular file. Returning false from `visit` stops
|
||||
* the walk. Skips entries lstat reports as symlinks (sync's no-symlink
|
||||
* policy). */
|
||||
function walkDir(root: string, visit: (absPath: string) => boolean | void): void {
|
||||
/**
|
||||
* Recursive directory walker with symlink-loop protection (via lstat) and
|
||||
* descent-time pruning of vendor / hidden / generated subtrees.
|
||||
*
|
||||
* Walks `root`, calling `visit(absPath)` for each regular file. Returning
|
||||
* `false` from `visit` stops the walk (used by scanOneSource for deadline
|
||||
* + abort).
|
||||
*
|
||||
* `pruneDir(name, parentDir)` is the canonical pruning gate (single source
|
||||
* of truth at src/core/sync.ts:258 — sync, extract, and transcript-discovery
|
||||
* all share it). Skipped subtrees: `node_modules`, `.git`, `.obsidian`,
|
||||
* `*.raw`, `ops`, all dot-prefix dirs, and git submodule dirs (`.git` as
|
||||
* FILE not DIRECTORY, the gitfile pattern from v0.37.7.0 #1169).
|
||||
*
|
||||
* Pre-v0.38.2.0 this walker descended into every subtree and let
|
||||
* `isSyncable` filter at the leaf — paying the IO cost of stat'ing hundreds
|
||||
* of thousands of vendor entries that were never going to be parsed. That
|
||||
* was the root cause of the `gbrain doctor` hang on 216K-page brains
|
||||
* reported in PR #1287.
|
||||
*
|
||||
* Optional `visitDir(dir)` is fired once per directory the walker decides
|
||||
* to DESCEND INTO (post-pruneDir, post-visited-set check). Production
|
||||
* callers don't pass it; tests use it to assert descent-time pruning
|
||||
* directly. Output-based tests pass under the original bug because
|
||||
* `isSyncable` filters at the leaf — `visitDir` is the load-bearing
|
||||
* observability hook for the regression suite.
|
||||
*/
|
||||
/** @internal — exported for the regression test suite (visitDir-based
|
||||
* descent-time pruning assertions). Production callers should go through
|
||||
* scanBrainSources / scanOneSource. */
|
||||
export function walkDir(
|
||||
root: string,
|
||||
visit: (absPath: string) => boolean | void,
|
||||
visitDir?: (dirPath: string) => void,
|
||||
): void {
|
||||
const stack: string[] = [root];
|
||||
const visited = new Set<string>();
|
||||
if (visitDir) visitDir(root);
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
@@ -501,9 +672,12 @@ function walkDir(root: string, visit: (absPath: string) => boolean | void): void
|
||||
}
|
||||
if (st.isSymbolicLink()) continue; // matches sync's no-symlink policy
|
||||
if (st.isDirectory()) {
|
||||
// Descent-time prune — the actual fix for the 216K-page hang.
|
||||
if (!pruneDir(name, dir)) continue;
|
||||
const real = resolve(full);
|
||||
if (visited.has(real)) continue;
|
||||
visited.add(real);
|
||||
if (visitDir) visitDir(full);
|
||||
stack.push(full);
|
||||
} else if (st.isFile()) {
|
||||
const result = visit(full);
|
||||
|
||||
196
test/brain-writer-partial-scan.test.ts
Normal file
196
test/brain-writer-partial-scan.test.ts
Normal file
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* v0.38.2.0 — partial-scan state tests for scanBrainSources.
|
||||
*
|
||||
* Codex outside-voice C1 caught that AbortSignal.timeout cannot interrupt
|
||||
* the sync walker (event loop blocked by readdirSync / readFileSync). The
|
||||
* load-bearing interruption mechanism is `deadline?: number` checked
|
||||
* inside scanOneSource's visit closure before parsing each file.
|
||||
*
|
||||
* These tests use `deadline: Date.now() - 1` (already-expired) to force
|
||||
* partial state deterministically — NOT AbortSignal, which doesn't fire
|
||||
* in the sync loop and would make this test flake or never trigger.
|
||||
*
|
||||
* They also cover codex C2 (`ok` after abort must be false even on clean
|
||||
* prefix), C4 (`files_scanned` numerator surfaced), and the
|
||||
* `aborted_at_source` field that lets doctor name the partial source.
|
||||
*/
|
||||
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { scanBrainSources } from '../src/core/brain-writer.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let sourceA: string;
|
||||
let sourceB: string;
|
||||
let sourceC: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Three source dirs, each with a few markdown files.
|
||||
sourceA = mkdtempSync(join(tmpdir(), 'partial-scan-a-'));
|
||||
sourceB = mkdtempSync(join(tmpdir(), 'partial-scan-b-'));
|
||||
sourceC = mkdtempSync(join(tmpdir(), 'partial-scan-c-'));
|
||||
for (const dir of [sourceA, sourceB, sourceC]) {
|
||||
mkdirSync(join(dir, 'people'), { recursive: true });
|
||||
for (let i = 0; i < 5; i++) {
|
||||
writeFileSync(
|
||||
join(dir, 'people', `p${i}.md`),
|
||||
`---\ntitle: Person ${i}\n---\n\nbody\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
for (const d of [sourceA, sourceB, sourceC]) {
|
||||
rmSync(d, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
// Register all three sources for each test.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('src-a', 'A', $1), ('src-b', 'B', $2), ('src-c', 'C', $3)`,
|
||||
[sourceA, sourceB, sourceC],
|
||||
);
|
||||
});
|
||||
|
||||
describe('scanBrainSources partial-scan state', () => {
|
||||
test('no deadline + no abort: every source scanned, partial=false, ok reflects grandTotal', async () => {
|
||||
const report = await scanBrainSources(engine);
|
||||
expect(report.partial).toBe(false);
|
||||
expect(report.aborted_at_source).toBe(null);
|
||||
expect(report.per_source.length).toBe(3);
|
||||
for (const src of report.per_source) {
|
||||
expect(src.status).toBe('scanned');
|
||||
expect(src.files_scanned).toBe(5);
|
||||
}
|
||||
expect(report.total).toBe(0);
|
||||
expect(report.ok).toBe(true);
|
||||
});
|
||||
|
||||
test('deadline expired before any source starts: all three skipped', async () => {
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: Date.now() - 1, // already expired
|
||||
});
|
||||
expect(report.partial).toBe(true);
|
||||
expect(report.per_source.length).toBe(3);
|
||||
for (const src of report.per_source) {
|
||||
expect(src.status).toBe('skipped');
|
||||
expect(src.files_scanned).toBe(0);
|
||||
}
|
||||
// ok must be false even though zero errors were found — partial state
|
||||
// means the clean count can't speak for unscanned files (codex C2).
|
||||
expect(report.ok).toBe(false);
|
||||
});
|
||||
|
||||
test('after-abort ok field is false even on clean prefix (codex C2 regression guard)', async () => {
|
||||
// Force the abort path: deadline already expired. Even though no
|
||||
// errors found (because no files scanned), `ok` must reflect the
|
||||
// partial-scan reality.
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: Date.now() - 1,
|
||||
});
|
||||
expect(report.total).toBe(0);
|
||||
expect(report.partial).toBe(true);
|
||||
expect(report.ok).toBe(false);
|
||||
});
|
||||
|
||||
test('files_scanned numerator populated on completed sources (codex C4 regression guard)', async () => {
|
||||
const report = await scanBrainSources(engine);
|
||||
for (const src of report.per_source) {
|
||||
// Each source has 5 .md files under people/; all syncable.
|
||||
expect(src.files_scanned).toBe(5);
|
||||
}
|
||||
});
|
||||
|
||||
test('dbPageCountForSource hook plumbed onto db_page_count; failure degrades to null', async () => {
|
||||
let calls = 0;
|
||||
const report = await scanBrainSources(engine, {
|
||||
dbPageCountForSource: async (sourceId) => {
|
||||
calls++;
|
||||
if (sourceId === 'src-b') throw new Error('synthetic query failure');
|
||||
return sourceId === 'src-a' ? 42 : 99;
|
||||
},
|
||||
});
|
||||
expect(calls).toBe(3);
|
||||
const a = report.per_source.find(r => r.source_id === 'src-a')!;
|
||||
const b = report.per_source.find(r => r.source_id === 'src-b')!;
|
||||
const c = report.per_source.find(r => r.source_id === 'src-c')!;
|
||||
expect(a.db_page_count).toBe(42);
|
||||
// Throw → null, no crash, scan continues.
|
||||
expect(b.db_page_count).toBe(null);
|
||||
expect(c.db_page_count).toBe(99);
|
||||
// files_scanned numerator still populated regardless of denominator outcome.
|
||||
expect(b.files_scanned).toBe(5);
|
||||
});
|
||||
|
||||
// Codex adversarial #3 regression: when the outer-loop deadline check fires
|
||||
// BEFORE any source starts, aborted_at_source MUST stamp the first
|
||||
// would-have-been-scanned source so the doctor message can name it.
|
||||
test('aborted_at_source stamped when deadline fires before any source starts', async () => {
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: Date.now() - 1,
|
||||
});
|
||||
expect(report.partial).toBe(true);
|
||||
// First source in deterministic order (sources ORDER BY id) is 'src-a'.
|
||||
expect(report.aborted_at_source).toBe('src-a');
|
||||
// Every source skipped, no scans started.
|
||||
expect(report.per_source.every(r => r.status === 'skipped')).toBe(true);
|
||||
});
|
||||
|
||||
// Codex adversarial #2 regression: a slow dbPageCountForSource that exceeds
|
||||
// the deadline must NOT result in scanOneSource running and reporting
|
||||
// status='partial' with files_scanned=0 (misleading — nothing was scanned).
|
||||
// The post-await deadline re-check should mark the source as 'skipped'.
|
||||
test('slow COUNT that exceeds deadline marks source skipped, not partial', async () => {
|
||||
const start = Date.now();
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: start + 50, // 50ms budget
|
||||
dbPageCountForSource: async () => {
|
||||
// Simulate a hung query: take 100ms (past the deadline).
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
return 42;
|
||||
},
|
||||
});
|
||||
// The first source should be skipped (post-await deadline re-check fires),
|
||||
// NOT marked partial with files_scanned=0.
|
||||
const firstSource = report.per_source.find(r => r.source_id === 'src-a')!;
|
||||
expect(firstSource.status).toBe('skipped');
|
||||
expect(firstSource.files_scanned).toBe(0);
|
||||
expect(report.partial).toBe(true);
|
||||
expect(report.aborted_at_source).toBe('src-a');
|
||||
});
|
||||
|
||||
// Codex adversarial #4 regression: even when dbPageCountForSource itself
|
||||
// would hang indefinitely, the Promise.race against the deadline must
|
||||
// resolve null and the scan must abort cleanly.
|
||||
test('hanging COUNT does not exceed deadline — Promise.race timeout fires', async () => {
|
||||
const start = Date.now();
|
||||
const report = await scanBrainSources(engine, {
|
||||
deadline: start + 100, // 100ms budget
|
||||
dbPageCountForSource: () => {
|
||||
// Never resolves — would hang forever without the deadline race.
|
||||
return new Promise<number | null>(() => {});
|
||||
},
|
||||
});
|
||||
const elapsed = Date.now() - start;
|
||||
// Generous bound: should complete within 2x the deadline budget (setup overhead).
|
||||
expect(elapsed).toBeLessThan(500);
|
||||
expect(report.partial).toBe(true);
|
||||
const firstSource = report.per_source.find(r => r.source_id === 'src-a')!;
|
||||
expect(firstSource.status).toBe('skipped');
|
||||
// Skipped sources never get db_page_count set — they weren't attempted.
|
||||
// (Either null from the race or undefined from never reaching the DB
|
||||
// path; both express "no denominator available" honestly.)
|
||||
expect(firstSource.db_page_count == null).toBe(true);
|
||||
});
|
||||
});
|
||||
150
test/brain-writer-walk-prune.test.ts
Normal file
150
test/brain-writer-walk-prune.test.ts
Normal file
@@ -0,0 +1,150 @@
|
||||
/**
|
||||
* v0.38.2.0 — descent-time pruning regression suite.
|
||||
*
|
||||
* The original bug (PR #1287 reported, this PR fixes): `gbrain doctor` hung
|
||||
* indefinitely on a 216K-page brain because the frontmatter walker descended
|
||||
* into every node_modules / .git / .obsidian / *.raw / ops subtree on disk
|
||||
* and let `isSyncable` filter at the leaf — paying the IO cost of stat'ing
|
||||
* hundreds of thousands of vendor entries that were never going to be parsed.
|
||||
*
|
||||
* Why output-based tests don't catch this: `isSyncable` rejects the
|
||||
* vendor-tree files at the leaf, so a test that just asserts "no bad
|
||||
* markdown reported" passes BOTH before and after Fix 1 (codex outside-voice
|
||||
* C6). The load-bearing assertion is `walker did NOT DESCEND` — fired by
|
||||
* the new `visitDir` test seam.
|
||||
*
|
||||
* This file covers both walkers that were missing pruneDir:
|
||||
* - brain-writer.ts:walkDir (driven by scanBrainSources / doctor)
|
||||
* - frontmatter.ts:collectFiles (driven by `gbrain frontmatter validate`)
|
||||
*/
|
||||
import { describe, expect, test, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { walkDir } from '../src/core/brain-writer.ts';
|
||||
import { collectFiles } from '../src/commands/frontmatter.ts';
|
||||
|
||||
let root: string;
|
||||
|
||||
beforeAll(() => {
|
||||
root = mkdtempSync(join(tmpdir(), 'walk-prune-'));
|
||||
// Real syncable files under regular dirs — walker MUST descend here.
|
||||
mkdirSync(join(root, 'people'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'alice.md'), '---\ntitle: Alice\n---\n\nbody\n');
|
||||
mkdirSync(join(root, 'concepts', 'subdir'), { recursive: true });
|
||||
writeFileSync(join(root, 'concepts', 'subdir', 'thing.md'), '---\ntitle: Thing\n---\n\nbody\n');
|
||||
// Vendor / hidden / generated trees — walker MUST NOT descend.
|
||||
mkdirSync(join(root, 'node_modules', 'fake-pkg'), { recursive: true });
|
||||
writeFileSync(join(root, 'node_modules', 'fake-pkg', 'README.md'), '# Should not be visited\n');
|
||||
mkdirSync(join(root, '.git', 'objects'), { recursive: true });
|
||||
writeFileSync(join(root, '.git', 'config'), '[core]\n');
|
||||
mkdirSync(join(root, '.obsidian'), { recursive: true });
|
||||
writeFileSync(join(root, '.obsidian', 'workspace.json'), '{}');
|
||||
mkdirSync(join(root, 'people', 'pedro.raw'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'pedro.raw', 'source.md'), '---\ntitle: should not visit\n---\n');
|
||||
mkdirSync(join(root, 'ops', 'logs'), { recursive: true });
|
||||
writeFileSync(join(root, 'ops', 'logs', 'run.md'), '# nope\n');
|
||||
// Nested node_modules — must also be pruned, not just at the root.
|
||||
mkdirSync(join(root, 'people', 'tools', 'node_modules', 'inner'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'tools', 'node_modules', 'inner', 'a.md'), '---\ntitle: nope\n---\n');
|
||||
// Git-submodule pattern: a dir containing `.git` as a FILE (gitfile).
|
||||
mkdirSync(join(root, 'people', 'submod'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'submod', '.git'), 'gitdir: ../../.git/modules/submod\n');
|
||||
writeFileSync(join(root, 'people', 'submod', 'README.md'), '---\ntitle: submod page\n---\n');
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('walkDir (brain-writer.ts) — descent-time pruning', () => {
|
||||
test('does NOT descend into node_modules at any depth', () => {
|
||||
const visited: string[] = [];
|
||||
walkDir(root, () => {}, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/node_modules'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into .git', () => {
|
||||
const visited: string[] = [];
|
||||
walkDir(root, () => {}, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/.git') || d.includes('/.git/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into .obsidian (dot-prefix heuristic)', () => {
|
||||
const visited: string[] = [];
|
||||
walkDir(root, () => {}, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/.obsidian'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into *.raw sidecar dirs', () => {
|
||||
const visited: string[] = [];
|
||||
walkDir(root, () => {}, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('.raw'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into git submodule directories (.git as FILE)', () => {
|
||||
const visited: string[] = [];
|
||||
walkDir(root, () => {}, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/people/submod'))).toBe(false);
|
||||
});
|
||||
|
||||
test('DOES descend into regular subdirs and visits .md files there', () => {
|
||||
const visited: string[] = [];
|
||||
const files: string[] = [];
|
||||
walkDir(root, (f) => { files.push(f); }, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/people'))).toBe(true);
|
||||
expect(visited.some(d => d.endsWith('/concepts/subdir'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true);
|
||||
// And explicitly does NOT visit the file under node_modules.
|
||||
expect(files.some(f => f.includes('/node_modules/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('regression: pre-v0.38.2.0 walker would have descended into node_modules and stat\'d every entry', () => {
|
||||
// This is the load-bearing assertion. If a future contributor removes
|
||||
// the `pruneDir(name, dir)` gate in walkDir, this test fails because
|
||||
// visitDir would be called with node_modules paths.
|
||||
const descents: string[] = [];
|
||||
walkDir(root, () => {}, (d) => descents.push(d));
|
||||
const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian|ops)(\/|$)/.test(d) || /\.raw$/.test(d));
|
||||
expect(vendor).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('collectFiles (frontmatter.ts) — descent-time pruning parity', () => {
|
||||
test('does NOT descend into node_modules at any depth', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/node_modules'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into .git, .obsidian, *.raw, or ops', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/.git'))).toBe(false);
|
||||
expect(visited.some(d => d.includes('/.obsidian'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('.raw'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into git submodule directories', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/people/submod'))).toBe(false);
|
||||
});
|
||||
|
||||
test('DOES collect .md files under regular subdirs', () => {
|
||||
const files = collectFiles(root);
|
||||
expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true);
|
||||
expect(files.some(f => f.includes('/node_modules/'))).toBe(false);
|
||||
expect(files.some(f => f.includes('/.git/'))).toBe(false);
|
||||
expect(files.some(f => f.includes('.raw/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('single-file target returns that file unchanged (no walk)', () => {
|
||||
const target = join(root, 'people', 'alice.md');
|
||||
const files = collectFiles(target);
|
||||
expect(files).toEqual([target]);
|
||||
});
|
||||
});
|
||||
@@ -293,7 +293,7 @@ describe('scanBrainSources (PGLite)', () => {
|
||||
expect(report.per_source[0]!.total).toBe(0);
|
||||
});
|
||||
|
||||
test('AbortSignal mid-scan stops walking', async () => {
|
||||
test('AbortSignal before scan: every source marked skipped (v0.38.2.0 partial-state contract)', async () => {
|
||||
const src = join(tmp, 'big');
|
||||
mkdirSync(src, { recursive: true });
|
||||
for (let i = 0; i < 50; i++) {
|
||||
@@ -303,7 +303,14 @@ describe('scanBrainSources (PGLite)', () => {
|
||||
const ctrl = new AbortController();
|
||||
ctrl.abort();
|
||||
const report = await scanBrainSources(engine, { signal: ctrl.signal });
|
||||
// Aborted before any source ran; per_source array stays empty (or has zero reports).
|
||||
expect(report.per_source.length).toBe(0);
|
||||
// v0.38.2.0 changed the contract: instead of an empty per_source array
|
||||
// (which hid the fact that sources weren't checked), the report now
|
||||
// includes a 'skipped' entry per source the outer loop never reached.
|
||||
// Doctor renders these as "NOT SCANNED" so the user knows.
|
||||
expect(report.per_source.length).toBe(1);
|
||||
expect(report.per_source[0].status).toBe('skipped');
|
||||
expect(report.per_source[0].files_scanned).toBe(0);
|
||||
expect(report.partial).toBe(true);
|
||||
expect(report.ok).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
84
test/doctor-frontmatter-partial.test.ts
Normal file
84
test/doctor-frontmatter-partial.test.ts
Normal file
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* v0.38.2.0 — doctor frontmatter_integrity rendering tests.
|
||||
*
|
||||
* Structural via source-grep. Behavioral coverage lives at two other
|
||||
* layers: (a) `test/brain-writer-partial-scan.test.ts` exercises the
|
||||
* scanBrainSources + deadline contract that doctor depends on, and
|
||||
* (b) `tests/heavy/frontmatter_scan_wallclock.sh` (manual / nightly)
|
||||
* subprocesses real `gbrain doctor` against a synthesized 60K-file brain.
|
||||
*
|
||||
* The unit layer here can't drive `runDoctor` directly because it calls
|
||||
* `process.exit(hasFail ? 1 : 0)` unconditionally, which terminates the
|
||||
* test runner. Refactoring runDoctor to return rather than exit is a
|
||||
* separate cleanup TODO (file: src/commands/doctor.ts:3885). Until then,
|
||||
* the source-grep tests below pin every load-bearing render string + the
|
||||
* codex D4 catch simplification.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
const DOCTOR_SOURCE = readFileSync(
|
||||
join(__dirname, '..', 'src', 'commands', 'doctor.ts'),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
describe('doctor frontmatter_integrity — structural rendering (source-grep)', () => {
|
||||
test('source contains GBRAIN_DOCTOR_FM_TIMEOUT_MS handling', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('GBRAIN_DOCTOR_FM_TIMEOUT_MS');
|
||||
});
|
||||
|
||||
test('source uses both deadline and AbortSignal.timeout (deadline is load-bearing per codex C1)', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('deadline: fmDeadline');
|
||||
expect(DOCTOR_SOURCE).toContain('AbortSignal.timeout(fmTimeoutMs)');
|
||||
});
|
||||
|
||||
test('source issues the DB COUNT(*) denominator query with deleted_at IS NULL', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('deleted_at IS NULL');
|
||||
expect(DOCTOR_SOURCE).toContain('FROM pages WHERE source_id');
|
||||
});
|
||||
|
||||
test('source renders honest "source has ~M pages in DB" wording', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('pages in DB');
|
||||
});
|
||||
|
||||
test('source renders NOT SCANNED per skipped source with remediation hint', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('NOT SCANNED');
|
||||
expect(DOCTOR_SOURCE).toContain('gbrain frontmatter validate');
|
||||
});
|
||||
|
||||
test('source has been simplified to remove the unreachable AbortError catch branch (codex D4)', () => {
|
||||
// The pre-v0.38.2.0 PR #1287 had a code-level branch:
|
||||
// const isTimeout = e instanceof DOMException && e.name === 'AbortError';
|
||||
// if (isTimeout) { ... }
|
||||
// Post-D4 there is no code-level isTimeout assignment OR DOMException
|
||||
// instanceof check. (Comments mentioning AbortError for explanation are
|
||||
// fine — only the code branch is the regression target.)
|
||||
expect(DOCTOR_SOURCE).not.toContain('const isTimeout');
|
||||
expect(DOCTOR_SOURCE).not.toContain("instanceof DOMException");
|
||||
});
|
||||
});
|
||||
|
||||
describe('doctor frontmatter_integrity — load-bearing render strings', () => {
|
||||
test('source includes "PARTIAL SCAN" wording for the warn message', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('PARTIAL SCAN');
|
||||
});
|
||||
|
||||
test('source includes "PARTIAL — scanned ~" per-source partial breakdown', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('PARTIAL — scanned ~');
|
||||
});
|
||||
|
||||
test('source threads files_scanned numerator into the render', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('src.files_scanned');
|
||||
});
|
||||
|
||||
test('source threads db_page_count denominator into the render', () => {
|
||||
expect(DOCTOR_SOURCE).toContain('src.db_page_count');
|
||||
});
|
||||
|
||||
test('source uses fallback hint pointing at GBRAIN_DOCTOR_FM_TIMEOUT_MS on partial', () => {
|
||||
// The fix hint when partial: raise the timeout OR run validate directly.
|
||||
const partialHintMatch = DOCTOR_SOURCE.includes('Raise GBRAIN_DOCTOR_FM_TIMEOUT_MS');
|
||||
expect(partialHintMatch).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -87,6 +87,8 @@ describe('v0.22.4 migration (B11)', () => {
|
||||
errors_by_code: { NESTED_QUOTES: 8 },
|
||||
sample: [],
|
||||
ignoredMissingOpen: 0,
|
||||
status: 'scanned' as const,
|
||||
files_scanned: 8,
|
||||
},
|
||||
{
|
||||
source_id: 'archive',
|
||||
@@ -95,6 +97,8 @@ describe('v0.22.4 migration (B11)', () => {
|
||||
errors_by_code: { NULL_BYTES: 4 },
|
||||
sample: [],
|
||||
ignoredMissingOpen: 0,
|
||||
status: 'scanned' as const,
|
||||
files_scanned: 4,
|
||||
},
|
||||
{
|
||||
source_id: 'clean-source',
|
||||
@@ -103,9 +107,13 @@ describe('v0.22.4 migration (B11)', () => {
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
ignoredMissingOpen: 0,
|
||||
status: 'scanned' as const,
|
||||
files_scanned: 10,
|
||||
},
|
||||
],
|
||||
scanned_at: new Date().toISOString(),
|
||||
partial: false,
|
||||
aborted_at_source: null,
|
||||
};
|
||||
const r = __testing.phaseCEmitTodo(
|
||||
{ yes: true, dryRun: false, noAutopilotInstall: true },
|
||||
|
||||
147
tests/heavy/frontmatter_scan_wallclock.sh
Executable file
147
tests/heavy/frontmatter_scan_wallclock.sh
Executable file
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env bash
|
||||
# tests/heavy/frontmatter_scan_wallclock.sh
|
||||
# Wall-clock smoke for the v0.38.2.0 doctor frontmatter-scan production fix.
|
||||
#
|
||||
# Reproduces the load case from PR #1287 (which this PR supersedes) at a
|
||||
# scale where the bug actually shows: a brain with N "real" markdown pages
|
||||
# under regular subdirs PLUS M dummy pages under node_modules/. Pre-v0.38.2.0
|
||||
# the walker descended into node_modules and paid the IO cost of stat'ing
|
||||
# every entry; with N+M ≈ 60K the scan would hang past doctor's default
|
||||
# 30s budget.
|
||||
#
|
||||
# Asserts post-v0.38.2.0:
|
||||
# 1. `gbrain doctor` exits 0 OR 1 (some warns OK), NOT killed by timeout.
|
||||
# 2. frontmatter_integrity status is `ok` (clean prefix, no partial), NOT `warn`.
|
||||
# 3. Total wall-clock under WALLCLOCK_BUDGET_S (default 15s).
|
||||
#
|
||||
# Why N+M=60K (not 200K like the original report): 60K is the smallest
|
||||
# fixture size where descent-into-node_modules adds at least 5s of wall-clock
|
||||
# (~50K extra stat calls on a modern SSD); above that, it scales linearly.
|
||||
# Beating 200K would be cleaner but takes ~30s to seed the fixture, which is
|
||||
# the line between "heavy test" and "torture test." If a future contributor
|
||||
# needs the larger fixture, just bump REAL_PAGES + NODE_MODULES_PAGES.
|
||||
#
|
||||
# Codex outside-voice C7 caught the original plan's 1500-file E2E budget:
|
||||
# at that scale the test passes BEFORE AND AFTER the fix, proving nothing.
|
||||
# This script's 60K-file budget is the minimum that catches the regression.
|
||||
#
|
||||
# Works on either PGLite (default, no DATABASE_URL required) or Postgres
|
||||
# (set DATABASE_URL to test the Postgres SQL paths).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
WALLCLOCK_BUDGET_S="${WALLCLOCK_BUDGET_S:-15}"
|
||||
REAL_PAGES="${REAL_PAGES:-10000}"
|
||||
NODE_MODULES_PAGES="${NODE_MODULES_PAGES:-50000}"
|
||||
|
||||
TS=$(date -u +%Y%m%d-%H%M%SZ)
|
||||
# Isolate from the developer's real ~/.gbrain. Each run uses a fresh tmpdir.
|
||||
TMP_GBRAIN_HOME=$(mktemp -d -t gbrain-fm-wallclock-home-XXXXXX)
|
||||
export GBRAIN_HOME="$TMP_GBRAIN_HOME"
|
||||
BRAIN_DIR=$(mktemp -d -t gbrain-fm-wallclock-brain-XXXXXX)
|
||||
LOG_DIR="$GBRAIN_HOME/audit"
|
||||
mkdir -p "$LOG_DIR"
|
||||
LOG="$LOG_DIR/heavy-frontmatter_scan_wallclock-$TS.log"
|
||||
SURFACE_LOG="${TMPDIR:-/tmp}/heavy-frontmatter_scan_wallclock-$TS.log"
|
||||
trap 'cp -f "$LOG" "$SURFACE_LOG" 2>/dev/null || true; rm -rf "$TMP_GBRAIN_HOME" "$BRAIN_DIR"' EXIT
|
||||
|
||||
echo "[fm_wallclock] log=$LOG"
|
||||
echo "[fm_wallclock] brain=$BRAIN_DIR (real_pages=$REAL_PAGES node_modules_pages=$NODE_MODULES_PAGES)"
|
||||
echo "[fm_wallclock] wallclock_budget=${WALLCLOCK_BUDGET_S}s"
|
||||
|
||||
# Step 1: seed the synthetic brain.
|
||||
echo "[fm_wallclock] seeding fixture..." | tee -a "$LOG"
|
||||
SEED_START=$SECONDS
|
||||
|
||||
mkdir -p "$BRAIN_DIR/people" "$BRAIN_DIR/concepts" "$BRAIN_DIR/node_modules/fake-pkg"
|
||||
|
||||
# Real syncable pages: split across two regular subdirs so the walker has
|
||||
# work to do on the legitimate side too.
|
||||
half_real=$((REAL_PAGES / 2))
|
||||
for i in $(seq 1 "$half_real"); do
|
||||
printf -- '---\ntitle: Person %s\n---\n\nbody\n' "$i" \
|
||||
> "$BRAIN_DIR/people/p$i.md"
|
||||
done
|
||||
remainder_real=$((REAL_PAGES - half_real))
|
||||
for i in $(seq 1 "$remainder_real"); do
|
||||
printf -- '---\ntitle: Concept %s\n---\n\nbody\n' "$i" \
|
||||
> "$BRAIN_DIR/concepts/c$i.md"
|
||||
done
|
||||
|
||||
# Vendor pages under node_modules/. Pre-v0.38.2.0 walker descended here and
|
||||
# stat'd every one. Post-fix the walker prunes at the node_modules boundary.
|
||||
for i in $(seq 1 "$NODE_MODULES_PAGES"); do
|
||||
printf '# README %s\n' "$i" > "$BRAIN_DIR/node_modules/fake-pkg/r$i.md"
|
||||
done
|
||||
SEED_ELAPSED=$((SECONDS - SEED_START))
|
||||
echo "[fm_wallclock] fixture seeded in ${SEED_ELAPSED}s" | tee -a "$LOG"
|
||||
|
||||
# Step 2: init brain + register the source.
|
||||
echo "[fm_wallclock] init brain..." | tee -a "$LOG"
|
||||
timeout 120s bun run src/cli.ts init --pglite --yes >> "$LOG" 2>&1 || {
|
||||
echo "[fm_wallclock] FAIL: gbrain init exited non-zero" >&2
|
||||
echo "Log tail:" >&2
|
||||
tail -30 "$LOG" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Register the brain dir as a source. Use raw SQL since `gbrain sources add`
|
||||
# might not exist in this version-window; the schema is what doctor reads.
|
||||
echo "[fm_wallclock] register source..." | tee -a "$LOG"
|
||||
bun run -e "
|
||||
import { PGLiteEngine } from './src/core/pglite-engine.ts';
|
||||
const e = new PGLiteEngine();
|
||||
await e.connect({});
|
||||
await e.initSchema();
|
||||
await e.executeRaw(
|
||||
\"INSERT INTO sources (id, name, local_path) VALUES ('fm-wallclock', 'Frontmatter wallclock test', \\\$1)\",
|
||||
['$BRAIN_DIR'],
|
||||
);
|
||||
await e.disconnect();
|
||||
console.log('source registered');
|
||||
" 2>&1 | tee -a "$LOG"
|
||||
|
||||
# Step 3: run gbrain doctor; capture wall-clock + exit + frontmatter_integrity status.
|
||||
echo "[fm_wallclock] running gbrain doctor (budget ${WALLCLOCK_BUDGET_S}s)..." | tee -a "$LOG"
|
||||
DOCTOR_START_NS=$(date +%s%N)
|
||||
set +e
|
||||
timeout "${WALLCLOCK_BUDGET_S}s" bun run src/cli.ts doctor --json > "$LOG.doctor" 2>>"$LOG"
|
||||
DOCTOR_RC=$?
|
||||
set -e
|
||||
DOCTOR_END_NS=$(date +%s%N)
|
||||
DOCTOR_MS=$(( (DOCTOR_END_NS - DOCTOR_START_NS) / 1000000 ))
|
||||
echo "[fm_wallclock] doctor exit=$DOCTOR_RC wallclock=${DOCTOR_MS}ms" | tee -a "$LOG"
|
||||
|
||||
# Step 4: assert.
|
||||
# RC 124 = `timeout` killed it. Other non-zero = doctor warns/fails, also FYI
|
||||
# but allowed for unrelated reasons. The load-bearing assertion is the
|
||||
# frontmatter_integrity status from the JSON.
|
||||
if [ "$DOCTOR_RC" = "124" ]; then
|
||||
echo "[fm_wallclock] FAIL: doctor exceeded ${WALLCLOCK_BUDGET_S}s budget — the v0.38.2.0 pruneDir wiring is broken" >&2
|
||||
echo " Log tail:" >&2
|
||||
tail -30 "$LOG" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
echo "[fm_wallclock] WARN: jq not installed; skipping JSON-shape assertion" >&2
|
||||
echo "[fm_wallclock] PASS (wall-clock check passed; JSON assertion skipped)" | tee -a "$LOG"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
FM_STATUS=$(jq -r '.checks[] | select(.name=="frontmatter_integrity") | .status' "$LOG.doctor")
|
||||
FM_MSG=$(jq -r '.checks[] | select(.name=="frontmatter_integrity") | .message' "$LOG.doctor")
|
||||
echo "[fm_wallclock] frontmatter_integrity: status=$FM_STATUS msg=$FM_MSG" | tee -a "$LOG"
|
||||
|
||||
if [ "$FM_STATUS" != "ok" ]; then
|
||||
# Pre-v0.38.2.0 would either timeout (caught above) or report PARTIAL when
|
||||
# the walker actually got into node_modules and ran out of budget. Either
|
||||
# is a regression.
|
||||
echo "[fm_wallclock] FAIL: frontmatter_integrity is not ok (got: $FM_STATUS)" >&2
|
||||
echo " Either the deadline fired (Fix 1 / pruneDir is broken) or the walk hit unexpected errors." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[fm_wallclock] PASS (doctor=${DOCTOR_MS}ms < ${WALLCLOCK_BUDGET_S}s budget, frontmatter_integrity=ok)" | tee -a "$LOG"
|
||||
Reference in New Issue
Block a user