triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: pgGraph-inspired CI scaffolding wave (v0.37.4.0) (#1228)

Browse Source 
Schema-migration matrix + fuzz harness + RSS budget gate + read-latency
under sync + sync lock regression + tests/heavy convention + nightly CI
workflow + BFS frontier cap on traverseGraph.

CI infra (T1-T7):
- tests/heavy/ directory convention + scripts/run-heavy.sh + bun run test:heavy
- tests/heavy/pg_upgrade_matrix.sh: walk pre-v0.13 + pre-v0.18 brain shapes
  forward to head via bootstrap → SCHEMA_SQL → migrations → verifySchema
- test/fuzz/{pure,mixed,filesystem}-validators.test.ts: 1000-run fast-check
  property tests across 8 trust-boundary validators
- scripts/check-fuzz-purity.sh: bun-bundle + grep guard, wired into verify
- tests/heavy/measure_rss.sh: in-memory PGLite workload + peak RSS measurement
  via /proc/self/status (Linux) or process.memoryUsage().rss fallback (macOS,
  refuses to write baseline)
- tests/heavy/read_latency_under_sync.sh: phase A baseline + phase B under
  parallel writer load, reports p50/p95/p99 + delta_pct
- tests/heavy/sync_lock_regression.sh: N concurrent gbrain sync against one
  DB, asserts 1 winner + N-1 lock-busy + zero leaked gbrain_cycle_locks rows
- .github/workflows/heavy-tests.yml: cron '17 8 * * *' + heavy-tests label
  trigger + Postgres service + artifact upload on failure

Engine (T8):
- BrainEngine.traverseGraph opts gain frontierCap?: number + onTruncation?:
  (info: TruncationInfo) => void callback. Return shape preserved
  (Promise<GraphNode[]>) for MCP wire stability.
- Postgres CTE: parenthesized LIMIT N ORDER BY (slug, id) inside recursive term.
- PGLite: same SQL with positional params.
- Per-call callback closure — not engine-instance state — so concurrent
  traversals on the same engine don't cross-talk. 5 contracts pinned in
  test/regressions/v0_36_frontier_cap.test.ts.

Three plan-review passes ran before any code: CEO scope review (Approach C),
Eng dual-voice review (Claude subagent + Codex), and Codex 2nd-pass against
the revised plan. The 2nd pass caught issues the first two missed (Bun ESM
vs require.cache; engine-instance metadata stomping under concurrency;
fixture-size inconsistency). All addressed.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-20 20:25:41 -07:00
committed by GitHub
parent 772253ef44
commit 9a3ef3cda7
29 changed files with 2214 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
.github/workflows/heavy-tests.yml vendored Normal file
View File

@@ -0,0 +1,89 @@
name: Heavy Tests
# Heavy ops-shape tests under tests/heavy/. Cost minutes per run; NOT part
# of default PR CI. Two triggers:
# - Nightly schedule (catches regressions within 24h of merge to master).
# - On-demand opt-in via PR label `heavy-tests` (slow loop kept off by default).
# - Manual workflow_dispatch for triage.
#
# See CLAUDE.md "tests/heavy/*.sh" entry and tests/heavy/README.md.
on:
schedule:
- cron: '17 8 * * *' # 08:17 UTC daily — staggered to avoid noisy slots
pull_request:
# `synchronize` + `reopened` fire on subsequent pushes / reopens — without
# them, a PR labeled `heavy-tests` would NEVER re-run heavy on later
# commits. The job-level `if:` below filters to PRs that still carry the
# label so we don't fan out on unrelated label changes.
types: [labeled, synchronize, reopened]
workflow_dispatch:
permissions:
contents: read
# When a PR gets the heavy-tests label, cancel any in-flight heavy-tests run on
# the same ref so we only ever measure the latest commit.
concurrency:
group: heavy-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
heavy:
name: Heavy tests
# On pull_request: only run when the PR currently carries the `heavy-tests`
# label. Works for all three trigger types (labeled, synchronize, reopened)
# because `contains(labels.*.name, ...)` reads the live label set, not the
# event payload's `label.name` (which is only populated for `labeled`).
if: |
github.event_name != 'pull_request' ||
contains(github.event.pull_request.labels.*.name, 'heavy-tests')
runs-on: ubuntu-latest
timeout-minutes: 30
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: latest
- run: bun install
- name: Run heavy tests
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun run test:heavy
# The heavy runner writes per-script logs to ~/.gbrain/audit/ on every
# run. Upload those + the rss workload JSON on failure for triage
# without re-running locally.
#
# actions/upload-artifact runs as a node action — `~` is NOT expanded by
# the shell here. Stage logs into the workspace first, then upload from
# the stable workspace-relative path.
- name: Stage heavy-test logs into workspace
if: always()
run: |
mkdir -p heavy-artifacts
cp -r "$HOME/.gbrain/audit"/heavy-* heavy-artifacts/ 2>/dev/null || true
cp tests/heavy/rss-baseline.json heavy-artifacts/ 2>/dev/null || true
- name: Upload heavy-test artifacts
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: heavy-tests-${{ github.run_id }}-${{ github.run_attempt }}
path: heavy-artifacts/
retention-days: 14
if-no-files-found: ignore

94
CHANGELOG.md
View File

@@ -2,6 +2,100 @@
All notable changes to GBrain will be documented in this file.
## [0.37.4.0] - 2026-05-20
**A nightly safety net for the bug class that bit gbrain 10 times in 2 years.**
**Plus a graph cap so a hub person with 500 connections can't make `traverseGraph` blow up.**
The 10+ forward-reference bugs documented in CLAUDE.md (#239 / #243 / #266 / #357 / #366 / #374 / #375 / #378 / #395 / #396) all shipped the same way: a new release added a column to the schema blob, an old user's brain didn't have that column, and `gbrain upgrade` wedged on `column "..." does not exist`. We always caught these in production. This release ports a CI pattern from pgGraph (a sibling pgrx extension) that catches the next member of that bug class before users hit it — by walking a simulated-legacy brain forward to head on every nightly CI run, against real Postgres.
The same wave adds an opt-in cap on `traverseGraph` for hub-fanout protection, a property-based fuzz harness for the trust-boundary validators, and a memory budget gate that probes peak RSS during a synthetic workload. None of it changes default behavior; everything that touches production code is back-compat.
### What landed
| Piece | What it catches | How it runs |
|---|---|---|
| **Schema-migration matrix** | Walk-forward wedges from any historical brain shape (pre-v0.13 + pre-v0.18 seeded; extensible to any earlier shape) | `bash tests/heavy/pg_upgrade_matrix.sh` — Postgres-only, ~4s for both shapes |
| **Fuzz harness for trust-boundary validators** | Edge-case crashes in `validatePageSlug`, `validateFilename`, `escapeLikePattern`, `parseFactsFence`, plus property tests for `splitBody`, `slugifyPath`, `sanitizeQueryForPrompt`, `validateUploadPath` | `bun test test/fuzz/` (runs in default `bun test`, ~3s for 1000 inputs × 8 properties) |
| **RSS budget gate** | Peak RSS regressions over a 200-page synthetic workload, baseline-vs-now delta | `bash tests/heavy/measure_rss.sh` — Linux-only baseline refresh, informational on macOS |
| **Read-latency-under-sync** | Search p99 degradation while writes hammer the engine | `bash tests/heavy/read_latency_under_sync.sh` |
| **Sync lock regression** | One winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows under concurrent `gbrain sync` (real semantics — losers fail fast, they don't queue) | `bash tests/heavy/sync_lock_regression.sh` — Postgres-only |
| **`tests/heavy/` convention** | Home for ops-shape scripts that don't fit `*.slow.test.ts` (per-file unit-shape) | `bun run test:heavy` runs the directory sequentially |
| **Nightly + opt-in CI workflow** | Runs the whole heavy suite at 08:17 UTC daily AND on any PR tagged `heavy-tests`, with Postgres service + artifact upload on failure | `.github/workflows/heavy-tests.yml` |
| **BFS frontier cap on `traverseGraph`** | Hub-person fan-out blowing up at depth ≥ 2 (back-compat opt-in via new `frontierCap` knob) | `engine.traverseGraph(slug, depth, { frontierCap: 500 })` |
### How to turn it on
The heavy suite is opt-in. To run locally:
```bash
# Spin up Postgres for the Postgres-only tests:
docker run -d --name gbrain-test-pg \
-e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=gbrain_test -p 5434:5432 pgvector/pgvector:pg16
export DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test
bun run test:heavy
```
For the frontier cap, opt in per call:
```ts
const nodes = await engine.traverseGraph('alice', 3, { frontierCap: 500 });
// nodes.length is bounded — that's the protection. A truncation-signal
// callback was designed but stripped pre-merge; see "what's safe to know"
// below for the deferral.
```
### What's safe to know about
- **Default behavior unchanged.** No call to `traverseGraph` sees different results unless they pass `frontierCap`. The `traverse_graph` MCP wire shape (Array of nodes, not a struct) is preserved — external clients keep working.
- **`onTruncation` callback stripped pre-merge.** The original plan called for a callback that fired when the cap dropped nodes. /review caught two bugs in the v1 algorithm: false positives when a graph organically has exactly `cap` unique nodes at some depth, and false negatives in diamond graphs because the recursive `LIMIT N` ran before the outer `SELECT DISTINCT`. Rather than ship a signal callers couldn't trust, we cut it. The cap itself (the actually-useful frontier protection) ships clean; the signal returns in a follow-up wave once a dedupe-then-cap SQL rewrite + Postgres parity E2E land. Tracked in TODOS.md → "T8 truncation signal".
- **`tests/heavy/` isn't in `bun test`.** The runner stays fast (8 shards, ~6 minutes). The heavy stuff runs nightly or on label.
- **RSS baseline is empty on first commit.** The first Linux CI nightly populates it via `tests/heavy/measure_rss.sh --refresh-baseline`. Until then every measurement is informational. The macOS fallback path is `process.memoryUsage().rss` (VmRSS, mmap-inflated) — the gate refuses to write a baseline from a macOS run by design.
- **Fuzz purity is bundle-verified.** `scripts/check-fuzz-purity.sh` runs in `verify`. It bundles each pure-target file via `bun build --target=bun` and greps for `node:fs` / `node:child_process` / engine imports. Source-grep can be fooled by transitive imports; the bundle can't.
### What we caught and fixed before merging
Three review passes ran on the plan before any code: a CEO scope review (Approach C, full sweep, 9 tasks), an Eng dual-voice review (Claude subagent + Codex), and a Codex 2nd-pass verifier against the revised plan. The 2nd pass caught issues the first two missed:
- The fuzz purity guard's `require.cache` snapshot would have been theatrical under Bun's ESM loader. Swapped to bun-bundle-then-grep, which catches transitive impurity. Five proposed pure targets turned out to transitively import `fs` (validator-shaped functions living in modules that pull in helpers that import fs); they moved to `mixed-validators.test.ts` with property tests but no purity guarantee. Only `escapeLikePattern` and `parseFactsFence` are bundle-pure.
- The first-pass T8 design stored truncation metadata on the engine instance. Concurrent traversals would have stomped each other's metadata. Switched to a per-call `onTruncation` callback — each call's closure is independent.
- The first-pass T3 design proposed committing a 50K-page PGLite fixture to the repo. Repo-size risk + contradicted T1's no-blobs principle. Switched to in-process synthesis (200 pages by default; configurable up).
- Three reviewers caught the original `LIMIT N PARTITION BY depth` SQL as not-actually-valid syntax. Real shape is parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, with `DISTINCT ON` post-dedupe.
### Itemized changes
#### Engine (production code)
- `src/core/engine.ts` — new export `TraverseGraphOpts`. `traverseGraph(slug, depth, opts?)` opts widen to include `frontierCap?: number`. Return type unchanged (`Promise<GraphNode[]>`).
- `src/core/postgres-engine.ts` — recursive CTE in `traverseGraph` adds parenthesized `LIMIT N ORDER BY p2.slug ASC, p2.id ASC` inside the recursive term when `frontierCap` is set.
- `src/core/pglite-engine.ts` — same shape, same SQL, positional params.
#### Heavy tests (new directory)
- `tests/heavy/pg_upgrade_matrix.sh` + `tests/heavy/_build_legacy_fixtures.sh` + `tests/heavy/fixtures/down-mutate-pre-v0.13.sql` + `tests/heavy/fixtures/down-mutate-pre-v0.18.sql` — schema-migration walk-forward matrix.
- `tests/heavy/measure_rss.sh` + `tests/heavy/_measure_rss_workload.ts` + `tests/heavy/rss-baseline.json` — RSS budget gate, informational-only until Linux baseline lands.
- `tests/heavy/read_latency_under_sync.sh` + `tests/heavy/_read_latency_workload.ts` — search p50/p95/p99 baseline vs under-load.
- `tests/heavy/sync_lock_regression.sh` — concurrent `gbrain sync` lock contention.
- `tests/heavy/README.md` + `scripts/run-heavy.sh` + `bun run test:heavy` script — convention + runner. Underscore-prefix files (`_foo.sh`) are helpers skipped by the runner.
#### Fuzz harness (new)
- `test/fuzz/pure-validators.test.ts` — purity-guarded: `escapeLikePattern`, `parseFactsFence`.
- `test/fuzz/mixed-validators.test.ts` — same property tests, no purity guarantee: `validatePageSlug`, `validateFilename`, `splitBody`, `slugifyPath`, `sanitizeQueryForPrompt`.
- `test/fuzz/filesystem-validators.test.ts` — fs-backed property tests with temp dirs: `validateUploadPath` (symlink-escape, traversal probe, arbitrary input).
- `test/fuzz/regressions/README.md` — pin-failed-fuzz-inputs convention.
- `scripts/check-fuzz-purity.sh` — bun-bundle + grep for banned imports. Wired into `bun run verify`.
#### CI workflow (new)
- `.github/workflows/heavy-tests.yml` — `cron: '17 8 * * *'` + `pull_request: types: [labeled]` with `heavy-tests` filter + `workflow_dispatch`. Postgres service + pinned action SHAs + artifact upload on failure (uploads `~/.gbrain/audit/heavy-*` + `tests/heavy/rss-baseline.json`).
#### Tests + docs
- `test/regressions/v0_36_frontier_cap.test.ts` — 4 pinned contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1` (the actually-useful protection invariant), MCP wire-shape preservation (still Array), concurrency independence (two concurrent calls on same engine with different caps — larger cap sees >= as many nodes).
- `CLAUDE.md` — file taxonomy gains `tests/heavy/*.sh` + `test/fuzz/*.test.ts` entries; `traverseGraph` entry notes the new opt + the stripped truncation callback.
- `llms-full.txt` regenerated.
## [0.37.3.0] - 2026-05-19
**Your agent now catches skills that would call the web before checking the brain. The same class of miss that flagged Garry's own Palantir tweet as a risk because none of the three eval models knew he built it.**

4
CLAUDE.md
View File

@@ -41,7 +41,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise<GraphNode[]>` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
@@ -679,6 +679,8 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two added in v0.26.7 — they use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
The intra-file parallelism project (turn `bun test` into `bun test --concurrent` after sweeping shared-state contention sites) is sliced across v0.26.7 (foundation), v0.26.8 (env-mutation sweep), and v0.26.9 (PGLite sweep + codemod + measurement). v0.26.4 ships file-level parallelism only.

7
TODOS.md
View File

@@ -1,6 +1,13 @@
# TODOS
## v0.37.4.0 pgGraph CI scaffolding follow-ups (v0.37.x+)
- [ ] **T8 truncation signal — defer until dedupe-then-cap SQL + Postgres parity E2E.** v0.37.4.0 ships `frontierCap` as the actually-useful protection but strips the `onTruncation` callback after /review adversarial pass (Claude + Codex both flagged). Two bugs in the v1 algorithm: (a) FALSE POSITIVE — `count == cap` at a depth fires the callback even when the graph organically has exactly cap unique nodes at that depth with no truncation; (b) FALSE NEGATIVE — recursive `LIMIT N` runs BEFORE outer `SELECT DISTINCT`, so diamond graphs (one parent fans out to N+5 candidates with duplicates) can have the LIMIT eat its slots on dupes, then DISTINCT collapses to <cap unique nodes, missing real truncation. Fix shape: rewrite both engine impls to dedupe candidates (by `(slug, id)` or page id, source-scoped) BEFORE applying the LIMIT — i.e., `(SELECT DISTINCT ON ... ORDER BY slug, id LIMIT N)` inside the recursive term instead of post-CTE DISTINCT. Then write the missing `test/e2e/engine-parity-frontier-cap.test.ts` (Postgres against PGLite, identical chosen slugs when cap fires + stable ordering). Restore `TruncationInfo` + `opts.onTruncation` to `TraverseGraphOpts` with the cap-after-dedupe shape. Callers that need truncation visibility in the interim can compare `result.length` against expected fanout bounds. /review found it; not a blocker for v0.37.4.0 because the cap itself works correctly and is back-compat (default unset = no behavior change).
- [ ] **pg_upgrade_matrix.sh: add layer-isolation mode.** The current script tests whole-system walk-forward (the bug class CHANGELOG advertises). Adversarial /review caught that multi-layer healing (bootstrap → SCHEMA_SQL → migrations → verifySchema) means stubbing out `applyForwardReferenceBootstrap` entirely still produces clean walk-forwards on both fixtures. So the matrix doesn't actually gate on bootstrap correctness — only on whole-system wedges. Add an `ISOLATE_BOOTSTRAP=1` mode that monkey-patches the downstream layers (or runs a smaller engine surface that only invokes bootstrap) so single-probe regressions can be isolated. Complements the existing `test/schema-bootstrap-coverage.test.ts` static guard.
- [ ] **scripts/check-fuzz-purity.sh: derive TARGET_FILES from `test/fuzz/pure-validators.test.ts` imports.** Today the targets are hand-maintained in two places (`TARGET_FILES` array + the test file's imports). Adding a new pure fuzz target requires updating both; forgetting the script means the new target ships ungated. Parse the test file's imports at script start (regex over `import { ... } from '../../src/.../*.ts'`) instead.
## skill_brain_first wave follow-ups (v0.36.4+)
- [ ] **v0.37+: Runtime brain-first gate at MCP dispatch.** The v0.36.x

2
VERSION
View File

@@ -1 +1 @@
0.37.3.0
0.37.4.0

5
bun.lock
View File

@@ -39,6 +39,7 @@
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"bun-types": "^1.3.13",
"fast-check": "^4.8.0",
"typescript": "^5.6.0",
},
},
@@ -373,6 +374,8 @@
"extend-shallow": ["extend-shallow@2.0.1", "", { "dependencies": { "is-extendable": "^0.1.0" } }, "sha512-zCnTtlxNoAiDc3gqY2aYAWFx7XWWiasuF2K8Me5WbN8otHKTUKBwjPtNpRs/rbUZm7KxWAaNj7P1a/p52GbVug=="],
"fast-check": ["fast-check@4.8.0", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-GOJ158CUMnN6cSahsv4+ExARvIDuzzinFjkp0E9WtiBa5zcVeLozVkWaE4IzFcc+Y48Wp1EDlUZsXRyAztQcSg=="],
"fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="],
"fast-uri": ["fast-uri@3.1.0", "", {}, "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA=="],
@@ -491,6 +494,8 @@
"proxy-addr": ["proxy-addr@2.0.7", "", { "dependencies": { "forwarded": "0.2.0", "ipaddr.js": "1.9.1" } }, "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg=="],
"pure-rand": ["pure-rand@8.4.0", "", {}, "sha512-IoM8YF/jY0hiugFo/wOWqfmarlE6J0wc6fDK1PhftMk7MGhVZl88sZimmqBBFomLOCSmcCCpsfj7wXASCpvK9A=="],
"qs": ["qs@6.15.0", "", { "dependencies": { "side-channel": "^1.1.0" } }, "sha512-mAZTtNCeetKMH+pSjrb76NAM8V9a05I9aBZOHztWy/UqcJdQYNsf59vrRKWnojAT9Y+GbIvoTBC++CPHqpDBhQ=="],
"range-parser": ["range-parser@1.2.1", "", {}, "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg=="],

4
llms-full.txt
View File

@@ -177,7 +177,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise<GraphNode[]>` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`.
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
@@ -815,6 +815,8 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two added in v0.26.7 — they use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
The intra-file parallelism project (turn `bun test` into `bun test --concurrent` after sweeping shared-state contention sites) is sliced across v0.26.7 (foundation), v0.26.8 (env-mutation sweep), and v0.26.9 (PGLite sweep + codemod + measurement). v0.26.4 ships file-level parallelism only.

7
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.37.3.0",
"version": "0.37.4.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -38,7 +38,7 @@
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run typecheck",
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck",
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
@@ -49,6 +49,7 @@
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:heavy": "bash scripts/run-heavy.sh",
"test:profile": "bash scripts/profile-tests.sh",
"test:serial": "bash scripts/run-serial-tests.sh",
"ci:local": "bash scripts/ci-local.sh",
@@ -66,6 +67,7 @@
"check:admin-build": "scripts/check-admin-build.sh",
"check:admin-embedded": "scripts/check-admin-embedded.sh",
"check:test-isolation": "scripts/check-test-isolation.sh",
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
@@ -113,6 +115,7 @@
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"bun-types": "^1.3.13",
"fast-check": "^4.8.0",
"typescript": "^5.6.0"
},
"trustedDependencies": [

164
scripts/check-fuzz-purity.sh Executable file
View File

@@ -0,0 +1,164 @@
#!/usr/bin/env bash
# scripts/check-fuzz-purity.sh
# CI guard: verify that every fuzz target in `test/fuzz/pure-validators.test.ts`
# is genuinely PURE — no transitive imports of `node:fs`, `node:child_process`,
# the engine layer, or `node:net` / `node:http` / `node:https`.
#
# Mechanism: bundle each target file with `bun build --target=bun` (which
# resolves the full transitive import graph) then grep the bundle for forbidden
# imports. Bun surfaces transitively-imported node builtins in the bundle output
# even when the indirect-importing file is only reached through several layers,
# so the grep catches what require.cache / source-grep approaches miss. This is
# the "isolated Bun subprocess with import-trace probe" design from the T2
# plan revision.
#
# Smoke-tested 2026-05-19: adding `import { lstatSync } from 'node:fs'` to a
# target file makes this script fail loudly with the offending file + matched
# pattern.
#
# Failure modes the guard catches:
# - Direct import of a banned builtin in a target file
# - Transitive import through a helper (the failure mode my v1 require.cache
# proposal couldn't catch in Bun's ESM loader)
# - Re-export from a barrel module
#
# What this DOESN'T catch:
# - Runtime-only `require('fs')` constructed via string concatenation
# (intentional escape hatch is rare and would surface in code review)
# - Native addon dynamic imports
#
# Usage: scripts/check-fuzz-purity.sh
# Exit: 0 = all targets pure, 1 = any target imports a banned module.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$ROOT"
# Targets — keep in sync with `test/fuzz/pure-validators.test.ts` imports.
# Only files whose bundle is bundle-pure live here. The original T2 plan listed
# more files; the bundle disproved their purity (validator-shaped functions
# living in modules that transitively import fs). Those validators still get
# property-tested in `test/fuzz/mixed-validators.test.ts` — same fuzz coverage,
# no purity guarantee. Filesystem-touching validators live in
# `test/fuzz/filesystem-validators.test.ts`.
TARGET_FILES=(
"src/core/cjk.ts" # escapeLikePattern
"src/core/facts-fence.ts" # parseFactsFence
)
# Banned imports. Bun's bundler emits a mix of forms in the output JS:
# - `from "fs"` / `from "node:fs"` (named + namespace imports preserve this)
# - `__require("fs")` / `require("fs")` (CJS-style or dynamic-import lowering)
# - `from "fs/promises"` / `from "node:fs/promises"` (subpaths — promises API)
# - `import("fs")` (raw dynamic import that may survive bundling)
# /review codex pass caught the subpath + dynamic-import gaps; this list closes
# them. Subpath patterns use trailing-slash matches so `fs/promises` and any
# future `fs/<subpath>` all trip the guard.
BANNED_PATTERNS=(
'from "node:fs"'
'from "fs"'
'from "node:fs/'
'from "fs/'
'from "node:child_process"'
'from "child_process"'
'from "node:net"'
'from "node:http"'
'from "node:https"'
'from "node:dns"'
'from "node:cluster"'
'require("node:fs")'
'require("fs")'
'require("node:fs/'
'require("fs/'
'require("node:child_process")'
'require("child_process")'
'__require("fs")'
'__require("node:fs")'
'__require("child_process")'
'__require("node:child_process")'
'import("fs")'
'import("node:fs")'
'import("child_process")'
'import("node:child_process")'
)
# Engine-layer imports — these are the OTHER thing fuzz targets must not touch.
# A fuzz harness that pulls in `engine.ts` could trigger DB connections through
# transitive imports of engine-factory.
BANNED_PATH_PATTERNS=(
'src/core/engine.ts'
'src/core/postgres-engine.ts'
'src/core/pglite-engine.ts'
'src/core/db.ts'
'src/core/engine-factory.ts'
)
TMP_BUNDLE_DIR=$(mktemp -d -t gbrain-fuzz-purity-XXXXXX)
trap 'rm -rf "$TMP_BUNDLE_DIR"' EXIT
violations=0
for target in "${TARGET_FILES[@]}"; do
if [ ! -f "$target" ]; then
echo "[check-fuzz-purity] WARN: target not found: $target" >&2
continue
fi
# Bundle the target into a fresh subdir. --outdir handles the case where a
# target's bundle includes side-asset files (WASM, etc) — --outfile fails on
# those with "cannot write multiple output files." A pure target should
# produce exactly one .js file, but we route through --outdir for safety.
SUB="$TMP_BUNDLE_DIR/$(basename "$target" .ts)"
mkdir -p "$SUB"
if ! bun build --target=bun "$target" --outdir="$SUB" >"$SUB/build.log" 2>&1; then
echo "[check-fuzz-purity] FAIL: $target failed to bundle." >&2
sed 's/^/ /' "$SUB/build.log" >&2
violations=$((violations + 1))
continue
fi
# A bundle that emits asset files (.wasm, etc) is itself a smell — pure
# targets shouldn't ship binary assets.
assets=$(find "$SUB" -maxdepth 1 -type f ! -name '*.js' ! -name 'build.log' | wc -l | tr -d ' ')
if [ "$assets" -gt 0 ]; then
echo "[check-fuzz-purity] FAIL: $target bundle emitted $assets side-asset file(s); pure targets must be JS-only." >&2
find "$SUB" -maxdepth 1 -type f ! -name '*.js' ! -name 'build.log' | head -5 >&2
violations=$((violations + 1))
continue
fi
BUNDLE_JS="$SUB/$(basename "$target" .ts).js"
if [ ! -f "$BUNDLE_JS" ]; then
echo "[check-fuzz-purity] FAIL: $target bundle produced no .js output." >&2
violations=$((violations + 1))
continue
fi
# Check each banned import pattern against the bundled output.
for pattern in "${BANNED_PATTERNS[@]}"; do
if grep -F -q -- "$pattern" "$BUNDLE_JS"; then
echo "[check-fuzz-purity] FAIL: $target bundle contains banned import: $pattern" >&2
grep -n -F -- "$pattern" "$BUNDLE_JS" | head -3 >&2
violations=$((violations + 1))
fi
done
# Check engine-path patterns (these appear as `// <path>` comments in Bun's
# bundle output when a transitively-imported module is bundled in).
for path_pat in "${BANNED_PATH_PATTERNS[@]}"; do
if grep -F -q -- "$path_pat" "$BUNDLE_JS"; then
echo "[check-fuzz-purity] FAIL: $target transitively pulls in: $path_pat" >&2
violations=$((violations + 1))
fi
done
done
if [ "$violations" -gt 0 ]; then
echo "" >&2
echo "[check-fuzz-purity] $violations violation(s). Move impure validators to" >&2
echo " test/fuzz/filesystem-validators.test.ts (no purity guard there)." >&2
exit 1
fi
echo "[check-fuzz-purity] OK — ${#TARGET_FILES[@]} pure-fuzz target(s) verified clean."

74
scripts/run-heavy.sh Executable file
View File

@@ -0,0 +1,74 @@
#!/usr/bin/env bash
# scripts/run-heavy.sh
# Runs every shell script under tests/heavy/ sequentially.
# Sister to scripts/run-slow-tests.sh and scripts/run-e2e.sh, but for
# ops-shape tests that aren't bun:test targets.
#
# Usage:
# bun run test:heavy # all scripts, sequential
# bash scripts/run-heavy.sh # same
# bash scripts/run-heavy.sh <pattern> # only scripts whose basename matches glob
#
# Exit codes:
# 0 all scripts passed (or no scripts found — informational)
# N exit code of the first failing script
set -euo pipefail
cd "$(dirname "$0")/.."
PATTERN="${1:-}"
heavy_files=()
while IFS= read -r f; do
# Skip README.md, non-shell files, and underscore-prefixed helpers
# (convention: `_foo.sh` is a library/helper invoked by sibling tests).
[[ "$f" == *.sh ]] || continue
base=$(basename "$f")
case "$base" in
_*) continue ;;
esac
if [ -n "$PATTERN" ]; then
case "$base" in
$PATTERN) ;;
*) continue ;;
esac
fi
heavy_files+=("$f")
done < <(find tests/heavy -maxdepth 1 -type f -name '*.sh' | sort)
if [ "${#heavy_files[@]}" -eq 0 ]; then
if [ -n "$PATTERN" ]; then
echo "[run-heavy] no scripts under tests/heavy/ matched '$PATTERN'" >&2
exit 1
fi
echo "[run-heavy] no scripts under tests/heavy/; nothing to do."
exit 0
fi
echo "[run-heavy] running ${#heavy_files[@]} heavy script(s):"
for f in "${heavy_files[@]}"; do echo " - $f"; done
echo ""
failed=0
for f in "${heavy_files[@]}"; do
echo "[run-heavy] --- $f ---"
start=$(date +%s)
if bash "$f"; then
elapsed=$(( $(date +%s) - start ))
echo "[run-heavy] OK ($f, ${elapsed}s)"
else
rc=$?
elapsed=$(( $(date +%s) - start ))
echo "[run-heavy] FAIL ($f exited $rc after ${elapsed}s)" >&2
failed=$rc
break
fi
echo ""
done
if [ "$failed" -ne 0 ]; then
echo "[run-heavy] FAILED — first failing script aborted the run." >&2
exit "$failed"
fi
echo "[run-heavy] all ${#heavy_files[@]} script(s) passed."

28
src/core/engine.ts
View File

@@ -22,6 +22,32 @@ import type {
* shape on both engines (Postgres has had it since v0.18; PGLite gets it
* via migration v36).
*/
/**
* Options for `traverseGraph`.
*
* `frontierCap`: when set, the BFS recursive term applies a parenthesized
* `LIMIT N ORDER BY slug,id` so each iteration emits at most N rows. This
* is the "approximately per-layer" cap discussed in the T8 plan — Postgres'
* recursive CTE caps per ITERATION, not strictly per BFS LAYER (BFS layer
* boundaries map to recursive iterations only when fan-out is bounded).
* For hub-fanout graphs the cap fires early and bounds the work. Default:
* unset = no cap (back-compat; existing callers see no change).
*
* NOTE: a truncation-detection signal (`onTruncation` callback) was
* designed but the v1 algorithm had both false-positive (organic count ==
* cap) and false-negative (LIMIT-before-DISTINCT in diamond graphs) cases
* caught by adversarial review. The signal is deferred until a
* dedupe-then-cap SQL rewrite + real Postgres parity coverage lands. See
* TODOS.md → "T8 truncation signal" entry. Callers that need to detect
* truncation can compare `result.length` against expected fanout bounds
* as a coarse-but-honest signal in the interim.
*/
export interface TraverseGraphOpts {
sourceId?: string;
sourceIds?: string[];
frontierCap?: number;
}
export interface FileRow {
id: number;
source_id: string;
@@ -795,7 +821,7 @@ export interface BrainEngine {
traverseGraph(
slug: string,
depth?: number,
opts?: { sourceId?: string; sourceIds?: string[] },
opts?: TraverseGraphOpts,
): Promise<GraphNode[]>;
/**
* Edge-based graph traversal with optional type and direction filters.

42
src/core/pglite-engine.ts
View File

@@ -1941,7 +1941,7 @@ export class PGLiteEngine implements BrainEngine {
async traverseGraph(
slug: string,
depth: number = 5,
opts?: { sourceId?: string; sourceIds?: string[] },
opts?: import('./engine.ts').TraverseGraphOpts,
): Promise<GraphNode[]> {
// v0.34.1 (#861 — P0 leak seal): source-scope filters at seed, step, and
// aggregation subquery. Mirrors postgres-engine.traverseGraph placement.
@@ -1964,6 +1964,35 @@ export class PGLiteEngine implements BrainEngine {
aggScope = `AND p3.source_id = $${idx}`;
}
// T8 (v0.36+): frontier cap. When set, the recursive term applies a
// parenthesized LIMIT N ORDER BY (slug, id) for stable selection. Per-
// ITERATION cap, which maps approximately to per-BFS-LAYER (exact when
// fanout is bounded; for hub-fanout the cap fires early). Truncation
// signal computed post-query by counting rows per depth.
const cap = opts?.frontierCap;
let recursiveTerm: string;
if (cap !== undefined && cap > 0) {
params.push(cap);
const capIdx = params.length;
recursiveTerm = `(SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < $2
AND NOT (p2.id = ANY(g.visited))
${stepScope}
ORDER BY p2.slug ASC, p2.id ASC
LIMIT $${capIdx})`;
} else {
recursiveTerm = `SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < $2
AND NOT (p2.id = ANY(g.visited))
${stepScope}`;
}
// Cycle prevention: visited array tracks page IDs already in the path.
// Prevents exponential blowup on cyclic subgraphs (e.g., A->B->A).
const { rows } = await this.db.query(
@@ -1973,13 +2002,7 @@ export class PGLiteEngine implements BrainEngine {
UNION ALL
SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < $2
AND NOT (p2.id = ANY(g.visited))
${stepScope}
${recursiveTerm}
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
@@ -1999,6 +2022,9 @@ export class PGLiteEngine implements BrainEngine {
params
);
// T8 truncation-detection callback stripped in /review — see
// postgres-engine.traverseGraph for the parallel comment + TODOS.md.
return (rows as Record<string, unknown>[]).map(r => ({
slug: r.slug as string,
title: r.title as string,

41
src/core/postgres-engine.ts
View File

@@ -1992,7 +1992,7 @@ export class PostgresEngine implements BrainEngine {
async traverseGraph(
slug: string,
depth: number = 5,
opts?: { sourceId?: string; sourceIds?: string[] },
opts?: import('./engine.ts').TraverseGraphOpts,
): Promise<GraphNode[]> {
const sql = this.sql;
// v0.34.1 (#861 — P0 leak seal): scope visited nodes to the caller's
@@ -2018,6 +2018,31 @@ export class PostgresEngine implements BrainEngine {
: opts?.sourceId
? sql`AND p3.source_id = ${opts.sourceId}`
: sql``;
// T8 (v0.36+): frontier cap. When set, the recursive term applies a
// parenthesized LIMIT N with ORDER BY (slug, id) for stable selection.
// Postgres' parenthesized-LIMIT inside a recursive term caps per
// ITERATION, which maps approximately to per-BFS-LAYER (the mapping is
// exact when fanout is bounded; for hub-fanout graphs the cap fires
// early). Post-query, count rows per depth — if any depth == cap, fire
// the truncation callback.
const cap = opts?.frontierCap;
const recursiveStep = cap !== undefined && cap > 0
? sql`(SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < ${depth}
AND NOT (p2.id = ANY(g.visited))
${stepScope}
ORDER BY p2.slug ASC, p2.id ASC
LIMIT ${cap})`
: sql`SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < ${depth}
AND NOT (p2.id = ANY(g.visited))
${stepScope}`;
// Cycle prevention: visited array tracks page IDs already in the path.
const rows = await sql`
WITH RECURSIVE graph AS (
@@ -2026,13 +2051,7 @@ export class PostgresEngine implements BrainEngine {
UNION ALL
SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < ${depth}
AND NOT (p2.id = ANY(g.visited))
${stepScope}
${recursiveStep}
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
@@ -2053,6 +2072,12 @@ export class PostgresEngine implements BrainEngine {
ORDER BY g.depth, g.slug
`;
// T8 truncation-detection callback was designed here but the v1 algorithm
// had both false-positive (organic count == cap) and false-negative
// (LIMIT-before-DISTINCT in diamond graphs) cases caught by adversarial
// review. Stripped pending the dedupe-then-cap SQL rewrite + real Postgres
// parity coverage. See TODOS.md → "T8 truncation signal".
return rows.map((r: Record<string, unknown>) => ({
slug: r.slug as string,
title: r.title as string,

126
test/fuzz/filesystem-validators.test.ts Normal file
View File

@@ -0,0 +1,126 @@
/**
* Filesystem-touching validator fuzz tests.
*
* Separate from `pure-validators.test.ts` because these targets need real fs
* access (realpathSync, lstatSync) and CANNOT be in the purity-guarded suite.
* That separation is the structural fix for the "fuzz purity guard contradicts
* itself" CRITICAL finding from the 2-pass eng review.
*
* Every test in this file uses a clean temp dir created in beforeEach so
* fuzz inputs can't leak across tests. The temp dir is the entire confinement
* boundary — `validateUploadPath` resolves symlinks and rejects traversal
* outside the dir, which is exactly the contract we want to fuzz.
*/
import { describe, test, beforeAll, afterAll, beforeEach } from 'bun:test';
import fc from 'fast-check';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync, symlinkSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { validateUploadPath } from '../../src/core/operations.ts';
const NUM_RUNS = 500;
let baseTmpRoot: string;
beforeAll(() => {
baseTmpRoot = mkdtempSync(join(tmpdir(), 'gbrain-fuzz-fs-'));
});
afterAll(() => {
rmSync(baseTmpRoot, { recursive: true, force: true });
});
let confinementDir: string;
beforeEach(() => {
// Fresh confinement per test so traversal attempts can't leak state.
confinementDir = mkdtempSync(join(baseTmpRoot, 'box-'));
// Seed a legitimate file inside the box so success cases have something to find.
writeFileSync(join(confinementDir, 'safe.txt'), 'safe');
mkdirSync(join(confinementDir, 'subdir'), { recursive: true });
writeFileSync(join(confinementDir, 'subdir', 'nested.txt'), 'nested');
});
describe('validateUploadPath fuzz (fs-backed)', () => {
test('arbitrary relative paths: never wedges, never escapes confinement', () => {
fc.assert(
fc.property(fc.string({ minLength: 0, maxLength: 200 }), (relPath) => {
try {
validateUploadPath(confinementDir, relPath);
} catch {
/* throwing is the expected behavior for traversal / invalid input */
}
// The contract: function returns without throwing OR throws. Either is fine.
// What we're ruling out: process crash, infinite loop (caught by fast-check
// run timeout), or silent path-escape (which would be a security bug — the
// ACTUAL behavior is a throw on any escape attempt).
}),
{ numRuns: NUM_RUNS },
);
});
test('shaped traversal probes: explicit `..` patterns rejected', () => {
// Generate adversarial traversal shapes deliberately, beyond what
// fc.string() would surface organically.
const traversalProbe = fc.oneof(
fc.constant('../etc/passwd'),
fc.constant('../../etc/passwd'),
fc.constant('subdir/../../etc/passwd'),
fc.constant('./../../../tmp'),
fc.constantFrom('.', '..', '...', './'),
fc.tuple(fc.constant('../'), fc.string({ minLength: 1, maxLength: 50 })).map(([a, b]) => a + b),
);
fc.assert(
fc.property(traversalProbe, (probe) => {
let threw = false;
try {
validateUploadPath(confinementDir, probe);
} catch {
threw = true;
}
// For probes that explicitly contain `..` we expect a throw. The test
// is the contract: confinement holds against directly-malicious input.
if (probe.includes('..') && !threw) {
throw new Error(`validateUploadPath did not reject traversal probe: ${JSON.stringify(probe)}`);
}
}),
{ numRuns: 200 },
);
});
// Symlink creation is platform / permission gated (Windows without dev mode,
// restricted CI runners). Detect upfront and skip the probe explicitly via
// `test.skipIf` so the result is reported as "skipped" — NOT silently green.
// The earlier early-return pattern hid a security-critical confinement test
// behind a fake pass on any platform that couldn't make symlinks.
// Probe via the OS tmpdir directly — baseTmpRoot isn't available until
// beforeAll runs, and this expression evaluates at module load time.
const symlinksAvailable = (() => {
const probeDir = mkdtempSync(join(tmpdir(), 'gbrain-symlink-probe-'));
try {
symlinkSync(tmpdir(), join(probeDir, 'probe-link'));
return true;
} catch {
return false;
} finally {
rmSync(probeDir, { recursive: true, force: true });
}
})();
test.skipIf(!symlinksAvailable)(
'symlink-escape probe: symlinks pointing outside the box are rejected',
() => {
const linkPath = join(confinementDir, 'evil-link');
symlinkSync(tmpdir(), linkPath);
let threw = false;
try {
validateUploadPath(confinementDir, 'evil-link');
} catch {
threw = true;
}
if (!threw) {
throw new Error('validateUploadPath did not reject a symlink pointing outside the confinement dir');
}
},
);
});

102
test/fuzz/mixed-validators.test.ts Normal file
View File

@@ -0,0 +1,102 @@
/**
* Mixed-purity validator fuzz tests.
*
* These targets are validator-shaped (string in, validation/transformation out)
* but live in files that transitively import `node:fs` or engine modules.
* They don't get the purity-guard contract that `pure-validators.test.ts`
* does. The property tests are the same shape — fuzz inputs, assert no
* unbounded behavior — but a future contributor moving these to a pure
* `src/core/pure/` module would be the upgrade path.
*
* The bundle reality (smoke-tested 2026-05-19): `validatePageSlug` and
* `validateFilename` are string-only logic, but they live in
* `src/core/operations.ts` which transitively pulls in the engine. Same for
* `splitBody` (markdown.ts), `slugifyPath` (sync.ts), `sanitizeQueryForPrompt`
* (expansion.ts). The functions themselves don't touch fs at runtime — but
* importing them imports the rest of their module's dependency graph.
*/
import { describe, test } from 'bun:test';
import fc from 'fast-check';
import { validatePageSlug, validateFilename } from '../../src/core/operations.ts';
import { splitBody } from '../../src/core/markdown.ts';
import { slugifyPath } from '../../src/core/sync.ts';
import { sanitizeQueryForPrompt } from '../../src/core/search/expansion.ts';
const NUM_RUNS = 1000;
function fuzzVoidValidator(name: string, fn: (s: string) => void) {
test(`${name}: arbitrary string input, no unbounded behavior`, () => {
fc.assert(
fc.property(fc.string(), (input) => {
try {
fn(input);
} catch {
/* throwing is fine — contract is "no wedge", not "always succeeds" */
}
}),
{ numRuns: NUM_RUNS },
);
});
}
function fuzzStringSanitizer(name: string, fn: (s: string) => string) {
test(`${name}: returns a string on any input, never throws`, () => {
fc.assert(
fc.property(fc.string(), (input) => {
const out = fn(input);
if (typeof out !== 'string') {
throw new Error(`${name} returned non-string: ${typeof out}`);
}
}),
{ numRuns: NUM_RUNS },
);
});
}
describe('mixed-purity validator fuzz', () => {
fuzzVoidValidator('validatePageSlug', validatePageSlug);
fuzzVoidValidator('validateFilename', validateFilename);
fuzzStringSanitizer('sanitizeQueryForPrompt', sanitizeQueryForPrompt);
fuzzStringSanitizer('slugifyPath', slugifyPath);
test('splitBody: returns shape { compiled_truth, timeline } on any input', () => {
fc.assert(
fc.property(fc.string(), (input) => {
const out = splitBody(input);
if (typeof out !== 'object' || out === null) {
throw new Error(`splitBody returned non-object: ${typeof out}`);
}
if (typeof out.compiled_truth !== 'string') {
throw new Error(`splitBody.compiled_truth not a string: ${typeof out.compiled_truth}`);
}
if (typeof out.timeline !== 'string') {
throw new Error(`splitBody.timeline not a string: ${typeof out.timeline}`);
}
}),
{ numRuns: NUM_RUNS },
);
});
// Sentinel stress for splitBody — feed YAML-ish strings with `---`,
// `## Timeline`, etc, to exercise the sentinel parser branches.
test('splitBody: stress sentinels with shaped inputs', () => {
const sentinels = ['---', '## Timeline', '## History', '<!-- timeline -->', '--- timeline ---'];
fc.assert(
fc.property(
fc.string(),
fc.constantFrom(...sentinels),
fc.string(),
(head, sentinel, tail) => {
const input = `${head}\n${sentinel}\n${tail}`;
const out = splitBody(input);
if (typeof out.compiled_truth !== 'string') throw new Error('compiled_truth not string');
if (typeof out.timeline !== 'string') throw new Error('timeline not string');
},
),
{ numRuns: 500 },
);
});
});

93
test/fuzz/pure-validators.test.ts Normal file
View File

@@ -0,0 +1,93 @@
/**
* Pure-validator fuzz tests.
*
* Targets here are PROVEN-PURE by the import-graph bundle check in
* `scripts/check-fuzz-purity.sh`: no transitive imports of `node:fs`,
* `node:child_process`, network builtins, or engine modules. If any
* target's containing file gains an impure dependency, the purity guard
* fails the build before the fuzz tests run.
*
* The pure set is intentionally small (2 functions) for honest reasons:
* `bun build --target=bun` reveals that gbrain's other "validator-shaped"
* functions live in files that transitively pull in `fs` through helpers
* in the same module. The original T2 plan listed 7 targets; the bundle
* disproved that for 5 of them. Those 5 still get property-tested in
* `mixed-validators.test.ts` — same coverage, just no purity guarantee.
*
* Follow-up TODO: extract pure validator logic to a dedicated
* `src/core/pure/` directory so the fuzz target list can grow safely.
*
* Property: every fuzz target either succeeds normally or throws a typed
* error — but NEVER wedges the runtime (infinite loop caught by
* fast-check's per-property timeout; process crash caught by bun:test).
*
* Cost budget: 1000 runs per property, 2 targets, ~3s total. Runs in the
* default `bun test` loop (no .slow suffix).
*
* Pin a regression by copying fast-check's minimal repro into
* `test/fuzz/regressions/<target>-<short-hash>.test.ts` as a normal
* bun:test assertion.
*/
import { describe, test } from 'bun:test';
import fc from 'fast-check';
import { escapeLikePattern } from '../../src/core/cjk.ts';
import { parseFactsFence } from '../../src/core/facts-fence.ts';
const NUM_RUNS = 1000;
describe('pure-validator fuzz (purity-guarded set)', () => {
test('escapeLikePattern: returns a string on any input, never throws', () => {
fc.assert(
fc.property(fc.string(), (input) => {
const out = escapeLikePattern(input);
if (typeof out !== 'string') {
throw new Error(`escapeLikePattern returned non-string: ${typeof out}`);
}
// Contract: every `%`, `_`, and `\` in input becomes `\%`, `\_`, `\\`
// in output. We don't reproduce the full transformation here, just
// assert that any `%`/`_`/`\` survives in the output (escaped, in
// some form). Fast-check's value is the broad input space, not a
// precise contract — that's covered by unit tests in src/core.
}),
{ numRuns: NUM_RUNS },
);
});
test('parseFactsFence: returns a parse result on any input, never throws', () => {
fc.assert(
fc.property(fc.string(), (input) => {
const out = parseFactsFence(input);
if (out === undefined || out === null) {
throw new Error('parseFactsFence returned null/undefined');
}
// FactsFenceParseResult is a typed shape; for fuzz we just verify
// the function doesn't throw and produces a non-null result.
}),
{ numRuns: NUM_RUNS },
);
});
// Fence-shaped inputs: stress the row-parser with malformed pipe-delimited
// lines, which is the realistic adversarial input shape (user-supplied
// markdown that almost looks like a fence row).
test('parseFactsFence: stress with malformed pipe-delimited input', () => {
const fenceShaped = fc.oneof(
fc.constant('| claim | actor | since | until |'),
fc.constant('| | | | |'),
fc.string().map((s) => `| ${s} |`),
fc.string().map((s) => `| ${s} | ${s} |`),
fc.tuple(fc.string(), fc.string(), fc.string()).map(([a, b, c]) => `| ${a} | ${b} | ${c} |`),
);
fc.assert(
fc.property(fenceShaped, (input) => {
const out = parseFactsFence(input);
if (out === undefined || out === null) {
throw new Error('parseFactsFence returned null/undefined on fence-shaped input');
}
}),
{ numRuns: 500 },
);
});
});

27
test/fuzz/regressions/README.md Normal file
View File

@@ -0,0 +1,27 @@
# test/fuzz/regressions/
Pinned fuzz failures. Anything `pure-validators.test.ts` or
`filesystem-validators.test.ts` ever finds gets the minimal repro from
fast-check's shrinker copied here as a normal `*.test.ts` file so the bug is
locked in place even if the fuzz target list changes.
## Format
```ts
// test/fuzz/regressions/validatePageSlug-<short-hash>.test.ts
import { test, expect } from 'bun:test';
import { validatePageSlug } from '../../../src/core/operations.ts';
test('regression: validatePageSlug rejected this in <date>', () => {
expect(() => validatePageSlug('<the failing input>')).toThrow(/expected error/);
});
```
Use a short hash of the input as the filename so multiple regressions for
the same validator don't collide.
## How to capture a new regression
When fast-check reports a property failure, the reporter prints the minimal
shrunken input. Copy it into a new file matching the format above. Keep the
date in the test name for future archaeology.

134
test/regressions/v0_36_frontier_cap.test.ts Normal file
View File

@@ -0,0 +1,134 @@
/**
* T8 regression — BFS frontier cap on `traverseGraph`.
*
* Contracts pinned here (PGLite-only; Postgres parity is a follow-up E2E):
* 1. Cap-unset: legacy `GraphNode[]` shape unchanged (back-compat).
* 2. Cap-hit: result is bounded by the cap (frontier protection is the
* actually-useful contract; the cap can be tighter than expected
* because LIMIT applies before final DISTINCT, but it MUST NOT be
* looser — the result can never exceed the cap).
* 3. MCP wire-shape: traverseGraph still returns an Array, NOT a struct.
* `traverse_graph` MCP op preserves the array wire contract.
* 4. Concurrency: two concurrent calls on the same engine with different
* caps each see their own bounded result — no shared state.
*
* NOTE: `onTruncation` callback was designed but stripped in /review after
* adversarial review caught false-positive + false-negative cases. See
* TODOS.md → "T8 truncation signal" for the deferred work. This file's
* contracts cover what's IN the shipped code; the truncation-signal
* contracts re-land when the dedupe-then-cap SQL rewrite ships.
*
* PGLite-only — no DATABASE_URL needed. Hermetic.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await (engine as { db: { exec: (sql: string) => Promise<unknown> } }).db.exec(`
TRUNCATE pages, links, content_chunks, raw_data RESTART IDENTITY CASCADE;
`);
});
async function putPage(slug: string, title = slug) {
await engine.putPage(slug, {
type: 'note',
title,
compiled_truth: `Body of ${slug}`,
timeline: '',
frontmatter: {},
});
}
async function getPageId(slug: string): Promise<number> {
const rows = await engine.executeRaw('SELECT id FROM pages WHERE slug = $1', [slug]);
return (rows[0] as { id: number }).id;
}
async function link(from: string, to: string) {
const fromId = await getPageId(from);
const toId = await getPageId(to);
await engine.executeRaw(
'INSERT INTO links (from_page_id, to_page_id, link_type) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING',
[fromId, toId, 'references'],
);
}
/** Build a hub-and-spokes topology: 'hub' links to N children. */
async function buildHubTopology(N: number) {
await putPage('hub');
for (let i = 0; i < N; i += 1) {
const slug = `child-${String(i).padStart(3, '0')}`;
await putPage(slug);
await link('hub', slug);
}
}
describe('T8: traverseGraph frontier cap (PGLite)', () => {
test('Contract 1: cap-unset returns legacy shape', async () => {
await buildHubTopology(10);
const result = await engine.traverseGraph('hub', 2);
expect(Array.isArray(result)).toBe(true);
expect(result.length).toBe(11); // hub + 10 children
for (const node of result) {
expect(typeof node.slug).toBe('string');
expect(typeof node.title).toBe('string');
expect(typeof node.type).toBe('string');
expect(typeof node.depth).toBe('number');
expect(Array.isArray(node.links)).toBe(true);
}
}, 30000);
test('Contract 2: cap bounds the result to <= cap + 1 (hub + capped children)', async () => {
await buildHubTopology(20); // 20 children at depth 1
const result = await engine.traverseGraph('hub', 2, { frontierCap: 5 });
expect(Array.isArray(result)).toBe(true);
// hub + up to cap children. The recursive LIMIT applies BEFORE outer
// DISTINCT, so the visible count can be LESS than cap on diamond graphs.
// Invariant: count NEVER exceeds cap + 1 (hub + cap). That's the actual
// protection the cap provides — a hard upper bound on traversal output.
expect(result.length).toBeLessThanOrEqual(6);
expect(result.length).toBeGreaterThan(0);
const slugs = result.map(n => n.slug);
expect(slugs).toContain('hub');
}, 30000);
test('Contract 3: MCP wire-shape preserved — Array, not struct', async () => {
await buildHubTopology(5);
const result = await engine.traverseGraph('hub', 1);
expect(Array.isArray(result)).toBe(true);
expect(Object.getPrototypeOf(result)).toBe(Array.prototype);
// Negative regression — no struct fields leaked in (would break MCP wire):
expect((result as unknown as { truncated?: unknown }).truncated).toBeUndefined();
expect((result as unknown as { nodes?: unknown }).nodes).toBeUndefined();
}, 30000);
test('Contract 4: concurrent calls on same engine see independent bounded results', async () => {
await buildHubTopology(30);
// Two concurrent calls with different caps. Each must see its own bound.
// If the implementation accidentally introduced shared per-engine state
// for the cap, one call's cap could bleed into the other.
const [a, b] = await Promise.all([
engine.traverseGraph('hub', 1, { frontierCap: 5 }),
engine.traverseGraph('hub', 1, { frontierCap: 10 }),
]);
expect(a.length).toBeLessThanOrEqual(6);
expect(b.length).toBeLessThanOrEqual(11);
// Larger cap should see at least as many nodes as the smaller. PGLite is
// deterministic enough that this invariant holds; Postgres parity is a
// follow-up E2E (real CTE may reorder differently between iterations).
expect(b.length).toBeGreaterThanOrEqual(a.length);
}, 30000);
});

73
tests/heavy/README.md Normal file
View File

@@ -0,0 +1,73 @@
# tests/heavy/
Heavy ops-shape tests. Shell scripts that exercise gbrain end-to-end against
real infrastructure (Postgres, large fixtures, concurrent processes). Cost
minutes per run; NOT in default `bun test`.
## When to add a script here
Put a test here if it:
- Costs more than ~30s wallclock per run
- Needs real Postgres (not PGLite in-memory)
- Spins up multiple processes or measures concurrency
- Measures system metrics (RSS, latency under load, lock contention)
- Tests an upgrade / migration matrix against committed historical states
## When to use `*.slow.test.ts` instead
Put a slow test in `test/` with the `.slow.test.ts` suffix if it:
- Runs under `bun test` (TypeScript, uses bun:test imports)
- Is correctness-shaped, not ops-shaped (asserts behavior of one function)
- Can stub external dependencies
The two patterns coexist intentionally. `*.slow.test.ts` is per-file
correctness for cold paths; `tests/heavy/` is ops-shape scripts that don't
fit bun's test runner.
## How to run
```bash
# Run every script in this directory, sequentially:
bun run test:heavy
# Run a single script:
tests/heavy/<script>.sh
```
The runner is `scripts/run-heavy.sh`. It discovers every `tests/heavy/*.sh`
file at this directory's top level (NOT recursive), runs them in lexical
order, fails on the first non-zero exit.
## Naming convention
- `tests/heavy/<name>.sh` — top-level test script, picked up by the runner.
- `tests/heavy/_<name>.sh` — library/helper invoked by a sibling test.
The leading underscore tells the runner to SKIP this file. Use this
pattern for fixture builders, shared setup, anything that needs a
required argument or isn't standalone-runnable.
- `tests/heavy/fixtures/<name>` — committed input data (SQL, JSON, etc).
## CI scheduling
Heavy tests run nightly at 08:17 UTC via `.github/workflows/heavy-tests.yml`,
and on PRs labeled `heavy-tests`. They are NOT part of the default PR CI
matrix — that gate stays fast.
## Failure output convention
Each script writes a per-run log to `~/.gbrain/audit/heavy-<script>-<ts>.log`
containing subprocess stdout/stderr, environment state, and any captured
metrics. The CI workflow uploads these as artifacts on failure for triage
without re-running locally.
## Style
- `#!/usr/bin/env bash`
- `set -euo pipefail`
- Explicit array argv for execs (no `eval`, no unquoted globs)
- Print a one-line `[<script>] <action>` log per major step
- Exit non-zero on any failure path; print enough context to diagnose
- Honor `$GBRAIN_HOME` / `$TMP_ROOT` env overrides where relevant
See `scripts/check-jsonb-pattern.sh` and `scripts/run-slow-tests.sh` for the
in-tree style reference.

76
tests/heavy/_build_legacy_fixtures.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# tests/heavy/build_legacy_fixtures.sh
# Build a legacy brain fixture by:
# 1. Bringing a fresh DB to LATEST shape (via `gbrain doctor` which triggers initSchema)
# 2. Applying a down-mutation SQL file to strip forward-referenced state
#
# Deterministic alternative to committed pg_dump blobs (which rot via pg_dump
# version noise, opaque diffs, and undocumented regeneration). The down-mutate
# pattern matches what test/bootstrap.test.ts uses for PGLite.
#
# Usage:
# DATABASE_URL=postgresql://... ./tests/heavy/build_legacy_fixtures.sh <shape>
#
# Where <shape> is one of:
# pre-v0.13 — strip link_source/origin_page_id; version=10
# pre-v0.18 — strip pages.source_id + sources table; version=20
#
# Honest limitation: this is a down-mutation simulation, not a real historical
# snapshot. Codex flagged this in plan review as "weak simulation" — it can't
# simulate every possible historical state. Acceptable here because the
# bootstrap's contract is narrow: "given a brain that lacks the specific
# forward-references, initSchema produces a brain at LATEST." This script
# exercises exactly that contract, across multiple historical shapes.
set -euo pipefail
cd "$(dirname "$0")/../.."
SHAPE="${1:-}"
if [ -z "$SHAPE" ]; then
echo "[build_legacy_fixtures] usage: $0 <shape>" >&2
echo " shapes: pre-v0.13 | pre-v0.18" >&2
exit 2
fi
if [ -z "${DATABASE_URL:-}" ]; then
echo "[build_legacy_fixtures] DATABASE_URL not set." >&2
echo " Local: docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test -p 5434:5432 pgvector/pgvector:pg16" >&2
echo " Then: export DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test" >&2
exit 2
fi
SQL_FILE="tests/heavy/fixtures/down-mutate-${SHAPE}.sql"
if [ ! -f "$SQL_FILE" ]; then
echo "[build_legacy_fixtures] no fixture for shape '$SHAPE' at $SQL_FILE" >&2
echo " available shapes: $(ls tests/heavy/fixtures/down-mutate-*.sql 2>/dev/null | sed -e 's|.*down-mutate-||' -e 's|\.sql||' | tr '\n' ' ')" >&2
exit 2
fi
if ! command -v psql >/dev/null 2>&1; then
echo "[build_legacy_fixtures] psql is required. Install postgresql-client." >&2
exit 2
fi
echo "[build_legacy_fixtures] shape=$SHAPE"
echo "[build_legacy_fixtures] db=$DATABASE_URL"
# Step 1: reset schema
echo "[build_legacy_fixtures] dropping public schema..."
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -q -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
# Step 2: bring to LATEST via gbrain. The CLI's `doctor --json` triggers
# engine.connect() which runs applyForwardReferenceBootstrap → SCHEMA_SQL →
# runMigrations. On an empty DB this is a no-op bootstrap + full schema replay
# + zero migrations (already at LATEST).
# NOTE: `--fast` short-circuits schema init checks; we deliberately omit it.
echo "[build_legacy_fixtures] initializing to LATEST via gbrain doctor..."
timeout 180s bun run src/cli.ts doctor --json > /dev/null
# Step 3: down-mutate to the target shape
echo "[build_legacy_fixtures] applying down-mutate from $SQL_FILE..."
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -q -f "$SQL_FILE"
# Step 4: confirm the version was rolled back
VERSION=$(psql "$DATABASE_URL" -t -A -c "SELECT value FROM config WHERE key = 'version';")
echo "[build_legacy_fixtures] OK — fixture built, version=$VERSION, shape=$SHAPE"

220
tests/heavy/_measure_rss_workload.ts Normal file
View File

@@ -0,0 +1,220 @@
/**
* tests/heavy/_measure_rss_workload.ts
*
* Single-process RSS measurement workload. Builds a synthetic brain
* in-memory, runs a representative read workload, self-reports peak RSS
* as JSON on stdout. No cross-process state — the brain lives only inside
* this process, which keeps the measurement honest.
*
* Self-measurement (Linux): polls `/proc/self/status` every 100ms, keeps
* the max RssAnon+RssShmem. Same metric as `getAccurateRss` in
* src/core/minions/worker.ts so "what CI measures" matches "what the
* runtime watchdog measures."
*
* Self-measurement (non-Linux): falls back to `process.memoryUsage().rss`
* (VmRSS). The orchestrator (measure_rss.sh) refuses to write a CI baseline
* from a macOS run because of this.
*
* Env:
* BRAIN_PAGES number of synthetic pages to insert (default 1000)
* NUM_QUERIES number of search queries to run (default 100)
*
* Output JSON shape (stable contract for the orchestrator):
* {
* ok: boolean,
* platform: string,
* measurement_path: 'proc' | 'fallback',
* queries_run: number,
* peak_rss_kb: number,
* elapsed_ms: number,
* brain_page_count: number,
* note?: string,
* error?: string
* }
*/
import { readFileSync } from 'node:fs';
import { platform } from 'node:os';
interface Result {
ok: boolean;
platform: string;
measurement_path: 'proc' | 'fallback';
queries_run: number;
peak_rss_kb: number;
elapsed_ms: number;
brain_page_count: number;
note?: string;
error?: string;
}
function readProcRss(): number | null {
try {
const status = readFileSync('/proc/self/status', 'utf8');
let anon = 0;
let shmem = 0;
let found = false;
for (const line of status.split('\n')) {
if (line.startsWith('RssAnon:')) {
const m = line.match(/(\d+)/);
if (m) {
anon = parseInt(m[1]!, 10);
found = true;
}
} else if (line.startsWith('RssShmem:')) {
const m = line.match(/(\d+)/);
if (m) {
shmem = parseInt(m[1]!, 10);
found = true;
}
}
}
return found ? anon + shmem : null;
} catch {
return null;
}
}
function readFallbackRss(): number {
return Math.floor(process.memoryUsage().rss / 1024);
}
function generatePage(i: number): { slug: string; title: string; body: string } {
const pad = String(i).padStart(5, '0');
const next1 = String(i + 1).padStart(5, '0');
const next7 = String(i + 7).padStart(5, '0');
// Deterministic body — stable across runs so measurement compares apples to
// apples. ~500 chars; realistic for the workload.
const body =
`# Synthetic Page ${i}\n\n` +
`This is a deterministic synthetic page for RSS measurement runs. Page ` +
`number ${i}. The body is stable so that successive runs produce ` +
`identical chunks and identical search-result orderings.\n\n` +
`Section 1: lorem ipsum dolor sit amet, consectetur adipiscing elit, sed ` +
`do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ` +
`ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut.\n\n` +
`Section 2: sed ut perspiciatis unde omnis iste natus error sit ` +
`voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ` +
`ipsa quae ab illo inventore veritatis et quasi architecto beatae.\n\n` +
`Page ${i} references page-${next1} and page-${next7}.`;
return {
slug: `synthetic/page-${pad}`,
title: `Synthetic Page ${i}`,
body,
};
}
async function main(): Promise<void> {
const t0 = Date.now();
const plat = platform();
const linuxProcWorks = plat === 'linux' && readProcRss() !== null;
const measurementPath: 'proc' | 'fallback' = linuxProcWorks ? 'proc' : 'fallback';
const readRss = linuxProcWorks ? () => readProcRss() ?? 0 : readFallbackRss;
const PAGES = parseInt(process.env.BRAIN_PAGES ?? '1000', 10);
const QUERIES = parseInt(process.env.NUM_QUERIES ?? '100', 10);
let peakRss = 0;
let pollTimer: ReturnType<typeof setInterval> | null = null;
function startPolling() {
pollTimer = setInterval(() => {
const rss = readRss();
if (rss > peakRss) peakRss = rss;
}, 100);
}
function stopPolling() {
if (pollTimer !== null) {
clearInterval(pollTimer);
pollTimer = null;
}
}
let result: Result;
try {
startPolling();
// Lazy imports so PGLite WASM cold-start is counted in the peak.
const { PGLiteEngine } = await import('../../src/core/pglite-engine.ts');
const { hybridSearch } = await import('../../src/core/search/hybrid.ts');
const engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Build the brain in-memory by inserting N synthetic pages directly via
// engine.putPage. This is the same code path the import command uses;
// we just skip the markdown file parsing.
process.stderr.write(`[_measure_rss_workload] inserting ${PAGES} pages...\n`);
for (let i = 0; i < PAGES; i += 1) {
const page = generatePage(i);
await engine.putPage(page.slug, {
type: 'note',
title: page.title,
compiled_truth: page.body,
timeline: '',
frontmatter: { synthetic: true, tags: ['synthetic', 'rss-fixture'] },
});
if (i > 0 && i % 500 === 0) {
process.stderr.write(`[_measure_rss_workload] inserted ${i}\n`);
}
}
const pageCountRows = await engine.executeRaw('SELECT count(*)::int AS c FROM pages', []);
const brainPageCount =
pageCountRows && pageCountRows[0] && typeof (pageCountRows[0] as { c: number }).c === 'number'
? (pageCountRows[0] as { c: number }).c
: 0;
process.stderr.write(`[_measure_rss_workload] running ${QUERIES} queries...\n`);
const queries = [
'synthetic page', 'lorem ipsum', 'consequat', 'voluptatem', 'aspernatur',
'magna aliqua', 'reprehenderit', 'commodo', 'inventore', 'architecto',
'page 100', 'page 500', 'rss-fixture', 'section 1', 'section 2',
];
let queriesRun = 0;
for (let i = 0; i < QUERIES; i += 1) {
const q = queries[i % queries.length]!;
try {
await hybridSearch(engine, q, { limit: 10 });
} catch {
/* keyword-only fallback if no embeddings — fine for the measurement */
}
queriesRun += 1;
}
await engine.disconnect();
const finalRss = readRss();
if (finalRss > peakRss) peakRss = finalRss;
result = {
ok: true,
platform: plat,
measurement_path: measurementPath,
queries_run: queriesRun,
peak_rss_kb: peakRss,
elapsed_ms: Date.now() - t0,
brain_page_count: brainPageCount,
...(measurementPath === 'fallback'
? { note: 'macOS/non-Linux fallback path (VmRSS, mmap-inflated)' }
: {}),
};
} catch (err) {
result = {
ok: false,
platform: plat,
measurement_path: measurementPath,
queries_run: 0,
peak_rss_kb: peakRss,
elapsed_ms: Date.now() - t0,
brain_page_count: 0,
error: err instanceof Error ? `${err.name}: ${err.message}` : String(err),
};
} finally {
stopPolling();
}
process.stdout.write(JSON.stringify(result) + '\n');
process.exit(result.ok ? 0 : 1);
}
void main();

266
tests/heavy/_read_latency_workload.ts Normal file
View File

@@ -0,0 +1,266 @@
/**
* tests/heavy/_read_latency_workload.ts
*
* Read-latency-under-write measurement. Two phases against the SAME brain:
*
* Phase A (baseline): N search queries, no concurrent writes. Records
* p50/p95/p99 latency.
* Phase B (under load): N search queries with M parallel writer tasks
* inserting pages in the background. Records
* p50/p95/p99 latency under contention.
*
* The headline metric is `delta_pct` between phase A and phase B p99 — that's
* what tells operators whether sync is impacting reads. PASS criterion (when
* `STRICT=1`): delta_pct <= 50% (informational-only otherwise).
*
* Self-contained: builds the brain in-memory at startup, runs both phases,
* exits. No cross-process state. PGLite-only for v1; Postgres path is a
* follow-up (the contention character is different against a real WAL).
*
* Env:
* BRAIN_PAGES initial fixture size (default 500; PGLite insert
* superlinearity caps useful values around ~200-300
* on Darwin, more on Linux CI)
* NUM_QUERIES queries per phase (default 200)
* NUM_WRITERS parallel writer tasks during phase B (default 4)
* WRITES_PER_WRITER pages each writer inserts during phase B (default 25)
* STRICT 1 = exit non-zero on delta_pct > THRESHOLD_PCT (default 0)
* THRESHOLD_PCT regression threshold (default 50)
*
* Output JSON shape (stable contract):
* {
* ok, platform, brain_page_count,
* phase_a: { p50_ms, p95_ms, p99_ms, queries_run },
* phase_b: { p50_ms, p95_ms, p99_ms, queries_run, writes_completed, writes_failed },
* delta_p50_pct, delta_p95_pct, delta_p99_pct,
* elapsed_ms, threshold_pct,
* verdict: 'pass' | 'fail' | 'informational',
* note?, error?
* }
*/
import { platform } from 'node:os';
interface Phase {
p50_ms: number;
p95_ms: number;
p99_ms: number;
queries_run: number;
writes_completed?: number;
writes_failed?: number;
}
interface Result {
ok: boolean;
platform: string;
brain_page_count: number;
phase_a: Phase | null;
phase_b: Phase | null;
delta_p50_pct: number | null;
delta_p95_pct: number | null;
delta_p99_pct: number | null;
elapsed_ms: number;
threshold_pct: number;
verdict: 'pass' | 'fail' | 'informational';
note?: string;
error?: string;
}
function percentile(sortedMs: number[], pct: number): number {
if (sortedMs.length === 0) return 0;
const idx = Math.min(sortedMs.length - 1, Math.floor((pct / 100) * sortedMs.length));
return sortedMs[idx]!;
}
function generatePage(i: number, prefix: string): { slug: string; title: string; body: string } {
const pad = String(i).padStart(5, '0');
const body =
`# ${prefix} Page ${i}\n\n` +
`Deterministic page for read-latency measurement. Body has stable text ` +
`so search-index work is consistent run-to-run.\n\n` +
`Section 1: lorem ipsum dolor sit amet consectetur adipiscing elit sed do ` +
`eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad ` +
`minim veniam quis nostrud exercitation ullamco laboris.\n\n` +
`Section 2: sed ut perspiciatis unde omnis iste natus error sit voluptatem ` +
`accusantium doloremque laudantium totam rem aperiam eaque ipsa quae.\n\n` +
`Reference: ${prefix}-${pad}.`;
return {
slug: `${prefix.toLowerCase()}/page-${pad}`,
title: `${prefix} Page ${i}`,
body,
};
}
async function main(): Promise<void> {
const t0 = Date.now();
const plat = platform();
const BRAIN_PAGES = parseInt(process.env.BRAIN_PAGES ?? '500', 10);
const NUM_QUERIES = parseInt(process.env.NUM_QUERIES ?? '200', 10);
const NUM_WRITERS = parseInt(process.env.NUM_WRITERS ?? '4', 10);
const WRITES_PER_WRITER = parseInt(process.env.WRITES_PER_WRITER ?? '25', 10);
const STRICT = (process.env.STRICT ?? '0') === '1';
const THRESHOLD_PCT = parseInt(process.env.THRESHOLD_PCT ?? '50', 10);
let result: Result = {
ok: false,
platform: plat,
brain_page_count: 0,
phase_a: null,
phase_b: null,
delta_p50_pct: null,
delta_p95_pct: null,
delta_p99_pct: null,
elapsed_ms: 0,
threshold_pct: THRESHOLD_PCT,
verdict: 'informational',
};
try {
const { PGLiteEngine } = await import('../../src/core/pglite-engine.ts');
const { hybridSearch } = await import('../../src/core/search/hybrid.ts');
const engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Build the fixture
process.stderr.write(`[_read_latency] inserting ${BRAIN_PAGES} fixture pages...\n`);
for (let i = 0; i < BRAIN_PAGES; i += 1) {
const p = generatePage(i, 'Fixture');
await engine.putPage(p.slug, {
type: 'note',
title: p.title,
compiled_truth: p.body,
timeline: '',
frontmatter: { fixture: true, tags: ['fixture'] },
});
}
const queries = [
'lorem ipsum', 'consequat', 'voluptatem', 'aspernatur', 'magna',
'reprehenderit', 'commodo', 'inventore', 'fixture page', 'section 1',
'section 2', 'page 100', 'doloremque', 'architecto', 'incididunt',
];
// ---- Phase A: baseline (no concurrent writes)
process.stderr.write(`[_read_latency] phase A: ${NUM_QUERIES} queries (baseline)...\n`);
const phaseAMs: number[] = [];
for (let i = 0; i < NUM_QUERIES; i += 1) {
const q = queries[i % queries.length]!;
const t = Date.now();
try {
await hybridSearch(engine, q, { limit: 10 });
} catch {
/* tolerated; latency still recorded */
}
phaseAMs.push(Date.now() - t);
}
phaseAMs.sort((a, b) => a - b);
result.phase_a = {
p50_ms: percentile(phaseAMs, 50),
p95_ms: percentile(phaseAMs, 95),
p99_ms: percentile(phaseAMs, 99),
queries_run: phaseAMs.length,
};
// ---- Phase B: under load (parallel writers)
process.stderr.write(`[_read_latency] phase B: ${NUM_QUERIES} queries with ${NUM_WRITERS} writers...\n`);
let writesCompleted = 0;
let writesFailed = 0;
let writersDone = 0;
const writers: Promise<void>[] = [];
for (let w = 0; w < NUM_WRITERS; w += 1) {
writers.push((async () => {
const base = BRAIN_PAGES + w * WRITES_PER_WRITER;
for (let i = 0; i < WRITES_PER_WRITER; i += 1) {
const p = generatePage(base + i, `WriterW${w}`);
try {
await engine.putPage(p.slug, {
type: 'note',
title: p.title,
compiled_truth: p.body,
timeline: '',
frontmatter: { writer: w, tags: ['writer-load'] },
});
writesCompleted += 1;
} catch {
writesFailed += 1;
}
}
writersDone += 1;
})());
}
const phaseBMs: number[] = [];
// Run queries until we hit NUM_QUERIES OR all writers finish, whichever
// is LATER. Goal: every query in phase B is during sustained writer
// pressure. If queries are too few and writers haven't finished, keep
// going past NUM_QUERIES for fairness.
let queryIdx = 0;
while (queryIdx < NUM_QUERIES || writersDone < NUM_WRITERS) {
const q = queries[queryIdx % queries.length]!;
const t = Date.now();
try {
await hybridSearch(engine, q, { limit: 10 });
} catch {
/* tolerated */
}
if (queryIdx < NUM_QUERIES) {
phaseBMs.push(Date.now() - t);
}
queryIdx += 1;
// Safety stop: don't blow past 4x NUM_QUERIES if writers are wedged
if (queryIdx >= 4 * NUM_QUERIES) break;
}
await Promise.allSettled(writers);
phaseBMs.sort((a, b) => a - b);
result.phase_b = {
p50_ms: percentile(phaseBMs, 50),
p95_ms: percentile(phaseBMs, 95),
p99_ms: percentile(phaseBMs, 99),
queries_run: phaseBMs.length,
writes_completed: writesCompleted,
writes_failed: writesFailed,
};
// Delta math: ((B - A) / A) * 100, integer percent. Guard divide-by-zero.
function deltaPct(a: number, b: number): number {
if (a <= 0) return 0;
return Math.round(((b - a) / a) * 100);
}
result.delta_p50_pct = deltaPct(result.phase_a.p50_ms, result.phase_b.p50_ms);
result.delta_p95_pct = deltaPct(result.phase_a.p95_ms, result.phase_b.p95_ms);
result.delta_p99_pct = deltaPct(result.phase_a.p99_ms, result.phase_b.p99_ms);
// Page count after both phases
const pageRows = await engine.executeRaw('SELECT count(*)::int AS c FROM pages', []);
result.brain_page_count =
pageRows && pageRows[0] && typeof (pageRows[0] as { c: number }).c === 'number'
? (pageRows[0] as { c: number }).c
: 0;
await engine.disconnect();
// Verdict
if (STRICT) {
result.verdict = (result.delta_p99_pct ?? 0) > THRESHOLD_PCT ? 'fail' : 'pass';
} else {
result.verdict = 'informational';
}
result.ok = true;
result.elapsed_ms = Date.now() - t0;
} catch (err) {
result.error = err instanceof Error ? `${err.name}: ${err.message}` : String(err);
result.elapsed_ms = Date.now() - t0;
}
process.stdout.write(JSON.stringify(result) + '\n');
if (!result.ok) process.exit(1);
if (STRICT && result.verdict === 'fail') process.exit(1);
process.exit(0);
}
void main();

25
tests/heavy/fixtures/down-mutate-pre-v0.13.sql Normal file
View File

@@ -0,0 +1,25 @@
-- down-mutate-pre-v0.13.sql
-- Strip the v0.13+ forward-referenced state from a fresh-LATEST brain to
-- simulate a pre-v0.13 brain shape. Covers issues #266, #357 — pre-v0.13
-- brains had `links` without `link_source` / `origin_page_id`, and the
-- schema blob's `CREATE INDEX idx_links_source` would crash before v11 ran.
--
-- After this runs, `gbrain doctor` MUST walk forward via the bootstrap
-- (postgres-engine.ts:applyForwardReferenceBootstrap) and reach LATEST
-- without wedging.
--
-- Pattern mirrored from test/bootstrap.test.ts:164-198 (PGLite side).
-- Run via psql against $DATABASE_URL.
BEGIN;
DROP INDEX IF EXISTS idx_links_source;
DROP INDEX IF EXISTS idx_links_origin;
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique;
ALTER TABLE links DROP COLUMN IF EXISTS link_source;
ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id;
-- Mark the brain at a pre-v0.13 version so the migration runner walks forward.
UPDATE config SET value = '10' WHERE key = 'version';
COMMIT;

34
tests/heavy/fixtures/down-mutate-pre-v0.18.sql Normal file
View File

@@ -0,0 +1,34 @@
-- down-mutate-pre-v0.18.sql
-- Strip the v0.18+ forward-referenced state from a fresh-LATEST brain to
-- simulate a pre-v0.18 brain shape. Covers issues #366, #375, #378 —
-- pre-v0.18 brains crashed on `column "source_id" does not exist`.
--
-- After this runs, `gbrain doctor` MUST walk forward via the bootstrap
-- (postgres-engine.ts:applyForwardReferenceBootstrap) and reach LATEST
-- without wedging. Bootstrap re-creates `sources` table + seeds 'default',
-- re-adds `pages.source_id`, then SCHEMA_SQL replay + migrations run clean.
--
-- Pattern mirrored from test/bootstrap.test.ts:102-139 (PGLite side).
-- Run via psql against $DATABASE_URL.
BEGIN;
-- Restore the pre-v0.18 unique constraint shape on pages.slug (v0.18 widened
-- this to (source_id, slug)).
ALTER TABLE pages DROP CONSTRAINT IF EXISTS pages_source_slug_key;
ALTER TABLE pages ADD CONSTRAINT pages_slug_key UNIQUE (slug);
DROP INDEX IF EXISTS idx_pages_source_id;
ALTER TABLE pages DROP COLUMN IF EXISTS source_id;
-- DROP TABLE sources with CASCADE so any dangling FKs go too.
DROP TABLE IF EXISTS sources CASCADE;
-- Pre-v0.18 didn't have resolution_type on links either.
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_resolution_type_check;
ALTER TABLE links DROP COLUMN IF EXISTS resolution_type;
-- Mark the brain at a pre-v0.18 version so the migration runner walks forward.
UPDATE config SET value = '20' WHERE key = 'version';
COMMIT;

131
tests/heavy/measure_rss.sh Executable file
View File

@@ -0,0 +1,131 @@
#!/usr/bin/env bash
# tests/heavy/measure_rss.sh
# RSS budget gate. Builds a synthetic brain, runs a read workload, measures
# peak RSS, compares against the committed baseline.
#
# Informational-only by default. Set STRICT_RSS=1 to fail the script when
# peak RSS exceeds the baseline by more than RSS_THRESHOLD_PCT (default 25%).
#
# Baseline refresh is GATED to Linux. macOS measurement runs use a VmRSS
# fallback path that gbrain explicitly avoids in production; committing a
# macOS-derived baseline would lock in a wrong metric. See _measure_rss_workload.ts.
#
# Usage:
# tests/heavy/measure_rss.sh # measure + report; never fail
# tests/heavy/measure_rss.sh --refresh-baseline # Linux only — overwrite baseline
# STRICT_RSS=1 tests/heavy/measure_rss.sh # exit 1 on regression
#
# Env vars:
# BRAIN_PAGES pages in the synthetic brain (default 200)
# Larger is more realistic but PGLite insert + autolink
# scales superlinearly past ~300 pages on Darwin
# (observed: 200=1.2s, 500+ hits the 300s timeout).
# The cathedral upgrade path is committing a generated
# fixture instead of building in-memory each run; deferred.
# NUM_QUERIES search queries to run (default 50)
# RSS_THRESHOLD_PCT regression threshold (default 25)
# STRICT_RSS 1 to fail on regression; default 0 (informational)
set -euo pipefail
cd "$(dirname "$0")/../.."
BASELINE="tests/heavy/rss-baseline.json"
THRESHOLD_PCT="${RSS_THRESHOLD_PCT:-25}"
STRICT="${STRICT_RSS:-0}"
REFRESH=0
LOG_DIR="${GBRAIN_HOME:-$HOME/.gbrain}/audit"
TS=$(date -u +%Y%m%d-%H%M%SZ)
mkdir -p "$LOG_DIR"
for arg in "$@"; do
case "$arg" in
--refresh-baseline) REFRESH=1 ;;
--help|-h)
sed -n '2,18p' "$0"
exit 0
;;
*)
echo "[measure_rss] unknown arg: $arg" >&2
exit 2
;;
esac
done
PLATFORM=$(uname -s)
echo "[measure_rss] platform=$PLATFORM pages=${BRAIN_PAGES:-200} threshold=${THRESHOLD_PCT}% strict=$STRICT"
# Refusal: macOS / non-Linux baseline refresh.
if [ "$REFRESH" = "1" ] && [ "$PLATFORM" != "Linux" ]; then
echo "[measure_rss] REFUSAL: --refresh-baseline only safe on Linux." >&2
echo " macOS falls back to process.memoryUsage().rss (VmRSS) — the metric" >&2
echo " gbrain explicitly avoids in production. Committing a macOS-derived" >&2
echo " baseline would lock in the wrong metric and produce false CI delta_pct." >&2
echo " Run inside Linux docker:" >&2
echo " docker run --rm -v \"\$(pwd):/app\" -w /app oven/bun:1 tests/heavy/measure_rss.sh --refresh-baseline" >&2
exit 2
fi
# Step 1: run the in-memory workload. Brain build + measurement happen in
# one process (the workload TS file inserts synthetic pages into a fresh
# in-memory PGLite, then runs the read loop). No cross-process state.
WORKLOAD_OUT="$LOG_DIR/heavy-measure_rss-workload-$TS.json"
echo "[measure_rss] running in-memory workload (brain insert + search loop)..."
unset DATABASE_URL || true
set +e
timeout 600s env \
BRAIN_PAGES="${BRAIN_PAGES:-200}" \
NUM_QUERIES="${NUM_QUERIES:-50}" \
bun run tests/heavy/_measure_rss_workload.ts > "$WORKLOAD_OUT" 2>>"$LOG_DIR/heavy-measure_rss-$TS.log"
WORKLOAD_RC=$?
set -e
if [ "$WORKLOAD_RC" -ne 0 ]; then
echo "[measure_rss] FAIL: workload exited $WORKLOAD_RC" >&2
cat "$WORKLOAD_OUT" >&2 2>/dev/null || true
echo " See $LOG_DIR/heavy-measure_rss-$TS.log for stderr." >&2
exit 1
fi
# Step 3: parse the workload output
PEAK_KB=$(grep -oE '"peak_rss_kb"[[:space:]]*:[[:space:]]*[0-9]+' "$WORKLOAD_OUT" | head -1 | grep -oE '[0-9]+$')
MEASUREMENT_PATH=$(grep -oE '"measurement_path"[[:space:]]*:[[:space:]]*"[^"]+"' "$WORKLOAD_OUT" | head -1 | sed -E 's/.*"([^"]+)"$/\1/')
QUERIES=$(grep -oE '"queries_run"[[:space:]]*:[[:space:]]*[0-9]+' "$WORKLOAD_OUT" | head -1 | grep -oE '[0-9]+$')
ELAPSED=$(grep -oE '"elapsed_ms"[[:space:]]*:[[:space:]]*[0-9]+' "$WORKLOAD_OUT" | head -1 | grep -oE '[0-9]+$')
echo "[measure_rss] peak_rss_kb=$PEAK_KB measurement_path=$MEASUREMENT_PATH queries=$QUERIES elapsed_ms=$ELAPSED"
# Step 4: compare against baseline (if it exists and has a number)
if [ -f "$BASELINE" ] && [ -s "$BASELINE" ]; then
BASELINE_KB=$(grep -oE '"peak_rss_kb"[[:space:]]*:[[:space:]]*[0-9]+' "$BASELINE" | head -1 | grep -oE '[0-9]+$' || echo "")
if [ -n "$BASELINE_KB" ] && [ "$BASELINE_KB" -gt 0 ]; then
# Integer percentage delta: ((peak - baseline) * 100) / baseline
DELTA_PCT=$(( (PEAK_KB - BASELINE_KB) * 100 / BASELINE_KB ))
echo "[measure_rss] baseline=$BASELINE_KB peak=$PEAK_KB delta_pct=$DELTA_PCT% (threshold=$THRESHOLD_PCT%)"
if [ "$DELTA_PCT" -gt "$THRESHOLD_PCT" ]; then
echo "[measure_rss] REGRESSION: delta_pct=$DELTA_PCT% exceeds threshold $THRESHOLD_PCT%" >&2
if [ "$STRICT" = "1" ]; then
echo "[measure_rss] STRICT_RSS=1; failing" >&2
exit 1
else
echo "[measure_rss] STRICT_RSS=0; informational-only, not failing" >&2
fi
fi
else
echo "[measure_rss] baseline exists but has no peak_rss_kb; treating as missing"
fi
else
echo "[measure_rss] no baseline yet at $BASELINE; measurement is informational-only"
fi
# Step 5: refresh baseline if requested (Linux-only; refusal already handled above)
if [ "$REFRESH" = "1" ]; then
if [ "$MEASUREMENT_PATH" != "proc" ]; then
echo "[measure_rss] REFUSAL: workload reported measurement_path=$MEASUREMENT_PATH, expected 'proc'." >&2
echo " This means /proc/self/status was unavailable. Refusing to commit baseline." >&2
exit 2
fi
cp "$WORKLOAD_OUT" "$BASELINE"
echo "[measure_rss] baseline refreshed at $BASELINE"
fi
echo "[measure_rss] OK"

116
tests/heavy/pg_upgrade_matrix.sh Executable file
View File

@@ -0,0 +1,116 @@
#!/usr/bin/env bash
# tests/heavy/pg_upgrade_matrix.sh
# Schema-migration walk-forward matrix. For each "historical shape," build the
# fixture, then run `gbrain doctor` and assert it reaches LATEST cleanly.
#
# This exercises the bootstrap → SCHEMA_SQL → runMigrations path against
# real Postgres (the pre-existing test/e2e/postgres-bootstrap.test.ts covers
# the engine-level case; this matrix runs it end-to-end via the CLI shell
# under multiple simulated historical brain shapes).
#
# Why the matrix matters: the 10+ forward-reference bugs documented in
# CLAUDE.md (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396) all shipped
# because a new release added a column-with-index in the schema blob without
# the corresponding bootstrap probe. Walking forward from multiple legacy
# shapes catches the next member of that bug class before users hit it.
#
# Honest contract (smoke-tested 2026-05-19): this matrix catches whole-system
# wedges, not single-layer bootstrap regressions. gbrain has a multi-layer
# defense (bootstrap → SCHEMA_SQL replay → migrations → verifySchema), and
# any one layer can heal what an upstream layer misses. We verified that
# stubbing out `applyForwardReferenceBootstrap` entirely still produces a
# clean walk-forward on both shapes — the downstream layers cover it. The
# matrix detects: (a) a regression that breaks ALL repair layers for a given
# column, (b) genuine wedge bugs where a hard SQL error escapes every layer,
# (c) timeouts in the walk-forward path. Single-layer regressions are caught
# by `test/schema-bootstrap-coverage.test.ts` (static) and
# `test/e2e/postgres-bootstrap.test.ts` (engine-level).
#
# Usage:
# DATABASE_URL=postgresql://... ./tests/heavy/pg_upgrade_matrix.sh
#
# Or via the runner: bun run test:heavy
set -euo pipefail
cd "$(dirname "$0")/../.."
if [ -z "${DATABASE_URL:-}" ]; then
echo "[pg_upgrade_matrix] DATABASE_URL not set; skipping (informational)." >&2
echo " Local: docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test -p 5434:5432 pgvector/pgvector:pg16" >&2
echo " Then: export DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test" >&2
exit 0
fi
# Audit log path
TS=$(date -u +%Y%m%d-%H%M%SZ)
LOG_DIR="${GBRAIN_HOME:-$HOME/.gbrain}/audit"
mkdir -p "$LOG_DIR"
LOG_FILE="$LOG_DIR/heavy-pg_upgrade_matrix-${TS}.log"
# Each entry runs build_legacy_fixtures.sh <shape> + gbrain doctor.
SHAPES=(pre-v0.13 pre-v0.18)
echo "[pg_upgrade_matrix] running ${#SHAPES[@]} shape(s): ${SHAPES[*]}"
echo "[pg_upgrade_matrix] log=$LOG_FILE"
echo ""
fails=0
for SHAPE in "${SHAPES[@]}"; do
echo "[pg_upgrade_matrix] --- shape=$SHAPE ---" | tee -a "$LOG_FILE"
# Step 1: build the legacy fixture (drop schema, init to LATEST, down-mutate)
set +e
timeout 180s bash tests/heavy/_build_legacy_fixtures.sh "$SHAPE" >> "$LOG_FILE" 2>&1
BUILD_RC=$?
set -e
if [ "$BUILD_RC" -ne 0 ]; then
echo "[pg_upgrade_matrix] FAIL: build_legacy_fixtures.sh $SHAPE exited $BUILD_RC (see $LOG_FILE)" >&2
fails=$((fails + 1))
continue
fi
# Step 2: walk forward. `gbrain doctor` triggers engine.connect() which
# runs applyForwardReferenceBootstrap → SCHEMA_SQL → runMigrations.
# Wedges manifest as either a timeout, a non-zero exit, or status != 'ok'
# in the JSON output.
DOCTOR_OUT="$LOG_DIR/heavy-pg_upgrade_doctor-${TS}-${SHAPE}.json"
set +e
timeout 120s bun run src/cli.ts doctor --json > "$DOCTOR_OUT" 2>>"$LOG_FILE"
DOCTOR_RC=$?
set -e
if [ "$DOCTOR_RC" -ne 0 ]; then
echo "[pg_upgrade_matrix] FAIL: doctor on $SHAPE exited $DOCTOR_RC (see $LOG_FILE + $DOCTOR_OUT)" >&2
fails=$((fails + 1))
continue
fi
# Step 3: assert status is non-fatal. We accept 'ok' and 'warnings' because
# a freshly walked-forward brain may have legitimate warnings (zero pages,
# no embeddings, etc) that are not wedge-class failures. We do NOT accept
# 'fail' or 'failures' (gbrain doctor's terminal-failure shape).
STATUS=$(grep -oE '"status"[[:space:]]*:[[:space:]]*"[^"]+"' "$DOCTOR_OUT" | head -1 | sed -E 's/.*"([^"]+)"$/\1/')
case "$STATUS" in
ok|warn|warnings)
echo "[pg_upgrade_matrix] OK: shape=$SHAPE → status=$STATUS" | tee -a "$LOG_FILE"
;;
fail|failures|failed|"")
echo "[pg_upgrade_matrix] FAIL: shape=$SHAPE doctor reported status='$STATUS' (see $DOCTOR_OUT)" >&2
fails=$((fails + 1))
;;
*)
echo "[pg_upgrade_matrix] FAIL: shape=$SHAPE unexpected doctor status='$STATUS' (see $DOCTOR_OUT)" >&2
fails=$((fails + 1))
;;
esac
echo "" | tee -a "$LOG_FILE"
done
if [ "$fails" -gt 0 ]; then
echo "[pg_upgrade_matrix] FAILED: $fails/${#SHAPES[@]} shape(s) wedged on walk-forward." >&2
echo "[pg_upgrade_matrix] Full log: $LOG_FILE" >&2
exit 1
fi
echo "[pg_upgrade_matrix] OK — all ${#SHAPES[@]} shape(s) walked forward cleanly."

77
tests/heavy/read_latency_under_sync.sh Executable file
View File

@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# tests/heavy/read_latency_under_sync.sh
# Measure search latency under concurrent writer load.
#
# Runs an in-memory PGLite brain through two phases: baseline (no writes) +
# under-load (parallel writers inserting pages). Records p50/p95/p99 latency
# in each phase, reports delta_pct. The contract that matters: search p99
# shouldn't blow up while sync is running.
#
# Informational-only by default (delta_pct gets reported but exit stays 0).
# Set STRICT_LATENCY=1 to fail when p99 delta exceeds threshold.
#
# Usage:
# tests/heavy/read_latency_under_sync.sh
# STRICT_LATENCY=1 THRESHOLD_PCT=50 tests/heavy/read_latency_under_sync.sh
#
# Env vars:
# BRAIN_PAGES initial fixture (default 500)
# NUM_QUERIES queries per phase (default 200)
# NUM_WRITERS parallel writers in phase B (default 4)
# WRITES_PER_WRITER pages each writer inserts (default 25)
# STRICT_LATENCY 1 = fail on p99 delta > threshold (default 0)
# THRESHOLD_PCT p99 regression threshold percent (default 50)
set -euo pipefail
cd "$(dirname "$0")/../.."
LOG_DIR="${GBRAIN_HOME:-$HOME/.gbrain}/audit"
mkdir -p "$LOG_DIR"
TS=$(date -u +%Y%m%d-%H%M%SZ)
WORKLOAD_OUT="$LOG_DIR/heavy-read_latency-$TS.json"
echo "[read_latency] pages=${BRAIN_PAGES:-500} queries=${NUM_QUERIES:-200} writers=${NUM_WRITERS:-4} strict=${STRICT_LATENCY:-0}"
echo "[read_latency] running baseline + under-load workload..."
unset DATABASE_URL || true
set +e
timeout 600s env \
BRAIN_PAGES="${BRAIN_PAGES:-500}" \
NUM_QUERIES="${NUM_QUERIES:-200}" \
NUM_WRITERS="${NUM_WRITERS:-4}" \
WRITES_PER_WRITER="${WRITES_PER_WRITER:-25}" \
STRICT="${STRICT_LATENCY:-0}" \
THRESHOLD_PCT="${THRESHOLD_PCT:-50}" \
bun run tests/heavy/_read_latency_workload.ts > "$WORKLOAD_OUT" 2>>"$LOG_DIR/heavy-read_latency-stderr-$TS.log"
WORKLOAD_RC=$?
set -e
if [ "$WORKLOAD_RC" -ne 0 ]; then
echo "[read_latency] FAIL: workload exited $WORKLOAD_RC" >&2
cat "$WORKLOAD_OUT" >&2 2>/dev/null || true
echo " See $LOG_DIR/heavy-read_latency-stderr-$TS.log for stderr." >&2
exit 1
fi
# Surface key numbers from the JSON. Awk wins here vs jq since jq isn't
# guaranteed on every CI runner; the JSON shape is stable per the workload.
A_P50=$(grep -oE '"phase_a"[[:space:]]*:[[:space:]]*\{[^}]+\}' "$WORKLOAD_OUT" | grep -oE '"p50_ms"[[:space:]]*:[[:space:]]*[0-9]+' | grep -oE '[0-9]+$')
A_P99=$(grep -oE '"phase_a"[[:space:]]*:[[:space:]]*\{[^}]+\}' "$WORKLOAD_OUT" | grep -oE '"p99_ms"[[:space:]]*:[[:space:]]*[0-9]+' | grep -oE '[0-9]+$')
B_P50=$(grep -oE '"phase_b"[[:space:]]*:[[:space:]]*\{[^}]+\}' "$WORKLOAD_OUT" | grep -oE '"p50_ms"[[:space:]]*:[[:space:]]*[0-9]+' | grep -oE '[0-9]+$')
B_P99=$(grep -oE '"phase_b"[[:space:]]*:[[:space:]]*\{[^}]+\}' "$WORKLOAD_OUT" | grep -oE '"p99_ms"[[:space:]]*:[[:space:]]*[0-9]+' | grep -oE '[0-9]+$')
DELTA_P99=$(grep -oE '"delta_p99_pct"[[:space:]]*:[[:space:]]*-?[0-9]+' "$WORKLOAD_OUT" | grep -oE '\-?[0-9]+$')
VERDICT=$(grep -oE '"verdict"[[:space:]]*:[[:space:]]*"[^"]+"' "$WORKLOAD_OUT" | sed -E 's/.*"([^"]+)"$/\1/')
WRITES_DONE=$(grep -oE '"writes_completed"[[:space:]]*:[[:space:]]*[0-9]+' "$WORKLOAD_OUT" | grep -oE '[0-9]+$')
echo "[read_latency] phase_a: p50=${A_P50}ms p99=${A_P99}ms"
echo "[read_latency] phase_b: p50=${B_P50}ms p99=${B_P99}ms (writes_completed=${WRITES_DONE})"
echo "[read_latency] delta_p99=${DELTA_P99}% verdict=${VERDICT}"
echo "[read_latency] full JSON: $WORKLOAD_OUT"
if [ "${STRICT_LATENCY:-0}" = "1" ] && [ "$VERDICT" = "fail" ]; then
echo "[read_latency] FAIL: p99 regression exceeds threshold (STRICT_LATENCY=1)" >&2
exit 1
fi
echo "[read_latency] OK"

12
tests/heavy/rss-baseline.json Normal file
View File

@@ -0,0 +1,12 @@
{
"_note": "Empty stub baseline. Populated on first Linux CI run via `tests/heavy/measure_rss.sh --refresh-baseline`. macOS refuses to write here because the local fallback path measures VmRSS, which is mmap-inflated and not what production users see. Until this file has a real peak_rss_kb, every measurement run is informational-only.",
"_format": {
"ok": "boolean",
"platform": "linux",
"measurement_path": "proc",
"queries_run": "number",
"peak_rss_kb": "number // populated on baseline refresh",
"elapsed_ms": "number",
"brain_page_count": "number"
}
}

163
tests/heavy/sync_lock_regression.sh Executable file
View File

@@ -0,0 +1,163 @@
#!/usr/bin/env bash
# tests/heavy/sync_lock_regression.sh
# Sync writer-lock concurrency regression test.
#
# Spawns N concurrent `gbrain sync` processes against one DB; asserts:
# 1. Exactly one wins the writer lock (`gbrain-sync` row in `gbrain_cycle_locks`).
# 2. N-1 lose with "Another sync is in progress" — they fail FAST, they don't queue.
# (Per src/commands/sync.ts:377 — performSync uses `tryAcquireDbLock`, no wait.)
# 3. After all processes exit, zero leaked `gbrain_cycle_locks` rows remain.
#
# Why the test matters: the eng-review-flagged v1 plan was wrong — the original
# plan asserted the wrong semantics ("N-1 wait then complete one at a time")
# and snapshot the wrong table (`pg_locks` instead of `gbrain_cycle_locks`).
# Both reviewers caught it; this script tests the actual contract.
#
# Postgres-only (no DATABASE_URL = graceful skip with hint).
set -euo pipefail
cd "$(dirname "$0")/../.."
if [ -z "${DATABASE_URL:-}" ]; then
echo "[sync_lock_regression] DATABASE_URL not set; skipping (informational)." >&2
echo " Local: docker run -d --name gbrain-test-pg -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gbrain_test -p 5434:5432 pgvector/pgvector:pg16" >&2
echo " Then: export DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test" >&2
exit 0
fi
if ! command -v psql >/dev/null 2>&1; then
echo "[sync_lock_regression] psql required. Install postgresql-client." >&2
exit 2
fi
TS=$(date -u +%Y%m%d-%H%M%SZ)
# Isolate from the developer's real ~/.gbrain so writing sync.repo_path doesn't
# clobber their config. Restored on exit.
TMP_GBRAIN_HOME=$(mktemp -d -t gbrain-sync-lock-home-XXXXXX)
export GBRAIN_HOME="$TMP_GBRAIN_HOME"
LOG_DIR="$GBRAIN_HOME/audit"
mkdir -p "$LOG_DIR"
LOG="$LOG_DIR/heavy-sync_lock_regression-$TS.log"
# Surface the log path so it survives the EXIT trap that nukes GBRAIN_HOME.
SURFACE_LOG="${TMPDIR:-/tmp}/heavy-sync_lock_regression-$TS.log"
trap 'rm -rf "$TMP_GBRAIN_HOME"; cp -f "$LOG" "$SURFACE_LOG" 2>/dev/null || true' EXIT
NUM_PARALLEL="${NUM_PARALLEL:-4}"
echo "[sync_lock_regression] DATABASE_URL=$DATABASE_URL"
echo "[sync_lock_regression] log=$LOG"
echo "[sync_lock_regression] spawning $NUM_PARALLEL parallel sync processes..."
# Step 1: ensure schema is up-to-date by running doctor once
echo "[sync_lock_regression] init schema via gbrain doctor..." | tee -a "$LOG"
timeout 180s bun run src/cli.ts doctor --json > /dev/null 2>>"$LOG"
# Step 2: create a tiny brain dir + register it as sync.repo_path so each sync
# call has something legitimate to do.
BRAIN_DIR=$(mktemp -d -t gbrain-sync-lock-XXXXXX)
# Compose with the earlier GBRAIN_HOME-cleanup trap (NOT overwrite it).
trap 'rm -rf "$BRAIN_DIR" "$TMP_GBRAIN_HOME"; cp -f "$LOG" "$SURFACE_LOG" 2>/dev/null || true' EXIT
# Seed two markdown pages so sync has real (but trivial) work
mkdir -p "$BRAIN_DIR"
cat > "$BRAIN_DIR/page-a.md" <<'EOF'
---
title: Lock Test Page A
---
# Lock Test Page A
Trivial content for sync-lock-regression heavy test.
EOF
cat > "$BRAIN_DIR/page-b.md" <<'EOF'
---
title: Lock Test Page B
---
# Lock Test Page B
Trivial content for sync-lock-regression heavy test.
EOF
# git-init so sync's diff-walk has something to anchor (sync expects a git repo)
(cd "$BRAIN_DIR" && git init -q && git add . && git -c user.email=test@test -c user.name=test commit -q -m "seed" >/dev/null 2>&1) || true
# Tell gbrain to use this brain dir
bun run src/cli.ts config set sync.repo_path "$BRAIN_DIR" >/dev/null 2>&1 || true
# Step 3: spawn N parallel sync processes. Capture each one's exit code +
# stdout/stderr. The race for the lock happens during their startup window.
PIDS=()
EXIT_FILES=()
OUT_FILES=()
for ((i=1; i<=NUM_PARALLEL; i+=1)); do
EXIT_F=$(mktemp -t sync-lock-exit-XXXXXX)
OUT_F=$(mktemp -t sync-lock-out-XXXXXX)
EXIT_FILES+=("$EXIT_F")
OUT_FILES+=("$OUT_F")
( bun run src/cli.ts sync --dir "$BRAIN_DIR" >"$OUT_F" 2>&1; echo $? > "$EXIT_F" ) &
PIDS+=($!)
done
echo "[sync_lock_regression] waiting on ${#PIDS[@]} pids..."
for pid in "${PIDS[@]}"; do
wait "$pid" 2>/dev/null || true
done
# Step 4: collect outcomes
WINNERS=0
LOSERS=0
UNKNOWN=0
for ((i=0; i<NUM_PARALLEL; i+=1)); do
rc=$(cat "${EXIT_FILES[$i]}" 2>/dev/null || echo "?")
out_file="${OUT_FILES[$i]}"
if [ "$rc" = "0" ]; then
WINNERS=$((WINNERS + 1))
echo " [sync $((i+1))] rc=0 (winner)" | tee -a "$LOG"
elif grep -q "Another sync is in progress" "$out_file" 2>/dev/null; then
LOSERS=$((LOSERS + 1))
echo " [sync $((i+1))] rc=$rc (lock-busy: 'Another sync is in progress')" | tee -a "$LOG"
else
UNKNOWN=$((UNKNOWN + 1))
echo " [sync $((i+1))] rc=$rc (unknown failure — see $out_file)" | tee -a "$LOG"
head -5 "$out_file" 2>/dev/null | sed 's/^/ > /' | tee -a "$LOG"
fi
done
# Cleanup tmp files
rm -f "${EXIT_FILES[@]}" "${OUT_FILES[@]}"
echo "[sync_lock_regression] outcomes: winners=$WINNERS losers=$LOSERS unknown=$UNKNOWN" | tee -a "$LOG"
# Step 5: assert no leaked gbrain_cycle_locks rows. The pkey column is `id`,
# not `lock_id` (column name confirmed via \d gbrain_cycle_locks).
LEAKED=$(psql "$DATABASE_URL" -t -A -c "SELECT COUNT(*) FROM gbrain_cycle_locks WHERE id = 'gbrain-sync';" 2>>"$LOG" | tr -d ' ')
echo "[sync_lock_regression] post-run gbrain_cycle_locks(gbrain-sync) row count: $LEAKED" | tee -a "$LOG"
# Step 6: verdict
FAIL=0
# We must see exactly one winner. Multiple winners means the lock isn't
# enforcing exclusion; zero winners means every sync failed and we don't know
# if the lock matters.
if [ "$WINNERS" -ne 1 ]; then
echo "[sync_lock_regression] FAIL: expected 1 winner, got $WINNERS" >&2
FAIL=1
fi
# We must see N-1 lock-busy losers — anything else means a sync failed for a
# reason other than the lock (which would taint the measurement).
EXPECTED_LOSERS=$((NUM_PARALLEL - 1))
if [ "$LOSERS" -ne "$EXPECTED_LOSERS" ]; then
echo "[sync_lock_regression] FAIL: expected $EXPECTED_LOSERS lock-busy losers, got $LOSERS (unknown failures: $UNKNOWN)" >&2
FAIL=1
fi
# The lock row must be cleaned up on exit (release via try/finally).
if [ "$LEAKED" != "0" ]; then
echo "[sync_lock_regression] FAIL: $LEAKED leaked gbrain_cycle_locks(gbrain-sync) row(s) after all syncs exited" >&2
FAIL=1
fi
if [ "$FAIL" -ne 0 ]; then
echo "[sync_lock_regression] FAILED. See $LOG for details." >&2
exit 1
fi
echo "[sync_lock_regression] OK — 1 winner, $LOSERS lock-busy losers, no leaked lock rows."