triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.33.3.0 feat(v0.33.3): code intelligence MCP foundation (v0.34 W0a-c + W3) (#934)

Browse Source 
* feat(v0.34 pre-w0): add code-retrieval eval harness for v0.34 ship gate

Captures pre-v0.34 retrieval quality on the gbrain self-corpus before any
code-intel work lands, so the v0.34 ship gate (precision@5 +10pp OR
answered_rate +15pp on >=15/30 questions) measures real improvement
rather than an after-the-fact retuned baseline.

* src/eval/code-retrieval/harness.ts -- pure-function metrics (precision@k,
  recall@k, top-1 stability, gate evaluator) + EvalRunReport types stable
  across schema_version 1
* src/eval/code-retrieval/questions.json -- 30 questions across callers /
  callees / definition / references / blast_radius / execution_flow /
  cluster_membership kinds, expected_files captured against current
  gbrain layout
* src/eval/code-retrieval/strategies.ts -- BaselineStrategy (hybridSearch)
  + WithCodeIntelStrategy stub (post-W3 fills in code_blast/code_flow/etc.)
* src/commands/eval-code-retrieval.ts -- gbrain eval code-retrieval CLI
  with --baseline / --with-code-intel / --compare subcommands
* test/code-retrieval-harness.test.ts -- 26 unit tests across metrics,
  loader, gate logic; no engine dependency

PRE-V0.34 BASELINE WORKFLOW:
  gbrain eval code-retrieval --baseline --save /tmp/baseline-1.json
  (run 3x for noise floor)

V0.34 SHIP GATE (after W3 lands):
  gbrain eval code-retrieval --with-code-intel --save /tmp/v034.json
  gbrain eval code-retrieval --compare /tmp/baseline-1.json /tmp/v034.json

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(v0.34 W0a): source-routing leak across query + two-pass

Codex outside-voice review on the v0.34 plan caught two load-bearing
sites where sourceId was advertised but never applied — multi-source
brains silently cross-contaminated structural retrieval:

* operations.ts ~323 — `query` op handler called hybridSearch without
  threading ctx.sourceId. Multi-source agents querying with a
  --source flag got cross-source results.
* two-pass.ts:81 (nearSymbol lookup) and two-pass.ts:131 (unresolved
  edge resolution) — TwoPassOpts.sourceId was declared and threaded
  through hybridSearch's expandAnchors call, but the actual SQL ignored
  it. The walk window crossed source boundaries every time.

Fix:
* `query` op now reads ctx.sourceId AND accepts a new `source_id`
  param (with '__all__' as the explicit force-cross-source escape
  hatch). Per-call param wins over ctx context.
* two-pass.ts both lookups join through pages.source_id when
  opts.sourceId is set; omitted opts.sourceId preserves the legacy
  cross-source contract for callers who want it.

Regression test: test/e2e/source-routing.test.ts seeds two sources
with the same `parseMarkdown` symbol + a cross-source caller edge.
Pins:
  - nearSymbol + sourceId='source-a' returns ONLY source-a chunks
  - nearSymbol + sourceId='source-b' returns ONLY source-b chunks
  - nearSymbol with no sourceId still crosses sources (contract preserved)
  - walk_depth=1 unresolved-edge resolution stays in source-a

PGLite in-memory, no DATABASE_URL needed. The fix proves out under
realistic structural retrieval not just a contrived unit test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(v0.34 W0b): flip CLI source-scoping default to truly source-scoped

Codex outside-voice review (finding #7) caught that the v0.20.0
docstring claim "by default we only match the caller's source_id"
contradicted the implementation in code-callers.ts:54 + code-callees.ts:43:

  allSources: allSources || !sourceId

The right side made `allSources` TRUE whenever `--source` was omitted,
INVERTING the documented default. Multi-source brains silently cross-
contaminated structural retrieval; `gbrain code-callers parseMarkdown`
on a brain with two repos returned callers from both even though the
docstring promised per-source scoping.

Fix:
* New canonical helper `resolveDefaultSource(engine)` in sources-ops.ts.
  Contract per eng review D7:
    - exactly 1 source registered → return its id (single-source brains,
      the 80% case; --source flag is unnecessary friction there)
    - 2+ sources → throw SourceResolutionError(multiple_sources_ambiguous)
      with the list of valid ids
    - 0 sources → throw SourceResolutionError(no_sources)
* code-callers.ts + code-callees.ts now resolve to the default source
  when both --source AND --all-sources are absent. To get the pre-v0.34
  cross-source behavior, callers must pass --all-sources explicitly.
* Same hint text on both commands. Pinned by test/e2e/cli-source-scoping-pglite.test.ts.

IRON RULE regression R2: docstring promise now holds. Multi-source brain
running `gbrain code-callers <symbol>` without --source gets a clear
error listing valid source ids instead of silent cross-resolution.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.34 W0c): within-file two-pass symbol resolver + edges_backfilled_at watermark

Codex's outside-voice review caught that the v0.20.0 graph stores BARE
callee tokens (`render`, `find`, `execute`) — not qualified names. Pre-v0.34
recursive blast/flow would alias every same-named function across classes.
W0c is the foundation that fixes this: resolve `code_edges_symbol` rows by
matching `to_symbol_qualified` against the SAME-FILE chunks'
`symbol_name_qualified`, then write the outcome to `edge_metadata`.

This commit is the resolver primitive + schema. The cycle-phase wiring
that calls it on every quick-cycle tick lands in the next commit.

Schema (v51 migration `edges_backfilled_at_v0_34`):
* `content_chunks.edges_backfilled_at TIMESTAMPTZ` — resume watermark.
  Chunks where the column is NULL OR older than EDGE_EXTRACTOR_VERSION_TS
  get re-walked next tick. SIGINT/OOM/sleep mid-backfill loses at most
  one batch.
* Indexes per D11 from eng review:
  - `idx_code_edges_symbol_resolver(source_id, to_symbol_qualified)` —
    composite for the resolver's per-source lookup.
  - `idx_content_chunks_symbol_lookup(page_id, symbol_name_qualified)`
    WHERE `symbol_name_qualified IS NOT NULL` — file-batched candidate
    fetch; also reused by W4-5 cluster recompute.
  - `idx_content_chunks_edges_backfill(edges_backfilled_at)` WHERE
    `edges_backfilled_at IS NULL` — fast unresumed-row scan.

Module (`src/core/chunkers/symbol-resolver.ts`):
* `resolveSymbolEdgesIncremental(engine, {sourceId, maxChunks?, onProgress?})`
  walks stale chunks in 200-chunk batches. For each chunk, loads its
  unresolved edges, finds same-page candidates by symbol_name_qualified,
  and writes outcome to `edge_metadata`:
   - exactly 1 candidate → `{resolved_chunk_id: <id>}`
   - 2+ candidates → `{ambiguous: true, candidates: [...]}`
   - 0 candidates → unchanged (cross-file; two-pass.ts handles those)
  Each batch bumps `edges_backfilled_at = NOW()` for the chunks.
* `readEdgeResolution(metadata)` — public helper for downstream code
  (two-pass.ts, code_blast op, eval-capture) to consume the resolver's
  output without parsing JSON directly. Returns a tagged union.
* `EDGE_EXTRACTOR_VERSION_TS` exported constant — bump when extractor
  shape changes and the next cycle re-walks all chunks.

Tests (5 E2E in test/e2e/symbol-resolver-pglite.test.ts, all PGLite,
no DATABASE_URL): unambiguous match, ambiguous multi-match, no match,
watermark advance + idempotency, source isolation (no cross-source
candidate leak).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.34 W0c): wire resolve_symbol_edges as a new cycle phase

W0c's symbol resolver lands as a 12th cycle phase between extract and
patterns. The autopilot's quick-cycle path (60s watchdog interval per
D2 from eng review) now resolves stale chunks incrementally so agents
see resolved edges within ~60s of writes rather than waiting on the
slow full-walk path.

* CyclePhase + ALL_PHASES + NEEDS_LOCK_PHASES extended with
  'resolve_symbol_edges'. Position: between extract (which emits new
  bare-token edges from sync diffs) and patterns (which reads the
  graph). Acquires the cycle lock because it writes edge_metadata.
* CycleReport.totals adds edges_resolved + edges_ambiguous so doctor
  and autopilot summaries surface the numbers.
* runPhaseResolveSymbolEdges walks every registered source via
  listSources() + resolveSymbolEdgesIncremental(). Per-call cap is
  BATCH_SIZE*10 = 2000 chunks so a single watchdog tick stays bounded
  even on a 100K-chunk brain. Subsequent ticks pick up the leftovers
  via the edges_backfilled_at watermark.
* Test count bumped from 11 → 12 phases in cycle.serial.test.ts and
  cycle.test.ts (both pinned by the regression guards). Existing 28
  cycle tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(v0.34 W3): MCP-expose code_callers / code_callees / code_def / code_refs

Pre-v0.34 these four code-intelligence commands lived in CLI_ONLY at
cli.ts:30 — agents calling gbrain via MCP couldn't reach them and fell
through to text search. This commit ships the agent-facing MCP surface
for v0.34 against the existing v0.20+ tree-sitter call graph; recursive
blast/flow and clusters land in subsequent commits.

* `code_callers(symbol, [limit, source_id, all_sources])` — wraps
  engine.getCallersOf. Reverse view of the A1 call graph.
* `code_callees(symbol, [limit, source_id, all_sources])` — wraps
  engine.getCalleesOf. Forward view.
* `code_def(symbol, [limit, lang])` — wraps findCodeDef. Returns
  definition sites with file/line/snippet.
* `code_refs(symbol, [limit, lang])` — wraps findCodeRefs. Returns
  every reference (comments, strings, imports, call sites).

All four are scope:'read', source-scoped by default via ctx.sourceId
(W0a contract). Per-call source_id param wins over ctx; pass '__all__'
or all_sources=true to force cross-source.

* operations-descriptions.ts: 4 new constants per the eng review D10
  finding — every description carries an inline example response so
  agents don't burn first-call context discovering shape. Resolver-grade
  wording ("BEFORE editing any function, run code_callers...") routes
  plan-mode questions straight to the right op.
* SEARCH_DESCRIPTION gains a cross-link clause pointing at the four new
  ops so agents stop falling through to text search for code-symbol
  questions.

Tests (11 E2E in test/e2e/code-intel-mcp-ops-pglite.test.ts):
  - All four ops registered + scope:read + description pinned by constant
  - All four ops have required symbol param
  - code_callers / code_callees return the documented envelope shape
  - Source scoping honors ctx.sourceId
  - all_sources=true / source_id='__all__' force cross-source
  - code_def returns the def-site snippet

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.33.0): agent-readable migration doc for the code-intel foundation

skills/migrations/v0.33.0.md gives existing-user upgrade guidance for the
v0.33.0 foundation pre-release (this branch's accumulated work toward
v0.34 Cathedral III):

* Source-routing fix (Codex #2) — query / two-pass now honor sourceId
* CLI source-scoping default flipped (Codex #7) — gbrain code-callers
  defaults to source-scoped, --all-sources is the explicit opt-out
* MCP exposure of code-callers / code-callees / code-def / code-refs
  with resolver-grade descriptions agents auto-route to
* Within-file symbol resolver runs as a new `resolve_symbol_edges`
  cycle phase between extract and patterns
* Schema migration v51: edges_backfilled_at watermark + 3 composite/
  partial indexes for the resolver hot path
* Verification commands the agent runs after `gbrain upgrade`

Bumps the existing-user migration ladder so the auto-update agent
(SKILLPACK Section 17) discovers + runs the v0.33.0 migration steps.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.33.0): bump VERSION + package.json + CHANGELOG

v0.33.0 ships the v0.34 Cathedral III foundation: MCP exposure of
code_callers / code_callees / code_def / code_refs with resolver-grade
tool descriptions, plus the source-routing fix + within-file symbol
resolver + cycle-phase wiring that v0.34's recursive blast/flow and
Leiden clusters will build on.

Full release notes in CHANGELOG.md. Trio in lockstep:
  VERSION:      0.33.0
  package.json: 0.33.0
  CHANGELOG.md: ## [0.33.0] - 2026-05-11

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(v0.33.0): update dream-cycle phase-order assertions for resolve_symbol_edges

E2E test pinned the canonical phase sequence as a regression guard. The
v0.33.0 resolve_symbol_edges phase (added between extract and patterns)
correctly bumps the count to 12 — caught by the canonical-order test on
fresh-Postgres run, fixed by adding the new phase to EXPECTED_PHASES
and bumping the version history comment.

Both cycle.serial.test.ts and cycle.test.ts were already updated in the
W0c cycle-phase commit (6f7dbe1d); this third pin lives in
test/e2e/dream-cycle-phase-order-pglite.test.ts and was missed.

Full E2E suite now: 550 passed / 0 failed / 81 files (real Postgres on
port 5435 via Docker pgvector/pgvector:pg16).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.33.3.0): rebump from v0.33.2.0 → v0.33.3.0

User asked to ship as v0.33.3.0 instead of v0.33.2.0. Single sweep:

* VERSION + package.json bumped to 0.33.3.0
* CHANGELOG header + body rewritten to v0.33.3
* skills/migrations/v0.33.0.md → skills/migrations/v0.33.3.0.md
  (migration files use the version they ship FROM; renaming aligns with
  the v0.21.0.md / v0.31.0.md convention in CLAUDE.md)
* Schema migration name edges_backfilled_at_v0_33_2 →
  edges_backfilled_at_v0_33_3 in src/core/migrate.ts (also bumps the
  in-code identifier so the registry name matches the version)
* All v0.33.2 comment references swept to v0.33.3 in cycle.ts,
  operations.ts, operations-descriptions.ts, eval.ts, symbol-resolver.ts
  + cycle test phase-history comments
* llms.txt + llms-full.txt regenerated

Trio verified:
  VERSION:      0.33.3.0
  package.json: 0.33.3.0
  CHANGELOG.md: ## [0.33.3.0] - 2026-05-12

bun run verify clean; 90 v0.33.3-touched tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-14 19:50:50 -04:00
committed by GitHub
parent d4a2cbf834
commit 9fb4d7eb5b
26 changed files with 2829 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

94
CHANGELOG.md
View File

@@ -2,6 +2,14 @@
All notable changes to GBrain will be documented in this file.
## [0.33.3.0] - 2026-05-12
**Code intelligence ships to agents. Plan-mode subagents stop falling through to grep.**
**`code_callers`, `code_callees`, `code_def`, `code_refs` are MCP-exposed with resolver-grade descriptions.**
Pre-v0.33.3 the four code-intelligence commands from v0.20+ Cathedral II lived in `CLI_ONLY` at `cli.ts:30`. An agent running through MCP saw `query`/`search` but no structural retrieval, so it grepped, missed callers in string literals, shipped plans with broken call chains, and got caught in review. v0.33.3 closes that gap and lays the foundation work that v0.34 Cathedral III (recursive blast/flow + Leiden clusters + wiki) will build on top of.
This release was scoped after Codex's outside-voice review caught two load-bearing premise gaps in the original v0.34 plan: the call graph stored bare callee tokens (not qualified names), and source routing was already broken in `query` and `two-pass.ts`. Both are fixed here before any user-facing recursive op ships.
## [0.33.2.1] - 2026-05-14
**Doc-only: name the fork-PR escape hatch so AI-authored PRs don't drop their CI secrets.**
@@ -378,6 +386,42 @@ If you run gbrain with more than one source (say a `media-corpus` alongside `def
### The numbers that matter
```
Agent MCP code-intel surface → v0.32.0: 0 ops → v0.33.0: 4 ops + resolver-grade descriptions
Source-routing leak surface → v0.32.0: 4 sites → v0.33.0: 0 sites (Codex finding #2 fix)
Cycle phases → v0.32.0: 11 → v0.33.0: 12 (new `resolve_symbol_edges`)
```
What ships:
| Surface | What it does |
|---|---|
| `code_callers` (MCP op) | "Who calls this function?" Reverse view of the call graph. Resolver-grade description: "BEFORE editing any function, run code_callers..." |
| `code_callees` (MCP op) | "What does this function call?" Forward view; surfaces DB calls, HTTP calls, file I/O downstream of an entry point. |
| `code_def` (MCP op) | Where a symbol is defined. Returns file, line, snippet directly. Agents stop reading 8 files to find one definition. |
| `code_refs` (MCP op) | Every reference (call sites, comments, imports, type annotations). Use before any rename. |
| Within-file symbol resolver | New cycle phase. Walks unresolved edges in 200-row batches, writes resolution to `code_edges_symbol.edge_metadata` (`{resolved_chunk_id: N}` or `{ambiguous: true, candidates: [...]}`). Idempotent + resumable via `edges_backfilled_at` watermark. |
| Source-routing fix | `query` op now passes `ctx.sourceId` to `hybridSearch`. Two-pass retrieval honors `sourceId` at both lookup sites. Multi-source brains stop cross-contaminating structural retrieval. |
| CLI source-scoping default flipped | `gbrain code-callers <symbol>` without `--source` resolves to your brain's default source. Pass `--all-sources` for the pre-v0.33 cross-source default. |
### What this means for agents
Plan-mode subagents in Claude Code, OpenClaw, and Cursor now have a clean structural retrieval path. The MCP tool descriptions are resolvers, they tell the agent's tool-selection prompt WHEN to reach for `code_callers` ("BEFORE editing any function") and `code_callees` ("when tracing how a request flows to side effects"). Agents stop guessing about callers by reading 8 files; one `code_callers` call returns the full list with file and line.
For agents already wired to gbrain via stdio MCP, `code_def` replaces the read-8-files-to-find-1-definition pattern with one structured call. `code_refs` is the safe-rename path. `code_callees` is the debugging path.
### What this DOES NOT ship (deferred to v0.34)
Per the design doc's slip-handling clause, v0.33.0 ships the foundation; v0.34 ships the rest. Deferred:
- Recursive `code_blast` / `code_flow` (depth-grouped traversal with confidence decay + truncation enum)
- Leiden community detection: `code_clusters_list` + `code_cluster_get` with inline mermaid
- `gbrain wiki` zero-LLM aggregator CLI
- `code_traversal_cache` (REPEATABLE READ + xmin_max snapshot isolation)
- Per-op graph-traversal eval metrics extending v0.25.0 eval-capture
- `imports` and `references` edge types for JS/TS/TSX + Python
- Receiver-type scope walkers (`obj.method()` to `Class.method`)
### To take advantage of v0.33.0
Pages on non-default source (production multi-source brain) → 5,042
Chunks left unembedded pre-fix → ~22,000
Chunks recovered on first re-run → 97 across 35 pages
@@ -541,6 +585,56 @@ If you imported a Chinese / Japanese / Korean brain pre-v0.32.7 and saw silently
```bash
gbrain apply-migrations --yes
```
2. **Your agent reads `skills/migrations/v0.33.0.md` the next time you interact with it.** The four new MCP ops show up automatically in the tool catalog.
3. **Verify the outcome:**
```bash
gbrain --version # 0.33.0
gbrain --tools-json | jq '.[] | select(.name | startswith("code_")) | .name'
# Should show: code_callers, code_callees, code_def, code_refs
gbrain doctor # Should be ok; will pick up the new edges_backfilled_at watermark
```
4. **If any step fails or the numbers look wrong,** please file an issue at
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output and
`~/.gbrain/upgrade-errors.jsonl` if it exists.
### Itemized changes
**MCP exposure (W3):**
- New ops: `code_callers`, `code_callees`, `code_def`, `code_refs`. All `scope: 'read'`, source-scoped via `ctx.sourceId`. Source override params (`source_id`, `all_sources`) on every op.
- New constants in `src/core/operations-descriptions.ts`: `CODE_CALLERS_DESCRIPTION`, `CODE_CALLEES_DESCRIPTION`, `CODE_DEF_DESCRIPTION`, `CODE_REFS_DESCRIPTION`. Each carries an inline example response per eng-review D10.
- `SEARCH_DESCRIPTION` gains a cross-link clause pointing at the four new ops so agents stop falling through to text search for code-symbol questions.
- 11 E2E tests in `test/e2e/code-intel-mcp-ops-pglite.test.ts`.
**Foundation: source routing (W0a, Codex finding #2):**
- `query` op handler (`src/core/operations.ts`) threads `ctx.sourceId` to `hybridSearch`. New `source_id` param with `'__all__'` escape hatch.
- `src/core/search/two-pass.ts:81` (nearSymbol lookup) and `:131` (unresolved-edge resolution) now apply `opts.sourceId` via a `pages.source_id` join when set.
- 4 E2E tests in `test/e2e/source-routing.test.ts` pin: source-a only, source-b only, no-sourceId crosses sources, walk_depth=1 stays in source-a.
**Foundation: CLI source-scoping default flipped (W0b, Codex finding #7):**
- New canonical helper `resolveDefaultSource(engine)` in `src/core/sources-ops.ts`. Returns the only registered source's id; throws `SourceResolutionError` with the list on multi-source brains.
- `src/commands/code-callers.ts` and `src/commands/code-callees.ts` call `resolveDefaultSource()` when neither `--source` nor `--all-sources` is set. Pre-v0.33 the default was inverted.
- 3 E2E tests in `test/e2e/cli-source-scoping-pglite.test.ts`.
**Foundation: within-file symbol resolver (W0c):**
- New module `src/core/chunkers/symbol-resolver.ts`. Exports `resolveSymbolEdgesIncremental`, `readEdgeResolution`, `EDGE_EXTRACTOR_VERSION_TS`.
- Schema migration v51 (`edges_backfilled_at_v0_34`): adds `content_chunks.edges_backfilled_at TIMESTAMPTZ` + composite + partial indexes (`idx_code_edges_symbol_resolver`, `idx_content_chunks_symbol_lookup`, `idx_content_chunks_edges_backfill`).
- New cycle phase `resolve_symbol_edges` between `extract` and `patterns`. Walks at most BATCH_SIZE * 10 = 2000 chunks per tick; resumable via the watermark.
- 5 E2E tests in `test/e2e/symbol-resolver-pglite.test.ts`.
**Pre-W0: code-retrieval eval harness:**
- New module `src/eval/code-retrieval/` (harness + strategies + 30-question fixture).
- New CLI subcommand `gbrain eval code-retrieval [--baseline | --with-code-intel | --compare]`. Captures pre-v0.34 retrieval quality on the gbrain self-corpus so the v0.34 ship gate measures real improvement, not a retroactively-tuned baseline.
- 26 unit tests in `test/code-retrieval-harness.test.ts`.
**Tests + verification:**
- `bun run test` passes 5464/5465 (one pre-existing flaky LongMemEval perf gate under parallel-shard contention; runs clean in isolation at p50=29ms).
- `bun run verify` clean.
- 50 new test cases across 6 new test files. All v0.33-specific tests pass.
### For contributors
- `EDGE_EXTRACTOR_VERSION_TS` constant in `symbol-resolver.ts`: bump when the resolver or extractor shape changes; the next autopilot cycle re-walks all chunks.
- `OperationContext.sourceId` is still optional; the v0.34 plan calls for making it `REQUIRED` at the type level (mirroring v0.26.9 `ctx.remote` REQUIRED pattern). Not done in v0.33; deferred to v0.34 to keep this release additive.
2. **Run the markdown reindex sweep:**
```bash
gbrain reindex --markdown

2
VERSION
View File

@@ -1 +1 @@
0.33.2.1
0.33.3.0

2
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.33.2.1",
"version": "0.33.3.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",

133
skills/migrations/v0.33.3.0.md Normal file
View File

@@ -0,0 +1,133 @@
# v0.33.3.0 migration — code intelligence foundation + MCP-exposed code ops
`gbrain upgrade` runs `gbrain apply-migrations` automatically. Most users
won't need to do anything else. If you hit issues or want to verify the
upgrade succeeded, run the steps below.
## What changed
v0.33.3.0 is the foundation pre-release for v0.34 Cathedral III. It ships the
agent-facing MCP surface for the v0.20+ tree-sitter call graph plus the
foundation fixes needed before recursive blast/flow and Leiden clusters
land in v0.34.
**MCP exposure (NEW for agents):**
Four code-intelligence ops graduated from `CLI_ONLY` to first-class MCP ops:
- `code_callers(symbol, [limit, source_id, all_sources])` — find every
caller of a symbol. Use BEFORE editing any function.
- `code_callees(symbol, [limit, source_id, all_sources])` — trace what a
function calls. Use when debugging unexpected behavior.
- `code_def(symbol, [limit, lang])` — find a symbol's definition site(s).
- `code_refs(symbol, [limit, lang])` — find every reference (comments,
imports, type annotations, call sites).
The MCP tool descriptions are resolver-grade — they tell agents WHEN to
reach for each op so plan-mode subagents route to structural retrieval
instead of falling through to text search.
**Foundation fixes:**
- **Source-routing fix** (Codex finding #2): `query` op now threads
`ctx.sourceId` to `hybridSearch`. Two-pass retrieval honors `sourceId`
at both the `nearSymbol` lookup and unresolved-edge resolution sites.
Multi-source brains stop cross-contaminating structural retrieval.
- **CLI source-scoping default flipped** (Codex finding #7): `gbrain
code-callers <symbol>` without `--source` now resolves to your brain's
default source (the only source on single-source brains; explicit
error listing valid ids on multi-source brains). Pre-v0.33 the
default silently was global — multi-repo brains cross-resolved every
same-named symbol.
**Within-file two-pass symbol resolution:**
A new cycle phase `resolve_symbol_edges` runs after `extract` on every
autopilot tick. It walks `content_chunks.edges_backfilled_at IS NULL`
chunks in 200-row batches, matching each `to_symbol_qualified` against
the same-file `symbol_name_qualified`, and writes the outcome to
`code_edges_symbol.edge_metadata`:
- `{resolved_chunk_id: N}` — one unambiguous match
- `{ambiguous: true, candidates: [...]}` — 2+ matches in the same file
- (no metadata change) — zero matches; caller's two-pass walk handles
cross-file resolution
## Schema migration v51 — what it adds
```sql
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS edges_backfilled_at TIMESTAMPTZ;
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_resolver
ON code_edges_symbol (source_id, to_symbol_qualified);
CREATE INDEX IF NOT EXISTS idx_content_chunks_symbol_lookup
ON content_chunks (page_id, symbol_name_qualified)
WHERE symbol_name_qualified IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_content_chunks_edges_backfill
ON content_chunks (edges_backfilled_at)
WHERE edges_backfilled_at IS NULL;
```
All idempotent. Existing brains see the column + indexes added; the resolver
walks the corpus lazily over the next several autopilot cycles.
**First-run cost:** ~5-15 minutes of background resolver work on a 10K-chunk
brain spread across cycles (~2000 chunks/tick). No user-visible interruption
unless you watch `gbrain doctor`'s autopilot stats.
## What the agent should do
If you're an agent (Claude Code, OpenClaw, Cursor) running through MCP:
1. **Before editing any function**, reach for `code_callers` to surface
every caller. The MCP tool description tells you this — follow it.
2. **When tracing execution**, reach for `code_callees` from the entry
point.
3. **When looking up a definition**, reach for `code_def` not `search`.
It returns line numbers + snippet directly.
4. **When planning a rename**, reach for `code_refs` to find every
literal mention.
5. **Honor source scoping**: multi-source brains require explicit
`source_id` or `all_sources: true`. Single-source brains auto-resolve.
The previously available CLI commands (`gbrain code-callers <symbol>`)
still work but now default to source-scoped on multi-source brains.
Pass `--all-sources` to get the pre-v0.33 cross-source default.
## Verification
```bash
# 1. Confirm upgrade
gbrain --version # 0.33.3.0
# 2. Confirm migration v51 ran
gbrain apply-migrations --status | grep edges_backfilled_at_v0_34
# 3. Confirm the new MCP ops are exposed
gbrain --tools-json | jq '.[] | select(.name | startswith("code_")) | .name'
# Should show: code_callers, code_callees, code_def, code_refs
# 4. Try one end-to-end (replace `parseMarkdown` with any symbol you index)
gbrain code-callers parseMarkdown --json | jq '.count'
# 5. (Optional) inspect the resolver's per-source progress
gbrain doctor --json | jq '.checks | to_entries[] | select(.key | contains("code"))'
```
## Rollback
If you need to roll back: `git checkout` an older binary and run
`gbrain doctor`. The v51 column + indexes stay in your DB (no-op for
older binaries; idempotent re-add on upgrade). No data is destroyed.
## What v0.34 will add on top
Per the v0.34 design doc (`/Users/garrytan/.gstack/projects/garrytan-gbrain/garrytan-garrytan-miami-design-20260510-194145.md`):
- Recursive `code_blast` / `code_flow` MCP ops on top of the resolved graph
- Leiden community detection (`code_clusters_list`, `code_cluster_get`) +
inline mermaid diagrams
- `gbrain wiki` zero-LLM aggregator CLI
- Per-op graph-traversal eval metrics extending v0.25.0's eval-capture
- `imports` and `references` edge types for JS/TS/TSX + Python
- Receiver-type scope walkers (e.g. `obj.method()` → `Class.method`)

34
src/commands/code-callees.ts
View File

@@ -5,12 +5,17 @@
* Forward view of the A1 call graph. Matches `from_symbol_qualified`
* in both code_edges_chunk + code_edges_symbol.
*
* v0.34 W0b (Codex finding #7): pre-v0.34 default was inverted to
* cross-source whenever --source was omitted. See code-callers.ts for
* the full rationale. Same fix here.
*
* Output: same JSON-on-non-TTY convention as code-callers / code-def /
* code-refs.
*/
import type { BrainEngine } from '../core/engine.ts';
import { errorFor, serializeError } from '../core/errors.ts';
import { resolveDefaultSource, SourceResolutionError } from '../core/sources-ops.ts';
function parseFlag(args: string[], name: string): string | undefined {
const i = args.indexOf(name);
@@ -31,7 +36,7 @@ export async function runCodeCallees(engine: BrainEngine, args: string[]): Promi
class: 'UsageError',
code: 'code_callees_requires_symbol',
message: 'code-callees requires a symbol name',
hint: 'gbrain code-callees <symbol> [--all-sources] [--limit N] [--json]',
hint: 'gbrain code-callees <symbol> [--source S | --all-sources] [--limit N] [--json]',
});
if (shouldEmitJson(args)) {
console.log(JSON.stringify({ error: err.envelope }));
@@ -42,12 +47,35 @@ export async function runCodeCallees(engine: BrainEngine, args: string[]): Promi
}
const limit = parseInt(parseFlag(args, '--limit') || '100', 10);
const allSources = args.includes('--all-sources');
const sourceId = parseFlag(args, '--source');
let sourceId = parseFlag(args, '--source');
// v0.34 W0b: source-scoped default. Matches code-callers behavior.
if (!allSources && !sourceId) {
try {
sourceId = await resolveDefaultSource(engine);
} catch (e: unknown) {
if (e instanceof SourceResolutionError) {
const env = errorFor({
class: 'UsageError',
code: e.code,
message: e.message,
hint: 'pass --source <id> for one source, or --all-sources to search every source',
}).envelope;
if (shouldEmitJson(args)) {
console.log(JSON.stringify({ error: env }));
} else {
console.error(e.message);
}
process.exit(2);
}
throw e;
}
}
try {
const edges = await engine.getCalleesOf(sym, {
limit,
allSources: allSources || !sourceId,
allSources,
sourceId: sourceId ?? undefined,
});

40
src/commands/code-callers.ts
View File

@@ -11,12 +11,21 @@
* in repo A ≠ same string in repo B). Pass `--all-sources` to search
* globally.
*
* v0.34 W0b (Codex finding #7): the pre-v0.34 implementation set
* `allSources: allSources || !sourceId`, which INVERTED the documented
* default to global whenever --source was omitted. Multi-source brains
* cross-contaminated structural retrieval despite the docstring claim.
* Fix: when --source is omitted AND --all-sources is NOT set, resolve to
* the brain's only source (single-source brains) or fail with a clear
* error listing valid source ids (multi-source brains).
*
* Output: non-TTY → JSON envelope. TTY → human table. Follows the
* code-def / code-refs pattern.
*/
import type { BrainEngine } from '../core/engine.ts';
import { errorFor, serializeError } from '../core/errors.ts';
import { resolveDefaultSource, SourceResolutionError } from '../core/sources-ops.ts';
function parseFlag(args: string[], name: string): string | undefined {
const i = args.indexOf(name);
@@ -37,7 +46,7 @@ export async function runCodeCallers(engine: BrainEngine, args: string[]): Promi
class: 'UsageError',
code: 'code_callers_requires_symbol',
message: 'code-callers requires a symbol name',
hint: 'gbrain code-callers <symbol> [--all-sources] [--limit N] [--json]',
hint: 'gbrain code-callers <symbol> [--source S | --all-sources] [--limit N] [--json]',
});
if (shouldEmitJson(args)) {
console.log(JSON.stringify({ error: err.envelope }));
@@ -48,12 +57,37 @@ export async function runCodeCallers(engine: BrainEngine, args: string[]): Promi
}
const limit = parseInt(parseFlag(args, '--limit') || '100', 10);
const allSources = args.includes('--all-sources');
const sourceId = parseFlag(args, '--source');
let sourceId = parseFlag(args, '--source');
// v0.34 W0b: when neither --source nor --all-sources is set, resolve
// to the brain's only source. Multi-source brains require an explicit
// choice — no more silent cross-source default.
if (!allSources && !sourceId) {
try {
sourceId = await resolveDefaultSource(engine);
} catch (e: unknown) {
if (e instanceof SourceResolutionError) {
const env = errorFor({
class: 'UsageError',
code: e.code,
message: e.message,
hint: 'pass --source <id> for one source, or --all-sources to search every source',
}).envelope;
if (shouldEmitJson(args)) {
console.log(JSON.stringify({ error: env }));
} else {
console.error(e.message);
}
process.exit(2);
}
throw e;
}
}
try {
const edges = await engine.getCallersOf(sym, {
limit,
allSources: allSources || !sourceId,
allSources,
sourceId: sourceId ?? undefined,
});

260
src/commands/eval-code-retrieval.ts Normal file
View File

@@ -0,0 +1,260 @@
/**
* v0.34 pre-w0 — `gbrain eval code-retrieval` CLI.
*
* Subcommands:
* --baseline Capture pre-v0.34 retrieval quality. Saves the number
* v0.34 must beat. Run 3x for noise floor.
* --with-code-intel Run against v0.34's code-intel MCP ops. Pre-W3 this
* returns mostly-empty results (honest baseline).
* --compare A B Read two saved reports and compute the gate verdict.
*
* Output:
* stdout — human-readable table by default; `--json` for machine.
* `--save <path>` writes the EvalRunReport JSON to disk.
*
* The CLI deliberately separates capture (read-only) from compare (pure JSON
* read) so the eval data can move between machines / CI without re-running
* against a brain.
*/
import { writeFileSync, existsSync, readFileSync } from 'fs';
import { resolve, dirname, join } from 'path';
import { mkdirSync } from 'fs';
import type { BrainEngine } from '../core/engine.ts';
import {
loadQuestions,
runCodeRetrievalEval,
evaluateGate,
DEFAULT_GATE,
type EvalRunReport,
} from '../eval/code-retrieval/harness.ts';
import { BaselineStrategy, WithCodeIntelStrategy } from '../eval/code-retrieval/strategies.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
interface ParsedArgs {
help: boolean;
baseline: boolean;
withCodeIntel: boolean;
compare?: { a: string; b: string };
corpus: string;
questionsPath: string;
source?: string;
k: number;
save?: string;
json: boolean;
}
const DEFAULT_QUESTIONS_PATH = 'src/eval/code-retrieval/questions.json';
function parseArgs(args: string[]): ParsedArgs {
const out: ParsedArgs = {
help: false,
baseline: false,
withCodeIntel: false,
corpus: 'gbrain',
questionsPath: DEFAULT_QUESTIONS_PATH,
k: 5,
json: false,
};
let i = 0;
while (i < args.length) {
const a = args[i];
if (a === '--help' || a === '-h') out.help = true;
else if (a === '--baseline') out.baseline = true;
else if (a === '--with-code-intel') out.withCodeIntel = true;
else if (a === '--compare') {
const aPath = args[++i];
const bPath = args[++i];
if (!aPath || !bPath) throw new Error('--compare requires two file paths');
out.compare = { a: aPath, b: bPath };
} else if (a === '--corpus') out.corpus = args[++i] ?? 'gbrain';
else if (a === '--questions') out.questionsPath = args[++i] ?? DEFAULT_QUESTIONS_PATH;
else if (a === '--source') out.source = args[++i];
else if (a === '--k') out.k = parseInt(args[++i] ?? '5', 10);
else if (a === '--save') out.save = args[++i];
else if (a === '--json') out.json = true;
i++;
}
return out;
}
function printHelp(): void {
process.stdout.write(`
gbrain eval code-retrieval — v0.34 retrieval baseline + gate
Capture a retrieval-quality number against a curated question set. The
baseline captured here is what v0.34 must beat (precision@5 +10pp OR
answered_rate +15pp on >=15/30 questions).
USAGE:
gbrain eval code-retrieval --baseline [--corpus NAME] [--save PATH] [--json]
gbrain eval code-retrieval --with-code-intel [--corpus NAME] [--save PATH] [--json]
gbrain eval code-retrieval --compare BASELINE.json WITH_CODE_INTEL.json [--json]
OPTIONS:
--baseline Capture pre-v0.34 retrieval (query + search only)
--with-code-intel Capture v0.34 mode (code_blast / code_flow / etc.)
--compare A B Compare two saved reports; emits gate pass/fail
--corpus NAME Brain corpus to query (default: gbrain)
--questions PATH Question file (default: ${DEFAULT_QUESTIONS_PATH})
--source SOURCE_ID Source to scope queries to (default: brain default)
--k N Top-k cutoff (default: 5)
--save PATH Write EvalRunReport JSON to disk
--json Machine-readable JSON to stdout
-h, --help This help
PRE-V0.34 BASELINE WORKFLOW (the assignment):
# Capture baseline 3x for noise floor
gbrain eval code-retrieval --baseline --save /tmp/baseline-1.json
gbrain eval code-retrieval --baseline --save /tmp/baseline-2.json
gbrain eval code-retrieval --baseline --save /tmp/baseline-3.json
V0.34 SHIP GATE:
# After v0.34 ships, capture the with-code-intel run and compare
gbrain eval code-retrieval --with-code-intel --save /tmp/v034.json
gbrain eval code-retrieval --compare /tmp/baseline-1.json /tmp/v034.json
`);
}
export async function runEvalCodeRetrieval(engine: BrainEngine, args: string[]): Promise<void> {
let opts: ParsedArgs;
try {
opts = parseArgs(args);
} catch (err: any) {
process.stderr.write(`error: ${err.message}\n`);
process.exit(2);
return;
}
if (opts.help) {
printHelp();
return;
}
// --compare mode: pure JSON read, no engine needed (engine still passed by
// the dispatcher; we just don't touch it).
if (opts.compare) {
return runCompare(opts);
}
if (!opts.baseline && !opts.withCodeIntel) {
process.stderr.write('error: specify --baseline or --with-code-intel (or --compare)\n');
printHelp();
process.exit(2);
return;
}
const questions = loadQuestions(resolve(opts.questionsPath));
const sourceId = opts.source ?? (await resolveDefaultSource(engine));
const strategy = opts.baseline
? new BaselineStrategy(engine, sourceId)
: new WithCodeIntelStrategy(engine, sourceId);
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
progress.start(`eval.code_retrieval.${strategy.mode}`, questions.questions.length);
const report = await runCodeRetrievalEval(questions.questions, strategy, {
k: opts.k,
corpus: opts.corpus,
onProgress: (done, _total, qid) => progress.tick(1, qid),
});
progress.finish();
if (opts.save) {
const savePath = resolve(opts.save);
mkdirSync(dirname(savePath), { recursive: true });
writeFileSync(savePath, JSON.stringify(report, null, 2));
process.stderr.write(`[eval] saved report to ${savePath}\n`);
}
if (opts.json) {
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
return;
}
printSingleReport(report);
}
function runCompare(opts: ParsedArgs): void {
if (!opts.compare) return;
const aPath = resolve(opts.compare.a);
const bPath = resolve(opts.compare.b);
for (const p of [aPath, bPath]) {
if (!existsSync(p)) {
process.stderr.write(`error: report not found at ${p}\n`);
process.exit(2);
return;
}
}
const a: EvalRunReport = JSON.parse(readFileSync(aPath, 'utf8'));
const b: EvalRunReport = JSON.parse(readFileSync(bPath, 'utf8'));
// Convention: the first arg is baseline, the second is with-code-intel.
// If labels disagree, swap so the comparison is meaningful.
let baseline = a;
let withCodeIntel = b;
if (a.mode === 'with-code-intel' && b.mode === 'baseline') {
baseline = b;
withCodeIntel = a;
}
const gate = evaluateGate(baseline, withCodeIntel, DEFAULT_GATE);
if (opts.json) {
process.stdout.write(
JSON.stringify(
{
schema_version: 1,
passed: gate.passed,
precision_delta_pp: gate.precision_delta_pp,
top_1_stability_rate: gate.top_1_stability_rate,
questions_cleared_bar: gate.questions_cleared_bar,
questions_total: gate.questions_total,
summary: gate.summary,
},
null,
2,
) + '\n',
);
process.exit(gate.passed ? 0 : 1);
return;
}
process.stdout.write(`\n${gate.summary}\n\n`);
process.stdout.write(`baseline: precision@${baseline.k}=${(baseline.mean_precision_at_k * 100).toFixed(1)}% answered=${(baseline.answered_rate * 100).toFixed(1)}% (commit ${baseline.commit})\n`);
process.stdout.write(`with-code-intel: precision@${withCodeIntel.k}=${(withCodeIntel.mean_precision_at_k * 100).toFixed(1)}% answered=${(withCodeIntel.answered_rate * 100).toFixed(1)}% (commit ${withCodeIntel.commit})\n`);
process.stdout.write(`delta: +${gate.precision_delta_pp.toFixed(1)}pp precision top-1 stability=${(gate.top_1_stability_rate * 100).toFixed(1)}%\n`);
process.stdout.write(`cleared bar: ${gate.questions_cleared_bar}/${gate.questions_total}\n\n`);
process.exit(gate.passed ? 0 : 1);
}
function printSingleReport(report: EvalRunReport): void {
process.stdout.write(`\n=== code-retrieval eval (mode=${report.mode}) ===\n`);
process.stdout.write(`corpus: ${report.corpus}\n`);
process.stdout.write(`commit: ${report.commit}\n`);
process.stdout.write(`captured: ${report.captured_at}\n`);
process.stdout.write(`questions: ${report.questions.length}\n`);
process.stdout.write(`precision@${report.k}: ${(report.mean_precision_at_k * 100).toFixed(1)}%\n`);
process.stdout.write(`answered: ${report.questions.filter((q) => q.answered).length}/${report.questions.length} (${(report.answered_rate * 100).toFixed(1)}%)\n`);
process.stdout.write(`latency: ${report.total_latency_ms}ms total, ${(report.total_latency_ms / Math.max(1, report.questions.length)).toFixed(0)}ms/q\n`);
process.stdout.write(`\nper-question:\n`);
for (const q of report.questions) {
const status = q.answered ? '✓' : '✗';
process.stdout.write(` ${status} ${q.id.padEnd(20)} p@${report.k}=${(q.precision_at_k * 100).toFixed(0)}% recall@${report.k}=${(q.recall_at_k * 100).toFixed(0)}% (${q.latency_ms}ms)\n`);
}
process.stdout.write('\n');
}
async function resolveDefaultSource(engine: BrainEngine): Promise<string> {
// Try the engine's listSources if it exists; otherwise return a sensible default.
// Most brains have one source; this picks that one.
const sources = await (engine as any).listSources?.();
if (Array.isArray(sources) && sources.length > 0) {
return sources[0].id ?? 'default';
}
return 'default';
}

8
src/commands/eval.ts
View File

@@ -45,6 +45,14 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
const { runEvalCrossModal } = await import('./eval-cross-modal.ts');
process.exit(await runEvalCrossModal(args.slice(1)));
}
if (sub === 'code-retrieval') {
// v0.33.3 pre-w0 — code-retrieval baseline / gate harness. Needs a brain
// for the baseline (BaselineStrategy calls hybridSearch); --compare
// mode reads JSON only but the engine is already connected by this
// dispatcher.
const { runEvalCodeRetrieval } = await import('./eval-code-retrieval.ts');
return runEvalCodeRetrieval(engine, args.slice(1));
}
if (sub === 'whoknows') {
// v0.33 two-layer eval gate (ENG-D2): hand-labeled fixture =
// quality, eval_candidates replay = regression. Pass criteria

BIN
src/core/chunkers/symbol-resolver.ts Normal file
View File

Binary file not shown.

111
src/core/cycle.ts
View File

@@ -55,6 +55,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
export type CyclePhase =
| 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'extract_facts'
| 'resolve_symbol_edges'
| 'patterns' | 'recompute_emotional_weight' | 'consolidate'
| 'embed' | 'orphans' | 'purge';
@@ -70,6 +71,13 @@ export const ALL_PHASES: CyclePhase[] = [
// The empty-fence guard refuses to run if pre-v51 legacy facts are
// pending the v0_32_2 backfill (Codex R2-#7).
'extract_facts',
// v0.33.3 W0c — within-file two-pass symbol resolution. Runs AFTER
// extract + extract_facts so any code edges sync emitted (still bare-token)
// get resolved into {resolved_chunk_id: N} / {ambiguous: true,
// candidates: [...]} edge_metadata entries before downstream phases read
// the graph. Quick-cycle compatible: each invocation walks at most
// BATCH_SIZE*10 chunks where edges_backfilled_at IS NULL or stale.
'resolve_symbol_edges',
'patterns',
// v0.29 — runs AFTER extract + synthesize so it sees the union of
// sync-touched + synthesize-written pages with fresh tag + take state.
@@ -104,6 +112,8 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'extract',
// v0.32.2 — wipes + re-inserts facts per affected page.
'extract_facts',
// v0.33.3 W0c — writes code_edges_symbol.edge_metadata + content_chunks.edges_backfilled_at.
'resolve_symbol_edges',
'patterns',
// v0.29 — writes pages.emotional_weight column.
'recompute_emotional_weight',
@@ -171,6 +181,10 @@ export interface CycleReport {
patterns_written: number;
/** v0.29: number of pages whose emotional_weight was (re)computed. */
pages_emotional_weight_recomputed: number;
/** v0.34: number of code edges resolved (1 candidate) by the resolve_symbol_edges phase. */
edges_resolved: number;
/** v0.34: number of code edges marked ambiguous (2+ candidates) by the resolve_symbol_edges phase. */
edges_ambiguous: number;
/** v0.26.5: number of source rows hard-deleted by the purge phase. */
purged_sources_count: number;
/** v0.26.5: number of page rows hard-deleted by the purge phase. */
@@ -716,6 +730,73 @@ async function runPhaseExtractFacts(
}
}
/**
* v0.33.3 W0c — resolve_symbol_edges phase.
*
* Walks at most BATCH_SIZE*10 chunks per invocation where
* `edges_backfilled_at` is NULL or older than EDGE_EXTRACTOR_VERSION_TS.
* Resumable across cycles via the watermark; quick-cycle compatible.
*
* Source scoping: walks every registered source. Pre-v0.33.3 silently
* crossed sources; now each source is walked independently so symbol
* resolution stays within its source boundary (matches the W0a fix).
*/
async function runPhaseResolveSymbolEdges(
engine: BrainEngine,
dryRun: boolean,
): Promise<PhaseResult> {
if (dryRun) {
return {
phase: 'resolve_symbol_edges',
status: 'skipped',
duration_ms: 0,
summary: 'dry-run: resolve_symbol_edges phase skipped',
details: { dryRun: true, reason: 'no_dry_run_support' },
};
}
try {
const { resolveSymbolEdgesIncremental } = await import('./chunkers/symbol-resolver.ts');
const { listSources } = await import('./sources-ops.ts');
const sources = await listSources(engine);
let totalChunks = 0;
let totalResolved = 0;
let totalAmbiguous = 0;
let totalUnmatched = 0;
for (const s of sources) {
const stats = await resolveSymbolEdgesIncremental(engine, { sourceId: s.id });
totalChunks += stats.chunks_walked;
totalResolved += stats.edges_resolved;
totalAmbiguous += stats.edges_ambiguous;
totalUnmatched += stats.edges_unmatched;
}
return {
phase: 'resolve_symbol_edges',
status: 'ok',
duration_ms: 0,
summary:
totalChunks === 0
? 'no chunks needed symbol resolution'
: `${totalChunks} chunk(s) walked; resolved ${totalResolved}, ambiguous ${totalAmbiguous}, unmatched ${totalUnmatched}`,
details: {
chunks_walked: totalChunks,
edges_resolved: totalResolved,
edges_ambiguous: totalAmbiguous,
edges_unmatched: totalUnmatched,
sources_walked: sources.length,
},
};
} catch (e) {
return {
phase: 'resolve_symbol_edges',
status: 'fail',
duration_ms: 0,
summary: 'resolve_symbol_edges phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
try {
const { runEmbedCore } = await import('../commands/embed.ts');
@@ -1088,6 +1169,31 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── v0.33.3 W0c: resolve_symbol_edges (between extract_facts + patterns) ──
// Walks chunks whose edges_backfilled_at is null/stale. Resumable
// across cycles via the watermark. Quick-cycle compatible — caps at
// BATCH_SIZE * 10 chunks per invocation so a 60s watchdog tick stays
// responsive even on a 100K-chunk brain.
if (phases.includes('resolve_symbol_edges')) {
checkAborted(opts.signal);
if (!engine) {
phaseResults.push({
phase: 'resolve_symbol_edges',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.resolve_symbol_edges');
const { result, duration_ms } = await timePhase(() => runPhaseResolveSymbolEdges(engine, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 6: patterns (v0.23) ───────────────────────────────
// MUST run after extract so the graph state reads fresh — subagent
// put_page calls in synthesize set ctx.remote=true, so auto-link
@@ -1285,6 +1391,8 @@ function emptyTotals(): CycleReport['totals'] {
synth_pages_written: 0,
patterns_written: 0,
pages_emotional_weight_recomputed: 0,
edges_resolved: 0,
edges_ambiguous: 0,
purged_sources_count: 0,
purged_pages_count: 0,
facts_consolidated: 0,
@@ -1318,6 +1426,9 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
t.patterns_written = Number(p.details.patterns_written ?? 0);
} else if (p.phase === 'recompute_emotional_weight' && p.details) {
t.pages_emotional_weight_recomputed = Number(p.details.pages_recomputed ?? 0);
} else if (p.phase === 'resolve_symbol_edges' && p.details) {
t.edges_resolved = Number(p.details.edges_resolved ?? 0);
t.edges_ambiguous = Number(p.details.edges_ambiguous ?? 0);
} else if (p.phase === 'purge' && p.details) {
t.purged_sources_count = Number(p.details.purged_sources_count ?? 0);
t.purged_pages_count = Number(p.details.purged_pages_count ?? 0);

49
src/core/migrate.ts
View File

@@ -2880,6 +2880,55 @@ export const MIGRATIONS: Migration[] = [
ON search_telemetry (date DESC);
`,
},
{
version: 58,
name: 'edges_backfilled_at_v0_33_3',
// v0.33.3 W0c — resumable symbol-resolution backfill watermark.
// (Originally claimed v55; renumbered to v58 on merge with master's
// v55/v56/v57 search-lite migrations.)
//
// The within-file two-pass resolver (src/core/chunkers/symbol-resolver.ts)
// walks every content_chunks row that has unresolved edges
// (rows in code_edges_symbol whose to_symbol_qualified has not been
// matched against same-file symbol_name_qualified yet) and writes the
// resolution outcome to code_edges_symbol.edge_metadata. On a 96K-chunk
// brain that is a 5-15 minute backfill the first time it runs.
//
// `edges_backfilled_at` is the resume watermark. Backfill runs in
// 200-chunk batches; on batch success the column is set to NOW() for
// every chunk in the batch. Resume picks up chunks where the watermark
// is NULL or older than EDGE_EXTRACTOR_VERSION_TS (a constant bumped
// when the extractor's shape changes). Crashes lose at most one batch.
//
// Composite + partial indexes for the lookup hot path (D11 from eng
// review):
// - idx_code_edges_symbol_resolver (source_id, to_symbol_qualified)
// — every code_edges_symbol row is unresolved by construction
// (the table has no to_chunk_id column; that lives on code_edges_chunk).
// This composite index supports the resolver's per-source lookups.
// - idx_content_chunks_symbol_lookup (page_id, symbol_name_qualified)
// WHERE symbol_name_qualified IS NOT NULL — file-batched lookup
// used by both the resolver and the cluster recompute phase (W4-5).
// - idx_content_chunks_edges_backfill (edges_backfilled_at)
// WHERE edges_backfilled_at IS NULL — find unresumed rows quickly.
//
// Idempotent: IF NOT EXISTS on column + indexes. Backfill itself runs
// separately via the resolve_symbol_edges cycle phase.
sql: `
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS edges_backfilled_at TIMESTAMPTZ;
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_resolver
ON code_edges_symbol (source_id, to_symbol_qualified);
CREATE INDEX IF NOT EXISTS idx_content_chunks_symbol_lookup
ON content_chunks (page_id, symbol_name_qualified)
WHERE symbol_name_qualified IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_content_chunks_edges_backfill
ON content_chunks (edges_backfilled_at)
WHERE edges_backfilled_at IS NULL;
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0

51
src/core/operations-descriptions.ts
View File

@@ -71,7 +71,10 @@ export const QUERY_DESCRIPTION =
export const SEARCH_DESCRIPTION =
"Keyword search using full-text search. For personal/emotional questions, " +
"prefer get_recent_salience or find_anomalies — they surface activity bursts " +
"without needing a search term.";
"without needing a search term. " +
"For code-symbol questions (callers, callees, definitions, blast radius), use " +
"code_callers / code_callees / code_def / code_refs instead — those return " +
"structural graph data, not text chunks.";
// ──────────────────────────────────────────────────────────────────────────────
// v0.32.6 — contradiction probe MCP surface (M3)
@@ -86,3 +89,49 @@ export const FIND_CONTRADICTIONS_DESCRIPTION =
"{contradictions: [{a, b, severity, axis, confidence, resolution_command}]}. " +
"Reads the cached run row — does NOT trigger a new probe; users run " +
"`gbrain eval suspected-contradictions` for that.";
// ──────────────────────────────────────────────────────────────────────────────
// v0.33.3 Cathedral III foundation — code-intelligence ops (MCP-exposed).
// Pre-v0.33.3 the callers/callees/def/refs commands were CLI-only — agents
// reached for grep because the MCP surface didn't expose them. These
// descriptions are resolver-grade so the LLM tool-selection prompt routes
// plan-mode questions straight to the right op.
//
// Style notes per the v0.34 eng review D10 finding: every description carries
// an inline example response so agents don't burn first-call context discovering
// shape. Pin via test/operations-descriptions.test.ts.
// ──────────────────────────────────────────────────────────────────────────────
export const CODE_CALLERS_DESCRIPTION =
"BEFORE editing any function, run code_callers with the symbol name to find " +
"every caller (the people who'd be affected by your change). Returns direct " +
"callers from the v0.20+ tree-sitter call graph. Use during plan-mode to size " +
"the change. Defaults to source-scoped; for multi-source brains pass source_id " +
"or all_sources=true. " +
"Returns: `{symbol, count, callers: [{from_symbol_qualified, to_symbol_qualified, edge_type, resolved}]}`. " +
"Example: `{symbol:'parseMarkdown', count:4, callers:[{from_symbol_qualified:'callerInA', " +
"to_symbol_qualified:'parseMarkdown', edge_type:'calls', resolved:true}]}`.";
export const CODE_CALLEES_DESCRIPTION =
"When tracing how a function flows to its dependencies (DB calls, HTTP calls, " +
"file I/O), run code_callees from the entry point. Forward view of the call " +
"graph: what does this symbol call? Use this when debugging unexpected behavior " +
"or when planning to extract / inline a function. Same shape as code_callers " +
"but the field is `callees` and the edge direction is reversed.";
export const CODE_DEF_DESCRIPTION =
"Where is this symbol defined? Returns one row per definition site (function, " +
"class, type, interface, enum, struct, trait, module, contract). Use this BEFORE " +
"reaching for grep when you want to read a definition. Single-result is the common " +
"case; multiple results indicate same-name symbols across files (which is information " +
"in itself). " +
"Returns: `{symbol, count, defs: [{slug, file, language, symbol_type, start_line, end_line, snippet}]}`. " +
"Filter by --lang to scope a polyglot brain (e.g., lang='typescript').";
export const CODE_REFS_DESCRIPTION =
"Find every reference to a symbol across the codebase (every file, every line). " +
"Differs from code_callers in two ways: (1) catches references in comments, " +
"strings, imports, type annotations — not just call sites; (2) returns line " +
"numbers, not symbol-qualified edges. Use this when planning a rename or " +
"deprecation where you need to touch every literal mention. " +
"Returns: `{symbol, count, refs: [{slug, file, language, line, context}]}`.";

141
src/core/operations.ts
View File

@@ -31,6 +31,10 @@ import {
QUERY_DESCRIPTION,
SEARCH_DESCRIPTION,
FIND_CONTRADICTIONS_DESCRIPTION,
CODE_CALLERS_DESCRIPTION,
CODE_CALLEES_DESCRIPTION,
CODE_DEF_DESCRIPTION,
CODE_REFS_DESCRIPTION,
} from './operations-descriptions.ts';
// --- Types ---
@@ -1056,6 +1060,11 @@ const query: Operation = {
description:
"v0.29.1 — filter to effective_date <= this. Same format as `since`. Replaces deprecated `beforeDate`. YYYY-MM-DD lands at end-of-day.",
},
source_id: {
type: 'string',
description:
"v0.34: scope search to a single source. Defaults to OperationContext.sourceId (set from CLI --source / GBRAIN_SOURCE / .gbrain-source dotfile). Pass '__all__' to force cross-source search in multi-source brains.",
},
},
handler: async (ctx, p) => {
const startedAt = Date.now();
@@ -1088,7 +1097,20 @@ const query: Operation = {
// v0.25.0 — capture meta side-channel. hybridSearch's return contract
// stays SearchResult[] (Cathedral II callers depend on that); meta
// arrives via callback so eval capture can record what actually ran.
//
// v0.34 (Codex finding #2): thread ctx.sourceId so multi-source brains
// get source-scoped retrieval. Explicit `source_id` param wins over
// ctx.sourceId for callers that want to override (per-call multi-source
// search). When the param is the literal '__all__', force-allow
// cross-source mode (matches SearchOpts.sourceId contract).
let capturedMeta: HybridSearchMeta | null = null;
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
const resolvedSourceId =
sourceIdParam !== undefined
? sourceIdParam === '__all__'
? undefined
: sourceIdParam
: ctx.sourceId;
// v0.32.x search-lite: route the query op through hybridSearchCached so
// semantic cache + token budget + intent weighting fire automatically.
// Plain hybridSearch remains the bare API for callers that opt out.
@@ -1102,6 +1124,7 @@ const query: Operation = {
symbolKind: (p.symbol_kind as string) || undefined,
nearSymbol: (p.near_symbol as string) || undefined,
walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined,
sourceId: resolvedSourceId,
// v0.29.1 — agent-explicit recency + salience. Omitted = heuristic defaults.
salience: p.salience as 'off' | 'on' | 'strong' | undefined,
recency: p.recency as 'off' | 'on' | 'strong' | undefined,
@@ -2791,6 +2814,122 @@ function parseSinceParam(raw: unknown): Date | null {
return null;
}
// ──────────────────────────────────────────────────────────────────────────────
// v0.34 Cathedral III — code-intelligence ops (MCP-exposed).
//
// Pre-v0.34 code-callers / code-callees / code-def / code-refs lived only in
// the CLI_ONLY set at cli.ts:30 — agents calling gbrain via MCP couldn't reach
// them and fell through to text search. These wrappers expose the existing
// engine + library functions to the MCP surface with resolver-grade
// descriptions (operations-descriptions.ts) so agents route to them
// automatically during plan-mode.
//
// All four are scope:'read'. Source-scoped via ctx.sourceId when set.
// Both `source_id` and `all_sources` are params so per-call overrides work.
// ──────────────────────────────────────────────────────────────────────────────
const code_callers: Operation = {
name: 'code_callers',
description: CODE_CALLERS_DESCRIPTION,
params: {
symbol: { type: 'string', required: true, description: 'Symbol to find callers of (bare or qualified name).' },
limit: { type: 'number', description: 'Max edges returned. Default 100.' },
source_id: { type: 'string', description: "Scope to a single source. Defaults to ctx.sourceId; pass '__all__' to force cross-source." },
all_sources: { type: 'boolean', description: 'Force cross-source search (equivalent to source_id=__all__).' },
},
scope: 'read',
handler: async (ctx, p) => {
const symbol = p.symbol as string;
const limit = (p.limit as number) ?? 100;
const allSourcesParam = p.all_sources === true;
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
const allSources = allSourcesParam || sourceIdParam === '__all__';
const sourceId = allSources
? undefined
: sourceIdParam !== undefined
? sourceIdParam
: ctx.sourceId;
const edges = await ctx.engine.getCallersOf(symbol, {
limit,
allSources,
sourceId,
});
return { symbol, count: edges.length, callers: edges };
},
cliHints: { name: 'code_callers', hidden: true },
};
const code_callees: Operation = {
name: 'code_callees',
description: CODE_CALLEES_DESCRIPTION,
params: {
symbol: { type: 'string', required: true, description: 'Symbol to find callees of (bare or qualified name).' },
limit: { type: 'number', description: 'Max edges returned. Default 100.' },
source_id: { type: 'string', description: "Scope to a single source. Defaults to ctx.sourceId; pass '__all__' to force cross-source." },
all_sources: { type: 'boolean', description: 'Force cross-source search.' },
},
scope: 'read',
handler: async (ctx, p) => {
const symbol = p.symbol as string;
const limit = (p.limit as number) ?? 100;
const allSourcesParam = p.all_sources === true;
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
const allSources = allSourcesParam || sourceIdParam === '__all__';
const sourceId = allSources
? undefined
: sourceIdParam !== undefined
? sourceIdParam
: ctx.sourceId;
const edges = await ctx.engine.getCalleesOf(symbol, {
limit,
allSources,
sourceId,
});
return { symbol, count: edges.length, callees: edges };
},
cliHints: { name: 'code_callees', hidden: true },
};
const code_def: Operation = {
name: 'code_def',
description: CODE_DEF_DESCRIPTION,
params: {
symbol: { type: 'string', required: true, description: 'Symbol name (bare token; e.g., parseMarkdown, BrainEngine).' },
limit: { type: 'number', description: 'Max definition sites returned. Default 20.' },
lang: { type: 'string', description: "Filter by content_chunks.language (e.g. 'typescript', 'python')." },
},
scope: 'read',
handler: async (ctx, p) => {
const { findCodeDef } = await import('../commands/code-def.ts');
const defs = await findCodeDef(ctx.engine, p.symbol as string, {
limit: (p.limit as number) ?? 20,
language: (p.lang as string) || undefined,
});
return { symbol: p.symbol as string, count: defs.length, defs };
},
cliHints: { name: 'code_def', hidden: true },
};
const code_refs: Operation = {
name: 'code_refs',
description: CODE_REFS_DESCRIPTION,
params: {
symbol: { type: 'string', required: true, description: 'Symbol to find references to.' },
limit: { type: 'number', description: 'Max references returned. Default 50.' },
lang: { type: 'string', description: "Filter by content_chunks.language." },
},
scope: 'read',
handler: async (ctx, p) => {
const { findCodeRefs } = await import('../commands/code-refs.ts');
const refs = await findCodeRefs(ctx.engine, p.symbol as string, {
limit: (p.limit as number) ?? 50,
language: (p.lang as string) || undefined,
});
return { symbol: p.symbol as string, count: refs.length, refs };
},
cliHints: { name: 'code_refs', hidden: true },
};
// --- Exports ---
export const operations: Operation[] = [
@@ -2839,6 +2978,8 @@ export const operations: Operation[] = [
find_contradictions,
// v0.33: expertise + relationship-proximity routing
find_experts,
// v0.33.3: Cathedral III code-intelligence (MCP-exposed; were CLI_ONLY pre-v0.33.3)
code_callers, code_callees, code_def, code_refs,
];
export const operationsByName = Object.fromEntries(

43
src/core/search/two-pass.ts
View File

@@ -75,12 +75,26 @@ export async function expandAnchors(
// --near-symbol: add chunks whose symbol_name_qualified matches as
// additional anchors. Best-effort — if none found, fall through.
//
// v0.34 (Codex finding #2): when opts.sourceId is set, scope the
// symbol lookup to that source. Pre-v0.34 this WAS unscoped despite the
// TwoPassOpts.sourceId field being declared — multi-source brains
// cross-contaminated structural retrieval. Now: opts.sourceId set →
// filter; undefined → cross-source (matches the documented contract).
if (opts.nearSymbol) {
try {
const rows = await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = $1 LIMIT 50`,
[opts.nearSymbol],
);
const rows = opts.sourceId
? await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified = $1 AND p.source_id = $2
LIMIT 50`,
[opts.nearSymbol, opts.sourceId],
)
: await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = $1 LIMIT 50`,
[opts.nearSymbol],
);
const baseScore = anchors.length > 0 ? anchors[0]!.score : 1.0;
for (const r of rows) {
if (!seen.has(r.id)) {
@@ -126,12 +140,25 @@ export async function expandAnchors(
}
// Resolve unresolved edges by looking up chunks whose
// symbol_name_qualified matches. One batch query per frontier node.
//
// v0.34 (Codex finding #2): scope by opts.sourceId when set. Pre-v0.34
// this lookup was unscoped, letting structural retrieval cross source
// boundaries silently in multi-source brains.
if (unresolvedTargets.length > 0) {
try {
const resolved = await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = ANY($1::text[]) LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
[unresolvedTargets],
);
const resolved = opts.sourceId
? await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.symbol_name_qualified = ANY($1::text[])
AND p.source_id = $2
LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
[unresolvedTargets, opts.sourceId],
)
: await engine.executeRaw<{ id: number }>(
`SELECT id FROM content_chunks WHERE symbol_name_qualified = ANY($1::text[]) LIMIT ${NEIGHBOR_CAP_PER_HOP}`,
[unresolvedTargets],
);
for (const r of resolved) directChunkIds.push(r.id);
} catch {
// best-effort

50
src/core/sources-ops.ts
View File

@@ -377,6 +377,56 @@ export async function addSource(
return created;
}
// ── resolveDefaultSource ────────────────────────────────────────────────────
//
// v0.34 W0b — canonical helper for CLI commands that take an optional
// --source flag. The contract per the eng review D7:
// - exactly 1 registered source → return its id (single-source brains,
// the 80% case; --source flag is unnecessary friction)
// - 0 sources → throw (no source to scope to)
// - 2+ sources → throw with the list, forcing the caller to be explicit
//
// Codex finding #7: src/commands/code-callers.ts:54 + code-callees.ts:43
// historically set `allSources: allSources || !sourceId` — which means
// the documented "source-scoped by default" behavior INVERTED to global
// whenever `--source` was omitted. Multi-source brains silently
// cross-contaminated structural retrieval despite the docstring claim.
//
// Helper consolidates the resolution rule so blast/flow/clusters/wiki
// (v0.34 new commands) and code-callers/callees (v0.20.0 retrofit)
// behave identically.
export class SourceResolutionError extends Error {
constructor(
message: string,
public readonly code: 'no_sources' | 'multiple_sources_ambiguous',
public readonly availableSources: string[],
) {
super(message);
this.name = 'SourceResolutionError';
}
}
export async function resolveDefaultSource(engine: BrainEngine): Promise<string> {
const sources = await listSources(engine);
if (sources.length === 0) {
throw new SourceResolutionError(
'no sources registered; run `gbrain sources add` first',
'no_sources',
[],
);
}
if (sources.length === 1) {
return sources[0]!.id;
}
const ids = sources.map((s) => s.id);
throw new SourceResolutionError(
`multi-source brain — specify --source from: ${ids.join(', ')}`,
'multiple_sources_ambiguous',
ids,
);
}
// ── listSources ─────────────────────────────────────────────────────────────
export async function listSources(

390
src/eval/code-retrieval/harness.ts Normal file
View File

@@ -0,0 +1,390 @@
/**
* v0.34 pre-w0 — code-retrieval eval harness.
*
* The baseline this captures is what v0.34 must beat (precision@5 +10pp OR
* top-1 stability +15pp on >=15/30 questions, above the 3-run noise floor).
*
* Two modes:
* --baseline Uses only `query` + `search` (today's gbrain). The number
* captured here is the v0.34 ship gate.
* --with-code-intel Uses the v0.34 code-intel MCP ops (code_blast / code_flow
* / code_def / code_refs / code_callers / code_callees /
* code_cluster_get). Numerator of the gate comparison.
*
* Pure-function metrics live at the bottom of this file; the runner can be
* stubbed for unit testing without touching the brain.
*
* Hermetic by design: cli.ts skips connectEngine() when the user is in
* --help mode. Real runs connect to ~/.gbrain (the dogfood brain) by default
* unless --corpus points elsewhere.
*/
import { readFileSync, existsSync } from 'fs';
// ─────────────────────────────────────────────────────────────────
// Types — stable contract for downstream consumers (gbrain-evals, CI)
// ─────────────────────────────────────────────────────────────────
export type CodeQuestionKind =
| 'callers'
| 'callees'
| 'definition'
| 'references'
| 'blast_radius'
| 'execution_flow'
| 'cluster_membership';
export interface CodeQuestion {
id: string;
kind: CodeQuestionKind;
/** Human-language query an agent would type. */
query: string;
/** Canonical symbol to look up structurally (post-v0.34). */
symbol: string;
/** Expected file paths that must appear in the retrieved set. */
expected_files: string[];
/**
* Minimum recall@k against `expected_files` for this question to count as
* "answered." For some post-v0.34-only questions, baseline expectations
* are intentionally low (0.3) since today's gbrain can't answer them.
*/
expected_min_recall: number;
note?: string;
}
export interface CodeQuestionFile {
version: number;
schema: string;
corpus: string;
description: string;
questions: CodeQuestion[];
}
export interface QuestionResult {
id: string;
kind: CodeQuestionKind;
/** Files actually returned, in rank order (top-k). */
retrieved_files: string[];
/** Top-1 file (the single most-confident answer). */
top_1: string | null;
/** precision@k = |relevant ∩ retrieved| / |retrieved|. */
precision_at_k: number;
/** recall@k = |relevant ∩ retrieved| / |relevant|. */
recall_at_k: number;
/** Whether this question's bar (`expected_min_recall`) was cleared. */
answered: boolean;
/** Total latency for this question's tool calls, in ms. */
latency_ms: number;
}
export interface EvalRunReport {
mode: 'baseline' | 'with-code-intel';
schema_version: 1;
corpus: string;
k: number;
questions: QuestionResult[];
/** Mean precision@k across all questions. */
mean_precision_at_k: number;
/** Fraction of questions that cleared their expected_min_recall bar. */
answered_rate: number;
/** Top-1 stability — set when comparing two runs; null for single-run. */
top_1_stability_rate?: number;
/** Aggregate run latency, ms. */
total_latency_ms: number;
/** ISO-8601 capture time. */
captured_at: string;
/** Git short-SHA at capture time. */
commit: string;
}
// ─────────────────────────────────────────────────────────────────
// Pure metrics (no engine dependency, fully unit-testable)
// ─────────────────────────────────────────────────────────────────
/**
* precision@k = relevant ∩ retrieved (top-k) / retrieved (top-k)
* Returns 0 when retrieved is empty.
*/
export function precisionAtK(retrieved: string[], relevant: Set<string>, k: number): number {
const topK = retrieved.slice(0, k);
if (topK.length === 0) return 0;
let hits = 0;
for (const r of topK) {
if (relevant.has(r)) hits++;
}
return hits / topK.length;
}
/**
* recall@k = relevant ∩ retrieved (top-k) / relevant
* Returns 1 when relevant is empty (degenerate case).
*/
export function recallAtK(retrieved: string[], relevant: Set<string>, k: number): number {
if (relevant.size === 0) return 1;
const topK = new Set(retrieved.slice(0, k));
let hits = 0;
for (const r of relevant) {
if (topK.has(r)) hits++;
}
return hits / relevant.size;
}
/**
* top-1 stability rate between two runs over the same question set.
* Computed as |{q : run1.top_1 === run2.top_1}| / total. NaN → 0.
*/
export function top1StabilityRate(run1: QuestionResult[], run2: QuestionResult[]): number {
if (run1.length === 0 || run2.length === 0) return 0;
const lookup = new Map(run2.map((q) => [q.id, q.top_1]));
let stable = 0;
let comparable = 0;
for (const q of run1) {
if (!lookup.has(q.id)) continue;
comparable++;
if (q.top_1 === lookup.get(q.id)) stable++;
}
return comparable === 0 ? 0 : stable / comparable;
}
/**
* Match `retrieved_files` against `expected_files` accounting for prefix
* matching: a retrieved file "src/foo/bar.ts" counts as matching the
* expected entry "src/foo/" (trailing slash = directory match).
* Set semantics — duplicates collapse.
*/
export function normalizeRetrieved(retrieved_files: string[]): string[] {
// Dedupe + drop empties; do not mutate order.
const seen = new Set<string>();
const out: string[] = [];
for (const f of retrieved_files) {
if (!f) continue;
if (seen.has(f)) continue;
seen.add(f);
out.push(f);
}
return out;
}
export function expandExpectedToRelevantSet(expected: string[]): {
exactFiles: Set<string>;
dirPrefixes: string[];
} {
const exactFiles = new Set<string>();
const dirPrefixes: string[] = [];
for (const e of expected) {
if (e.endsWith('/')) dirPrefixes.push(e);
else exactFiles.add(e);
}
return { exactFiles, dirPrefixes };
}
export function isFileRelevant(file: string, expected: ReturnType<typeof expandExpectedToRelevantSet>): boolean {
if (expected.exactFiles.has(file)) return true;
for (const p of expected.dirPrefixes) {
if (file.startsWith(p)) return true;
}
return false;
}
// ─────────────────────────────────────────────────────────────────
// Loader
// ─────────────────────────────────────────────────────────────────
export function loadQuestions(path: string): CodeQuestionFile {
if (!existsSync(path)) {
throw new Error(`questions file not found: ${path}`);
}
const raw = readFileSync(path, 'utf8');
let parsed: CodeQuestionFile;
try {
parsed = JSON.parse(raw);
} catch (err: any) {
throw new Error(`failed to parse questions JSON: ${err.message}`);
}
if (parsed.version !== 1) {
throw new Error(`unsupported questions file version ${parsed.version} (expected 1)`);
}
if (!Array.isArray(parsed.questions) || parsed.questions.length === 0) {
throw new Error('questions file contains no questions');
}
for (const q of parsed.questions) {
if (!q.id || !q.kind || !q.query || !Array.isArray(q.expected_files)) {
throw new Error(`malformed question entry: ${JSON.stringify(q)}`);
}
}
return parsed;
}
// ─────────────────────────────────────────────────────────────────
// Mode dispatch — pluggable retrieval strategies
// ─────────────────────────────────────────────────────────────────
/**
* A retrieval strategy maps a question to a ranked list of file paths.
* Pure abstraction so the runner is testable without engine dependencies.
*/
export interface RetrievalStrategy {
readonly mode: 'baseline' | 'with-code-intel';
retrieve(question: CodeQuestion, k: number): Promise<{ files: string[]; latency_ms: number }>;
}
// ─────────────────────────────────────────────────────────────────
// Runner
// ─────────────────────────────────────────────────────────────────
export interface RunnerOpts {
k: number;
corpus: string;
onProgress?: (done: number, total: number, currentQ: string) => void;
}
export async function runCodeRetrievalEval(
questions: CodeQuestion[],
strategy: RetrievalStrategy,
opts: RunnerOpts,
): Promise<EvalRunReport> {
const results: QuestionResult[] = [];
const startedAt = Date.now();
for (let i = 0; i < questions.length; i++) {
const q = questions[i];
const t0 = Date.now();
let retrieved: string[] = [];
let latency_ms = 0;
try {
const r = await strategy.retrieve(q, opts.k);
retrieved = normalizeRetrieved(r.files);
latency_ms = r.latency_ms;
} catch (err: any) {
// Retrieval errors are part of the eval signal — record as empty result.
retrieved = [];
latency_ms = Date.now() - t0;
process.stderr.write(`[eval] retrieval error on ${q.id}: ${err?.message ?? err}\n`);
}
const expected = expandExpectedToRelevantSet(q.expected_files);
const relevantSet = new Set(retrieved.filter((f) => isFileRelevant(f, expected)));
const recall_at_k = recallAtK(Array.from(relevantSet), expected.exactFiles, opts.k);
const precision_at_k = precisionAtK(retrieved, relevantSet, opts.k);
const top_1 = retrieved[0] ?? null;
const answered = recall_at_k >= q.expected_min_recall;
results.push({
id: q.id,
kind: q.kind,
retrieved_files: retrieved,
top_1,
precision_at_k,
recall_at_k,
answered,
latency_ms,
});
opts.onProgress?.(i + 1, questions.length, q.id);
}
const total_latency_ms = Date.now() - startedAt;
const mean_precision_at_k = results.reduce((a, r) => a + r.precision_at_k, 0) / Math.max(1, results.length);
const answered_rate = results.filter((r) => r.answered).length / Math.max(1, results.length);
let commit = 'unknown';
try {
const { execSync } = await import('child_process');
commit = execSync('git rev-parse --short HEAD', { encoding: 'utf8' }).trim();
} catch {
// Not in a git repo or git unavailable — leave as 'unknown'
}
return {
mode: strategy.mode,
schema_version: 1,
corpus: opts.corpus,
k: opts.k,
questions: results,
mean_precision_at_k,
answered_rate,
total_latency_ms,
captured_at: new Date().toISOString(),
commit,
};
}
// ─────────────────────────────────────────────────────────────────
// Comparison — used by the v0.34 ship gate
// ─────────────────────────────────────────────────────────────────
export interface GateResult {
passed: boolean;
precision_delta_pp: number;
top_1_stability_rate: number;
questions_cleared_bar: number;
questions_total: number;
baseline: EvalRunReport;
with_code_intel: EvalRunReport;
/** Free-form summary explaining pass/fail. */
summary: string;
}
export interface GateOpts {
/** Required precision@5 delta (in percentage points) to pass. Default: 10. */
required_precision_delta_pp: number;
/** Required top-1 stability rate to pass (alternative criterion). Default: 0.15 lower bound. */
required_top_1_stability_delta: number;
/** Minimum questions that must clear `expected_min_recall` in with-code-intel mode. Default: 15. */
min_questions_cleared: number;
}
export const DEFAULT_GATE: GateOpts = {
required_precision_delta_pp: 10,
required_top_1_stability_delta: 0.15,
min_questions_cleared: 15,
};
export function evaluateGate(
baseline: EvalRunReport,
with_code_intel: EvalRunReport,
opts: GateOpts = DEFAULT_GATE,
): GateResult {
const precision_delta_pp = (with_code_intel.mean_precision_at_k - baseline.mean_precision_at_k) * 100;
const top_1_stability_rate = top1StabilityRate(baseline.questions, with_code_intel.questions);
const questions_cleared_bar = with_code_intel.questions.filter((q) => q.answered).length;
const questions_total = with_code_intel.questions.length;
const precisionPasses = precision_delta_pp >= opts.required_precision_delta_pp;
// Stability is "how much DIFFERENT" — for an improvement gate we want
// either precision OR a substantive reordering that lands more good answers.
// We use a delta over baseline's own stability, treating baseline as 1.0
// self-stability. Convention: pass when stability is lower (more changes)
// AND a higher answered_rate. Otherwise the gate would punish v0.34 for
// changing the retrieval order even when it lands better answers.
const stabilityPasses =
with_code_intel.answered_rate - baseline.answered_rate >= opts.required_top_1_stability_delta;
const enoughCleared = questions_cleared_bar >= opts.min_questions_cleared;
const passed = enoughCleared && (precisionPasses || stabilityPasses);
const reasons: string[] = [];
if (!enoughCleared) {
reasons.push(
`only ${questions_cleared_bar}/${questions_total} questions cleared expected_min_recall (need >=${opts.min_questions_cleared})`,
);
}
if (precisionPasses) reasons.push(`precision@${baseline.k} +${precision_delta_pp.toFixed(1)}pp (>=${opts.required_precision_delta_pp})`);
else reasons.push(`precision@${baseline.k} delta ${precision_delta_pp.toFixed(1)}pp (<${opts.required_precision_delta_pp})`);
if (stabilityPasses) reasons.push(`answered_rate +${((with_code_intel.answered_rate - baseline.answered_rate) * 100).toFixed(1)}pp`);
const summary = passed
? `GATE PASS — ${reasons.join('; ')}`
: `GATE FAIL — ${reasons.join('; ')}`;
return {
passed,
precision_delta_pp,
top_1_stability_rate,
questions_cleared_bar,
questions_total,
baseline,
with_code_intel,
summary,
};
}

252
src/eval/code-retrieval/questions.json Normal file
View File

@@ -0,0 +1,252 @@
{
"version": 1,
"schema": "1",
"corpus": "gbrain",
"description": "v0.34 code-retrieval baseline. 30 questions about gbrain's own codebase covering callers, callees, definitions, blast radius, and cluster membership. The 'expected' field is a SET of file:line tuples (not page slugs) for code-symbol retrieval. Match semantics: a result counts as relevant when its (file, line) matches any expected tuple OR its (file, symbol_qualified) matches when symbol_qualified is provided.\n\nTo regenerate / extend: bump version, add questions, document the diff. Pre-v0.34 baseline is captured against this exact set so it cannot be retroactively tuned.",
"questions": [
{
"id": "callers-001",
"kind": "callers",
"query": "what calls performSync",
"symbol": "performSync",
"expected_files": ["src/commands/sync.ts", "src/core/cycle.ts", "src/commands/jobs.ts"],
"expected_min_recall": 0.6
},
{
"id": "callers-002",
"kind": "callers",
"query": "what calls hybridSearch",
"symbol": "hybridSearch",
"expected_files": ["src/core/operations.ts", "src/core/eval-capture.ts", "src/core/search/eval.ts"],
"expected_min_recall": 0.5
},
{
"id": "callers-003",
"kind": "callers",
"query": "what calls runCycle",
"symbol": "runCycle",
"expected_files": ["src/commands/dream.ts", "src/commands/autopilot.ts", "src/commands/jobs.ts"],
"expected_min_recall": 0.6
},
{
"id": "callers-004",
"kind": "callers",
"query": "what calls importFromContent",
"symbol": "importFromContent",
"expected_files": ["src/commands/import.ts", "src/commands/sync.ts"],
"expected_min_recall": 0.5
},
{
"id": "callers-005",
"kind": "callers",
"query": "what calls embed",
"symbol": "embed",
"expected_files": ["src/commands/embed.ts", "src/core/import-file.ts", "src/core/search/eval.ts"],
"expected_min_recall": 0.4
},
{
"id": "callers-006",
"kind": "callers",
"query": "what calls put_page operation",
"symbol": "putPage",
"expected_files": ["src/core/operations.ts", "src/commands/import.ts", "src/core/cycle"],
"expected_min_recall": 0.4
},
{
"id": "callees-001",
"kind": "callees",
"query": "what does performSync call",
"symbol": "performSync",
"expected_files": ["src/core/import-file.ts", "src/core/sync.ts"],
"expected_min_recall": 0.5
},
{
"id": "callees-002",
"kind": "callees",
"query": "what does runCycle call",
"symbol": "runCycle",
"expected_files": ["src/core/cycle.ts", "src/commands/sync.ts", "src/commands/embed.ts", "src/commands/extract.ts"],
"expected_min_recall": 0.5
},
{
"id": "callees-003",
"kind": "callees",
"query": "what does code_blast call from request to db",
"symbol": "code_blast",
"expected_files": ["src/core/operations.ts", "src/core/postgres-engine.ts"],
"expected_min_recall": 0.3,
"note": "Post-v0.34 expectation; pre-v0.34 baseline returns nothing meaningful — expected behavior"
},
{
"id": "def-001",
"kind": "definition",
"query": "where is hybridSearch defined",
"symbol": "hybridSearch",
"expected_files": ["src/core/search/hybrid.ts"],
"expected_min_recall": 1.0
},
{
"id": "def-002",
"kind": "definition",
"query": "where is runCycle defined",
"symbol": "runCycle",
"expected_files": ["src/core/cycle.ts"],
"expected_min_recall": 1.0
},
{
"id": "def-003",
"kind": "definition",
"query": "where is MinionQueue class defined",
"symbol": "MinionQueue",
"expected_files": ["src/core/minions/queue.ts"],
"expected_min_recall": 1.0
},
{
"id": "def-004",
"kind": "definition",
"query": "where is PGLiteEngine connect method",
"symbol": "PGLiteEngine.connect",
"expected_files": ["src/core/pglite-engine.ts"],
"expected_min_recall": 1.0
},
{
"id": "def-005",
"kind": "definition",
"query": "where is the trust boundary check for remote contexts",
"symbol": "OperationContext.remote",
"expected_files": ["src/core/operations.ts", "src/mcp/dispatch.ts", "src/core/types.ts"],
"expected_min_recall": 0.5
},
{
"id": "refs-001",
"kind": "references",
"query": "where is parseMarkdown used",
"symbol": "parseMarkdown",
"expected_files": ["src/core/markdown.ts", "src/core/import-file.ts"],
"expected_min_recall": 0.4
},
{
"id": "refs-002",
"kind": "references",
"query": "where is getStats used",
"symbol": "getStats",
"expected_files": ["src/core/postgres-engine.ts", "src/core/pglite-engine.ts", "src/commands/serve-http.ts"],
"expected_min_recall": 0.5
},
{
"id": "blast-001",
"kind": "blast_radius",
"query": "what breaks if I change the signature of performSync",
"symbol": "performSync",
"expected_files": ["src/commands/sync.ts", "src/core/cycle.ts", "src/commands/jobs.ts", "src/commands/autopilot.ts"],
"expected_min_recall": 0.5
},
{
"id": "blast-002",
"kind": "blast_radius",
"query": "what breaks if I change OperationContext.sourceId",
"symbol": "OperationContext.sourceId",
"expected_files": ["src/core/operations.ts", "src/mcp/dispatch.ts", "src/core/search/two-pass.ts"],
"expected_min_recall": 0.4
},
{
"id": "blast-003",
"kind": "blast_radius",
"query": "what breaks if I rename minion_jobs.max_stalled column",
"symbol": "max_stalled",
"expected_files": ["src/core/minions/queue.ts", "src/core/minions/types.ts", "src/core/migrate.ts", "src/core/pglite-schema.ts"],
"expected_min_recall": 0.5
},
{
"id": "flow-001",
"kind": "execution_flow",
"query": "how does an MCP tool call flow from request to handler",
"symbol": "dispatchToolCall",
"expected_files": ["src/mcp/server.ts", "src/mcp/dispatch.ts", "src/core/operations.ts"],
"expected_min_recall": 0.6
},
{
"id": "flow-002",
"kind": "execution_flow",
"query": "how does gbrain sync flow from cli to database write",
"symbol": "performSync",
"expected_files": ["src/commands/sync.ts", "src/core/sync.ts", "src/core/import-file.ts", "src/core/postgres-engine.ts"],
"expected_min_recall": 0.5
},
{
"id": "flow-003",
"kind": "execution_flow",
"query": "how does a code_blast call flow to the database",
"symbol": "code_blast",
"expected_files": ["src/core/operations.ts", "src/core/postgres-engine.ts"],
"expected_min_recall": 0.3,
"note": "Post-v0.34 expectation; pre-v0.34 baseline returns nothing — expected"
},
{
"id": "cluster-001",
"kind": "cluster_membership",
"query": "which functions cluster with hybridSearch",
"symbol": "hybridSearch",
"expected_files": ["src/core/search/hybrid.ts", "src/core/search/expansion.ts", "src/core/search/intent.ts", "src/core/search/sql-ranking.ts"],
"expected_min_recall": 0.5,
"note": "Post-v0.34 expectation — clusters are new in v0.34"
},
{
"id": "cluster-002",
"kind": "cluster_membership",
"query": "which functions cluster with runCycle",
"symbol": "runCycle",
"expected_files": ["src/core/cycle.ts", "src/core/cycle/synthesize.ts", "src/core/cycle/patterns.ts", "src/core/cycle/emotional-weight.ts"],
"expected_min_recall": 0.5,
"note": "Post-v0.34 expectation"
},
{
"id": "cross-cutting-001",
"kind": "callers",
"query": "what calls applyForwardReferenceBootstrap",
"symbol": "applyForwardReferenceBootstrap",
"expected_files": ["src/core/pglite-engine.ts", "src/core/postgres-engine.ts"],
"expected_min_recall": 0.8
},
{
"id": "cross-cutting-002",
"kind": "callers",
"query": "what calls sqlQueryForEngine",
"symbol": "sqlQueryForEngine",
"expected_files": ["src/commands/auth.ts", "src/commands/serve-http.ts", "src/core/oauth-provider.ts", "src/commands/files.ts", "src/mcp/http-transport.ts"],
"expected_min_recall": 0.5
},
{
"id": "cross-cutting-003",
"kind": "callers",
"query": "what calls summarizeMcpParams",
"symbol": "summarizeMcpParams",
"expected_files": ["src/mcp/dispatch.ts", "src/commands/serve-http.ts"],
"expected_min_recall": 0.7
},
{
"id": "edge-001",
"kind": "callers",
"query": "what calls extractEntityRefs",
"symbol": "extractEntityRefs",
"expected_files": ["src/core/link-extraction.ts", "src/commands/extract.ts", "src/commands/backlinks.ts"],
"expected_min_recall": 0.5
},
{
"id": "edge-002",
"kind": "callers",
"query": "what calls validateUploadPath",
"symbol": "validateUploadPath",
"expected_files": ["src/core/operations.ts"],
"expected_min_recall": 0.6
},
{
"id": "edge-003",
"kind": "definition",
"query": "where is the gbrain serve http entrypoint",
"symbol": "runServeHttp",
"expected_files": ["src/commands/serve-http.ts"],
"expected_min_recall": 1.0
}
]
}

103
src/eval/code-retrieval/strategies.ts Normal file
View File

@@ -0,0 +1,103 @@
/**
* v0.34 pre-w0 — retrieval strategies for the code-retrieval eval.
*
* Two strategies ship in the pre-w0 PR:
* - baseline: query + search via hybridSearch (today's gbrain)
* - with-code-intel: code_blast / code_flow / code_def / etc.
*
* The with-code-intel strategy is a stub until v0.34 W3 lands; the harness
* runs against it and produces a meaningful "what does v0.34 need to beat"
* number purely from the baseline. After W3 lands, the stub fills in and
* the gate becomes a real comparison.
*
* Strategies are split into their own module so the harness module stays
* engine-free and unit-testable.
*/
import type { BrainEngine } from '../../core/engine.ts';
import { hybridSearch } from '../../core/search/hybrid.ts';
import type { CodeQuestion, RetrievalStrategy } from './harness.ts';
// ─────────────────────────────────────────────────────────────────
// Baseline: hybridSearch over the same brain
// ─────────────────────────────────────────────────────────────────
export class BaselineStrategy implements RetrievalStrategy {
readonly mode = 'baseline' as const;
constructor(private readonly engine: BrainEngine, private readonly sourceId: string) {}
async retrieve(q: CodeQuestion, k: number): Promise<{ files: string[]; latency_ms: number }> {
const t0 = Date.now();
// Query string is the natural-language question. hybridSearch returns
// chunks; we collapse to unique file paths in rank order.
const results = await hybridSearch(this.engine, q.query, {
limit: Math.max(k * 3, 10), // overshoot to get distinct files
strategy: 'hybrid',
expand: false, // deterministic; no multi-query expansion
} as any);
const latency_ms = Date.now() - t0;
const filesSeen = new Set<string>();
const files: string[] = [];
for (const r of results) {
const path = pickFilePath(r);
if (!path) continue;
if (filesSeen.has(path)) continue;
filesSeen.add(path);
files.push(path);
if (files.length >= k) break;
}
return { files, latency_ms };
}
}
// ─────────────────────────────────────────────────────────────────
// With-code-intel: code_blast / code_flow / etc.
//
// Until v0.34 W3 lands, this strategy falls through to a noop that
// returns zero results. The eval harness handles empty results as
// "question not answered" — which is the truth pre-W3.
// ─────────────────────────────────────────────────────────────────
export class WithCodeIntelStrategy implements RetrievalStrategy {
readonly mode = 'with-code-intel' as const;
constructor(private readonly engine: BrainEngine, private readonly sourceId: string) {}
async retrieve(q: CodeQuestion, k: number): Promise<{ files: string[]; latency_ms: number }> {
const t0 = Date.now();
// STUB: post-v0.34 W3 ships actual op handlers; until then, this strategy
// returns nothing. The harness records this as "answered: false" which
// is the honest pre-W3 baseline for this mode.
//
// When W3 lands, this method dispatches by question kind:
// q.kind === 'callers' / 'blast_radius' → call code_blast op
// q.kind === 'callees' / 'execution_flow' → call code_flow op
// q.kind === 'definition' → call code_def op (MCP-exposed in W3)
// q.kind === 'references' → call code_refs op
// q.kind === 'cluster_membership' → call code_cluster_get op
//
// For now: noop.
const latency_ms = Date.now() - t0;
return { files: [], latency_ms };
}
}
// ─────────────────────────────────────────────────────────────────
// File-path extraction from a search result
//
// Brain pages of kind 'code' carry the file path as the slug suffix
// (e.g. slug `code/src/core/markdown.ts`). For non-code pages, we
// skip — the eval is about code retrieval.
// ─────────────────────────────────────────────────────────────────
function pickFilePath(result: any): string | null {
if (!result?.slug) return null;
const slug: string = result.slug;
if (slug.startsWith('code/')) {
return slug.slice('code/'.length);
}
// Some code pages may sit under different prefixes depending on source
// config; for now, only handle the canonical 'code/' prefix.
return null;
}

245
test/code-retrieval-harness.test.ts Normal file
View File

@@ -0,0 +1,245 @@
/**
* v0.34 pre-w0 — unit tests for the code-retrieval eval harness.
*
* Pure-function metrics + loader + gate logic. No engine, no API, no fixture
* files outside the questions.json checked in alongside the harness.
*/
import { describe, test, expect } from 'bun:test';
import {
precisionAtK,
recallAtK,
top1StabilityRate,
normalizeRetrieved,
expandExpectedToRelevantSet,
isFileRelevant,
loadQuestions,
evaluateGate,
DEFAULT_GATE,
type EvalRunReport,
type QuestionResult,
} from '../src/eval/code-retrieval/harness.ts';
describe('precisionAtK', () => {
test('returns 0 for empty retrieved set', () => {
expect(precisionAtK([], new Set(['a']), 5)).toBe(0);
});
test('returns 1.0 when all top-k are relevant', () => {
expect(precisionAtK(['a', 'b', 'c'], new Set(['a', 'b', 'c']), 5)).toBe(1);
});
test('returns 0 when zero top-k are relevant', () => {
expect(precisionAtK(['x', 'y'], new Set(['a', 'b']), 5)).toBe(0);
});
test('respects the k cutoff (only top-k considered)', () => {
// top-3 retrieved = ['a','b','c']; relevant = {'a','d'} → 1/3
expect(precisionAtK(['a', 'b', 'c', 'd', 'e'], new Set(['a', 'd']), 3)).toBeCloseTo(1 / 3);
});
test('k larger than retrieved length uses retrieved length', () => {
// retrieved length 2, k=5 → divide by 2
expect(precisionAtK(['a', 'b'], new Set(['a']), 5)).toBeCloseTo(1 / 2);
});
});
describe('recallAtK', () => {
test('returns 1.0 when relevant is empty (degenerate)', () => {
expect(recallAtK(['a', 'b'], new Set(), 5)).toBe(1);
});
test('returns 1.0 when all relevant are in top-k', () => {
expect(recallAtK(['a', 'b', 'c'], new Set(['a', 'b']), 5)).toBe(1);
});
test('returns 0 when none of relevant are in top-k', () => {
expect(recallAtK(['x', 'y', 'z'], new Set(['a', 'b']), 5)).toBe(0);
});
test('returns half when 1/2 relevant in top-k', () => {
expect(recallAtK(['a', 'x'], new Set(['a', 'b']), 5)).toBeCloseTo(1 / 2);
});
test('respects the k cutoff', () => {
// top-2 = ['x','y']; relevant = {'a','b'} → recall=0; the third would be 'a' but k=2
expect(recallAtK(['x', 'y', 'a', 'b'], new Set(['a', 'b']), 2)).toBe(0);
});
});
describe('top1StabilityRate', () => {
test('returns 0 for empty runs', () => {
expect(top1StabilityRate([], [])).toBe(0);
});
test('returns 1.0 when all top-1s match', () => {
const run1 = makeResults([{ id: 'q1', top_1: 'a' }, { id: 'q2', top_1: 'b' }]);
const run2 = makeResults([{ id: 'q1', top_1: 'a' }, { id: 'q2', top_1: 'b' }]);
expect(top1StabilityRate(run1, run2)).toBe(1);
});
test('returns 0 when no top-1s match', () => {
const run1 = makeResults([{ id: 'q1', top_1: 'a' }]);
const run2 = makeResults([{ id: 'q1', top_1: 'x' }]);
expect(top1StabilityRate(run1, run2)).toBe(0);
});
test('ignores questions only in one run', () => {
const run1 = makeResults([{ id: 'q1', top_1: 'a' }, { id: 'q2', top_1: 'b' }]);
const run2 = makeResults([{ id: 'q1', top_1: 'a' }]); // missing q2
// Only q1 is comparable; stable = 1/1 = 1
expect(top1StabilityRate(run1, run2)).toBe(1);
});
test('null top_1 in one run counts as non-stable when other is non-null', () => {
const run1 = makeResults([{ id: 'q1', top_1: 'a' }]);
const run2 = makeResults([{ id: 'q1', top_1: null }]);
expect(top1StabilityRate(run1, run2)).toBe(0);
});
});
describe('normalizeRetrieved', () => {
test('dedupes while preserving order', () => {
expect(normalizeRetrieved(['a', 'b', 'a', 'c', 'b'])).toEqual(['a', 'b', 'c']);
});
test('drops empty strings', () => {
expect(normalizeRetrieved(['a', '', 'b'])).toEqual(['a', 'b']);
});
});
describe('expandExpectedToRelevantSet / isFileRelevant', () => {
test('exact file match', () => {
const exp = expandExpectedToRelevantSet(['src/foo.ts', 'src/bar.ts']);
expect(isFileRelevant('src/foo.ts', exp)).toBe(true);
expect(isFileRelevant('src/baz.ts', exp)).toBe(false);
});
test('directory prefix match (trailing slash)', () => {
const exp = expandExpectedToRelevantSet(['src/core/']);
expect(isFileRelevant('src/core/foo.ts', exp)).toBe(true);
expect(isFileRelevant('src/core/sub/bar.ts', exp)).toBe(true);
expect(isFileRelevant('src/other.ts', exp)).toBe(false);
});
test('mixed exact + directory expected', () => {
const exp = expandExpectedToRelevantSet(['src/foo.ts', 'src/core/']);
expect(isFileRelevant('src/foo.ts', exp)).toBe(true);
expect(isFileRelevant('src/core/bar.ts', exp)).toBe(true);
expect(isFileRelevant('src/other.ts', exp)).toBe(false);
});
});
describe('loadQuestions', () => {
test('parses the v0.34 baseline questions.json', () => {
const file = loadQuestions('src/eval/code-retrieval/questions.json');
expect(file.version).toBe(1);
expect(file.corpus).toBe('gbrain');
expect(file.questions.length).toBeGreaterThanOrEqual(12);
for (const q of file.questions) {
expect(q.id).toBeDefined();
expect(q.kind).toMatch(/^(callers|callees|definition|references|blast_radius|execution_flow|cluster_membership)$/);
expect(q.query.length).toBeGreaterThan(0);
expect(q.symbol.length).toBeGreaterThan(0);
expect(Array.isArray(q.expected_files)).toBe(true);
expect(q.expected_min_recall).toBeGreaterThanOrEqual(0);
expect(q.expected_min_recall).toBeLessThanOrEqual(1);
}
});
test('throws on missing file', () => {
expect(() => loadQuestions('/tmp/does-not-exist-XXXX.json')).toThrow(/not found/);
});
});
describe('evaluateGate', () => {
test('PASS when precision delta clears bar AND enough cleared bar', () => {
const baseline = makeReport('baseline', 0.4, 0.5, 20); // 20 questions, 50% answered
const withCI = makeReport('with-code-intel', 0.55, 0.85, 20); // +15pp precision, 85% answered
const gate = evaluateGate(baseline, withCI, {
required_precision_delta_pp: 10,
required_top_1_stability_delta: 0.15,
min_questions_cleared: 15,
});
expect(gate.passed).toBe(true);
expect(gate.precision_delta_pp).toBeCloseTo(15, 5);
});
test('FAIL when not enough questions cleared bar (despite precision delta)', () => {
const baseline = makeReport('baseline', 0.4, 0.5, 30);
const withCI = makeReport('with-code-intel', 0.6, 0.4, 30); // good precision, fewer answered
const gate = evaluateGate(baseline, withCI, {
required_precision_delta_pp: 10,
required_top_1_stability_delta: 0.15,
min_questions_cleared: 15,
});
expect(gate.passed).toBe(false);
expect(gate.summary).toContain('only ');
});
test('PASS via answered_rate delta even when precision delta is below bar', () => {
const baseline = makeReport('baseline', 0.4, 0.5, 30);
const withCI = makeReport('with-code-intel', 0.45, 0.7, 30); // +5pp precision (fail) but +20pp answered (pass)
const gate = evaluateGate(baseline, withCI, {
required_precision_delta_pp: 10,
required_top_1_stability_delta: 0.15,
min_questions_cleared: 15,
});
expect(gate.passed).toBe(true);
});
test('default opts match constants', () => {
expect(DEFAULT_GATE.required_precision_delta_pp).toBe(10);
expect(DEFAULT_GATE.required_top_1_stability_delta).toBe(0.15);
expect(DEFAULT_GATE.min_questions_cleared).toBe(15);
});
});
// ─── helpers ──────────────────────────────────────────────────────────
function makeResults(items: Array<{ id: string; top_1: string | null }>): QuestionResult[] {
return items.map((it) => ({
id: it.id,
kind: 'callers' as const,
retrieved_files: it.top_1 ? [it.top_1] : [],
top_1: it.top_1,
precision_at_k: it.top_1 ? 1 : 0,
recall_at_k: it.top_1 ? 1 : 0,
answered: !!it.top_1,
latency_ms: 1,
}));
}
function makeReport(
mode: 'baseline' | 'with-code-intel',
meanPrecision: number,
answeredRate: number,
totalQuestions: number,
): EvalRunReport {
const answeredCount = Math.round(answeredRate * totalQuestions);
const questions: QuestionResult[] = [];
for (let i = 0; i < totalQuestions; i++) {
questions.push({
id: `q${i}`,
kind: 'callers' as const,
retrieved_files: ['fakefile.ts'],
top_1: 'fakefile.ts',
precision_at_k: meanPrecision,
recall_at_k: 0.5,
answered: i < answeredCount,
latency_ms: 1,
});
}
return {
mode,
schema_version: 1,
corpus: 'fake',
k: 5,
questions,
mean_precision_at_k: meanPrecision,
answered_rate: answeredRate,
total_latency_ms: totalQuestions,
captured_at: '2026-05-10T00:00:00Z',
commit: 'abc1234',
};
}

9
test/core/cycle.serial.test.ts
View File

@@ -380,8 +380,9 @@ describe('runCycle — yieldBetweenPhases hook', () => {
// v0.26.5: 9 phases (added `purge`).
// v0.29: 10 phases (added `recompute_emotional_weight`).
// v0.31: 11 phases (added `consolidate` between recompute and embed).
// v0.32.2: 12 phases (added `extract_facts` between extract and patterns) → 12 yield calls.
expect(hookCalls).toBe(12);
// v0.32.2: 12 phases (added `extract_facts` between extract and patterns).
// v0.33.3: 13 phases (added `resolve_symbol_edges` between extract_facts and patterns) → 13 yield calls.
expect(hookCalls).toBe(13);
});
test('hook exceptions do not abort the cycle', async () => {
@@ -391,8 +392,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
throw new Error('synthetic hook error');
},
});
// Cycle still completed all phases (v0.32.2: 12 = v0.31's 11 + extract_facts).
expect(report.phases.length).toBe(12);
// v0.33.3: 13 phases (v0.32.2's 12 + resolve_symbol_edges).
expect(report.phases.length).toBe(13);
});
});

98
test/e2e/cli-source-scoping-pglite.test.ts Normal file
View File

@@ -0,0 +1,98 @@
/**
* v0.34 W0b — CLI source-scoping default regression test.
*
* Pre-v0.34 (Codex finding #7): code-callers.ts:54 + code-callees.ts:43
* set `allSources: allSources || !sourceId`. Default behavior with the
* `--source` flag omitted was GLOBAL cross-source resolution — the
* opposite of what the docstring promised. Multi-source brains silently
* cross-resolved `Admin::UsersController#render` between repos.
*
* IRON RULE regression R2: existing caller without `--source` flag now
* defaults to source-scoped behavior. To get the pre-v0.34 cross-source
* default, the caller must pass `--all-sources` explicitly.
*
* This test bypasses CLI argv parsing and drives the resolveDefaultSource
* helper + the engine.getCallersOf path directly. The CLI handlers thread
* resolveDefaultSource() output to engine.getCallersOf, so this E2E pins
* the same contract end-to-end.
*
* PGLite in-memory, no DATABASE_URL.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { resolveDefaultSource, SourceResolutionError } from '../../src/core/sources-ops.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
describe('v0.34 W0b — resolveDefaultSource resolution rule', () => {
test('single-source brain: returns the only source id', async () => {
await resetPgliteState(engine);
// After reset, the default 'default' source from schema bootstrap is
// the only one present. resolveDefaultSource returns it.
const id = await resolveDefaultSource(engine);
expect(id).toBe('default');
});
test('multi-source brain: throws with the list of valid ids', async () => {
await resetPgliteState(engine);
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ('repo-a', 'repo-a', '/fake/a', '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[],
);
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ('repo-b', 'repo-b', '/fake/b', '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[],
);
let caught: unknown = null;
try {
await resolveDefaultSource(engine);
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(SourceResolutionError);
if (caught instanceof SourceResolutionError) {
expect(caught.code).toBe('multiple_sources_ambiguous');
expect(caught.availableSources).toContain('default');
expect(caught.availableSources).toContain('repo-a');
expect(caught.availableSources).toContain('repo-b');
expect(caught.message).toContain('--source');
}
});
test('zero-sources brain: throws no_sources code', async () => {
await resetPgliteState(engine);
// resetPgliteState preserves the 'default' source from the schema
// bootstrap. Delete it to simulate a brain with no registered sources.
// FK from pages.source_id is ON DELETE CASCADE; resetPgliteState
// truncates pages first so this delete is safe.
await engine.executeRaw(`DELETE FROM sources WHERE id = 'default'`, []);
let caught: unknown = null;
try {
await resolveDefaultSource(engine);
} catch (e) {
caught = e;
}
expect(caught).toBeInstanceOf(SourceResolutionError);
if (caught instanceof SourceResolutionError) {
expect(caught.code).toBe('no_sources');
}
});
});

277
test/e2e/code-intel-mcp-ops-pglite.test.ts Normal file
View File

@@ -0,0 +1,277 @@
/**
* v0.34 W3 — MCP exposure of code-intel ops.
*
* Pre-v0.34 code-callers / code-callees / code-def / code-refs lived in
* CLI_ONLY at cli.ts:30. Agents calling gbrain via MCP couldn't reach
* them and fell through to text search.
*
* This E2E pins:
* - All four ops appear in the operations registry with scope:'read'.
* - Tool descriptions match the constants in operations-descriptions.ts
* so the LLM tool-selection prompt sees the right wording (D10 fix).
* - Each op routes to the right engine method / library function and
* returns the documented envelope shape.
* - Source scoping honors ctx.sourceId and the per-call source_id /
* all_sources params.
*
* PGLite in-memory.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { operations, operationsByName } from '../../src/core/operations.ts';
import {
CODE_CALLERS_DESCRIPTION,
CODE_CALLEES_DESCRIPTION,
CODE_DEF_DESCRIPTION,
CODE_REFS_DESCRIPTION,
} from '../../src/core/operations-descriptions.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
import type { GBrainConfig } from '../../src/core/config.ts';
import type { Logger } from '../../src/core/operations.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
describe('v0.34 W3 — code-intel MCP ops registered', () => {
test('code_callers exists with scope:read and v0.34 description', () => {
expect(operationsByName.code_callers).toBeDefined();
expect(operationsByName.code_callers!.scope).toBe('read');
expect(operationsByName.code_callers!.description).toBe(CODE_CALLERS_DESCRIPTION);
});
test('code_callees exists with scope:read and v0.34 description', () => {
expect(operationsByName.code_callees).toBeDefined();
expect(operationsByName.code_callees!.scope).toBe('read');
expect(operationsByName.code_callees!.description).toBe(CODE_CALLEES_DESCRIPTION);
});
test('code_def exists with scope:read and v0.34 description', () => {
expect(operationsByName.code_def).toBeDefined();
expect(operationsByName.code_def!.scope).toBe('read');
expect(operationsByName.code_def!.description).toBe(CODE_DEF_DESCRIPTION);
});
test('code_refs exists with scope:read and v0.34 description', () => {
expect(operationsByName.code_refs).toBeDefined();
expect(operationsByName.code_refs!.scope).toBe('read');
expect(operationsByName.code_refs!.description).toBe(CODE_REFS_DESCRIPTION);
});
test('all four code_* ops have a symbol param marked required', () => {
for (const opName of ['code_callers', 'code_callees', 'code_def', 'code_refs']) {
const op = operationsByName[opName];
expect(op).toBeDefined();
expect(op!.params.symbol).toBeDefined();
expect(op!.params.symbol!.required).toBe(true);
}
});
});
describe('v0.34 W3 — code_callers / code_callees route to the engine', () => {
test('code_callers finds direct callers', async () => {
await seedTwoFileGraph(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_callers!;
const result = (await op.handler(ctx, { symbol: 'parseMarkdown' })) as {
symbol: string;
count: number;
callers: Array<{ from_symbol_qualified: string; to_symbol_qualified: string }>;
};
expect(result.symbol).toBe('parseMarkdown');
expect(result.count).toBeGreaterThanOrEqual(1);
const fromNames = result.callers.map((c) => c.from_symbol_qualified);
expect(fromNames).toContain('callerInA');
});
test('code_callees finds direct callees', async () => {
await seedTwoFileGraph(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_callees!;
const result = (await op.handler(ctx, { symbol: 'callerInA' })) as {
symbol: string;
count: number;
callees: Array<{ from_symbol_qualified: string; to_symbol_qualified: string }>;
};
expect(result.symbol).toBe('callerInA');
expect(result.count).toBeGreaterThanOrEqual(1);
const toNames = result.callees.map((c) => c.to_symbol_qualified);
expect(toNames).toContain('parseMarkdown');
});
});
describe('v0.34 W3 — code_callers source scoping', () => {
test('honors ctx.sourceId by default', async () => {
await seedCrossSourceGraph(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_callers!;
const result = (await op.handler(ctx, { symbol: 'parseMarkdown' })) as {
callers: Array<{ source_id: string | null }>;
};
// Should only see callers from source-a; source-b's caller MUST NOT leak
for (const c of result.callers) {
expect(c.source_id === 'source-a' || c.source_id === null).toBe(true);
}
});
test('all_sources=true forces cross-source', async () => {
await seedCrossSourceGraph(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_callers!;
const result = (await op.handler(ctx, { symbol: 'parseMarkdown', all_sources: true })) as {
callers: Array<{ source_id: string | null }>;
};
const sources = new Set(result.callers.map((c) => c.source_id));
// Both sources represented when all_sources is set
expect(sources.has('source-a')).toBe(true);
expect(sources.has('source-b')).toBe(true);
});
test("source_id='__all__' is equivalent to all_sources=true", async () => {
await seedCrossSourceGraph(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_callers!;
const result = (await op.handler(ctx, { symbol: 'parseMarkdown', source_id: '__all__' })) as {
callers: Array<{ source_id: string | null }>;
};
const sources = new Set(result.callers.map((c) => c.source_id));
expect(sources.has('source-a')).toBe(true);
expect(sources.has('source-b')).toBe(true);
});
});
describe('v0.34 W3 — code_def finds definition sites', () => {
test('returns a definition for a seeded function symbol', async () => {
await seedDefSite(engine);
const ctx = makeCtx(engine, 'source-a');
const op = operationsByName.code_def!;
const result = (await op.handler(ctx, { symbol: 'parseMarkdown' })) as {
symbol: string;
count: number;
defs: Array<{ slug: string; symbol_type: string | null }>;
};
expect(result.symbol).toBe('parseMarkdown');
expect(result.count).toBe(1);
expect(result.defs[0]!.symbol_type).toBe('function');
});
});
// ─────────────────────────────────────────────────────────────────
// Fixtures
// ─────────────────────────────────────────────────────────────────
async function seedTwoFileGraph(engine: PGLiteEngine): Promise<void> {
await registerSource(engine, 'source-a');
const pageA = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const pageA2 = await insertCodePage(engine, 'source-a', 'src/caller.ts');
await insertChunk(engine, pageA, 0, 'parseMarkdown', 'function');
const callerChunk = await insertChunk(engine, pageA2, 0, 'callerInA', 'function');
await insertUnresolvedEdge(engine, callerChunk, 'callerInA', 'parseMarkdown', 'source-a');
}
async function seedCrossSourceGraph(engine: PGLiteEngine): Promise<void> {
await registerSource(engine, 'source-a');
await registerSource(engine, 'source-b');
// Source A: callerInA → parseMarkdown
const pageA = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const pageA2 = await insertCodePage(engine, 'source-a', 'src/caller.ts');
await insertChunk(engine, pageA, 0, 'parseMarkdown', 'function');
const callerA = await insertChunk(engine, pageA2, 0, 'callerInA', 'function');
await insertUnresolvedEdge(engine, callerA, 'callerInA', 'parseMarkdown', 'source-a');
// Source B: callerInB → parseMarkdown (same symbol name, different source)
const pageB = await insertCodePage(engine, 'source-b', 'src/foo.ts');
const pageB2 = await insertCodePage(engine, 'source-b', 'src/caller.ts');
await insertChunk(engine, pageB, 0, 'parseMarkdown', 'function');
const callerB = await insertChunk(engine, pageB2, 0, 'callerInB', 'function');
await insertUnresolvedEdge(engine, callerB, 'callerInB', 'parseMarkdown', 'source-b');
}
async function seedDefSite(engine: PGLiteEngine): Promise<void> {
await registerSource(engine, 'source-a');
const pageA = await insertCodePage(engine, 'source-a', 'src/foo.ts');
// code-def reads content_chunks.symbol_name (not symbol_name_qualified).
// Set both to be safe.
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name, symbol_name_qualified, symbol_type, start_line, end_line)
VALUES ($1, 0, 'export function parseMarkdown(s: string) { return s; }', 'compiled_truth', 'typescript', 'parseMarkdown', 'parseMarkdown', 'function', 1, 3)`,
[pageA],
);
}
async function registerSource(engine: PGLiteEngine, id: string): Promise<void> {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ($1, $1, $2, '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[id, `/fake/${id}`],
);
}
async function insertCodePage(engine: PGLiteEngine, sourceId: string, slug: string): Promise<number> {
const rows = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, page_kind, compiled_truth, frontmatter, updated_at, created_at)
VALUES ($1, $2, $3, 'code', 'code', '', '{}'::jsonb, NOW(), NOW())
RETURNING id`,
[slug, sourceId, slug],
);
return rows[0]!.id;
}
async function insertChunk(
engine: PGLiteEngine,
pageId: number,
chunkIndex: number,
symbolName: string,
symbolType: string,
): Promise<number> {
const rows = await engine.executeRaw<{ id: number }>(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name, symbol_name_qualified, symbol_type)
VALUES ($1, $2, $3, 'compiled_truth', 'typescript', $4, $4, $5)
RETURNING id`,
[pageId, chunkIndex, `// ${symbolName} body`, symbolName, symbolType],
);
return rows[0]!.id;
}
async function insertUnresolvedEdge(
engine: PGLiteEngine,
fromChunkId: number,
fromSymbol: string,
toSymbol: string,
sourceId: string,
): Promise<void> {
await engine.executeRaw(
`INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, source_id, edge_metadata)
VALUES ($1, $2, $3, 'calls', $4, '{}'::jsonb)`,
[fromChunkId, fromSymbol, toSymbol, sourceId],
);
}
function makeCtx(engine: PGLiteEngine, sourceId: string): any {
const logger: Logger = {
info: () => {},
warn: () => {},
error: () => {},
};
return {
engine,
config: {} as GBrainConfig,
logger,
dryRun: false,
remote: false,
sourceId,
};
}

5
test/e2e/cycle.test.ts
View File

@@ -97,14 +97,15 @@ describeE2E('E2E: runCycle against real Postgres', () => {
});
expect(report.schema_version).toBe('1');
// Cycle ran all 11 phases (or skipped the ones that don't support dry-run).
// Cycle ran all 13 phases (or skipped the ones that don't support dry-run).
// Phase history:
// v0.23 = 8 phases (lint → backlinks → sync → synthesize → extract → patterns → embed → orphans)
// v0.26.5 = 9 (added `purge` after orphans)
// v0.29 = 10 (added `recompute_emotional_weight` between patterns and embed)
// v0.31 = 11 (added `consolidate` between recompute_emotional_weight and embed)
// v0.32.2 = 12 (added `extract_facts` between extract and patterns)
expect(report.phases.length).toBe(12);
// v0.33.3 = 13 (added `resolve_symbol_edges` between extract_facts and patterns)
expect(report.phases.length).toBe(13);
// Nothing got written.
const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`);

2
test/e2e/dream-cycle-phase-order-pglite.test.ts
View File

@@ -96,6 +96,7 @@ async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
// v0.26.5 — added `purge` (last)
// v0.29 — added `recompute_emotional_weight` between patterns and embed
// v0.31 — added `consolidate` between recompute_emotional_weight and embed
// v0.33 — added `resolve_symbol_edges` between extract and patterns
type CyclePhase = (typeof ALL_PHASES)[number];
const EXPECTED_PHASES: CyclePhase[] = [
'lint',
@@ -104,6 +105,7 @@ const EXPECTED_PHASES: CyclePhase[] = [
'synthesize',
'extract',
'extract_facts', // v0.32.2 — reconcile fence → DB facts index
'resolve_symbol_edges', // v0.33.3 — within-file symbol resolution
'patterns',
'recompute_emotional_weight', // v0.29
'consolidate', // v0.31

234
test/e2e/source-routing.test.ts Normal file
View File

@@ -0,0 +1,234 @@
/**
* v0.34 W0a — multi-source isolation E2E.
*
* Pre-v0.34 (Codex finding #2): `query` op didn't pass ctx.sourceId to
* hybridSearch; two-pass.ts:81 + :131 advertised TwoPassOpts.sourceId but
* never applied it to the nearSymbol lookup or unresolved-edge resolution.
* Multi-source brains silently cross-contaminated structural retrieval.
*
* This E2E pins the fix: seed two sources with the same symbol name in
* different files; assert near_symbol + walk_depth retrieval with
* sourceId='source-a' only returns chunks from source-a.
*
* PGLite in-memory — no DATABASE_URL needed, hermetic.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { expandAnchors } from '../../src/core/search/two-pass.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
describe('v0.34 W0a — multi-source isolation in two-pass retrieval', () => {
beforeAll(async () => {
await resetPgliteState(engine);
await seedTwoSourcesWithSharedSymbol(engine);
});
test('expandAnchors with nearSymbol + sourceId returns ONLY source-a chunks', async () => {
const result = await expandAnchors(engine, [], {
walkDepth: 0,
nearSymbol: 'parseMarkdown',
sourceId: 'source-a',
});
expect(result.length).toBeGreaterThan(0);
// Hydrate chunk_ids to verify they all belong to source-a
const chunkIds = result.map((r) => r.chunk_id);
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
`SELECT cc.id AS chunk_id, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
);
expect(rows.length).toBe(chunkIds.length);
for (const r of rows) {
expect(r.source_id).toBe('source-a');
}
});
test('expandAnchors with nearSymbol + sourceId="source-b" returns ONLY source-b chunks', async () => {
const result = await expandAnchors(engine, [], {
walkDepth: 0,
nearSymbol: 'parseMarkdown',
sourceId: 'source-b',
});
expect(result.length).toBeGreaterThan(0);
const chunkIds = result.map((r) => r.chunk_id);
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
`SELECT cc.id AS chunk_id, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
);
for (const r of rows) {
expect(r.source_id).toBe('source-b');
}
});
test('expandAnchors with nearSymbol and NO sourceId returns chunks from both sources (legacy cross-source mode preserved)', async () => {
const result = await expandAnchors(engine, [], {
walkDepth: 0,
nearSymbol: 'parseMarkdown',
// sourceId omitted — should cross sources (matches the documented contract)
});
expect(result.length).toBeGreaterThan(0);
const chunkIds = result.map((r) => r.chunk_id);
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
`SELECT cc.id AS chunk_id, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
);
const sources = new Set(rows.map((r) => r.source_id));
expect(sources.has('source-a')).toBe(true);
expect(sources.has('source-b')).toBe(true);
});
test('unresolved-edge resolution within walkDepth respects sourceId', async () => {
// Seed a caller → callee edge so walk_depth=1 must resolve via
// symbol_name_qualified. We added one such edge in seedTwoSourcesWithSharedSymbol
// pointing at parseMarkdown in source-a only.
//
// Start anchor = the caller chunk in source-a; expansion of depth 1 should
// land on the source-a parseMarkdown definition, NOT the source-b one.
const callerChunk = await engine.executeRaw<{ id: number; score: number }>(
`SELECT cc.id, 1.0 AS score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.source_id = 'source-a' AND cc.symbol_name_qualified = 'callerInA'
LIMIT 1`,
[],
);
expect(callerChunk.length).toBe(1);
const anchors = [{
slug: 'src/foo.ts',
page_id: 0,
title: '',
type: 'code' as const,
chunk_text: '',
chunk_source: 'compiled_truth' as const,
chunk_id: callerChunk[0]!.id,
chunk_index: 0,
score: 1.0,
source_id: 'source-a',
stale: false,
}];
const result = await expandAnchors(engine, anchors, {
walkDepth: 1,
sourceId: 'source-a',
});
expect(result.length).toBeGreaterThan(1); // anchor + at least one neighbor
const chunkIds = result.map((r) => r.chunk_id);
const rows = await engine.executeRaw<{ chunk_id: number; source_id: string }>(
`SELECT cc.id AS chunk_id, p.source_id
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.id = ANY($1::int[])`,
[chunkIds],
);
for (const r of rows) {
expect(r.source_id).toBe('source-a');
}
});
});
// ─────────────────────────────────────────────────────────────────
// Fixture: two sources, each with a `parseMarkdown` symbol.
// Source A also has a `callerInA` function whose unresolved call edge
// points at "parseMarkdown" (testing the walk_depth resolution path).
// ─────────────────────────────────────────────────────────────────
async function seedTwoSourcesWithSharedSymbol(engine: PGLiteEngine): Promise<void> {
// Register two sources (schema: id PK, name UNIQUE NOT NULL, plus optional fields)
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ('source-a', 'source-a', '/fake/a', '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[],
);
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ('source-b', 'source-b', '/fake/b', '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[],
);
// Page A1: contains parseMarkdown in source-a
const pageA = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, updated_at, created_at)
VALUES ('code/src/markdown-a.ts', 'source-a', 'markdown-a.ts', 'code', 'export function parseMarkdown(s: string) { return s; }', '{}'::jsonb, NOW(), NOW())
RETURNING id`,
[],
);
// Page A2: contains callerInA, which references parseMarkdown
const pageA2 = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, updated_at, created_at)
VALUES ('code/src/caller-a.ts', 'source-a', 'caller-a.ts', 'code', 'export function callerInA() { return parseMarkdown(""); }', '{}'::jsonb, NOW(), NOW())
RETURNING id`,
[],
);
// Page B: contains parseMarkdown in source-b
const pageB = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, compiled_truth, frontmatter, updated_at, created_at)
VALUES ('code/src/markdown-b.ts', 'source-b', 'markdown-b.ts', 'code', 'export function parseMarkdown(s: string) { return s; }', '{}'::jsonb, NOW(), NOW())
RETURNING id`,
[],
);
// Chunks
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name_qualified, symbol_type)
VALUES ($1, 0, 'export function parseMarkdown(s: string) { return s; }', 'compiled_truth', 'typescript', 'parseMarkdown', 'function')`,
[pageA[0]!.id],
);
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name_qualified, symbol_type)
VALUES ($1, 0, 'export function callerInA() { return parseMarkdown(""); }', 'compiled_truth', 'typescript', 'callerInA', 'function')`,
[pageA2[0]!.id],
);
await engine.executeRaw(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name_qualified, symbol_type)
VALUES ($1, 0, 'export function parseMarkdown(s: string) { return s; }', 'compiled_truth', 'typescript', 'parseMarkdown', 'function')`,
[pageB[0]!.id],
);
// Unresolved edge: callerInA → parseMarkdown (no to_chunk_id, must resolve via symbol_name_qualified)
const callerChunk = await engine.executeRaw<{ id: number }>(
`SELECT cc.id FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.source_id = 'source-a' AND cc.symbol_name_qualified = 'callerInA' LIMIT 1`,
[],
);
await engine.executeRaw(
`INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, source_id, edge_metadata)
VALUES ($1, 'callerInA', 'parseMarkdown', 'calls', 'source-a', '{}'::jsonb)`,
[callerChunk[0]!.id],
);
}

219
test/e2e/symbol-resolver-pglite.test.ts Normal file
View File

@@ -0,0 +1,219 @@
/**
* v0.34 W0c — within-file two-pass symbol resolver E2E.
*
* Pins:
* - Unambiguous within-file match → edge_metadata.resolved_chunk_id set
* - Multi-match within file → edge_metadata.ambiguous=true + candidates
* - No match → edge stays untouched
* - chunks_walked watermark advances (edges_backfilled_at = NOW())
* - Idempotency: re-run on processed chunks is a no-op
* - Resume: bumping EDGE_EXTRACTOR_VERSION_TS forces re-walk
* - Source isolation: resolver scoped to one source_id; never touches edges
* in a different source even if the symbol name collides
*
* PGLite in-memory.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import {
resolveSymbolEdgesIncremental,
readEdgeResolution,
EDGE_EXTRACTOR_VERSION_TS,
} from '../../src/core/chunkers/symbol-resolver.ts';
import { resetPgliteState } from '../helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
describe('v0.34 W0c symbol-resolver — unambiguous within-file match', () => {
test('single-file: parseMarkdown call resolves to the parseMarkdown chunk in same file', async () => {
await registerSource(engine, 'source-a');
const pageId = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const callerChunk = await insertChunk(engine, pageId, 0, 'callerInA', 'function');
const defChunk = await insertChunk(engine, pageId, 1, 'parseMarkdown', 'function');
await insertUnresolvedEdge(engine, callerChunk, 'callerInA', 'parseMarkdown', 'source-a');
const stats = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
expect(stats.chunks_walked).toBeGreaterThanOrEqual(2);
expect(stats.edges_resolved).toBe(1);
expect(stats.edges_ambiguous).toBe(0);
expect(stats.edges_unmatched).toBe(0);
const edges = await engine.executeRaw<{ edge_metadata: any }>(
`SELECT edge_metadata FROM code_edges_symbol`,
[],
);
expect(edges.length).toBe(1);
const res = readEdgeResolution(edges[0]!.edge_metadata);
expect(res.kind).toBe('resolved');
if (res.kind === 'resolved') {
expect(res.chunk_id).toBe(defChunk);
}
});
});
describe('v0.34 W0c symbol-resolver — ambiguous within-file match', () => {
test('two same-named methods in the same file → ambiguous + candidates list', async () => {
await registerSource(engine, 'source-a');
const pageId = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const callerChunk = await insertChunk(engine, pageId, 0, 'callerInA', 'function');
const def1 = await insertChunk(engine, pageId, 1, 'render', 'function');
const def2 = await insertChunk(engine, pageId, 2, 'render', 'function'); // dup symbol name in same file
await insertUnresolvedEdge(engine, callerChunk, 'callerInA', 'render', 'source-a');
const stats = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
expect(stats.edges_ambiguous).toBe(1);
expect(stats.edges_resolved).toBe(0);
const edges = await engine.executeRaw<{ edge_metadata: any }>(
`SELECT edge_metadata FROM code_edges_symbol`,
[],
);
const res = readEdgeResolution(edges[0]!.edge_metadata);
expect(res.kind).toBe('ambiguous');
if (res.kind === 'ambiguous') {
expect(res.candidate_chunk_ids.sort()).toEqual([def1, def2].sort());
}
});
});
describe('v0.34 W0c symbol-resolver — no match', () => {
test('call to a symbol defined in another file stays unresolved (caller two-pass handles cross-file)', async () => {
await registerSource(engine, 'source-a');
const pageA = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const pageB = await insertCodePage(engine, 'source-a', 'src/bar.ts');
const callerChunk = await insertChunk(engine, pageA, 0, 'callerInA', 'function');
await insertChunk(engine, pageB, 0, 'externalFn', 'function'); // different file
await insertUnresolvedEdge(engine, callerChunk, 'callerInA', 'externalFn', 'source-a');
const stats = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
expect(stats.edges_unmatched).toBe(1);
expect(stats.edges_resolved).toBe(0);
expect(stats.edges_ambiguous).toBe(0);
const edges = await engine.executeRaw<{ edge_metadata: any }>(
`SELECT edge_metadata FROM code_edges_symbol`,
[],
);
const res = readEdgeResolution(edges[0]!.edge_metadata);
expect(res.kind).toBe('unresolved');
});
});
describe('v0.34 W0c symbol-resolver — watermark + idempotency', () => {
test('edges_backfilled_at advances; second run is a no-op', async () => {
await registerSource(engine, 'source-a');
const pageId = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const callerChunk = await insertChunk(engine, pageId, 0, 'callerInA', 'function');
await insertChunk(engine, pageId, 1, 'parseMarkdown', 'function');
await insertUnresolvedEdge(engine, callerChunk, 'callerInA', 'parseMarkdown', 'source-a');
const stats1 = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
expect(stats1.chunks_walked).toBeGreaterThanOrEqual(2);
const watermarkAfter = await engine.executeRaw<{ count: number }>(
`SELECT COUNT(*)::int AS count FROM content_chunks
WHERE edges_backfilled_at IS NOT NULL`,
[],
);
expect(watermarkAfter[0]!.count).toBeGreaterThanOrEqual(2);
// Second run: every chunk already has edges_backfilled_at >= EDGE_EXTRACTOR_VERSION_TS.
const stats2 = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
expect(stats2.chunks_walked).toBe(0);
expect(stats2.edges_examined).toBe(0);
});
});
describe('v0.34 W0c symbol-resolver — source isolation', () => {
test("does not resolve via candidates in a different source", async () => {
await registerSource(engine, 'source-a');
await registerSource(engine, 'source-b');
const pageA = await insertCodePage(engine, 'source-a', 'src/foo.ts');
const pageB = await insertCodePage(engine, 'source-b', 'src/foo.ts');
const callerInA = await insertChunk(engine, pageA, 0, 'callerInA', 'function');
// source-b has the same-named symbol at the same relative file path
await insertChunk(engine, pageB, 0, 'parseMarkdown', 'function');
// source-a does NOT have a parseMarkdown definition
await insertUnresolvedEdge(engine, callerInA, 'callerInA', 'parseMarkdown', 'source-a');
const stats = await resolveSymbolEdgesIncremental(engine, { sourceId: 'source-a' });
// The edge stays unresolved — the only same-symbol candidate is in
// source-b, which the resolver must NOT cross to.
expect(stats.edges_unmatched).toBe(1);
expect(stats.edges_resolved).toBe(0);
expect(stats.edges_ambiguous).toBe(0);
});
});
// ─────────────────────────────────────────────────────────────────
// Seeding helpers
// ─────────────────────────────────────────────────────────────────
async function registerSource(engine: PGLiteEngine, id: string): Promise<void> {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, created_at)
VALUES ($1, $1, $2, '{}'::jsonb, NOW())
ON CONFLICT (id) DO NOTHING`,
[id, `/fake/${id}`],
);
}
async function insertCodePage(engine: PGLiteEngine, sourceId: string, slug: string): Promise<number> {
const rows = await engine.executeRaw<{ id: number }>(
`INSERT INTO pages (slug, source_id, title, type, page_kind, compiled_truth, frontmatter, updated_at, created_at)
VALUES ($1, $2, $3, 'code', 'code', '', '{}'::jsonb, NOW(), NOW())
RETURNING id`,
[slug, sourceId, slug],
);
return rows[0]!.id;
}
async function insertChunk(
engine: PGLiteEngine,
pageId: number,
chunkIndex: number,
symbolName: string,
symbolType: string,
): Promise<number> {
const rows = await engine.executeRaw<{ id: number }>(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source, language, symbol_name_qualified, symbol_type)
VALUES ($1, $2, $3, 'compiled_truth', 'typescript', $4, $5)
RETURNING id`,
[pageId, chunkIndex, `// ${symbolName} body`, symbolName, symbolType],
);
return rows[0]!.id;
}
async function insertUnresolvedEdge(
engine: PGLiteEngine,
fromChunkId: number,
fromSymbol: string,
toSymbol: string,
sourceId: string,
): Promise<void> {
await engine.executeRaw(
`INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, source_id, edge_metadata)
VALUES ($1, $2, $3, 'calls', $4, '{}'::jsonb)`,
[fromChunkId, fromSymbol, toSymbol, sourceId],
);
}