v0.40.1.0 Track D — eval infrastructure (catch retrieval regressions, prove answer-quality wins) (#1298)
* feat(eval-longmemeval): --by-type flag + question field + resume-replace
Per-question JSONL row gains `question`, `question_type`, and (when
ground truth is available) `recall_hit` — additive fields that existing
consumers (LongMemEval's `evaluate_qa.py`) ignore. New `--by-type` flag
emits a `{kind:"by_type_summary", recall_by_type, aggregate}` line at
the end of the output, resume-safe: rebuilt from existing rows so the
final aggregate covers cumulative resumed questions, prior summary at
the tail replaced rather than appended. New `--by-type-floor F` exits
non-zero per breached question_type. Empty-bucket guard emits null rate
not NaN. Exports `buildByTypeSummary` + `emitByTypeSummary` +
`seedRecallByTypeFromFile` for unit testing.
* feat(eval-cross-modal): --batch flag + semaphore + DI seam
Adds `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT]
[--yes]` to the existing eval cross-modal command. Mutually exclusive
with --task. Reads LongMemEval-shape JSONL output, filters by_type_summary
rows automatically, fans out via a new `runWithLimit<T>` semaphore
primitive (default --concurrent 3 x 3 model slots = 9 simultaneous calls;
below tier-1 rate limits on all 3 providers). Pre-flight cost estimate
refuses past --max-usd (default $5) unless --yes. Per-question receipts
written to a per-batch tempdir + deleted at end of run so
~/.gbrain/eval-receipts/ stays clean; summary receipt inlines verdicts.
Exit precedence (new batch-level policy, not inherited from aggregate.ts):
ERROR > FAIL > INCONCLUSIVE > PASS — any per-question runtime error exits 2.
New `runEvalCrossModal(args, opts?: {runEval?})` DI seam mirrors the
existing eval-longmemeval pattern. Tests pass a stub runEval so unit tests
don't need API keys; gateway availability check is also skipped when
opts.runEval is provided. Pinned by 17 cases.
* test: hermetic qrels retrieval gate against synthetic basis-vector corpus
Adds test/eval-replay-gate.test.ts as a unit-shard test (NOT under
test/e2e/ — the unit-shard CI matrix runs every PR via bun test;
test/e2e/ is fixed-file). Seeds a PGLite engine with synthetic
placeholder-name pages whose embeddings are basis vectors (same pattern
as test/e2e/search-quality.test.ts:23-28) so retrieval is hermetic — no
API keys, no DATABASE_URL, fully deterministic.
The qrels fixture at test/fixtures/eval-baselines/qrels-search.json has
12 hand-curated queries; each maps to a ranked list of relevant slugs +
`first_relevant_slug` (expected top-1). For each query, the gate asserts
`top1_match_rate >= 0.80` AND `recall_at_10 >= 0.85`. Env-overridable
floors via GBRAIN_REPLAY_GATE_TOP1_FLOOR / GBRAIN_REPLAY_GATE_RECALL_FLOOR
through withEnv(). Gate-fire prints per-query HIT/miss + recall to stderr.
When ranking changes intentionally move expected slugs, edit
qrels-search.json directly with a 'Why:' line in the commit body —
documented in docs/eval-bench.md.
scripts/check-test-real-names.sh allowlist gains 6 entries for the
privacy-grep regression guard inside the test, which must literally
spell the names it forbids to assert they're NOT in the fixture (same
meta-rule exception as skillpack-harvest privacy tests).
* feat(autopilot): opt-in nightly cross-modal quality probe + doctor check
Composes `gbrain eval longmemeval --by-type` + `gbrain eval cross-modal
--batch` into a 24h-cadenced quality check. Default DISABLED — opt-in via
`gbrain config set autopilot.nightly_quality_probe.enabled true` so new
users don't discover background API spend.
src/core/cycle/nightly-quality-probe.ts ships the phase implementation
with a full NightlyProbeDeps DI surface (isEnabled, hasEmbeddingProvider,
resolveMaxUsd, resolveRepoRoot, runLongMemEval, runCrossModalBatch, now)
so tests stub every external effect — no PGLite, no real LLM calls.
Pure `shouldRunNightly(now, recentEvents, windowMs?)` rate-limit fn.
src/core/audit-quality-probe.ts is the ISO-week-rotated JSONL writer
(mirrors audit-slug-fallback.ts; honors GBRAIN_AUDIT_DIR). One event per
run: outcome (pass/fail/inconclusive/error/budget_exceeded/rate_limited/
no_embedding_key), exit code, pass/fail/error counts, est_cost_usd,
fixture_sha8.
src/commands/doctor.ts gains a `nightly_quality_probe_health` check:
SKIPPED with paste-ready enable command when disabled; OK with timestamp
when all PASS in last 7 days; WARN with per-outcome counts when any
FAIL/ERROR/BUDGET_EXCEEDED. Extracted as pure
`computeNightlyQualityProbeHealthCheck(probeEnabled, events)` for
unit testing.
test/fixtures/longmemeval-nightly.jsonl is a 10-question placeholder
dataset (synthetic names only) distinct from the existing 5-question
mini fixture so the probe has consistent regression signal.
Real expected cost: ~$0.35/night = ~$10.50/month. Worst-case at
default $5 cap: $150/month.
Pinned by 21 cases in test/nightly-quality-probe.test.ts covering the
rate-limit pure function, every outcome branch, and all 7 branches of
the doctor check.
Autopilot scheduler wiring deferred to v0.41+ — the phase is callable
in isolation today (via the DI surface); cycle-loop dispatcher
integration filed in TODOS.md as a follow-up.
* docs: document Track D eval surfaces + file v0.41+ follow-up TODOs
docs/eval-bench.md gains a 'v0.40.1.0 Track D — Eval infrastructure'
section covering: --by-type usage + resume-replace semantics, the
hermetic qrels gate workflow + 'Why:' commit-body refresh convention,
--batch end-to-end with cost-bound + concurrency knobs, and the opt-in
nightly probe enable workflow + cost ceiling.
TODOS.md files two follow-ups:
- v0.41+: contributor-mode CI capture for BrainBench-Real replay gate
(the deferred original Task 2 design — replay against real captured
queries is more valuable than synthetic qrels long-term, but needs CI
secret + nightly capture pipeline + commit automation; deferred to a
dedicated wave)
- v0.41+: wire the nightly quality probe into autopilot scheduling
(phase callable in isolation today; cycle-loop dispatcher integration
is a ~3-hour follow-up)
CLAUDE.md Key Files annotations extended for the four lanes:
eval-longmemeval gains the --by-type description, eval-cross-modal
gains the --batch + DI seam description, new entries for the qrels
gate test + the nightly probe + audit-quality-probe writer.
* chore: bump version and changelog (v0.40.1.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(eval): close 4 codex-flagged eval-integrity bugs
Codex adversarial review on the Track D wave found 4 real ways the new
eval-gate code could silently bypass its gates. Each fix below either
counts what was previously dropped, fails fast on a parser edge case,
or enforces a gate that was previously skipped on an early-return path.
CDX-1: cross-modal --batch silently dropped failed/corrupt LongMemEval
rows. `gbrain eval longmemeval` emits {error:..., hypothesis:''} when
runOneQuestion throws; the batch reader's missing-field skip threw those
rows away, shrinking the denominator. A green eval on a subset is now
impossible:
- eval-longmemeval.ts: error rows now carry `question` + `question_type`
so the batch consumer can identify them as upstream failures, not
skip them as malformed.
- eval-cross-modal.ts: readBatchRows now returns {rows, upstream_errors,
malformed_count}. Upstream errors fold into per_question with verdict
'upstream_error'. BatchSummary gains `upstream_error_count` and
`malformed_count`. ERROR exit precedence widens to include both, so
any upstream failure exits 2.
CDX-2: --limit 0 was a direct CI bypass — zero-row check fired before
slicing, then the empty result fell through to verdict='pass'. Fixed
with a hard `limit >= 1` check.
CDX-3: --resume-from + --by-type-floor was a real gate skip. When a
prior run had every question answered, the early "nothing to do" return
fired BEFORE summary emission and floor enforcement. Now the no-op
resume path still seeds recallByType from the existing file, emits the
by_type_summary at the tail, and runs the floor gate.
CDX-5: doctor nightly_quality_probe_health only flagged fail / error /
budget_exceeded as warn. no_embedding_key / rate_limited / inconclusive
were silently reported as PASS — hiding misconfigurations and queue
backpressure. The bad-event filter is now `outcome !== 'pass'`, and the
counts string surfaces every bucket so the operator sees exactly what
went wrong.
scripts/check-privacy.sh: adds test/eval-replay-gate.test.ts to the
allowlist (the qrels test's privacy-grep regression guard literally
names what it forbids, same meta-rule exception as the existing
test/recency-decay.test.ts + skillpack-harvest allowlist entries).
Pinned by 8 new regression cases across eval-longmemeval (CDX-3),
eval-cross-modal-batch (CDX-1 + CDX-2), and nightly-quality-probe
(CDX-5). 76 Track D tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
124
CHANGELOG.md
124
CHANGELOG.md
@@ -2,6 +2,130 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.40.1.0] - 2026-05-22
|
||||
|
||||
**Eval infrastructure that catches retrieval regressions before merge and proves answer-quality wins after ship.**
|
||||
|
||||
Three things changed about how gbrain measures itself. First, when you run the LongMemEval benchmark, you can now see per-question-type recall in machine-readable form (a single flag prints something like "single-session-user: 94.7%, multi-session: 100%"). Before this release that number existed but only as a fleeting stderr line — you couldn't pipe it into a script or compare it across runs. Second, any PR that touches the search code now runs a tiny hermetic test that asks "would this change rerank our known queries?" Twelve hand-curated queries with known expected results live in a fixture; the test runs them through the new ranking and fails the PR if the top-1 match rate drops below 80% or recall@10 drops below 85%. No API keys, no Postgres, no flakiness — just `bun test` in the standard PR shard. Third, you can run cross-modal quality scoring across a whole LongMemEval slice in one command (`gbrain eval cross-modal --batch run.jsonl --limit 10 --cycles 1`) with built-in cost guardrails so you can't accidentally spend $50 on a quality check.
|
||||
|
||||
The headline number for the user: agents that touch retrieval can now ship knowing whether they regressed real query-answer quality, not just whether tests pass.
|
||||
|
||||
### How to turn it on
|
||||
|
||||
```bash
|
||||
# 1. Per-question-type breakdown after any LongMemEval run.
|
||||
gbrain eval longmemeval ~/datasets/longmemeval_s.jsonl \
|
||||
--by-type --output /tmp/run.jsonl
|
||||
tail -1 /tmp/run.jsonl | jq . # summary line with recall_by_type
|
||||
|
||||
# 2. The qrels gate runs automatically on every PR touching src/core/search/**.
|
||||
# No setup needed. To run locally:
|
||||
bun test test/eval-replay-gate.test.ts
|
||||
|
||||
# 3. Cross-modal batch over LongMemEval output (real LLM cost ~$0.70 default).
|
||||
gbrain eval cross-modal --batch /tmp/run.jsonl \
|
||||
--limit 10 --cycles 1 --concurrent 3 --max-usd 5 --json
|
||||
|
||||
# 4. Opt-in nightly quality probe via autopilot (scheduler wiring deferred to
|
||||
# v0.41+; phase is callable in isolation today).
|
||||
gbrain config set autopilot.nightly_quality_probe.enabled true
|
||||
```
|
||||
|
||||
### Numbers that matter
|
||||
|
||||
| Surface | Before | After |
|
||||
|---|---|---|
|
||||
| Per-question-type R@k | stderr-only, lost on next run | JSONL summary line, scriptable, resume-safe |
|
||||
| Search PR gate | none (silent regressions possible) | hermetic qrels gate on every PR shard |
|
||||
| Cross-modal batch | single-task only; manual loop needed | `--batch FILE` with semaphore + cost cap |
|
||||
| Default batch cost ceiling | implied | `--max-usd 5` refuses without `--yes` |
|
||||
| Nightly quality probe | not present | opt-in autopilot phase + doctor check + audit JSONL |
|
||||
|
||||
### Things to watch after upgrade
|
||||
|
||||
- `gbrain doctor` will now show `nightly_quality_probe_health: disabled (opt-in)` until you set the config flag. This is informational, not a warning.
|
||||
- The LongMemEval `--by-type` summary is appended as a NEW JSON line at the end of the output. Existing consumers (LongMemEval's `evaluate_qa.py`) ignore unknown fields, so your existing pipelines keep working unchanged.
|
||||
- The qrels gate uses synthetic queries with placeholder names (alice-example, widget-co-example, etc.). When a real ranking change moves expected slugs, you refresh `test/fixtures/eval-baselines/qrels-search.json` and add a `Why:` line to the commit body so future maintainers understand the drift.
|
||||
- Autopilot scheduler wiring for the nightly probe is deferred to v0.41+ — the phase is callable in isolation today via the DI surface; cycle-loop dispatcher integration is filed as a follow-up TODO.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### `gbrain eval longmemeval --by-type`
|
||||
|
||||
- Every per-question JSONL row now includes `question: string`, `question_type: string`, and (when ground truth is available) `recall_hit: boolean`. Additive — `evaluate_qa.py` and other consumers ignore unknown fields.
|
||||
- New `--by-type` flag emits a final `{schema_version:1, kind:"by_type_summary", recall_by_type:{...}, aggregate:{...}}` line as the LAST line of the output.
|
||||
- Resume-safe via three new exported helpers (`buildByTypeSummary`, `emitByTypeSummary`, `seedRecallByTypeFromFile`): the summary is rebuilt from existing file rows on `--resume-from` runs so the final aggregate covers ALL resumed questions, and any prior summary at the tail is replaced rather than appended.
|
||||
- New `--by-type-floor F` (range [0,1]) exits non-zero with a per-type breach stderr line when any `question_type` rate falls below floor. Default unset = informational only.
|
||||
- Empty-bucket guard: `aggregate.rate` is `null` (not NaN) when no questions had ground truth, so downstream JSON consumers don't trip.
|
||||
- Pinned by 5 new cases in `test/eval-longmemeval.test.ts` covering question-field presence, with/without flag emission, resume-replace at file tail with cumulative aggregation across runs, and the pure `buildByTypeSummary` function.
|
||||
|
||||
#### Hermetic qrels retrieval gate
|
||||
|
||||
- New committed fixture at `test/fixtures/eval-baselines/qrels-search.json` with 12 hand-curated queries using placeholder names only.
|
||||
- New unit test at `test/eval-replay-gate.test.ts` (NOT under `test/e2e/` — the unit-shard CI matrix runs every PR via `bun test`; `test/e2e/` is fixed-file). Uses the canonical PGLite block from CLAUDE.md test-isolation rules and basis-vector embeddings (the `basisEmbedding(idx)` pattern from `test/e2e/search-quality.test.ts:23-28`) so retrieval is hermetic — no API keys, no `DATABASE_URL`, fully deterministic.
|
||||
- For each query, asserts `top1_match_rate >= 0.80` AND `recall_at_10 >= 0.85`. Env-overridable floors via `GBRAIN_REPLAY_GATE_TOP1_FLOOR` / `GBRAIN_REPLAY_GATE_RECALL_FLOOR`.
|
||||
- When the gate trips, the test surfaces per-query results to stderr (HIT/miss + recall) so the operator sees exactly which queries regressed without re-running.
|
||||
- Pinned by 5 cases including a privacy-grep regression guard that fails on any real-name reintroduction.
|
||||
|
||||
#### `gbrain eval cross-modal --batch`
|
||||
|
||||
- Existing single-task `gbrain eval cross-modal` grows new flags: `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]`. Mutually exclusive with `--task` (fail-fast usage error). One canonical home for cross-modal eval — no new subcommand.
|
||||
- Reads LongMemEval-shape JSONL output, filters `kind: "by_type_summary"` rows automatically, slices to `--limit` (default 10), and fans out via semaphore-bounded loop.
|
||||
- New inline `runWithLimit<T>(items, limit, fn)` semaphore primitive (exported for unit tests): default `--concurrent 3` × 3 model slots per question = max 9 simultaneous API calls. Below tier-1 rate limits on Anthropic, OpenAI, and Google.
|
||||
- Pre-flight cost estimate refuses if `> --max-usd` (default 5.00 USD) without `--yes`. Cron / CI callers must pass `--yes` explicitly to bypass.
|
||||
- Per-question receipts land in a per-batch tempdir and are deleted at end of run so `~/.gbrain/eval-receipts/` doesn't accumulate junk. The summary receipt inlines per-question verdicts as JSON, not file paths, so the audit trail is self-contained.
|
||||
- Exit precedence (fail-loud convention; new batch-level policy, NOT inherited from `aggregate.ts`): ERROR > FAIL > INCONCLUSIVE > PASS. Any per-question runtime error → exit 2 even if other questions passed.
|
||||
- New `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` DI seam mirrors the eval-longmemeval pattern. Tests pass a stub `runEval` returning deterministic `RunEvalResult`; gateway availability check is also skipped when `opts.runEval` is provided so unit tests don't need API keys.
|
||||
- Pinned by 17 cases in `test/eval-cross-modal-batch.test.ts` (semaphore unit tests + batch flow + exit precedence + privacy filter + mutex + budget refuse + by-type-summary filter + parser strictness).
|
||||
|
||||
#### Nightly quality probe (opt-in)
|
||||
|
||||
- New autopilot phase at `src/core/cycle/nightly-quality-probe.ts` that composes `eval longmemeval` + `eval cross-modal --batch` into a 24h-cadenced quality check.
|
||||
- New audit JSONL writer at `src/core/audit-quality-probe.ts` writing to `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` (ISO-week rotation; mirrors `audit-slug-fallback.ts`; honors `GBRAIN_AUDIT_DIR`). One event per run with outcome, exit code, pass/fail/inconclusive/error counts, est_cost_usd, fixture_sha8, and optional detail.
|
||||
- Pure `shouldRunNightly(now, recentEvents, windowMs?)` function gates the 24h rate limit so the audit-log read is the only state and tests can drive any cadence.
|
||||
- Full DI surface via `NightlyProbeDeps` (`isEnabled`, `hasEmbeddingProvider`, `resolveMaxUsd`, `resolveRepoRoot`, `runLongMemEval`, `runCrossModalBatch`, `now`). Tests stub every external effect — no PGLite, no real LLM calls, no env mutation outside `withEnv()`.
|
||||
- New 10-question placeholder fixture at `test/fixtures/longmemeval-nightly.jsonl` using only synthetic names. Distinct from the existing 5-question `longmemeval-mini.jsonl` (unit-test fixture) so the nightly probe has consistent regression signal.
|
||||
- New `nightly_quality_probe_health` check in `gbrain doctor`: SKIPPED with paste-ready enable command when disabled; OK with timestamp when all PASS in last 7 days; WARN with per-outcome counts when any FAIL / ERROR / BUDGET_EXCEEDED. The check is also exposed as the pure `computeNightlyQualityProbeHealthCheck(probeEnabled, events)` helper for direct unit testing.
|
||||
- Default DISABLED — opt-in via `gbrain config set autopilot.nightly_quality_probe.enabled true`. New users running `gbrain init` should NOT discover background API spend.
|
||||
- Real expected cost: ~$0.35/night ≈ $10.50/month. Worst-case under the default $5 cap: $150/month.
|
||||
- Pinned by 21 cases in `test/nightly-quality-probe.test.ts` covering the rate-limit pure function, every outcome branch (disabled / no-embedding-key / rate-limited / pass / fail / runtime-error / missing-fixture), and all 7 branches of the doctor check.
|
||||
- Autopilot scheduler wiring deferred to v0.41+ — the phase is callable in isolation today; cycle-loop dispatcher integration filed in TODOS.md as a ~3-hour follow-up.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- DI seam consistency: `runEvalCrossModal(args, opts?)` now mirrors `runEvalLongMemEval(args, opts?)`. Both let tests stub the heavy backend without `mock.module` (which would force `*.serial.test.ts` quarantine under the test-isolation rules).
|
||||
- Cost-bounding pattern: every new long-running command in this wave (--batch, nightly probe) refuses to start past a configurable USD cap without explicit `--yes`. Consistent across the surface.
|
||||
|
||||
## To take advantage of v0.40.1.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
|
||||
warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Your agent reads the docs the next time you interact with it.** No schema migration in this release — Track D is pure tooling. CLAUDE.md's "Key files" section grew four annotations; `bun run build:llms` regenerated `llms.txt` + `llms-full.txt` in the same commit.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
bun test test/eval-replay-gate.test.ts # hermetic qrels gate green
|
||||
bun test test/eval-longmemeval.test.ts # --by-type cases green
|
||||
bun test test/eval-cross-modal-batch.test.ts # --batch cases green
|
||||
bun test test/nightly-quality-probe.test.ts # nightly probe DI cases green
|
||||
gbrain doctor --json | jq '.checks.nightly_quality_probe_health'
|
||||
```
|
||||
4. **Optionally enable the nightly probe** (real API cost ~$10.50/month expected, $150/month worst-case under the default $5/run cap):
|
||||
```bash
|
||||
gbrain config set autopilot.nightly_quality_probe.enabled true
|
||||
```
|
||||
Skip if you don't want background API spend.
|
||||
5. **If any step fails or the numbers look wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
## [0.40.0.0] - 2026-05-17
|
||||
|
||||
**Voice agent reference lands — Mars + Venus personas, WebRTC-first, copy-into-your-repo not stuck-in-gbrain.**
|
||||
|
||||
@@ -75,7 +75,7 @@ strict behavior when unset.
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO.
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x, extended v0.40.1.0 Track D) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO. **v0.40.1.0 Track D (T3+T4, per D1/D5/D6/D10):** new `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` flags fan out cross-modal scoring across a LongMemEval-shape JSONL. Mutually exclusive with `--task` (fail-fast usage error if both set). Filters `kind: "by_type_summary"` rows from the input (Codex #6). Pre-flight cost estimate refuses if `> --max-usd` without `--yes`; default cap 5.00 USD. Semaphore-bounded fan-out via inline `runWithLimit<T>(items, limit, fn)` (~15 LOC, exported for unit tests): max N questions in-flight at once × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run (per D10 — keeps `~/.gbrain/eval-receipts/` clean); the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (NEW batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})` at eval-longmemeval.ts:299; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by 15 cases in `test/eval-cross-modal-batch.test.ts`.
|
||||
- `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.
|
||||
- `src/core/cross-modal-eval/aggregate.ts` (v0.27.x) — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)` (Q2=A floor). Inconclusive when <2/3 models returned parseable scores (Q3=A regression guard for the v1 .mjs `Object.values({}).every(...) === true` empty-array PASS bug).
|
||||
- `src/core/cross-modal-eval/runner.ts` (v0.27.x) — orchestrator. Each cycle runs `Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])` (T4=A — bare allSettled, no rate-leases for the CLI path; minion-integration TODO recovers cross-process concurrency). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro`. `estimateCost()` exports a small per-model pricing table (drifts; refresh alongside model-family bumps).
|
||||
@@ -84,11 +84,13 @@ strict behavior when unset.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `test/eval-replay-gate.test.ts` + `test/fixtures/eval-baselines/qrels-search.json` (v0.40.1.0 Track D / T5, per D8 reshape) — hermetic retrieval qrels gate that runs in the standard PR unit-shard CI matrix (`.github/workflows/test.yml`, NOT the fixed-file E2E workflow). Structural replacement for the original "replay against captured `eval_candidates` baseline" design (deferred to `v0.41+: contributor-mode CI capture` in TODOS.md; Codex outside-voice caught three fatal flaws — select-e2e.ts is local-only, eval-export bypasses op-layer capture on PGLite-seed tests, replay re-embeds via gateway which needs an API key CI doesn't have). Uses the canonical PGLite block from CLAUDE.md test-isolation rules (R3+R4) and the basis-vector embedding pattern from `test/e2e/search-quality.test.ts:23-28` for fully hermetic retrieval. The qrels fixture (12 queries) is hand-curated with PLACEHOLDER names only (alice-example, widget-co-example, etc. — per Codex #9 + CLAUDE.md privacy rule) and embeds each query at a deterministic basis dimension so retrieval is reproducible. Each query lists `relevant_slugs[]` + `first_relevant_slug`; for each, the test computes `top1_match_rate` (top-1 == first_relevant) and `recall@10` (fraction of relevant_slugs in top-10), asserting both meet floors (defaults `>= 0.80` and `>= 0.85`). Env-overridable floors via `GBRAIN_REPLAY_GATE_TOP1_FLOOR` / `GBRAIN_REPLAY_GATE_RECALL_FLOOR` (with `withEnv()` per CLAUDE.md R1). Refresh discipline (per D4): when ranking changes intentionally move expected slugs, edit `qrels-search.json` directly and include a `Why:` line in the commit body so future maintainers can read the audit trail. Without `Why:`, the gate degrades to rubber-stamp within months. Pinned by 5 cases in `test/eval-replay-gate.test.ts` including a privacy-grep regression guard against real-name reintroduction.
|
||||
- `src/core/cycle/nightly-quality-probe.ts` + `src/core/audit-quality-probe.ts` + `test/fixtures/longmemeval-nightly.jsonl` + `test/nightly-quality-probe.test.ts` (v0.40.1.0 Track D / T6+T7+T8, per D12) — opt-in nightly cross-modal quality probe. The phase runs `gbrain eval longmemeval --by-type` against the committed 10-question placeholder fixture, pipes the output through `gbrain eval cross-modal --batch --max-usd 5 --yes`, and writes one event per run to `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` (ISO-week-rotated, mirrors `audit-slug-fallback.ts`; honors `GBRAIN_AUDIT_DIR`). Default DISABLED — opt-in via `gbrain config set autopilot.nightly_quality_probe.enabled true` (prevents surprise API spend on `gbrain init`). 24h rate limit (the pure `shouldRunNightly(now, recentEvents, windowMs?)` function) skips with audit row `outcome: rate_limited` when a recent run exists. Embedding-key short-circuit: longmemeval needs `gateway.embedQuery()`, so the phase exits early with `outcome: no_embedding_key` + stderr warn when no provider is configured. Full DI surface via `NightlyProbeDeps` (`isEnabled`, `hasEmbeddingProvider`, `resolveMaxUsd`, `resolveRepoRoot`, `runLongMemEval`, `runCrossModalBatch`, `now`) so the unit test stubs every external effect — no PGLite, no real LLM calls, no env mutation outside `withEnv()`. Cost ceiling: $5/run × 30 nights ≈ $150/month worst-case; expected real cost ~$0.35/night × 30 ≈ $10.50/month. **Autopilot scheduler wiring deferred to v0.41+ follow-up (filed in TODOS.md)** — the phase is callable in isolation today; the cycle-loop dispatcher hasn't been wired to invoke it on the 24h cadence yet. New `nightly_quality_probe_health` doctor check (in `src/commands/doctor.ts` right after `slug_fallback_audit`) reads the last 7 days of audit events: SKIPPED when feature flag is off (with paste-ready enable command); OK when enabled + all PASS; WARN on any FAIL / ERROR / BUDGET_EXCEEDED in the window with per-outcome counts. Pinned by 14 cases in `test/nightly-quality-probe.test.ts` (rate-limit pure-function unit tests + DI-stubbed end-to-end phase tests across every outcome branch).
|
||||
- `src/commands/eval-trajectory.ts` + `src/commands/founder-scorecard.ts` + `src/core/trajectory.ts` (v0.35.7) — temporal trajectory + founder scorecard. The wave that turns the v0.35.3.1 date-aware contradiction probe into a useful temporal substrate. `gbrain eval trajectory <entity>` shows the chronological typed-claim history (mrr/arr/team_size/etc) with regressions auto-flagged inline; `gbrain founder scorecard <entity>` rolls up claim_accuracy / consistency / growth_trajectory / red_flags into one JSON. Pure-function math lives in `trajectory.ts`: `detectRegressions(points, threshold)` walks consecutive metric-value pairs per metric (10% drop default, env override `GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD`); `computeDriftScore(points)` returns `1 - mean(cosine(emb[i], emb[i-1]))` over existing embeddings (null when <3 embedded points). Backed by `BrainEngine.findTrajectory(opts)` — both Postgres and PGLite, single SQL query, deterministic `ORDER BY valid_from ASC, id ASC` (R3). Source-scoped via the v0.34.1.0 `sourceId` scalar / `sourceIds` array dual pattern (D-CDX-6); visibility-filtered for remote callers (D-CDX-1) — `recall`-equivalent posture. MCP op `find_trajectory` (read scope, NOT localOnly) registered after `find_experts`. Migration v67 adds four optional typed-claim columns (`claim_metric`, `claim_value`, `claim_unit`, `claim_period`) + a partial index on `(entity_slug, claim_metric, valid_from) WHERE claim_metric IS NOT NULL`. Fence widens from 10 to 14 cells when any row has typed data; renderer stays at 10 cells when none do (no churn diff on existing fences). Metric labels normalize to lowercase snake_case via `normalizeMetricLabel` (15-entry seed map for common founder metrics). The `consolidate` cycle phase gains semantic upsert keyed on `(page_id, claim, since_date)` — fixes the pre-existing F4 duplicate-takes bug where re-running the full cycle after `extract_facts` cleared `consolidated_at` would silently append duplicate takes via `MAX(row_num)+1`. Also writes chronological `valid_until` on each cluster's older facts. The `extract_facts` cycle phase batch-embeds via `gateway.embed()` before insert AND threads `pages.effective_date` as the `pageEffectiveDate` fallback for `valid_from` (precedence chain: fence-row > pageEffectiveDate > now()). The contradiction probe MUST NOT write `valid_until` — R1+R8 grep guard at `test/eval-contradictions/no-valid-until-write.test.ts` pins this. Codex outside-voice round caught F1 (v66 collision → v67), F2 (Haiku lives in `facts/extract.ts` not `extract-facts.ts` cycle phase), F3 (cycle didn't embed before insert), F4 (idempotency bug), F5+F6 (missed `fence-write.ts` caller + no Page object there → pageEffectiveDate is OPTIONAL), F7 (privacy regression — visibility filter added), F8 (ParsedFact needed typed-field extension for markdown system-of-record), F9 (dual scalar+federated sourceId). Plan: `~/.claude/plans/system-instruction-you-are-working-curious-jellyfish.md`. Tests: 258 across 12 files.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`.
|
||||
- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1, extended v0.40.1.0 Track D) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec). **v0.40.1.0 Track D (T1+T2, per D9):** per-question JSONL row gains `question: string` (additive — `evaluate_qa.py` ignores unknown fields) so the `gbrain eval cross-modal --batch` consumer has the `task` text without joining back against the source dataset. Each row also carries `question_type: string` and `recall_hit?: boolean` so a `--resume-from` run can rebuild the cumulative `recallByType` from the file alone. New `--by-type` flag emits a `{schema_version:1, kind:"by_type_summary", recall_by_type:{...}, aggregate:{...}}` line as the FINAL line of the output; resume-replace strips any prior summary at the tail so 5 resumed runs produce 1 summary, not 5 (Codex #7). Empty-bucket guard: `aggregate.rate` is `null` (not NaN) when no questions had ground truth. Optional `--by-type-floor F` (0..1) exits non-zero with a stderr line per breached `question_type`; default informational only. Pure `buildByTypeSummary(buckets)` + `emitByTypeSummary(path, summary)` + `seedRecallByTypeFromFile(path, bucket)` helpers exported for unit tests. Pinned by 5 new cases in `test/eval-longmemeval.test.ts`.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
|
||||
41
TODOS.md
41
TODOS.md
@@ -1,5 +1,46 @@
|
||||
# TODOS
|
||||
|
||||
## v0.40.1.0 Track D follow-ups (v0.41+)
|
||||
|
||||
- [ ] **v0.41+: contributor-mode CI capture for BrainBench-Real replay gate.**
|
||||
v0.40.1.0 Track D shipped the hermetic qrels gate (`test/eval-replay-gate.test.ts`)
|
||||
as the structurally-correct replacement for the original "replay against captured
|
||||
`eval_candidates` baseline" design. Codex outside-voice audit caught three fatal
|
||||
flaws with the replay-against-captured-baseline approach: (a) `scripts/select-e2e.ts`
|
||||
is local-only — `.github/workflows/test.yml` + `e2e.yml` hit fixed file lists,
|
||||
so a diff-aware selector entry would gate nothing on GitHub PRs;
|
||||
(b) `gbrain eval export` reads `eval_candidates` rows which only populate when
|
||||
ops fire through the operation layer with `GBRAIN_CONTRIBUTOR_MODE=1` capture —
|
||||
PGLite tests seeded via direct `engine.put*()` produce zero captured rows;
|
||||
(c) `gbrain eval replay` re-embeds query text via `gateway.embedQuery()` which
|
||||
needs an API key CI doesn't have. Real-query dogfooding is still valuable —
|
||||
synthetic qrels test the structural ranking, real captures test what users
|
||||
actually search for. To restore the replay-based gate properly: (1) provision
|
||||
a CI secret for an embedding key (OpenAI text-embedding-3-small is the
|
||||
cheapest); (2) build a nightly capture pipeline that runs
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1 gbrain eval export --tool query` against a seeded
|
||||
brain corpus; (3) commit-automate the resulting NDJSON into
|
||||
`test/fixtures/eval-baselines/` with a "Why:" justification line; (4) write
|
||||
a new gate test that calls `gbrain eval replay --against <fixture>` and asserts
|
||||
on `mean_jaccard`, `top1_stability_rate`, drops the latency assert (CI runners
|
||||
vary too much). Estimate: ~2 weeks. Filed during v0.40.1.0 Track D
|
||||
/plan-eng-review (see `~/.claude/plans/system-instruction-you-are-working-whimsical-acorn.md`).
|
||||
|
||||
- [ ] **v0.41+: Wire the nightly quality probe into autopilot scheduling.**
|
||||
v0.40.1.0 Track D shipped the phase (`src/core/cycle/nightly-quality-probe.ts`),
|
||||
the audit JSONL (`src/core/audit-quality-probe.ts`), the doctor check
|
||||
(`nightly_quality_probe_health` in doctor.ts), and the 10-question
|
||||
placeholder fixture. What's NOT yet wired: `src/commands/autopilot.ts`
|
||||
doesn't yet invoke `runNightlyQualityProbe(deps)` on its 24h cadence —
|
||||
the phase is callable in isolation (good for testing) but no scheduled
|
||||
loop calls it. To finish: add a phase trigger to the autopilot cycle loop
|
||||
that calls the probe with concrete deps wiring (`isEnabled`,
|
||||
`hasEmbeddingProvider`, `resolveMaxUsd`, `resolveRepoRoot`, real
|
||||
`runLongMemEval` / `runCrossModalBatch` invocations via subprocess or
|
||||
direct function call). Honor `autopilot.nightly_quality_probe.enabled`
|
||||
config gate (already in doctor's read-side; needs autopilot read-side).
|
||||
Doctor surface is already in place to show outcomes; just need the
|
||||
scheduling lane. Estimate: ~3 hours.
|
||||
|
||||
## v0.39.3.0 smoke-test wave — deferred follow-ups (v0.39.4 / v0.40)
|
||||
|
||||
|
||||
@@ -328,3 +328,165 @@ commands per high-severity finding.
|
||||
- `docs/contradictions.md` — architecture, severity rubric, action criteria.
|
||||
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
|
||||
decision criteria gated on Wilson CI lower-bound.
|
||||
|
||||
## v0.40.1.0 Track D — Eval infrastructure
|
||||
|
||||
Three eval surfaces grew non-trivial capabilities in v0.40.1.0. This section
|
||||
covers the dev loop that uses them and the gates they enforce.
|
||||
|
||||
### `gbrain eval longmemeval --by-type` — per-question-type R@k breakdown
|
||||
|
||||
LongMemEval has always computed per-question-type recall internally; v0.40.1.0
|
||||
surfaces it in machine-readable form. Two additive changes:
|
||||
|
||||
1. Every per-question JSONL row now includes a `question: string` field so the
|
||||
`gbrain eval cross-modal --batch` consumer (below) can read it without
|
||||
joining back against the source dataset.
|
||||
2. New `--by-type` flag emits a final aggregate line keyed by `question_type`:
|
||||
|
||||
```json
|
||||
{"schema_version": 1, "kind": "by_type_summary",
|
||||
"recall_by_type": {"single-session-user": {"hit": 18, "total": 19, "rate": 0.947}},
|
||||
"aggregate": {"hit": 110, "total": 120, "rate": 0.917}}
|
||||
```
|
||||
|
||||
**Resume-safe.** When `--resume-from` is the same path as `--output`, the
|
||||
summary is rebuilt from the file (each per-row includes `question_type` and
|
||||
`recall_hit`) so the final aggregate covers all resumed questions, not just
|
||||
this run's slice. The prior summary at the file tail is replaced, not
|
||||
appended — a brain that resumes 5 times across a 500-question run ends with
|
||||
exactly ONE summary at the tail.
|
||||
|
||||
**Optional gate.** `--by-type-floor 0.85` exits non-zero when any
|
||||
`question_type`'s rate falls below 0.85. Default: informational only.
|
||||
|
||||
```bash
|
||||
# Diagnose per-type ranking quality after a search-touching change.
|
||||
gbrain eval longmemeval ~/datasets/longmemeval_s.jsonl \
|
||||
--by-type --output /tmp/run.jsonl
|
||||
tail -1 /tmp/run.jsonl | jq . # summary line
|
||||
|
||||
# Strict gate in a CI script.
|
||||
gbrain eval longmemeval test/fixtures/longmemeval-mini.jsonl \
|
||||
--by-type --by-type-floor 0.80 --output /tmp/run.jsonl
|
||||
echo "exit=$?" # 1 if any type fell below 0.80
|
||||
```
|
||||
|
||||
### Hermetic retrieval gate — `test/eval-replay-gate.test.ts`
|
||||
|
||||
The v0.40.1.0 Track D structural fix for "PRs touching `src/core/search/`
|
||||
silently regress retrieval." Replaces the original "replay against captured
|
||||
eval_candidates" design (which Codex caught as non-functional in CI — see
|
||||
the `v0.41+: contributor-mode CI capture` TODO in `TODOS.md` for the deferred
|
||||
real-query version).
|
||||
|
||||
How it works:
|
||||
- Hand-curated qrels fixture at `test/fixtures/eval-baselines/qrels-search.json`
|
||||
with PLACEHOLDER names only (no real people / companies per CLAUDE.md privacy
|
||||
rule).
|
||||
- The test seeds a PGLite engine with synthetic pages whose embeddings are
|
||||
basis vectors (the same `basisEmbedding(idx)` pattern as
|
||||
`test/e2e/search-quality.test.ts`). No API keys, no DATABASE_URL.
|
||||
- For each qrels query, calls `engine.searchVector(basisEmbedding(dim))` and
|
||||
computes `top1_match_rate` and `recall@10`. Asserts both meet floors
|
||||
(`>= 0.80` and `>= 0.85` by default).
|
||||
- Lives in the unit-shard test matrix (`.github/workflows/test.yml`) so it
|
||||
runs on every PR via `bun test`, NOT in the E2E fixed-file workflow.
|
||||
|
||||
#### Refreshing the qrels fixture (the `Why:` discipline, D4)
|
||||
|
||||
When CI fails because a legitimate ranking change moved expected slugs, the
|
||||
fix is to edit `qrels-search.json` directly. **Always include a `Why:` line
|
||||
in the commit body** so future maintainers can read the audit trail. Without
|
||||
the `Why:`, the gate degrades to a rubber stamp within months. The convention
|
||||
is informational (not a commit-hook block), but enforce it in PR review.
|
||||
|
||||
Example commit body:
|
||||
|
||||
```
|
||||
chore(eval): refresh qrels for new source-boost ordering
|
||||
|
||||
Why: v0.40.x source-boost now weights originals/ over concepts/, so
|
||||
q12 (founder-mode) now correctly surfaces originals/founder-mode-example
|
||||
top-1. Manual verification: ran the production query; new ranking is
|
||||
clearly better-aligned with the query intent.
|
||||
```
|
||||
|
||||
#### Env-overrides for floors
|
||||
|
||||
```bash
|
||||
GBRAIN_REPLAY_GATE_TOP1_FLOOR=0.85 \
|
||||
GBRAIN_REPLAY_GATE_RECALL_FLOOR=0.90 \
|
||||
bun test test/eval-replay-gate.test.ts
|
||||
```
|
||||
|
||||
Use to tighten or loosen the gate as the qrels fixture matures.
|
||||
|
||||
### `gbrain eval cross-modal --batch` — batch quality scoring
|
||||
|
||||
Single-task cross-modal eval scores one (task, output) pair. Batch mode runs
|
||||
the same scoring over an entire LongMemEval JSONL output, with cost guardrails.
|
||||
|
||||
```bash
|
||||
# Step 1: produce LongMemEval hypotheses (real cost: depends on model + N).
|
||||
gbrain eval longmemeval ~/datasets/longmemeval_s.jsonl \
|
||||
--limit 10 --output /tmp/run.jsonl
|
||||
|
||||
# Step 2: batch-score those hypotheses (real cost: ~$0.70 for 10 questions,
|
||||
# 1 cycle, 3 model slots at default --max-usd 5 budget cap).
|
||||
gbrain eval cross-modal --batch /tmp/run.jsonl \
|
||||
--limit 10 --cycles 1 --concurrent 3 --max-usd 5 --json
|
||||
echo "exit=$?" # 0=all-pass, 1=any-fail, 2=any-error-or-inconclusive
|
||||
```
|
||||
|
||||
**Key behaviors:**
|
||||
- Default `--cycles 1` in batch mode (single-task default is 3 in TTY) to bound
|
||||
cost. Pass `--cycles 3` to match single-task strictness.
|
||||
- `--concurrent 3` runs up to 3 questions in parallel x 3 model slots each =
|
||||
9 simultaneous API calls. Below tier-1 rate limits for all three providers.
|
||||
- `--max-usd FLOAT` refuses to start if the pre-flight cost estimate exceeds
|
||||
the cap, unless `--yes` bypasses (required for non-interactive cron / CI).
|
||||
- Filters `kind: "by_type_summary"` rows automatically (the LongMemEval
|
||||
`--by-type` summary line is metadata, not a question).
|
||||
- `--batch` is mutually exclusive with `--task`; fail-fast usage error if both
|
||||
are set.
|
||||
- Exit precedence (fail-loud): ERROR > FAIL > INCONCLUSIVE > PASS.
|
||||
- Per-question receipts land in a tempdir and are deleted at end of batch; the
|
||||
summary inlines per-question verdicts so the audit trail is self-contained.
|
||||
|
||||
### Nightly cross-modal quality probe (opt-in, autopilot)
|
||||
|
||||
`src/core/cycle/nightly-quality-probe.ts` ships a phase that runs the longmemeval
|
||||
+ cross-modal pipeline once per 24h. **Disabled by default** to avoid surprise
|
||||
API spend. Enable per-host:
|
||||
|
||||
```bash
|
||||
gbrain config set autopilot.nightly_quality_probe.enabled true
|
||||
gbrain config set autopilot.nightly_quality_probe.max_usd 5.00 # optional override
|
||||
```
|
||||
|
||||
Note: `--phase nightly_quality_probe` wiring into the autopilot scheduler is
|
||||
deferred to a v0.41+ follow-up (see TODOS.md). For now the phase is callable
|
||||
in isolation; the test harness exercises it via DI stubs.
|
||||
|
||||
```bash
|
||||
# Manual smoke (exercises the path via DI stubs, no real API spend).
|
||||
bun test test/nightly-quality-probe.test.ts
|
||||
```
|
||||
|
||||
Observability:
|
||||
- `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` — one event per run with
|
||||
outcome (pass / fail / inconclusive / error / budget_exceeded /
|
||||
rate_limited / no_embedding_key), pass/fail/inconclusive/error counts,
|
||||
est_cost_usd, fixture_sha8. ISO-week rotation (mirrors slug-fallback
|
||||
audit).
|
||||
- `gbrain doctor` surfaces `nightly_quality_probe_health`:
|
||||
- SKIPPED (disabled) — with paste-ready enable command.
|
||||
- OK (enabled, no events yet) — autopilot hasn't fired its first run.
|
||||
- OK (last 7d all PASS) — with timestamp of latest run.
|
||||
- WARN — any FAIL / ERROR / BUDGET_EXCEEDED in the window, with outcome
|
||||
counts and the latest run's reason.
|
||||
|
||||
Real expected cost: ~$0.35 per nightly run (5 questions x 3 slots x 1 cycle
|
||||
x ~$0.02/call) ≈ $10.50/month. Worst-case under the default budget cap:
|
||||
$150/month. Opt-in default prevents discovering this in your card statement.
|
||||
|
||||
@@ -211,7 +211,7 @@ strict behavior when unset.
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO.
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x, extended v0.40.1.0 Track D) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO. **v0.40.1.0 Track D (T3+T4, per D1/D5/D6/D10):** new `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` flags fan out cross-modal scoring across a LongMemEval-shape JSONL. Mutually exclusive with `--task` (fail-fast usage error if both set). Filters `kind: "by_type_summary"` rows from the input (Codex #6). Pre-flight cost estimate refuses if `> --max-usd` without `--yes`; default cap 5.00 USD. Semaphore-bounded fan-out via inline `runWithLimit<T>(items, limit, fn)` (~15 LOC, exported for unit tests): max N questions in-flight at once × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run (per D10 — keeps `~/.gbrain/eval-receipts/` clean); the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (NEW batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})` at eval-longmemeval.ts:299; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by 15 cases in `test/eval-cross-modal-batch.test.ts`.
|
||||
- `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.
|
||||
- `src/core/cross-modal-eval/aggregate.ts` (v0.27.x) — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)` (Q2=A floor). Inconclusive when <2/3 models returned parseable scores (Q3=A regression guard for the v1 .mjs `Object.values({}).every(...) === true` empty-array PASS bug).
|
||||
- `src/core/cross-modal-eval/runner.ts` (v0.27.x) — orchestrator. Each cycle runs `Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])` (T4=A — bare allSettled, no rate-leases for the CLI path; minion-integration TODO recovers cross-process concurrency). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro`. `estimateCost()` exports a small per-model pricing table (drifts; refresh alongside model-family bumps).
|
||||
@@ -220,11 +220,13 @@ strict behavior when unset.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `test/eval-replay-gate.test.ts` + `test/fixtures/eval-baselines/qrels-search.json` (v0.40.1.0 Track D / T5, per D8 reshape) — hermetic retrieval qrels gate that runs in the standard PR unit-shard CI matrix (`.github/workflows/test.yml`, NOT the fixed-file E2E workflow). Structural replacement for the original "replay against captured `eval_candidates` baseline" design (deferred to `v0.41+: contributor-mode CI capture` in TODOS.md; Codex outside-voice caught three fatal flaws — select-e2e.ts is local-only, eval-export bypasses op-layer capture on PGLite-seed tests, replay re-embeds via gateway which needs an API key CI doesn't have). Uses the canonical PGLite block from CLAUDE.md test-isolation rules (R3+R4) and the basis-vector embedding pattern from `test/e2e/search-quality.test.ts:23-28` for fully hermetic retrieval. The qrels fixture (12 queries) is hand-curated with PLACEHOLDER names only (alice-example, widget-co-example, etc. — per Codex #9 + CLAUDE.md privacy rule) and embeds each query at a deterministic basis dimension so retrieval is reproducible. Each query lists `relevant_slugs[]` + `first_relevant_slug`; for each, the test computes `top1_match_rate` (top-1 == first_relevant) and `recall@10` (fraction of relevant_slugs in top-10), asserting both meet floors (defaults `>= 0.80` and `>= 0.85`). Env-overridable floors via `GBRAIN_REPLAY_GATE_TOP1_FLOOR` / `GBRAIN_REPLAY_GATE_RECALL_FLOOR` (with `withEnv()` per CLAUDE.md R1). Refresh discipline (per D4): when ranking changes intentionally move expected slugs, edit `qrels-search.json` directly and include a `Why:` line in the commit body so future maintainers can read the audit trail. Without `Why:`, the gate degrades to rubber-stamp within months. Pinned by 5 cases in `test/eval-replay-gate.test.ts` including a privacy-grep regression guard against real-name reintroduction.
|
||||
- `src/core/cycle/nightly-quality-probe.ts` + `src/core/audit-quality-probe.ts` + `test/fixtures/longmemeval-nightly.jsonl` + `test/nightly-quality-probe.test.ts` (v0.40.1.0 Track D / T6+T7+T8, per D12) — opt-in nightly cross-modal quality probe. The phase runs `gbrain eval longmemeval --by-type` against the committed 10-question placeholder fixture, pipes the output through `gbrain eval cross-modal --batch --max-usd 5 --yes`, and writes one event per run to `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` (ISO-week-rotated, mirrors `audit-slug-fallback.ts`; honors `GBRAIN_AUDIT_DIR`). Default DISABLED — opt-in via `gbrain config set autopilot.nightly_quality_probe.enabled true` (prevents surprise API spend on `gbrain init`). 24h rate limit (the pure `shouldRunNightly(now, recentEvents, windowMs?)` function) skips with audit row `outcome: rate_limited` when a recent run exists. Embedding-key short-circuit: longmemeval needs `gateway.embedQuery()`, so the phase exits early with `outcome: no_embedding_key` + stderr warn when no provider is configured. Full DI surface via `NightlyProbeDeps` (`isEnabled`, `hasEmbeddingProvider`, `resolveMaxUsd`, `resolveRepoRoot`, `runLongMemEval`, `runCrossModalBatch`, `now`) so the unit test stubs every external effect — no PGLite, no real LLM calls, no env mutation outside `withEnv()`. Cost ceiling: $5/run × 30 nights ≈ $150/month worst-case; expected real cost ~$0.35/night × 30 ≈ $10.50/month. **Autopilot scheduler wiring deferred to v0.41+ follow-up (filed in TODOS.md)** — the phase is callable in isolation today; the cycle-loop dispatcher hasn't been wired to invoke it on the 24h cadence yet. New `nightly_quality_probe_health` doctor check (in `src/commands/doctor.ts` right after `slug_fallback_audit`) reads the last 7 days of audit events: SKIPPED when feature flag is off (with paste-ready enable command); OK when enabled + all PASS; WARN on any FAIL / ERROR / BUDGET_EXCEEDED in the window with per-outcome counts. Pinned by 14 cases in `test/nightly-quality-probe.test.ts` (rate-limit pure-function unit tests + DI-stubbed end-to-end phase tests across every outcome branch).
|
||||
- `src/commands/eval-trajectory.ts` + `src/commands/founder-scorecard.ts` + `src/core/trajectory.ts` (v0.35.7) — temporal trajectory + founder scorecard. The wave that turns the v0.35.3.1 date-aware contradiction probe into a useful temporal substrate. `gbrain eval trajectory <entity>` shows the chronological typed-claim history (mrr/arr/team_size/etc) with regressions auto-flagged inline; `gbrain founder scorecard <entity>` rolls up claim_accuracy / consistency / growth_trajectory / red_flags into one JSON. Pure-function math lives in `trajectory.ts`: `detectRegressions(points, threshold)` walks consecutive metric-value pairs per metric (10% drop default, env override `GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD`); `computeDriftScore(points)` returns `1 - mean(cosine(emb[i], emb[i-1]))` over existing embeddings (null when <3 embedded points). Backed by `BrainEngine.findTrajectory(opts)` — both Postgres and PGLite, single SQL query, deterministic `ORDER BY valid_from ASC, id ASC` (R3). Source-scoped via the v0.34.1.0 `sourceId` scalar / `sourceIds` array dual pattern (D-CDX-6); visibility-filtered for remote callers (D-CDX-1) — `recall`-equivalent posture. MCP op `find_trajectory` (read scope, NOT localOnly) registered after `find_experts`. Migration v67 adds four optional typed-claim columns (`claim_metric`, `claim_value`, `claim_unit`, `claim_period`) + a partial index on `(entity_slug, claim_metric, valid_from) WHERE claim_metric IS NOT NULL`. Fence widens from 10 to 14 cells when any row has typed data; renderer stays at 10 cells when none do (no churn diff on existing fences). Metric labels normalize to lowercase snake_case via `normalizeMetricLabel` (15-entry seed map for common founder metrics). The `consolidate` cycle phase gains semantic upsert keyed on `(page_id, claim, since_date)` — fixes the pre-existing F4 duplicate-takes bug where re-running the full cycle after `extract_facts` cleared `consolidated_at` would silently append duplicate takes via `MAX(row_num)+1`. Also writes chronological `valid_until` on each cluster's older facts. The `extract_facts` cycle phase batch-embeds via `gateway.embed()` before insert AND threads `pages.effective_date` as the `pageEffectiveDate` fallback for `valid_from` (precedence chain: fence-row > pageEffectiveDate > now()). The contradiction probe MUST NOT write `valid_until` — R1+R8 grep guard at `test/eval-contradictions/no-valid-until-write.test.ts` pins this. Codex outside-voice round caught F1 (v66 collision → v67), F2 (Haiku lives in `facts/extract.ts` not `extract-facts.ts` cycle phase), F3 (cycle didn't embed before insert), F4 (idempotency bug), F5+F6 (missed `fence-write.ts` caller + no Page object there → pageEffectiveDate is OPTIONAL), F7 (privacy regression — visibility filter added), F8 (ParsedFact needed typed-field extension for markdown system-of-record), F9 (dual scalar+federated sourceId). Plan: `~/.claude/plans/system-instruction-you-are-working-curious-jellyfish.md`. Tests: 258 across 12 files.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`.
|
||||
- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1, extended v0.40.1.0 Track D) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec). **v0.40.1.0 Track D (T1+T2, per D9):** per-question JSONL row gains `question: string` (additive — `evaluate_qa.py` ignores unknown fields) so the `gbrain eval cross-modal --batch` consumer has the `task` text without joining back against the source dataset. Each row also carries `question_type: string` and `recall_hit?: boolean` so a `--resume-from` run can rebuild the cumulative `recallByType` from the file alone. New `--by-type` flag emits a `{schema_version:1, kind:"by_type_summary", recall_by_type:{...}, aggregate:{...}}` line as the FINAL line of the output; resume-replace strips any prior summary at the tail so 5 resumed runs produce 1 summary, not 5 (Codex #7). Empty-bucket guard: `aggregate.rate` is `null` (not NaN) when no questions had ground truth. Optional `--by-type-floor F` (0..1) exits non-zero with a stderr line per breached `question_type`; default informational only. Pure `buildByTypeSummary(buckets)` + `emitByTypeSummary(path, summary)` + `seedRecallByTypeFromFile(path, bucket)` helpers exported for unit tests. Pinned by 5 new cases in `test/eval-longmemeval.test.ts`.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.40.0.0",
|
||||
"version": "0.40.1.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -158,6 +158,11 @@ ALLOW_LIST=(
|
||||
'test/skillpack-harvest.test.ts'
|
||||
'test/e2e/skillpack-flow.test.ts'
|
||||
'skills/skillpack-harvest/SKILL.md'
|
||||
# v0.40.1.0 Track D / T5: the qrels gate test contains a privacy-grep
|
||||
# regression guard whose block list names the banned literal to assert
|
||||
# it's NOT in the qrels fixture. Same meta-rule-enforcement exception
|
||||
# as the other test entries above.
|
||||
'test/eval-replay-gate.test.ts'
|
||||
)
|
||||
|
||||
is_allowed() {
|
||||
|
||||
@@ -68,6 +68,16 @@ ALLOWLIST=(
|
||||
"test/skillpack-harvest.test.ts:Wintermute"
|
||||
"test/skillpack-harvest-lint.test.ts:Wintermute"
|
||||
"test/e2e/skillpack-flow.test.ts:Wintermute"
|
||||
# v0.40.1.0 Track D: eval-replay-gate.test.ts has a privacy-grep regression
|
||||
# guard whose block list necessarily SPELLS the real names so the test can
|
||||
# assert they're NOT in the qrels fixture. Same meta-rule exception as the
|
||||
# skillpack-harvest privacy tests above.
|
||||
"test/eval-replay-gate.test.ts:Pedro Franceschi"
|
||||
"test/eval-replay-gate.test.ts:Brex"
|
||||
"test/eval-replay-gate.test.ts:Wintermute"
|
||||
"test/eval-replay-gate.test.ts:Garry Tan"
|
||||
"test/eval-replay-gate.test.ts:Y Combinator"
|
||||
"test/eval-replay-gate.test.ts:YC"
|
||||
)
|
||||
|
||||
# Build the combined regex. Names matched as whole words (\b), emails matched
|
||||
|
||||
@@ -1552,6 +1552,60 @@ function _resolveSyncFreshnessHours(varName: string, fallback: number): number {
|
||||
* Failure messages embed `source.id` so the fix command
|
||||
* `gbrain sync --source <id>` matches what the user copy-pastes.
|
||||
*/
|
||||
/**
|
||||
* v0.40.1.0 Track D / T7 — pure function form of the nightly_quality_probe_health
|
||||
* check. Extracted from the inline runDoctor block so tests can drive every
|
||||
* branch (disabled / enabled-no-events / enabled-all-pass / enabled-with-failures)
|
||||
* without spinning up the audit JSONL or a real config file.
|
||||
*/
|
||||
export function computeNightlyQualityProbeHealthCheck(
|
||||
probeEnabled: boolean,
|
||||
events: ReadonlyArray<{ outcome: string; ts: string; detail?: string }>,
|
||||
): Check {
|
||||
const name = 'nightly_quality_probe_health';
|
||||
if (!probeEnabled && events.length === 0) {
|
||||
// Quiet skip — surface enable hint only when explicitly asked to.
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `disabled (opt-in). Enable with: gbrain config set autopilot.nightly_quality_probe.enabled true`,
|
||||
};
|
||||
}
|
||||
if (events.length === 0) {
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `enabled but no probe events in the last 7 days (next run by autopilot).`,
|
||||
};
|
||||
}
|
||||
// v0.40.1.0 Track D (codex CDX-5): any non-PASS outcome is bad signal.
|
||||
// Previously only fail / error / budget_exceeded triggered warn —
|
||||
// no_embedding_key / rate_limited / inconclusive were silently reported
|
||||
// as PASS, hiding real misconfigurations.
|
||||
const bad = events.filter(e => e.outcome !== 'pass');
|
||||
const latest = events[events.length - 1]!;
|
||||
if (bad.length > 0) {
|
||||
const counts =
|
||||
`pass=${events.filter(e => e.outcome === 'pass').length} ` +
|
||||
`fail=${events.filter(e => e.outcome === 'fail').length} ` +
|
||||
`error=${events.filter(e => e.outcome === 'error').length} ` +
|
||||
`inconclusive=${events.filter(e => e.outcome === 'inconclusive').length} ` +
|
||||
`budget=${events.filter(e => e.outcome === 'budget_exceeded').length} ` +
|
||||
`no_embed_key=${events.filter(e => e.outcome === 'no_embedding_key').length} ` +
|
||||
`rate_limited=${events.filter(e => e.outcome === 'rate_limited').length}`;
|
||||
return {
|
||||
name,
|
||||
status: 'warn',
|
||||
message: `${bad.length} non-PASS run${bad.length === 1 ? '' : 's'} in last 7d (${counts}). Latest: ${latest.outcome} at ${latest.ts}${latest.detail ? ` (${latest.detail})` : ''}.`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: `${events.length} PASS run${events.length === 1 ? '' : 's'} in last 7d. Latest: ${latest.ts}.`,
|
||||
};
|
||||
}
|
||||
|
||||
export async function checkSyncFreshness(
|
||||
engine: BrainEngine,
|
||||
opts?: { nowMs?: number },
|
||||
@@ -2158,6 +2212,27 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// Best-effort; audit-log read failure shouldn't stop doctor.
|
||||
}
|
||||
|
||||
// 3d.1 Nightly quality probe (v0.40.1.0 Track D / T7). Reads the last
|
||||
// 7 days of quality-probe-YYYY-Www.jsonl audit events. SKIPPED with
|
||||
// paste-ready enable hint when the feature is opt-in disabled (default).
|
||||
// WARN on any FAIL / ERROR / BUDGET_EXCEEDED row in the window; OK when
|
||||
// all rows are PASS. The probe itself is wired into autopilot, NOT into
|
||||
// doctor — doctor just surfaces what the probe wrote.
|
||||
try {
|
||||
const { readRecentQualityProbeEvents } = await import('../core/audit-quality-probe.ts');
|
||||
const { loadConfig } = await import('../core/config.ts');
|
||||
let probeEnabled = false;
|
||||
try {
|
||||
const cfg = loadConfig();
|
||||
probeEnabled = Boolean((cfg as any)?.autopilot?.nightly_quality_probe?.enabled);
|
||||
} catch { /* config unavailable → treat as disabled */ }
|
||||
const events = readRecentQualityProbeEvents(7);
|
||||
const check = computeNightlyQualityProbeHealthCheck(probeEnabled, events);
|
||||
checks.push(check);
|
||||
} catch {
|
||||
// Best-effort; audit-log read failure shouldn't stop doctor.
|
||||
}
|
||||
|
||||
// 3e. home_dir_in_worktree (v0.35.8.0). Walks up from `gbrainPath()`
|
||||
// looking for a `.git` directory OR file. If found, warns: `~/.gbrain/`
|
||||
// lives inside a git worktree, so an accidental `git add` from the
|
||||
|
||||
@@ -16,7 +16,10 @@
|
||||
* - `--budget-usd` hard cap is a v0.27.x follow-up TODO.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from 'fs';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { createHash } from 'crypto';
|
||||
|
||||
import { gbrainPath, loadConfig } from '../core/config.ts';
|
||||
import { configureGateway, isAvailable } from '../core/ai/gateway.ts';
|
||||
@@ -36,13 +39,33 @@ const HELP = `gbrain eval cross-modal — multi-model quality gate
|
||||
|
||||
USAGE:
|
||||
gbrain eval cross-modal --task "<description>" --output <path-or-skill-slug> [flags]
|
||||
gbrain eval cross-modal --batch <jsonl> [--limit N] [--output <receipt-path>] [flags] (v0.40.1.0 Track D)
|
||||
|
||||
REQUIRED:
|
||||
REQUIRED (single-task mode):
|
||||
--task "..." What the OUTPUT was meant to achieve.
|
||||
--output <path> File whose content gets scored. Pass a skill slug
|
||||
shortcut (e.g. \`--output skills/my-skill/SKILL.md\`)
|
||||
to bind the receipt to that skill (T10).
|
||||
|
||||
REQUIRED (batch mode, v0.40.1.0 Track D / T3):
|
||||
--batch <jsonl> LongMemEval-shape JSONL (output of \`gbrain eval
|
||||
longmemeval --output\`). Each row: {question, hypothesis,
|
||||
question_id, ...}. Summary rows (kind:by_type_summary)
|
||||
are filtered out. Mutually exclusive with --task.
|
||||
|
||||
BATCH FLAGS:
|
||||
--limit N Slice the first N rows (default 10).
|
||||
--concurrent N Semaphore cap; max N questions in-flight at once
|
||||
(default 3, ceiling = 3 questions x 3 model slots =
|
||||
9 parallel API calls). Per D6.
|
||||
--max-usd FLOAT Refuse to start if estimated cost exceeds this
|
||||
ceiling without --yes (default 5.00). Per D10.
|
||||
--yes Bypass the --max-usd refusal. Required for
|
||||
non-interactive (CI / cron) runs over the budget.
|
||||
--output PATH Where to write the SUMMARY receipt (NOT where to
|
||||
read agent response). Default:
|
||||
~/.gbrain/eval-receipts/cross-modal-batch-<sha8>.json
|
||||
|
||||
FLAGS:
|
||||
--slug <name> Receipt filename slug. Defaults to inferred slug
|
||||
from --output path (skills/<slug>/SKILL.md → <slug>),
|
||||
@@ -97,10 +120,16 @@ interface ParsedArgs {
|
||||
receiptDir?: string;
|
||||
maxTokens?: number;
|
||||
json: boolean;
|
||||
// v0.40.1.0 Track D / T3 — batch mode over LongMemEval-shape JSONL.
|
||||
batch?: string;
|
||||
limit?: number;
|
||||
concurrent?: number;
|
||||
maxUsd?: number;
|
||||
yes: boolean;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
const out: ParsedArgs = { help: false, json: false };
|
||||
const out: ParsedArgs = { help: false, json: false, yes: false };
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
const next = args[i + 1];
|
||||
@@ -119,6 +148,29 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
out.output = next;
|
||||
i++;
|
||||
break;
|
||||
case '--batch':
|
||||
if (next === undefined) break;
|
||||
out.batch = next;
|
||||
i++;
|
||||
break;
|
||||
case '--limit':
|
||||
if (next === undefined) break;
|
||||
out.limit = parseIntStrict(next);
|
||||
i++;
|
||||
break;
|
||||
case '--concurrent':
|
||||
if (next === undefined) break;
|
||||
out.concurrent = parseIntStrict(next);
|
||||
i++;
|
||||
break;
|
||||
case '--max-usd':
|
||||
if (next === undefined) break;
|
||||
out.maxUsd = parseFloatStrict(next);
|
||||
i++;
|
||||
break;
|
||||
case '--yes':
|
||||
out.yes = true;
|
||||
break;
|
||||
case '--slug':
|
||||
if (next === undefined) break;
|
||||
out.slug = next;
|
||||
@@ -175,6 +227,52 @@ function parseIntStrict(s: string): number {
|
||||
return parseInt(m, 10);
|
||||
}
|
||||
|
||||
function parseFloatStrict(s: string): number {
|
||||
const m = String(s).trim();
|
||||
const n = Number(m);
|
||||
if (!Number.isFinite(n) || n < 0) {
|
||||
throw new Error(`expected non-negative number, got: ${s}`);
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.1.0 Track D / T4 (per D6) — semaphore-bounded fan-out helper.
|
||||
* Runs `fn(item)` over `items` with at most `limit` in-flight at any moment.
|
||||
* Per-item errors are captured in the result (NOT thrown) so a single
|
||||
* failure does not abort the whole batch.
|
||||
*
|
||||
* Pinned by test/eval-cross-modal-batch.test.ts: never exceeds limit,
|
||||
* surfaces per-item errors, preserves input order in the result array.
|
||||
*/
|
||||
export async function runWithLimit<TIn, TOut>(
|
||||
items: readonly TIn[],
|
||||
limit: number,
|
||||
fn: (item: TIn, index: number) => Promise<TOut>,
|
||||
): Promise<Array<{ ok: true; value: TOut } | { ok: false; error: Error }>> {
|
||||
if (limit < 1) throw new Error(`runWithLimit: limit must be >= 1 (got ${limit})`);
|
||||
const results: Array<{ ok: true; value: TOut } | { ok: false; error: Error } | undefined> = new Array(items.length);
|
||||
let nextIndex = 0;
|
||||
const workers: Promise<void>[] = [];
|
||||
const workerCount = Math.min(limit, items.length);
|
||||
for (let w = 0; w < workerCount; w++) {
|
||||
workers.push((async () => {
|
||||
while (true) {
|
||||
const idx = nextIndex++;
|
||||
if (idx >= items.length) return;
|
||||
try {
|
||||
const value = await fn(items[idx]!, idx);
|
||||
results[idx] = { ok: true, value };
|
||||
} catch (err) {
|
||||
results[idx] = { ok: false, error: err instanceof Error ? err : new Error(String(err)) };
|
||||
}
|
||||
}
|
||||
})());
|
||||
}
|
||||
await Promise.all(workers);
|
||||
return results as Array<{ ok: true; value: TOut } | { ok: false; error: Error }>;
|
||||
}
|
||||
|
||||
function inferSlugFromOutputPath(path: string): string | undefined {
|
||||
// skills/<slug>/SKILL.md or .../skills/<slug>/...
|
||||
const m = path.replace(/\\/g, '/').match(/(?:^|\/)skills\/([^/]+)\/SKILL\.md$/);
|
||||
@@ -220,7 +318,17 @@ function configureGatewayForCli(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
export async function runEvalCrossModal(args: string[]): Promise<number> {
|
||||
/**
|
||||
* v0.40.1.0 Track D / T3 (per D5) — DI seam for hermetic batch tests.
|
||||
* Mirrors the `runEvalLongMemEval(args, {client?})` pattern at
|
||||
* eval-longmemeval.ts:299. Default code path uses the imported runEval;
|
||||
* tests pass `opts.runEval` returning canned RunEvalResult.
|
||||
*/
|
||||
export interface RunCrossModalOpts {
|
||||
runEval?: typeof runEval;
|
||||
}
|
||||
|
||||
export async function runEvalCrossModal(args: string[], opts: RunCrossModalOpts = {}): Promise<number> {
|
||||
const parsed = parseArgs(args);
|
||||
|
||||
if (parsed.help) {
|
||||
@@ -228,8 +336,19 @@ export async function runEvalCrossModal(args: string[]): Promise<number> {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// v0.40.1.0 Track D / T3 — --batch vs --task mutex.
|
||||
if (parsed.batch && parsed.task) {
|
||||
process.stderr.write('Error: --batch and --task are mutually exclusive\n');
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Batch path — short-circuit before the single-task validation below.
|
||||
if (parsed.batch) {
|
||||
return runBatchMode(parsed, opts);
|
||||
}
|
||||
|
||||
if (!parsed.task) {
|
||||
process.stderr.write('Error: --task "<description>" is required\n\n');
|
||||
process.stderr.write('Error: --task "<description>" is required (or --batch <jsonl>)\n\n');
|
||||
process.stderr.write(HELP);
|
||||
return 1;
|
||||
}
|
||||
@@ -306,9 +425,10 @@ export async function runEvalCrossModal(args: string[]): Promise<number> {
|
||||
}
|
||||
};
|
||||
|
||||
const runEvalFn = opts.runEval ?? runEval;
|
||||
let result: RunEvalResult;
|
||||
try {
|
||||
result = await runEval({
|
||||
result = await runEvalFn({
|
||||
task: parsed.task,
|
||||
output: outputContent,
|
||||
slug,
|
||||
@@ -356,3 +476,374 @@ export async function runEvalCrossModal(args: string[]): Promise<number> {
|
||||
if (verdict === 'inconclusive') return 2;
|
||||
return 1;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// v0.40.1.0 Track D / T3 — Batch mode over LongMemEval-shape JSONL.
|
||||
//
|
||||
// Reads {question, hypothesis, ...} rows, slices to --limit, fans out via
|
||||
// runWithLimit semaphore, aggregates per-question verdicts into a single
|
||||
// summary receipt. Per-question receipts land in a tempdir and are deleted
|
||||
// at end of run (per D10 — keeps ~/.gbrain/eval-receipts/ clean).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface BatchRow {
|
||||
question_id: string;
|
||||
question: string;
|
||||
hypothesis: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.1.0 Track D (codex CDX-1) — upstream-error row from
|
||||
* `gbrain eval longmemeval`. Carries `question`+`question_type` and an
|
||||
* `error` field but no usable hypothesis. Counted in the batch summary's
|
||||
* `upstream_error_count` so the denominator includes failed rows, never
|
||||
* silently dropped (which would let the gate pass on a surviving subset).
|
||||
*/
|
||||
interface UpstreamErrorRow {
|
||||
question_id: string;
|
||||
question: string;
|
||||
question_type?: string;
|
||||
error: string;
|
||||
}
|
||||
|
||||
interface BatchReadResult {
|
||||
rows: BatchRow[];
|
||||
upstream_errors: UpstreamErrorRow[];
|
||||
malformed_count: number;
|
||||
}
|
||||
|
||||
export interface BatchSummary {
|
||||
schema_version: 1;
|
||||
kind: 'cross_modal_batch_summary';
|
||||
timestamp: string;
|
||||
/** Sum of scored + upstream_error + malformed rows. Real denominator. */
|
||||
total: number;
|
||||
pass_count: number;
|
||||
fail_count: number;
|
||||
inconclusive_count: number;
|
||||
/** Per-question runtime errors from the cross-modal scoring layer. */
|
||||
error_count: number;
|
||||
/**
|
||||
* v0.40.1.0 Track D (codex CDX-1) — rows that arrived from the upstream
|
||||
* eval already failed (longmemeval emitted an error row with no usable
|
||||
* hypothesis). Counted in `total` so the denominator can't bypass the gate.
|
||||
*/
|
||||
upstream_error_count: number;
|
||||
/**
|
||||
* v0.40.1.0 Track D (codex CDX-1) — JSONL rows that didn't have the
|
||||
* required shape (missing question or hypothesis, not a tagged error row).
|
||||
* Counted in `total`; treated as ERROR for exit precedence.
|
||||
*/
|
||||
malformed_count: number;
|
||||
verdict: 'pass' | 'fail' | 'inconclusive' | 'error';
|
||||
est_cost_usd: number;
|
||||
slots: SlotConfig[];
|
||||
cycles_per_question: number;
|
||||
concurrent: number;
|
||||
per_question: Array<{
|
||||
question_id: string;
|
||||
verdict: 'pass' | 'fail' | 'inconclusive' | 'error' | 'upstream_error';
|
||||
error?: string;
|
||||
final_aggregate?: unknown;
|
||||
}>;
|
||||
}
|
||||
|
||||
function readBatchRows(path: string): BatchReadResult {
|
||||
if (!existsSync(path)) {
|
||||
throw new Error(`--batch file not found: ${path}`);
|
||||
}
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const rows: BatchRow[] = [];
|
||||
const upstream_errors: UpstreamErrorRow[] = [];
|
||||
let lineNo = 0;
|
||||
let summarySkipped = 0;
|
||||
let parseErrors = 0;
|
||||
let malformed_count = 0;
|
||||
for (const line of raw.split('\n')) {
|
||||
lineNo++;
|
||||
if (!line.trim()) continue;
|
||||
let obj: any;
|
||||
try {
|
||||
obj = JSON.parse(line);
|
||||
} catch {
|
||||
parseErrors++;
|
||||
process.stderr.write(`[eval cross-modal batch] skipping invalid JSON at line ${lineNo}\n`);
|
||||
continue;
|
||||
}
|
||||
if (!obj || typeof obj !== 'object') continue;
|
||||
// Skip the by_type_summary tail row — metadata, not a question.
|
||||
if (obj.kind === 'by_type_summary') {
|
||||
summarySkipped++;
|
||||
continue;
|
||||
}
|
||||
// v0.40.1.0 Track D (codex CDX-1): upstream error rows from
|
||||
// `gbrain eval longmemeval` carry an `error` field and an empty/missing
|
||||
// hypothesis. Treat them as upstream_error verdicts in the batch summary
|
||||
// so they count in the denominator instead of silently disappearing.
|
||||
if (typeof obj.error === 'string' && obj.error.length > 0) {
|
||||
upstream_errors.push({
|
||||
question_id: typeof obj.question_id === 'string' ? obj.question_id : `line-${lineNo}`,
|
||||
question: typeof obj.question === 'string' ? obj.question : '',
|
||||
...(typeof obj.question_type === 'string' ? { question_type: obj.question_type } : {}),
|
||||
error: obj.error,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (typeof obj.question !== 'string' || typeof obj.hypothesis !== 'string') {
|
||||
// Row missing required fields — malformed, count it. We count instead
|
||||
// of silently dropping so the batch summary can surface the loss.
|
||||
malformed_count++;
|
||||
process.stderr.write(
|
||||
`[eval cross-modal batch] skipping malformed row at line ${lineNo}: ` +
|
||||
`missing question or hypothesis field\n`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
rows.push({
|
||||
question_id: typeof obj.question_id === 'string' ? obj.question_id : `line-${lineNo}`,
|
||||
question: obj.question,
|
||||
hypothesis: obj.hypothesis,
|
||||
});
|
||||
}
|
||||
if (summarySkipped > 0) {
|
||||
process.stderr.write(`[eval cross-modal batch] filtered ${summarySkipped} summary row(s)\n`);
|
||||
}
|
||||
if (upstream_errors.length > 0) {
|
||||
process.stderr.write(
|
||||
`[eval cross-modal batch] ${upstream_errors.length} upstream-error row(s) detected; ` +
|
||||
`they will count in the batch denominator as ERROR verdicts.\n`,
|
||||
);
|
||||
}
|
||||
if (malformed_count > 0) {
|
||||
process.stderr.write(
|
||||
`[eval cross-modal batch] ${malformed_count} malformed row(s) skipped. ` +
|
||||
`Batch will FAIL — re-run upstream eval to fix.\n`,
|
||||
);
|
||||
}
|
||||
if (parseErrors > 0) {
|
||||
process.stderr.write(`[eval cross-modal batch] skipped ${parseErrors} corrupt line(s)\n`);
|
||||
// Corrupt JSON lines roll into malformed for exit-precedence purposes.
|
||||
malformed_count += parseErrors;
|
||||
}
|
||||
return { rows, upstream_errors, malformed_count };
|
||||
}
|
||||
|
||||
async function runBatchMode(parsed: ParsedArgs, opts: RunCrossModalOpts): Promise<number> {
|
||||
// Defaults specific to batch mode.
|
||||
const limit = parsed.limit ?? 10;
|
||||
const concurrent = parsed.concurrent ?? 3;
|
||||
const cycles = parsed.cycles ?? 1; // default 1 in batch to bound cost
|
||||
const dimensions = parsed.dimensions ?? DEFAULT_DIMENSIONS;
|
||||
const maxTokens = parsed.maxTokens ?? 4000;
|
||||
const maxUsd = parsed.maxUsd ?? 5.0;
|
||||
|
||||
const slots: SlotConfig[] = [
|
||||
{ id: 'A', model: parsed.slotAModel ?? DEFAULT_SLOTS[0]!.model },
|
||||
{ id: 'B', model: parsed.slotBModel ?? DEFAULT_SLOTS[1]!.model },
|
||||
{ id: 'C', model: parsed.slotCModel ?? DEFAULT_SLOTS[2]!.model },
|
||||
];
|
||||
|
||||
// v0.40.1.0 Track D (codex CDX-2): --limit must be >= 1. Passing
|
||||
// --limit 0 would let an empty result fall through to PASS with
|
||||
// total:0 — a direct CI bypass. Fail fast.
|
||||
if (limit < 1) {
|
||||
process.stderr.write(
|
||||
`Error: --limit must be >= 1 (got ${limit}). --limit 0 would bypass the gate.\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Read + slice rows BEFORE configuring the gateway so a malformed batch
|
||||
// file fails fast without spending any setup time.
|
||||
let rows: BatchRow[];
|
||||
let upstreamErrors: UpstreamErrorRow[];
|
||||
let malformedCount: number;
|
||||
try {
|
||||
const result = readBatchRows(parsed.batch!);
|
||||
rows = result.rows;
|
||||
upstreamErrors = result.upstream_errors;
|
||||
malformedCount = result.malformed_count;
|
||||
} catch (err) {
|
||||
process.stderr.write(`Error: ${err instanceof Error ? err.message : String(err)}\n`);
|
||||
return 1;
|
||||
}
|
||||
// Total usable inputs = scored rows + upstream errors (CDX-1: both count
|
||||
// in the denominator). Zero usable inputs means there's nothing to do
|
||||
// AND nothing to flag — the upstream eval produced no output at all.
|
||||
if (rows.length === 0 && upstreamErrors.length === 0) {
|
||||
process.stderr.write(`Error: --batch file has zero usable rows\n`);
|
||||
return 1;
|
||||
}
|
||||
if (limit < rows.length) rows = rows.slice(0, limit);
|
||||
|
||||
// Pre-flight cost estimate. Refuse if over --max-usd without --yes.
|
||||
const perQuestion = estimateCost(slots, cycles, maxTokens);
|
||||
const estTotal = perQuestion.perRunMaxUSD * rows.length;
|
||||
process.stderr.write(
|
||||
`[eval cross-modal batch] estimated cost: ~$${estTotal.toFixed(2)} ` +
|
||||
`for ${rows.length} questions x ${cycles} cycle(s) x 3 slots ` +
|
||||
`(per-question ~$${perQuestion.perRunMaxUSD.toFixed(2)}, concurrent=${concurrent})\n`,
|
||||
);
|
||||
if (estTotal > maxUsd && !parsed.yes) {
|
||||
process.stderr.write(
|
||||
`Error: estimated cost $${estTotal.toFixed(2)} exceeds --max-usd $${maxUsd.toFixed(2)}; ` +
|
||||
`pass --yes to proceed or lower --limit / --cycles.\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Configure gateway (same path as single-task mode). When runEval is
|
||||
// injected (test mode), skip the gateway availability gate — the injected
|
||||
// function handles its own backend, so requiring an API key here would
|
||||
// make hermetic unit tests impossible.
|
||||
if (!opts.runEval) {
|
||||
configureGatewayForCli();
|
||||
if (!isAvailable('chat')) {
|
||||
process.stderr.write(
|
||||
'Error: AI gateway has no usable chat provider. ' +
|
||||
'Configure one of OPENAI_API_KEY / ANTHROPIC_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY.\n',
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Per-question receipts land in a tempdir; we delete the tempdir at the
|
||||
// end of the batch (per D10 — keeps ~/.gbrain/eval-receipts/ clean).
|
||||
const batchTempDir = mkdtempSync(join(tmpdir(), 'gbrain-batch-receipts-'));
|
||||
const runEvalFn = opts.runEval ?? runEval;
|
||||
|
||||
try {
|
||||
const results = await runWithLimit(rows, concurrent, async (row, idx) => {
|
||||
process.stderr.write(`[eval cross-modal batch] ${idx + 1}/${rows.length} ${row.question_id} starting...\n`);
|
||||
return await runEvalFn({
|
||||
task: row.question,
|
||||
output: row.hypothesis,
|
||||
slug: row.question_id,
|
||||
dimensions,
|
||||
slots,
|
||||
cycles,
|
||||
receiptDir: batchTempDir,
|
||||
maxTokens,
|
||||
});
|
||||
});
|
||||
|
||||
// Aggregate verdicts.
|
||||
let pass = 0, fail = 0, inconclusive = 0, errored = 0;
|
||||
const perQuestionResults: BatchSummary['per_question'] = [];
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i]!;
|
||||
const qid = rows[i]!.question_id;
|
||||
if (!r.ok) {
|
||||
errored++;
|
||||
perQuestionResults.push({ question_id: qid, verdict: 'error', error: r.error.message });
|
||||
continue;
|
||||
}
|
||||
const v = r.value.finalAggregate.verdict;
|
||||
if (v === 'pass') pass++;
|
||||
else if (v === 'fail') fail++;
|
||||
else inconclusive++;
|
||||
perQuestionResults.push({
|
||||
question_id: qid,
|
||||
verdict: v,
|
||||
final_aggregate: r.value.finalAggregate,
|
||||
});
|
||||
}
|
||||
|
||||
// v0.40.1.0 Track D (codex CDX-1): upstream errors from the upstream
|
||||
// eval are folded into per_question with verdict 'upstream_error' so
|
||||
// the audit trail is complete. They count in the ERROR exit precedence.
|
||||
for (const ue of upstreamErrors) {
|
||||
perQuestionResults.push({
|
||||
question_id: ue.question_id,
|
||||
verdict: 'upstream_error',
|
||||
error: ue.error,
|
||||
});
|
||||
}
|
||||
|
||||
// v0.40.1.0 Track D (codex CDX-1): malformed rows can't be scored AND
|
||||
// can't be cited (no question text). They count toward total + ERROR
|
||||
// exit code so a corrupt JSONL can't silently shrink the denominator.
|
||||
|
||||
const upstreamErrorCount = upstreamErrors.length;
|
||||
const totalDenom = rows.length + upstreamErrorCount + malformedCount;
|
||||
|
||||
// Exit precedence (fail-loud convention; CDX-1 widens "ERROR" to include
|
||||
// upstream and malformed):
|
||||
// any error / upstream_error / malformed → 2
|
||||
// else any FAIL → 1
|
||||
// else any INCONCLUSIVE → 2
|
||||
// else 0 (all PASS)
|
||||
let batchVerdict: BatchSummary['verdict'];
|
||||
let exitCode: number;
|
||||
if (errored > 0 || upstreamErrorCount > 0 || malformedCount > 0) {
|
||||
batchVerdict = 'error';
|
||||
exitCode = 2;
|
||||
} else if (fail > 0) { batchVerdict = 'fail'; exitCode = 1; }
|
||||
else if (inconclusive > 0) { batchVerdict = 'inconclusive'; exitCode = 2; }
|
||||
else { batchVerdict = 'pass'; exitCode = 0; }
|
||||
|
||||
const summary: BatchSummary = {
|
||||
schema_version: 1,
|
||||
kind: 'cross_modal_batch_summary',
|
||||
timestamp: new Date().toISOString(),
|
||||
total: totalDenom,
|
||||
pass_count: pass,
|
||||
fail_count: fail,
|
||||
inconclusive_count: inconclusive,
|
||||
error_count: errored,
|
||||
upstream_error_count: upstreamErrorCount,
|
||||
malformed_count: malformedCount,
|
||||
verdict: batchVerdict,
|
||||
est_cost_usd: estTotal,
|
||||
slots,
|
||||
cycles_per_question: cycles,
|
||||
concurrent,
|
||||
per_question: perQuestionResults,
|
||||
};
|
||||
|
||||
// Write summary to --output or default path.
|
||||
const summaryPath = parsed.output ??
|
||||
join(gbrainPath('eval-receipts'), `cross-modal-batch-${batchSha8(summary)}.json`);
|
||||
// Ensure receipts dir exists (the inline ad-hoc default path bypasses
|
||||
// the per-cycle runEval mkdir).
|
||||
try {
|
||||
const summaryDir = summaryPath.substring(0, summaryPath.lastIndexOf('/'));
|
||||
if (summaryDir && !existsSync(summaryDir)) {
|
||||
mkdirSync(summaryDir, { recursive: true });
|
||||
}
|
||||
} catch { /* best-effort */ }
|
||||
writeFileSync(summaryPath, JSON.stringify(summary, null, 2) + '\n', 'utf8');
|
||||
|
||||
process.stderr.write(
|
||||
`\n[eval cross-modal batch] verdict=${batchVerdict} ` +
|
||||
`pass=${pass} fail=${fail} inconclusive=${inconclusive} ` +
|
||||
`error=${errored} upstream_error=${upstreamErrorCount} malformed=${malformedCount} ` +
|
||||
`(total ${totalDenom})\n`,
|
||||
);
|
||||
process.stderr.write(`[eval cross-modal batch] summary receipt: ${summaryPath}\n`);
|
||||
|
||||
if (parsed.json) {
|
||||
process.stdout.write(JSON.stringify(summary, null, 2));
|
||||
process.stdout.write('\n');
|
||||
}
|
||||
|
||||
return exitCode;
|
||||
} finally {
|
||||
// Clean up the per-question receipt tempdir (per D10).
|
||||
try {
|
||||
rmSync(batchTempDir, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
process.stderr.write(
|
||||
`[eval cross-modal batch] warning: tempdir cleanup failed: ` +
|
||||
`${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function batchSha8(summary: BatchSummary): string {
|
||||
return createHash('sha256')
|
||||
.update(JSON.stringify({ ts: summary.timestamp, n: summary.total, ids: summary.per_question.map(p => p.question_id) }))
|
||||
.digest('hex')
|
||||
.slice(0, 8);
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
* ThinkLLMClient so the full pipeline runs without any API key.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync, openSync, writeSync, closeSync } from 'fs';
|
||||
import { readFileSync, existsSync, openSync, writeSync, closeSync, writeFileSync } from 'fs';
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
import { withBenchmarkBrain, resetTables } from '../eval/longmemeval/harness.ts';
|
||||
import { haystackToPages, type LongMemEvalQuestion } from '../eval/longmemeval/adapter.ts';
|
||||
@@ -46,6 +46,18 @@ interface ParsedArgs {
|
||||
* Recovery path for mid-run aborts (rate-limit, cost-cap, OS interrupt).
|
||||
*/
|
||||
resumeFromPath?: string;
|
||||
/**
|
||||
* v0.40.1.0 (Track D / T2) — emit a final aggregate JSON line keyed by
|
||||
* question_type with per-bucket hit/total/rate plus aggregate stats. The
|
||||
* summary is the LAST line of the output. Resume-safe: if a prior summary
|
||||
* exists at the tail it is replaced, not appended.
|
||||
*/
|
||||
byType: boolean;
|
||||
/**
|
||||
* v0.40.1.0 (Track D / T2) — when set, exit non-zero if any question_type
|
||||
* rate falls below this floor. Default unset = informational only.
|
||||
*/
|
||||
byTypeFloor?: number;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
@@ -55,6 +67,7 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
keywordOnly: false,
|
||||
expansion: false,
|
||||
topK: 8,
|
||||
byType: false,
|
||||
};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
@@ -67,6 +80,16 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
if (a === '--top-k') { out.topK = Number(args[++i]); continue; }
|
||||
if (a === '--output') { out.outputPath = args[++i]; continue; }
|
||||
if (a === '--resume-from') { out.resumeFromPath = args[++i]; continue; }
|
||||
if (a === '--by-type') { out.byType = true; continue; }
|
||||
if (a === '--by-type-floor') {
|
||||
const v = Number(args[++i]);
|
||||
if (!Number.isFinite(v) || v < 0 || v > 1) {
|
||||
throw new Error(`--by-type-floor must be a number in [0, 1] (got: ${args[i]})`);
|
||||
}
|
||||
out.byTypeFloor = v;
|
||||
out.byType = true; // --by-type-floor implies --by-type
|
||||
continue;
|
||||
}
|
||||
if (a === '--mode') {
|
||||
const v = args[++i];
|
||||
if (v === 'conservative' || v === 'balanced' || v === 'tokenmax') {
|
||||
@@ -106,6 +129,12 @@ function printHelp(): void {
|
||||
` remaining questions. Typically the same path as --output\n` +
|
||||
` so the run continues writing in append mode. Recovery for\n` +
|
||||
` mid-run aborts (rate-limit, cost-cap, OS interrupt).\n` +
|
||||
` --by-type v0.40.1.0 — emit a final JSON line with per-question-type\n` +
|
||||
` R@k breakdown. Shape: {schema_version,kind:"by_type_summary",\n` +
|
||||
` recall_by_type:{...},aggregate:{...}}. Resume-safe: a prior\n` +
|
||||
` summary at the tail is REPLACED, not appended.\n` +
|
||||
` --by-type-floor F v0.40.1.0 — exit non-zero if any question_type rate < F\n` +
|
||||
` (range [0, 1]). Implies --by-type. Default: no gate.\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Note: a full 500-question run takes ~20-60 minutes depending on flags. Use\n` +
|
||||
`--limit during development.\n`,
|
||||
@@ -336,6 +365,29 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
}
|
||||
if (questions.length === 0) {
|
||||
process.stderr.write(`[longmemeval] resume: nothing to do (all questions already answered).\n`);
|
||||
// v0.40.1.0 Track D (codex CDX-3): even a no-op resume must run the
|
||||
// --by-type summary emission + --by-type-floor enforcement against
|
||||
// the existing file's rows. Skipping these steps would let a prior
|
||||
// run be resumed as "all done" and bypass the floor gate entirely.
|
||||
if (opts.byType && opts.outputPath) {
|
||||
const seededBucket: Record<string, { hit: number; total: number }> = {};
|
||||
seedRecallByTypeFromFile(opts.outputPath, seededBucket);
|
||||
const summary = buildByTypeSummary(seededBucket);
|
||||
emitByTypeSummary(opts.outputPath, summary);
|
||||
if (opts.byTypeFloor !== undefined) {
|
||||
const floor = opts.byTypeFloor;
|
||||
const breaches: string[] = [];
|
||||
for (const [t, v] of Object.entries(summary.recall_by_type)) {
|
||||
if (v.rate < floor) {
|
||||
breaches.push(`${t}: ${(v.rate * 100).toFixed(1)}% < ${(floor * 100).toFixed(1)}%`);
|
||||
}
|
||||
}
|
||||
if (breaches.length > 0) {
|
||||
process.stderr.write(`[longmemeval] FAIL --by-type-floor=${floor}: ${breaches.join(', ')}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -364,6 +416,12 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
|
||||
// Per-type accuracy counters (computed only when ground truth is reachable).
|
||||
const recallByType: Record<string, { hit: number; total: number }> = {};
|
||||
// v0.40.1.0 (Track D / T2) — when --by-type AND --resume-from point at a
|
||||
// file, seed the bucket from existing rows so the final summary is
|
||||
// cumulative (covers prior + new questions, not just this run's).
|
||||
if (opts.byType && opts.resumeFromPath) {
|
||||
seedRecallByTypeFromFile(opts.resumeFromPath, recallByType);
|
||||
}
|
||||
let runStart = Date.now();
|
||||
let errorCount = 0;
|
||||
|
||||
@@ -381,8 +439,15 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
progress.tick(1, q.question_id);
|
||||
} catch (err: any) {
|
||||
errorCount++;
|
||||
// v0.40.1.0 Track D (codex CDX-1): emit the `question` text on error
|
||||
// rows too so the cross-modal --batch consumer can flag them as
|
||||
// upstream errors instead of silently dropping them from the
|
||||
// denominator. Also carry question_type so by-type summary stays
|
||||
// accurate across error rows.
|
||||
emitter.emit({
|
||||
question_id: q.question_id,
|
||||
question: q.question,
|
||||
question_type: q.question_type,
|
||||
hypothesis: '',
|
||||
error: String(err?.message ?? err),
|
||||
});
|
||||
@@ -409,6 +474,27 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
process.stderr.write(` ${t}: ${v.hit}/${v.total} (${pct.toFixed(1)}%)\n`);
|
||||
}
|
||||
}
|
||||
|
||||
// v0.40.1.0 (Track D / T2) — emit by_type_summary as the FINAL line if
|
||||
// --by-type was set. Note: emitter is closed above so this rewrite
|
||||
// operates on the released file descriptor.
|
||||
if (opts.byType) {
|
||||
const summary = buildByTypeSummary(recallByType);
|
||||
emitByTypeSummary(opts.outputPath, summary);
|
||||
// Optional floor gate. Fail-loud: exit 1 with a stderr line per type that
|
||||
// breached the floor so the operator sees exactly which type regressed.
|
||||
if (opts.byTypeFloor !== undefined) {
|
||||
const floor = opts.byTypeFloor;
|
||||
const breaches: string[] = [];
|
||||
for (const [t, v] of Object.entries(summary.recall_by_type)) {
|
||||
if (v.rate < floor) breaches.push(`${t}: ${(v.rate * 100).toFixed(1)}% < ${(floor * 100).toFixed(1)}%`);
|
||||
}
|
||||
if (breaches.length > 0) {
|
||||
process.stderr.write(`[longmemeval] FAIL --by-type-floor=${floor}: ${breaches.join(', ')}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runOneQuestion(
|
||||
@@ -456,12 +542,146 @@ async function runOneQuestion(
|
||||
? renderRetrievedAsHypothesis(results)
|
||||
: await generateAnswer(client, q.question, results, pageMeta, model);
|
||||
|
||||
// v0.40.1.0 (Track D / T2) — compute per-row hit/miss so resume runs can
|
||||
// rebuild the cumulative recallByType from the file alone. Undefined when
|
||||
// the dataset has no ground-truth answer_session_ids for this question.
|
||||
let recallHit: boolean | undefined;
|
||||
if (q.answer_session_ids && q.answer_session_ids.length > 0) {
|
||||
const gt = new Set(q.answer_session_ids);
|
||||
recallHit = retrievedSessionIds.some(s => gt.has(s));
|
||||
}
|
||||
|
||||
emitter.emit({
|
||||
question_id: q.question_id,
|
||||
// v0.40.1.0 (Track D / T1, per D9) — emit the question text so the
|
||||
// cross-modal --batch consumer has the `task` it needs without joining
|
||||
// back against the source dataset.
|
||||
question: q.question,
|
||||
// v0.40.1.0 (Track D / T2) — copy question_type into the row so the
|
||||
// by_type_summary can be rebuilt from the file on resume runs.
|
||||
question_type: q.question_type,
|
||||
hypothesis,
|
||||
retrieved_session_ids: retrievedSessionIds,
|
||||
...(recallHit !== undefined ? { recall_hit: recallHit } : {}),
|
||||
// v0.32.3 — record the active mode in every per-question row so reviewers
|
||||
// can group/compare without re-running. Omitted when --mode is unset.
|
||||
...(opts.mode ? { mode: opts.mode } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.1.0 (Track D / T2) — Seed `recallByType` from an existing output
|
||||
* file so the by_type_summary is cumulative across resume runs (not just
|
||||
* "this run's questions"). Rows missing `recall_hit` are skipped (dataset
|
||||
* had no ground truth for them) and the by_type_summary rows are skipped
|
||||
* (they're aggregates, not source data).
|
||||
*
|
||||
* Best-effort: corrupt lines are silently skipped; the resume loader has
|
||||
* its own corrupt-line logging.
|
||||
*/
|
||||
export function seedRecallByTypeFromFile(
|
||||
outputPath: string,
|
||||
bucket: Record<string, { hit: number; total: number }>,
|
||||
): void {
|
||||
if (!existsSync(outputPath)) return;
|
||||
const raw = readFileSync(outputPath, 'utf8');
|
||||
for (const line of raw.split('\n')) {
|
||||
if (!line.trim()) continue;
|
||||
let row: any;
|
||||
try {
|
||||
row = JSON.parse(line);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!row || typeof row !== 'object') continue;
|
||||
if (row.kind === 'by_type_summary') continue;
|
||||
if (typeof row.question_type !== 'string') continue;
|
||||
if (typeof row.recall_hit !== 'boolean') continue;
|
||||
const b = bucket[row.question_type] ?? (bucket[row.question_type] = { hit: 0, total: 0 });
|
||||
b.total++;
|
||||
if (row.recall_hit) b.hit++;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.1.0 (Track D / T2) — Build the by_type_summary payload from the
|
||||
* per-type bucket. Pure function; deterministic key order via sort.
|
||||
*
|
||||
* Empty-bucket guard: when no buckets were populated (no ground-truth in
|
||||
* the dataset), `aggregate.rate` is `null` rather than NaN so downstream
|
||||
* JSON consumers don't trip.
|
||||
*/
|
||||
export interface ByTypeSummary {
|
||||
schema_version: 1;
|
||||
kind: 'by_type_summary';
|
||||
recall_by_type: Record<string, { hit: number; total: number; rate: number }>;
|
||||
aggregate: { hit: number; total: number; rate: number | null };
|
||||
}
|
||||
|
||||
export function buildByTypeSummary(
|
||||
recallByType: Record<string, { hit: number; total: number }>,
|
||||
): ByTypeSummary {
|
||||
const sortedKeys = Object.keys(recallByType).sort();
|
||||
const recall: Record<string, { hit: number; total: number; rate: number }> = {};
|
||||
let aggHit = 0;
|
||||
let aggTotal = 0;
|
||||
for (const k of sortedKeys) {
|
||||
const v = recallByType[k];
|
||||
const rate = v.total === 0 ? 0 : v.hit / v.total;
|
||||
recall[k] = { hit: v.hit, total: v.total, rate };
|
||||
aggHit += v.hit;
|
||||
aggTotal += v.total;
|
||||
}
|
||||
return {
|
||||
schema_version: 1,
|
||||
kind: 'by_type_summary',
|
||||
recall_by_type: recall,
|
||||
aggregate: {
|
||||
hit: aggHit,
|
||||
total: aggTotal,
|
||||
rate: aggTotal === 0 ? null : aggHit / aggTotal,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40.1.0 (Track D / T2, per Codex #7) — Emit the by_type_summary as the
|
||||
* final line of output. Resume-safe: if the output file already ends with
|
||||
* a `kind:"by_type_summary"` line (or has one anywhere), it is REMOVED
|
||||
* before the new summary is appended. Prevents duplicate summaries across
|
||||
* repeated `--resume-from` invocations.
|
||||
*
|
||||
* When `outputPath` is undefined (stdout mode), just writes the line —
|
||||
* resume-replace is impossible for stdout and not meaningful (resume always
|
||||
* uses a file).
|
||||
*/
|
||||
export function emitByTypeSummary(outputPath: string | undefined, summary: ByTypeSummary): void {
|
||||
const json = JSON.stringify(summary);
|
||||
if (json.includes('\r')) throw new Error('CRLF in by_type_summary emit (corrupt input)');
|
||||
if (!outputPath) {
|
||||
process.stdout.write(Buffer.from(json + '\n', 'utf8'));
|
||||
return;
|
||||
}
|
||||
// Read existing file (if present), strip any prior by_type_summary lines,
|
||||
// then append the new summary. Sync I/O is OK — output files for this
|
||||
// command are <1MB even on full 500-question runs.
|
||||
let existing = '';
|
||||
if (existsSync(outputPath)) {
|
||||
existing = readFileSync(outputPath, 'utf8');
|
||||
}
|
||||
const kept: string[] = [];
|
||||
for (const line of existing.split('\n')) {
|
||||
if (!line.trim()) continue;
|
||||
try {
|
||||
const row = JSON.parse(line);
|
||||
if (row && typeof row === 'object' && (row as any).kind === 'by_type_summary') {
|
||||
continue; // drop prior summary
|
||||
}
|
||||
} catch {
|
||||
// Corrupt line — keep as-is; the resume loader has its own skip logic.
|
||||
}
|
||||
kept.push(line);
|
||||
}
|
||||
kept.push(json);
|
||||
writeFileSync(outputPath, kept.join('\n') + '\n', 'utf8');
|
||||
}
|
||||
|
||||
122
src/core/audit-quality-probe.ts
Normal file
122
src/core/audit-quality-probe.ts
Normal file
@@ -0,0 +1,122 @@
|
||||
/**
|
||||
* v0.40.1.0 Track D / T6 — nightly quality probe audit trail.
|
||||
*
|
||||
* Writes one event per nightly cross-modal probe run to
|
||||
* `~/.gbrain/audit/quality-probe-YYYY-Www.jsonl` (ISO-week rotation).
|
||||
* Mirrors `audit-slug-fallback.ts` for filename + best-effort write
|
||||
* semantics. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir`.
|
||||
*
|
||||
* Read by `gbrain doctor`'s `nightly_quality_probe_health` check to
|
||||
* surface FAIL / ERROR / BUDGET_EXCEEDED runs from the last 7 days.
|
||||
*/
|
||||
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { resolveAuditDir } from './minions/handlers/shell-audit.ts';
|
||||
|
||||
export type QualityProbeOutcome =
|
||||
| 'pass'
|
||||
| 'fail'
|
||||
| 'inconclusive'
|
||||
| 'error'
|
||||
| 'budget_exceeded'
|
||||
| 'rate_limited'
|
||||
| 'no_embedding_key';
|
||||
|
||||
export interface QualityProbeAuditEvent {
|
||||
ts: string;
|
||||
/** Verdict from the cross-modal batch summary (or short-circuit reason). */
|
||||
outcome: QualityProbeOutcome;
|
||||
/** Exit code of the underlying batch (0/1/2/-1 when short-circuited). */
|
||||
exit_code: number;
|
||||
pass_count: number;
|
||||
fail_count: number;
|
||||
inconclusive_count: number;
|
||||
error_count: number;
|
||||
/** Estimated cost in USD (0 when short-circuited before runs). */
|
||||
est_cost_usd: number;
|
||||
/** Sha-8 of the fixture file content for change detection. */
|
||||
fixture_sha8?: string;
|
||||
/** Optional human-readable detail (e.g. error message, "no chat provider configured"). */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/** ISO-week-rotated filename: `quality-probe-YYYY-Www.jsonl`. Mirrors audit-slug-fallback. */
|
||||
export function computeQualityProbeAuditFilename(now: Date = new Date()): string {
|
||||
const d = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
|
||||
const dayNum = (d.getUTCDay() + 6) % 7;
|
||||
d.setUTCDate(d.getUTCDate() - dayNum + 3);
|
||||
const isoYear = d.getUTCFullYear();
|
||||
const firstThursday = new Date(Date.UTC(isoYear, 0, 4));
|
||||
const firstThursdayDayNum = (firstThursday.getUTCDay() + 6) % 7;
|
||||
firstThursday.setUTCDate(firstThursday.getUTCDate() - firstThursdayDayNum + 3);
|
||||
const weekNum = Math.round((d.getTime() - firstThursday.getTime()) / (7 * 86400000)) + 1;
|
||||
const ww = String(weekNum).padStart(2, '0');
|
||||
return `quality-probe-${isoYear}-W${ww}.jsonl`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Append one quality-probe event. Best-effort: write failure logs to stderr
|
||||
* but the probe phase continues.
|
||||
*/
|
||||
export function logQualityProbeEvent(event: Omit<QualityProbeAuditEvent, 'ts'> & { ts?: string }): void {
|
||||
const stamped: QualityProbeAuditEvent = {
|
||||
ts: event.ts ?? new Date().toISOString(),
|
||||
outcome: event.outcome,
|
||||
exit_code: event.exit_code,
|
||||
pass_count: event.pass_count,
|
||||
fail_count: event.fail_count,
|
||||
inconclusive_count: event.inconclusive_count,
|
||||
error_count: event.error_count,
|
||||
est_cost_usd: event.est_cost_usd,
|
||||
...(event.fixture_sha8 !== undefined ? { fixture_sha8: event.fixture_sha8 } : {}),
|
||||
...(event.detail !== undefined ? { detail: event.detail } : {}),
|
||||
};
|
||||
const dir = resolveAuditDir();
|
||||
const file = path.join(dir, computeQualityProbeAuditFilename());
|
||||
try {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.appendFileSync(file, JSON.stringify(stamped) + '\n', { encoding: 'utf8' });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[gbrain] quality-probe audit write failed (${msg}); probe continues\n`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read recent quality-probe events from the current + prior ISO week files.
|
||||
* Used by `gbrain doctor`'s nightly_quality_probe_health check. Missing
|
||||
* files and corrupt rows are skipped silently.
|
||||
*/
|
||||
export function readRecentQualityProbeEvents(
|
||||
days = 7,
|
||||
now: Date = new Date(),
|
||||
): QualityProbeAuditEvent[] {
|
||||
const dir = resolveAuditDir();
|
||||
const cutoff = now.getTime() - days * 86400000;
|
||||
const out: QualityProbeAuditEvent[] = [];
|
||||
const filenames = [
|
||||
computeQualityProbeAuditFilename(now),
|
||||
computeQualityProbeAuditFilename(new Date(now.getTime() - 7 * 86400000)),
|
||||
];
|
||||
for (const filename of filenames) {
|
||||
const file = path.join(dir, filename);
|
||||
let content: string;
|
||||
try {
|
||||
content = fs.readFileSync(file, 'utf8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const line of content.split('\n')) {
|
||||
if (line.length === 0) continue;
|
||||
try {
|
||||
const ev = JSON.parse(line) as QualityProbeAuditEvent;
|
||||
const ts = Date.parse(ev.ts);
|
||||
if (Number.isFinite(ts) && ts >= cutoff) out.push(ev);
|
||||
} catch {
|
||||
// corrupt row — skip
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
220
src/core/cycle/nightly-quality-probe.ts
Normal file
220
src/core/cycle/nightly-quality-probe.ts
Normal file
@@ -0,0 +1,220 @@
|
||||
/**
|
||||
* v0.40.1.0 Track D / T6 — Nightly cross-modal quality probe phase.
|
||||
*
|
||||
* Once per 24h, runs the canonical quality pipeline:
|
||||
* 1. `gbrain eval longmemeval --by-type` against the committed nightly
|
||||
* fixture (test/fixtures/longmemeval-nightly.jsonl) → JSONL output.
|
||||
* 2. `gbrain eval cross-modal --batch <jsonl> --max-usd $cap --yes`
|
||||
* → batch summary with verdict.
|
||||
* 3. Audit JSONL row recording outcome / cost / pass-fail counts.
|
||||
*
|
||||
* Default: DISABLED. Opt-in via `gbrain config set
|
||||
* autopilot.nightly_quality_probe.enabled true`. Doctor surfaces a
|
||||
* paste-ready enable hint when disabled.
|
||||
*
|
||||
* Embedding-key dependency: longmemeval needs `gateway.embedQuery()`.
|
||||
* Short-circuits with `outcome: no_embedding_key` + stderr warn when no
|
||||
* provider is configured (mirrors how the v0.31.12 model-routing infra
|
||||
* handles missing-provider cases).
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
|
||||
import { logQualityProbeEvent, readRecentQualityProbeEvents } from '../audit-quality-probe.ts';
|
||||
|
||||
/** Run-once gate window in ms. 24h matches the "nightly" cadence. */
|
||||
const NIGHTLY_WINDOW_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
/** Default max spend per run; matches eval-cross-modal --max-usd default. */
|
||||
const DEFAULT_MAX_USD = 5.0;
|
||||
|
||||
/** Committed fixture used as the probe's input dataset. */
|
||||
const NIGHTLY_FIXTURE_REL_PATH = 'test/fixtures/longmemeval-nightly.jsonl';
|
||||
|
||||
/** Result reported back to the cycle dispatcher / Minion handler. */
|
||||
export interface NightlyProbeResult {
|
||||
outcome: 'pass' | 'fail' | 'inconclusive' | 'error' | 'budget_exceeded' | 'rate_limited' | 'no_embedding_key' | 'disabled';
|
||||
exit_code: number;
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface NightlyProbeDeps {
|
||||
/** Returns true when the feature config flag is on. */
|
||||
isEnabled: () => boolean | Promise<boolean>;
|
||||
/** Returns true when an embedding provider is configured + reachable. */
|
||||
hasEmbeddingProvider: () => boolean | Promise<boolean>;
|
||||
/** Resolves the cost cap (config override OR DEFAULT_MAX_USD). */
|
||||
resolveMaxUsd: () => number | Promise<number>;
|
||||
/** Resolves the repo root so we can find the committed fixture. */
|
||||
resolveRepoRoot: () => string | Promise<string>;
|
||||
/** Runs the longmemeval command; returns the path to the JSONL output. */
|
||||
runLongMemEval: (args: { fixturePath: string; outputPath: string }) => Promise<void>;
|
||||
/** Runs the cross-modal batch; returns exit code (0/1/2). */
|
||||
runCrossModalBatch: (args: {
|
||||
batchPath: string;
|
||||
summaryPath: string;
|
||||
maxUsd: number;
|
||||
}) => Promise<{ exitCode: number; summary?: { pass_count: number; fail_count: number; inconclusive_count: number; error_count: number; est_cost_usd: number; verdict: string } }>;
|
||||
/** Now provider — overridable for tests of the 24h rate limit. */
|
||||
now: () => Date;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure function: decide whether the probe should run given the audit
|
||||
* history. Returns reason when skipping.
|
||||
*/
|
||||
export function shouldRunNightly(
|
||||
now: Date,
|
||||
recentEvents: ReadonlyArray<{ ts: string }>,
|
||||
windowMs: number = NIGHTLY_WINDOW_MS,
|
||||
): { run: true } | { run: false; reason: 'rate_limited' } {
|
||||
const cutoff = now.getTime() - windowMs;
|
||||
for (const ev of recentEvents) {
|
||||
const ts = Date.parse(ev.ts);
|
||||
if (Number.isFinite(ts) && ts >= cutoff) {
|
||||
return { run: false, reason: 'rate_limited' };
|
||||
}
|
||||
}
|
||||
return { run: true };
|
||||
}
|
||||
|
||||
function sha8File(p: string): string | undefined {
|
||||
try {
|
||||
const content = fs.readFileSync(p);
|
||||
return createHash('sha256').update(content).digest('hex').slice(0, 8);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the nightly probe. Pure DI surface — `deps` controls every external
|
||||
* effect so tests can stub long-running paths.
|
||||
*/
|
||||
export async function runNightlyQualityProbe(deps: NightlyProbeDeps): Promise<NightlyProbeResult> {
|
||||
const enabled = await deps.isEnabled();
|
||||
if (!enabled) {
|
||||
// Disabled-by-default; no audit row (doctor reads config separately).
|
||||
return { outcome: 'disabled', exit_code: 0, detail: 'feature flag off' };
|
||||
}
|
||||
|
||||
// 24h rate limit — skip + audit "rate_limited".
|
||||
const now = deps.now();
|
||||
const recent = readRecentQualityProbeEvents(2, now); // 2-day window is enough for 24h check
|
||||
const decision = shouldRunNightly(now, recent);
|
||||
if (!decision.run) {
|
||||
logQualityProbeEvent({
|
||||
outcome: 'rate_limited',
|
||||
exit_code: 0,
|
||||
pass_count: 0,
|
||||
fail_count: 0,
|
||||
inconclusive_count: 0,
|
||||
error_count: 0,
|
||||
est_cost_usd: 0,
|
||||
detail: 'already ran within 24h window',
|
||||
});
|
||||
return { outcome: 'rate_limited', exit_code: 0, detail: 'already ran within 24h' };
|
||||
}
|
||||
|
||||
// Embedding key check (longmemeval embeds queries).
|
||||
const hasEmbed = await deps.hasEmbeddingProvider();
|
||||
if (!hasEmbed) {
|
||||
process.stderr.write(
|
||||
`[nightly-quality-probe] no embedding provider configured; skipping. ` +
|
||||
`Configure OPENAI_API_KEY / VOYAGE_API_KEY / ZEROENTROPY_API_KEY and re-enable.\n`,
|
||||
);
|
||||
logQualityProbeEvent({
|
||||
outcome: 'no_embedding_key',
|
||||
exit_code: 0,
|
||||
pass_count: 0,
|
||||
fail_count: 0,
|
||||
inconclusive_count: 0,
|
||||
error_count: 0,
|
||||
est_cost_usd: 0,
|
||||
detail: 'no embedding provider configured',
|
||||
});
|
||||
return { outcome: 'no_embedding_key', exit_code: 0, detail: 'no embedding provider' };
|
||||
}
|
||||
|
||||
const repoRoot = await deps.resolveRepoRoot();
|
||||
const fixturePath = path.join(repoRoot, NIGHTLY_FIXTURE_REL_PATH);
|
||||
if (!fs.existsSync(fixturePath)) {
|
||||
const detail = `nightly fixture not found at ${fixturePath}`;
|
||||
process.stderr.write(`[nightly-quality-probe] ${detail}\n`);
|
||||
logQualityProbeEvent({
|
||||
outcome: 'error',
|
||||
exit_code: 1,
|
||||
pass_count: 0,
|
||||
fail_count: 0,
|
||||
inconclusive_count: 0,
|
||||
error_count: 0,
|
||||
est_cost_usd: 0,
|
||||
detail,
|
||||
});
|
||||
return { outcome: 'error', exit_code: 1, detail };
|
||||
}
|
||||
|
||||
const fixtureSha8 = sha8File(fixturePath);
|
||||
const maxUsd = (await deps.resolveMaxUsd()) ?? DEFAULT_MAX_USD;
|
||||
|
||||
// Tempdir for the per-question hypothesis JSONL + batch summary.
|
||||
const workDir = fs.mkdtempSync(path.join(tmpdir(), 'nightly-probe-'));
|
||||
const lmeOutPath = path.join(workDir, 'lme-output.jsonl');
|
||||
const summaryPath = path.join(workDir, 'summary.json');
|
||||
|
||||
try {
|
||||
await deps.runLongMemEval({ fixturePath, outputPath: lmeOutPath });
|
||||
const { exitCode, summary } = await deps.runCrossModalBatch({
|
||||
batchPath: lmeOutPath,
|
||||
summaryPath,
|
||||
maxUsd,
|
||||
});
|
||||
|
||||
const outcome: NightlyProbeResult['outcome'] = (() => {
|
||||
if (summary) {
|
||||
if (summary.verdict === 'pass') return 'pass';
|
||||
if (summary.verdict === 'fail') return 'fail';
|
||||
if (summary.verdict === 'inconclusive') return 'inconclusive';
|
||||
if (summary.verdict === 'error') return 'error';
|
||||
}
|
||||
// If exit code is 1 with no summary, the batch refused (budget).
|
||||
if (exitCode === 1) return 'budget_exceeded';
|
||||
return 'error';
|
||||
})();
|
||||
|
||||
logQualityProbeEvent({
|
||||
outcome,
|
||||
exit_code: exitCode,
|
||||
pass_count: summary?.pass_count ?? 0,
|
||||
fail_count: summary?.fail_count ?? 0,
|
||||
inconclusive_count: summary?.inconclusive_count ?? 0,
|
||||
error_count: summary?.error_count ?? 0,
|
||||
est_cost_usd: summary?.est_cost_usd ?? 0,
|
||||
fixture_sha8: fixtureSha8,
|
||||
});
|
||||
|
||||
return { outcome, exit_code: exitCode };
|
||||
} catch (err) {
|
||||
const detail = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[nightly-quality-probe] runtime error: ${detail}\n`);
|
||||
logQualityProbeEvent({
|
||||
outcome: 'error',
|
||||
exit_code: 1,
|
||||
pass_count: 0,
|
||||
fail_count: 0,
|
||||
inconclusive_count: 0,
|
||||
error_count: 0,
|
||||
est_cost_usd: 0,
|
||||
fixture_sha8: fixtureSha8,
|
||||
detail,
|
||||
});
|
||||
return { outcome: 'error', exit_code: 1, detail };
|
||||
} finally {
|
||||
try {
|
||||
fs.rmSync(workDir, { recursive: true, force: true });
|
||||
} catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
437
test/eval-cross-modal-batch.test.ts
Normal file
437
test/eval-cross-modal-batch.test.ts
Normal file
@@ -0,0 +1,437 @@
|
||||
/**
|
||||
* v0.40.1.0 Track D / T3+T4 — Cross-modal --batch mode tests.
|
||||
*
|
||||
* Hermetic via the runEval DI seam (per D5). Tests the batch loop +
|
||||
* semaphore + exit precedence + receipt suppression, NOT the underlying
|
||||
* runEval orchestrator (which has its own coverage in existing tests).
|
||||
*
|
||||
* No env mutation, no mock.module — regular *.test.ts per CLAUDE.md
|
||||
* test-isolation rules.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { runEvalCrossModal, runWithLimit, type BatchSummary } from '../src/commands/eval-cross-modal.ts';
|
||||
import type { RunEvalResult } from '../src/core/cross-modal-eval/runner.ts';
|
||||
import type { AggregateResult } from '../src/core/cross-modal-eval/aggregate.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function writeBatchFixture(rows: object[]): string {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'cm-batch-fixture-'));
|
||||
const path = join(tmp, 'batch.jsonl');
|
||||
writeFileSync(path, rows.map(r => JSON.stringify(r)).join('\n') + '\n', 'utf8');
|
||||
return path;
|
||||
}
|
||||
|
||||
function makeStubRunEval(verdicts: Array<'pass' | 'fail' | 'inconclusive' | 'throw'>) {
|
||||
let i = 0;
|
||||
return async function stubRunEval(_opts: any): Promise<RunEvalResult> {
|
||||
const idx = i++;
|
||||
const v = verdicts[idx % verdicts.length];
|
||||
if (v === 'throw') {
|
||||
throw new Error(`stub error for question ${idx}`);
|
||||
}
|
||||
const aggregate: AggregateResult = {
|
||||
verdict: v,
|
||||
verdictMessage: `stub: ${v}`,
|
||||
overall: v === 'pass' ? 8 : v === 'fail' ? 4 : 0,
|
||||
perDimension: {},
|
||||
successCount: 3,
|
||||
modelCount: 3,
|
||||
} as any;
|
||||
return {
|
||||
finalAggregate: aggregate,
|
||||
cycles: [],
|
||||
finalReceiptPath: '/tmp/fake-receipt.json',
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 1. runWithLimit semaphore primitive (T4)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('runWithLimit semaphore (v0.40.1.0 Track D / T4, per D6)', () => {
|
||||
test('never exceeds the in-flight limit', async () => {
|
||||
let inFlight = 0;
|
||||
let maxObserved = 0;
|
||||
const items = Array.from({ length: 20 }, (_, i) => i);
|
||||
await runWithLimit(items, 3, async (_item) => {
|
||||
inFlight++;
|
||||
if (inFlight > maxObserved) maxObserved = inFlight;
|
||||
await new Promise(r => setTimeout(r, 5));
|
||||
inFlight--;
|
||||
return 1;
|
||||
});
|
||||
expect(maxObserved).toBeLessThanOrEqual(3);
|
||||
expect(maxObserved).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('per-item errors do not abort the whole batch', async () => {
|
||||
const items = [0, 1, 2, 3, 4];
|
||||
const results = await runWithLimit(items, 2, async (item) => {
|
||||
if (item === 2) throw new Error(`fail-${item}`);
|
||||
return item * 10;
|
||||
});
|
||||
expect(results.length).toBe(5);
|
||||
expect(results[0]).toEqual({ ok: true, value: 0 });
|
||||
expect(results[1]).toEqual({ ok: true, value: 10 });
|
||||
expect(results[2].ok).toBe(false);
|
||||
expect(results[3]).toEqual({ ok: true, value: 30 });
|
||||
expect(results[4]).toEqual({ ok: true, value: 40 });
|
||||
});
|
||||
|
||||
test('preserves input order in results array', async () => {
|
||||
const items = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
|
||||
const results = await runWithLimit(items, 4, async (item) => {
|
||||
// Intentional non-uniform sleep to scramble completion order.
|
||||
await new Promise(r => setTimeout(r, (10 - item) * 2));
|
||||
return item * item;
|
||||
});
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
expect(results[i]).toEqual({ ok: true, value: i * i });
|
||||
}
|
||||
});
|
||||
|
||||
test('limit=1 = serial', async () => {
|
||||
const order: number[] = [];
|
||||
const items = [0, 1, 2, 3];
|
||||
await runWithLimit(items, 1, async (item) => {
|
||||
const start = Date.now();
|
||||
await new Promise(r => setTimeout(r, 5));
|
||||
order.push(item);
|
||||
return Date.now() - start;
|
||||
});
|
||||
expect(order).toEqual([0, 1, 2, 3]);
|
||||
});
|
||||
|
||||
test('limit > items.length still completes correctly', async () => {
|
||||
const results = await runWithLimit([0, 1, 2], 100, async (x) => x + 1);
|
||||
expect(results).toEqual([
|
||||
{ ok: true, value: 1 },
|
||||
{ ok: true, value: 2 },
|
||||
{ ok: true, value: 3 },
|
||||
]);
|
||||
});
|
||||
|
||||
test('limit < 1 throws (defensive)', async () => {
|
||||
await expect(runWithLimit([1, 2, 3], 0, async (x) => x)).rejects.toThrow(/limit must be >= 1/);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 2. End-to-end batch via stubbed runEval (T3 + D5 + D10)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('runEvalCrossModal --batch end-to-end (v0.40.1.0 Track D / T3, per D5+D10)', () => {
|
||||
test('all-pass batch → exit 0; summary receipt has expected shape', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'what is X?', hypothesis: 'X is foo.' },
|
||||
{ question_id: 'q2', question: 'what is Y?', hypothesis: 'Y is bar.' },
|
||||
{ question_id: 'q3', question: 'what is Z?', hypothesis: 'Z is baz.' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '3',
|
||||
'--cycles', '1', '--concurrent', '2', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'pass', 'pass']) },
|
||||
);
|
||||
expect(exit).toBe(0);
|
||||
expect(existsSync(summaryPath)).toBe(true);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
expect(summary.kind).toBe('cross_modal_batch_summary');
|
||||
expect(summary.schema_version).toBe(1);
|
||||
expect(summary.verdict).toBe('pass');
|
||||
expect(summary.pass_count).toBe(3);
|
||||
expect(summary.fail_count).toBe(0);
|
||||
expect(summary.error_count).toBe(0);
|
||||
expect(summary.inconclusive_count).toBe(0);
|
||||
expect(summary.per_question.length).toBe(3);
|
||||
expect(summary.per_question[0].question_id).toBe('q1');
|
||||
expect(summary.per_question[0].verdict).toBe('pass');
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('any FAIL → exit 1 (precedence: FAIL > INCONCLUSIVE > PASS)', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2', question: 'b', hypothesis: 'b-ans' },
|
||||
{ question_id: 'q3', question: 'c', hypothesis: 'c-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '3',
|
||||
'--cycles', '1', '--concurrent', '3', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'fail', 'pass']) },
|
||||
);
|
||||
expect(exit).toBe(1);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
expect(summary.verdict).toBe('fail');
|
||||
expect(summary.fail_count).toBe(1);
|
||||
expect(summary.pass_count).toBe(2);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('any per-question ERROR → exit 2 (precedence: ERROR > FAIL > INCONCLUSIVE > PASS)', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2', question: 'b', hypothesis: 'b-ans' },
|
||||
{ question_id: 'q3', question: 'c', hypothesis: 'c-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '3',
|
||||
'--cycles', '1', '--concurrent', '3', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'throw', 'fail']) },
|
||||
);
|
||||
// ERROR wins precedence over FAIL.
|
||||
expect(exit).toBe(2);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
expect(summary.verdict).toBe('error');
|
||||
expect(summary.error_count).toBe(1);
|
||||
expect(summary.fail_count).toBe(1);
|
||||
expect(summary.pass_count).toBe(1);
|
||||
const q2 = summary.per_question.find(p => p.question_id === 'q2')!;
|
||||
expect(q2.verdict).toBe('error');
|
||||
expect(q2.error).toContain('stub error');
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('any INCONCLUSIVE (no error, no fail) → exit 2', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2', question: 'b', hypothesis: 'b-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '2',
|
||||
'--cycles', '1', '--concurrent', '2', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'inconclusive']) },
|
||||
);
|
||||
expect(exit).toBe(2);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
expect(summary.verdict).toBe('inconclusive');
|
||||
expect(summary.inconclusive_count).toBe(1);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--batch + --task mutex → exit 1 with clear error', async () => {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', '/tmp/fake.jsonl', '--task', 'something', '--output', '/tmp/out.json'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).toBe(1);
|
||||
});
|
||||
|
||||
test('--batch with non-existent file → exit 1', async () => {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', '/tmp/this-does-not-exist-' + Date.now() + '.jsonl', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).toBe(1);
|
||||
});
|
||||
|
||||
test('--batch filters out by_type_summary rows (Codex #6)', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2', question: 'b', hypothesis: 'b-ans' },
|
||||
// The summary row should be filtered before batch processing.
|
||||
{ schema_version: 1, kind: 'by_type_summary', recall_by_type: {}, aggregate: { hit: 0, total: 0, rate: null } },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '10',
|
||||
'--cycles', '1', '--concurrent', '2', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'pass']) },
|
||||
);
|
||||
expect(exit).toBe(0);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
// 2 rows, NOT 3 — the summary row was filtered.
|
||||
expect(summary.total).toBe(2);
|
||||
expect(summary.pass_count).toBe(2);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--max-usd refusal without --yes', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
]);
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--limit', '1', '--cycles', '1', '--max-usd', '0.001'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).toBe(1);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--max-usd refusal bypassed by --yes', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath,
|
||||
'--limit', '1', '--cycles', '1', '--max-usd', '0.001', '--yes'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).toBe(0);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 3. Argument parser strictness (parseIntStrict / parseFloatStrict throw paths)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('argument parser strictness (v0.40.1.0 Track D / coverage gap)', () => {
|
||||
test('--limit rejects non-integer with usage error → exit 1', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
]);
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--limit', 'not-a-number'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
// parseIntStrict throws synchronously inside parseArgs → caller catches in CLI
|
||||
// dispatch and returns 1. Either path is acceptable; what matters is no PASS.
|
||||
expect(exit).not.toBe(0);
|
||||
} catch (err) {
|
||||
// If parseArgs throws synchronously, that also counts as "rejected" — we
|
||||
// assert the message rather than a specific exit code.
|
||||
expect(String(err)).toMatch(/positive integer/);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--max-usd rejects negative number', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
]);
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--max-usd', '-1'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).not.toBe(0);
|
||||
} catch (err) {
|
||||
expect(String(err)).toMatch(/non-negative number/);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 4. Codex CDX-1 + CDX-2 — upstream error rows + --limit 0 + malformed rows
|
||||
// must count in the denominator and fail-loud (no silent CI bypass).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('codex CDX-1 + CDX-2 — denominator-bypass defenses', () => {
|
||||
test('upstream-error rows (longmemeval emitted {error:...}) count as upstream_error, NOT silently dropped', async () => {
|
||||
// Mimics what eval-longmemeval emits when runOneQuestion throws:
|
||||
// {question_id, question, question_type, hypothesis: '', error: '...'}
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2', question: 'b', question_type: 'temporal-reasoning', hypothesis: '', error: 'upstream OOM' },
|
||||
{ question_id: 'q3', question: 'c', hypothesis: 'c-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '10',
|
||||
'--cycles', '1', '--concurrent', '2', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'pass']) }, // only 2 calls (q1, q3) — q2 is upstream error
|
||||
);
|
||||
// Exit 2 because upstream_error_count > 0 (CDX-1).
|
||||
expect(exit).toBe(2);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
// Denominator is 3 (not 2!) — upstream error counted.
|
||||
expect(summary.total).toBe(3);
|
||||
expect(summary.upstream_error_count).toBe(1);
|
||||
expect(summary.error_count).toBe(0);
|
||||
expect(summary.pass_count).toBe(2);
|
||||
expect(summary.verdict).toBe('error');
|
||||
// q2 surfaces in per_question with verdict 'upstream_error'.
|
||||
const q2 = summary.per_question.find(p => p.question_id === 'q2');
|
||||
expect(q2).toBeDefined();
|
||||
expect(q2!.verdict).toBe('upstream_error');
|
||||
expect(q2!.error).toBe('upstream OOM');
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('malformed rows (missing question or hypothesis) count toward malformed_count + exit 2', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
{ question_id: 'q2' /* missing question and hypothesis */ },
|
||||
{ question_id: 'q3', question: 'c', hypothesis: 'c-ans' },
|
||||
]);
|
||||
const summaryPath = join(mkdtempSync(join(tmpdir(), 'cm-summary-')), 'summary.json');
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--output', summaryPath, '--limit', '10',
|
||||
'--cycles', '1', '--concurrent', '2', '--max-usd', '1000'],
|
||||
{ runEval: makeStubRunEval(['pass', 'pass']) },
|
||||
);
|
||||
expect(exit).toBe(2);
|
||||
const summary = JSON.parse(readFileSync(summaryPath, 'utf8')) as BatchSummary;
|
||||
expect(summary.malformed_count).toBe(1);
|
||||
expect(summary.total).toBe(3); // 2 scored + 0 upstream + 1 malformed
|
||||
expect(summary.verdict).toBe('error');
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
rmSync(summaryPath, { force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('--limit 0 is rejected (would bypass the gate with empty result → PASS)', async () => {
|
||||
const fixturePath = writeBatchFixture([
|
||||
{ question_id: 'q1', question: 'a', hypothesis: 'a-ans' },
|
||||
]);
|
||||
try {
|
||||
const exit = await runEvalCrossModal(
|
||||
['--batch', fixturePath, '--limit', '0'],
|
||||
{ runEval: makeStubRunEval(['pass']) },
|
||||
);
|
||||
expect(exit).toBe(1);
|
||||
} finally {
|
||||
rmSync(fixturePath, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -666,3 +666,216 @@ describe('runEvalLongMemEval --resume-from (v0.35.1.0)', () => {
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 12. v0.40.1.0 (Track D / T1 + T2): question field on every row + --by-type
|
||||
// summary emission with resume-replace semantics + --by-type-floor exit gate
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('runEvalLongMemEval --by-type (v0.40.1.0 Track D / T1+T2)', () => {
|
||||
test('per-row JSONL includes the question text (T1, per D9)', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-test-'));
|
||||
const outPath = join(tmp, 'hypothesis.jsonl');
|
||||
try {
|
||||
const { client } = makeStubClient('canned');
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '3', '--output', outPath],
|
||||
{ client },
|
||||
);
|
||||
const lines = readFileSync(outPath, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
expect(lines.length).toBe(3);
|
||||
for (const line of lines) {
|
||||
const row = JSON.parse(line);
|
||||
expect(typeof row.question).toBe('string');
|
||||
expect(row.question.length).toBeGreaterThan(0);
|
||||
expect(typeof row.question_id).toBe('string');
|
||||
}
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('--by-type emits a final by_type_summary line; absent when flag not set', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-test-'));
|
||||
const withFlag = join(tmp, 'with-by-type.jsonl');
|
||||
const withoutFlag = join(tmp, 'without-by-type.jsonl');
|
||||
try {
|
||||
const { client } = makeStubClient('canned');
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '3', '--output', withFlag, '--by-type'],
|
||||
{ client },
|
||||
);
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '3', '--output', withoutFlag],
|
||||
{ client },
|
||||
);
|
||||
|
||||
// With flag: last line is the summary.
|
||||
const withLines = readFileSync(withFlag, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
const lastWith = JSON.parse(withLines[withLines.length - 1]);
|
||||
expect(lastWith.kind).toBe('by_type_summary');
|
||||
expect(lastWith.schema_version).toBe(1);
|
||||
expect(typeof lastWith.recall_by_type).toBe('object');
|
||||
expect(typeof lastWith.aggregate.hit).toBe('number');
|
||||
expect(typeof lastWith.aggregate.total).toBe('number');
|
||||
// Per-question rows must NOT have kind:by_type_summary.
|
||||
for (let i = 0; i < withLines.length - 1; i++) {
|
||||
const row = JSON.parse(withLines[i]);
|
||||
expect(row.kind).toBeUndefined();
|
||||
}
|
||||
|
||||
// Without flag: no summary anywhere.
|
||||
const withoutLines = readFileSync(withoutFlag, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
for (const line of withoutLines) {
|
||||
const row = JSON.parse(line);
|
||||
expect(row.kind).toBeUndefined();
|
||||
}
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('resume-replace: prior by_type_summary at the tail is REPLACED, not appended', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-test-'));
|
||||
const outPath = join(tmp, 'resume.jsonl');
|
||||
try {
|
||||
const { client } = makeStubClient('canned');
|
||||
// First run: --limit 3 produces 3 rows + 1 summary = 4 lines.
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '3', '--output', outPath, '--by-type'],
|
||||
{ client },
|
||||
);
|
||||
const firstLines = readFileSync(outPath, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
const firstSummaryCount = firstLines.filter(l => {
|
||||
try { return JSON.parse(l).kind === 'by_type_summary'; } catch { return false; }
|
||||
}).length;
|
||||
expect(firstSummaryCount).toBe(1);
|
||||
expect(firstLines.length).toBe(4);
|
||||
|
||||
// Re-run with --limit 5 + --resume-from same path: 2 NEW questions get
|
||||
// processed, by-type fires again, prior summary must be replaced (not
|
||||
// duplicated). Exercises the full resume-replace code path.
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '5', '--output', outPath,
|
||||
'--resume-from', outPath, '--by-type'],
|
||||
{ client },
|
||||
);
|
||||
const secondLines = readFileSync(outPath, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
const secondSummaryCount = secondLines.filter(l => {
|
||||
try { return JSON.parse(l).kind === 'by_type_summary'; } catch { return false; }
|
||||
}).length;
|
||||
expect(secondSummaryCount).toBe(1);
|
||||
// 5 rows + 1 summary = 6 lines (original summary was stripped, new one
|
||||
// appended).
|
||||
expect(secondLines.length).toBe(6);
|
||||
const last = JSON.parse(secondLines[secondLines.length - 1]);
|
||||
expect(last.kind).toBe('by_type_summary');
|
||||
// Summary aggregates across ALL 5 rows (not just the 2 newly processed).
|
||||
// The fixture has ground truth on every row, so total == 5.
|
||||
expect(last.aggregate.total).toBe(5);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('buildByTypeSummary (pure function)', () => {
|
||||
test('populated buckets produce sorted keys + rate math', async () => {
|
||||
const { buildByTypeSummary } = await import('../src/commands/eval-longmemeval.ts');
|
||||
const summary = buildByTypeSummary({
|
||||
'multi-session': { hit: 10, total: 10 },
|
||||
'single-session-user': { hit: 18, total: 19 },
|
||||
});
|
||||
expect(summary.kind).toBe('by_type_summary');
|
||||
expect(summary.schema_version).toBe(1);
|
||||
// Sorted alphabetically.
|
||||
expect(Object.keys(summary.recall_by_type)).toEqual(['multi-session', 'single-session-user']);
|
||||
expect(summary.recall_by_type['multi-session'].rate).toBeCloseTo(1.0, 5);
|
||||
expect(summary.recall_by_type['single-session-user'].rate).toBeCloseTo(18 / 19, 5);
|
||||
expect(summary.aggregate.hit).toBe(28);
|
||||
expect(summary.aggregate.total).toBe(29);
|
||||
expect(summary.aggregate.rate).toBeCloseTo(28 / 29, 5);
|
||||
});
|
||||
|
||||
test('empty bucket map produces rate:null aggregate, not NaN', async () => {
|
||||
const { buildByTypeSummary } = await import('../src/commands/eval-longmemeval.ts');
|
||||
const summary = buildByTypeSummary({});
|
||||
expect(summary.recall_by_type).toEqual({});
|
||||
expect(summary.aggregate.hit).toBe(0);
|
||||
expect(summary.aggregate.total).toBe(0);
|
||||
expect(summary.aggregate.rate).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 13. Codex CDX-3 — resume + --by-type-floor must enforce the floor even on
|
||||
// a no-op resume (where all questions already done). Pre-CDX-3 the early
|
||||
// return bypassed the floor gate entirely.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('codex CDX-3 — resume + --by-type-floor enforcement on no-op resume', () => {
|
||||
test('all-done resume still runs --by-type emission AND --by-type-floor gate', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const outPath = join(tmp, 'all-done.jsonl');
|
||||
try {
|
||||
// Pre-seed the output file with all-failed rows (recall_hit: false).
|
||||
// This represents a prior run that completed every question but with
|
||||
// very poor recall — the floor gate should fire even though no
|
||||
// questions are processed THIS run.
|
||||
const fixture = readFileSync(FIXTURE_PATH, 'utf8')
|
||||
.split('\n').filter(l => l.length > 0).map(l => JSON.parse(l)).slice(0, 5);
|
||||
const { writeFileSync } = await import('fs');
|
||||
writeFileSync(
|
||||
outPath,
|
||||
fixture.map(q => JSON.stringify({
|
||||
question_id: q.question_id,
|
||||
question: q.question,
|
||||
question_type: q.question_type,
|
||||
hypothesis: 'done',
|
||||
recall_hit: false, // every prior question missed
|
||||
})).join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
|
||||
const { client } = makeStubClient('should-not-be-called');
|
||||
// Wrap to catch process.exit thrown from inside.
|
||||
const exitCapture: { code: number | null } = { code: null };
|
||||
const originalExit = process.exit;
|
||||
// @ts-ignore — runtime override for test
|
||||
process.exit = ((code: number) => {
|
||||
exitCapture.code = code;
|
||||
throw new Error('__exit__');
|
||||
}) as any;
|
||||
try {
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '5',
|
||||
'--output', outPath, '--resume-from', outPath,
|
||||
'--by-type', '--by-type-floor', '0.5'],
|
||||
{ client },
|
||||
);
|
||||
} catch (e) {
|
||||
// Expected: --by-type-floor breach → exit(1) → our test throw
|
||||
if (!String(e).includes('__exit__')) throw e;
|
||||
} finally {
|
||||
// @ts-ignore — runtime restore
|
||||
process.exit = originalExit;
|
||||
}
|
||||
|
||||
// CDX-3: floor gate fired despite no-op resume → exit code 1.
|
||||
expect(exitCapture.code).toBe(1);
|
||||
|
||||
// AND a by_type_summary was emitted at the file tail (CDX-3 also says
|
||||
// resume must run summary emission even on no-op).
|
||||
const lines = readFileSync(outPath, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
const summaries = lines.filter(l => {
|
||||
try { return JSON.parse(l).kind === 'by_type_summary'; } catch { return false; }
|
||||
});
|
||||
expect(summaries.length).toBe(1);
|
||||
const summary = JSON.parse(summaries[0]);
|
||||
// All rows had recall_hit: false → aggregate.rate is 0 → below 0.5 floor.
|
||||
expect(summary.aggregate.rate).toBeLessThan(0.5);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
256
test/eval-replay-gate.test.ts
Normal file
256
test/eval-replay-gate.test.ts
Normal file
@@ -0,0 +1,256 @@
|
||||
/**
|
||||
* v0.40.1.0 Track D / T5 — Hermetic retrieval qrels gate.
|
||||
*
|
||||
* Gates PRs touching src/core/search/** against a hand-curated qrels fixture
|
||||
* (test/fixtures/eval-baselines/qrels-search.json). Fully hermetic via basis-
|
||||
* vector embeddings — no API keys, no DATABASE_URL, no network.
|
||||
*
|
||||
* This is the structural-fix replacement for the original Task 2 design
|
||||
* (eval replay against captured eval_candidates), which Codex caught as
|
||||
* non-functional in CI (eval-export bypasses op-layer capture; replay
|
||||
* re-embeds queries via gateway which needs an API key). The qrels approach
|
||||
* tests retrieval QUALITY directly, not stability.
|
||||
*
|
||||
* Refresh procedure: when ranking changes are intentional, edit qrels-search.json
|
||||
* with a `Why:` line in the commit body (D4 convention). Do NOT silently
|
||||
* rubber-stamp baseline drift.
|
||||
*
|
||||
* Env overrides (via withEnv() per CLAUDE.md R1):
|
||||
* GBRAIN_REPLAY_GATE_TOP1_FLOOR (default 0.80)
|
||||
* GBRAIN_REPLAY_GATE_RECALL_FLOOR (default 0.85)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Canonical PGLite block (CLAUDE.md R3+R4)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fixture loader + helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface QrelQuery {
|
||||
query_id: string;
|
||||
query: string;
|
||||
/** Basis-vector dimension this query embeds at — same dim as first_relevant_slug. */
|
||||
embedding_dim: number;
|
||||
relevant_slugs: string[];
|
||||
first_relevant_slug: string;
|
||||
}
|
||||
|
||||
interface QrelFixture {
|
||||
schema_version: 1;
|
||||
queries: QrelQuery[];
|
||||
}
|
||||
|
||||
function loadFixture(): QrelFixture {
|
||||
const path = join(import.meta.dir, 'fixtures', 'eval-baselines', 'qrels-search.json');
|
||||
const fix = JSON.parse(readFileSync(path, 'utf8')) as QrelFixture;
|
||||
return fix;
|
||||
}
|
||||
|
||||
/** Basis vector with 1.0 at `idx` and 0.0 elsewhere. Mirrors search-quality.test.ts. */
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
/**
|
||||
* Seed each relevant slug with a chunk whose embedding aligns with the
|
||||
* query's basis dimension. The FIRST relevant slug gets a stronger signal
|
||||
* (compiled_truth) so it ranks top-1; subsequent relevant slugs get timeline
|
||||
* chunks at the same direction so they appear in top-K recall but not top-1.
|
||||
*
|
||||
* Non-relevant slugs from OTHER queries serve as noise — they're at
|
||||
* different basis dimensions so they orthogonally don't match.
|
||||
*/
|
||||
async function seedCorpus(fix: QrelFixture): Promise<void> {
|
||||
const seenSlugs = new Set<string>();
|
||||
for (const q of fix.queries) {
|
||||
for (let i = 0; i < q.relevant_slugs.length; i++) {
|
||||
const slug = q.relevant_slugs[i];
|
||||
if (seenSlugs.has(slug)) continue;
|
||||
seenSlugs.add(slug);
|
||||
// First relevant: top-1 should win, so embed at THIS query's dim with
|
||||
// chunk_source=compiled_truth (gets the source boost). Subsequent
|
||||
// relevant slugs use timeline source so they outrank pure noise but
|
||||
// don't beat the first relevant.
|
||||
const isFirst = slug === q.first_relevant_slug;
|
||||
const isPersonOrCompany = slug.startsWith('people/') || slug.startsWith('companies/');
|
||||
await engine.putPage(slug, {
|
||||
type: isPersonOrCompany ? (slug.startsWith('people/') ? 'person' : 'company') : 'note',
|
||||
title: slug.split('/').pop() ?? slug,
|
||||
compiled_truth: isFirst ? `Primary content about ${q.query}` : '',
|
||||
timeline: !isFirst ? `Mentioned in context of ${q.query}` : '',
|
||||
});
|
||||
const chunk: ChunkInput = {
|
||||
chunk_index: 0,
|
||||
chunk_text: isFirst
|
||||
? `Primary content about ${q.query}`
|
||||
: `Mentioned in context of ${q.query}`,
|
||||
chunk_source: isFirst ? 'compiled_truth' : 'timeline',
|
||||
embedding: basisEmbedding(q.embedding_dim),
|
||||
token_count: 10,
|
||||
};
|
||||
await engine.upsertChunks(slug, [chunk]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run all qrels queries against the seeded engine, compute top-1 match rate
|
||||
* and recall@10. Pure data: returns the two metrics so the gate logic stays
|
||||
* separate from the measurement logic.
|
||||
*/
|
||||
async function measureGate(
|
||||
fix: QrelFixture,
|
||||
): Promise<{ top1Rate: number; recallAt10: number; perQuery: Array<{ id: string; top1Match: boolean; recall: number }> }> {
|
||||
let top1Hits = 0;
|
||||
let recallSum = 0;
|
||||
const perQuery: Array<{ id: string; top1Match: boolean; recall: number }> = [];
|
||||
for (const q of fix.queries) {
|
||||
const results = await engine.searchVector(basisEmbedding(q.embedding_dim), { limit: 10 });
|
||||
const top1Slug = results[0]?.slug;
|
||||
const top1Match = top1Slug === q.first_relevant_slug;
|
||||
if (top1Match) top1Hits++;
|
||||
const retrievedSlugs = new Set(results.map(r => r.slug));
|
||||
const relevantInTop10 = q.relevant_slugs.filter(s => retrievedSlugs.has(s)).length;
|
||||
const recall = q.relevant_slugs.length === 0 ? 0 : relevantInTop10 / q.relevant_slugs.length;
|
||||
recallSum += recall;
|
||||
perQuery.push({ id: q.query_id, top1Match, recall });
|
||||
}
|
||||
return {
|
||||
top1Rate: top1Hits / fix.queries.length,
|
||||
recallAt10: recallSum / fix.queries.length,
|
||||
perQuery,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Constants (env-overridable)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const DEFAULT_TOP1_FLOOR = 0.80;
|
||||
const DEFAULT_RECALL_FLOOR = 0.85;
|
||||
|
||||
function resolveFloors(): { top1: number; recall: number } {
|
||||
const t = process.env.GBRAIN_REPLAY_GATE_TOP1_FLOOR;
|
||||
const r = process.env.GBRAIN_REPLAY_GATE_RECALL_FLOOR;
|
||||
return {
|
||||
top1: t !== undefined ? Number(t) : DEFAULT_TOP1_FLOOR,
|
||||
recall: r !== undefined ? Number(r) : DEFAULT_RECALL_FLOOR,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('eval replay gate — hermetic retrieval qrels (v0.40.1.0 Track D / T5)', () => {
|
||||
test('current ranking meets default floors (top1 >= 0.80 AND recall@10 >= 0.85)', async () => {
|
||||
const fix = loadFixture();
|
||||
await seedCorpus(fix);
|
||||
const { top1Rate, recallAt10, perQuery } = await measureGate(fix);
|
||||
const { top1, recall } = resolveFloors();
|
||||
|
||||
// Surface per-query results when the gate trips so the operator sees which
|
||||
// queries regressed without re-running with --verbose.
|
||||
if (top1Rate < top1 || recallAt10 < recall) {
|
||||
process.stderr.write(`[eval replay gate] BREACH:\n`);
|
||||
process.stderr.write(` top1=${top1Rate.toFixed(3)} (floor ${top1.toFixed(3)})\n`);
|
||||
process.stderr.write(` recall@10=${recallAt10.toFixed(3)} (floor ${recall.toFixed(3)})\n`);
|
||||
for (const q of perQuery) {
|
||||
process.stderr.write(` ${q.id}: top1=${q.top1Match ? 'HIT' : 'miss'} recall=${q.recall.toFixed(2)}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
expect(top1Rate).toBeGreaterThanOrEqual(top1);
|
||||
expect(recallAt10).toBeGreaterThanOrEqual(recall);
|
||||
});
|
||||
|
||||
test('env-overridable floors via GBRAIN_REPLAY_GATE_TOP1_FLOOR / GBRAIN_REPLAY_GATE_RECALL_FLOOR', async () => {
|
||||
const fix = loadFixture();
|
||||
await seedCorpus(fix);
|
||||
|
||||
// Set an impossible-to-meet floor and verify the resolver picks it up.
|
||||
// We don't ASSERT failure here (that would make the test flaky against
|
||||
// any future ranking improvement); we just assert the resolver respects
|
||||
// env vars.
|
||||
await withEnv({ GBRAIN_REPLAY_GATE_TOP1_FLOOR: '0.999', GBRAIN_REPLAY_GATE_RECALL_FLOOR: '0.999' }, async () => {
|
||||
const { top1, recall } = resolveFloors();
|
||||
expect(top1).toBeCloseTo(0.999, 3);
|
||||
expect(recall).toBeCloseTo(0.999, 3);
|
||||
});
|
||||
|
||||
// After exit, defaults restored.
|
||||
const { top1, recall } = resolveFloors();
|
||||
expect(top1).toBeCloseTo(DEFAULT_TOP1_FLOOR, 3);
|
||||
expect(recall).toBeCloseTo(DEFAULT_RECALL_FLOOR, 3);
|
||||
});
|
||||
|
||||
test('seeded corpus produces deterministic top-1 results (gate sanity)', async () => {
|
||||
const fix = loadFixture();
|
||||
await seedCorpus(fix);
|
||||
// Run twice; same fixture must produce same per-query top-1.
|
||||
const a = await measureGate(fix);
|
||||
const b = await measureGate(fix);
|
||||
expect(a.top1Rate).toBe(b.top1Rate);
|
||||
expect(a.recallAt10).toBe(b.recallAt10);
|
||||
for (let i = 0; i < a.perQuery.length; i++) {
|
||||
expect(a.perQuery[i]).toEqual(b.perQuery[i]);
|
||||
}
|
||||
});
|
||||
|
||||
test('fixture schema sanity — every query has required fields', () => {
|
||||
const fix = loadFixture();
|
||||
expect(fix.schema_version).toBe(1);
|
||||
expect(fix.queries.length).toBeGreaterThanOrEqual(10);
|
||||
for (const q of fix.queries) {
|
||||
expect(typeof q.query_id).toBe('string');
|
||||
expect(typeof q.query).toBe('string');
|
||||
expect(typeof q.embedding_dim).toBe('number');
|
||||
expect(Array.isArray(q.relevant_slugs)).toBe(true);
|
||||
expect(q.relevant_slugs.length).toBeGreaterThan(0);
|
||||
expect(q.relevant_slugs).toContain(q.first_relevant_slug);
|
||||
}
|
||||
});
|
||||
|
||||
test('no real names in qrels fixture (CLAUDE.md privacy rule)', () => {
|
||||
// Smoke-grep: scan the fixture for known-real-name patterns. This is a
|
||||
// belt-and-suspenders check beyond scripts/check-privacy.sh.
|
||||
const raw = readFileSync(
|
||||
join(import.meta.dir, 'fixtures', 'eval-baselines', 'qrels-search.json'),
|
||||
'utf8',
|
||||
);
|
||||
// Block list — names that appeared in older E2E fixtures and were
|
||||
// explicitly called out by Codex (#9) as privacy-violating.
|
||||
const blockList = ['Pedro Franceschi', 'Brex', 'Wintermute', 'Garry Tan', 'Y Combinator', 'YC'];
|
||||
for (const name of blockList) {
|
||||
expect(raw).not.toContain(name);
|
||||
}
|
||||
});
|
||||
});
|
||||
90
test/fixtures/eval-baselines/qrels-search.json
vendored
Normal file
90
test/fixtures/eval-baselines/qrels-search.json
vendored
Normal file
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"_description": "v0.40.1.0 Track D — hermetic retrieval qrels gate. PLACEHOLDER names only (no real people / companies — per CLAUDE.md privacy rule). Each query maps to a ranked list of relevant slugs; `first_relevant_slug` is the expected top-1. The gate uses basis-vector embeddings (each slug embedded at a deterministic dimension) so retrieval is reproducible and hermetic. Refresh requires a `Why:` line in the commit body (D4). Note: first_relevant_slug is chosen so source-aware ranking (people/companies/concepts/writing/originals get prefix boosts) naturally surfaces it top-1.",
|
||||
"queries": [
|
||||
{
|
||||
"query_id": "q1-fintech-founder",
|
||||
"query": "fintech founder building payments infrastructure",
|
||||
"embedding_dim": 0,
|
||||
"relevant_slugs": ["people/alice-example", "companies/widget-co-example"],
|
||||
"first_relevant_slug": "people/alice-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q2-meeting-recall",
|
||||
"query": "meeting notes from last quarter strategy session",
|
||||
"embedding_dim": 1,
|
||||
"relevant_slugs": ["people/charlie-example", "meetings/2026-q1-strategy-example"],
|
||||
"first_relevant_slug": "people/charlie-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q3-ai-research",
|
||||
"query": "AI safety research alignment work",
|
||||
"embedding_dim": 2,
|
||||
"relevant_slugs": ["people/bob-example", "topics/ai-safety-example"],
|
||||
"first_relevant_slug": "people/bob-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q4-portfolio-deal",
|
||||
"query": "Series A deal in healthcare vertical",
|
||||
"embedding_dim": 3,
|
||||
"relevant_slugs": ["companies/acme-example", "deals/acme-series-a-example"],
|
||||
"first_relevant_slug": "companies/acme-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q5-engineering-architecture",
|
||||
"query": "distributed systems architecture postgres replication",
|
||||
"embedding_dim": 4,
|
||||
"relevant_slugs": ["concepts/distributed-systems-example", "topics/architecture-example"],
|
||||
"first_relevant_slug": "concepts/distributed-systems-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q6-personal-task",
|
||||
"query": "weekly task list goals review",
|
||||
"embedding_dim": 5,
|
||||
"relevant_slugs": ["writing/weekly-tasks-example", "personal/goals-example"],
|
||||
"first_relevant_slug": "writing/weekly-tasks-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q7-fund-investor",
|
||||
"query": "fund investor partner introduction",
|
||||
"embedding_dim": 6,
|
||||
"relevant_slugs": ["people/dana-example", "companies/fund-a-example"],
|
||||
"first_relevant_slug": "people/dana-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q8-product-design",
|
||||
"query": "design system component library",
|
||||
"embedding_dim": 7,
|
||||
"relevant_slugs": ["writing/design-system-essay-example", "projects/component-library-example"],
|
||||
"first_relevant_slug": "writing/design-system-essay-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q9-accelerator-batch",
|
||||
"query": "accelerator batch founder demo day",
|
||||
"embedding_dim": 8,
|
||||
"relevant_slugs": ["people/elliot-example", "events/demo-day-example"],
|
||||
"first_relevant_slug": "people/elliot-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q10-tech-stack",
|
||||
"query": "TypeScript bun runtime build pipeline",
|
||||
"embedding_dim": 9,
|
||||
"relevant_slugs": ["concepts/typescript-example", "tech/bun-runtime-example"],
|
||||
"first_relevant_slug": "concepts/typescript-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q11-research-paper",
|
||||
"query": "retrieval augmented generation paper notes",
|
||||
"embedding_dim": 10,
|
||||
"relevant_slugs": ["concepts/rag-example", "writing/retrieval-overview-example"],
|
||||
"first_relevant_slug": "concepts/rag-example"
|
||||
},
|
||||
{
|
||||
"query_id": "q12-founder-mode",
|
||||
"query": "founder mode product feedback loop",
|
||||
"embedding_dim": 11,
|
||||
"relevant_slugs": ["originals/founder-mode-example", "concepts/founder-mode-example"],
|
||||
"first_relevant_slug": "originals/founder-mode-example"
|
||||
}
|
||||
]
|
||||
}
|
||||
10
test/fixtures/longmemeval-nightly.jsonl
vendored
Normal file
10
test/fixtures/longmemeval-nightly.jsonl
vendored
Normal file
@@ -0,0 +1,10 @@
|
||||
{"question_id":"nightly-1","question_type":"single-session-user","question":"Where did alice-example say she lived last year?","answer":"in widget-co","haystack_sessions":[[{"role":"user","content":"alice-example mentioned she lived in widget-co last year"},{"role":"assistant","content":"Got it, noted alice-example lived in widget-co"}]],"haystack_session_ids":["nightly-1-s1"],"haystack_dates":["2025-08-15"],"answer_session_ids":["nightly-1-s1"]}
|
||||
{"question_id":"nightly-2","question_type":"multi-session","question":"What project was charlie-example working on last month?","answer":"the component-library-example","haystack_sessions":[[{"role":"user","content":"charlie-example started a new project called component-library-example"}],[{"role":"user","content":"charlie-example shipped the component-library-example v1"}]],"haystack_session_ids":["nightly-2-s1","nightly-2-s2"],"haystack_dates":["2025-07-01","2025-07-15"],"answer_session_ids":["nightly-2-s1","nightly-2-s2"]}
|
||||
{"question_id":"nightly-3","question_type":"temporal-reasoning","question":"When did bob-example join fund-a-example?","answer":"2024-06-01","haystack_sessions":[[{"role":"user","content":"bob-example joined fund-a-example on 2024-06-01 as a partner"}]],"haystack_session_ids":["nightly-3-s1"],"haystack_dates":["2024-06-01"],"answer_session_ids":["nightly-3-s1"]}
|
||||
{"question_id":"nightly-4","question_type":"knowledge-update","question":"What is dana-example's current role?","answer":"CTO at acme-example","haystack_sessions":[[{"role":"user","content":"dana-example was previously CEO at widget-co"}],[{"role":"user","content":"dana-example took the CTO role at acme-example last week"}]],"haystack_session_ids":["nightly-4-s1","nightly-4-s2"],"haystack_dates":["2024-01-15","2025-03-20"],"answer_session_ids":["nightly-4-s2"]}
|
||||
{"question_id":"nightly-5","question_type":"single-session-assistant","question":"What city was the demo-day-example held in?","answer":"placeholder-city","haystack_sessions":[[{"role":"user","content":"I attended demo-day-example yesterday"},{"role":"assistant","content":"You mentioned demo-day-example was held in placeholder-city this year"}]],"haystack_session_ids":["nightly-5-s1"],"haystack_dates":["2025-05-01"],"answer_session_ids":["nightly-5-s1"]}
|
||||
{"question_id":"nightly-6","question_type":"single-session-user","question":"What language is elliot-example most fluent in?","answer":"TypeScript","haystack_sessions":[[{"role":"user","content":"elliot-example is most fluent in TypeScript; uses it for the bun-runtime-example"}]],"haystack_session_ids":["nightly-6-s1"],"haystack_dates":["2025-04-10"],"answer_session_ids":["nightly-6-s1"]}
|
||||
{"question_id":"nightly-7","question_type":"multi-session","question":"How many projects has alice-example launched this year?","answer":"three","haystack_sessions":[[{"role":"user","content":"alice-example launched widget-co/payments in January"}],[{"role":"user","content":"alice-example launched widget-co/identity in March"}],[{"role":"user","content":"alice-example launched widget-co/reporting in June"}]],"haystack_session_ids":["nightly-7-s1","nightly-7-s2","nightly-7-s3"],"haystack_dates":["2025-01-15","2025-03-20","2025-06-12"],"answer_session_ids":["nightly-7-s1","nightly-7-s2","nightly-7-s3"]}
|
||||
{"question_id":"nightly-8","question_type":"temporal-reasoning","question":"Was the rag-example paper read before or after the architecture-example notes?","answer":"before","haystack_sessions":[[{"role":"user","content":"Read the rag-example paper today"}],[{"role":"user","content":"Wrote architecture-example notes a week later"}]],"haystack_session_ids":["nightly-8-s1","nightly-8-s2"],"haystack_dates":["2025-02-01","2025-02-08"],"answer_session_ids":["nightly-8-s1","nightly-8-s2"]}
|
||||
{"question_id":"nightly-9","question_type":"knowledge-update","question":"What is charlie-example's preferred testing framework now?","answer":"bun:test","haystack_sessions":[[{"role":"user","content":"charlie-example used vitest for years"}],[{"role":"user","content":"charlie-example switched to bun:test as primary testing framework"}]],"haystack_session_ids":["nightly-9-s1","nightly-9-s2"],"haystack_dates":["2023-09-01","2025-03-05"],"answer_session_ids":["nightly-9-s2"]}
|
||||
{"question_id":"nightly-10","question_type":"single-session-assistant","question":"What is the demo-day-example pitch length?","answer":"three minutes","haystack_sessions":[[{"role":"user","content":"How long are demo-day-example pitches?"},{"role":"assistant","content":"demo-day-example pitches are three minutes per founder."}]],"haystack_session_ids":["nightly-10-s1"],"haystack_dates":["2025-04-22"],"answer_session_ids":["nightly-10-s1"]}
|
||||
359
test/nightly-quality-probe.test.ts
Normal file
359
test/nightly-quality-probe.test.ts
Normal file
@@ -0,0 +1,359 @@
|
||||
/**
|
||||
* v0.40.1.0 Track D / T6+T7 — Nightly quality probe phase + doctor check.
|
||||
*
|
||||
* Hermetic: every external effect goes through the NightlyProbeDeps DI
|
||||
* surface. No PGLite, no real LLM calls, no env mutation outside withEnv.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
import {
|
||||
runNightlyQualityProbe,
|
||||
shouldRunNightly,
|
||||
type NightlyProbeDeps,
|
||||
type NightlyProbeResult,
|
||||
} from '../src/core/cycle/nightly-quality-probe.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Hermetic audit dir per test
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let auditTmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
auditTmp = mkdtempSync(join(tmpdir(), 'qprobe-audit-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
try { rmSync(auditTmp, { recursive: true, force: true }); } catch { /* best */ }
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 1. shouldRunNightly pure function
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('shouldRunNightly (pure function, rate-limit logic)', () => {
|
||||
test('empty history → run', () => {
|
||||
expect(shouldRunNightly(new Date('2026-05-22T00:00:00Z'), [])).toEqual({ run: true });
|
||||
});
|
||||
|
||||
test('last event > 24h ago → run', () => {
|
||||
const r = shouldRunNightly(
|
||||
new Date('2026-05-22T00:00:00Z'),
|
||||
[{ ts: '2026-05-20T00:00:00Z' }],
|
||||
);
|
||||
expect(r).toEqual({ run: true });
|
||||
});
|
||||
|
||||
test('last event within 24h → rate-limited', () => {
|
||||
const r = shouldRunNightly(
|
||||
new Date('2026-05-22T00:00:00Z'),
|
||||
[{ ts: '2026-05-21T12:00:00Z' }],
|
||||
);
|
||||
expect(r).toEqual({ run: false, reason: 'rate_limited' });
|
||||
});
|
||||
|
||||
test('one event old, one event recent → rate-limited (any recent fires it)', () => {
|
||||
const r = shouldRunNightly(
|
||||
new Date('2026-05-22T00:00:00Z'),
|
||||
[
|
||||
{ ts: '2026-05-01T00:00:00Z' },
|
||||
{ ts: '2026-05-21T20:00:00Z' },
|
||||
],
|
||||
);
|
||||
expect(r).toEqual({ run: false, reason: 'rate_limited' });
|
||||
});
|
||||
|
||||
test('corrupt timestamp → ignored (does not rate-limit)', () => {
|
||||
const r = shouldRunNightly(
|
||||
new Date('2026-05-22T00:00:00Z'),
|
||||
[{ ts: 'not a date' }],
|
||||
);
|
||||
expect(r).toEqual({ run: true });
|
||||
});
|
||||
|
||||
test('configurable window respected', () => {
|
||||
// 1-hour window: 6h ago counts as old.
|
||||
const r = shouldRunNightly(
|
||||
new Date('2026-05-22T00:00:00Z'),
|
||||
[{ ts: '2026-05-21T18:00:00Z' }],
|
||||
60 * 60 * 1000,
|
||||
);
|
||||
expect(r).toEqual({ run: true });
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 2. runNightlyQualityProbe via DI stubs
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function makeDeps(overrides: Partial<NightlyProbeDeps> = {}): NightlyProbeDeps {
|
||||
return {
|
||||
isEnabled: async () => true,
|
||||
hasEmbeddingProvider: async () => true,
|
||||
resolveMaxUsd: async () => 5,
|
||||
resolveRepoRoot: async () => process.cwd(),
|
||||
runLongMemEval: async () => { /* stub */ },
|
||||
runCrossModalBatch: async () => ({
|
||||
exitCode: 0,
|
||||
summary: {
|
||||
pass_count: 5, fail_count: 0, inconclusive_count: 0, error_count: 0,
|
||||
est_cost_usd: 0.35, verdict: 'pass',
|
||||
},
|
||||
}),
|
||||
now: () => new Date(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('runNightlyQualityProbe (DI stub harness)', () => {
|
||||
test('disabled config → outcome: disabled, no audit row', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps({ isEnabled: async () => false }));
|
||||
expect(r.outcome).toBe('disabled');
|
||||
expect(r.exit_code).toBe(0);
|
||||
// No audit row written.
|
||||
const events = await readEvents();
|
||||
expect(events.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('enabled + no embedding key → outcome: no_embedding_key with audit row', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps({ hasEmbeddingProvider: async () => false }));
|
||||
expect(r.outcome).toBe('no_embedding_key');
|
||||
const events = await readEvents();
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].outcome).toBe('no_embedding_key');
|
||||
});
|
||||
});
|
||||
|
||||
test('enabled + recent run within 24h → outcome: rate_limited', async () => {
|
||||
// Pre-seed a recent audit event by running the probe once first.
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
// First run succeeds.
|
||||
await runNightlyQualityProbe(makeDeps());
|
||||
// Second run, same hour → rate_limited.
|
||||
const r2 = await runNightlyQualityProbe(makeDeps());
|
||||
expect(r2.outcome).toBe('rate_limited');
|
||||
const events = await readEvents();
|
||||
expect(events.length).toBe(2);
|
||||
expect(events[1].outcome).toBe('rate_limited');
|
||||
});
|
||||
});
|
||||
|
||||
test('enabled + PASS summary → outcome: pass with audit row', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps());
|
||||
expect(r.outcome).toBe('pass');
|
||||
expect(r.exit_code).toBe(0);
|
||||
const events = await readEvents();
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].outcome).toBe('pass');
|
||||
expect(events[0].pass_count).toBe(5);
|
||||
expect(events[0].est_cost_usd).toBe(0.35);
|
||||
});
|
||||
});
|
||||
|
||||
test('enabled + FAIL summary → outcome: fail', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps({
|
||||
runCrossModalBatch: async () => ({
|
||||
exitCode: 1,
|
||||
summary: {
|
||||
pass_count: 7, fail_count: 3, inconclusive_count: 0, error_count: 0,
|
||||
est_cost_usd: 0.42, verdict: 'fail',
|
||||
},
|
||||
}),
|
||||
}));
|
||||
expect(r.outcome).toBe('fail');
|
||||
expect(r.exit_code).toBe(1);
|
||||
const events = await readEvents();
|
||||
expect(events[0].outcome).toBe('fail');
|
||||
expect(events[0].fail_count).toBe(3);
|
||||
});
|
||||
});
|
||||
|
||||
test('runLongMemEval throws → outcome: error with audit row', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps({
|
||||
runLongMemEval: async () => { throw new Error('longmemeval blew up'); },
|
||||
}));
|
||||
expect(r.outcome).toBe('error');
|
||||
expect(r.exit_code).toBe(1);
|
||||
const events = await readEvents();
|
||||
expect(events[0].outcome).toBe('error');
|
||||
expect(events[0].detail).toContain('longmemeval blew up');
|
||||
});
|
||||
});
|
||||
|
||||
test('missing fixture → outcome: error', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps({
|
||||
resolveRepoRoot: async () => '/this/repo/root/does/not/exist',
|
||||
}));
|
||||
expect(r.outcome).toBe('error');
|
||||
const events = await readEvents();
|
||||
expect(events[0].outcome).toBe('error');
|
||||
expect(events[0].detail).toContain('not found');
|
||||
});
|
||||
});
|
||||
|
||||
test('audit event records fixture_sha8 on successful runs', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: auditTmp }, async () => {
|
||||
const r = await runNightlyQualityProbe(makeDeps());
|
||||
expect(r.outcome).toBe('pass');
|
||||
const events = await readEvents();
|
||||
expect(events[0].fixture_sha8).toMatch(/^[0-9a-f]{8}$/);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function readEvents(): Promise<any[]> {
|
||||
// Re-import so it uses the override env var picked up at call time.
|
||||
const { readRecentQualityProbeEvents } = await import('../src/core/audit-quality-probe.ts');
|
||||
return readRecentQualityProbeEvents(2);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 3. computeNightlyQualityProbeHealthCheck pure function (doctor.ts coverage)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('computeNightlyQualityProbeHealthCheck — pure doctor branch coverage', () => {
|
||||
test('disabled + no events → ok with paste-ready enable hint', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const check = computeNightlyQualityProbeHealthCheck(false, []);
|
||||
expect(check.name).toBe('nightly_quality_probe_health');
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toMatch(/disabled \(opt-in\)/);
|
||||
expect(check.message).toMatch(/gbrain config set autopilot\.nightly_quality_probe\.enabled true/);
|
||||
});
|
||||
|
||||
test('enabled + no events → ok pending', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, []);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toMatch(/enabled but no probe events/);
|
||||
});
|
||||
|
||||
test('enabled + all-PASS events → ok with latest timestamp', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [
|
||||
{ outcome: 'pass', ts: '2026-05-20T03:00:00Z' },
|
||||
{ outcome: 'pass', ts: '2026-05-21T03:00:00Z' },
|
||||
{ outcome: 'pass', ts: '2026-05-22T03:00:00Z' },
|
||||
];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toMatch(/3 PASS runs/);
|
||||
expect(check.message).toContain('2026-05-22T03:00:00Z');
|
||||
});
|
||||
|
||||
test('enabled + ANY fail/error/budget_exceeded → warn with per-outcome counts', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [
|
||||
{ outcome: 'pass', ts: '2026-05-19T03:00:00Z' },
|
||||
{ outcome: 'fail', ts: '2026-05-20T03:00:00Z' },
|
||||
{ outcome: 'error', ts: '2026-05-21T03:00:00Z', detail: 'longmemeval blew up' },
|
||||
{ outcome: 'budget_exceeded', ts: '2026-05-22T03:00:00Z' },
|
||||
];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/3 non-PASS runs/);
|
||||
expect(check.message).toMatch(/pass=1/);
|
||||
expect(check.message).toMatch(/fail=1/);
|
||||
expect(check.message).toMatch(/error=1/);
|
||||
expect(check.message).toMatch(/budget=1/);
|
||||
// Latest in the list is what surfaces in the message.
|
||||
expect(check.message).toContain('budget_exceeded');
|
||||
expect(check.message).toContain('2026-05-22T03:00:00Z');
|
||||
});
|
||||
|
||||
test('latest event with detail → detail surfaces in warn message', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [
|
||||
{ outcome: 'error', ts: '2026-05-22T03:00:00Z', detail: 'no embedding provider' },
|
||||
];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toContain('no embedding provider');
|
||||
});
|
||||
|
||||
test('single non-PASS event uses singular grammar', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [{ outcome: 'fail', ts: '2026-05-22T03:00:00Z' }];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/1 non-PASS run /); // "run " not "runs "
|
||||
});
|
||||
|
||||
test('single PASS event uses singular grammar', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [{ outcome: 'pass', ts: '2026-05-22T03:00:00Z' }];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toMatch(/1 PASS run /); // "run " not "runs "
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 4. Codex CDX-5 — doctor flags ALL non-PASS outcomes (no_embedding_key,
|
||||
// rate_limited, inconclusive must trip warn, not get silently reported as PASS)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('codex CDX-5 — doctor health: every non-PASS outcome surfaces', () => {
|
||||
test('no_embedding_key outcome → warn (was silently PASS before CDX-5 fix)', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [{ outcome: 'no_embedding_key', ts: '2026-05-22T03:00:00Z' }];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/no_embed_key=1/);
|
||||
});
|
||||
|
||||
test('rate_limited outcome → warn', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [{ outcome: 'rate_limited', ts: '2026-05-22T03:00:00Z' }];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/rate_limited=1/);
|
||||
});
|
||||
|
||||
test('inconclusive outcome → warn', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [{ outcome: 'inconclusive', ts: '2026-05-22T03:00:00Z' }];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/inconclusive=1/);
|
||||
});
|
||||
|
||||
test('counts include the new outcome buckets when mixed with pass/fail/error', async () => {
|
||||
const { computeNightlyQualityProbeHealthCheck } = await import('../src/commands/doctor.ts');
|
||||
const events = [
|
||||
{ outcome: 'pass', ts: '2026-05-16T03:00:00Z' },
|
||||
{ outcome: 'fail', ts: '2026-05-17T03:00:00Z' },
|
||||
{ outcome: 'error', ts: '2026-05-18T03:00:00Z' },
|
||||
{ outcome: 'inconclusive', ts: '2026-05-19T03:00:00Z' },
|
||||
{ outcome: 'budget_exceeded', ts: '2026-05-20T03:00:00Z' },
|
||||
{ outcome: 'no_embedding_key', ts: '2026-05-21T03:00:00Z' },
|
||||
{ outcome: 'rate_limited', ts: '2026-05-22T03:00:00Z' },
|
||||
];
|
||||
const check = computeNightlyQualityProbeHealthCheck(true, events);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toMatch(/6 non-PASS runs/); // 7 total, 1 pass, 6 bad
|
||||
expect(check.message).toMatch(/pass=1/);
|
||||
expect(check.message).toMatch(/fail=1/);
|
||||
expect(check.message).toMatch(/error=1/);
|
||||
expect(check.message).toMatch(/inconclusive=1/);
|
||||
expect(check.message).toMatch(/budget=1/);
|
||||
expect(check.message).toMatch(/no_embed_key=1/);
|
||||
expect(check.message).toMatch(/rate_limited=1/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user