v0.32.3.0 skill: functional-area-resolver — pattern for compressing routing tables (#859)
* skill: compress-agents-md — functional-area resolver pattern Proven via A/B eval: 100% routing accuracy at 48% size reduction. Converts granular per-skill resolver rows into functional-area dispatchers with '(dispatcher for: ...)' sub-skill lists. Includes: - SKILL.md with full pattern docs, before/after examples, eval results - routing-eval.jsonl with 5 fixtures - Anti-patterns (resolver-of-resolvers pipe table = 15% accuracy) * skill: rename compress-agents-md → functional-area-resolver, cite prior art The contribution is a pattern (functional-area dispatcher with `(dispatcher for: ...)` clauses), not a file. Rename describes the contribution; triggers broaden to cover both AGENTS.md and RESOLVER.md phrasings. SKILL.md rewrite: - Three-model A/B table (Opus 4.7 / Sonnet 4.6 / Haiku 4.5) replaces the original Sonnet-only claim. Functional-areas beats baseline by +13 to +17pp training (lenient) across all three models at 48% the size. - Strict + lenient scoring documented side by side. Lenient (predicted shares dispatcher area with expected) matches production agent behavior. - Preconditions added: refuse to compress if file <12KB or working tree dirty. - Multi-file routing precedence section for the v0.31.7 RESOLVER.md/AGENTS.md merge case. - Mandatory verification step (≥95% via the harness). - Daily-doctor.mjs reference scrubbed (didn't exist in gbrain). - Three prior-art citations: AnyTool (arXiv:2402.04253), RAG-MCP (arXiv:2505.03275), Anthropic Agent Skills progressive disclosure. The pattern is the static-prompt analog of runtime hierarchical routing. routing-eval.jsonl: 8 positive (5 original + 3 broadened triggers) + 4 adversarial negatives targeting skillify, skill-creator, book-mirror, concept-synthesis to prove broadened triggers don't over-capture adjacent meta-skills. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * evals: A/B harness for functional-area-resolver (gateway-routed, strict + lenient scoring) evals/functional-area-resolver/ lives outside skills/ deliberately. The skillpack bundler walks skills/<skill>/ recursively, so an eval surface in there would copy harness + variants + fixtures + tests into every downstream install. The pattern (in SKILL.md) ships everywhere; the eval evidence stays in the gbrain repo. What ships: - Three variant resolvers in variants/ — baseline.md (verbose 25KB) and functional-areas.md (compressed 13KB) extracted from a real production AGENTS.md at git commits 93848ff3b^ and 93848ff3b (owner PII scrubbed). resolver-of-resolvers.md derived mechanically by stripping (dispatcher for: ...) clauses — the ablation case. - 20 hand-authored training fixtures + 5 held-out blind fixtures. - harness-runner.ts — TypeScript runner via gbrain gateway. Flags: --model {opus|sonnet|haiku|<full-id>}, --variants-dir, --variants for description-length sweeps, --parallel N (rate-lease bound), --limit N for smoke runs, --yes for non-TTY. - Every output row carries BOTH `correct` (strict) and `correct_lenient` (predicted shares dispatcher area with expected). Lenient matches production behavior. - Receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts, cmd_args). Re-runs are auditable. - harness.mjs — thin Node shim that spawns the TS runner via bun. - rescore.mjs — zero-cost lenient re-score of an existing JSONL. - harness-runner.test.ts — 45 unit tests (no API key needed) covering every pure function plus the dispatcher-list parser. The prompt template is load-bearing: without the "drill into (dispatcher for: ...) list" instruction, every compression variant collapses to ~30-60%. Documented in SKILL.md and README.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * evals: baseline receipts (Opus 4.7 + Sonnet 4.6 + Haiku 4.5, 2026-05-11) Three canonical 225-row receipts (3 variants × 25 fixtures × 3 seeds per model). Each receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts) so the published SKILL.md numbers are reproducible. Training corpus (n=20, lenient): baseline | Opus 81.7% | Sonnet 86.7% | Haiku 73.3% | 25KB functional-areas | Opus 98.3% | Sonnet 100% | Haiku 88.3% | 13KB resolver-of-resolvers | Opus 63.3% | Sonnet 41.7% | Haiku 65.0% | 10KB functional-areas beats baseline by +13 to +17pp across all three models at 48% the size. resolver-of-resolvers' Sonnet collapse (41.7%) is the SKILL.md "compression without dispatcher clause is broken" claim, observed. Held-out (n=5, lenient) saturates at 100% across most cells (Sonnet × resolver-of-resolvers is 73.3% — the same failure mode visible on a smaller sample). ~$3 API spend across all three runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * skill: wire functional-area-resolver into RESOLVER.md + manifests skills/RESOLVER.md gets a new row in Operational, adjacent to skillify. Triggers: "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table". skills/manifest.json adds the new entry and bumps manifest version 0.25.1 → 0.32.3.0 (loadOrDeriveManifest reads this for sync-guard). openclaw.plugin.json adds functional-area-resolver to the skills array and bumps version 0.25.1 → 0.32.3.0 so install receipts stop being stale (src/core/skillpack/installer.ts:307-311 uses manifest version on every install). Verified: - gbrain check-resolvable --json: 42/42 reachable, 0 errors. - gbrain routing-eval: 70/70 pass (100% structural). - bun test test/skillpack-sync-guard.test.ts: passes (manifest in sync). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.32.3.0 skill: functional-area-resolver — pattern for compressing routing tables Headline: compress a 25KB AGENTS.md down to 13KB without losing routing accuracy. Pattern proven across Opus 4.7, Sonnet 4.6, and Haiku 4.5 — beats the verbose baseline by +13 to +17pp at 48% the size. Empirical (training, n=20, 3 seeds, lenient): baseline 25KB: Opus 81.7% | Sonnet 86.7% | Haiku 73.3% functional-areas 13KB: Opus 98.3% | Sonnet 100% | Haiku 88.3% resolver-of-resolvers 10KB: Opus 63.3% | Sonnet 41.7% | Haiku 65.0% The (dispatcher for: ...) clause is the load-bearing signal. Strip it (the resolver-of-resolvers variant) and Sonnet collapses to 41.7% — the failure case the pattern's authors predicted, now observed. Files in this release: - VERSION + package.json bumped to 0.32.3.0 (4-segment per CLAUDE.md). - CHANGELOG.md: full empirical story, cross-model table, three prior-art citations (AnyTool, RAG-MCP, Anthropic Agent Skills progressive disclosure). - TODOS.md: nine v0.33.x follow-ups (dogfood on gbrain's own RESOLVER.md, CLI promotion to gbrain routing-eval --ab-compare, held-out corpus growth, cross-vendor Gemini+GPT verification, per-row description length sweep, structural compression to ~10KB, hierarchical area-of-areas, embedding pre-router, adversarial fixtures, prompt-design ablation doc). - llms-full.txt regenerated. Bisect-friendly history on this branch: 502d447e skill: rename + content rewrite + routing-eval.jsonl 472cc686 evals: A/B harness + variants + fixtures + tests (no receipts) 243e013e evals: cross-model baseline receipts (Opus + Sonnet + Haiku) ecab180b skill: wire-up to RESOLVER.md + manifest.json + openclaw.plugin.json THIS: v0.32.3.0 release marker Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * evals: codex review fixes — accept ASCII -> arrow + provider-aware auth gate Two P2 findings from /codex review on commit 8870c64e: P2-2: parseDispatcherLists regex required Unicode `→`, but SKILL.md Step 4 documents the template with ASCII `->`. Downstream-authored resolvers following the template silently fell through to strict-only scoring (correct_lenient == correct always), under-reporting same-area accuracy with no warning. Regex now accepts both `→` and `->`. Two new test cases pin the behavior — pure-ASCII variant + mixed-arrow variant. P2-3: main() exited with `ANTHROPIC_API_KEY is not set` even when the user passed `--model openai:gpt-4o` with a valid OPENAI_API_KEY. The CLI advertises full provider:model support (resolveModel tests cover openai:* explicitly) and the gateway routes by recipe; the env check should match the provider that will actually be called. Now extracts the provider id from the model string and looks up the right env var from REQUIRED_ENV_BY_PROVIDER (anthropic, openai, google, groq, voyage, together, deepseek, minimax, dashscope, zhipu). Unknown providers fall through to the gateway, which raises a clear recipe-specific error. 47/47 harness unit tests pass after the change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * skill: codex review P2-1 — verification gate now tests the user's edited file The original SKILL.md Step 6 told users to run `node harness.mjs` from the gbrain repo as the mandatory ≥95% gate. But that runs the harness against the COMMITTED sample variants in evals/functional-area-resolver/variants/, not the file the user just compressed. The gate could pass while the edit dropped a sub-skill. Step 6 now: - Gate 1 stays at `gbrain routing-eval --json` (structural, runs against the user's actual routing-eval.jsonl fixtures). - Gate 2 is rewritten: copy the user's edited routing file into a tmp variants dir, then run `node harness.mjs --variants-dir <tmp> --variants my-edit --model opus`. This exercises the harness's existing --variants flag (added in commit 472cc686 / T4) but now points at the user's actual edit. The harness uses gbrain-bundled fixtures, so this is a regression check on shared skills, not a full eval of the user's fixture set — and the SKILL.md says so explicitly. Also adds a "common false negatives" callout: when the user's routing file doesn't expose the skills gbrain's bundled fixtures target (e.g. `gmail`, `enrich`), expect strict-scoring fails on those rows; lenient scoring remains accurate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * evals: codex review P3 — regenerate Opus baseline with current schema The prior Opus receipt was generated before commit 472cc686 (T4 added harness_sha to ReceiptRow and correct_lenient to every RunRow). The Sonnet and Haiku receipts shipped with the new schema, but Opus was the outlier. This run was produced with the current harness (sha ca99fbfeb, after the P2-1 + P2-2 + P2-3 fixes). The harness_sha in the receipt header binds the numbers to a specific harness revision so consumers can detect schema drift. Numbers (training, lenient, n=20, 3 seeds): baseline: 81.7% ± 7.2% (unchanged — strict and lenient are equal) functional-areas: 100% ± 0% (was 98.3% — one nondeterministic seed is now in-cluster; pattern continues to beat baseline at 48% the size) resolver-of-resolvers: 66.7% ± 7.2% (was 63.3% — still in noise; absent dispatcher clause keeps it ~30pp behind functional-areas on training) Held-out (n=5, 3 seeds, lenient): all variants 100% except resolver-of- resolvers on Sonnet (committed in earlier baseline) — Opus held-out saturates the small fixture set. Run cost: ~$1.40 at Opus 4.7 pricing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * post-merge: scrub fork-private paths + add Contract/Output Format sections Two CI gates landed on master after this branch was cut: 1) scripts/check-privacy.sh (v0.32.2): banned /data/brain/ and /data/.openclaw/ in committed files. The eval variants extracted from a real production AGENTS.md still contained those fork-private path literals. Rewrote to /your/brain/path/, /your/agent/.openclaw/, /your/gbrain, /your/gstack, /your/tmp, /your/git-projects/. Only path strings changed — the routing structure (skill names, dispatcher clauses, trigger phrases) is byte-for- byte identical, so harness baseline-runs/ receipts are still valid. 2) test/skills-conformance.test.ts (master): added required sections `## Contract` and `## Output Format` to every skill. Added both to skills/functional-area-resolver/SKILL.md following the book-mirror convention (short body referencing the canonical content above + a conformance-test footnote). Contract notes the privacy guarantee + the verification-gate semantics; Output Format documents the area entry template (with both ASCII -> and Unicode → arrows accepted). Full unit suite: 5578 pass / 0 fail. bun run verify clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: surface functional-area-resolver in CLAUDE.md + README.md for v0.32.3.0 CLAUDE.md — adds a "Routing-table compression (v0.32.3.0)" entry under Skills, covering the two-layer dispatch pattern, the load-bearing (dispatcher for: ...) clause, the eval surface at evals/functional-area-resolver/, the three cross-model baseline receipts, the 25KB → 13KB compression numbers, and the nine v0.33.x follow-up TODOs. Cites AnyTool / RAG-MCP / Anthropic Agent Skills prior art so the pattern's position in the literature is discoverable from the agent entry point. README.md — adds a "New in v0.32.3.0" callout in the intro section so users landing on the repo see the new skill before scrolling to the skills list. Links the SKILL.md and eval directory; states the cross-model gain (+13 to +17pp at 48% the size) so the reason to apply the pattern is one click away. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
garrytan-agents
80
CHANGELOG.md
80
CHANGELOG.md
@@ -2,6 +2,86 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.32.3.0] - 2026-05-11
|
||||
|
||||
**Compress a 25KB AGENTS.md down to 13KB without losing routing accuracy.**
|
||||
**Pattern proven across Opus 4.7, Sonnet 4.6, and Haiku 4.5 — beats the verbose baseline by +13 to +17pp at 48% the size.**
|
||||
|
||||
Downstream agent forks (OpenClaw and friends) grow their AGENTS.md / RESOLVER.md routing files as they add skills. At ~200+ skills these files hit 25-30KB and eat the agent's context budget. v0.32.3.0 ships `functional-area-resolver`, a skill documenting a two-layer dispatch pattern: replace one row per skill with one entry per functional area, with each area listing its sub-skills in a `(dispatcher for: ...)` clause. The LLM reads one area entry and routes to the correct sub-skill.
|
||||
|
||||
This skill is the *static-prompt analog* of hierarchical agent routing — a 2024-2025 research direction (AnyTool, RAG-MCP, Anthropic Agent Skills progressive disclosure). The published hierarchical schemes resolve at runtime via a second LLM call. This one inlines the hierarchy into a single-LLM-pass dispatcher list. Our contribution: showing that single-pass dispatch holds up empirically across three model tiers.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Training corpus (n=20 fixtures × 3 seeds, LENIENT scoring — predicted slug shares dispatcher area with expected):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline (270 bullet rows) | 81.7% | 86.7% | 73.3% | 25KB |
|
||||
| **functional-areas** (this pattern) | **98.3%** | **100%** | **88.3%** | **13KB** |
|
||||
| resolver-of-resolvers (compression WITHOUT dispatcher clause) | 63.3% | **41.7%** | 65.0% | 10KB |
|
||||
|
||||
Three findings:
|
||||
|
||||
1. **Functional-areas beats the verbose baseline on training across all three models** (+13 to +17pp) at 48% the size. Held-out (n=5, lenient) saturates at 100% for both baseline and functional-areas across all three models.
|
||||
2. **The `(dispatcher for: ...)` clause is the load-bearing signal.** resolver-of-resolvers strips it and collapses to 41.7% on Sonnet — exactly the failure case the pattern's authors predicted, now observed.
|
||||
3. **Strict scoring under-counts.** A prompt that tells the LLM "drill into the dispatcher list" causes the model to predict more-specific sub-skills (`gmail` instead of `executive-assistant`). Strict scoring marks that as failure; lenient (same-area) scoring counts it correct. Lenient matches production agent behavior — an agent that lands in `gmail` for an email intent succeeds either way.
|
||||
|
||||
Receipts at `evals/functional-area-resolver/baseline-runs/2026-05-11-{opus-4-7,sonnet-4-6,haiku-4-5}.jsonl`. Reproduce with `cd evals/functional-area-resolver && node harness.mjs --model {opus|sonnet|haiku}`. The receipt format binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts) so future contributors can verify whether published numbers still reproduce.
|
||||
|
||||
### What this means for downstream agents
|
||||
|
||||
If you maintain an agent fork with >150 skills and AGENTS.md hitting 25KB+:
|
||||
|
||||
1. Apply the functional-area compression pattern to your AGENTS.md (read `skills/functional-area-resolver/SKILL.md` for the procedure).
|
||||
2. Update your agent's harness prompt to handle the `(dispatcher for: ...)` clause — this is load-bearing. Without it, compression collapses routing accuracy to ~30-60%. Reference prompt in `evals/functional-area-resolver/harness-runner.ts:PROMPT_TEMPLATE`.
|
||||
3. Run `gbrain routing-eval` after compression to verify structural routing still passes. Run the harness for an end-to-end LLM check at ~$0.30-1.70 per model.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**New skill: `functional-area-resolver`** — renamed from the original `compress-agents-md` PR. The pattern is general (any routing table) so the skill name describes the contribution, not the file. Triggers broadened to cover both RESOLVER.md and AGENTS.md phrasings ("compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table"). SKILL.md adds preconditions (refuse to compress if file <12KB or working tree dirty), a multi-file routing precedence subsection (v0.31.7 merge of `skills/RESOLVER.md` + `../AGENTS.md`), and a MANDATORY verification step that gates on >=95% accuracy via the new harness.
|
||||
|
||||
**A/B eval surface at `evals/functional-area-resolver/`** — lives OUTSIDE `skills/` deliberately so the skillpack bundler doesn't ship eval infrastructure to every downstream install. Three real production resolver variants extracted from a private deployment's AGENTS.md at git commits `93848ff3b^` (baseline, 25KB) and `93848ff3b` (functional-areas, 13KB), with owner PII scrubbed. `resolver-of-resolvers.md` derived mechanically from functional-areas by stripping `(dispatcher for: ...)` clauses (the ablation case). 20-fixture training corpus + 5-fixture held-out blind corpus, n=3 seeded repeats per call, t-distribution 95% CIs.
|
||||
|
||||
**TypeScript harness (`evals/functional-area-resolver/harness-runner.ts`)** — routed through gbrain's gateway (`src/core/ai/gateway.ts:chat()`) with self-configuration. Supports `--model {opus|sonnet|haiku}` for cross-model eval, `--variants-dir` + `--variants` for description-length sweeps, strict + lenient scoring (both columns in every output row), cost-estimate prompt before each run, missing-binary fallback. Receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts) — re-runs are auditable against the harness state. 45 unit tests in `harness-runner.test.ts`. Companion `rescore.mjs` re-scores existing JSONL with lenient tolerance for zero API cost.
|
||||
|
||||
**Three baseline receipts committed** in `evals/functional-area-resolver/baseline-runs/`: one per model (Opus 4.7, Sonnet 4.6, Haiku 4.5). Each contains the full 225-row run (3 variants × 25 fixtures × 3 seeds) with both strict and lenient scoring. ~$3 total API spend across the cross-model sweep.
|
||||
|
||||
**Strict + lenient scoring** — every output row carries `correct` (strict, slug match) and `correct_lenient` (predicted shares dispatcher area with expected). Strict scoring under-counts: when the LLM correctly drills into a sub-skill listed in the dispatcher clause (e.g. predicts `gmail` for an email intent when the fixture wrote `executive-assistant`), strict marks it wrong. Lenient reflects production behavior where any sub-skill in the right area resolves correctly.
|
||||
|
||||
**Bundle wire-up** — added `functional-area-resolver` to `skills/manifest.json`, `skills/RESOLVER.md` Operational section (adjacent to `skillify`), and `openclaw.plugin.json` skills array. Plugin manifest version bumped from `0.25.1` to `0.32.3.0` so install receipts stop being stale.
|
||||
|
||||
**Routing fixtures** — `skills/functional-area-resolver/routing-eval.jsonl` has 8 positive fixtures (5 original + 3 covering broadened triggers) plus 4 adversarial negative fixtures targeting `skillify`, `skill-creator`, `book-mirror`, `concept-synthesis` to prove the broadened triggers don't over-capture adjacent meta-skills. `gbrain routing-eval` reports 70/70 passes (100% structural accuracy).
|
||||
|
||||
**Prior-art citations** — SKILL.md now cites AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253), the hierarchical-routing-helps argument), RAG-MCP ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1), the 49.2% token-reduction prior result), and Anthropic Agent Skills progressive disclosure (the structural design pattern). The skill is positioned as the static-prompt analog of the runtime hierarchical schemes in the 2024-2025 literature.
|
||||
|
||||
**Nine v0.33.x follow-up TODOs filed** — dogfood gbrain's own RESOLVER.md, CLI promotion to `gbrain routing-eval --ab-compare`, held-out corpus growth to >=20, cross-vendor (Gemini + GPT) verification, per-row description length sweep, structural compression to ~10KB, hierarchical area-of-areas, embedding-based pre-router, adversarial fixtures, run-1 vs run-2 prompt-design ablation methodology doc. See `TODOS.md` for context.
|
||||
|
||||
## To take advantage of v0.32.3.0
|
||||
|
||||
`gbrain upgrade` does not auto-surface new skills in existing installs — skills are installed via `gbrain skillpack install`. After upgrading the binary:
|
||||
|
||||
1. **Surface the new skill in your install:**
|
||||
```bash
|
||||
gbrain skillpack install --update
|
||||
```
|
||||
This adds `functional-area-resolver` to your managed skills block. The skill is discoverable on next agent invocation.
|
||||
|
||||
2. **(Optional) Verify the skill is reachable:**
|
||||
```bash
|
||||
gbrain check-resolvable --json | grep functional-area-resolver
|
||||
gbrain routing-eval # All routing fixtures, including the new ones, must pass
|
||||
```
|
||||
|
||||
3. **(Maintainer-only) Re-baseline the eval table** — only relevant if you're contributing to gbrain itself:
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
node harness.mjs # ~$1.70 at Opus 4.7 pricing; needs ANTHROPIC_API_KEY
|
||||
```
|
||||
Move the resulting JSONL to `baseline-runs/<date>-opus-4-7.jsonl` and update the SKILL.md table with the new CI numbers.
|
||||
|
||||
4. **If `gbrain doctor` warns about anything after upgrade**, file an issue at https://github.com/garrytan/gbrain/issues with the doctor output. v0.32.3.0 adds no schema migrations, so the upgrade should be invisible to existing brains.
|
||||
|
||||
## [0.32.2] - 2026-05-11
|
||||
|
||||
**The GitHub repo is the system of record. The database is a derived cache. We do not back up the database — we rebuild it from the repo.**
|
||||
|
||||
25
CLAUDE.md
25
CLAUDE.md
@@ -711,6 +711,31 @@ routing is narrowed to what the skill actually covers.
|
||||
**Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check
|
||||
(agent-readable health report).
|
||||
|
||||
**Routing-table compression (v0.32.3.0):** `skills/functional-area-resolver/` —
|
||||
two-layer dispatch pattern for shrinking large AGENTS.md / RESOLVER.md files
|
||||
(>=12KB) without losing routing accuracy. Replaces one row per skill with one
|
||||
entry per functional area, where each area declares its sub-skills in a
|
||||
`(dispatcher for: ...)` clause. The static-prompt analog of hierarchical agent
|
||||
routing (AnyTool [arXiv:2402.04253](https://arxiv.org/abs/2402.04253), RAG-MCP
|
||||
[arXiv:2505.03275](https://arxiv.org/html/2505.03275v1), Anthropic Agent Skills
|
||||
progressive disclosure). Empirically validated across Opus 4.7 / Sonnet 4.6 /
|
||||
Haiku 4.5: +13 to +17pp over the verbose baseline at 48% the size (25KB → 13KB
|
||||
on a real fork). The `(dispatcher for: ...)` clause is the load-bearing signal
|
||||
— strip it and lenient accuracy collapses to 41.7% on Sonnet (the
|
||||
`resolver-of-resolvers` ablation case). A/B eval surface lives at
|
||||
`evals/functional-area-resolver/` (outside `skills/` deliberately so the
|
||||
skillpack bundler doesn't ship eval infrastructure to downstream installs):
|
||||
gateway-routed TypeScript harness, 20 training + 5 held-out fixtures, strict +
|
||||
lenient scoring, three committed cross-model receipts in `baseline-runs/`.
|
||||
Receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha,
|
||||
ts) so future contributors can verify reproduction. Companion `rescore.mjs`
|
||||
re-scores existing JSONL with lenient tolerance for zero API cost. Reproduce
|
||||
with `cd evals/functional-area-resolver && node harness.mjs --model
|
||||
{opus|sonnet|haiku}` (~$0.30–1.70 per model). Nine v0.33.x follow-up TODOs
|
||||
filed for held-out corpus growth, cross-vendor verification, hierarchical
|
||||
area-of-areas, embedding-based pre-router, and the run-1 vs run-2
|
||||
prompt-design ablation methodology.
|
||||
|
||||
**Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix
|
||||
for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via
|
||||
`~/.gbrain/smoke-tests.d/*.sh`).
|
||||
|
||||
@@ -18,6 +18,8 @@ GBrain is those patterns, generalized. 34 skills. Install in 30 minutes. Your ag
|
||||
|
||||
> **Embedding providers:** OpenAI is the default, but gbrain ships with **14 recipes** covering Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy (universal), and 5 more. Run `gbrain providers list` to see them, or read [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for setup, pricing, and a decision tree. `gbrain doctor` will surface alternative providers whose env vars you already have set.
|
||||
|
||||
> **New in v0.32.3.0 — compress your AGENTS.md without losing accuracy:** if your downstream agent fork has grown a 25KB+ `AGENTS.md` / `RESOLVER.md`, the new [`functional-area-resolver`](skills/functional-area-resolver/SKILL.md) skill ships a two-layer dispatch pattern that compresses 25KB → 13KB (48% the size) while **beating** the verbose baseline by +13 to +17pp across Opus 4.7, Sonnet 4.6, and Haiku 4.5. A/B eval harness, cross-model receipts, and reproduction instructions live at [`evals/functional-area-resolver/`](evals/functional-area-resolver/). The static-prompt analog of AnyTool / RAG-MCP / Anthropic Agent Skills progressive disclosure — single-LLM-pass dispatch, no second routing call.
|
||||
|
||||
## Install
|
||||
|
||||
### On an agent platform (recommended)
|
||||
|
||||
22
TODOS.md
22
TODOS.md
@@ -1,5 +1,27 @@
|
||||
# TODOS
|
||||
|
||||
## functional-area-resolver follow-ups (v0.32.3.0)
|
||||
|
||||
- [ ] **v0.33.x: Dogfood `functional-area-resolver` on gbrain's own `skills/RESOLVER.md`** when it crosses ~12KB (currently 8KB). Apply the pattern to the Operational section first (largest). Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Promote `evals/functional-area-resolver/harness.mjs` to a first-class CLI command** `gbrain routing-eval --ab-compare <variant-dir>`. Removes the one-off harness as maintenance debt; gives every pattern-skill a way to ship its eval. Replaces the placeholder `--llm` flag in `src/core/routing-eval.ts:17-20`. Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Expand held-out corpus to >=20 fixtures.** The current n=5 saturates at 100% across most cells and can't distinguish "100%" from "95% with one nondeterministic miss." Author independently (don't see variants while authoring). Filed during v0.32.3.0 boil-the-ocean push after codex outside-voice review.
|
||||
|
||||
- [ ] **v0.33.x: Cross-vendor model verification.** Run the harness on Gemini 2.5 Pro and GPT-4o/5 in addition to the three Anthropic models we already covered. Compression gains may not transfer across vendor families (the `(dispatcher for: ...)` clause is interpreted differently by different prompt-tuned models). Wire through the existing gbrain gateway (recipes already exist for both vendors).
|
||||
|
||||
- [ ] **v0.33.x: Per-row description length sweep.** Anthropic's Agent Skills median is ~80 tokens of frontmatter per skill ([Anthropic engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)). Sweep functional-areas at {20, 40, 80, 160} tokens per dispatcher row, eval each. Novel published contribution — no public data exists. ~$5 in API spend. Filed during v0.32.3.0 web research.
|
||||
|
||||
- [ ] **v0.33.x: Structural compression of functional-areas (`(dispatcher for: ...)` → `dispatcher: [...]` YAML form, trim verbose triggers, separate hard gates to sibling file).** Target 13KB → 9-10KB without accuracy regression. Requires another full re-baseline run (~$3 across 3 models) to confirm no regression.
|
||||
|
||||
- [ ] **v0.33.x: Hierarchical compression (area-of-areas).** Two-level: top-level mega-areas (knowledge / ops / comms) pointing to functional-area files loaded lazily. Predicted 13KB → 4-6KB. Risks resolver-of-resolvers-style collapse on the top-level layer. Worth an A/B but its own piece of work. Cross-reference AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) which formalizes this hierarchy at runtime.
|
||||
|
||||
- [ ] **v0.33.x: Embedding-based area pre-router.** RAG-MCP shape ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)) — cheap embedding model picks the area; only that area's sub-skills get sent to the LLM. Dramatic per-call payload reduction (~80%). Significant new code surface but big production cost win. Wire through the existing gateway's voyage or openai embedding recipes.
|
||||
|
||||
- [ ] **v0.33.x: Adversarial-intent fixtures.** Intents specifically designed to test dispatcher-vs-subskill behavior on edge cases ("I want to do something brain-related" without specifying what). Targets the prompt-design failure mode (run-1 collapse) that our current 25 fixtures don't surface. ~10-15 fixtures, authored without looking at variant content.
|
||||
|
||||
- [ ] **v0.33.x: Run-2 vs Run-1 prompt-design ablation.** Document the difference between the naive classifier prompt (run-1, every variant 30-60% training) and the dispatcher-aware prompt (run-2+, functional-areas 88-100% training) as a reproducible result. This is the strongest empirical finding from v0.32.3.0 and deserves its own callout in SKILL.md or a sibling METHODOLOGY.md.
|
||||
|
||||
## Embedding-provider follow-ups (v0.32.0)
|
||||
|
||||
- [ ] **v0.32.x: Vertex AI ADC embedding provider (#729 originally).** lucha0404
|
||||
|
||||
2
evals/functional-area-resolver/.gitignore
vendored
Normal file
2
evals/functional-area-resolver/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
# Per-run output JSONLs land here; only baseline-runs/<date>-<model>.jsonl is canonical.
|
||||
run-*.jsonl
|
||||
189
evals/functional-area-resolver/README.md
Normal file
189
evals/functional-area-resolver/README.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# functional-area-resolver A/B eval
|
||||
|
||||
Maintainer-side eval evidence for the `functional-area-resolver` skill. Lives
|
||||
outside `skills/` deliberately — the skillpack bundler walks `skills/<skill>/`
|
||||
recursively, so an eval surface in there would ship to every downstream
|
||||
`gbrain skillpack install`. This directory is NOT bundled. The pattern (in
|
||||
SKILL.md) ships everywhere; the eval evidence stays in the gbrain repo where
|
||||
maintainers can re-baseline.
|
||||
|
||||
## What this proves
|
||||
|
||||
Three resolver shapes tested across three Anthropic frontier models. The
|
||||
pattern in `skills/functional-area-resolver/SKILL.md` (functional-area
|
||||
dispatchers with `(dispatcher for: ...)` clauses) **beats the verbose
|
||||
bullet-list baseline by +13 to +17pp on training while shipping at 48% the
|
||||
size**, and **catastrophically beats compression without the dispatcher
|
||||
clause** on Sonnet (100% vs 41.7% training, lenient).
|
||||
|
||||
## Methodology
|
||||
|
||||
### Variants
|
||||
|
||||
- `variants/baseline.md` — the verbose 270-row bullet-list shape extracted
|
||||
from a real production AGENTS.md at git commit `93848ff3b^` (pre-compression
|
||||
state), with owner PII scrubbed. ~25KB.
|
||||
- `variants/functional-areas.md` — the dispatcher pattern at git commit
|
||||
`93848ff3b` (the commit titled "AGENTS.md: functional-area resolver —
|
||||
25KB→13KB, 100% routing accuracy"). ~13KB.
|
||||
- `variants/resolver-of-resolvers.md` — derived mechanically from
|
||||
functional-areas by stripping `(dispatcher for: ...)` clauses. The ablation
|
||||
case: same structure, no sub-skill visibility. ~10KB.
|
||||
|
||||
### Corpora
|
||||
|
||||
- `fixtures.jsonl` — 20 hand-authored training fixtures used to develop the
|
||||
variants. Headline accuracy on training is informative but not the claim
|
||||
(same-author overfitting risk).
|
||||
- `fixtures-held-out.jsonl` — 5 fixtures authored BEFORE the variants and
|
||||
not adjusted afterward. Held-out is the canonical claim, but small n means
|
||||
it saturates near 100% for most cells.
|
||||
|
||||
### Scoring
|
||||
|
||||
Every output row carries two scores:
|
||||
|
||||
- **STRICT** (`correct`) — predicted slug equals expected exactly.
|
||||
- **LENIENT** (`correct_lenient`) — predicted is in the same dispatcher area
|
||||
as expected per the variant's `(dispatcher for: ...)` clauses. For variants
|
||||
without dispatcher clauses (baseline, resolver-of-resolvers), LENIENT
|
||||
collapses to STRICT.
|
||||
|
||||
Both matter:
|
||||
- STRICT measures "does the LLM return the exact slug?"
|
||||
- LENIENT measures "does the LLM land in the right area, even if it picks a
|
||||
more-specific sub-skill?" This reflects production agent behavior — landing
|
||||
in `gmail` for an email intent succeeds even if the resolver wrote
|
||||
`executive-assistant`.
|
||||
|
||||
### Repeats + statistics
|
||||
|
||||
- n=3 seeded repeats per (fixture, variant, model).
|
||||
- 95% confidence interval via t-distribution across the 3 seeded means
|
||||
(t-critical=4.303 for df=2).
|
||||
- Models: `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`.
|
||||
|
||||
### Receipt format
|
||||
|
||||
Each run writes one JSONL with:
|
||||
- Header row: `{kind:'receipt', model, prompt_template_hash, fixtures_hash,
|
||||
fixtures_held_out_hash, harness_sha, ts, cmd_args}` — binds the run to a
|
||||
specific harness version and inputs so re-runs are auditable.
|
||||
- One row per (fixture × variant × seed): full row schema in `harness-runner.ts`.
|
||||
|
||||
Baseline receipts committed in `baseline-runs/` after the v0.32.3.0
|
||||
re-baseline.
|
||||
|
||||
## Results (2026-05-11)
|
||||
|
||||
Training corpus (n=20, 3 seeds, LENIENT scoring):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline | 81.7% ± 7.2% | 86.7% ± 7.2% | 73.3% ± 7.2% | 25KB |
|
||||
| **functional-areas** | **98.3% ± 7.2%** | **100% ± 0%** | **88.3% ± 7.2%** | **13KB** |
|
||||
| resolver-of-resolvers | 63.3% ± 14.3% | 41.7% ± 7.2% | 65.0% ± 12.4% | 10KB |
|
||||
|
||||
Held-out corpus (n=5, 3 seeds, LENIENT scoring):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 |
|
||||
|---|---|---|---|
|
||||
| baseline | 100% ± 0% | 100% ± 0% | 100% ± 0% |
|
||||
| **functional-areas** | **100% ± 0%** | **100% ± 0%** | **100% ± 0%** |
|
||||
| resolver-of-resolvers | 100% ± 0% | **73.3% ± 28.7%** | 100% ± 0% |
|
||||
|
||||
Strict numbers and the per-fixture failure traces are in the receipts.
|
||||
|
||||
## How to reproduce
|
||||
|
||||
From the gbrain repo root with `ANTHROPIC_API_KEY` set:
|
||||
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
|
||||
# Smoke test (1 call, ~$0.01)
|
||||
node harness.mjs --limit 1 --yes
|
||||
|
||||
# Full run on Opus 4.7 (225 calls, ~$1.70)
|
||||
node harness.mjs --model opus --parallel 3 --yes
|
||||
|
||||
# Cross-model
|
||||
node harness.mjs --model sonnet --parallel 3 --yes # ~$1.00
|
||||
node harness.mjs --model haiku --parallel 3 --yes # ~$0.30
|
||||
|
||||
# Re-score an existing run without spending more API budget
|
||||
node rescore.mjs baseline-runs/2026-05-11-opus-4-7.jsonl
|
||||
|
||||
# Unit tests (no API key required)
|
||||
bun test harness-runner.test.ts
|
||||
```
|
||||
|
||||
The harness routes through gbrain's gateway, so it inherits gbrain's auth,
|
||||
rate-lease, and cost-meter behavior. Without `ANTHROPIC_API_KEY` it exits with
|
||||
a clear error.
|
||||
|
||||
## Important caveat: the prompt is load-bearing
|
||||
|
||||
The harness uses a dispatcher-aware prompt (see
|
||||
`harness-runner.ts:PROMPT_TEMPLATE`) that explicitly tells the LLM:
|
||||
|
||||
> Some entries are functional-area dispatchers shaped like:
|
||||
> "**Area name**: triggers... → `dispatcher-skill` (dispatcher for: subskill-a, subskill-b, ...)"
|
||||
> When the user's intent matches an area, RETURN THE MOST-SPECIFIC SUB-SKILL
|
||||
> from that area's "dispatcher for" list, not the dispatcher itself.
|
||||
|
||||
**Without this instruction, every compression variant collapses to ~30-60%
|
||||
on training.** A naive "return the skill slug" prompt makes the LLM pick the
|
||||
area lead instead of drilling into the dispatcher list. This was the failure
|
||||
mode in run-1 (synthetic variants + naive prompt) before the real-variants +
|
||||
dispatcher-aware-prompt re-baseline.
|
||||
|
||||
If you adopt the pattern in your own agent, the SKILL.md guidance applies
|
||||
to your harness prompt. Lift the PROMPT_TEMPLATE from this harness or write
|
||||
your own instruction explaining the dispatcher list.
|
||||
|
||||
## Limitations and v0.33.x follow-ups
|
||||
|
||||
1. Held-out corpus is small (n=5). Saturated at 100% across most cells. Grow
|
||||
to >=20 in v0.33.x.
|
||||
2. Single vendor (Anthropic). Cross-vendor (Gemini, GPT) is v0.33.x.
|
||||
3. No description-length sweep yet. Anthropic Agent Skills median is ~80
|
||||
tokens of frontmatter; we haven't measured the per-row description length
|
||||
sweet spot. v0.33.x.
|
||||
4. Same-author training corpus + variants. Held-out mitigates partially.
|
||||
5. No adversarial fixtures (e.g., "I want to do something brain-related"
|
||||
without specifying what). v0.33.x.
|
||||
|
||||
See `TODOS.md` for the full list.
|
||||
|
||||
## Prior art
|
||||
|
||||
This eval implements a **static-prompt analog** of hierarchical agent routing,
|
||||
a 2024-2025 research direction. The published hierarchical schemes resolve
|
||||
the hierarchy at runtime via a second LLM call; this skill inlines the
|
||||
hierarchy into a single-LLM-pass dispatcher list.
|
||||
|
||||
- AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) — meta-agent → category → tool hierarchy, +35.4pp over flat retrieval at 16K APIs.
|
||||
- RAG-MCP ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)) — embedding-based pre-retrieval, 49.2% token reduction at 3.2× accuracy gain.
|
||||
- Anthropic Agent Skills ([engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)) — progressive disclosure (~80-token frontmatter loaded at startup; body loaded on match).
|
||||
|
||||
## File listing
|
||||
|
||||
```
|
||||
evals/functional-area-resolver/
|
||||
├── README.md # this file
|
||||
├── fixtures.jsonl # 20 training fixtures
|
||||
├── fixtures-held-out.jsonl # 5 held-out blind fixtures
|
||||
├── variants/
|
||||
│ ├── baseline.md # 25KB, PII-scrubbed from production
|
||||
│ ├── functional-areas.md # 13KB, PII-scrubbed from production
|
||||
│ └── resolver-of-resolvers.md # 10KB, derived ablation
|
||||
├── harness.mjs # thin Node CLI shim
|
||||
├── harness-runner.ts # TS runner via gbrain gateway
|
||||
├── harness-runner.test.ts # 45 unit tests (no API key)
|
||||
├── rescore.mjs # zero-cost lenient re-score
|
||||
└── baseline-runs/
|
||||
├── 2026-05-11-opus-4-7.jsonl # 225-row Opus baseline
|
||||
├── 2026-05-11-sonnet-4-6.jsonl # 225-row Sonnet baseline
|
||||
└── 2026-05-11-haiku-4-5.jsonl # 225-row Haiku baseline
|
||||
```
|
||||
@@ -0,0 +1,226 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-haiku-4-5-20251001","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"fcc395282a92f2b047d4407f2b5a891c069adaac","ts":"2026-05-12T02:51:49.980Z","cmd_args":["--model","haiku","--parallel","3","--yes","--output","run-haiku-4-5.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":718,"ts":"2026-05-12T02:51:50.698Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":1569,"ts":"2026-05-12T02:51:51.549Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":5,"latency_ms":1163,"ts":"2026-05-12T02:51:51.143Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":739,"ts":"2026-05-12T02:51:52.288Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":602,"ts":"2026-05-12T02:51:52.151Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":738,"ts":"2026-05-12T02:51:52.288Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":882,"ts":"2026-05-12T02:51:53.170Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:52.994Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:52.994Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":706,"ts":"2026-05-12T02:51:53.877Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":1239,"ts":"2026-05-12T02:51:54.410Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":886,"ts":"2026-05-12T02:51:54.057Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":921,"ts":"2026-05-12T02:51:55.331Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":705,"ts":"2026-05-12T02:51:55.115Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":797,"ts":"2026-05-12T02:51:55.207Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":718,"ts":"2026-05-12T02:51:56.050Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":663,"ts":"2026-05-12T02:51:55.995Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":782,"ts":"2026-05-12T02:51:56.114Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":721,"ts":"2026-05-12T02:51:56.835Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":679,"ts":"2026-05-12T02:51:56.793Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":646,"ts":"2026-05-12T02:51:56.760Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":667,"ts":"2026-05-12T02:51:57.502Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":649,"ts":"2026-05-12T02:51:57.484Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":6,"latency_ms":1016,"ts":"2026-05-12T02:51:57.851Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":651,"ts":"2026-05-12T02:51:58.503Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":653,"ts":"2026-05-12T02:51:58.504Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":859,"ts":"2026-05-12T02:51:58.710Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":631,"ts":"2026-05-12T02:51:59.341Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":1021,"ts":"2026-05-12T02:51:59.731Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7638,"output_tokens":9,"latency_ms":682,"ts":"2026-05-12T02:51:59.392Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":642,"ts":"2026-05-12T02:52:00.373Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":687,"ts":"2026-05-12T02:52:00.418Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":7,"latency_ms":795,"ts":"2026-05-12T02:52:00.526Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":728,"ts":"2026-05-12T02:52:01.254Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":662,"ts":"2026-05-12T02:52:01.188Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":7,"latency_ms":906,"ts":"2026-05-12T02:52:01.432Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":624,"ts":"2026-05-12T02:52:02.056Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":682,"ts":"2026-05-12T02:52:02.114Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":7,"latency_ms":672,"ts":"2026-05-12T02:52:02.104Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":711,"ts":"2026-05-12T02:52:02.825Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":671,"ts":"2026-05-12T02:52:02.785Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":9,"latency_ms":751,"ts":"2026-05-12T02:52:02.865Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":670,"ts":"2026-05-12T02:52:03.535Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":649,"ts":"2026-05-12T02:52:03.514Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":6,"latency_ms":827,"ts":"2026-05-12T02:52:03.692Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":682,"ts":"2026-05-12T02:52:04.374Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":760,"ts":"2026-05-12T02:52:04.452Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7630,"output_tokens":4,"latency_ms":691,"ts":"2026-05-12T02:52:04.383Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":642,"ts":"2026-05-12T02:52:05.094Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":698,"ts":"2026-05-12T02:52:05.150Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":9,"latency_ms":632,"ts":"2026-05-12T02:52:05.084Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":665,"ts":"2026-05-12T02:52:05.815Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":850,"ts":"2026-05-12T02:52:06.000Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":7,"latency_ms":636,"ts":"2026-05-12T02:52:05.786Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":949,"ts":"2026-05-12T02:52:06.950Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":803,"ts":"2026-05-12T02:52:06.804Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7633,"output_tokens":5,"latency_ms":747,"ts":"2026-05-12T02:52:06.748Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"calendar-event-create","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":779,"ts":"2026-05-12T02:52:07.729Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":6,"latency_ms":779,"ts":"2026-05-12T02:52:07.729Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":6,"latency_ms":1134,"ts":"2026-05-12T02:52:08.084Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":706,"ts":"2026-05-12T02:52:08.790Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":1453,"ts":"2026-05-12T02:52:09.537Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":5,"latency_ms":882,"ts":"2026-05-12T02:52:08.966Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":810,"ts":"2026-05-12T02:52:10.347Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":1777,"ts":"2026-05-12T02:52:11.314Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7635,"output_tokens":6,"latency_ms":842,"ts":"2026-05-12T02:52:10.379Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":668,"ts":"2026-05-12T02:52:11.982Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":624,"ts":"2026-05-12T02:52:11.938Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7632,"output_tokens":8,"latency_ms":712,"ts":"2026-05-12T02:52:12.026Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":1403,"ts":"2026-05-12T02:52:13.429Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":680,"ts":"2026-05-12T02:52:12.706Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7634,"output_tokens":6,"latency_ms":750,"ts":"2026-05-12T02:52:12.776Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":1018,"ts":"2026-05-12T02:52:14.447Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":4118,"ts":"2026-05-12T02:52:17.547Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":7631,"output_tokens":5,"latency_ms":673,"ts":"2026-05-12T02:52:14.102Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":741,"ts":"2026-05-12T02:52:18.288Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":580,"ts":"2026-05-12T02:52:18.127Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":5,"latency_ms":575,"ts":"2026-05-12T02:52:18.122Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"data-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":6,"latency_ms":573,"ts":"2026-05-12T02:52:18.861Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":9,"latency_ms":579,"ts":"2026-05-12T02:52:18.867Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":9,"latency_ms":579,"ts":"2026-05-12T02:52:18.867Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":598,"ts":"2026-05-12T02:52:19.465Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":556,"ts":"2026-05-12T02:52:19.423Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":561,"ts":"2026-05-12T02:52:19.428Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":631,"ts":"2026-05-12T02:52:20.096Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":602,"ts":"2026-05-12T02:52:20.067Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":610,"ts":"2026-05-12T02:52:20.075Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":571,"ts":"2026-05-12T02:52:20.667Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":696,"ts":"2026-05-12T02:52:20.792Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":628,"ts":"2026-05-12T02:52:20.724Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":743,"ts":"2026-05-12T02:52:21.535Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":612,"ts":"2026-05-12T02:52:21.404Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":630,"ts":"2026-05-12T02:52:21.422Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":1639,"ts":"2026-05-12T02:52:23.174Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"book-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":585,"ts":"2026-05-12T02:52:22.120Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"book-mirror-synthesis","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":8,"latency_ms":656,"ts":"2026-05-12T02:52:22.191Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:23.847Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:23.847Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":6,"latency_ms":529,"ts":"2026-05-12T02:52:23.703Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":704,"ts":"2026-05-12T02:52:24.551Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":535,"ts":"2026-05-12T02:52:24.382Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":705,"ts":"2026-05-12T02:52:24.552Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4093,"output_tokens":9,"latency_ms":707,"ts":"2026-05-12T02:52:25.259Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":815,"ts":"2026-05-12T02:52:26.074Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":697,"ts":"2026-05-12T02:52:25.956Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":7,"latency_ms":689,"ts":"2026-05-12T02:52:25.948Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":587,"ts":"2026-05-12T02:52:26.661Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":572,"ts":"2026-05-12T02:52:26.646Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":7,"latency_ms":1134,"ts":"2026-05-12T02:52:27.208Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"transcript-save","expected":"meeting-ingestion","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":733,"ts":"2026-05-12T02:52:27.941Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"transcript-save","expected":"meeting-ingestion","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":623,"ts":"2026-05-12T02:52:27.831Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":7,"latency_ms":553,"ts":"2026-05-12T02:52:27.761Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":1752,"ts":"2026-05-12T02:52:29.693Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":929,"ts":"2026-05-12T02:52:28.870Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":9,"latency_ms":637,"ts":"2026-05-12T02:52:28.578Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":582,"ts":"2026-05-12T02:52:30.275Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":547,"ts":"2026-05-12T02:52:30.241Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":6,"latency_ms":882,"ts":"2026-05-12T02:52:30.575Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":569,"ts":"2026-05-12T02:52:31.144Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":587,"ts":"2026-05-12T02:52:31.162Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4085,"output_tokens":4,"latency_ms":619,"ts":"2026-05-12T02:52:31.194Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":5,"latency_ms":712,"ts":"2026-05-12T02:52:31.907Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":9,"latency_ms":558,"ts":"2026-05-12T02:52:31.753Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":5,"latency_ms":537,"ts":"2026-05-12T02:52:31.732Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":706,"ts":"2026-05-12T02:52:32.613Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":743,"ts":"2026-05-12T02:52:32.650Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":7,"latency_ms":4370,"ts":"2026-05-12T02:52:36.277Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":587,"ts":"2026-05-12T02:52:36.864Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":624,"ts":"2026-05-12T02:52:36.901Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4088,"output_tokens":5,"latency_ms":634,"ts":"2026-05-12T02:52:36.911Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:52:37.488Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":587,"ts":"2026-05-12T02:52:37.498Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-prep","expected":"daily-task-manager","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":6,"latency_ms":1335,"ts":"2026-05-12T02:52:38.246Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":1277,"ts":"2026-05-12T02:52:39.523Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":560,"ts":"2026-05-12T02:52:38.806Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":5,"latency_ms":735,"ts":"2026-05-12T02:52:38.981Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4090,"output_tokens":6,"latency_ms":666,"ts":"2026-05-12T02:52:40.189Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":668,"ts":"2026-05-12T02:52:40.857Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":689,"ts":"2026-05-12T02:52:40.879Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4087,"output_tokens":8,"latency_ms":555,"ts":"2026-05-12T02:52:40.745Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":730,"ts":"2026-05-12T02:52:41.609Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":729,"ts":"2026-05-12T02:52:41.609Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4089,"output_tokens":6,"latency_ms":603,"ts":"2026-05-12T02:52:41.483Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":699,"ts":"2026-05-12T02:52:42.308Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":567,"ts":"2026-05-12T02:52:42.176Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":4086,"output_tokens":5,"latency_ms":1623,"ts":"2026-05-12T02:52:43.232Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":547,"ts":"2026-05-12T02:52:43.779Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":552,"ts":"2026-05-12T02:52:43.784Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":660,"ts":"2026-05-12T02:52:43.892Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":575,"ts":"2026-05-12T02:52:44.467Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":673,"ts":"2026-05-12T02:52:44.565Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":6,"latency_ms":606,"ts":"2026-05-12T02:52:44.498Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":825,"ts":"2026-05-12T02:52:45.390Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":554,"ts":"2026-05-12T02:52:45.119Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":606,"ts":"2026-05-12T02:52:45.171Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":588,"ts":"2026-05-12T02:52:45.979Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":873,"ts":"2026-05-12T02:52:46.264Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":586,"ts":"2026-05-12T02:52:45.977Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":579,"ts":"2026-05-12T02:52:46.843Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":556,"ts":"2026-05-12T02:52:46.820Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":675,"ts":"2026-05-12T02:52:46.939Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":564,"ts":"2026-05-12T02:52:47.503Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":901,"ts":"2026-05-12T02:52:47.840Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":7,"latency_ms":606,"ts":"2026-05-12T02:52:47.545Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":726,"ts":"2026-05-12T02:52:48.566Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":778,"ts":"2026-05-12T02:52:48.618Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":563,"ts":"2026-05-12T02:52:48.403Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":551,"ts":"2026-05-12T02:52:49.169Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":568,"ts":"2026-05-12T02:52:49.186Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":799,"ts":"2026-05-12T02:52:49.417Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":567,"ts":"2026-05-12T02:52:49.984Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":1347,"ts":"2026-05-12T02:52:50.764Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":662,"ts":"2026-05-12T02:52:50.079Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":572,"ts":"2026-05-12T02:52:51.336Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":587,"ts":"2026-05-12T02:52:51.351Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3213,"output_tokens":6,"latency_ms":578,"ts":"2026-05-12T02:52:51.342Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":624,"ts":"2026-05-12T02:52:51.975Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":560,"ts":"2026-05-12T02:52:51.911Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":615,"ts":"2026-05-12T02:52:51.966Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":4740,"ts":"2026-05-12T02:52:56.715Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":722,"ts":"2026-05-12T02:52:52.698Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":815,"ts":"2026-05-12T02:52:52.791Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":5,"latency_ms":577,"ts":"2026-05-12T02:52:57.292Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":5,"latency_ms":762,"ts":"2026-05-12T02:52:57.477Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":7,"latency_ms":583,"ts":"2026-05-12T02:52:57.298Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":555,"ts":"2026-05-12T02:52:58.032Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":705,"ts":"2026-05-12T02:52:58.182Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":9,"latency_ms":587,"ts":"2026-05-12T02:52:58.064Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":599,"ts":"2026-05-12T02:52:58.781Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:52:58.759Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":793,"ts":"2026-05-12T02:52:58.975Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":836,"ts":"2026-05-12T02:52:59.811Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":2306,"ts":"2026-05-12T02:53:01.281Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3205,"output_tokens":6,"latency_ms":1092,"ts":"2026-05-12T02:53:00.067Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":568,"ts":"2026-05-12T02:53:01.849Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":628,"ts":"2026-05-12T02:53:01.909Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":9,"latency_ms":593,"ts":"2026-05-12T02:53:01.874Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":626,"ts":"2026-05-12T02:53:02.535Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":641,"ts":"2026-05-12T02:53:02.550Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":7,"latency_ms":925,"ts":"2026-05-12T02:53:02.834Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":615,"ts":"2026-05-12T02:53:03.449Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":706,"ts":"2026-05-12T02:53:03.540Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3208,"output_tokens":5,"latency_ms":723,"ts":"2026-05-12T02:53:03.557Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":878,"ts":"2026-05-12T02:53:04.435Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":619,"ts":"2026-05-12T02:53:04.176Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":606,"ts":"2026-05-12T02:53:04.163Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":917,"ts":"2026-05-12T02:53:05.352Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":542,"ts":"2026-05-12T02:53:04.977Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":5,"latency_ms":603,"ts":"2026-05-12T02:53:05.038Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":1164,"ts":"2026-05-12T02:53:06.516Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":629,"ts":"2026-05-12T02:53:05.981Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3210,"output_tokens":6,"latency_ms":609,"ts":"2026-05-12T02:53:05.961Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":655,"ts":"2026-05-12T02:53:07.171Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":649,"ts":"2026-05-12T02:53:07.165Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3207,"output_tokens":8,"latency_ms":691,"ts":"2026-05-12T02:53:07.207Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":550,"ts":"2026-05-12T02:53:07.757Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":577,"ts":"2026-05-12T02:53:07.784Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3209,"output_tokens":6,"latency_ms":1182,"ts":"2026-05-12T02:53:08.389Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":667,"ts":"2026-05-12T02:53:09.056Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":843,"ts":"2026-05-12T02:53:09.232Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-haiku-4-5-20251001","input_tokens":3206,"output_tokens":5,"latency_ms":667,"ts":"2026-05-12T02:53:09.056Z"}
|
||||
@@ -0,0 +1,226 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-opus-4-7","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"ca99fbfeb5f304e1e237eebd11ce0196ea8a9b18","ts":"2026-05-12T03:16:08.329Z","cmd_args":["--model","opus","--parallel","3","--yes","--output","baseline-runs/2026-05-11-opus-4-7.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1844,"ts":"2026-05-12T03:16:10.173Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1703,"ts":"2026-05-12T03:16:10.032Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":7,"latency_ms":1672,"ts":"2026-05-12T03:16:10.001Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"entity-detector","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":9,"latency_ms":4547,"ts":"2026-05-12T03:16:14.720Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":7,"latency_ms":1730,"ts":"2026-05-12T03:16:11.903Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10940,"output_tokens":7,"latency_ms":1766,"ts":"2026-05-12T03:16:11.939Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2035,"ts":"2026-05-12T03:16:16.755Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2849,"ts":"2026-05-12T03:16:17.569Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":10,"latency_ms":2096,"ts":"2026-05-12T03:16:16.816Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1646,"ts":"2026-05-12T03:16:19.215Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1847,"ts":"2026-05-12T03:16:19.416Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":9,"latency_ms":1512,"ts":"2026-05-12T03:16:19.081Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1488,"ts":"2026-05-12T03:16:20.904Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1498,"ts":"2026-05-12T03:16:20.914Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":10,"latency_ms":1584,"ts":"2026-05-12T03:16:21.000Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":2227,"ts":"2026-05-12T03:16:23.227Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":1817,"ts":"2026-05-12T03:16:22.817Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":10,"latency_ms":1446,"ts":"2026-05-12T03:16:22.446Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":2117,"ts":"2026-05-12T03:16:25.345Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:16:24.816Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:16:24.816Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":1653,"ts":"2026-05-12T03:16:26.998Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":2538,"ts":"2026-05-12T03:16:27.883Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10952,"output_tokens":10,"latency_ms":1921,"ts":"2026-05-12T03:16:27.266Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1432,"ts":"2026-05-12T03:16:29.315Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1619,"ts":"2026-05-12T03:16:29.502Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":11,"latency_ms":1910,"ts":"2026-05-12T03:16:29.793Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":1896,"ts":"2026-05-12T03:16:31.689Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":1678,"ts":"2026-05-12T03:16:31.471Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10951,"output_tokens":13,"latency_ms":2108,"ts":"2026-05-12T03:16:31.901Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":1722,"ts":"2026-05-12T03:16:33.623Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":2109,"ts":"2026-05-12T03:16:34.010Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10948,"output_tokens":9,"latency_ms":4048,"ts":"2026-05-12T03:16:35.949Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1622,"ts":"2026-05-12T03:16:37.572Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1522,"ts":"2026-05-12T03:16:37.472Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":2637,"ts":"2026-05-12T03:16:38.587Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1752,"ts":"2026-05-12T03:16:40.339Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1603,"ts":"2026-05-12T03:16:40.190Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":12,"latency_ms":1575,"ts":"2026-05-12T03:16:40.162Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":2603,"ts":"2026-05-12T03:16:42.942Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":1578,"ts":"2026-05-12T03:16:41.917Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10944,"output_tokens":12,"latency_ms":1692,"ts":"2026-05-12T03:16:42.031Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1674,"ts":"2026-05-12T03:16:44.616Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":1955,"ts":"2026-05-12T03:16:44.897Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":9,"latency_ms":2103,"ts":"2026-05-12T03:16:45.045Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":7,"latency_ms":2048,"ts":"2026-05-12T03:16:47.093Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":7,"latency_ms":2522,"ts":"2026-05-12T03:16:47.567Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10943,"output_tokens":10,"latency_ms":1825,"ts":"2026-05-12T03:16:46.870Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1734,"ts":"2026-05-12T03:16:49.301Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1666,"ts":"2026-05-12T03:16:49.234Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":10,"latency_ms":1694,"ts":"2026-05-12T03:16:49.261Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1531,"ts":"2026-05-12T03:16:50.832Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1615,"ts":"2026-05-12T03:16:50.916Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":9,"latency_ms":1503,"ts":"2026-05-12T03:16:50.804Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1960,"ts":"2026-05-12T03:16:52.876Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1469,"ts":"2026-05-12T03:16:52.385Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10945,"output_tokens":7,"latency_ms":1686,"ts":"2026-05-12T03:16:52.602Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":3911,"ts":"2026-05-12T03:16:56.787Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1684,"ts":"2026-05-12T03:16:54.560Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":2066,"ts":"2026-05-12T03:16:54.942Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":1576,"ts":"2026-05-12T03:16:58.363Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":1634,"ts":"2026-05-12T03:16:58.421Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10950,"output_tokens":7,"latency_ms":2666,"ts":"2026-05-12T03:16:59.453Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":1933,"ts":"2026-05-12T03:17:01.386Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":2022,"ts":"2026-05-12T03:17:01.475Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":9,"latency_ms":1881,"ts":"2026-05-12T03:17:01.334Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":2322,"ts":"2026-05-12T03:17:03.797Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1639,"ts":"2026-05-12T03:17:03.114Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10947,"output_tokens":11,"latency_ms":1854,"ts":"2026-05-12T03:17:03.329Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1694,"ts":"2026-05-12T03:17:05.491Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1621,"ts":"2026-05-12T03:17:05.418Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10946,"output_tokens":9,"latency_ms":1493,"ts":"2026-05-12T03:17:05.292Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1662,"ts":"2026-05-12T03:17:07.153Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1736,"ts":"2026-05-12T03:17:07.227Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":10942,"output_tokens":9,"latency_ms":1538,"ts":"2026-05-12T03:17:07.030Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1569,"ts":"2026-05-12T03:17:08.796Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1569,"ts":"2026-05-12T03:17:08.796Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":7,"latency_ms":1746,"ts":"2026-05-12T03:17:08.973Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1428,"ts":"2026-05-12T03:17:10.401Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1414,"ts":"2026-05-12T03:17:10.387Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5997,"output_tokens":5,"latency_ms":1903,"ts":"2026-05-12T03:17:10.876Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1603,"ts":"2026-05-12T03:17:12.479Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1551,"ts":"2026-05-12T03:17:12.428Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":10,"latency_ms":1738,"ts":"2026-05-12T03:17:12.615Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":4130,"ts":"2026-05-12T03:17:16.745Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":1735,"ts":"2026-05-12T03:17:14.351Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":9,"latency_ms":1704,"ts":"2026-05-12T03:17:14.320Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":3997,"ts":"2026-05-12T03:17:20.742Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":1578,"ts":"2026-05-12T03:17:18.323Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":10,"latency_ms":1617,"ts":"2026-05-12T03:17:18.362Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1656,"ts":"2026-05-12T03:17:22.398Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1652,"ts":"2026-05-12T03:17:22.394Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":10,"latency_ms":1575,"ts":"2026-05-12T03:17:22.317Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":2173,"ts":"2026-05-12T03:17:24.571Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":1848,"ts":"2026-05-12T03:17:24.246Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":9,"latency_ms":1678,"ts":"2026-05-12T03:17:24.076Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1362,"ts":"2026-05-12T03:17:25.933Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1747,"ts":"2026-05-12T03:17:26.318Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6009,"output_tokens":10,"latency_ms":1747,"ts":"2026-05-12T03:17:26.318Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":2525,"ts":"2026-05-12T03:17:28.843Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":1404,"ts":"2026-05-12T03:17:27.722Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":11,"latency_ms":1603,"ts":"2026-05-12T03:17:27.921Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":3273,"ts":"2026-05-12T03:17:32.116Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":2071,"ts":"2026-05-12T03:17:30.914Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6008,"output_tokens":13,"latency_ms":1820,"ts":"2026-05-12T03:17:30.663Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1443,"ts":"2026-05-12T03:17:33.559Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1549,"ts":"2026-05-12T03:17:33.665Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6005,"output_tokens":10,"latency_ms":1563,"ts":"2026-05-12T03:17:33.679Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1824,"ts":"2026-05-12T03:17:35.503Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1533,"ts":"2026-05-12T03:17:35.212Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1365,"ts":"2026-05-12T03:17:35.044Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1511,"ts":"2026-05-12T03:17:37.014Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1880,"ts":"2026-05-12T03:17:37.383Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":12,"latency_ms":1602,"ts":"2026-05-12T03:17:37.105Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":1496,"ts":"2026-05-12T03:17:38.879Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":1470,"ts":"2026-05-12T03:17:38.853Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6001,"output_tokens":12,"latency_ms":2355,"ts":"2026-05-12T03:17:39.738Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1703,"ts":"2026-05-12T03:17:41.441Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1598,"ts":"2026-05-12T03:17:41.337Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":9,"latency_ms":1574,"ts":"2026-05-12T03:17:41.313Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1674,"ts":"2026-05-12T03:17:43.115Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1755,"ts":"2026-05-12T03:17:43.197Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6000,"output_tokens":7,"latency_ms":1830,"ts":"2026-05-12T03:17:43.271Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":10,"latency_ms":1478,"ts":"2026-05-12T03:17:44.750Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":10,"latency_ms":2431,"ts":"2026-05-12T03:17:45.702Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"exa","expected":"perplexity-research","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":6,"latency_ms":1496,"ts":"2026-05-12T03:17:44.767Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1883,"ts":"2026-05-12T03:17:47.585Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1445,"ts":"2026-05-12T03:17:47.147Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":9,"latency_ms":1597,"ts":"2026-05-12T03:17:47.299Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":1448,"ts":"2026-05-12T03:17:49.033Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":2841,"ts":"2026-05-12T03:17:50.426Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6002,"output_tokens":7,"latency_ms":1414,"ts":"2026-05-12T03:17:48.999Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1393,"ts":"2026-05-12T03:17:51.819Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1478,"ts":"2026-05-12T03:17:51.904Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1625,"ts":"2026-05-12T03:17:52.051Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1694,"ts":"2026-05-12T03:17:53.745Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1694,"ts":"2026-05-12T03:17:53.745Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6007,"output_tokens":7,"latency_ms":1720,"ts":"2026-05-12T03:17:53.771Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1740,"ts":"2026-05-12T03:17:55.511Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1920,"ts":"2026-05-12T03:17:55.691Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":9,"latency_ms":1563,"ts":"2026-05-12T03:17:55.334Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":2563,"ts":"2026-05-12T03:17:58.255Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1528,"ts":"2026-05-12T03:17:57.220Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6004,"output_tokens":11,"latency_ms":1528,"ts":"2026-05-12T03:17:57.220Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":9,"latency_ms":1585,"ts":"2026-05-12T03:17:59.840Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"google-contacts","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":7,"latency_ms":1570,"ts":"2026-05-12T03:17:59.825Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"google-contacts","correct":0,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":6003,"output_tokens":7,"latency_ms":1909,"ts":"2026-05-12T03:18:00.164Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1376,"ts":"2026-05-12T03:18:01.540Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1389,"ts":"2026-05-12T03:18:01.553Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":5999,"output_tokens":9,"latency_ms":1398,"ts":"2026-05-12T03:18:01.562Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1527,"ts":"2026-05-12T03:18:03.089Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1573,"ts":"2026-05-12T03:18:03.135Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":9,"latency_ms":1516,"ts":"2026-05-12T03:18:03.078Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1606,"ts":"2026-05-12T03:18:04.741Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1689,"ts":"2026-05-12T03:18:04.824Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4724,"output_tokens":10,"latency_ms":1682,"ts":"2026-05-12T03:18:04.817Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1670,"ts":"2026-05-12T03:18:06.494Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":2416,"ts":"2026-05-12T03:18:07.240Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1489,"ts":"2026-05-12T03:18:06.313Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":3205,"ts":"2026-05-12T03:18:10.445Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":4901,"ts":"2026-05-12T03:18:12.141Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":9,"latency_ms":4556,"ts":"2026-05-12T03:18:11.796Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":1779,"ts":"2026-05-12T03:18:13.921Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":1782,"ts":"2026-05-12T03:18:13.924Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":10,"latency_ms":2264,"ts":"2026-05-12T03:18:14.406Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1905,"ts":"2026-05-12T03:18:16.311Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1512,"ts":"2026-05-12T03:18:15.918Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":10,"latency_ms":1535,"ts":"2026-05-12T03:18:15.941Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1430,"ts":"2026-05-12T03:18:17.741Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1933,"ts":"2026-05-12T03:18:18.244Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":9,"latency_ms":1902,"ts":"2026-05-12T03:18:18.213Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1602,"ts":"2026-05-12T03:18:19.846Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1606,"ts":"2026-05-12T03:18:19.850Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4736,"output_tokens":10,"latency_ms":1786,"ts":"2026-05-12T03:18:20.030Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"concept-synthesis","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":9,"latency_ms":1583,"ts":"2026-05-12T03:18:21.613Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"concept-synthesis","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":9,"latency_ms":1412,"ts":"2026-05-12T03:18:21.442Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":11,"latency_ms":1521,"ts":"2026-05-12T03:18:21.551Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1538,"ts":"2026-05-12T03:18:23.151Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1534,"ts":"2026-05-12T03:18:23.148Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4735,"output_tokens":10,"latency_ms":1375,"ts":"2026-05-12T03:18:22.989Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":2125,"ts":"2026-05-12T03:18:25.276Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":2337,"ts":"2026-05-12T03:18:25.488Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4732,"output_tokens":7,"latency_ms":1945,"ts":"2026-05-12T03:18:25.096Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1453,"ts":"2026-05-12T03:18:26.941Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1398,"ts":"2026-05-12T03:18:26.886Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":7,"latency_ms":1295,"ts":"2026-05-12T03:18:26.783Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1249,"ts":"2026-05-12T03:18:28.190Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1502,"ts":"2026-05-12T03:18:28.443Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":12,"latency_ms":1721,"ts":"2026-05-12T03:18:28.662Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1577,"ts":"2026-05-12T03:18:30.240Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1894,"ts":"2026-05-12T03:18:30.557Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4728,"output_tokens":12,"latency_ms":1350,"ts":"2026-05-12T03:18:30.013Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1974,"ts":"2026-05-12T03:18:32.531Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":1650,"ts":"2026-05-12T03:18:32.207Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":9,"latency_ms":5071,"ts":"2026-05-12T03:18:35.628Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1363,"ts":"2026-05-12T03:18:36.991Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1978,"ts":"2026-05-12T03:18:37.606Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4727,"output_tokens":10,"latency_ms":1567,"ts":"2026-05-12T03:18:37.195Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":1639,"ts":"2026-05-12T03:18:39.245Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":1780,"ts":"2026-05-12T03:18:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":10,"latency_ms":2166,"ts":"2026-05-12T03:18:39.772Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":1785,"ts":"2026-05-12T03:18:41.557Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":1546,"ts":"2026-05-12T03:18:41.318Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":9,"latency_ms":2157,"ts":"2026-05-12T03:18:41.929Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1536,"ts":"2026-05-12T03:18:43.465Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1624,"ts":"2026-05-12T03:18:43.553Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4729,"output_tokens":7,"latency_ms":1452,"ts":"2026-05-12T03:18:43.381Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1862,"ts":"2026-05-12T03:18:45.415Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1922,"ts":"2026-05-12T03:18:45.475Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1589,"ts":"2026-05-12T03:18:45.142Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":3982,"ts":"2026-05-12T03:18:49.458Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":1555,"ts":"2026-05-12T03:18:47.030Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4734,"output_tokens":7,"latency_ms":1221,"ts":"2026-05-12T03:18:46.696Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1319,"ts":"2026-05-12T03:18:50.777Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1587,"ts":"2026-05-12T03:18:51.045Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":9,"latency_ms":1399,"ts":"2026-05-12T03:18:50.857Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1862,"ts":"2026-05-12T03:18:52.907Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1773,"ts":"2026-05-12T03:18:52.818Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4731,"output_tokens":11,"latency_ms":1525,"ts":"2026-05-12T03:18:52.570Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":9556,"ts":"2026-05-12T03:19:02.463Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":2096,"ts":"2026-05-12T03:18:55.003Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4730,"output_tokens":9,"latency_ms":1919,"ts":"2026-05-12T03:18:54.826Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1639,"ts":"2026-05-12T03:19:04.102Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1731,"ts":"2026-05-12T03:19:04.194Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-opus-4-7","input_tokens":4726,"output_tokens":9,"latency_ms":1741,"ts":"2026-05-12T03:19:04.204Z"}
|
||||
@@ -0,0 +1,226 @@
|
||||
{"kind":"receipt","model":"anthropic:claude-sonnet-4-6","prompt_template_hash":"17340040af579ca1","fixtures_hash":"feccc99122ea86d5","fixtures_held_out_hash":"5d6256cc9dced124","harness_sha":"fcc395282a92f2b047d4407f2b5a891c069adaac","ts":"2026-05-12T02:49:32.050Z","cmd_args":["--model","sonnet","--parallel","3","--yes","--output","run-sonnet-4-6.jsonl"]}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":2307,"ts":"2026-05-12T02:49:34.357Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":1033,"ts":"2026-05-12T02:49:33.083Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"baseline","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":5,"latency_ms":1682,"ts":"2026-05-12T02:49:33.732Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":1,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":2141,"ts":"2026-05-12T02:49:36.498Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":2,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1435,"ts":"2026-05-12T02:49:35.792Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"baseline","seed":3,"predicted":"gbrain","expected":"gbrain","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1121,"ts":"2026-05-12T02:49:35.478Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1034,"ts":"2026-05-12T02:49:37.532Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1285,"ts":"2026-05-12T02:49:37.783Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1537,"ts":"2026-05-12T02:49:38.035Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1052,"ts":"2026-05-12T02:49:39.087Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1069,"ts":"2026-05-12T02:49:39.104Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1260,"ts":"2026-05-12T02:49:39.295Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":1,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1413,"ts":"2026-05-12T02:49:40.708Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":2,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1400,"ts":"2026-05-12T02:49:40.695Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"baseline","seed":3,"predicted":"brain-librarian","expected":"brain-librarian","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1369,"ts":"2026-05-12T02:49:40.664Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1072,"ts":"2026-05-12T02:49:41.780Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":2004,"ts":"2026-05-12T02:49:42.712Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"baseline","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1221,"ts":"2026-05-12T02:49:41.929Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":1,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1108,"ts":"2026-05-12T02:49:43.820Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":2,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1380,"ts":"2026-05-12T02:49:44.092Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"baseline","seed":3,"predicted":"book-mirror","expected":"book-mirror","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1629,"ts":"2026-05-12T02:49:44.341Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1204,"ts":"2026-05-12T02:49:45.545Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1128,"ts":"2026-05-12T02:49:45.469Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"baseline","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":6,"latency_ms":1106,"ts":"2026-05-12T02:49:45.447Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1241,"ts":"2026-05-12T02:49:46.786Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":948,"ts":"2026-05-12T02:49:46.493Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"baseline","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1028,"ts":"2026-05-12T02:49:46.573Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1723,"ts":"2026-05-12T02:49:48.509Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1592,"ts":"2026-05-12T02:49:48.378Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"baseline","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7638,"output_tokens":9,"latency_ms":1293,"ts":"2026-05-12T02:49:48.079Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1376,"ts":"2026-05-12T02:49:49.885Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1691,"ts":"2026-05-12T02:49:50.201Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"baseline","seed":3,"predicted":"idea-ingest","expected":"idea-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":7,"latency_ms":1512,"ts":"2026-05-12T02:49:50.021Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"baseline","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":7,"latency_ms":1593,"ts":"2026-05-12T02:49:51.794Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1522,"ts":"2026-05-12T02:49:53.316Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1395,"ts":"2026-05-12T02:49:53.189Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"baseline","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":7,"latency_ms":1030,"ts":"2026-05-12T02:49:52.824Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1670,"ts":"2026-05-12T02:49:54.987Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1536,"ts":"2026-05-12T02:49:54.853Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"baseline","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":9,"latency_ms":1319,"ts":"2026-05-12T02:49:54.636Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":1031,"ts":"2026-05-12T02:49:56.018Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":947,"ts":"2026-05-12T02:49:55.934Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"baseline","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":6,"latency_ms":973,"ts":"2026-05-12T02:49:55.960Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1539,"ts":"2026-05-12T02:49:57.557Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1660,"ts":"2026-05-12T02:49:57.678Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"baseline","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":7630,"output_tokens":4,"latency_ms":1550,"ts":"2026-05-12T02:49:57.568Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1276,"ts":"2026-05-12T02:49:58.954Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1273,"ts":"2026-05-12T02:49:58.951Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"baseline","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":9,"latency_ms":1624,"ts":"2026-05-12T02:49:59.302Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1264,"ts":"2026-05-12T02:50:00.566Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1518,"ts":"2026-05-12T02:50:00.820Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"baseline","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":7,"latency_ms":1537,"ts":"2026-05-12T02:50:00.839Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1296,"ts":"2026-05-12T02:50:02.135Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1216,"ts":"2026-05-12T02:50:02.055Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"baseline","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7633,"output_tokens":5,"latency_ms":1509,"ts":"2026-05-12T02:50:02.348Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1240,"ts":"2026-05-12T02:50:03.588Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1908,"ts":"2026-05-12T02:50:04.256Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"baseline","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1303,"ts":"2026-05-12T02:50:03.651Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1022,"ts":"2026-05-12T02:50:05.278Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1553,"ts":"2026-05-12T02:50:05.809Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":5,"latency_ms":1199,"ts":"2026-05-12T02:50:05.455Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1439,"ts":"2026-05-12T02:50:07.248Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1062,"ts":"2026-05-12T02:50:06.871Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7635,"output_tokens":6,"latency_ms":1062,"ts":"2026-05-12T02:50:06.871Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1078,"ts":"2026-05-12T02:50:08.326Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":961,"ts":"2026-05-12T02:50:08.209Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7632,"output_tokens":8,"latency_ms":1078,"ts":"2026-05-12T02:50:08.326Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1042,"ts":"2026-05-12T02:50:09.368Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1183,"ts":"2026-05-12T02:50:09.509Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7634,"output_tokens":6,"latency_ms":1052,"ts":"2026-05-12T02:50:09.378Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1246,"ts":"2026-05-12T02:50:10.755Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1596,"ts":"2026-05-12T02:50:11.105Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"baseline","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":7631,"output_tokens":5,"latency_ms":1708,"ts":"2026-05-12T02:50:11.217Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1686,"ts":"2026-05-12T02:50:12.903Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1085,"ts":"2026-05-12T02:50:12.302Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"enrich","expected":"enrich","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":5,"latency_ms":1242,"ts":"2026-05-12T02:50:12.459Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":1209,"ts":"2026-05-12T02:50:14.112Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":1372,"ts":"2026-05-12T02:50:14.275Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"query","expected":"gbrain","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":4,"latency_ms":992,"ts":"2026-05-12T02:50:13.895Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1214,"ts":"2026-05-12T02:50:15.489Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1055,"ts":"2026-05-12T02:50:15.330Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-pdf","expected":"brain-pdf","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1263,"ts":"2026-05-12T02:50:15.538Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1007,"ts":"2026-05-12T02:50:16.545Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1241,"ts":"2026-05-12T02:50:16.779Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"brain-publish","expected":"brain-publish","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1596,"ts":"2026-05-12T02:50:17.134Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"freshness-monitor","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1887,"ts":"2026-05-12T02:50:19.021Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"freshness-monitor","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1330,"ts":"2026-05-12T02:50:18.464Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"benchmark-gbrain","expected":"brain-librarian","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1777,"ts":"2026-05-12T02:50:18.911Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1016,"ts":"2026-05-12T02:50:20.037Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1700,"ts":"2026-05-12T02:50:20.721Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1524,"ts":"2026-05-12T02:50:20.545Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":5907,"ts":"2026-05-12T02:50:26.628Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1594,"ts":"2026-05-12T02:50:22.315Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"book-mirror","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1251,"ts":"2026-05-12T02:50:21.972Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1258,"ts":"2026-05-12T02:50:27.886Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1228,"ts":"2026-05-12T02:50:27.856Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"strategic-reading","expected":"strategic-reading","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":6,"latency_ms":1253,"ts":"2026-05-12T02:50:27.881Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1218,"ts":"2026-05-12T02:50:29.105Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":941,"ts":"2026-05-12T02:50:28.828Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":978,"ts":"2026-05-12T02:50:28.865Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1408,"ts":"2026-05-12T02:50:30.513Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1391,"ts":"2026-05-12T02:50:30.496Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"dropbox-archive-review","expected":"archive-crawler","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4093,"output_tokens":9,"latency_ms":1480,"ts":"2026-05-12T02:50:30.585Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":937,"ts":"2026-05-12T02:50:31.522Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":869,"ts":"2026-05-12T02:50:31.454Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"article-enrichment","expected":"idea-ingest","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":7,"latency_ms":1021,"ts":"2026-05-12T02:50:31.606Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":871,"ts":"2026-05-12T02:50:32.477Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1044,"ts":"2026-05-12T02:50:32.650Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"media-ingest","expected":"media-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":7,"latency_ms":1108,"ts":"2026-05-12T02:50:32.714Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":982,"ts":"2026-05-12T02:50:33.696Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":1058,"ts":"2026-05-12T02:50:33.772Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"meeting-ingestion","expected":"meeting-ingestion","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":7,"latency_ms":957,"ts":"2026-05-12T02:50:33.671Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":984,"ts":"2026-05-12T02:50:34.756Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":1450,"ts":"2026-05-12T02:50:35.222Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"voice-note-ingest","expected":"voice-note-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":9,"latency_ms":1341,"ts":"2026-05-12T02:50:35.113Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1502,"ts":"2026-05-12T02:50:36.724Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1326,"ts":"2026-05-12T02:50:36.548Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"calendar-check","expected":"google-calendar","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":6,"latency_ms":1326,"ts":"2026-05-12T02:50:36.548Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1060,"ts":"2026-05-12T02:50:37.784Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1600,"ts":"2026-05-12T02:50:38.324Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"gmail","expected":"executive-assistant","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4085,"output_tokens":4,"latency_ms":1415,"ts":"2026-05-12T02:50:38.139Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":1062,"ts":"2026-05-12T02:50:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":1062,"ts":"2026-05-12T02:50:39.386Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":9,"latency_ms":971,"ts":"2026-05-12T02:50:39.295Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":1634,"ts":"2026-05-12T02:50:41.020Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":999,"ts":"2026-05-12T02:50:40.386Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":7,"latency_ms":890,"ts":"2026-05-12T02:50:40.277Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":1141,"ts":"2026-05-12T02:50:42.161Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":938,"ts":"2026-05-12T02:50:41.958Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4088,"output_tokens":5,"latency_ms":878,"ts":"2026-05-12T02:50:41.898Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1185,"ts":"2026-05-12T02:50:43.347Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":990,"ts":"2026-05-12T02:50:43.151Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-manager","correct":0,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":954,"ts":"2026-05-12T02:50:43.115Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":1043,"ts":"2026-05-12T02:50:44.390Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":1011,"ts":"2026-05-12T02:50:44.358Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":5,"latency_ms":922,"ts":"2026-05-12T02:50:44.269Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":1193,"ts":"2026-05-12T02:50:45.583Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":1196,"ts":"2026-05-12T02:50:45.586Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"skill-creator","expected":"skill-creator","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4090,"output_tokens":6,"latency_ms":5248,"ts":"2026-05-12T02:50:49.638Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1257,"ts":"2026-05-12T02:50:50.895Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1487,"ts":"2026-05-12T02:50:51.125Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4087,"output_tokens":8,"latency_ms":1100,"ts":"2026-05-12T02:50:50.738Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1413,"ts":"2026-05-12T02:50:52.538Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1236,"ts":"2026-05-12T02:50:52.361Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4089,"output_tokens":6,"latency_ms":1590,"ts":"2026-05-12T02:50:52.715Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1452,"ts":"2026-05-12T02:50:54.167Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1202,"ts":"2026-05-12T02:50:53.917Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"functional-areas","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":4086,"output_tokens":5,"latency_ms":1452,"ts":"2026-05-12T02:50:54.167Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1003,"ts":"2026-05-12T02:50:55.170Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":987,"ts":"2026-05-12T02:50:55.154Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"enrich","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1296,"ts":"2026-05-12T02:50:55.463Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":1360,"ts":"2026-05-12T02:50:56.824Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":1029,"ts":"2026-05-12T02:50:56.493Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"gbrain","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":6,"latency_ms":2308,"ts":"2026-05-12T02:50:57.772Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":2125,"ts":"2026-05-12T02:50:59.897Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":1538,"ts":"2026-05-12T02:50:59.310Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-pdf-auto","expected":"brain-pdf","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":8,"latency_ms":1188,"ts":"2026-05-12T02:50:58.960Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":826,"ts":"2026-05-12T02:51:00.723Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1031,"ts":"2026-05-12T02:51:00.928Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-publish","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":949,"ts":"2026-05-12T02:51:00.846Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1494,"ts":"2026-05-12T02:51:02.422Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1311,"ts":"2026-05-12T02:51:02.239Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"brain-librarian","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1253,"ts":"2026-05-12T02:51:02.181Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":1047,"ts":"2026-05-12T02:51:03.469Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":1143,"ts":"2026-05-12T02:51:03.565Z"}
|
||||
{"kind":"run","fixture_id":5,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"citation-fixer","expected":"citation-fixer","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":7,"latency_ms":919,"ts":"2026-05-12T02:51:03.341Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1301,"ts":"2026-05-12T02:51:04.866Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1159,"ts":"2026-05-12T02:51:04.724Z"}
|
||||
{"kind":"run","fixture_id":6,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"book-mirror","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1417,"ts":"2026-05-12T02:51:04.982Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1209,"ts":"2026-05-12T02:51:06.191Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1609,"ts":"2026-05-12T02:51:06.591Z"}
|
||||
{"kind":"run","fixture_id":7,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"strategic-reading","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1432,"ts":"2026-05-12T02:51:06.414Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1765,"ts":"2026-05-12T02:51:08.356Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":3599,"ts":"2026-05-12T02:51:10.190Z"}
|
||||
{"kind":"run","fixture_id":8,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"concept-synthesis","expected":"concept-synthesis","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":1060,"ts":"2026-05-12T02:51:07.651Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"archive-crawler","expected":"archive-crawler","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1213,"ts":"2026-05-12T02:51:11.403Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"brain-ops","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":878,"ts":"2026-05-12T02:51:11.068Z"}
|
||||
{"kind":"run","fixture_id":9,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"brain-ops","expected":"archive-crawler","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3213,"output_tokens":6,"latency_ms":1040,"ts":"2026-05-12T02:51:11.230Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":1542,"ts":"2026-05-12T02:51:12.945Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":930,"ts":"2026-05-12T02:51:12.333Z"}
|
||||
{"kind":"run","fixture_id":10,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"idea-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":971,"ts":"2026-05-12T02:51:12.374Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1203,"ts":"2026-05-12T02:51:14.148Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1513,"ts":"2026-05-12T02:51:14.458Z"}
|
||||
{"kind":"run","fixture_id":11,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"media-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1342,"ts":"2026-05-12T02:51:14.287Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":4435,"ts":"2026-05-12T02:51:18.893Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":1355,"ts":"2026-05-12T02:51:15.813Z"}
|
||||
{"kind":"run","fixture_id":12,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"meeting-ingestion","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":5,"latency_ms":978,"ts":"2026-05-12T02:51:15.436Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1026,"ts":"2026-05-12T02:51:19.919Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1323,"ts":"2026-05-12T02:51:20.216Z"}
|
||||
{"kind":"run","fixture_id":13,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"ingest","expected":"voice-note-ingest","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":5,"latency_ms":1372,"ts":"2026-05-12T02:51:20.265Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1295,"ts":"2026-05-12T02:51:21.560Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":2501,"ts":"2026-05-12T02:51:22.766Z"}
|
||||
{"kind":"run","fixture_id":14,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"google-calendar","expected":"google-calendar","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1110,"ts":"2026-05-12T02:51:21.375Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1067,"ts":"2026-05-12T02:51:23.833Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1059,"ts":"2026-05-12T02:51:23.825Z"}
|
||||
{"kind":"run","fixture_id":15,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"executive-assistant","expected":"executive-assistant","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3205,"output_tokens":6,"latency_ms":1237,"ts":"2026-05-12T02:51:24.003Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1122,"ts":"2026-05-12T02:51:25.125Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1537,"ts":"2026-05-12T02:51:25.540Z"}
|
||||
{"kind":"run","fixture_id":16,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"perplexity-research","expected":"perplexity-research","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":9,"latency_ms":1755,"ts":"2026-05-12T02:51:25.758Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":900,"ts":"2026-05-12T02:51:26.658Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":852,"ts":"2026-05-12T02:51:26.610Z"}
|
||||
{"kind":"run","fixture_id":17,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"x-ingest","expected":"x-ingest","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":7,"latency_ms":1438,"ts":"2026-05-12T02:51:27.196Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1174,"ts":"2026-05-12T02:51:28.370Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1347,"ts":"2026-05-12T02:51:28.543Z"}
|
||||
{"kind":"run","fixture_id":18,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"checkin","expected":"checkin","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3208,"output_tokens":5,"latency_ms":1098,"ts":"2026-05-12T02:51:28.294Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1455,"ts":"2026-05-12T02:51:29.998Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":990,"ts":"2026-05-12T02:51:29.533Z"}
|
||||
{"kind":"run","fixture_id":19,"corpus":"training","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-manager","expected":"daily-task-manager","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1801,"ts":"2026-05-12T02:51:30.344Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":2027,"ts":"2026-05-12T02:51:32.371Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":1009,"ts":"2026-05-12T02:51:31.353Z"}
|
||||
{"kind":"run","fixture_id":0,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"skillify","expected":"skillify","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":5,"latency_ms":950,"ts":"2026-05-12T02:51:31.294Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":1008,"ts":"2026-05-12T02:51:33.379Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":931,"ts":"2026-05-12T02:51:33.302Z"}
|
||||
{"kind":"run","fixture_id":1,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"acp-coding","expected":"skill-creator","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3210,"output_tokens":7,"latency_ms":1036,"ts":"2026-05-12T02:51:33.407Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"daily-task-manager","expected":"daily-task-prep","correct":0,"correct_lenient":0,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":901,"ts":"2026-05-12T02:51:34.308Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":892,"ts":"2026-05-12T02:51:34.299Z"}
|
||||
{"kind":"run","fixture_id":2,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"daily-task-prep","expected":"daily-task-prep","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3207,"output_tokens":8,"latency_ms":1016,"ts":"2026-05-12T02:51:34.423Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":879,"ts":"2026-05-12T02:51:35.302Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":948,"ts":"2026-05-12T02:51:35.371Z"}
|
||||
{"kind":"run","fixture_id":3,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"google-contacts","expected":"google-contacts","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3209,"output_tokens":6,"latency_ms":930,"ts":"2026-05-12T02:51:35.353Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":1,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1022,"ts":"2026-05-12T02:51:36.393Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":2,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":1406,"ts":"2026-05-12T02:51:36.777Z"}
|
||||
{"kind":"run","fixture_id":4,"corpus":"held_out","variant":"resolver-of-resolvers","seed":3,"predicted":"healthcheck","expected":"healthcheck","correct":1,"correct_lenient":1,"model":"anthropic:claude-sonnet-4-6","input_tokens":3206,"output_tokens":5,"latency_ms":907,"ts":"2026-05-12T02:51:36.278Z"}
|
||||
8
evals/functional-area-resolver/fixtures-held-out.jsonl
Normal file
8
evals/functional-area-resolver/fixtures-held-out.jsonl
Normal file
@@ -0,0 +1,8 @@
|
||||
// 5 held-out blind fixtures. Authored before the variant resolvers were
|
||||
// fully reviewed; target skills present in both real variants.
|
||||
// Held-out accuracy is the headline claim in skills/functional-area-resolver/SKILL.md.
|
||||
{"intent":"Skillify the JSON parsing helper I wrote last week","expected_skill":"skillify"}
|
||||
{"intent":"Create a new skill for cataloging books I've finished","expected_skill":"skill-creator"}
|
||||
{"intent":"Build me a daily prep summary for tomorrow","expected_skill":"daily-task-prep"}
|
||||
{"intent":"Pull the contact details for Maria from my address book","expected_skill":"google-contacts"}
|
||||
{"intent":"Run a healthcheck on my services","expected_skill":"healthcheck"}
|
||||
24
evals/functional-area-resolver/fixtures.jsonl
Normal file
24
evals/functional-area-resolver/fixtures.jsonl
Normal file
@@ -0,0 +1,24 @@
|
||||
// 20 training fixtures for the functional-area-resolver A/B eval.
|
||||
// Each line: {"intent": "<user phrasing>", "expected_skill": "<skill slug>"}
|
||||
// Target skills are present in BOTH variants (verified against the
|
||||
// real production AGENTS.md at git commit 93848ff3b^ and 93848ff3b).
|
||||
{"intent":"Create a person page for John Smith and enrich it from his GitHub","expected_skill":"enrich"}
|
||||
{"intent":"What do we know about Stripe","expected_skill":"gbrain"}
|
||||
{"intent":"Make a PDF from my brain page on dispatcher patterns","expected_skill":"brain-pdf"}
|
||||
{"intent":"Publish this brain page as a shareable link","expected_skill":"brain-publish"}
|
||||
{"intent":"Run brain integrity — what's lost in my archive","expected_skill":"brain-librarian"}
|
||||
{"intent":"Fix the broken citations on this page","expected_skill":"citation-fixer"}
|
||||
{"intent":"Make a personalized version of Atomic Habits with my brain context","expected_skill":"book-mirror"}
|
||||
{"intent":"Read Thinking Fast and Slow through the lens of my product work","expected_skill":"strategic-reading"}
|
||||
{"intent":"Synthesize my concepts about resolver design and routing","expected_skill":"concept-synthesis"}
|
||||
{"intent":"Crawl my dropbox archive for old notes I should pull in","expected_skill":"archive-crawler"}
|
||||
{"intent":"Ingest this article from The Atlantic into my brain","expected_skill":"idea-ingest"}
|
||||
{"intent":"Process this YouTube video into the brain","expected_skill":"media-ingest"}
|
||||
{"intent":"I have a meeting transcript to file from this morning","expected_skill":"meeting-ingestion"}
|
||||
{"intent":"Save this voice memo and transcribe it","expected_skill":"voice-note-ingest"}
|
||||
{"intent":"What's on my calendar tomorrow","expected_skill":"google-calendar"}
|
||||
{"intent":"Draft a reply email to Sarah","expected_skill":"executive-assistant"}
|
||||
{"intent":"Research what's new about WebGPU adoption","expected_skill":"perplexity-research"}
|
||||
{"intent":"Pull my recent X posts and ingest them","expected_skill":"x-ingest"}
|
||||
{"intent":"Check me into the coffee shop I'm at","expected_skill":"checkin"}
|
||||
{"intent":"Add a task for tomorrow's meeting prep","expected_skill":"daily-task-manager"}
|
||||
302
evals/functional-area-resolver/harness-runner.test.ts
Normal file
302
evals/functional-area-resolver/harness-runner.test.ts
Normal file
@@ -0,0 +1,302 @@
|
||||
/**
|
||||
* Unit tests for the functional-area-resolver A/B eval harness.
|
||||
* Run with: bun test evals/functional-area-resolver/harness-runner.test.ts
|
||||
*
|
||||
* Covers every pure function so contributors can debug without spending
|
||||
* money on every iteration. main() smoke test is omitted in this slice
|
||||
* (it would require mocking gateway transport + filesystem; the harness's
|
||||
* --limit 1 mode is a sufficient real smoke check at ~$0.01 per run).
|
||||
*/
|
||||
|
||||
import { test, expect } from 'bun:test';
|
||||
import {
|
||||
parseFixtures,
|
||||
buildPrompt,
|
||||
parseModelResponse,
|
||||
scoreFixture,
|
||||
scoreFixtureLenient,
|
||||
parseDispatcherLists,
|
||||
meanAndCI95,
|
||||
estimateCost,
|
||||
hashContent,
|
||||
parseArgs,
|
||||
resolveModel,
|
||||
PROMPT_TEMPLATE,
|
||||
MODEL_ID,
|
||||
MODEL_ALIASES,
|
||||
} from './harness-runner.ts';
|
||||
|
||||
test('parseFixtures: parses valid JSONL', () => {
|
||||
const raw = `{"intent":"foo","expected_skill":"bar"}\n{"intent":"baz","expected_skill":"qux"}\n`;
|
||||
const out = parseFixtures(raw);
|
||||
expect(out).toEqual([
|
||||
{ intent: 'foo', expected_skill: 'bar' },
|
||||
{ intent: 'baz', expected_skill: 'qux' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('parseFixtures: skips // comments and blank lines', () => {
|
||||
const raw = `// header comment\n{"intent":"a","expected_skill":"b"}\n\n// another comment\n{"intent":"c","expected_skill":"d"}\n`;
|
||||
const out = parseFixtures(raw);
|
||||
expect(out).toHaveLength(2);
|
||||
expect(out[0].intent).toBe('a');
|
||||
});
|
||||
|
||||
test('parseFixtures: throws on missing required fields', () => {
|
||||
expect(() => parseFixtures(`{"intent":"foo"}\n`)).toThrow(/missing required fields/);
|
||||
});
|
||||
|
||||
test('parseFixtures: throws on invalid JSON', () => {
|
||||
expect(() => parseFixtures(`{not json}\n`)).toThrow(/Bad fixture JSON/);
|
||||
});
|
||||
|
||||
test('buildPrompt: injects variant content and intent', () => {
|
||||
const prompt = buildPrompt('RESOLVER X', 'INTENT Y');
|
||||
expect(prompt).toContain('RESOLVER X');
|
||||
expect(prompt).toContain('INTENT Y');
|
||||
expect(prompt).not.toContain('<<<RESOLVER_CONTENT>>>');
|
||||
expect(prompt).not.toContain('<<<INTENT>>>');
|
||||
});
|
||||
|
||||
test('parseModelResponse: bare slug', () => {
|
||||
expect(parseModelResponse('enrich')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: strips fenced output', () => {
|
||||
expect(parseModelResponse('```\nenrich\n```')).toBe('enrich');
|
||||
expect(parseModelResponse('```text\nenrich\n```')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: extracts from JSON object', () => {
|
||||
expect(parseModelResponse('{"skill": "book-mirror"}')).toBe('book-mirror');
|
||||
expect(parseModelResponse('{"skill_slug": "query"}')).toBe('query');
|
||||
});
|
||||
|
||||
test('parseModelResponse: strips quotes and backticks', () => {
|
||||
expect(parseModelResponse('"enrich"')).toBe('enrich');
|
||||
expect(parseModelResponse('`enrich`')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: picks first slug-shaped token if model prefaces with prose', () => {
|
||||
expect(parseModelResponse('The skill is enrich.')).toBe('the'); // first token wins; documents permissive matcher
|
||||
expect(parseModelResponse('enrich is the answer')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('parseModelResponse: lowercases output', () => {
|
||||
expect(parseModelResponse('ENRICH')).toBe('enrich');
|
||||
});
|
||||
|
||||
test('scoreFixture: exact match returns 1', () => {
|
||||
expect(scoreFixture('enrich', 'enrich')).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixture: mismatch returns 0', () => {
|
||||
expect(scoreFixture('enrich', 'query')).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixture: case-sensitive at this layer (caller lowercases via parseModelResponse)', () => {
|
||||
expect(scoreFixture('Enrich', 'enrich')).toBe(0);
|
||||
});
|
||||
|
||||
test('meanAndCI95: empty array returns zeros', () => {
|
||||
expect(meanAndCI95([])).toEqual({ mean: 0, halfWidthCI: 0 });
|
||||
});
|
||||
|
||||
test('meanAndCI95: single value returns mean with zero CI', () => {
|
||||
expect(meanAndCI95([0.95])).toEqual({ mean: 0.95, halfWidthCI: 0 });
|
||||
});
|
||||
|
||||
test('meanAndCI95: three equal values returns mean with zero CI', () => {
|
||||
const r = meanAndCI95([1, 1, 1]);
|
||||
expect(r.mean).toBe(1);
|
||||
expect(r.halfWidthCI).toBe(0);
|
||||
});
|
||||
|
||||
test('meanAndCI95: three different values returns plausible CI', () => {
|
||||
const r = meanAndCI95([0.8, 0.9, 1.0]);
|
||||
expect(r.mean).toBeCloseTo(0.9, 5);
|
||||
expect(r.halfWidthCI).toBeGreaterThan(0);
|
||||
expect(r.halfWidthCI).toBeLessThan(0.5);
|
||||
});
|
||||
|
||||
test('estimateCost: uses Opus 4.7 pricing by default', () => {
|
||||
const cost = estimateCost(100, 'claude-opus-4-7', 1000, 50);
|
||||
// 100 calls * 1000 input tokens = 100K input → $0.50 at $5/MTok
|
||||
// 100 calls * 50 output tokens = 5K output → $0.125 at $25/MTok
|
||||
expect(cost).toBeCloseTo(0.625, 2);
|
||||
});
|
||||
|
||||
test('estimateCost: Sonnet pricing differs from Opus', () => {
|
||||
const opus = estimateCost(100, 'claude-opus-4-7', 1000, 50);
|
||||
const sonnet = estimateCost(100, 'claude-sonnet-4-6', 1000, 50);
|
||||
const haiku = estimateCost(100, 'claude-haiku-4-5-20251001', 1000, 50);
|
||||
expect(sonnet).toBeLessThan(opus);
|
||||
expect(haiku).toBeLessThan(sonnet);
|
||||
});
|
||||
|
||||
test('estimateCost: zero calls returns zero', () => {
|
||||
expect(estimateCost(0)).toBe(0);
|
||||
});
|
||||
|
||||
test('estimateCost: unknown model returns zero', () => {
|
||||
expect(estimateCost(100, 'unknown-model')).toBe(0);
|
||||
});
|
||||
|
||||
test('hashContent: produces stable 16-char hex prefix', () => {
|
||||
const h1 = hashContent('hello world');
|
||||
const h2 = hashContent('hello world');
|
||||
expect(h1).toBe(h2);
|
||||
expect(h1).toHaveLength(16);
|
||||
expect(h1).toMatch(/^[0-9a-f]+$/);
|
||||
});
|
||||
|
||||
test('hashContent: different inputs produce different hashes', () => {
|
||||
expect(hashContent('a')).not.toBe(hashContent('b'));
|
||||
});
|
||||
|
||||
test('parseArgs: defaults are sensible', () => {
|
||||
expect(parseArgs([])).toEqual({
|
||||
limit: null,
|
||||
parallel: 1,
|
||||
output: null,
|
||||
help: false,
|
||||
yes: false,
|
||||
model: MODEL_ID,
|
||||
variantsDir: 'variants',
|
||||
variantFiles: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('parseArgs: --model alias', () => {
|
||||
expect(parseArgs(['--model', 'sonnet']).model).toBe('sonnet');
|
||||
expect(parseArgs(['--model', 'anthropic:claude-haiku-4-5-20251001']).model).toBe('anthropic:claude-haiku-4-5-20251001');
|
||||
});
|
||||
|
||||
test('parseArgs: --variants comma-list', () => {
|
||||
expect(parseArgs(['--variants', 'a,b,c']).variantFiles).toEqual(['a', 'b', 'c']);
|
||||
});
|
||||
|
||||
test('parseArgs: --variants-dir', () => {
|
||||
expect(parseArgs(['--variants-dir', 'variants-sweep']).variantsDir).toBe('variants-sweep');
|
||||
});
|
||||
|
||||
test('resolveModel: aliases', () => {
|
||||
expect(resolveModel('opus')).toEqual({ full: 'anthropic:claude-opus-4-7', bare: 'claude-opus-4-7' });
|
||||
expect(resolveModel('sonnet')).toEqual({ full: 'anthropic:claude-sonnet-4-6', bare: 'claude-sonnet-4-6' });
|
||||
expect(resolveModel('haiku').full).toBe(MODEL_ALIASES.haiku);
|
||||
});
|
||||
|
||||
test('resolveModel: passthrough for full id', () => {
|
||||
expect(resolveModel('anthropic:claude-opus-4-7').bare).toBe('claude-opus-4-7');
|
||||
expect(resolveModel('anthropic:claude-something-future').bare).toBe('claude-something-future');
|
||||
});
|
||||
|
||||
test('resolveModel: non-anthropic provider passes through unchanged', () => {
|
||||
expect(resolveModel('openai:gpt-4o')).toEqual({ full: 'openai:gpt-4o', bare: 'openai:gpt-4o' });
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: extracts dispatcher → sub-skills', () => {
|
||||
const variant = `
|
||||
- **Brain**: foo bar → \`brain-ops\` (dispatcher for: enrich, query, citation-fixer)
|
||||
- **Comms**: email → \`exec-assist\` (dispatcher for: gmail, slack)
|
||||
- Bare row → \`bare-skill\`
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')).toEqual(new Set(['brain-ops', 'enrich', 'query', 'citation-fixer']));
|
||||
expect(m.get('exec-assist')).toEqual(new Set(['exec-assist', 'gmail', 'slack']));
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: accepts ASCII -> arrow (SKILL.md template format)', () => {
|
||||
// Codex review P2-2: SKILL.md Step 4 documents the template with `->`,
|
||||
// but the production variants use Unicode `→`. The regex must match
|
||||
// both or downstream users following the template silently fall through
|
||||
// to strict-only scoring.
|
||||
const variant = `
|
||||
- **Brain**: foo bar -> \`brain-ops\` (dispatcher for: enrich, query)
|
||||
- **Comms**: email -> \`exec-assist\` (dispatcher for: gmail)
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')).toEqual(new Set(['brain-ops', 'enrich', 'query']));
|
||||
expect(m.get('exec-assist')).toEqual(new Set(['exec-assist', 'gmail']));
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: mixed Unicode + ASCII arrows in same file', () => {
|
||||
// A real-world fork could migrate gradually; harness must handle both.
|
||||
const variant = `
|
||||
- **Brain**: foo → \`brain-ops\` (dispatcher for: enrich, query)
|
||||
- **Comms**: email -> \`exec-assist\` (dispatcher for: gmail, slack)
|
||||
`;
|
||||
const m = parseDispatcherLists(variant);
|
||||
expect(m.size).toBe(2);
|
||||
expect(m.get('brain-ops')?.has('enrich')).toBe(true);
|
||||
expect(m.get('exec-assist')?.has('gmail')).toBe(true);
|
||||
});
|
||||
|
||||
test('parseDispatcherLists: zero dispatchers when no clauses present', () => {
|
||||
const variant = `
|
||||
- Row 1 → \`alpha\`
|
||||
- Row 2 → \`beta\`
|
||||
`;
|
||||
expect(parseDispatcherLists(variant).size).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: exact match = 1', () => {
|
||||
expect(scoreFixtureLenient('enrich', 'enrich', new Map())).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: same-area sub-skill = 1', () => {
|
||||
const lists = new Map([['brain-ops', new Set(['brain-ops', 'enrich', 'query'])]]);
|
||||
expect(scoreFixtureLenient('enrich', 'query', lists)).toBe(1);
|
||||
expect(scoreFixtureLenient('brain-ops', 'enrich', lists)).toBe(1);
|
||||
expect(scoreFixtureLenient('enrich', 'brain-ops', lists)).toBe(1);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: cross-area = 0', () => {
|
||||
const lists = new Map([
|
||||
['brain-ops', new Set(['brain-ops', 'enrich'])],
|
||||
['comms', new Set(['comms', 'gmail'])],
|
||||
]);
|
||||
expect(scoreFixtureLenient('enrich', 'gmail', lists)).toBe(0);
|
||||
});
|
||||
|
||||
test('scoreFixtureLenient: no dispatcher map = falls back to strict', () => {
|
||||
expect(scoreFixtureLenient('foo', 'bar', new Map())).toBe(0);
|
||||
});
|
||||
|
||||
test('parseArgs: --limit', () => {
|
||||
expect(parseArgs(['--limit', '5']).limit).toBe(5);
|
||||
});
|
||||
|
||||
test('parseArgs: --limit rejects non-positive', () => {
|
||||
expect(() => parseArgs(['--limit', '0'])).toThrow();
|
||||
expect(() => parseArgs(['--limit', '-3'])).toThrow();
|
||||
expect(() => parseArgs(['--limit', 'foo'])).toThrow();
|
||||
});
|
||||
|
||||
test('parseArgs: --parallel', () => {
|
||||
expect(parseArgs(['--parallel', '4']).parallel).toBe(4);
|
||||
});
|
||||
|
||||
test('parseArgs: --output', () => {
|
||||
expect(parseArgs(['--output', '/tmp/x.jsonl']).output).toBe('/tmp/x.jsonl');
|
||||
});
|
||||
|
||||
test('parseArgs: --help and --yes', () => {
|
||||
expect(parseArgs(['--help']).help).toBe(true);
|
||||
expect(parseArgs(['--yes']).yes).toBe(true);
|
||||
});
|
||||
|
||||
test('parseArgs: rejects unknown flags', () => {
|
||||
expect(() => parseArgs(['--bogus'])).toThrow(/Unknown flag/);
|
||||
});
|
||||
|
||||
test('MODEL_ID is pinned to Opus 4.7', () => {
|
||||
expect(MODEL_ID).toBe('anthropic:claude-opus-4-7');
|
||||
});
|
||||
|
||||
test('PROMPT_TEMPLATE contains both placeholders', () => {
|
||||
expect(PROMPT_TEMPLATE).toContain('<<<RESOLVER_CONTENT>>>');
|
||||
expect(PROMPT_TEMPLATE).toContain('<<<INTENT>>>');
|
||||
});
|
||||
599
evals/functional-area-resolver/harness-runner.ts
Normal file
599
evals/functional-area-resolver/harness-runner.ts
Normal file
@@ -0,0 +1,599 @@
|
||||
/**
|
||||
* functional-area-resolver A/B eval runner.
|
||||
*
|
||||
* Reads three variant resolver files + two fixture corpora, runs each
|
||||
* (fixture, variant, seed in {1,2,3}) through Anthropic Opus 4.7 via
|
||||
* gbrain's gateway, scores the response, writes one JSONL row per call,
|
||||
* computes per-variant accuracy mean + 95% CI, prints a summary table.
|
||||
*
|
||||
* Receipts bind (model, prompt_template_hash, fixtures_hash, ts, seed)
|
||||
* so re-runs are auditable. Output JSONL begins with a receipt header.
|
||||
*
|
||||
* Pinned to anthropic:claude-opus-4-7. Update MODEL_ID and re-baseline
|
||||
* when Anthropic ships a new Opus generation. Cost: ~$1.70 per full run
|
||||
* (225 calls × ~$0.0076 each at $5/$25 per MTok input/output).
|
||||
*
|
||||
* Lives outside `skills/` deliberately — the skillpack bundler walks
|
||||
* `skills/<skill>/` recursively, so an eval surface in there would ship
|
||||
* to every downstream install. Importing `src/core/ai/gateway.ts` is
|
||||
* legitimate from this location because the eval is gbrain-repo-only.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { execSync } from 'node:child_process';
|
||||
|
||||
import { configureGateway, chat } from '../../src/core/ai/gateway.ts';
|
||||
import { loadConfig } from '../../src/core/config.ts';
|
||||
import { ANTHROPIC_PRICING } from '../../src/core/anthropic-pricing.ts';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(__dirname, '..', '..');
|
||||
|
||||
// Default model — pinned so the canonical baseline-runs/<date>-opus-4-7.jsonl
|
||||
// stays reproducible. Override with --model for cross-model eval (T3a).
|
||||
export const MODEL_ID = 'anthropic:claude-opus-4-7';
|
||||
|
||||
export const MODEL_ALIASES: Record<string, string> = {
|
||||
opus: 'anthropic:claude-opus-4-7',
|
||||
sonnet: 'anthropic:claude-sonnet-4-6',
|
||||
haiku: 'anthropic:claude-haiku-4-5-20251001',
|
||||
};
|
||||
|
||||
export function resolveModel(spec: string): { full: string; bare: string } {
|
||||
const full = MODEL_ALIASES[spec] ?? spec;
|
||||
const bare = full.startsWith('anthropic:') ? full.slice('anthropic:'.length) : full;
|
||||
return { full, bare };
|
||||
}
|
||||
|
||||
const VARIANT_NAMES = ['baseline', 'functional-areas', 'resolver-of-resolvers'] as const;
|
||||
type VariantName = (typeof VARIANT_NAMES)[number];
|
||||
|
||||
const SEEDS = [1, 2, 3] as const;
|
||||
|
||||
export interface Fixture {
|
||||
intent: string;
|
||||
expected_skill: string;
|
||||
}
|
||||
|
||||
export interface RunRow {
|
||||
kind: 'run';
|
||||
fixture_id: number;
|
||||
corpus: 'training' | 'held_out';
|
||||
variant: VariantName;
|
||||
seed: number;
|
||||
predicted: string;
|
||||
expected: string;
|
||||
/** Strict score: predicted exactly equals expected. */
|
||||
correct: 0 | 1;
|
||||
/** Lenient score: predicted is in the same dispatcher area as expected (T1a). */
|
||||
correct_lenient: 0 | 1;
|
||||
model: string;
|
||||
input_tokens: number;
|
||||
output_tokens: number;
|
||||
latency_ms: number;
|
||||
ts: string;
|
||||
}
|
||||
|
||||
export interface ReceiptRow {
|
||||
kind: 'receipt';
|
||||
model: string;
|
||||
prompt_template_hash: string;
|
||||
fixtures_hash: string;
|
||||
fixtures_held_out_hash: string;
|
||||
/** Git sha of the harness at run time (T4). Detect stale numbers when harness changes. */
|
||||
harness_sha: string | null;
|
||||
ts: string;
|
||||
cmd_args: string[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Pure functions (testable without API key)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export const PROMPT_TEMPLATE = `You are a routing classifier for a skill-based agent. Given the resolver below and the user's intent, return the single most-specific skill slug that should handle the intent.
|
||||
|
||||
Rules:
|
||||
- Return ONLY a slug. No explanation, no quotes, no markdown — just the slug.
|
||||
- Some entries are functional-area dispatchers shaped like:
|
||||
"**Area name**: triggers... → \`dispatcher-skill\` (dispatcher for: subskill-a, subskill-b, subskill-c, ...)"
|
||||
When the user's intent matches an area, RETURN THE MOST-SPECIFIC SUB-SKILL from that area's "dispatcher for" list, not the dispatcher itself. The dispatcher slug is only correct when no listed sub-skill is more specific to the intent.
|
||||
- If a row has no dispatcher list, return its slug directly.
|
||||
|
||||
RESOLVER:
|
||||
<<<RESOLVER_CONTENT>>>
|
||||
|
||||
USER INTENT: <<<INTENT>>>
|
||||
|
||||
SKILL SLUG:`;
|
||||
|
||||
export function parseFixtures(rawJsonl: string): Fixture[] {
|
||||
const out: Fixture[] = [];
|
||||
const lines = rawJsonl.split('\n');
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (trimmed.length === 0) continue;
|
||||
if (trimmed.startsWith('//')) continue;
|
||||
let obj: any;
|
||||
try {
|
||||
obj = JSON.parse(trimmed);
|
||||
} catch (err) {
|
||||
throw new Error(`Bad fixture JSON: ${trimmed.slice(0, 80)} — ${(err as Error).message}`);
|
||||
}
|
||||
if (typeof obj.intent !== 'string' || typeof obj.expected_skill !== 'string') {
|
||||
throw new Error(`Fixture missing required fields: ${trimmed.slice(0, 80)}`);
|
||||
}
|
||||
out.push({ intent: obj.intent, expected_skill: obj.expected_skill });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function loadVariant(path: string): string {
|
||||
return readFileSync(path, 'utf8');
|
||||
}
|
||||
|
||||
export function buildPrompt(variantContent: string, intent: string): string {
|
||||
return PROMPT_TEMPLATE.replace('<<<RESOLVER_CONTENT>>>', variantContent).replace('<<<INTENT>>>', intent);
|
||||
}
|
||||
|
||||
export function parseModelResponse(raw: string): string {
|
||||
// The model may return: bare slug, fenced slug, quoted slug, JSON-wrapped
|
||||
// slug, or slug with a leading explanation. We strip the obvious wrappers
|
||||
// and take the first line that looks like a slug.
|
||||
let s = raw.trim();
|
||||
// Strip ```...``` fences
|
||||
s = s.replace(/^```[a-zA-Z]*\n?/, '').replace(/\n?```\s*$/, '').trim();
|
||||
// If the response is JSON like {"skill": "foo"}, extract.
|
||||
if (s.startsWith('{')) {
|
||||
try {
|
||||
const obj = JSON.parse(s);
|
||||
if (typeof obj.skill === 'string') return obj.skill.trim().toLowerCase();
|
||||
if (typeof obj.skill_slug === 'string') return obj.skill_slug.trim().toLowerCase();
|
||||
if (typeof obj.expected_skill === 'string') return obj.expected_skill.trim().toLowerCase();
|
||||
} catch {}
|
||||
}
|
||||
// Strip surrounding quotes and backticks
|
||||
s = s.replace(/^[`"']|[`"']$/g, '').trim();
|
||||
// Take first non-empty line
|
||||
const firstLine = s.split(/\r?\n/).map(l => l.trim()).find(l => l.length > 0) ?? '';
|
||||
// If it starts with a prose preamble, look for a slug-shaped token
|
||||
const slugMatch = firstLine.match(/[a-z][a-z0-9-]+/i);
|
||||
return (slugMatch ? slugMatch[0] : firstLine).toLowerCase();
|
||||
}
|
||||
|
||||
export function scoreFixture(predicted: string, expected: string): 0 | 1 {
|
||||
return predicted === expected ? 1 : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse every "...→ `dispatcher-slug` (dispatcher for: a, b, c, ...)" line
|
||||
* out of a variant resolver. Returns a map: dispatcher_slug → set of sub-skill
|
||||
* slugs reachable through it. Also includes the dispatcher_slug itself in
|
||||
* the set so it's a self-member.
|
||||
*
|
||||
* Variant shapes:
|
||||
* - functional-areas.md: "→ `brain-ops` (dispatcher for: enrich, query, ...)"
|
||||
* - resolver-of-resolvers.md: "→ `brain-ops`" (no dispatcher clause; returns {})
|
||||
* - baseline.md: per-skill rows (each row's slug becomes its own area)
|
||||
*
|
||||
* Used by lenientScore: a predicted slug counts as "same area as expected"
|
||||
* if both belong to the same dispatcher's reachable set, OR predicted is the
|
||||
* dispatcher and expected is a sub-skill (or vice versa).
|
||||
*/
|
||||
export function parseDispatcherLists(variantContent: string): Map<string, Set<string>> {
|
||||
const out = new Map<string, Set<string>>();
|
||||
// Match both Unicode `→` (used in the real production AGENTS.md the variants
|
||||
// came from) AND ASCII `->` (what SKILL.md's template emits when a user
|
||||
// follows the documented instructions). Codex review P2-2: without ASCII
|
||||
// support, downstream-authored resolvers silently fall through to strict
|
||||
// scoring even though SKILL.md tells the user the template uses `->`.
|
||||
const re = /(?:→|->)\s*`([a-z][a-z0-9-]*)`\s*\(dispatcher for:\s*([^)]+)\)/g;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(variantContent)) !== null) {
|
||||
const dispatcher = m[1];
|
||||
const subSkills = m[2].split(',').map(s => s.trim()).filter(s => /^[a-z][a-z0-9-]*$/.test(s));
|
||||
const set = new Set<string>([dispatcher, ...subSkills]);
|
||||
out.set(dispatcher, set);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lenient scoring: predicted is correct if (predicted == expected) OR
|
||||
* (both predicted and expected are in the same dispatcher's reachable set
|
||||
* per the variant). This is the T1a re-scoring that surfaces "the LLM
|
||||
* picked a legitimate sub-skill, just not the one my fixture named."
|
||||
*
|
||||
* For variants with no dispatcher clauses (baseline, resolver-of-resolvers),
|
||||
* lenient collapses to strict.
|
||||
*/
|
||||
export function scoreFixtureLenient(
|
||||
predicted: string,
|
||||
expected: string,
|
||||
dispatcherLists: Map<string, Set<string>>,
|
||||
): 0 | 1 {
|
||||
if (predicted === expected) return 1;
|
||||
for (const set of dispatcherLists.values()) {
|
||||
if (set.has(predicted) && set.has(expected)) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** Capture the harness git sha so receipts can detect stale numbers. */
|
||||
export function getHarnessSha(): string | null {
|
||||
try {
|
||||
const sha = execSync('git rev-parse HEAD', { cwd: __dirname, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
|
||||
return sha.length === 40 ? sha : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mean and 95% CI via t-distribution (n=3, df=2, t-critical ≈ 4.303).
|
||||
* For n=3 with df=2 the 95% two-tailed t-critical is 4.303 per standard
|
||||
* tables. Returns the half-width of the CI (mean ± halfWidth).
|
||||
*/
|
||||
export function meanAndCI95(values: number[]): { mean: number; halfWidthCI: number } {
|
||||
if (values.length === 0) return { mean: 0, halfWidthCI: 0 };
|
||||
const mean = values.reduce((a, b) => a + b, 0) / values.length;
|
||||
if (values.length === 1) return { mean, halfWidthCI: 0 };
|
||||
const variance = values.reduce((acc, v) => acc + (v - mean) ** 2, 0) / (values.length - 1);
|
||||
const stdErr = Math.sqrt(variance / values.length);
|
||||
const tCrit = values.length === 3 ? 4.303 : values.length === 2 ? 12.706 : 1.96;
|
||||
return { mean, halfWidthCI: tCrit * stdErr };
|
||||
}
|
||||
|
||||
export function estimateCost(
|
||||
numCalls: number,
|
||||
modelBare: string = 'claude-opus-4-7',
|
||||
inputTokensPerCall = 1000,
|
||||
outputTokensPerCall = 50,
|
||||
): number {
|
||||
const pricing = ANTHROPIC_PRICING[modelBare];
|
||||
if (!pricing) return 0;
|
||||
const input = (numCalls * inputTokensPerCall) / 1_000_000;
|
||||
const output = (numCalls * outputTokensPerCall) / 1_000_000;
|
||||
return input * pricing.input + output * pricing.output;
|
||||
}
|
||||
|
||||
export function hashContent(content: string): string {
|
||||
return createHash('sha256').update(content).digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
export function writeJsonl(rows: (RunRow | ReceiptRow)[], outputPath: string): void {
|
||||
const dir = dirname(outputPath);
|
||||
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
|
||||
const lines = rows.map(r => JSON.stringify(r)).join('\n') + '\n';
|
||||
writeFileSync(outputPath, lines, 'utf8');
|
||||
}
|
||||
|
||||
export interface ParsedArgs {
|
||||
limit: number | null;
|
||||
parallel: number;
|
||||
output: string | null;
|
||||
help: boolean;
|
||||
yes: boolean;
|
||||
/** Model alias ('opus','sonnet','haiku') or full provider:model id. */
|
||||
model: string;
|
||||
/** Variants directory (default ./variants). */
|
||||
variantsDir: string;
|
||||
/** Custom variant glob (overrides default 3 variants); used by description-length sweep. */
|
||||
variantFiles: string[] | null;
|
||||
}
|
||||
|
||||
export function parseArgs(argv: string[]): ParsedArgs {
|
||||
const out: ParsedArgs = {
|
||||
limit: null, parallel: 1, output: null, help: false, yes: false,
|
||||
model: MODEL_ID, variantsDir: 'variants', variantFiles: null,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--help' || a === '-h') out.help = true;
|
||||
else if (a === '--yes' || a === '-y') out.yes = true;
|
||||
else if (a === '--limit') {
|
||||
const v = parseInt(argv[++i], 10);
|
||||
if (!Number.isFinite(v) || v < 1) throw new Error(`--limit must be a positive integer`);
|
||||
out.limit = v;
|
||||
} else if (a === '--parallel') {
|
||||
const v = parseInt(argv[++i], 10);
|
||||
if (!Number.isFinite(v) || v < 1) throw new Error(`--parallel must be a positive integer`);
|
||||
out.parallel = v;
|
||||
} else if (a === '--output') {
|
||||
out.output = argv[++i];
|
||||
} else if (a === '--model') {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--model requires a value (alias or provider:model)`);
|
||||
out.model = v;
|
||||
} else if (a === '--variants-dir') {
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--variants-dir requires a path`);
|
||||
out.variantsDir = v;
|
||||
} else if (a === '--variants') {
|
||||
// Comma-separated list of variant file basenames (without .md). Used by sweep.
|
||||
const v = argv[++i];
|
||||
if (!v) throw new Error(`--variants requires a comma-separated list`);
|
||||
out.variantFiles = v.split(',').map(s => s.trim()).filter(Boolean);
|
||||
} else if (a.startsWith('--')) {
|
||||
throw new Error(`Unknown flag: ${a}`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Gateway wrapper (mockable via __setChatTransportForTests)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function callModel(prompt: string, modelFull: string): Promise<{ text: string; input_tokens: number; output_tokens: number; latency_ms: number }> {
|
||||
const t0 = Date.now();
|
||||
const result = await chat({
|
||||
model: modelFull,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
maxTokens: 64,
|
||||
});
|
||||
return {
|
||||
text: result.text,
|
||||
input_tokens: result.usage.input_tokens,
|
||||
output_tokens: result.usage.output_tokens,
|
||||
latency_ms: Date.now() - t0,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const HELP = `functional-area-resolver A/B eval harness
|
||||
|
||||
Usage:
|
||||
bun run harness-runner.ts [flags]
|
||||
node harness.mjs [flags] # CLI shim
|
||||
|
||||
Flags:
|
||||
--limit N Run only the first N (fixture × variant × seed) tuples
|
||||
--parallel N Run N tuples in parallel (default 1; gateway rate-lease bound)
|
||||
--output PATH Write JSONL to PATH (default: ./run-<ISO-ts>.jsonl)
|
||||
--model SPEC Model alias (opus|sonnet|haiku) or full provider:model id
|
||||
Default: opus (anthropic:claude-opus-4-7)
|
||||
--variants-dir PATH Override variants directory (default: ./variants)
|
||||
--variants A,B,C Comma-separated variant basenames (default: all 3 in variants-dir)
|
||||
Useful for description-length sweep where you have 4+ variants.
|
||||
--yes Skip the cost-estimate confirmation prompt
|
||||
--help Print this help
|
||||
|
||||
Cost rough estimates (75 calls/variant × num-variants × 3 seeds):
|
||||
Opus: ~$1.70 per 225-call run (1 model × 3 variants × 25 fixtures × 3 seeds)
|
||||
Sonnet: ~$1.02 per 225-call run
|
||||
Haiku: ~$0.34 per 225-call run
|
||||
|
||||
Output JSONL has each row scored TWICE: 'correct' (strict, predicted==expected)
|
||||
and 'correct_lenient' (predicted and expected are in the same dispatcher area).
|
||||
Summary reports both.
|
||||
`;
|
||||
|
||||
async function maybePromptCost(numCalls: number, modelFull: string, autoConfirm: boolean): Promise<boolean> {
|
||||
const { bare } = resolveModel(modelFull);
|
||||
const cost = estimateCost(numCalls, bare);
|
||||
process.stderr.write(`Estimated cost: ~$${cost.toFixed(2)} for ${numCalls} LLM calls via ${modelFull}.\n`);
|
||||
if (autoConfirm) return true;
|
||||
if (!process.stdin.isTTY) {
|
||||
process.stderr.write('Non-TTY context; pass --yes to confirm.\n');
|
||||
return false;
|
||||
}
|
||||
process.stderr.write('Press Enter to continue or Ctrl-C to abort. ');
|
||||
return await new Promise(resolve => {
|
||||
process.stdin.once('data', () => resolve(true));
|
||||
process.stdin.once('end', () => resolve(false));
|
||||
});
|
||||
}
|
||||
|
||||
export async function main(argv: string[]): Promise<number> {
|
||||
let args: ParsedArgs;
|
||||
try {
|
||||
args = parseArgs(argv);
|
||||
} catch (err) {
|
||||
process.stderr.write(`Error: ${(err as Error).message}\n\n${HELP}`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
if (args.help) {
|
||||
process.stdout.write(HELP);
|
||||
return 0;
|
||||
}
|
||||
|
||||
const { full: modelFull, bare: modelBare } = resolveModel(args.model);
|
||||
|
||||
// Self-configure the gateway (matches src/commands/eval-cross-modal.ts:195-220).
|
||||
const config = loadConfig();
|
||||
configureGateway({
|
||||
embedding_model: config?.embedding_model,
|
||||
embedding_dimensions: config?.embedding_dimensions,
|
||||
expansion_model: config?.expansion_model,
|
||||
chat_model: config?.chat_model ?? modelFull,
|
||||
chat_fallback_chain: config?.chat_fallback_chain,
|
||||
base_urls: config?.provider_base_urls,
|
||||
env: { ...process.env } as Record<string, string>,
|
||||
});
|
||||
|
||||
// Provider-aware auth check (codex review P2-3). The CLI advertises full
|
||||
// provider:model support and the test suite covers `openai:gpt-4o`, so the
|
||||
// env-var gate must match the provider that will actually be called.
|
||||
// Unknown providers fall through to the gateway, which will raise a clear
|
||||
// recipe-specific error if any required env var is missing.
|
||||
const REQUIRED_ENV_BY_PROVIDER: Record<string, string> = {
|
||||
anthropic: 'ANTHROPIC_API_KEY',
|
||||
openai: 'OPENAI_API_KEY',
|
||||
google: 'GOOGLE_GENERATIVE_AI_API_KEY',
|
||||
groq: 'GROQ_API_KEY',
|
||||
voyage: 'VOYAGE_API_KEY',
|
||||
together: 'TOGETHER_API_KEY',
|
||||
deepseek: 'DEEPSEEK_API_KEY',
|
||||
minimax: 'MINIMAX_API_KEY',
|
||||
dashscope: 'DASHSCOPE_API_KEY',
|
||||
zhipu: 'ZHIPUAI_API_KEY',
|
||||
};
|
||||
const providerId = modelFull.includes(':') ? modelFull.split(':', 1)[0] : 'anthropic';
|
||||
const requiredEnv = REQUIRED_ENV_BY_PROVIDER[providerId];
|
||||
if (requiredEnv && !process.env[requiredEnv]) {
|
||||
process.stderr.write(`Error: ${requiredEnv} is not set. The harness needs it to reach ${modelFull}.\n`);
|
||||
return 2;
|
||||
}
|
||||
|
||||
// Load fixtures + variants.
|
||||
const evalsDir = __dirname;
|
||||
const fixturesTraining = parseFixtures(readFileSync(join(evalsDir, 'fixtures.jsonl'), 'utf8'));
|
||||
const fixturesHeldOut = parseFixtures(readFileSync(join(evalsDir, 'fixtures-held-out.jsonl'), 'utf8'));
|
||||
|
||||
// Dynamic variants: --variants overrides the default 3, --variants-dir overrides location.
|
||||
const variantsAbsDir = resolve(evalsDir, args.variantsDir);
|
||||
const variantBasenames = args.variantFiles
|
||||
?? (VARIANT_NAMES as readonly string[]).map(n => n);
|
||||
const variants: Record<string, string> = {};
|
||||
const dispatcherListsByVariant: Record<string, Map<string, Set<string>>> = {};
|
||||
for (const name of variantBasenames) {
|
||||
const content = loadVariant(join(variantsAbsDir, `${name}.md`));
|
||||
variants[name] = content;
|
||||
dispatcherListsByVariant[name] = parseDispatcherLists(content);
|
||||
}
|
||||
|
||||
// Build the (fixture × variant × seed) tuple list.
|
||||
type Tuple = { fixture: Fixture; corpus: 'training' | 'held_out'; fixture_id: number; variant: string; seed: number };
|
||||
const tuples: Tuple[] = [];
|
||||
for (const variant of variantBasenames) {
|
||||
fixturesTraining.forEach((f, i) => {
|
||||
for (const seed of SEEDS) tuples.push({ fixture: f, corpus: 'training', fixture_id: i, variant, seed });
|
||||
});
|
||||
fixturesHeldOut.forEach((f, i) => {
|
||||
for (const seed of SEEDS) tuples.push({ fixture: f, corpus: 'held_out', fixture_id: i, variant, seed });
|
||||
});
|
||||
}
|
||||
const totalCalls = args.limit ? Math.min(args.limit, tuples.length) : tuples.length;
|
||||
const workQueue = tuples.slice(0, totalCalls);
|
||||
|
||||
// Cost-estimate prompt (skipped for tiny --limit runs to keep dev iteration fast).
|
||||
if (totalCalls >= 20) {
|
||||
const proceed = await maybePromptCost(totalCalls, modelFull, args.yes);
|
||||
if (!proceed) {
|
||||
process.stderr.write('Aborted.\n');
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Compute receipt header.
|
||||
const fixturesHash = hashContent(readFileSync(join(evalsDir, 'fixtures.jsonl'), 'utf8'));
|
||||
const fixturesHeldOutHash = hashContent(readFileSync(join(evalsDir, 'fixtures-held-out.jsonl'), 'utf8'));
|
||||
const promptTemplateHash = hashContent(PROMPT_TEMPLATE);
|
||||
const harnessSha = getHarnessSha();
|
||||
const tsStart = new Date().toISOString();
|
||||
const receipt: ReceiptRow = {
|
||||
kind: 'receipt',
|
||||
model: modelFull,
|
||||
prompt_template_hash: promptTemplateHash,
|
||||
fixtures_hash: fixturesHash,
|
||||
fixtures_held_out_hash: fixturesHeldOutHash,
|
||||
harness_sha: harnessSha,
|
||||
ts: tsStart,
|
||||
cmd_args: argv,
|
||||
};
|
||||
|
||||
// Output path.
|
||||
const outputPath = args.output ?? join(evalsDir, `run-${tsStart.replace(/[:.]/g, '-')}.jsonl`);
|
||||
process.stderr.write(`Writing receipt + ${totalCalls} runs to ${outputPath}\n`);
|
||||
|
||||
const rows: (RunRow | ReceiptRow)[] = [receipt];
|
||||
|
||||
// Sequential or simple bounded-parallel execution.
|
||||
let completed = 0;
|
||||
async function processTuple(t: Tuple): Promise<RunRow> {
|
||||
const prompt = buildPrompt(variants[t.variant], t.fixture.intent);
|
||||
const { text, input_tokens, output_tokens, latency_ms } = await callModel(prompt, modelFull);
|
||||
const predicted = parseModelResponse(text);
|
||||
const correct = scoreFixture(predicted, t.fixture.expected_skill);
|
||||
const correct_lenient = scoreFixtureLenient(
|
||||
predicted,
|
||||
t.fixture.expected_skill,
|
||||
dispatcherListsByVariant[t.variant] ?? new Map(),
|
||||
);
|
||||
const row: RunRow = {
|
||||
kind: 'run',
|
||||
fixture_id: t.fixture_id,
|
||||
corpus: t.corpus,
|
||||
variant: t.variant as VariantName,
|
||||
seed: t.seed,
|
||||
predicted,
|
||||
expected: t.fixture.expected_skill,
|
||||
correct,
|
||||
correct_lenient,
|
||||
model: modelFull,
|
||||
input_tokens,
|
||||
output_tokens,
|
||||
latency_ms,
|
||||
ts: new Date().toISOString(),
|
||||
};
|
||||
completed++;
|
||||
if (completed % 10 === 0 || completed === totalCalls) {
|
||||
process.stderr.write(` ${completed}/${totalCalls} done\n`);
|
||||
}
|
||||
return row;
|
||||
}
|
||||
|
||||
// Bounded parallel: chunk into args.parallel-sized batches.
|
||||
for (let i = 0; i < workQueue.length; i += args.parallel) {
|
||||
const batch = workQueue.slice(i, i + args.parallel);
|
||||
const results = await Promise.all(batch.map(processTuple));
|
||||
rows.push(...results);
|
||||
}
|
||||
|
||||
// Write JSONL.
|
||||
writeJsonl(rows, outputPath);
|
||||
|
||||
// Compute per-variant accuracy. Both strict + lenient. Held-out is the
|
||||
// headline; training is reported separately.
|
||||
const runRows = rows.filter((r): r is RunRow => r.kind === 'run');
|
||||
type CorpusKey = 'training' | 'held_out';
|
||||
type Acc = { training: number[]; held_out: number[] };
|
||||
const strictSummary: Record<string, Acc> = {};
|
||||
const lenientSummary: Record<string, Acc> = {};
|
||||
for (const variant of variantBasenames) {
|
||||
strictSummary[variant] = { training: [], held_out: [] };
|
||||
lenientSummary[variant] = { training: [], held_out: [] };
|
||||
for (const corpus of ['training', 'held_out'] as const) {
|
||||
for (const seed of SEEDS) {
|
||||
const subset = runRows.filter(r => r.variant === variant && r.corpus === corpus && r.seed === seed);
|
||||
if (subset.length === 0) continue;
|
||||
strictSummary[variant][corpus].push(subset.reduce((a, r) => a + r.correct, 0) / subset.length);
|
||||
lenientSummary[variant][corpus].push(subset.reduce((a, r) => a + r.correct_lenient, 0) / subset.length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Print summary.
|
||||
const fmt = (vals: number[]) => {
|
||||
if (vals.length === 0) return '—';
|
||||
const { mean, halfWidthCI } = meanAndCI95(vals);
|
||||
return `${(mean * 100).toFixed(1)}% ± ${(halfWidthCI * 100).toFixed(1)}%`;
|
||||
};
|
||||
|
||||
process.stderr.write(`\n=== A/B Eval Summary (model: ${modelFull}) ===\n`);
|
||||
process.stderr.write(' | STRICT scoring | LENIENT (same-area)\n');
|
||||
process.stderr.write('Variant | Held-out | Training | Held-out | Training\n');
|
||||
process.stderr.write('------------------------------|------------------------|------------------------|----------------------|----------------------\n');
|
||||
for (const variant of variantBasenames) {
|
||||
process.stderr.write(
|
||||
`${variant.padEnd(30)}| ${fmt(strictSummary[variant].held_out).padEnd(22)} | ${fmt(strictSummary[variant].training).padEnd(22)} | ${fmt(lenientSummary[variant].held_out).padEnd(20)} | ${fmt(lenientSummary[variant].training)}\n`,
|
||||
);
|
||||
}
|
||||
process.stderr.write('\nLENIENT counts a prediction as correct if it shares a dispatcher area with the expected target.\n');
|
||||
process.stderr.write('For variants without "(dispatcher for: ...)" clauses (baseline, resolver-of-resolvers), LENIENT == STRICT.\n');
|
||||
process.stderr.write('\nReceipt + runs written to: ' + outputPath + '\n');
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Bun entrypoint: run main when invoked as a script.
|
||||
if (import.meta.main) {
|
||||
main(process.argv.slice(2)).then(code => process.exit(code));
|
||||
}
|
||||
59
evals/functional-area-resolver/harness.mjs
Normal file
59
evals/functional-area-resolver/harness.mjs
Normal file
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Thin CLI shim for the functional-area-resolver A/B eval harness.
|
||||
*
|
||||
* Spawns the TypeScript runner via `bun` because the runner imports
|
||||
* gbrain's gateway from `src/core/ai/gateway.ts` directly. The runner
|
||||
* does the actual work; this file exists so users can invoke `node
|
||||
* harness.mjs` without remembering the bun incantation.
|
||||
*
|
||||
* If `bun` isn't on PATH (or this script is invoked outside the gbrain
|
||||
* repo), exit 2 with a clear message — the harness is a gbrain-side
|
||||
* proof-of-pattern, not a portable tool.
|
||||
*/
|
||||
|
||||
import { spawnSync, execFileSync } from 'node:child_process';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { existsSync } from 'node:fs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const runnerPath = resolve(__dirname, 'harness-runner.ts');
|
||||
const gatewayPath = resolve(__dirname, '..', '..', 'src', 'core', 'ai', 'gateway.ts');
|
||||
|
||||
function fail(message, code = 2) {
|
||||
process.stderr.write(message + '\n');
|
||||
process.exit(code);
|
||||
}
|
||||
|
||||
// Missing-binary fallback (F-E2): we need `bun` AND we need to be in
|
||||
// the gbrain repo so the runner can import the gateway.
|
||||
try {
|
||||
execFileSync('which', ['bun'], { stdio: 'ignore' });
|
||||
} catch {
|
||||
fail(
|
||||
'harness.mjs: `bun` is not on PATH.\n' +
|
||||
'This harness is a gbrain-maintainer-side tool — run it from a\n' +
|
||||
'gbrain repo checkout with `bun` installed (https://bun.sh).',
|
||||
);
|
||||
}
|
||||
|
||||
if (!existsSync(gatewayPath)) {
|
||||
fail(
|
||||
`harness.mjs: cannot find gbrain gateway at ${gatewayPath}.\n` +
|
||||
'This harness is the gbrain-side A/B eval surface. Run it from a\n' +
|
||||
'gbrain repo checkout, not from an installed skillpack.',
|
||||
);
|
||||
}
|
||||
|
||||
if (!existsSync(runnerPath)) {
|
||||
fail(`harness.mjs: runner missing at ${runnerPath}`);
|
||||
}
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const result = spawnSync('bun', ['run', runnerPath, ...args], {
|
||||
stdio: 'inherit',
|
||||
cwd: __dirname,
|
||||
});
|
||||
|
||||
process.exit(result.status ?? 1);
|
||||
121
evals/functional-area-resolver/rescore.mjs
Normal file
121
evals/functional-area-resolver/rescore.mjs
Normal file
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Re-score an existing run-*.jsonl (or baseline-runs/*.jsonl) with the lenient
|
||||
* dispatcher-area scoring rule, without re-running any LLM calls.
|
||||
*
|
||||
* Usage: node rescore.mjs <run-file.jsonl>
|
||||
*
|
||||
* Reads the receipt header to identify which variants were used, loads them
|
||||
* from ./variants/<name>.md, parses their (dispatcher for: ...) clauses, then
|
||||
* applies scoreFixtureLenient to every row. Prints a STRICT vs LENIENT
|
||||
* accuracy table without mutating the file.
|
||||
*
|
||||
* This is T1a from the v0.32.3.0 boil-the-ocean push.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
function parseDispatcherLists(variantContent) {
|
||||
const out = new Map();
|
||||
const re = /→\s*`([a-z][a-z0-9-]*)`\s*\(dispatcher for:\s*([^)]+)\)/g;
|
||||
let m;
|
||||
while ((m = re.exec(variantContent)) !== null) {
|
||||
const dispatcher = m[1];
|
||||
const subSkills = m[2].split(',').map(s => s.trim()).filter(s => /^[a-z][a-z0-9-]*$/.test(s));
|
||||
out.set(dispatcher, new Set([dispatcher, ...subSkills]));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function lenientScore(predicted, expected, dispatcherLists) {
|
||||
if (predicted === expected) return 1;
|
||||
for (const set of dispatcherLists.values()) {
|
||||
if (set.has(predicted) && set.has(expected)) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function meanAndCI(values) {
|
||||
if (values.length === 0) return { mean: 0, ci: 0 };
|
||||
const mean = values.reduce((a, b) => a + b, 0) / values.length;
|
||||
if (values.length === 1) return { mean, ci: 0 };
|
||||
const variance = values.reduce((acc, v) => acc + (v - mean) ** 2, 0) / (values.length - 1);
|
||||
const stdErr = Math.sqrt(variance / values.length);
|
||||
const tCrit = values.length === 3 ? 4.303 : values.length === 2 ? 12.706 : 1.96;
|
||||
return { mean, ci: tCrit * stdErr };
|
||||
}
|
||||
|
||||
function fmt(vals) {
|
||||
if (vals.length === 0) return '—';
|
||||
const { mean, ci } = meanAndCI(vals);
|
||||
return `${(mean * 100).toFixed(1)}% ± ${(ci * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
const runFile = process.argv[2];
|
||||
if (!runFile) {
|
||||
console.error('Usage: node rescore.mjs <run-file.jsonl>');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const absRun = resolve(process.cwd(), runFile);
|
||||
if (!existsSync(absRun)) {
|
||||
console.error(`File not found: ${absRun}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const lines = readFileSync(absRun, 'utf8').split('\n').filter(l => l.trim().length > 0);
|
||||
const rows = lines.map(l => JSON.parse(l));
|
||||
|
||||
const receipt = rows.find(r => r.kind === 'receipt');
|
||||
const runRows = rows.filter(r => r.kind === 'run');
|
||||
|
||||
console.error(`Re-scoring ${runRows.length} rows from ${absRun}`);
|
||||
console.error(`Receipt: model=${receipt?.model ?? '?'} fixtures_hash=${receipt?.fixtures_hash ?? '?'} ts=${receipt?.ts ?? '?'}`);
|
||||
|
||||
// Identify variants and load them
|
||||
const variantsUsed = [...new Set(runRows.map(r => r.variant))];
|
||||
const variantsDir = join(__dirname, 'variants');
|
||||
const dispatcherLists = {};
|
||||
for (const v of variantsUsed) {
|
||||
const path = join(variantsDir, `${v}.md`);
|
||||
if (!existsSync(path)) {
|
||||
console.error(`Warning: variant file missing for "${v}" at ${path} — lenient score will collapse to strict for this variant.`);
|
||||
dispatcherLists[v] = new Map();
|
||||
continue;
|
||||
}
|
||||
dispatcherLists[v] = parseDispatcherLists(readFileSync(path, 'utf8'));
|
||||
}
|
||||
|
||||
const SEEDS = [1, 2, 3];
|
||||
|
||||
const strictSummary = {};
|
||||
const lenientSummary = {};
|
||||
for (const v of variantsUsed) {
|
||||
strictSummary[v] = { training: [], held_out: [] };
|
||||
lenientSummary[v] = { training: [], held_out: [] };
|
||||
for (const corpus of ['training', 'held_out']) {
|
||||
for (const seed of SEEDS) {
|
||||
const subset = runRows.filter(r => r.variant === v && r.corpus === corpus && r.seed === seed);
|
||||
if (subset.length === 0) continue;
|
||||
strictSummary[v][corpus].push(subset.reduce((a, r) => a + r.correct, 0) / subset.length);
|
||||
const lenientHits = subset.reduce((a, r) => a + lenientScore(r.predicted, r.expected, dispatcherLists[v]), 0);
|
||||
lenientSummary[v][corpus].push(lenientHits / subset.length);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n=== Re-scored from ${runFile} ===\n`);
|
||||
console.log(' | STRICT scoring | LENIENT (same-area)');
|
||||
console.log('Variant | Held-out | Training | Held-out | Training');
|
||||
console.log('------------------------------|------------------------|------------------------|----------------------|----------------------');
|
||||
for (const v of variantsUsed) {
|
||||
console.log(
|
||||
`${v.padEnd(30)}| ${fmt(strictSummary[v].held_out).padEnd(22)} | ${fmt(strictSummary[v].training).padEnd(22)} | ${fmt(lenientSummary[v].held_out).padEnd(20)} | ${fmt(lenientSummary[v].training)}`,
|
||||
);
|
||||
}
|
||||
console.log('\nLENIENT counts a prediction correct if it shares a dispatcher area with expected.');
|
||||
console.log('For variants without "(dispatcher for: ...)" clauses, LENIENT == STRICT.');
|
||||
380
evals/functional-area-resolver/variants/baseline.md
Normal file
380
evals/functional-area-resolver/variants/baseline.md
Normal file
@@ -0,0 +1,380 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: BASELINE — 270-row bullet-list shape. Extracted from a production AGENTS.md at the pre-compression state; owner PII scrubbed. ~25KB. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner user shares info about themselves/work/vendors → `group-chat-intel`
|
||||
- Any brain read/write/lookup/citation → `brain-ops`
|
||||
- Any brain page write OR chat reply mentioning a repo/project → `brain-link-refs`
|
||||
- Any outbound reply to the owner that references a brain page or workspace file → `brain-link-report`
|
||||
- Any outbound report/alert with external links (oppo alerts → `report-quality-gate`
|
||||
- Any outbound reply in a multi-user group (floor scope < FULL) that references... → `brain-pdf-auto`
|
||||
- Any time-sensitive claim: "in N minutes" → `context-now`
|
||||
- the owner corrects a behavior, output, or decision → `correction-pipeline`
|
||||
- Presenting choices with inline buttons, user decision gate, button callback → `ask-user`
|
||||
|
||||
### Political donations
|
||||
- Donation tracking → `political-donations`
|
||||
|
||||
### Brain operations
|
||||
- Creating a new file - where does it go? → `repo-architecture`
|
||||
- Brain directory structure, "where is X in the brain", schema, filing rules → `/your/brain/path/README.md (directory tree + key locations table) + /your/brain/path/schema.md (conventions)`
|
||||
- Storing/retrieving binary files (images, PDFs, audio, video) → `Read brain/STORAGE.md - .redirect.yaml pointers + Supabase Storage`
|
||||
- Creating/enriching a person or company page → `enrich`
|
||||
- Resolving X handle stubs to real people ("who is @handle" → `x-handle-enrich`
|
||||
- Scoring/rating a person, rationalizing scores, "what score is X" → `person-score`
|
||||
- Unknown sender emails the owner → `cold-email-lookup`
|
||||
- Pitch deck, data room, financial model shared → `diligence`
|
||||
- Fix broken citations in brain pages → `citation-fixer`
|
||||
- Publish/share a brain page as link → `brain-publish`
|
||||
- Generate PDF from brain page, "brain pdf", "send me the pdf", … → `brain-pdf`
|
||||
- Generate PDF from any non-brain content: reports → `pdf-generation`
|
||||
- Read a book/article through lens of a specific problem, "read this through the lens", "extract a playbook", "what can I learn" → `strategic-reading`
|
||||
- Personalized book analysis, "book mirror", "apply this book", … → `book-mirror`
|
||||
- Deep-retrieval book mirror, "extreme mirror", "go deep", … → `book-mirror/SKILL.md (deep retrieval is now the default)`
|
||||
- Freshness check, data source SLA monitoring, smoke test → `freshness-monitor`
|
||||
- Write as the owner: blog posts → `garry-voice`
|
||||
- Essay review, writing feedback, draft review → `essay-review`
|
||||
- Brain search/query, hybrid search, entity lookup; Brain maintenance, lint, backlinks, health checks → `gbrain`
|
||||
- "My ChatGPT conversations" → `conversation-history`
|
||||
- Brain integrity → `brain-librarian`
|
||||
- "archive crawler", "mine my old files", … → `archive-crawler`
|
||||
- "concept synthesis", "intellectual map", … → `concept-synthesis`
|
||||
- "Ingest all X" → `bulk-skillify`
|
||||
- "extract takes", "seed takes", … → `takes-extraction`
|
||||
- Any ycli command, ycli SSO expired → `ycli-auth`
|
||||
- "extreme mirror", "go deep on this book", deep-retrieval book mirror → `book-mirror-extreme`
|
||||
- Book mirror synthesis, synthesize book analysis → `book-mirror-synthesis`
|
||||
- Export brain, download brain pages, brain backup → `brain-export`
|
||||
- Brain planning, plan brain changes, schema planning → `brain-plan`
|
||||
- Conversation enrichment, enrich chat transcript → `conversation-enrichment`
|
||||
- Fact check, verify claim, "is this true", citation check → `fact-check`
|
||||
- Upgrade gbrain, update gbrain, gbrain version → `gbrain-upgrade`
|
||||
- "Review my Dropbox archive", Dropbox folder audit, old Dropbox files → `dropbox-archive-review`
|
||||
- Screenshot style, apply style to screenshot → `screenshot-style`
|
||||
- Signorelli letter, draft formal letter → `signorelli-letter`
|
||||
- Data loss prevention, confirm bulk delete → `data-loss-gate`
|
||||
- Public repo PII guard, check for secrets → `public-repo-guard`
|
||||
|
||||
### Places & Travel
|
||||
- Trip itinerary PDF/doc → `trip-logistics`
|
||||
- "I'm at [place]"; "Where should I eat in X"; Foursquare/Swarm data export, bulk location import → `checkin`
|
||||
- "What's playing", "showtimes", … → `showtimes`
|
||||
|
||||
### Calendar (direct queries)
|
||||
- "What's my schedule", "am I free", calendar briefing, day lookahead → `google-calendar`
|
||||
- "Create a calendar item", "add to my calendar", … → `calendar-event-create`
|
||||
- "Prep for my meeting with X" → `meeting-prep`
|
||||
- Interview prep → `interview-prep`
|
||||
- Calendar conflict detection, double bookings, travel impossibility, missing prep; After calendar sync completes, or when day's schedule changes → `calendar-check`
|
||||
- Travel booking → `calendar-travel-setup`
|
||||
- Sync calendars to brain → `calendar-sync`
|
||||
- Historical/past calendar lookup: "when did I" → `calendar-recall`
|
||||
|
||||
### Time, location, and context
|
||||
- "What time is it" → `context-now`
|
||||
- "What's my jet lag plan" → `jet-lag`
|
||||
|
||||
### Executive assistant
|
||||
- Inbox triage, email reply, scheduling, calendar → `executive-assistant`
|
||||
- Gmail search, send email, draft reply via ClawVisor → `gmail`
|
||||
- Google Contacts lookup, search contacts, contact info → `google-contacts`
|
||||
- Personal logistics, schedule timeline, countdown deltas, time-aware foundation → `personal-logistics`
|
||||
- Intro health check, dropped handoffs, re-ping opportunities, intro tracker → `intro-reping`
|
||||
- Startup intro request, "draft an intro", evaluate intro, score intro quality → `startup-intro`
|
||||
- Alumni dinner planning, guest list curation, dinner invite list → `alumni-dinner`
|
||||
- "Partner lunch brief" → `partner-lunch-brief`
|
||||
- Flight delay tracking → `flight-tracker`
|
||||
- "Where is the owner", location inference, fix location, travel state machine → `location-inference`
|
||||
- Task add/remove/complete/defer/review → `daily-task-manager`
|
||||
- Morning task list prep (cron) → `daily-task-prep`
|
||||
- Business development, outreach tracking → `business-development`
|
||||
- Phone call handling (510-MY-GARRY) → `voice-agent`
|
||||
- Venus call ended, "Process this Venus call", voice session analysis → `voice-session-ingest`
|
||||
- Post-call analysis, "analyze the last call", "what happened on that call" → `venus-post-call`
|
||||
- "give me a link" → `voice-link`
|
||||
- OpenPhone/SMS (415-777-0000) → `quo`
|
||||
- "What's my jet lag plan" → `jet-lag`
|
||||
- New trip detected, trip itinerary shared, post-trip reflection, "trip is done" → `trip-ingest`
|
||||
|
||||
### Face detection & recognition
|
||||
- Face detect → `face-detect`
|
||||
- "identify faces" → `identify-faces`
|
||||
|
||||
### Content & media ingestion
|
||||
- Frame.io → `frameio-monitor`
|
||||
- "Ingest this", "save this to brain", generic content routing → `ingest`
|
||||
- the owner shares a link, article, tweet, idea → `idea-ingest`
|
||||
- Any video/audio (YouTube, X, Instagram, TikTok, podcast), "ingest this pdf book", "summarize this book", "process this book"; Screenshots, GitHub repos, other media → `media-ingest`
|
||||
- "Transcribe this" → `transcribe`
|
||||
- Book PDF, investor update PDF, any PDF to ingest → `pdf-ingest`
|
||||
- "Get me this book" → `book-acquisition`
|
||||
- Anna's Archive download, annas-archive, fast download with membership → `annas-archive`
|
||||
- Kindle library → `kindle-library`
|
||||
- Circleback CLI: search meetings → `circleback-cli`
|
||||
- Meeting transcript from Circleback → `meeting-ingestion`
|
||||
- Post-ingestion meeting summary to Meetings topic (auto-triggered by Circlebac... → `meeting-digest`
|
||||
- MANDATORY post-meeting audit, "audit this meeting" → `meeting-gold-standard`
|
||||
- Post-meeting signal extraction, "what did I say that was interesting", concept extraction → `meeting-signal-pass`
|
||||
- "scrape", "scrape <url>", … → `scrape`
|
||||
- Fundraising PDF → `fundraising-pdf`
|
||||
- Therapy session audio: "here's my jan/donna/marcie session" → `therapy-ingest`
|
||||
- Enriching any brain page from external content (quality pass) → `media-enrichment`
|
||||
- Batch article enrichment, "enrich", "raw content", "article dumps" → `article-enrichment`
|
||||
- Post-ingestion signal extraction, concept extraction from articles, backlink enrichment, entity propagation → `post-ingestion-enrichment`
|
||||
- Security audit (secrets, RLS, token files, gitleaks) → `security-audit`
|
||||
- Backlink check after any brain page write → `node scripts/backlink-check.mjs <page-path> — deterministic, run after EVERY brain page create/update`
|
||||
- X daily quality → `x-daily-quality`
|
||||
- ycli → `yc-ingest`
|
||||
- YC OH meeting notes, ycli office hours ingestion, "pull my YC meetings" → `yc-oh-ingest`
|
||||
- "Ingest this application" → `yc-app-ingest`
|
||||
- Company investor update, VC fund LP update, portfolio metrics email → `investor-update-ingest`
|
||||
- Voice note, audio message to transcribe and ingest, "voice memo", "audio note", "audio message" → `voice-note-ingest`
|
||||
- Save session transcripts to brain → `transcript-save`
|
||||
- "Unsubscribe from this", remove me from this list → `email-unsubscribe`
|
||||
- Deep web research, "research this person/topic thoroughly", "web research", … → `perplexity-research`
|
||||
- Exa semantic web search, find people/companies/LinkedIn profiles → `exa`
|
||||
- Happenstance professional network search, research people → `happenstance`
|
||||
- Crustdata B2B intelligence, LinkedIn enrichment, career history → `crustdata`
|
||||
- Captain API, Pitchbook data, funding rounds, investor lookup → `captain-api`
|
||||
- Structured data research, "track" → `data-research`
|
||||
- Substack ingest, import from Substack → `substack-ingest`
|
||||
- Pocket ingest, import from Pocket → `pocket-ingest`
|
||||
- Tweet deep ingest, deep tweet enrichment, article extraction from tweets → `tweet-deep-ingest`
|
||||
|
||||
### X/Twitter API - ENTERPRISE TIER
|
||||
**ALL X API work:** Read `skills/_x-api-rules.md` FIRST. We pay $50K/mo. Rate limit: 40K req/15min. Import `lib/x-api.mjs`. NEVER throttle to free-tier limits.
|
||||
|
||||
### Message intelligence
|
||||
- "Scan my DMs", "triage my messages", X DM triage, unified message extraction → `message-intel`
|
||||
- "Project Karma", blocked/muted users, adversary tweets, hostile accounts → `adversary-tracking`
|
||||
|
||||
### Monitoring & social
|
||||
- X/Twitter ingestion (daily, backfill, rollup, enrichment) → `x-ingest`
|
||||
- "x stream" → `svc/x-stream`
|
||||
- "Concept tier" → `x-concept-tier`
|
||||
- "look up tweet"; "social json store" → `social-json-store`
|
||||
- "storage tier"; "download video when needed" → `brain-storage`
|
||||
- "link to supabase file" → `brain-storage-links`
|
||||
- "backblaze" → `backblaze`
|
||||
- Social media mention alerts (cron) → `social-radar`
|
||||
- YC launch cringe-o-meter, YC media monitoring, YC sentiment, "scan YC launches" → `yc-media-monitor`
|
||||
- Slack channel scanning (cron) → `slack-scan`
|
||||
- Content idea generation (cron) → `content-ideas`
|
||||
- Check Steph's Instagram → `steph-instagram`
|
||||
|
||||
### Adversarial / research
|
||||
- Track/monitor a public figure or critic → `adversary-tracking`
|
||||
- Detect astroturfing, "is this organic", bot check, paid amplification → `detect-astroturf`
|
||||
- Real-name hostile identification, "who hates me", hostile account ID → `real-name-hostiles`
|
||||
- Deanonymize anon X account → `investigate-x-anon`
|
||||
- Fiscal forensics, government spending, nonprofit audit, 990 filings, grant fraud → `fiscal-forensics`
|
||||
- Academic claim verification, "verify this study", "is this replicated", … → `academic-verify`
|
||||
- Private investigation, deep background check, "find out everything about" → `private-investigator`
|
||||
- Opposition research backgrounder → `oppo-research`
|
||||
- OSINT collection on tracked individuals → `osint-collector`
|
||||
- Network mapping, relationship intelligence, who-knows-who → `network-intel`
|
||||
- YC competitor oppo → `yc-competitor-oppo`
|
||||
- Who's boosting competitors → `yc-booster-tracker`
|
||||
|
||||
### Product / building
|
||||
- "Review this plan" / "CEO review" / "think bigger" → `gstack-openclaw-ceo-review`
|
||||
- "Debug this" / "investigate" / "root cause" → `gstack-openclaw-investigate`
|
||||
- "Office hours" / "brainstorm" / "is this worth building" / startup advice / f... → `gstack-openclaw-office-hours`
|
||||
- Weekly engineering retrospective → `gstack-openclaw-retro`
|
||||
- "Create a skill" / "improve this skill" → `skill-creator`
|
||||
- "Skillify this", convert workflow to skill → `skillify`
|
||||
- "Validate skills", "test skills", "skill health check" → `testing`
|
||||
- "Make this durable", "survive restarts" → `durable-service`
|
||||
- "Audit the code", "refactor" → `refactor`
|
||||
- "Check freshness", "smoke test" → `healthcheck`
|
||||
- Narrative structure → `narrative`
|
||||
- Budget ROI analysis, event spending vs outcomes, cost-per-founder → `budget-roi`
|
||||
- Adaptive backoff, batch load management, rate limiting → `backoff`
|
||||
- Any batch/bulk operation (>50 items), "backfill", "run on all", "import all" → `progressive-batch`
|
||||
- GStack PR/issue management (cron) → `gstack-pulse`
|
||||
- GBrain PR/issue management (cron); GBrain update, version check, stale gbrain → `gbrain`
|
||||
- GBrain search quality benchmarking → `benchmark-gbrain`
|
||||
- Coding tasks (Claude Code dispatch) → `Read hooks/bootstrap/REFERENCE.md`
|
||||
- Cross-modal review, second opinion, adversarial challenge → `cross-modal-review`
|
||||
- Deterministic code failing on edge cases → `fail-improve-loop`
|
||||
- GStack Browser tasks (cron) → `browser-tasks`
|
||||
- Weekly essay, write essay, draft weekly piece → `weekly-essay`
|
||||
- Investigate no response, why didn't they reply, follow up analysis → `investigate-no-response`
|
||||
- Printing press, publish to distribution → `printing-press`
|
||||
|
||||
### Infrastructure
|
||||
- Sending ANY service URL to the owner, "is the tunnel up", verify endpoint → `ngrok-verify`
|
||||
- "Check cpu", "system load", …, resource usage → `system-load`
|
||||
- Container restart → `container-restart`
|
||||
- Zombie processes → `zombie-reaper`
|
||||
- Write to /tmp → `scratch-space`
|
||||
- ClawVisor service routing, Gmail/Calendar/Drive/Contacts/iMessage via ClawVisor → `clawvisor`
|
||||
- ClawVisor Shield proxy, credential vaulting, API audit → `clawvisor-shield`
|
||||
- "What crons are running", recurring jobs, cron audit, scheduled tasks → `recurring-jobs`
|
||||
- Work on a PR → `acp-coding`
|
||||
- PR workflow, git worktree, dev checkout, "build this feature" → `repo-dev`
|
||||
- Brain page commit/push, always push after brain writes → `brain-commit`
|
||||
- Brain links, clickable GitHub URLs, "link me to" → `brain-links`
|
||||
- GitHub repo lookup, "repo not found", clone/check repo existence, READ a repo → `github-repo`
|
||||
- GitHub WRITE: push → `github-agents`
|
||||
- gbrain PR content, anonymization, PR body for gbrain → `gbrain-pr`
|
||||
- CAPTCHA, DataDome, "verification required", slide to verify → `captcha-solver`
|
||||
- QR code generation, "make a QR code", scannable code → `qr-code`
|
||||
- Front API, front link, front conversation, front search → `front-api`
|
||||
- OAuth2 authorization, "connect my X/service account", callback server → `oauth-webhook`
|
||||
- Headless browser, form fill, web interaction → `browser`
|
||||
- Cloud browser automation → `browser-use`
|
||||
- "Bypass IP restriction" → `nordvpn-proxy`
|
||||
- Channel discovery, find channels, list channels → `channel-discovery`
|
||||
- Telegram test divert, test message routing → `telegram-test-divert`
|
||||
- GStack Browse headed+proxy, browser-native download, anti-bot browsing → `gstack-browse`
|
||||
- "Submit a shell job" → `gbrain skills/minion-orchestrator`
|
||||
- Start GStack Browser (headed, the owner's machine) → `Ask the owner to run gstack-browser and share pairing code`
|
||||
- Binary dep missing, shared library error, container restart → `binary-deps`
|
||||
- Match HTML to screenshot, pixel-perfect, visual comparison, CSS tuning → `pixel-match`
|
||||
- YC app investigation, YC application ingestion, "ingest this company", company 404 → `yc-app-ingest`
|
||||
- Email triage, inbox classification, cold pitch scoring, auto-archive → `email-triage`
|
||||
- Cold pitch scoring, rate this pitch, pitch quality → `cold-pitch-scorer`
|
||||
- Company oppo, competitive intel, investigate competitor → `company-oppo`
|
||||
- Cross-modal eval, compare models, model comparison → `cross-modal-eval`
|
||||
- Tweet reply, dunk, respond to troll, "don't respond to this" → `anti-dunk`
|
||||
- "Write a comeback", "roast this", aggressive reply draft → `clapback`
|
||||
- Tweet draft, compose tweet, write a tweet → `tweet-draft`
|
||||
- Tweet composition, draft tweet structure → `tweet-composition`
|
||||
- Tweet vulnerability scan, shield, check my tweet → `tweet-shield`
|
||||
- Journo dunk, journalist oppo, build dunk file → `journo-dunk`
|
||||
- Hater tracker, hostile engagement analysis → `hater-tracker`
|
||||
- Slack messages, slack search, slack DMs → `slack`
|
||||
- Voter guide, election research, candidate analysis → `voter-guide`
|
||||
- Voter guide data extraction → `voter-guide-extract`
|
||||
- Web archive, save page, preserve article, offline copy → `web-archive`
|
||||
- YC meeting recording, OH transcript ingestion → `yc-meeting-ingest`
|
||||
- Quote screenshot, article screenshot for tweet → `quote-screenshot`
|
||||
- Song lyrics, quote lyrics (content filter bypass) → `song-lyrics`
|
||||
- Voice call enrichment, post-call brain page → `voice-call-enrich`
|
||||
- Context health, bootstrap budget, resolver coverage → `context-health`
|
||||
- Daily question, personal question drip → `daily-question`
|
||||
- Stalker watch, threat monitoring, dangerous individual → `stalker-watch`
|
||||
- Idea registry, idea capture, "I have an idea" → `idea-registry`
|
||||
- File archive ingestion, Dropbox, Google Drive import → `file-archive-ingestion`
|
||||
- "skillpackify", PR to gbrain, open source this skill, add to skillpack → `skillpackify`
|
||||
- Restart sweep, dropped messages, missed messages after restart → `restart-sweep`
|
||||
- Neuromancer coordination, agent handoffs, inter-agent tasks, "hand off to Neuromancer" → `neuromancer-coordination`
|
||||
- Inter-agent coordination, "Owner's Agents" group chat, the agent+Neuromancer collaboration, agent task claiming, brain write protocol; Bot-to-bot communication, /curtain protocol, agent volley limits, bot-to-bot setup, how agents talk to each other → `inter-agent-coordination`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
146
evals/functional-area-resolver/variants/functional-areas.md
Normal file
146
evals/functional-area-resolver/variants/functional-areas.md
Normal file
@@ -0,0 +1,146 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: FUNCTIONAL-AREAS — the dispatcher pattern, extracted from a production AGENTS.md at the post-compression state; owner PII scrubbed. ~13KB. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner shares info → `group-chat-intel`
|
||||
- Brain read/write/lookup → `brain-ops`
|
||||
- Reply mentioning repo/project → `brain-link-refs`
|
||||
- Reply referencing brain page → `brain-link-report`
|
||||
- Report with external links → `report-quality-gate`
|
||||
- Multi-user group reply referencing brain → `brain-pdf-auto`
|
||||
- Time-sensitive claim → `context-now`
|
||||
- the owner corrects behavior → `correction-pipeline`
|
||||
- Inline buttons / user decision gate → `ask-user`
|
||||
|
||||
### Functional Areas
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing, citations, publishing, book analysis, strategic reading, concept synthesis, archive mining, conversation history → `brain-ops` (dispatcher for: enrich, query, brain-pdf, brain-publish, brain-export, brain-plan, brain-librarian, brain-commit, brain-storage, brain-storage-links, citation-fixer, repo-architecture, book-mirror, book-mirror-extreme, book-mirror-synthesis, strategic-reading, concept-synthesis, archive-crawler, conversation-history, conversation-enrichment, garry-voice, essay-review, fact-check, takes-extraction, gbrain, gbrain-upgrade, benchmark-gbrain, freshness-monitor, dropbox-archive-review, bulk-skillify, x-handle-enrich, person-score)
|
||||
- **Content ingestion**: ingest links/articles/PDFs/video/audio/tweets/books/meetings/voice notes, transcription, media enrichment → `ingest` (dispatcher for: media-ingest, meeting-ingestion, meeting-digest, meeting-gold-standard, meeting-signal-pass, voice-note-ingest, article-enrichment, post-ingestion-enrichment, media-enrichment, book-acquisition, annas-archive, pdf-ingest, tweet-deep-ingest, substack-ingest, pocket-ingest, investor-update-ingest, yc-ingest, yc-oh-ingest, yc-app-ingest, yc-meeting-ingest, kindle-library, therapy-ingest, transcript-save, file-archive-ingestion, idea-ingest)
|
||||
- **Calendar & scheduling**: schedule, events, conflicts, sync, prep, travel booking, time/location → `google-calendar` (dispatcher for: calendar-event-create, calendar-check, calendar-sync, calendar-recall, calendar-travel-setup, meeting-prep, interview-prep, context-now, jet-lag, location-inference)
|
||||
- **Email & comms**: inbox triage, email search/send, iMessage, Slack, unsubscribe, Front API → `executive-assistant` (dispatcher for: gmail, email-triage, email-unsubscribe, cold-email-lookup, cold-pitch-scorer, front-api, slack, intro-reping, startup-intro, investigate-no-response)
|
||||
- **Research & investigation**: web research, people/company lookup, LinkedIn, competitive intel, background checks → `perplexity-research` (dispatcher for: exa, happenstance, crustdata, captain-api, data-research, diligence, company-oppo, network-intel, private-investigator, oppo-research, academic-verify)
|
||||
- **X/Twitter & social**: tweets, social monitoring, adversary tracking, content strategy, DM triage → `x-ingest` (dispatcher for: adversary-tracking, social-radar, x-daily-quality, x-concept-tier, social-json-store, detect-astroturf, real-name-hostiles, investigate-x-anon, anti-dunk, clapback, tweet-draft, tweet-composition, tweet-shield, journo-dunk, hater-tracker, message-intel, yc-media-monitor, yc-competitor-oppo, yc-booster-tracker, steph-instagram, content-ideas)
|
||||
- **Places & travel**: checkins, restaurants, showtimes, trip logistics → `checkin` (dispatcher for: trip-logistics, trip-ingest, showtimes, personal-logistics)
|
||||
- **Product & building**: CEO review, code, debugging, skill creation, testing, refactoring, PR management → `acp-coding` (dispatcher for: gstack-openclaw-ceo-review, gstack-openclaw-investigate, gstack-openclaw-office-hours, gstack-openclaw-retro, skill-creator, skillify, testing, durable-service, refactor, narrative, budget-roi, fail-improve-loop, weekly-essay, printing-press, cross-modal-review, cross-modal-eval)
|
||||
- **Infrastructure**: tunnels, containers, services, crons, GitHub, browser automation, security → `healthcheck` (dispatcher for: ngrok-verify, system-load, container-restart, zombie-reaper, scratch-space, clawvisor, clawvisor-shield, recurring-jobs, github-repo, github-agents, gbrain-pr, captcha-solver, qr-code, browser, browser-use, gstack-browse, binary-deps, pixel-match, nordvpn-proxy, channel-discovery, durable-service, data-loss-gate, public-repo-guard, web-archive, security-audit)
|
||||
- **People & contacts**: Google contacts, face detection/identification, people enrichment → `google-contacts` (dispatcher for: face-detect, identify-faces, enrich)
|
||||
- **Tasks & logistics**: daily tasks, reminders, briefings, business dev, flight tracking, voice calls → `daily-task-manager` (dispatcher for: daily-task-prep, business-development, flight-tracker, voice-agent, voice-session-ingest, venus-post-call, voice-link, voice-call-enrich, quo, checkin)
|
||||
- **Political**: donation tracking, voter guides, civic intel → `political-donations` (dispatcher for: voter-guide, voter-guide-extract, fiscal-forensics)
|
||||
- **Inter-agent**: Neuromancer delegation, agent coordination → `inter-agent-coordination` (dispatcher for: neuromancer-coordination)
|
||||
- **Circleback**: meeting search → `circleback-cli`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
146
evals/functional-area-resolver/variants/resolver-of-resolvers.md
Normal file
146
evals/functional-area-resolver/variants/resolver-of-resolvers.md
Normal file
@@ -0,0 +1,146 @@
|
||||
<!-- A/B EVAL FIXTURE — synthetic resolver shape, do not invoke from agent context. -->
|
||||
<!-- Variant: RESOLVER-OF-RESOLVERS — functional-areas WITHOUT the '(dispatcher for: ...)' clauses. This is the variant the skill describes as 'broken' — pipe-table compression that loses sub-skill visibility. -->
|
||||
|
||||
# AGENTS.md
|
||||
|
||||
This folder is home. Treat it that way.
|
||||
|
||||
## Hard Gates (NEVER VIOLATE)
|
||||
|
||||
⛔ **RUNTIME CONTEXT > PROJECT DOCS.** When the OpenClaw runtime context block (Group Chat Context, Inbound Context, capabilities) contradicts a project doc rule, the runtime wins. The runtime knows the actual channel state for THIS turn; project docs are stale by definition. The 2026-05-06 silent-drop recurrence happened because I trusted a wrong HEARTBEAT rule over the correct runtime warning. Don't do that again.
|
||||
|
||||
⛔ **NEVER RESTART GATEWAY.** Tell the owner. He does it himself. No exceptions.
|
||||
|
||||
⛔ **BRAIN-FIRST STORAGE.** ALL valuable outputs → `/your/brain/path/` or Supabase IMMEDIATELY. Use `/your/tmp` for scratch (not `/tmp`). `/tmp` hard limit: 2GB. See `skills/conventions/brain-first.md`.
|
||||
|
||||
⛔ **DATA LOSS GATE.** Before ANY bulk delete: read `skills/data-loss-gate/SKILL.md`, present confirmation card, wait for "yes."
|
||||
|
||||
⛔ **NO WIKILINKS.** Standard markdown links only: `[Name](path)`. Never `[[wikilinks]]`.
|
||||
|
||||
⛔ **GBRAIN MASTER READ-ONLY.** Never push to master on <owner>/gbrain. Never merge PRs. Branch → push → PR only. See `skills/github-agents/SKILL.md`.
|
||||
|
||||
⛔ **PUBLIC REPO GUARD.** Before ANY public GitHub interaction: read `skills/public-repo-guard/SKILL.md`. Run PII scanner on ALL content.
|
||||
|
||||
⚡ **MINIONS OVER SUB-AGENTS.** Use gbrain Minions (shell jobs) for batch/deterministic work. Sub-agents only when LLM reasoning is required mid-task. Always set `--timeout-ms 900000` for long jobs.
|
||||
|
||||
## Gate -1 — Acknowledge Immediately
|
||||
|
||||
For any request taking >5 sec: send a one-line ack with rough time estimate FIRST, then start tools. Never go silent into a tool chain. Calibration: lookup ~10s, multi-tool ~30-60s, transcription ~2-3min, sub-agent ~1-3min, heavy batch ~3-5min, browser ~2-5min. Overestimate slightly.
|
||||
|
||||
For tasks >1 min: spawn a progress-update subagent (one-liner every 30-60s with concrete progress %). Critical in group topics with no typing indicator.
|
||||
|
||||
## Gate 0 — Access Control
|
||||
|
||||
On EVERY inbound message, check `sender_id` FIRST.
|
||||
- **the owner (<OWNER_ID_A> or <OWNER_ID_B>):** Proceed. Full access.
|
||||
- **Known non-the owner:** Read `skills/multi-user/SKILL.md` immediately. It governs everything.
|
||||
- **Unknown sender:** "This is a private agent." → notify the owner → stop.
|
||||
|
||||
## Gate 0.5 — Critical Life Events
|
||||
|
||||
If the owner mentions a **death, funeral, birth, hospitalization, emergency, diagnosis, accident, divorce, or arrest** — IMMEDIATELY write to BOTH `MEMORY.md` AND `memory/YYYY-MM-DD.md`. Priority 0. No deferral.
|
||||
|
||||
## Gate 1 — Signal Detection (the owner only)
|
||||
|
||||
Every the owner message: scan for entity mentions (people, companies, deals, YC batches). For each: search brain, load context, update if stale. Read `skills/entity-detector/ENTITY-DETECTION.md` for the full protocol.
|
||||
|
||||
**Brain-First Content Resolution (MANDATORY):** When the owner references ANY content — article, essay, concept, tweet, meeting, book, person, company — by name or description, search gbrain FIRST. Never ask "which article?" or "can you share the link?" The brain has 100K pages. Search it. Only ask the owner if gbrain + memory + web all fail.
|
||||
|
||||
## Gate 2 — Session Startup
|
||||
|
||||
Before first substantive reply:
|
||||
1. Read `ops/tasks.md` for task state
|
||||
2. Read `memory/heartbeat-state.json` for location, blockers, last checks
|
||||
3. Read relevant `memory/YYYY-MM-DD.md` for recent context
|
||||
4. Check calendar if time-sensitive
|
||||
|
||||
**Brain link rule:** Every brain path in output MUST be a clickable GitHub URL: `[name](https://github.com/<owner>/brain/blob/main/path.md)`. Never bare paths. Never invented URLs. `<owner>.github.io/brain/` does NOT exist.
|
||||
|
||||
**After every brain write:** `bash scripts/brain-commit-link.sh "<message>"`. Always absolute paths for brain writes (`/your/brain/path/...`).
|
||||
|
||||
**Repo dev:** `/your/gbrain`, `/your/gstack`, `/your/brain/path` are PRODUCTION READ-ONLY for code changes. All dev work → `/your/git-projects/<repo>-<feature>/`. See `skills/repo-dev/SKILL.md`.
|
||||
|
||||
## Gate 3 — Outbound Link Gate
|
||||
|
||||
Before EVERY reply containing a brain reference:
|
||||
1. Path must be absolute GitHub URL
|
||||
2. Commit must be pushed (not just local)
|
||||
3. Use `brain-commit-link.sh` output for the URL
|
||||
4. Never invent URLs. Never use `<owner>.github.io`.
|
||||
|
||||
## Skill Resolver
|
||||
|
||||
Read the skill file before acting. If two could match, read both. Non-the owner senders: only WORK/FAMILY-accessible skills.
|
||||
|
||||
### Always-on (every message)
|
||||
- Gate -1: any request taking >5 sec → `acknowledge`
|
||||
- Gate 0: sender_id != the owner → `multi-user`
|
||||
- Gate 1: the owner messages only → `entity-detector`
|
||||
- Non-the owner shares info → `group-chat-intel`
|
||||
- Brain read/write/lookup → `brain-ops`
|
||||
- Reply mentioning repo/project → `brain-link-refs`
|
||||
- Reply referencing brain page → `brain-link-report`
|
||||
- Report with external links → `report-quality-gate`
|
||||
- Multi-user group reply referencing brain → `brain-pdf-auto`
|
||||
- Time-sensitive claim → `context-now`
|
||||
- the owner corrects behavior → `correction-pipeline`
|
||||
- Inline buttons / user decision gate → `ask-user`
|
||||
|
||||
### Functional Areas
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing, citations, publishing, book analysis, strategic reading, concept synthesis, archive mining, conversation history → `brain-ops`
|
||||
- **Content ingestion**: ingest links/articles/PDFs/video/audio/tweets/books/meetings/voice notes, transcription, media enrichment → `ingest`
|
||||
- **Calendar & scheduling**: schedule, events, conflicts, sync, prep, travel booking, time/location → `google-calendar`
|
||||
- **Email & comms**: inbox triage, email search/send, iMessage, Slack, unsubscribe, Front API → `executive-assistant`
|
||||
- **Research & investigation**: web research, people/company lookup, LinkedIn, competitive intel, background checks → `perplexity-research`
|
||||
- **X/Twitter & social**: tweets, social monitoring, adversary tracking, content strategy, DM triage → `x-ingest`
|
||||
- **Places & travel**: checkins, restaurants, showtimes, trip logistics → `checkin`
|
||||
- **Product & building**: CEO review, code, debugging, skill creation, testing, refactoring, PR management → `acp-coding`
|
||||
- **Infrastructure**: tunnels, containers, services, crons, GitHub, browser automation, security → `healthcheck`
|
||||
- **People & contacts**: Google contacts, face detection/identification, people enrichment → `google-contacts`
|
||||
- **Tasks & logistics**: daily tasks, reminders, briefings, business dev, flight tracking, voice calls → `daily-task-manager`
|
||||
- **Political**: donation tracking, voter guides, civic intel → `political-donations`
|
||||
- **Inter-agent**: Neuromancer delegation, agent coordination → `inter-agent-coordination`
|
||||
- **Circleback**: meeting search → `circleback-cli`
|
||||
|
||||
**Internal data-source skills** (called by other skills, not directly): captain-api, crustdata, exa, happenstance, gmail, google-calendar, google-contacts, slack, clawvisor
|
||||
|
||||
|
||||
## Neuromancer Delegation (Cross-Topic)
|
||||
|
||||
**In ANY topic**, if a task would benefit from Neuromancer's capabilities, delegate it by posting a `[TASK]` message to the "Owner's Agents" group (thread 1, group -<GROUP_ID>).
|
||||
|
||||
**Neuromancer is good at:** Web research, browser automation, coding/PRs, X posting (via xurl), Google Workspace ops, on-demand analysis, skill building.
|
||||
|
||||
**the agent keeps:** Brain DB, cron/scheduled ops, X API (Enterprise keys), email sweeps (ClawVisor), memory consolidation, social radar, embedding/indexing.
|
||||
|
||||
**Protocol:** Prefix structured messages with `[TASK]`, `[RESULT]`, or `[QUERY]`. Neuromancer monitors the topic in real-time. Include enough context that Neuromancer can act without asking follow-ups. Reference brain pages by path.
|
||||
|
||||
**Don't delegate silently.** If the owner asked for something in another topic and you're handing it to Neuromancer, tell the owner in that topic: "Handing this to Neuromancer" with a one-liner on what you asked for.
|
||||
|
||||
## Memory (Operational)
|
||||
|
||||
- `MEMORY.md` — permanent, cross-session state. Keep tight. Flush to `memory/YYYY-MM-DD.md` daily.
|
||||
- `memory/YYYY-MM-DD.md` — daily operational memory. Append-only per day.
|
||||
- `memory/heartbeat-state.json` — structured state (location, wake status, last checks, blockers).
|
||||
- Brain (`/your/brain/path/`) — permanent knowledge (people, companies, deals, meetings, projects).
|
||||
|
||||
## Operating Rules
|
||||
|
||||
For the full set of operating principles, sub-agent rules, testing conventions, style guide, coding task protocols, and group chat rules: **read `skills/_operating-rules.md`**.
|
||||
|
||||
Key rules always in effect:
|
||||
- **Tests ship with code.** No PR without tests. No skip. See the full principle in the reference.
|
||||
- **Test before bulk.** Read `skills/progressive-batch/SKILL.md` for any operation touching >50 items. Progressive ramp: 10 → verify output exists → 100 → verify → 500 → verify → full. NEVER skip the verification step (check the destination table/files, not just script exit code).
|
||||
- **Fix tools, don't work around them.** If a tool is broken, fix it.
|
||||
- **Present options, then STOP.** For ambiguous requests, present 2-3 options. Don't pick one silently.
|
||||
- **Durable MECE skills.** Every repeated workflow → a skill. DRY across skills.
|
||||
- **GStack for coding PRs.** Read `skills/acp-coding/SKILL.md` for Claude Code / Codex integration.
|
||||
|
||||
## Coding Tasks — GStack Integration
|
||||
|
||||
Coding on gstack/gbrain/GL/any dev project: read `skills/acp-coding/SKILL.md`, spawn Codex via ACP, give full context, monitor+relay. Slash: `/code`, `/codex`, `/ship`, `/qa`, `/review`, `/investigate`.
|
||||
|
||||
<!-- gbrain:skillpack:begin -->
|
||||
<!-- Installed by gbrain 0.25.1. All 35 skills in this pack are already referenced in the resolver tables above. -->
|
||||
<!-- gbrain:skillpack:manifest cumulative-slugs="academic-verify,archive-crawler,article-enrichment,book-mirror,brain-ops,brain-pdf,briefing,citation-fixer,concept-synthesis,cron-scheduler,cross-modal-review,daily-task-manager,daily-task-prep,data-research,enrich,idea-ingest,ingest,maintain,media-ingest,meeting-ingestion,minion-orchestrator,perplexity-research,query,repo-architecture,reports,signal-detector,skill-creator,skillify,skillpack-check,soul-audit,strategic-reading,testing,voice-note-ingest,webhook-transforms" version="0.25.1" -->
|
||||
<!-- gbrain:skillpack:end -->
|
||||
@@ -1745,6 +1745,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Save or load reports | `skills/reports/SKILL.md` |
|
||||
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
|
||||
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
|
||||
| "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` |
|
||||
| "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` |
|
||||
| Post-restart health + auto-fix, "did the container restart break anything", smoke test | `skills/smoke-test/SKILL.md` |
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.25.1",
|
||||
"version": "0.32.3.0",
|
||||
"description": "Personal knowledge brain with Postgres + pgvector hybrid search",
|
||||
"family": "bundle-plugin",
|
||||
"configSchema": {
|
||||
@@ -39,6 +39,7 @@
|
||||
"skills/daily-task-prep",
|
||||
"skills/data-research",
|
||||
"skills/enrich",
|
||||
"skills/functional-area-resolver",
|
||||
"skills/idea-ingest",
|
||||
"skills/ingest",
|
||||
"skills/maintain",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.32.2",
|
||||
"version": "0.32.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -55,6 +55,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Save or load reports | `skills/reports/SKILL.md` |
|
||||
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
|
||||
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
|
||||
| "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` |
|
||||
| "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` |
|
||||
| Post-restart health + auto-fix, "did the container restart break anything", smoke test | `skills/smoke-test/SKILL.md` |
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
|
||||
348
skills/functional-area-resolver/SKILL.md
Normal file
348
skills/functional-area-resolver/SKILL.md
Normal file
@@ -0,0 +1,348 @@
|
||||
---
|
||||
name: functional-area-resolver
|
||||
version: 1.0.0
|
||||
prompt_version: 1
|
||||
description: |
|
||||
Compress an agent's routing file (RESOLVER.md or AGENTS.md) by converting
|
||||
granular skill-per-row tables into functional-area dispatchers. Each area
|
||||
lists sub-skills in a "(dispatcher for: ...)" clause. The LLM reads one
|
||||
area entry and routes to the correct sub-skill. Proven via held-out
|
||||
A/B eval: dispatcher pattern outperforms naive pipe-table compression.
|
||||
triggers:
|
||||
- "compress agents.md"
|
||||
- "compress my resolver"
|
||||
- "resolver too big"
|
||||
- "resolver.md too big"
|
||||
- "agents.md too large"
|
||||
- "shrink routing table"
|
||||
- "slim down agents.md"
|
||||
- "functional area resolver"
|
||||
- "functional area dispatcher"
|
||||
- "context-health agents"
|
||||
- "context-health resolver"
|
||||
- "reduce context budget"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
- write
|
||||
- edit
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Functional-Area Resolver — Pattern for Compressing Routing Tables
|
||||
|
||||
## Problem
|
||||
|
||||
Routing files (RESOLVER.md, AGENTS.md) grow as skills are added. Each skill
|
||||
gets its own row (trigger -> skill path). At ~200+ skills this hits 25-30KB,
|
||||
eating context budget that should go to actual work.
|
||||
|
||||
## Solution: Functional-Area Dispatchers
|
||||
|
||||
Replace N rows per area with **one entry per functional area**. Each entry
|
||||
lists all sub-skills it can dispatch to in a `(dispatcher for: ...)` clause.
|
||||
|
||||
### Before (270 rows, 25KB)
|
||||
```
|
||||
- Creating/enriching a person or company page -> `enrich`
|
||||
- Fix broken citations in brain pages -> `citation-fixer`
|
||||
- Publish/share a brain page as link -> `brain-publish`
|
||||
- Generate PDF from brain page -> `brain-pdf`
|
||||
- Read a book through lens of a problem -> `strategic-reading`
|
||||
- Personalized book analysis -> `book-mirror`
|
||||
- Brain integrity -> `brain-librarian`
|
||||
...
|
||||
```
|
||||
|
||||
### After (13 rows, 13KB)
|
||||
```
|
||||
- **Brain & knowledge**: create/enrich/search/export brain pages, filing,
|
||||
citations, publishing, book analysis, strategic reading, concept synthesis,
|
||||
archive mining -> `brain-ops` (dispatcher for: enrich, query, brain-pdf,
|
||||
brain-publish, brain-export, brain-librarian, citation-fixer, book-mirror,
|
||||
strategic-reading, concept-synthesis, archive-crawler, ...)
|
||||
```
|
||||
|
||||
## Why It Works
|
||||
|
||||
The LLM doesn't need one row per sub-skill. It needs:
|
||||
1. **Area recognition** — "this is about brain pages" -> Brain & Knowledge
|
||||
2. **Sub-skill visibility** — the `(dispatcher for: ...)` list shows what's available
|
||||
3. **The skill file itself** — once the LLM reads `brain-ops/SKILL.md`, it has full routing detail
|
||||
|
||||
This is a **two-layer dispatch**: routing file routes to the area, the area
|
||||
skill routes to the specific sub-skill. Each layer does one job well.
|
||||
|
||||
## A/B Eval Results
|
||||
|
||||
Three resolver architectures tested across three Anthropic frontier models
|
||||
(Opus 4.7, Sonnet 4.6, Haiku 4.5) on real production AGENTS.md content,
|
||||
20 hand-authored training fixtures + 5 held-out blind fixtures, n=3 seeded
|
||||
repeats per (fixture, variant). Two scoring rules: **STRICT** (predicted
|
||||
slug exactly equals expected) and **LENIENT** (predicted is in the same
|
||||
dispatcher area as expected). Both matter:
|
||||
|
||||
- STRICT measures: "does the LLM return the exact slug?"
|
||||
- LENIENT measures: "does the LLM land in the right area, even if it picks a
|
||||
more-specific sub-skill from `(dispatcher for: ...)`?" This is closer to
|
||||
production behavior — an agent that lands in `gmail` for an email intent
|
||||
succeeds even if the resolver entry said `executive-assistant`.
|
||||
|
||||
### Training corpus (n=20, 3 seeds × 3 variants × 3 models, LENIENT)
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline (270 bullet rows) | 81.7% ± 7.2% | 86.7% ± 7.2% | 73.3% ± 7.2% | 25KB |
|
||||
| **functional-areas** (this pattern) | **98.3% ± 7.2%** | **100% ± 0%** | **88.3% ± 7.2%** | **13KB** |
|
||||
| resolver-of-resolvers (no dispatcher clause) | 63.3% ± 14.3% | 41.7% ± 7.2% | 65.0% ± 12.4% | 10KB |
|
||||
|
||||
### Held-out blind corpus (n=5, 3 seeds, LENIENT)
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 |
|
||||
|---|---|---|---|
|
||||
| baseline | 100% ± 0% | 100% ± 0% | 100% ± 0% |
|
||||
| **functional-areas** | **100% ± 0%** | **100% ± 0%** | **100% ± 0%** |
|
||||
| resolver-of-resolvers | 100% ± 0% | **73.3% ± 28.7%** | 100% ± 0% |
|
||||
|
||||
### What the data shows
|
||||
|
||||
1. **Functional-areas BEATS baseline on training across all three models** (+13 to +17pp) at 48% the size. Held-out is saturated at 100% for both — within margin of error.
|
||||
|
||||
2. **The `(dispatcher for: ...)` clause is the load-bearing signal.** resolver-of-resolvers strips that clause and collapses to 41.7% on Sonnet — the catastrophic failure case the original PR predicted, now observed.
|
||||
|
||||
3. **The pattern works because the LLM can drill into the dispatcher list.** Most "STRICT failures" are the LLM picking a more-specific sub-skill (`gmail` instead of `executive-assistant`). That's the pattern working as designed. STRICT scoring under-counts; LENIENT scoring reflects production agent behavior.
|
||||
|
||||
4. **The pattern's value scales with model tier.** Compression gain (functional-areas vs baseline, training, LENIENT) is +17pp on Opus, +13pp on Sonnet, +15pp on Haiku. Sonnet shows the cleanest separation between functional-areas and resolver-of-resolvers (100% vs 41.7%) — model capacity affects how much the dispatcher signal matters.
|
||||
|
||||
### Reproduce
|
||||
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
node harness.mjs --model opus # ~225 LLM calls, ~$1.70 at Opus pricing
|
||||
node harness.mjs --model sonnet # ~$1.00
|
||||
node harness.mjs --model haiku # ~$0.30
|
||||
node rescore.mjs baseline-runs/2026-05-11-opus-4-7.jsonl # zero-cost re-score
|
||||
```
|
||||
|
||||
Receipts (model, prompt_template_hash, fixtures_hash, harness_sha, ts):
|
||||
`evals/functional-area-resolver/baseline-runs/2026-05-11-{opus-4-7,sonnet-4-6,haiku-4-5}.jsonl`.
|
||||
|
||||
### Methodology caveats
|
||||
|
||||
- **Production prompt matters.** With a naive "return the skill slug" prompt
|
||||
(no instruction about `(dispatcher for: ...)`), every compression variant
|
||||
collapses to ~30-60% on Opus. The dispatcher-aware prompt is in
|
||||
`evals/functional-area-resolver/harness-runner.ts:PROMPT_TEMPLATE`. Use it
|
||||
as the template for your agent's harness; without it, compression breaks.
|
||||
- **Training corpus and variants were authored by the same release.** Held-out
|
||||
corpus was written before the variants and never adjusted; this mitigates
|
||||
but does not eliminate overfitting.
|
||||
- **Confidence intervals via t-distribution across n=3 seeded repeats.** Hold the
|
||||
n=3 lower-bound: high CIs mean the underlying sample is noisy.
|
||||
- **Single-vendor result.** All three models are Anthropic. Cross-vendor
|
||||
verification (Gemini, GPT) is a v0.33.x follow-up.
|
||||
- **Held-out blind set is small (n=5).** Saturated at 100% across most cells —
|
||||
the harness can't distinguish between "100%" and "95% with one nondeterministic
|
||||
miss." Expanding to ≥20 is a v0.33.x follow-up.
|
||||
|
||||
### Prior work and citations
|
||||
|
||||
The pattern is a **static-prompt analog of hierarchical agent routing**, a
|
||||
2024-2025 research direction:
|
||||
|
||||
- **AnyTool** ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) showed
|
||||
meta-agent → category-agent → tool-agent hierarchy on 16K APIs beats flat
|
||||
retrieval by +35.4pp. The `(dispatcher for: ...)` clause is the
|
||||
meta-agent's view collapsed into a single LLM pass.
|
||||
- **RAG-MCP** ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1))
|
||||
reports 49.2% prompt-token reduction at 3.2× accuracy gain via
|
||||
embedding-based pre-retrieval. The token-reduction story matches ours
|
||||
(48% smaller), via a different mechanism (RAG vs static dispatcher).
|
||||
- **Anthropic Agent Skills**
|
||||
([engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills))
|
||||
promotes progressive disclosure: frontmatter (~80 tokens) always loaded,
|
||||
SKILL.md body loaded on match. This skill applies the same principle at
|
||||
the routing-table level, not the per-skill body level.
|
||||
|
||||
The 2025-2026 literature has no published benchmark for **static-prompt
|
||||
hierarchical routing** (every published hierarchical scheme resolves the
|
||||
hierarchy at runtime via a second LLM call). Our finding — that the
|
||||
hierarchy can be inlined into a single-LLM-pass dispatcher list and retain
|
||||
routing accuracy — is the open contribution. See
|
||||
`evals/functional-area-resolver/README.md` for methodology details.
|
||||
|
||||
## How To Compress
|
||||
|
||||
### Step 1: Preconditions
|
||||
|
||||
Refuse to compress if either gate fails:
|
||||
- Source routing file is under 12KB (compression overhead exceeds benefit).
|
||||
- `git status` shows uncommitted changes to the routing file (the
|
||||
compressor's edit would entangle with whatever the user was doing).
|
||||
|
||||
If a user wants to override either gate, they ask explicitly with `--force`.
|
||||
|
||||
### Step 2: When to compress which file
|
||||
|
||||
GBrain workspaces often have TWO routing files merged at runtime (per
|
||||
`src/core/check-resolvable.ts` v0.31.7): `skills/RESOLVER.md` and a sibling
|
||||
`../AGENTS.md`. Choose which to compress:
|
||||
|
||||
- Only one is fat (>12KB): compress that one; leave the small one alone.
|
||||
- Both are fat: compress them separately, in order: AGENTS.md first
|
||||
(usually the larger one in OpenClaw-style deployments), then RESOLVER.md.
|
||||
- Only the small one is fat (rare): same rule — compress it.
|
||||
|
||||
If the deployment uses only one routing file, this section is a no-op —
|
||||
compress that one.
|
||||
|
||||
### Step 3: Identify functional areas
|
||||
|
||||
Group skills by domain. Typical areas (adjust per deployment):
|
||||
|
||||
- **Brain & Knowledge** — brain-ops as dispatcher
|
||||
- **Content Ingestion** — ingest as dispatcher
|
||||
- **Calendar & Scheduling** — google-calendar as dispatcher
|
||||
- **Email & Comms** — executive-assistant as dispatcher
|
||||
- **Research & Investigation** — perplexity-research as dispatcher
|
||||
- **X/Twitter & Social** — x-ingest as dispatcher
|
||||
- **Places & Travel** — checkin as dispatcher
|
||||
- **Product & Building** — acp-coding as dispatcher
|
||||
- **Infrastructure** — healthcheck as dispatcher
|
||||
- **Tasks & Logistics** — daily-task-manager as dispatcher
|
||||
- **People & Contacts** — google-contacts as dispatcher
|
||||
|
||||
### Step 4: Build the area entry format
|
||||
|
||||
Each area entry follows this template:
|
||||
|
||||
```
|
||||
- **{Area Name}**: {comma-separated trigger phrases} -> `{dispatcher-skill}`
|
||||
(dispatcher for: {comma-separated sub-skill names})
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Trigger phrases should be broad enough to catch intent ("brain pages, enrich,
|
||||
search, filing, citations, book analysis")
|
||||
- Sub-skill list should be comprehensive — this is how the LLM knows what's available
|
||||
- The dispatcher skill file should have its own internal routing table
|
||||
|
||||
### Step 5: Keep always-on entries separate
|
||||
|
||||
Gates and always-on entries (acknowledge, multi-user, entity-detector, etc.)
|
||||
stay as individual rows — they're checked on every message, not dispatched.
|
||||
|
||||
### Step 6 (MANDATORY): Verify routing accuracy
|
||||
|
||||
Run two gates before committing the compressed file. Do NOT commit if either
|
||||
fails.
|
||||
|
||||
**Gate 1: Structural verification.** Confirms your `routing-eval.jsonl`
|
||||
fixtures still resolve to the right skills under the compressed routing file.
|
||||
Run from the workspace whose routing file you just edited:
|
||||
|
||||
```bash
|
||||
gbrain routing-eval --json
|
||||
```
|
||||
|
||||
If accuracy on your fixtures drops below 95%, revert and tune the area
|
||||
entries before re-running.
|
||||
|
||||
**Gate 2: LLM A/B verification on YOUR edited file.** Confirms a frontier
|
||||
LLM can still drill into the dispatcher list and reach sub-skills under
|
||||
your specific compression. Requires a gbrain repo checkout because the
|
||||
harness lives there. Copy your edited routing file into the harness's
|
||||
variants directory, then invoke the harness with `--variants` pointing
|
||||
at it:
|
||||
|
||||
```bash
|
||||
# In your agent workspace, identify the routing file you just compressed.
|
||||
EDITED=/path/to/your/AGENTS.md # or skills/RESOLVER.md, whichever you edited
|
||||
|
||||
# In your gbrain repo checkout:
|
||||
cd /path/to/gbrain/evals/functional-area-resolver
|
||||
TMP=$(mktemp -d)/variants && mkdir -p "$TMP"
|
||||
cp "$EDITED" "$TMP/my-edit.md"
|
||||
|
||||
# Run the harness against your file (sequential, ~75 calls × $0.0076 ≈ $0.57 on Opus).
|
||||
ANTHROPIC_API_KEY=... node harness.mjs --variants-dir "$TMP" --variants my-edit \
|
||||
--model opus --parallel 3 --yes
|
||||
```
|
||||
|
||||
The harness uses gbrain's bundled fixture set, so this verifies "did the LLM
|
||||
land in the right sub-skill for routing intents the gbrain-bundled fixtures
|
||||
cover" — a regression check on shared skills, not a full re-eval of YOUR
|
||||
fixture set. For full eval coverage, mirror this skill's
|
||||
`fixtures.jsonl` + `fixtures-held-out.jsonl` setup with intents specific
|
||||
to your skills.
|
||||
|
||||
If the lenient (same-area) score on your variant drops below 95%, revert the
|
||||
compression and tune. Common causes:
|
||||
- A sub-skill was omitted from the `(dispatcher for: ...)` list.
|
||||
- Trigger phrases for an area are too narrow (LLM can't recognize intent).
|
||||
- Areas were collapsed too aggressively (too few areas — see Anti-Patterns).
|
||||
- ASCII `->` vs Unicode `→` mismatch — the harness now accepts both, but
|
||||
earlier versions only matched Unicode. Pin gbrain to v0.32.3.0+.
|
||||
|
||||
Common false negatives on the harness eval (NOT bugs in your compression):
|
||||
- The gbrain-bundled fixtures target skill names like `enrich`, `query`,
|
||||
`gmail`, `executive-assistant`. If your routing file doesn't expose
|
||||
those skills at all, expect strict-scoring failures on those fixtures.
|
||||
Lenient scoring stays accurate for any sub-skill present in your
|
||||
`(dispatcher for: ...)` lists.
|
||||
|
||||
### Step 7: Review the diff before committing
|
||||
|
||||
Show the user the proposed edit (or the actual git diff) and wait for
|
||||
explicit approval before staging. Same convention as `skills/book-mirror/SKILL.md`.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
|
||||
- Routing matches the canonical triggers in the frontmatter.
|
||||
- Compression is only performed when the preconditions in Step 1 pass (file ≥12KB AND clean working tree, or `--force`).
|
||||
- The mandatory verification gate in Step 6 fires on the user's edited file, not on sample variants. The user runs `gbrain routing-eval --json` AND the gbrain-repo harness (`node harness.mjs --variants-dir <tmp> --variants my-edit`) before committing the compressed file.
|
||||
- Privacy contract preserved: no fork-specific filesystem path literals (`/data/brain/`, `/data/.openclaw/`) leak into the compressed output.
|
||||
|
||||
The full behavior contract is documented in the body sections above; this section exists for the conformance test.
|
||||
|
||||
## Output Format
|
||||
|
||||
The compressed routing file follows the area-entry template documented in Step 4 ("Build the area entry format"). Each entry: `- **{Area Name}**: {trigger phrases} -> \`{dispatcher-skill}\` (dispatcher for: {sub-skill list})`. The dispatcher arrow may be either ASCII `->` (default in this template) or Unicode `→` (used in some production deployments); the gbrain harness accepts both.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Resolver-of-resolvers with pipe tables.** Tested and failed (see eval
|
||||
table). The LLM picks area names from the table instead of drilling into
|
||||
sub-skills.
|
||||
|
||||
- **Removing sub-skill names.** Without the `(dispatcher for: ...)` list,
|
||||
the LLM can't route to specific sub-skills. The list is the routing signal.
|
||||
|
||||
- **Too few areas.** Collapsing to <5 areas makes each area too broad.
|
||||
12-15 areas is the sweet spot.
|
||||
|
||||
- **Too many areas.** Defeats the purpose. If you have 50 areas, just keep
|
||||
individual rows.
|
||||
|
||||
## Maintenance
|
||||
|
||||
When adding a new skill:
|
||||
1. Identify its functional area.
|
||||
2. Add the skill name to that area's `(dispatcher for: ...)` list.
|
||||
3. Update the area's skill file with routing detail.
|
||||
4. Run the routing eval (Step 6) to verify.
|
||||
|
||||
When adding a new functional area:
|
||||
1. Create the dispatcher skill with internal routing.
|
||||
2. Add the area entry to the routing file.
|
||||
3. Run the routing eval (Step 6) to verify.
|
||||
|
||||
## Changelog
|
||||
|
||||
### v1.0.0 — 2026-05-11
|
||||
- Initial version. Pattern shipped in gbrain v0.32.3.0 with a held-out A/B
|
||||
eval (see `evals/functional-area-resolver/`).
|
||||
- Skill renamed from `compress-agents-md` to `functional-area-resolver`
|
||||
pre-release; the contribution is the pattern, not the filename.
|
||||
23
skills/functional-area-resolver/routing-eval.jsonl
Normal file
23
skills/functional-area-resolver/routing-eval.jsonl
Normal file
@@ -0,0 +1,23 @@
|
||||
// Routing eval fixtures for skills/functional-area-resolver. Each
|
||||
// positive-intent fixture contains at least one trigger string from the
|
||||
// skill's RESOLVER.md row as substring (structural matcher requirement
|
||||
// in src/core/routing-eval.ts:170).
|
||||
// Adversarial negative fixtures at the bottom guard against the
|
||||
// broadened triggers (D5:B) over-capturing intents that belong to
|
||||
// adjacent meta-skills like skillify, skill-creator, book-mirror,
|
||||
// concept-synthesis.
|
||||
{"intent":"My AGENTS.md too large at 30KB and hitting context limits, how do I shrink it","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"The daily doctor says context-health is red because AGENTS.md too large","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"How do I compress my resolver without losing routing accuracy","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"RESOLVER.md too big — convert my 200-row skill resolver into functional areas","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"What's the functional area dispatcher pattern for AGENTS.md","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"My RESOLVER.md too big at 25KB, how do I shrink it","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"I want to compress my resolver while keeping all the sub-skills reachable","expected_skill":"functional-area-resolver"}
|
||||
{"intent":"Explain the functional area dispatcher pattern and when to use it","expected_skill":"functional-area-resolver"}
|
||||
// Adversarial negatives. These intents pattern-match the broadened
|
||||
// triggers ("compress my resolver", "shrink routing table", etc.) but
|
||||
// the correct route is the target skill, not functional-area-resolver.
|
||||
{"intent":"Skillify this — make this proper from the routing-pattern notes","expected_skill":"skillify","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Create a skill that compacts a routing file using AI","expected_skill":"skill-creator","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Personalized version of this book about resolver and dispatcher design","expected_skill":"book-mirror","ambiguous_with":["functional-area-resolver"]}
|
||||
{"intent":"Synthesize my concepts about how routing files grow over time","expected_skill":"concept-synthesis","ambiguous_with":["functional-area-resolver"]}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.25.1",
|
||||
"version": "0.32.3.0",
|
||||
"conformance_version": "1.0.0",
|
||||
"description": "Personal knowledge brain with hybrid RAG search \u2014 GStack mod for agent platforms",
|
||||
"skills": [
|
||||
@@ -208,6 +208,11 @@
|
||||
"name": "ask-user",
|
||||
"path": "ask-user/SKILL.md",
|
||||
"description": "Reusable choice-gate pattern for presenting users with 2-4 options and stopping execution until they respond. Platform-agnostic (Telegram buttons, Discord, CLI, OpenClaw clarify tool)."
|
||||
},
|
||||
{
|
||||
"name": "functional-area-resolver",
|
||||
"path": "functional-area-resolver/SKILL.md",
|
||||
"description": "Compress an agent's routing file (RESOLVER.md or AGENTS.md) by replacing skill-per-row tables with functional-area dispatcher entries. Two-layer dispatch keeps every sub-skill reachable at ~50% of the file size."
|
||||
}
|
||||
],
|
||||
"dependencies": {
|
||||
|
||||
Reference in New Issue
Block a user