triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
258 Commits 2 Branches 0 Tags
8ab733471bcefcd56d6ad4fd7b5eec0c12da366d
Commit Graph

4 Commits

Author SHA1 Message Date
Garry Tan
8ab733471b v0.41.17.0 feat: --workers N on every bulk command + facts dim doctor parity (#1519) 
* feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper

T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is
the canonical primitive every --workers N bulk command in this wave (and
future bulk commands) builds on. Atomic-claim invariant enforced by
scripts/check-worker-pool-atomicity.sh (wired into bun run verify).
BudgetExhausted bypass + AbortSignal composition baked into the helper so
budget caps are a structural ceiling under concurrency, not a per-caller
convention.

The new resolveWorkersWithClamp wrapper composes existing autoConcurrency
with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately
NOT a modification to shared autoConcurrency (silent today, used by sync
+ import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per
codex #13.

23 + 12 + 9 = 44 hermetic tests pin every contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: structural + dim-check regression suites for v0.41.16.0 wave

- test/embed-helper-migration.test.ts (T3): asserts embed.ts's two
  sliding-pool sites are migrated to runSlidingPool, pre-migration
  shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone,
  GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel
  threads page.slug. Per codex #16/#17 these are invariant assertions,
  not byte-equality on progress event ORDERING.
- test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim
  covers vector(N) + halfvec(N), halfvec-before-vector regex ordering
  pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER
  USING + CREATE INDEX (codex #18, not bare REINDEX),
  FactsEmbeddingDimMismatchError tagged class shape,
  assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent-
  column skip, doctor check + insert-cast wiring assertions.
- test/extract-conversation-facts-workers.test.ts (T5): helper
  exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES),
  structural wiring (runSlidingPool, resolveWorkersWithClamp,
  withRefreshingLock, LockUnavailableError, delete-orphans-first
  before segment loop, preflight before pool, exit 3 when lock_skipped
  > 0), Minion handler round-trip.
- test/extract-workers.test.ts (T7): --workers wiring on all 3 inner
  fs-walk loops (extractForSlugs, extractLinksFromDir,
  extractTimelineFromDir) + CLI parse + opts threading through
  runExtractCore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510)

PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0
on master in parallel. Advancing this wave to v0.41.17.0 so both can land
cleanly. Pure mechanical version bump:

- VERSION + package.json → 0.41.17.0
- CHANGELOG.md header + "To take advantage of v0.41.17.0" block
- TODOS.md section header + v0.41.18+ forward references
- CLAUDE.md inline version tags
- Regenerated llms-full.txt / llms.txt

No code changes. The actual workers cathedral feature set is unchanged
from the two prior commits in this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): search-image-column probes column dim at runtime

CI shard 5 failed on `searchVector column routing (v0.27.1)` with:
  error: expected 1280 dimensions, not 1536

The test had a hardcoded `fakeText1536` helper that seeded chunks at
1536-d vectors. Master's default embedding model switched from OpenAI
text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh
PGLite brain on CI now sizes content_chunks.embedding at 1280; the
test's 1536-d INSERT trips pgvector's CheckExpectedDim.

Fix: probe `content_chunks.embedding` width via
`readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in
`TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test
now passes regardless of which default ships (the model has flipped
twice and may flip again). Local dev (1536 from older config) and CI
fresh-install (1280 from new default) both pass.

Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the
column's fixed width on the image side).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): bump PGLite hook timeout for shard-4 deep-process files

facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI
shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0
master merge brought migration count to 99. When these files run deep in
a shard process that has already created ~20 PGLite engines, the WASM
cold-start + 95-migration replay legitimately exceeds bun's 5s default
hook timeout (observed 5.6s and 7.3s locally when reproducing).

Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts
but NOT hook timeouts; those default to 5s and must be set per-hook via
the optional 2nd arg to beforeAll/afterAll.

Reproduced locally by running the first 21 shard-4 files via
  head -21 /tmp/shard4-list.txt | xargs bun test
  → 179 pass, 2 fail (both with hook-timeout error)

After fix:
  → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover)

Full shard 4 with fix:  955 pass, 0 fail.
Full shard 5 with fix:  1261 pass, 0 fail.

Also added a defensive diagnostic to the two put_page tests: if
facts_backstop is missing in the response payload, throw with the full
payload + isError so future failures surface the actual handler error
instead of a bare "expected {...} got undefined" assertion. No-op when
the test passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 18:29:03 -07:00
Garry Tan
ca68633faa v0.41.2.0 feat: lens packs + epistemology unification — atoms + concepts as first-class units, calibration profile widening, gstack-learnings bridge (#1364) 
* feat(schema): migration v93 take_domain_assignments (v0.41 T1)

Adds the JOIN table backing per-pack calibration domain aggregation
in the v0.41 lens-packs wave. Replaces the originally-planned scalar
`takes.domain` column after codex outside-voice review caught that
one take can legitimately belong to multiple domains (a take about
"Sequoia's investment in Anthropic" lands in deal_success AND
market_call), and that scalar attribution bakes today's pack→domain
mapping into permanent fact.

Schema: composite PK (take_id, domain) for idempotent re-assignment,
FK CASCADE so deleting a take cascades assignments, confidence CHECK
in [0,1], idx_take_domain_assignments_domain for the aggregator JOIN
direction. RLS guard matches takes/synthesis_evidence pattern (enable
when running as BYPASSRLS role). PGLite parity via sqlFor.pglite.

Backward-compat: pre-existing takes carry no assignments; aggregator
LEFT JOIN skips them gracefully. No backfill required at migration
time — propose_takes (T10) populates new rows; greenfield assignment
of historical takes is a v0.42 follow-up.

R-MIG IRON-RULE regression at test/migrations-v93.test.ts pins 12
contracts: existence/name, LATEST_VERSION advance, table queryable
after initSchema, column shape, composite PK rejects duplicate
(take_id, domain), multi-domain assignment permitted, FK ON DELETE
CASCADE, CHECK rejects out-of-range confidence, index presence,
aggregator JOIN direction returns per-domain counts, sql/sqlFor.pglite
parity grep, backward-compat LEFT JOIN handles unassigned takes.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
First of 13 sequencing tasks in v0.41 lens packs + epistemology
unification wave (decisions D9-B → T1-B per codex challenge).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(contracts): IngestionSource.mode + pack manifest phases/calibration_domains (v0.41 T2+T3)

Two independent contract extensions, batched because both are pre-
requisites for T4 (pack YAML manifests) and T9 (cycle.ts orchestrator
gate). Neither is load-bearing alone; together they form the surface
the four lens-pack manifests will declare against.

T2 — IngestionSource.mode discriminator (codex outside-voice fix):
  src/core/ingestion/types.ts grows an optional `mode: 'trickle' |
  'migration'` field on IngestionSource. Defaults to 'trickle' when
  unset — v0.38 sources unchanged. New IngestionSourceMode export.
  src/core/ingestion/daemon.ts handleEmit() branches on the mode:
  trickle keeps the 24h DedupWindow.mark() path; migration bypasses
  dedup entirely (the source owns permanent slug-keyed idempotency
  via op_checkpoint or similar). Validation, rate limit, and dispatch
  apply uniformly to both modes.

  Why: the 24h content-hash dedup window is wrong for bulk historical
  migration. 24K wintermute pages over hours, retries days apart, and
  same-hash collisions across the window are expected. Trickle
  semantics (file-watcher, inbox-folder, webhook) want dedup to catch
  at-least-once replay; migration semantics want EVERY explicitly-
  emitted event to land because the source already gated it.

T3 — SchemaPackManifestSchema phases + calibration_domains:
  src/core/schema-pack/manifest-v1.ts grows two optional fields. New
  AGGREGATOR_KINDS closed enum (4 v1 algorithms: scalar_brier,
  weighted_brier, count_based, cluster_summary) backing
  AggregatorKind type. New CalibrationDomain {name, aggregator,
  page_types} schema with snake_case regex on name, .strict on extra
  fields, page_types.min(1).

  `phases: string[]` declares which cycle phases the active pack
  participates in (D4-B orchestrator gate; runCycle will consult this
  in T9). Validated as string here, against runtime CyclePhase union
  at the registry layer (avoids circular import). `borrow_from` does
  NOT borrow phases — each pack declares explicitly.

  `calibration_domains: CalibrationDomain[]` declares per-pack
  scorecard buckets. Closed registry of algorithm `aggregator` values
  keeps SQL injection surface closed; open `name` strings let third-
  party packs add domains without a gbrain release (T3 codex
  refinement of D6).

  Backward compat: both fields default to []. Existing v0.38 manifests
  parse unchanged (pinned by 2 regression cases).

Tests:
  test/ingestion/migration-mode.test.ts (8 cases): mode type accepts
  literals, defaults to trickle, daemon branches correctly across
  trickle/migration/default-undefined, validation still runs in
  migration mode, mixed dual-source independence.

  test/schema-pack-manifest-v041.test.ts (19 cases): aggregator enum
  shape, phases default + accept + reject (non-string, empty, non-
  array), calibration_domains default + accept (single + multi entry,
  multi page_types), reject (unknown aggregator, kebab/uppercase/
  digit-start names, empty page_types, unknown extra field), v0.38
  back-compat regressions.

  All 27 cases pass first-green after API surface alignment.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Tasks T2 + T3 of 13 in v0.41 lens packs + epistemology unification wave.
Unblocks: T4 (pack manifests reference both fields), T9 (cycle.ts gate
reads phases:), T10 (calibration widening reads calibration_domains).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(packs): 4 bundled lens pack manifests + registry wiring (v0.41 T4)

Authors gbrain-creator + gbrain-investor + gbrain-engineer +
gbrain-everything as bundled YAML manifests in
src/core/schema-pack/base/, registers them in the BUNDLED array in
load-active.ts, exports AGGREGATOR_KINDS + AggregatorKind +
CalibrationDomain types through the schema-pack barrel.

gbrain-creator: atom (NEW page type) + concept (reuse from base).
  phases: [extract_atoms, synthesize_concepts]. One calibration
  domain: concept_themes / cluster_summary / [concept]. Retires
  wintermute's atom-pipeline-coordinator cron (T12 follow-up).

gbrain-investor: thesis + bet_resolution_log (NEW). Borrows
  deal/person/company/yc from base. No new cycle phases (consumes
  existing extract_facts/propose_takes/grade_takes pipeline). Three
  calibration domains: deal_success/scalar_brier/[deal],
  founder_evaluation/scalar_brier/[person], market_call/weighted_brier
  /[thesis]. Filing rules mirror wintermute's existing investing/deals
  + investing/theses + investing/bets layout.

gbrain-engineer: bridge-only per D8-C. ONLY declares `learning`
  page type (primitive: annotation); borrows code+project from base.
  No new cycle phases (gstack-learnings IngestionSource is daemon-
  side per T8). Three calibration domains: architecture_calls/
  scalar_brier/[code, learning], effort_estimates/weighted_brier/
  [project], risk_assessment/scalar_brier/[project].

gbrain-everything: meta-pack extending gbrain-investor + borrowing
  atom (from creator) + learning (from engineer). Codex outside-voice
  T4 resolution to the multi-lens problem: composes via the v0.38-
  shipped extends + borrow_from chain instead of inventing an
  active-multi-pack architecture. Single-active-pack constraint
  preserved. Explicitly re-declares phases + calibration_domains
  (borrow_from borrows types/link_types only — phases must be
  declared per pack per D4-B).

Frontmatter validators (atom_type closed 11-value enum, virality_
score range, etc.) are NOT declared in these manifests — that
contract surface (per-page-type frontmatter_validators on
PageTypeSchema) is a v0.42 follow-up filed in plan TODOs. For
v0.41, extract_atoms hardcodes the enum with a TODO comment
pointing at the eventual manifest read path (D11).

YAML parser caveat: src/core/schema-pack/loader.ts uses a hand-
rolled parseYamlMini (per loader.ts:86 explicit non-support of `|`
block scalars). Initial descriptions used `|` blocks and broke
parsing silently (description was 'literal "|"', everything after
collapsed). Reauthored to single-line "..." strings. Pinned by
the manifest-load tests asserting page_types/phases/calibration_
domains all resolve.

Tests:
  test/lens-pack-manifests.test.ts (31 cases): one file covers all
  4 packs to avoid 4x boilerplate. Pins parse cleanly, registry
  inclusion, per-pack page_types/phases/calibration_domains/filing_
  rules shape, every aggregator value falls in AGGREGATOR_KINDS,
  meta-pack unions correctly (7 calibration domains across all
  three lens packs).

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T4 of 13. Unblocks T5/T6 (phases now declared; phases read
from active pack at runtime), T7 (importer writes atom-typed
pages against creator manifest), T8 (gstack-learnings emits
learning-typed pages against engineer manifest), T9 (orchestrator
gate reads phases: declaration), T10 (calibration_profile walks
calibration_domains).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): orchestrator-level pack gate for lens-pack phases (v0.41 T9)

Wires extract_atoms + synthesize_concepts into runCycle with the D4-B
orchestrator-level pack gate. Five surgical edits to src/core/cycle.ts:

  1. CyclePhase union grows by 2 names.
  2. ALL_PHASES inserts extract_atoms after extract_facts (Haiku 3-check
     has fresh fact context, BEFORE resolve_symbol_edges to avoid
     interrupting the symbol resolution sweep mid-flight) and
     synthesize_concepts after patterns (cluster pass sees fresh
     cross-session themes).
  3. PHASE_SCOPE entries: extract_atoms='source' (per-source transcript
     walk), synthesize_concepts='global' (concept clusters cross sources
     by nature).
  4. NEEDS_LOCK_PHASES adds both (put_page writes mutate DB).
  5. runCycle dispatch blocks for both phases consult packDeclaresPhase
     before invoking. When the active pack doesn't declare the phase,
     skipped with reason='not_in_active_pack' marker. When it does,
     lazy-imports extract-atoms.ts / synthesize-concepts.ts and runs.

The packDeclaresPhase helper is new at module-private scope. Loads the
active pack via loadActivePack({cfg, remote:false}); reads
resolved.manifest.phases (local only — D4-B). Fail-open: any registry
error (pack not found, malformed manifest) returns false. Skipping >
crashing for an orchestrator gate.

Local-only phase semantics (not extends-chain inherited) preserves user
sovereignty: a downstream pack extending gbrain-creator may NOT want
extract_atoms to run (e.g. derives atoms differently). Inheriting phases
would force them into a no-op-or-fork choice. The gbrain-everything
meta-pack therefore RE-DECLARES creator's phases verbatim in its own
manifest, asserted by the T4 test.

Stub phase modules ship in this commit:
  src/core/cycle/extract-atoms.ts → returns skipped with reason=
    'stub_pending_t5'
  src/core/cycle/synthesize-concepts.ts → returns skipped with reason=
    'stub_pending_t6'

T5/T6 replace the stub bodies with real LLM-driven phases. The
orchestrator dispatch is fully wired today and exercised by the test.

Manifest schema follow-on: phases + calibration_domains were originally
.default([]) but the type narrowing broke v0.38 fixture casts in
test/schema-pack-{lint-rules,registry,registry-reload}.test.ts.
Reverted to .optional(); consumers apply `?? []` at the read site.
Same pattern as IngestionSource.mode in T2. Updated T3 + T4 tests
to use `!` non-null assertion at sites that explicitly declared the
fields (typechecker can't narrow array literals through optional
boundaries).

Tests:
  test/cycle-pack-gating.test.ts (19 cases, R-GATE IRON RULE):
  ALL_PHASES + PHASE_SCOPE shape, ordering invariants (extract_atoms
  after extract_facts, synthesize_concepts after patterns), exhaustive
  PHASE_SCOPE map, NEEDS_LOCK_PHASES static-source assertion (both new
  phases included), dispatch consults packDeclaresPhase for BOTH new
  phases (and ONLY those two), packDeclaresPhase helper exists +
  reads manifest.phases (not merged chain) + fail-open returns false
  on catch, pre-existing 17 phases NEVER consult packDeclaresPhase
  (extract_facts + calibration_profile spot-checked), not_in_active_pack
  reason marker appears exactly 2x (semantic consistency across
  both gated phases).

  Adjacent test fixes: T3 + T4 tests updated for optional-field
  semantics. T2 dispatch type narrowed to DispatchOutcome shape from
  daemon.ts ({kind: 'queued'} for success path).

89/89 across T1+T2+T3+T4+T9 tests pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T9 of 13. Unblocks: T5 (extract-atoms.ts body replaces stub),
T6 (synthesize-concepts.ts body replaces stub).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(calibration): domain_scorecards widening + 4 aggregators (v0.41 T10)

Replaces the v0.36.1.0 placeholder `JSON.stringify({})` in
calibration-profile.ts:336 with a real aggregator pass over the active
pack's calibration_domains declarations. domain_scorecards JSONB now
populates per declared domain with {n, brier, accuracy, aggregator,
page_types, extras}.

New module: src/core/calibration/domain-aggregators.ts
  - aggregateDomainScorecards(engine, holder, domains, sourceId) → JSONB-shape
  - 4 aggregator implementations matching the AggregatorKind closed enum:
    - scalar_brier: AVG(POWER(weight - outcome::int, 2)). The default for
      most predictive domains. Filters by holder + page_types +
      resolved_outcome IS NOT NULL + active=TRUE + source_id.
    - weighted_brier: Brier weighted by ABS(weight - 0.5) * 2 (conviction
      proxy since takes table has no separate confidence column). A
      0.95-conviction miss weights 9x more than a 0.55-conviction one.
      Matches the investor pack's market_call semantics.
    - count_based: simple SUM(hit)/COUNT(*) accuracy without Brier.
      For domains where probability isn't natural.
    - cluster_summary: page count + tier histogram via
      frontmatter->>'tier' JSONB read. For concept_themes where there's
      no binary outcome to score. Returns {n, tier_counts: {T1, T2,
      T3, T4}}.

Wiring in src/core/cycle/calibration-profile.ts:
  Try/catch wraps the loadActivePack → aggregator chain. Empty {}
  scorecard on any pack-resolution error (R1 IRON RULE: byte-identical
  v0.36.1.0 baseline when no active pack declares domains). Warning
  appended to result.warnings so doctor surfaces silent failures
  instead of crashing the phase.

Per-domain fail-soft: aggregateOneDomain's try/catch returns
{n: 0, brier: null, accuracy: null, extras: {error}} for any single
malformed domain. The other domains still aggregate. Phase keeps
running.

Tests (test/domain-aggregators.test.ts, 13 cases):
  - R1 IRON RULE: empty domain list returns {} (byte-identical)
  - scalar_brier: empty no-takes returns n:0/null/null; 2-take
    Brier computed correctly (0.5 over (0, 1) sq_errs); accuracy
    matches weight>=0.5 hit/miss; filters by holder; filters by
    page_types; ignores unresolved takes
  - weighted_brier: high-conviction miss weighted 9x more; accuracy
    independent of conviction weighting
  - count_based: accuracy without Brier
  - cluster_summary: tier histogram from frontmatter; zero-concepts
    returns n:0 + all-zero tiers
  - Multi-domain: aggregates all declared in one call
  - Fail-soft per domain: nonexistent page_type produces n:0 without
    blocking other domains

89/89 across T1+T2+T3+T4+T9+T10 tests; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T10 of 13. The propose_takes-side wiring (populate
take_domain_assignments at write time from active pack's page_type→
domain mapping) is deferred to T5/T6 phase implementations, since
they are the natural producers of takes. Manual propose_takes via
fence write covers the operator path. v0.42+ adds a takes-fence
parser extension to read domain[] from fence rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): gstack-learnings bridge source (v0.41 T8)

Implements GstackLearningsSource — the daemon-side IngestionSource
that watches ~/.gstack/projects/{repo}/learnings.jsonl and emits
each new line as a `learning`-typed IngestionEvent.

Closes the v0.40-and-earlier gap where gstack's typed engineering
knowledge base (7 learning types: pattern, pitfall, preference,
architecture, tool, operational, investigation) lived in JSONL files
the brain never queried. After T8 + the engineer-pack manifest
activation, every gstack-logged learning surfaces as a first-class
gbrain page within seconds of being written.

Lifecycle:
  - constructor: discovers JSONL files via ~/.gstack/projects/*&#47;
    learnings.jsonl (cross-project mode, default) or just the current
    project (per-project mode). Test seam: _readFile/_existsSync/_skipWatch.
  - start(ctx): seeds seenLines with content_hashes of EVERY existing
    line so first-run-after-install does NOT replay thousands of
    historical lines as fresh emits. Then installs fs.watch handlers
    (one per discovered file) that fire rescanFile on 'change'.
  - rescanFile: O(N) per change event; re-reads the whole file,
    canonical-JSON content_hash on each line, emits any line not in
    seenLines. Malformed JSONL lines skip+warn.
  - stop(): closes all watchers; JSONL state preserved (gstack owns
    the files, gbrain only reads).
  - healthCheck(): reports warn when no files discovered (gstack not
    installed) OR when watched files have disappeared; ok otherwise
    with counter of lines seen.

mode: 'trickle' (the v0.41 T2 default). Line-level content_hash via
canonical-JSON serialization means whitespace reformatting doesn't
trigger re-emit. Re-emit of an identical line is a silent dedup hit
via the daemon's 24h DedupWindow (T2 trickle path).

Frontmatter rendered into the emitted markdown body preserves the
original JSONL fields verbatim: type=learning, learning_type
(one of the 7 types), confidence (1-10), source (one of: observed,
user-stated, inferred, cross-model), skill, key, optional files[]
+ branch + ts. Body is `# <key>\n\n<insight>` so search hits surface
the insight prose against semantic queries.

Pack activation: this source is intended to register with the daemon
when the active pack is gbrain-engineer or gbrain-everything (which
borrows learning from engineer). The daemon's startup probe layer
that consults active pack's page_types to decide which built-in
sources to construct lands in a follow-up wave; for now the source
is wired and tested but not auto-activated.

Tests (test/ingestion/gstack-learnings.test.ts, 14 cases):
  - Basic contract: mode='trickle', id includes pid, kind='gstack-learnings'
  - Start seeds seenLines (historical lines NOT replayed)
  - Malformed JSONL lines skip without crashing
  - Blank lines + trailing newlines OK
  - emitLine: new line emits, identical line is silent dedup hit
  - Emitted body carries proper frontmatter (type, learning_type,
    confidence, source, skill, key, files, branch, ts)
  - Canonical-JSON content_hash dedup (whitespace reformat = hit)
  - healthCheck warn/ok states
  - describePaths diagnostic per-file existence + size

All 14 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T8 of 13.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): wintermute-greenfield migration-mode importer (v0.41 T7)

Implements WintermuteGreenfieldSource — the one-shot bulk importer
for migrating the user's existing wintermute brain (13K atoms + 11K
concepts + ~30 ideas) into gbrain via the v0.41 lens packs.

mode: 'migration' (per T2 codex outside-voice challenge): bypasses
the 24h DedupWindow trickle dedup. Permanent slug-keyed idempotency
is owned by op_checkpoint (caller-wired via gbrain capture --source
wintermute-greenfield) + the imported_from frontmatter marker that
gates re-extraction by extract_atoms + synthesize_concepts (D7).

@one-shot doc comment per D10: this module stays in src/core/
ingestion/sources/ forever, not deleted post-migration. Future
similar migrations (other downstream agents, brain merges, schema-
pack upgrades) reuse the IngestionSource pattern shipped here.
Deleting the working example is short-sighted.

Walk:
  - ~/git/brain/atoms/{YYYY-MM-DD}/*.md (atoms, date-bucketed)
  - ~/git/brain/concepts/*.md (concepts, flat)
  - ~/git/brain/ideas/*.md (ideas, flat)
  Recursive directory walk via injected _readdirSync + _statSync
  (test seam). Alphabetical sort by relative path so --limit
  produces deterministic slices.

Per file:
  1. Read content; gray-matter parses frontmatter + body
  2. Skip when no `type:` frontmatter (skipped_no_type — not invalid,
     just not a gbrain page)
  3. Stamp imported_from='wintermute-greenfield' + imported_at ISO
     timestamp; preserve ALL other frontmatter fields verbatim
  4. Re-stringify via matter.stringify
  5. Emit IngestionEvent with content_type='text/markdown',
     untrusted_payload=false (local user-owned files), metadata
     carrying slug + page_type + original_path + original_frontmatter
     + importer + importer_version

Per-row validation failure → JSONL audit at
~/.gbrain/audit/wintermute-greenfield-failures-YYYY-Www.jsonl per
D12. Failed-file processing continues (don't fail-fast on one bad
row). Audit dir created lazily via mkdirSync recursive on first
write.

CLI flags supported via opts:
  --dry-run: walks + validates + stamps but doesn't emit
  --limit N: processes only the first N files (alphabetical)

The CLI surface lands via gbrain capture --source wintermute-greenfield
in a follow-up commit (capture.ts allow-list extension); for now the
source is instantiable + testable but not registered with the daemon.

Tests (test/ingestion/wintermute-greenfield.test.ts, 16 cases):
  - Basic contract: mode='migration', kind, start throws on missing
    repo
  - Walk: atoms+concepts+ideas, all 3 dirs visited
  - Frontmatter stamping: imported_from marker + imported_at present;
    original fields preserved (virality_score, source_slug, etc.)
  - Event shape: source_id/source_kind/source_uri/content_type/
    untrusted_payload all correct
  - Metadata: slug/page_type/original_path/original_frontmatter/
    importer/importer_version
  - Validation: no-type counts as skipped_no_type (not invalid);
    audit JSONL not appended for benign skips
  - Dry-run: counts tracked but no events emitted (3 stats but 0
    ctx.emitted)
  - --limit: only N files processed
  - Deterministic ordering: alphabetical relative-path sort means
    --limit 1 always picks the alphabetically-first file
  - healthCheck: ok after clean run; warn before start

All 16 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Task T7 of 13.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): extract_atoms + synthesize_concepts minimal-viable bodies (v0.41 T5+T6)

Replaces the T9-shipped stub modules with working LLM-driven phase
bodies. v0.41 ships the right SHAPE — Haiku per transcript producing
1-3 atoms, atoms grouped by concept frontmatter ref, tier assignment
by count, Sonnet narrative for T1/T2. The richer 3-check quality gate
(truism/punchline/entity multi-pass), embedding-similarity dedup, voice
gate integration, op_checkpoint resumability all land in v0.41.1+ —
filed as inline TODOs and plan follow-ups.

T5 extract_atoms (src/core/cycle/extract-atoms.ts):
  - Takes transcripts via _transcripts test seam OR discoverTranscripts
    production path (lazy-imports transcript-discovery.ts to avoid
    circular module loads through cycle.ts).
  - Per transcript: ONE Haiku call with the 11-value atom_type enum
    embedded in the prompt (matches gbrain-creator.yaml declaration;
    v0.42 reads from active pack manifest at runtime per D11).
  - parseAtomsResponse tolerates markdown fences + trailing prose;
    rejects invalid atom_type values; clamps virality_score to [0,100];
    rejects malformed entries silently (skip don't crash).
  - Per atom: putPage atom-typed page under atoms/{YYYY-MM-DD}/
    {slug-from-title}. Frontmatter preserves atom_type, source_quote,
    lesson, virality_score, emotional_register from the LLM output.
  - Budget cap $0.30/source/run (DEFAULT_BUDGET_USD); over-budget
    transcripts counted as budget-skipped, phase returns status='warn'
    if any failures occurred.
  - Source-scoped: opts.sourceId routes corpus dir + write target.
  - dry-run: counts but doesn't writePages.
  - Failures tracked per-transcript without halting the run.

T6 synthesize_concepts (src/core/cycle/synthesize-concepts.ts):
  - Takes atoms via _atoms test seam OR DB query for type='atom' pages
    excluding imported_from frontmatter marker (D7 skip).
  - Groups atoms by frontmatter `concepts:` array ref.
  - Tier by count: T1 >=10, T2 >=5, T3 >=2, T4 deferred (no <2 groups).
  - T1/T2 groups: Sonnet call with up to 10 sample titles + 5 sample
    bodies → 1-paragraph narrative. Budget cap $1.50/run; over-budget
    or LLM-failed groups fall back to deterministic narrative.
  - T3 groups: deterministic narrative (no LLM call).
  - Per group: putPage concept-typed page at concepts/{title-from-slug}
    with tier + mention_count + composite_score frontmatter.
  - dry-run + yieldDuringPhase honored.

Tests (test/cycle/extract-atoms-synthesize-concepts.test.ts, 19 cases):
  parseAtomsResponse: well-formed JSON, markdown fences stripped,
  trailing prose tolerated, invalid atom_type rejected, missing fields
  rejected, garbage returns [], all 11 atom_type values accepted,
  virality_score clamped to [0,100].

  runPhaseExtractAtoms: no-op without transcripts, extracts via stub
  chat + writes pages, dry-run counts without writing, failures
  tracked per-transcript without halting.

  runPhaseSynthesizeConcepts: no-op without atoms, groups by concept
  ref + tier assignment by count (T1=12 atoms, T2=6, T3=3), atoms
  without concept refs filtered out, <T3 threshold (1 atom) filtered,
  T3 uses deterministic (no LLM call), dry-run counts without writing,
  T1 narrative comes from LLM stub verbatim.

All 19 pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Tasks T5 + T6 of 13. v0.41.1 follow-ups inline:
  - extract_atoms: read atom_type enum from active pack at runtime (D11)
  - extract_atoms: 3-check quality gate as multi-pass refinement
  - synthesize_concepts: embedding-similarity dedup (currently exact-
    string concept ref match only)
  - synthesize_concepts: voice gate for T1 Canon narratives
  - Both: op_checkpoint resumability for cross-cycle continuation

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.41): CHANGELOG + lens-packs architecture + wintermute migration guide + eval scaffolds (T11+T12+T13)

Closes out the v0.41 lens packs + epistemology unification wave with
docs, eval command surfaces, and the version bump. Three tasks batched
because each is small standalone:

T11 — 3 eval command scaffolds:
  src/commands/eval-extract-atoms.ts
  src/commands/eval-synthesize-concepts.ts
  src/commands/eval-wintermute-greenfield.ts

  Each command surfaces the stable schema_version=1 envelope shape
  with status='not_yet_implemented' for v0.41. The real parity-baseline
  implementations (compare new phase output against wintermute's
  existing 13K atoms + 11K concepts on a 500-page sample subset; pass
  rate floor enforcement on greenfield import) land in v0.41.1. The
  scaffolds let users discover the commands AND give the v0.41.1 work
  a clear extension point. Pinned by 7 scaffold tests.

T12 — wintermute-side cleanup deferred to wintermute repo:
  The wintermute-side edits (shrink content-atom-extractor +
  concept-synthesis SKILL.md to thin wrappers; delete atom-backfill-
  coordinator; retire atom-pipeline-coordinator + atom-backfill-
  coordinator cron entries) live in ~/git/wintermute, not this repo.
  The migration guide (docs/migrations/v0.41-wintermute-greenfield.md
  below) documents the cleanup steps. Operator runs them after
  verifying the greenfield import.

T13 — Documentation:
  CHANGELOG.md: full v0.41.0.0 entry in the GStack/Garry voice with
  ELI10 lead, locked-decisions narrative explaining the 4 codex
  outside-voice tensions that reshaped the design, To-take-advantage-
  of-v0.41 paste-ready upgrade commands, itemized changes covering
  all 13 plan tasks, v0.41.1 follow-ups list.

  docs/architecture/lens-packs.md: four-pack diagram (creator/
  investor/engineer/everything via extends+borrow chain), per-pack
  shape (page types, phases, calibration domains), calibration
  profile widening + 4 aggregator algorithms (scalar_brier /
  weighted_brier / count_based / cluster_summary), take_domain_
  assignments table explanation, v0.41.1 follow-ups.

  docs/migrations/v0.41-wintermute-greenfield.md: operator guide
  for the bulk 24K-page migration. Dry-run flow, audit JSONL
  inspection, the actual import command, post-import verification,
  retiring wintermute's parallel atom-pipeline-coordinator + atom-
  backfill-coordinator crons, rollback procedure, re-running after
  partial failures.

Version bump: VERSION + package.json → 0.41.0.0.

All 158 tests across 10 v0.41 test files pass; typecheck clean.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Final tasks T11 + T12 + T13 of 13. Wave shipped end-to-end across
11 commits on this branch:
  9e17d007  T1: migration v93 take_domain_assignments
  f4b2648b  T2+T3: IngestionSource.mode + manifest schema extensions
  cefaad31  T4: 4 bundled lens pack manifests
  1850613e  T9: cycle.ts orchestrator-level pack gate
  c6f33491  T10: calibration_profile widening + 4 aggregators
  d1964ef2  T8: gstack-learnings bridge source
  adcaf4ac  T7: wintermute-greenfield migration-mode importer
  0318229f  T5+T6: extract_atoms + synthesize_concepts bodies
  (this)    T11+T12+T13: eval scaffolds + docs + version bump

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tests): bump phase-count assertions from 17→19 (v0.41 follow-on)

v0.41 added extract_atoms + synthesize_concepts to ALL_PHASES.
Three existing tests pinned the count at 17 via load-bearing
regression assertions:

  test/phase-scope-coverage.test.ts:48-49
    expect(ALL_PHASES.length).toBe(17)
    expect(Object.keys(PHASE_SCOPE).length).toBe(17)

  test/core/cycle.serial.test.ts:393
    expect(hookCalls).toBe(17)  // yieldBetweenPhases hook fires per phase

  test/core/cycle.serial.test.ts:406
    expect(report.phases.length).toBe(17)

  test/e2e/cycle.test.ts:110
    expect(report.phases.length).toBe(17)

These are the correct fix: the assertions exist precisely to catch
this case (a PR that adds a phase without updating downstream
consumers). The wave's v0.41 commit (T9) updated ALL_PHASES but
missed these three sites. Updating them to 19 with comment
breadcrumbs preserving the version history (v0.26.5 → 9,
v0.29 → 10, v0.31 → 11, v0.32.2 → 12, v0.33.3 → 13,
v0.36.1.0 → 16, v0.39.0.0 → 17, v0.41.0.0 → 19).

Without this fix: full unit test suite (`bun run test`) shows 3
failures from these assertions. Underlying v0.41 logic was already
green; this is pure pin-bumping.

After fix: 9059 unit tests pass. 0 actual test failures. (3 shard
wedges remain from unrelated long-running parallel-runner tests
that exceed the 600s per-shard cap — infra concern, not test
logic, pre-dates this wave.)

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Wave gate: all 13 plan tasks done; all v0.41 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): update EXPECTED_PHASES for v0.41 (extract_atoms + synthesize_concepts + schema-suggest)

E2E test/e2e/dream-cycle-phase-order-pglite.test.ts pinned the canonical
phase sequence at 16 entries. v0.41 added extract_atoms (after
extract_facts) and synthesize_concepts (after patterns); v0.39 had
already added schema-suggest between orphans and purge. EXPECTED_PHASES
was missing all three.

This is the correct fix — the test exists specifically to catch a PR
that adds a phase without updating consumers, and it fired exactly as
designed. Updating EXPECTED_PHASES to the v0.41 19-phase sequence with
comment breadcrumbs (v0.39.0.0 schema-suggest, v0.41.0.0 extract_atoms
+ synthesize_concepts).

Verification (run with --timeout 60000 per E2E convention):
  DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \
    bun test test/e2e/dream-cycle-phase-order-pglite.test.ts --timeout 60000
  → 5 pass, 0 fail

Other E2E failures observed in the full run are pre-existing /
environmental and not v0.41 regressions:
  - dream-synthesize-chunking: existing flake (synthesize details
    shape under withoutAnthropicKey)
  - fresh-install-pglite: env has multiple embedding providers
    configured; requires explicit --embedding-model disambiguation
  - http-transport: last_used_at debounce timing flake
  - ingestion-roundtrip: file-watcher trickle-mode timing flake
  - mechanical: gbrain doctor exits 1 because user's persistent
    ~/.gbrain has wedged migrations + reranker auth warnings
  - autopilot-fanout-postgres: pre-existing dispatch-selector
    timestamp semantics

None of those 6 are touched by the v0.41 wave. Filing them as
unrelated maintenance items.

Plan: ~/.claude/plans/system-instruction-you-are-working-toasty-milner.md
Wave gate: 13 plan tasks done; v0.41 unit tests green; v0.41 E2E
green; pre-existing E2E flakes unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(e2e): 4 root-cause fixes for pre-existing E2E flakes (master polish)

After merging origin/master (which landed v0.40.8.0's flake-fix wave),
re-ran the 6 E2E files previously called out as pre-existing failures.
v0.40.8.0 had already fixed 3; the remaining 3 had real root causes:

1. autopilot-fanout-postgres — hardcoded date 2026-05-22 was 30min ago
   when the test was written; today (2026-05-24) it's 2 days past the
   60-min freshness window. selectSourcesForDispatch correctly classifies
   the source as STALE (dispatch.length=1) instead of FRESH (length=0).
   Fix: replace literal date with Date.now() - 30 * 60 * 1000 so the
   timestamp stays relative-fresh forever.

2. ingestion-roundtrip — chokidar cross-test contamination on macOS
   FSEvents. Tests share OS-level fd resources across describe blocks;
   the first test's watcher hasn't fully released when the second
   test's watcher attaches, so the new watcher's events queue behind
   pending cleanup and the waitFor(15s) for the first file drop times
   out. Fixes:
     - Move fs.mkdirSync(inboxDir) BEFORE createInboxFolderSource +
       daemon.start to eliminate the chokidar attach race (chokidar
       can watch non-existent dirs but the timing is unreliable
       under test load).
     - Add 200ms grace period in beforeEach after resetPgliteState
       to let prior watchers fully release FSEvents handles.
     - mkdirSync both inboxA + inboxB BEFORE source registration in
       the multi-source test (same race shape).
     - Bump waitFor timeouts 6s → 15s for fs.watch flake tolerance.

3. fresh-install-pglite — dev machines with multi-provider env
   (OPENAI_API_KEY + VOYAGE_API_KEY + ZEROENTROPY_API_KEY set in zsh)
   fail init's disambiguation gate with "Multiple embedding providers
   env-ready". The test sets ZE_API_KEY but doesn't NEGATE the others.
   Fix: beforeEach saves + clears OPENAI_API_KEY + VOYAGE_API_KEY so
   init sees only ZE. afterEach restores. Hermetic per dev machine.

4. dream-synthesize-chunking — TIER_DEFAULTS + DEFAULT_ALIASES in
   src/core/model-config.ts had BARE Anthropic model ids (e.g.
   'claude-sonnet-4-6' instead of 'anthropic:claude-sonnet-4-6'). The
   v0.40.8+ subagent queue's classifyCapabilities() now validates that
   submitted models have a provider prefix via resolveRecipe(), which
   throws "unknown provider" on bare ids. The synthesize phase
   resolveModel → bare 'claude-sonnet-4-6' → submit_job → REJECT →
   phase 'fail' status with empty details (test expected children_submitted=1).
   Fix: prefix all 4 TIER_DEFAULTS + 5 DEFAULT_ALIASES with their
   provider (anthropic:claude-*, google:gemini-3-pro, openai:gpt-5).
   Production paths already worked because user pack manifests have
   explicit `models.tier.subagent = anthropic:...`; only the fallback
   path (used in tests with no API key + no model config) hit the
   bare-id format and broke.

Verification (all run against DATABASE_URL=...:5434/gbrain_test):
  test/e2e/autopilot-fanout-postgres.test.ts → 6/6 pass
  test/e2e/dream-cycle-phase-order-pglite.test.ts → 5/5 pass
  test/e2e/dream-synthesize-chunking.test.ts → 4/4 pass
  test/e2e/fresh-install-pglite.test.ts → 2/2 pass
  test/e2e/http-transport.test.ts → 8/8 pass
  test/e2e/ingestion-roundtrip.test.ts → 3/3 pass
  test/e2e/mechanical.test.ts → 78/78 pass
  Total: 106/106 pass, 0 fail.

Adjacent unit tests verified green:
  test/anthropic-model-ids.test.ts → 6/6 pass
  test/model-config.serial.test.ts → 19/19 pass

typecheck clean.

Plan: v0.41 wave (~/.claude/plans/system-instruction-you-are-working-toasty-milner.md).
Post-merge polish — every E2E failure surfaced in the v0.41 ship reports is now green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(v0.42.0.0): privacy sweep + queue rebump + 5 pre-existing test fixes

Privacy: rename `wintermute-greenfield` → `markdown-greenfield` identifier
across 13 files + 4 file renames per CLAUDE.md:550 (banned private-fork name
in public artifacts). Identifier shipped through the lens-pack wave as the
long-lived migration-mode source kind; sweep includes class names
(MarkdownGreenfieldSource), frontmatter marker, audit JSONL path, eval
command, and operator doc filename. Reframe contextual mentions per
OpenClaw substitution rule ("your OpenClaw"/"upstream OpenClaw").

Queue: rebump v0.41.0.0 → v0.42.0.0 (PR #1352 claims v0.41.0.0 in queue);
sweeps 38 v0.41 → v0.42 references across branch-introduced files; renames
docs/migrations/v0.41-markdown-greenfield.md → v0.42-markdown-greenfield.md,
test/schema-pack-manifest-v041.test.ts → -v042, test/eval-v041-scaffolds →
test/eval-v042-scaffolds. Pre-existing master files referencing v0.41 left
untouched (those describe master's own anticipated wave).

Test fixes (5 pre-existing failures + 1 shard wedge, all unrelated to lens
packs but caught by the post-merge run):
- src/core/anthropic-pricing.ts: estimateMaxCostUsd strips `anthropic:`
  provider prefix before ANTHROPIC_PRICING lookup. v0.31.12 introduced
  provider-prefixed model strings; the budget meter wasn't updated and
  fell through to BUDGET_METER_NO_PRICING (budget gate disabled), letting
  auto-think submissions complete when the test expected budget exhaustion
  to force partial/skipped.
- test/longmemeval-trajectory-routing.test.ts: perf-gate cap 10s → 30s.
  Test runs ~4s isolated; parallel-shard CPU contention pushes it to 16s.
  30s still catches genuine cold-path regressions.
- test/search/embedding-column.test.ts → .serial.test.ts: quarantine to
  serial pass (depends on gateway module-state set by bunfig.toml preload;
  other parallel tests' resetGateway() leaves stale state).
- scripts/run-unit-parallel.sh: SHARD_TIMEOUT 600s → 900s. Shard 8's
  migration test suite runs 1369 tests in 807s (all pass); 600s wrapper
  cap was killing healthy shards.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: update project documentation for v0.42.0.0

Sweep v0.41 → v0.42.0.0 drift across the wave's release-summary and the
two new doc files. The wave shipped under its planning-time name (v0.41);
the queue rebump to v0.42.0.0 left a handful of factual references
pointing at the wrong version.

- CHANGELOG.md v0.42.0.0 entry: doc-ref filename, follow-up version
  label, and 4 in-prose v0.41 cites corrected to v0.42.0.0 / v0.42.0.1.
- docs/architecture/lens-packs.md: title + body + follow-up section
  corrected to v0.42.0.0 / v0.42.0.1.
- docs/migrations/v0.42-markdown-greenfield.md: title + upgrade
  command text corrected to v0.42.0.0; fixed two prose typos
  ("your existing your OpenClaw" → "your existing OpenClaw";
   "The your OpenClaw skills" → "The OpenClaw skills").

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: rebump v0.42.0.0 → v0.41.2.0 (per user; patch slot on v0.41 line)

PRs #1352 and #1367 both claim v0.41.0.0 in queue (the .0 slot is contested);
v0.41.2.0 is unclaimed and represents this wave as a PATCH on the v0.41 line
rather than a separate minor wave.

Sweeps v0.42.0.0 → v0.41.2.0 across CHANGELOG + 2 docs + 4 yaml + 4 ts + 2
test files; renames docs/migrations/v0.42-markdown-greenfield.md →
v0.41.2-markdown-greenfield.md and 2 test files (-v042 → -v041_2).

Wave-identity tags ("v0.41 T4" etc) in test/code comments correctly
preserved — this IS a v0.41 wave patch, not a new wave. macOS sed `\b`
limitation means those tags were never converted in the first place;
verified intentional preservation.

Forward references to v0.42 in TODOS.md + CHANGELOG D3 section + future-
wave declarations in code comments are untouched (they describe the NEXT
minor wave, not this one).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(audit-writer): route log() to event-ts ISO-week file, not wall-clock now

CI shard 3 failed `createAuditWriter — readRecent() > returns events from
current week, filtered by ts cutoff` at audit-writer.test.ts:229 with
`Expected: 2, Received: 0`.

Root cause: `log()` computed the destination filename from `new Date()`
(wall-clock now) instead of the event's own `ts`. Back-dated events
(written with an explicit ts in the past) landed in the wrong ISO-week
file. `readRecent(days, now)` walks the current + previous week files
keyed on `now`, so events whose own ts pointed at a different week
became unreachable.

The test passes ts=2026-05-21/16/14 and now=2026-05-22 (week 21 + 20).
CI runs on wall-clock 2026-05-25 (week 22). The writer routed all 3
events to the week-22 file; readRecent walked weeks 21 + 20 and found
0 events. Locally on 2026-05-22 the bug was invisible because
wall-clock-now and event-ts fell in the same week.

Fix in src/core/audit/audit-writer.ts:log(): derive the destination
filename from `new Date(ts)` (the event's ts) so events always land in
their own ISO-week file. NaN-guard falls back to wall-clock-now on
unparseable ts.

Test update at test/audit/audit-writer.test.ts:132: the 'honors
caller-supplied ts override' case had encoded the bug as a contract
("writer.log writes to current-week file regardless of event ts").
Updated to compute the file path from the event's ts, matching the
corrected behavior.

All 22 audit-writer tests pass. All 103 audit-writer-consumer tests
(rerank, phantom, slug-fallback, shell, supervisor, content-sanity,
graph-signals-failures, bench-publish) pass — none of them assert on
the file path the writer chose; they all read via readRecent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 22:56:21 -07:00
Garry Tan
41ab138462 v0.40.8.0 test: e2e + unit gap coverage + master flake root-cause fixes (#1313) 
* fix(tests): root-cause two master test-infra flakes

gateway.test.ts: add afterAll(resetGateway) hook. The file's tests use
beforeEach(resetGateway) for per-test isolation, but the FINAL test was
leaving configureGateway({env: {OPENAI_API_KEY: 'openai-fake'}}) in the
gateway module state. Sibling files in the same bun shard (e.g.
test/ingestion/ingest-capture.test.ts) then triggered embed() against
the real OpenAI endpoint with the leaked fake key, wedging the shard
with 'Incorrect API key provided: openai-fake'.

header-transport.test.ts → .serial.test.ts: the file mutates RECIPES
(gateway's module-scoped recipe map) plus configureGateway, then asserts
fakeChatFetch was invoked. Under bun's intra-shard parallelism, sibling
files like test/ai/rerank.test.ts race the same state — chat would see
result.text === '[]' instead of 'ok' because another test called
resetGateway between this test's configureGateway and chat. Quarantining
as .serial.test.ts moves the file into the post-parallel serial pass
at --max-concurrency=1 per repo convention.

* refactor(doctor): extract buildChecks seam + behavioral coverage

src/commands/doctor.ts: extract buildChecks(engine, args, dbSource):
Promise<Check[]> from runDoctor. The check-building logic moves into
the new exported function; existing exported computeDoctorReport(checks)
at line 78 stays untouched. runDoctor becomes a thin wrapper:
buildChecks → computeDoctorReport → render + process.exit. All 10
process.exit sites stay in place. The two early-return paths drop
their inline outputResults+process.exit calls and return the partial
check list; the wrapper still produces identical observable output.

test/doctor-behavioral.test.ts (13 cases): pure pure-aggregation cases
pin computeDoctorReport math (3 fails → -60 points, score clamped at 0,
mixed outcome → unhealthy with fail dominating). Orchestrator cases
assert --fast flag honors the skip set, --json doesn't alter the list,
no-engine path returns partial without process.exit, and the snapshot
of load-bearing check names catches accidental drop-outs during
future refactors.

test/doctor-cli-smoke.serial.test.ts (1 case): subprocess smoke spawning
'bun run src/cli.ts doctor --json' against a fresh PGLite tempdir brain.
Catches render-path bugs that buildChecks-only tests miss — the class
the v0.38.2.0 partial-scan wave exposed. Quarantined as .serial because
PGLite write-locks don't play well with parallel runners; skippable via
GBRAIN_SKIP_SUBPROCESS_TESTS=1.

* feat(operations): trust-boundary contract test + filter-bypass shell guard

test/operations-trust-boundary.test.ts (14 cases): hybrid design per
plan D7. Pure assertions over all 74 ops cover the drift-detection
win (every op has a scope; every mutating op has a non-read scope;
hasScope(['read'], op.scope) correctly rejects 'admin' or 'write').
Plus the canonical filter contract: every localOnly: true op is
excluded from operations.filter(op => !op.localOnly). Plus targeted
handler-invocation cases for the two historically-broken HTTP-callable
classes: submit_job(name='shell', ctx.remote=true) MUST reject (F7b
HTTP MCP shell-job RCE class), and search_by_image(image_path,
ctx.remote=true) MUST reject (D18 P0 image-leak class). file_upload
and sync_brain are deliberately omitted from handler-invocation tests
because they're localOnly — calling their handlers directly tests an
impossible production path (codex CMT-3). All 7 localOnly ops are
snapshot-pinned by name to catch future flag-flips.

scripts/check-operations-filter-bypass.sh: greps src/ for any module
that imports the 'operations' value from core/operations.ts outside
the canonical filter site. Three import shapes detected: destructured,
aliased ('as ops'), namespace ('import * as'). Explicit allow-list of
10 known-safe importers with one-line rationale per entry. Plus a
filter-presence check on serve-http.ts that fails if the canonical
filter expression is refactored out. Codex /ship adversarial review
caught the original narrow regex missed aliased + namespace bypasses;
the expanded regex closes that class. Type-only imports of sibling
exports (sourceScopeOpts, OperationContext) are not flagged.

package.json: wires check:operations-filter-bypass into the verify
chain alongside check:jsonb and check:progress.

* refactor(cycle): export runPhaseLint+runPhaseBacklinks; add wrapper tests

src/core/cycle.ts: adds 'export' keyword to two existing phase
functions so behavioral tests can drive them without going through
runCycle's full setup cost. No body changes; no behavior change.
Documented as internal helpers exposed for test-only consumption —
downstream code should NOT take a dependency on them; existing
plan-eng-review D9 explicitly accepted the API-widening tax for
testability.

test/cycle-legacy-phases.test.ts (11 cases): combined file with two
describe blocks per plan D5 (DRY — shared setup, future phase
wrappers land as additional describes). Narrowed to result-mapping
+ error envelope per codex CMT-1 (legacy phases don't extend
BaseCyclePhase and don't take a progress reporter or AbortSignal
directly, so the contract surface is counter → status enum + try/catch
envelope). Cases: clean run → status='ok', partial fix → status='warn'
with dryRun in details, dry-run path doesn't write, throw-from-lib
→ status='fail' with envelope populated (no exception escape).
Verified runLintCore and runBacklinksCore both throw on missing dir,
so the throw-from-lib cases are deterministic.

* chore: bump version and changelog (v0.40.4.1)

E2E + unit test gap coverage wave. Closes 4 audit gaps with new
behavioral coverage (doctor orchestrator + subprocess smoke, operations
trust-boundary contract + filter-bypass guard, cycle phase wrapper
result-mapping). Verifies 3 audit gaps were already covered (ingestion
dedup/daemon/skillpack-load, phantom-redirect, ingestion test-harness).
Root-causes 2 pre-existing master flakes (gateway state leak, header-
transport cross-shard race). Files 5 follow-up TODOs from codex
adversarial-review findings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: note v0.40.4.1 doctor.buildChecks + cycle phase exports in CLAUDE.md

Adds three Key-files entries pinning the v0.40.4.1 test-wave additions:
- doctor.ts extension: buildChecks seam + behavioral tests (13+1 cases)
- cycle.ts extension: runPhaseLint + runPhaseBacklinks exports (11 cases)
- operations-trust-boundary contract + check-operations-filter-bypass.sh

Regenerates llms-full.txt to match (CLAUDE.md edits require build:llms per
project rule, otherwise test/build-llms.test.ts fails in CI shard 1).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(tests): reset gateway in put_page write-through tests to skip embed in CI

CI failure mode: 9 tests in test/ingestion/put-page-write-through.test.ts
failed with `AIConfigError: [embed(zeroentropyai:zembed-1)] Unauthorized`
because put_page's handler at src/core/operations.ts:622 computes
`noEmbed = !isAvailable('embedding')`. When the gateway state has been
configured by a sibling test (or by the cli.ts module-load path reading
.env.testing) with a fake/stale ZEROENTROPY_API_KEY, isAvailable returns
true → put_page tries to embed → the real ZeroEntropy API returns 401.

Local dev passes because real ZE keys are present; CI doesn't have them.

Fix: call resetGateway() in beforeEach so isAvailable('embedding')
returns false → put_page's noEmbed path activates → no network call.
Also reset in afterAll to avoid leaking the cleared state to sibling
files in the same bun shard (the v0.40.4.1 gateway state-leak class
that motivated the earlier gateway.test.ts fix).

The test exercises write-through behavior, not embedding. No need for
a real or fake embed transport — bypass entirely.

* fix(tests): widen brain-writer partial-scan deadlines to absorb CI timing variance

CI failure: scanBrainSources partial-scan state > "hanging COUNT does not
exceed deadline — Promise.race timeout fires" failed once on a GitHub
Actions runner. The test asserted a 100ms deadline budget with a 500ms
bound; observed test duration was 187ms on CI (passes locally 20/20
runs at the original budget).

Root cause: Node.js timer drift under shard parallelism. The deadline
check at src/core/brain-writer.ts:503 uses strict `Date.now() > deadline`,
so when the setTimeout in Promise.race fires exactly at the boundary
(e.g. start+100ms when deadline is start+100ms), the post-await check
sees equality and skips the markRemainingSkipped branch. The test
also asserts elapsed < 500ms; CI overhead can push elapsed past that
bound when setTimeout drifts.

Fix: widen the deadline budget on both deadline-race tests proportionally
(keeps the same 2x ratio that proves "query exceeds deadline"). No
src/ changes — this is purely a test robustness widening.

  - "hanging COUNT" test: 100ms → 500ms deadline, 500ms → 2500ms bound
  - "slow COUNT" test: 50ms → 250ms deadline, 100ms → 500ms query delay

Verified locally: 20/20 stress runs at the widened budgets, no fails.

* fix(brain-writer): deadline check is >= not > (closes CI flake at boundary)

CI failure recurred: same "hanging COUNT does not exceed deadline" test
failed again at 588ms (past my previous 500ms deadline + 2500ms bound
widening). The root cause isn't test timing — it's an off-by-one in
the source.

src/core/brain-writer.ts had two deadline checks using strict `>`:
- line 445 (between-source abort)
- line 503 (post-COUNT-await re-check)

The Promise.race setTimeout resolves null at exactly `remainingMs` from
now, so post-await Date.now() OFTEN equals the deadline within
integer-ms precision. With `>`, the check skipped → scanOneSource ran
on the source whose budget had just been eaten → that source got
status='scanned' instead of 'skipped'. The test's `expect(firstSource
.status).toBe('skipped')` failed.

Fix: both checks now use `>=`. When Date.now() equals deadline exactly,
the budget IS exhausted — proceeding would let the next source eat its
own budget on top of what's already spent. Matches the boundary the
Promise.race's remainingMs <= 0 immediate-null path uses (line 481).

This is the real fix for the v0.40.x CI flakes; my earlier test-budget
widening papered over the symptom without closing the boundary. Kept
the wider 500ms deadline for headroom but added a comment pointing at
the operator fix as the load-bearing change.

Verified: 20/20 stress runs green locally after the operator fix.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 18:43:10 -07:00
Garry Tan
26c54588fe v0.38.0.0 ingestion cathedral — gbrain capture + write-through + IngestionSource contract (#1275) 
* feat(ingestion): v0.38 substrate — daemon + IngestionSource contract + 2 sources

The foundation for the ingestion cathedral (CEO+DX+Eng plan-reviewed).
Plan: ~/.claude/plans/system-instruction-you-are-working-ethereal-riddle.md

WHAT YOU CAN NOW DO
The IngestionSource public contract is locked. Skillpack publishers can
build third-party ingestion sources (Granola, Linear, Mail, voice, OCR,
etc.) and ship them through the v0.37 skillpack registry. The locked
surface lives at the new package subpaths:

  import { IngestionSource, IngestionEvent } from 'gbrain/ingestion';
  import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness';

Both subpaths are pinned by test/public-exports.test.ts — breaking either
is a major-version change.

WHAT THIS COMMIT BUILDS
Foundation:
- src/core/ingestion/types.ts (IngestionSource, IngestionEvent,
  IngestionSourceContext, validateIngestionEvent, computeContentHash,
  INGESTION_SOURCE_API_VERSION, INGESTION_CONTENT_TYPES)
- src/core/ingestion/dedup.ts (24h content-hash LRU, 5000-entry cap)
- src/core/ingestion/skillpack-load.ts (gbrain.plugin.json discovery for
  third-party sources, api_version compat with paste-ready upgrade hints,
  in-process trust model for v1)
- src/core/ingestion/daemon.ts (IngestionDaemon: in-process source
  supervision sibling to v0.34.3.0 ChildWorkerSupervisor pattern, plus
  validate -> dedup -> rate-limit -> dispatch pipeline + health surface)
- src/core/ingestion/test-harness.ts (publisher-facing test utility with
  fake clock + in-memory event bus + expectEvent matchers + engine proxy
  that throws on access so publishers know what they're depending on)
- src/core/ingestion/index.ts (barrel for gbrain/ingestion subpath)

First two built-in sources prove the abstraction:
- file-watcher (chokidar over the brain repo; 1s debounce; honors
  pruneDir from src/core/sync.ts; symlinks rejected; Linux ENOSPC
  surfaces a paste-ready sysctl hint at runtime)
- inbox-folder (~/.gbrain/inbox/ target for iOS Shortcuts / AirDrop /
  Drafts; auto-archives processed files into .archived/YYYY-MM-DD/;
  symlink rejection; world-writable dir warning; routes content-type by
  extension)

Public exports surface (count 18 -> 20) pinned in:
- package.json exports map
- test/public-exports.test.ts EXPECTED_EXPORTS + count gate
- scripts/check-exports-count.sh baseline

ARCHITECTURE-LOCKED DECISIONS (from /plan-eng-review)
E1 webhook source process boundary: webhook source will live INSIDE
serve --http (NOT this daemon) when it lands in the next commit. Daemon
supervises only daemon-side sources.
E2 content-type processor execution: hybrid by size (inline <1MB,
Minion handlers >1MB). Processors land in a later commit.
E3 publisher TTHW: chokidar v4.0.3 across platforms; ephemeral PGLite
persistence and Linux inotify-limit doctor probe land in later commits.
E4 migration v80 (provenance columns) + forward-reference bootstrap:
lands with put_page write-through in a later commit.

DX-locked decisions (from /plan-devex-review):
- Source error semantics: throws bubble to daemon; supervisor backoff.
- IngestionTestHarness exported as gbrain/ingestion/test-harness.
- api_version field on gbrain.plugin.json with loud-fail on mismatch.

TESTS
192 cases across 8 test files, 0 failures:
- test/ingestion/types.test.ts (28 cases pinning the contract)
- test/ingestion/dedup.test.ts (15 cases for LRU + TTL + collision)
- test/ingestion/skillpack-load.test.ts (22 cases for manifest
  validation + api_version compat + collision policy + module load)
- test/ingestion/test-harness.test.ts (24 cases for harness lifecycle +
  clock + healthCheck + every expectEvent matcher)
- test/ingestion/daemon.test.ts (19 cases for supervision + dispatch
  pipeline + health surface + per-source config + logger wrapping)
- test/ingestion/sources/file-watcher.test.ts (10 cases including
  ENOSPC sysctl-hint surfacing)
- test/ingestion/sources/inbox-folder.test.ts (24 cases including
  symlink rejection + world-writable warning + archive-loop-prevention)
- test/public-exports.test.ts (2 new cases for the new subpaths)

typecheck clean. bun run verify gate passes.

NEXT IN WAVE
Subsequent commits in this PR ship webhook source (serve --http route),
cron-scheduler refactor + OpenClaw credential auto-migrate, content-type
processors (PDF + image OCR + audio transcribe + video keyframe), put_page
write-through with serializePageToMarkdown DRY extract, migration v80
+ bootstrap probes, gbrain capture verb, publisher DX cathedral (init
scaffold extension + gbrain ingest test [--watch] + tail + validate),
daemon rename autopilot -> ingest with forever-alias, doctor inotify
probe on Linux, skillpack contract docs + reference pack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): webhook source — POST /ingest + ingest_capture Minion handler

Lands the v0.38 ingestion cathedral's webhook source. Per the
/plan-eng-review E1 decision, the webhook source lives INSIDE
`serve --http` (NOT the ingestion daemon) so there is no new IPC: the
HTTP route submits Minion jobs directly into the existing queue, and
the daemon supervises only daemon-side sources.

WHAT YOU CAN NOW DO
With `gbrain serve --http` running and an OAuth client minted, any
HTTP caller (Zapier, IFTTT, n8n, Make, Apple Shortcuts) can POST a
captured thought into the brain:

  curl -X POST https://your-brain.example.com/ingest \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: text/markdown" \
    -d "# captured from my Shortcut"

The route auths via OAuth (write scope required), validates the
content-type, enforces a 1MB payload cap and per-IP rate limit
(100 events / 10s), submits an `ingest_capture` Minion job tagged
`untrusted_payload: true`, and returns 202 Accepted with the job id.
The job materializes the page under `inbox/YYYY-MM-DD-<hash6>` by
default (overridable via X-Gbrain-Slug header) so the user has a
predictable triage location.

WHAT THIS COMMIT BUILDS
- src/core/minions/handlers/ingest-capture.ts (new) — handler that
  takes an IngestionEvent payload, resolves a slug via fallback chain
  (job.data.slug -> event.metadata.slug -> inbox/<date>-<hash6>),
  validates the event at the handler boundary, REJECTS binary
  content_types with a paste-ready hint to install a processor
  skillpack, and routes through importFromContent. Defaults
  noEmbed: true (embed is a separate Minion job, matching the sync
  handler's pattern).
- src/commands/jobs.ts — registers `ingest_capture` in
  registerBuiltinHandlers alongside sync/embed/extract.
- src/commands/serve-http.ts — POST /ingest route with:
    - OAuth write-scope gate via requireBearerAuth({requiredScopes:['write']})
    - 100 events / 10s rate limiter (sibling to ccRateLimiter)
    - Content-type allowlist: text/markdown, text/plain, text/html,
      application/json; binary REJECTED with HTTP 415
    - 1 MB payload cap (configurable via GBRAIN_INGEST_MAX_BYTES)
    - Caller-overridable source identity via X-Gbrain-Source-Id /
      X-Gbrain-Source-Uri / X-Gbrain-Content-Type / X-Gbrain-Slug
      headers — useful for downstream tools that want clean provenance
    - untrusted_payload: true ALWAYS (network input)
    - Idempotency on (client_id, content_hash) so simultaneous retries
      collapse to one job
    - maxWaiting: 50 per client so a runaway integration can't
      monopolize the queue
    - Audit row in mcp_request_log + SSE broadcast for the admin feed

TESTS
test/ingestion/ingest-capture.test.ts (15 cases against PGLite):
- defaultSlugForEvent helper (3 cases pinning shape + UTC + determinism)
- slug resolution fallback chain (3 cases)
- validation + content-type routing (5 cases including binary rejection
  + untrusted_payload round-trip)
- importFromContent integration (3 cases including content_hash dedup
  via status='skipped' on repeat)

207 total ingestion tests passing. typecheck clean.

NEXT IN WAVE
cron-scheduler refactor + OpenClaw credential auto-migrate; content-type
processors (PDF + image OCR + audio transcribe + video keyframe);
put_page write-through + serializePageToMarkdown DRY extract +
migration v80 + bootstrap probes; gbrain capture verb; publisher DX
cathedral (init scaffold + gbrain ingest test --watch + tail + validate);
daemon rename autopilot -> ingest with forever-alias; doctor inotify
probe; skillpack contract docs + reference pack + VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): put_page write-through + migration v80 + DRY extract

WHAT YOU CAN NOW DO
The drift class is dead. Every `gbrain put_page` (CLI or MCP, local
or remote) now lands its markdown file on disk alongside the DB row
whenever `sync.repo_path` is configured. The page is queryable
immediately AND visible to git, your editor, and downstream tools.

Pre-v0.38, put_page wrote ONLY to the DB and synthesize/extract paths
had to reverse-render later. The v0.35.6.0 phantom-redirect pass was
the cleanup for what THIS commit prevents in the first place.

  # local CLI
  gbrain put inbox/test < my-thought.md
  # file lands at ${sync.repo_path}/inbox/test.md AND in the DB

  # MCP remote (Zapier / Cursor / Claude Desktop)
  curl -X POST /mcp ... '{"method":"tools/call","params":{"name":"put_page",...}}'
  # server-side write-through fires, agent gets a normal success response
  # untrusted_payload tagging applied (no auto-link, slug-allowlist gate)

Provenance frontmatter stamped on every write so future sync round-trips
know where the page came from:
  ingested_via: put_page         # local CLI
  ingested_via: 'mcp:put_page'   # MCP remote
  ingested_at: 2026-05-21T04:...

WHAT THIS COMMIT BUILDS
1. Migration v80 — `pages_provenance_columns` adds four nullable
   columns to `pages`: `ingested_via`, `ingested_at`, `source_uri`,
   `source_kind`. ADD COLUMN with no DEFAULT is metadata-only on
   Postgres 11+ and PGLite 17.5; instant on tables of any size. The
   four columns get NULL on every historical page (pre-v0.38 pages
   never had provenance).

2. DRY extract — `serializePageToMarkdown(page, tags, opts)` and
   `resolvePageFilePath(brainDir, slug, sourceId)` in `src/core/markdown.ts`.
   The dream-cycle's `renderPageToMarkdown` (synthesize.ts) and the new
   put_page write-through path were going to have 90% duplicate bodies.
   They now share one foundation; the dream version is a 4-line wrapper
   that passes `frontmatterOverrides: {dream_generated: true, ...}`.
   Future markdown-shape changes happen in one place.

3. put_page write-through (`src/core/operations.ts`) — after
   importFromContent succeeds, resolves sync.repo_path, computes the
   v0.32.8 source-aware path layout (default: brainDir/<slug>.md;
   non-default: brainDir/.sources/<id>/<slug>.md), serializes the
   freshly-written Page via `serializePageToMarkdown`, writes the file.
   Returns a `write_through: {written, path}` field in the put_page
   response so callers can see what happened.

   Trust gating:
   - subagent sandbox (viaSubagent without allowedSlugPrefixes) → DB-only
   - dry-run → DB-only (handler's early-return short-circuits before
     write-through; documented via the dry_run response field)
   - no sync.repo_path configured → DB-only, skipped reason returned
   - sync.repo_path points at a non-existent dir → DB-only, skipped
   - all other writes → write-through

   Failure isolation: disk-write failures are LOGGED loud but do NOT
   roll back the DB write. DB is the durable record; the
   phantom-redirect pass exists for drift cleanup if it ever shows up.

TESTS
- test/ingestion/put-page-write-through.test.ts (10 cases against PGLite):
  happy path (file land, provenance stamp local + remote), trust gating
  (subagent sandbox, dry-run, trusted-workspace), config edges (no
  repo_path, missing dir), multi-source filing (.sources/<id>/),
  failure isolation (DB write survives a disk failure).
- Migration v80 verified across both engines via the existing
  test/migrate.test.ts + test/bootstrap.test.ts coverage (~125 cases).

369 total tests passing in the ingestion + markdown + migrate bundle.
typecheck clean.

NOTES
- Bootstrap probes for the v80 provenance columns are NOT yet added
  to applyForwardReferenceBootstrap on either engine. This is safe
  for v0.38 because no SCHEMA_SQL CREATE INDEX or FK references the
  new columns — migration v80 is the only consumer, and it runs
  AFTER SCHEMA_SQL replay. A future commit may add bootstrap probes
  + REQUIRED_BOOTSTRAP_COVERAGE entries as defense-in-depth (eng
  review E4).

- The trusted-workspace path (dream cycle's reverseWriteRefs in
  synthesize.ts) still runs its own write at synthesize phase time.
  Both paths writing the same file is idempotent (byte-identical
  serialization), but a future commit may simplify reverseWriteRefs
  to skip pages whose file already matches.

NEXT IN WAVE
gbrain capture verb (the single human-facing entrypoint); daemon
rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): gbrain capture — the single human-facing entrypoint

WHAT YOU CAN NOW DO
One command, local or thin-client, synchronous receipt with the resulting
page slug. The answer to "what is the best way to get data into the brain?"
is now: just type `gbrain capture` and the right thing happens.

  # the basic case
  gbrain capture "remember to follow up on the X deal"

  # from a file
  gbrain capture --file ./notes/today.md --slug daily/2026-05-21

  # from a pipe (shell pipelines)
  echo "from stdin" | gbrain capture --stdin

  # script-friendly: print just the slug
  SLUG=$(gbrain capture "a thought" --quiet)

  # JSON for agents
  gbrain capture "..." --json

Default slug is `inbox/YYYY-MM-DD-<hash8>` — deterministic for the same
content so re-running idempotently lands the same page. Receipt block
on stdout shows slug + status + content_hash + on-disk path so you
can confirm where the page went without rerunning `gbrain query`.

The local-install path routes through the put_page operation with the
v0.38 write-through plumbing landed in the prior commit, so the page
hits both the DB AND the file tree in one move. Thin-client installs
route through `callRemoteTool('put_page', ...)` so the server's
write-through handles disk persistence the same way.

WHAT THIS COMMIT BUILDS
- src/commands/capture.ts (new ~290 LOC):
  - `defaultSlug(content)` — UTC-stable `inbox/YYYY-MM-DD-<hash8>`
  - `parseArgs(args)` — positional + flag parsing with --file / --stdin
    / --slug / --type / --source / --quiet / --json / --help
  - `buildContent(rawBody, opts)` — wraps unstructured prose in
    frontmatter (type + title + captured_via + captured_at) and a
    leading `# Title` heading; passes through if the body already
    looks like markdown
  - `runCapture(engine, args)` — local install routes through the
    in-process put_page operation; thin-client routes through MCP.
    `--quiet` prints just the slug; `--json` prints structured output;
    default prints a 5-line receipt block.

- src/cli.ts:
  - Adds `case 'capture'` dispatch
  - Adds `'capture'` to the CLI_ONLY set so cli.ts wires it correctly

TESTS
test/commands/capture.test.ts (21 cases against PGLite):
- defaultSlug helper: shape + determinism + UTC math
- parseArgs: positional + multi-token join + every flag
- buildContent: prose wrapping, --type override, no double-wrap
  for pre-frontmattered content, title cap at 80 chars,
  --source provenance stamp
- Integration: inline content lands in DB + on disk, default slug
  shape, --file reads from disk, --json structured output,
  --help returns without engine roundtrip

271 total tests passing in the bundle. typecheck clean.

NOTES
- Thin-client routing relies on `callRemoteTool('put_page', ...)` from
  src/core/mcp-client.ts. Identical UX to the local path because the
  server's put_page handler runs the same write-through plumbing.

- buildContent's "looks like markdown" heuristic is intentionally
  simple — first-line heading or frontmatter delimiter is the trigger.
  Users who care about exact formatting pass a pre-formatted --file.

NEXT IN WAVE
Daemon rename autopilot -> ingest with forever-alias + plist migration;
doctor inotify probe (Linux); content-type processor router
(PDF + image OCR + audio transcribe stubs); cron-scheduler refactor
+ OpenClaw credential auto-migrate; skillpack contract docs +
reference pack; VERSION bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ingestion): v0.38.0.0 release hygiene + e2e roundtrip + capture skill

VERSION 0.37.1.0 → 0.38.0.0 (trio: VERSION, package.json, CHANGELOG header).
CHANGELOG entry written in user-facing ELI10-lead voice per CLAUDE.md
release-summary rules. README's pre-loop section gains a new "How to get
data in (v0.38+)" block leading with `gbrain capture`.

skills/capture/SKILL.md (NEW) so agents route "capture this" / "save this
thought" / "remember this" / "drop this in the inbox" / "save to brain" to
the capture verb. RESOLVER.md updated with the new triggers (sits above
idea-ingest/media-ingest/meeting-ingestion in the content-ingestion
section as the "simple thought" path).

E2E roundtrip test (test/e2e/ingestion-roundtrip.test.ts) covers the gap:
inbox-folder source -> daemon -> ingest_capture handler -> DB page,
including:
- Full pipeline: file drop appears as page in DB + file moves to .archived/
- Dedup catches byte-identical content from a different filename
- Multi-source coordination: two distinct inbox dirs, two sources, daemon
  ingests both events independently

The test runs against an in-memory PGLite (no DATABASE_URL needed) so it
exercises the substrate-level wiring in the standard test suite. A
follow-up commit can add a full-process e2e (gbrain serve --http + real
OAuth client + POST /ingest) that requires DATABASE_URL.

399/399 v0.38 wave tests passing (910 assertions). typecheck clean.
bun run verify gate green across all 14 shell checks.

DEFERRED TO FOLLOW-UP RELEASES (called out in CHANGELOG)
- Daemon rename autopilot -> ingest + forever-alias + plist migration
- cron-scheduler skill refactor + OpenClaw credential auto-migrate
- Content-type processors (PDF / OCR / audio / video)
- gbrain doctor inotify probe (Linux)
- Publisher DX cathedral: gbrain skillpack init --kind=ingestion-source,
  gbrain ingest test --watch, ingest tail, ingest validate
- Reference pack at examples/skillpack-ingestion-reference/ + 3-stage
  tutorial in docs/ingestion-source-skillpack.md

These are polish items; the substrate is shipped and queryable, and
skillpack publishers can build sources against the IngestionTestHarness
public export today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ingestion): test-gap fills — bootstrap probes + manifest entry + conformance

The v0.38.0.0 release-hygiene commit landed cleanly against the v0.38 wave
suite but tripped 3 categories of full-suite tests. This commit fixes
each. The remaining failure (doctorReportRemote > "healthy" status) was
verified pre-existing via `git stash + bun test` and is not caused by
v0.38; left alone.

Fix 1 — `schema-bootstrap-coverage.test.ts` (s1)
The test parses MIGRATIONS for ALTER TABLE ADD COLUMN statements and
fails if any column is not covered by `applyForwardReferenceBootstrap`
on both engines. Migration v80's four provenance columns triggered
the failure. Bootstrap probes added to both engines + 4 entries
appended to REQUIRED_BOOTSTRAP_COVERAGE:

- src/core/pglite-engine.ts — 4 EXISTS probes + state field + needs
  flag + ALTER TABLE block when bootstrap fires
- src/core/postgres-engine.ts — same pattern
- test/schema-bootstrap-coverage.test.ts — 4 coverage entries

Fix 2 — `check-resolvable.test.ts` (s3 — orphan_trigger)
RESOLVER.md references skills via name; check-resolvable cross-checks
against skills/manifest.json. The new `capture` skill was missing the
manifest entry; added between `brain-ops` and `idea-ingest` so the
manifest order mirrors the resolver order.

Fix 3 — `skills-conformance.test.ts` (s8)
Every SKILL.md must have `## Contract`, `## Output Format`, and
`## Anti-Patterns` sections. skills/capture/SKILL.md was missing all
three (initial draft skipped them); now compliant with concrete
content per the v0.38 contract.

Fix 4 — `build-llms.test.ts` (s6)
README + CHANGELOG edits in the release-hygiene commit caused
llms-full.txt to drift behind. Regenerated via `bun run build:llms`.
Per CLAUDE.md: any user-facing docs edit MUST run build:llms before
push.

The full bun-test parallel runner now passes everywhere except the
pre-existing `doctorReportRemote > healthy status` failure (50/100
score on an empty fresh brain — this is a pre-v0.38 health-score
tuning issue and orthogonal to ingestion work).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(version): bump 0.38.0.0 → 0.38.1.0

Renumbers the in-flight ingestion-cathedral release to v0.38.1.0.
Trio (VERSION, package.json, CHANGELOG.md) bumped together.

bun run typecheck → clean.

* chore(version): bump 0.38.1.0 → 0.38.0.0

Master sits at 0.37.11.0; 0.38.0.0 is the natural next slot rather than
skipping a release. Trio (VERSION, package.json, CHANGELOG.md) bumped
together. Migration v81 + ingestion substrate stay identical — this is a
header-only renumber.

bun run typecheck → clean.

* test(ingestion): fill v0.38 test gaps — markdown helpers + migration v81 + webhook E2E

Three gaps surfaced from a v0.38 audit against what shipped vs what was
covered. All three filled:

1. **test/markdown-serializer.test.ts** (NEW, 19 cases) — pure-function
   coverage of `serializePageToMarkdown` + `resolvePageFilePath`, the
   DRY extract that the dream-cycle reverse-render and put_page
   write-through both consume. Pre-fix nothing pinned the
   frontmatter-override merge precedence, the type/title defaults, or
   the source-aware filing layout (default → `<brainDir>/<slug>.md`,
   non-default → `<brainDir>/.sources/<source_id>/<slug>.md`). Future
   schema-shape changes to either helper now surface immediately.

2. **test/migrate.test.ts — v81 cases** (10 new cases, two describe
   blocks) — structural assertions on `pages_provenance_columns`
   (four nullable columns, no NOT NULL, no DEFAULT, no index — the
   ADD COLUMN stays metadata-only) plus a PGLite round-trip that
   asserts the columns appear post-`initSchema`, accept direct UPDATEs,
   and survive the historical-page NULL scenario. The
   schema-bootstrap-coverage test already pinned the forward-reference
   probe contract; this fills the migrate.test.ts contract gap.

3. **test/e2e/serve-http-ingest-webhook.test.ts** (NEW, 16 cases) — HTTP
   contract coverage for POST /ingest. The pre-existing
   ingestion-roundtrip E2E explicitly notes "e2e (gbrain serve --http +
   POST /ingest + real OAuth) is a separate" thing — it covers the
   in-process daemon → handler → DB pipeline, NOT the real HTTP route.
   This file fills that gap. Spawns real gbrain serve --http against
   real Postgres, mints OAuth tokens with various scopes, exercises:
     - Auth gate (missing → 401; read-only → 403)
     - Body validation (empty → 400 with error: empty_body)
     - Content-type allowlist (image/png → 415 with skillpack hint;
       application/pdf → 415; text/plain + application/json + text/html
       all accepted; unknown text/* falls through to text/plain)
     - X-Gbrain-Content-Type / Source-Id / Source-Uri / Slug header
       overrides
     - Idempotency (same content + same client = identical job_id via
       queue dedup on content_hash)

Also wires three new entries into `scripts/e2e-test-map.ts` so changes
to `src/commands/serve-http.ts`, `src/core/ingestion/**`, or the
`ingest-capture` Minion handler auto-trigger the relevant E2Es under
`bun run ci:local:diff`.

Verified locally:
- bun test test/markdown-serializer.test.ts → 19/19 green
- bun test test/migrate.test.ts -t "v81" → 10/10 green
- bun test test/e2e/serve-http-ingest-webhook.test.ts (real Postgres on
  ephemeral 5435) → 16/16 green
- bun test test/select-e2e.test.ts → 24/24 green (selector test still
  honors the v0.38 entries)
- bun run typecheck → clean

E2E DB lifecycle handled per CLAUDE.md (spin up pgvector:pg16 on a free
port, bootstrap via `gbrain doctor --json`, run, tear down).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 08:00:42 -07:00