v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health (#1322)
* wip: federated sync v2 pre-merge snapshot
* v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health
Bump VERSION + package.json + CHANGELOG header + migration walkthrough filename
to v0.40.5.0 (claiming the next free slot in the v0.40.x patch series after
master's v0.40.1.0).
What ships (6 components, all behind sync.federated_v2 feature flag default-on):
1. Per-source sync lock — syncLockId(sourceId), phantom-redirect parity
2. Parallel sync --all — pMapAllSettled fan-out, --max-sources N cap
3. embed-backfill minion handler — D2 per-source lock + D6 $10/job budget + D15.1
fire-and-forget submission + D19 source-level cooldown + 24h $25 rolling cap
4. sync trigger CLI + POST /webhooks/github — HMAC-verified (60 req/min/IP),
X-GitHub-Event=push + ref filter against tracked_branch
5. sources status + federation_health doctor — batched GROUP BY pipeline
(4 queries instead of 6×N per-source roundtrips)
6. sources federate/unfederate hook — auto-submit embed-backfill on flip
Correctness fixes (unconditional):
- D21: sync.ts:959 facts backstop now passes sourceId to engine.getPage
- D15.4: redactSourceConfig + CI guard prevent webhook_secret leak
- D15.5: safeHexEqual extracted to src/core/timing-safe.ts
Schema:
- Migration v89 (sources_github_repo_index): partial expression index on
config->>'github_repo' for fast webhook source-lookup
Tests:
- 14 new test files, 112 cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID
back-compat, phantom per-source lock, embed-backfill kill+resume,
webhook HMAC prefix-strip). All 9449 unit tests pass.
Caught at test-write time: the webhook handler had a Buffer.from('sha256=...',
'hex') truncation bug — without the prefix-strip, every signature would have
"matched" empty buffers. Pinned by a test/sources-webhook.test.ts IRON-RULE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(check-source-config-leak): tighten regex to source-row patterns only
The v0.40.5.0 wave added scripts/check-source-config-leak.sh with a
too-broad pattern (JSON\.stringify\(.*config) that flagged any variable
named 'config' — catching the GLOBAL gbrain config.json serializers in
src/commands/init.ts (status envelopes) and src/core/config.ts (the
config-file write site). On the CI runner without rg installed, the
grep -rE fallback fired correctly and produced 4 false positives that
broke the `verify` script.
Tightened the patterns to specifically match `(source|src|row|s).config`
property access — the actual risk shape (a sources-table row being
serialized whole). The global gbrain config has a different shape and
threat model (file-mode 0o600 at the write site), so it's safe to
exempt at the regex level rather than per-file whitelist.
Also fixed a latent bug: the rg branch used `--include='*.ts'` (grep's
flag, not rg's). rg silently rejected it and CANDIDATES came back empty,
so the local-dev runs (which have rg) would never have caught a real
leak. Now branches on tool availability: `-g '*.ts'` for rg, `--include`
for grep -rE. Both branches verified against a synthetic leak fixture.
Also added init.ts + config.ts to the whitelist as a belt-and-suspenders
since they handle gbrain-global config (not source rows) and could
otherwise reflect-back via regex iteration.
CI: `bun run verify` exit 0 locally with both the original false-positive
fixture (clean repo) and a synthetic leak fixture (correctly caught,
exit 1).
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
113
CHANGELOG.md
113
CHANGELOG.md
@@ -2,6 +2,118 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.40.5.0] - 2026-05-23
|
||||
|
||||
**Your federated brain syncs every source at once instead of one-by-one, reacts to GitHub pushes within seconds instead of minutes, and stops blocking on a slow source when you onboard a new one.**
|
||||
|
||||
If you've ever watched `gbrain sync --all` chew through a 200K-page default brain before it even starts looking at your 13K-page zion-brain, this release is for you. Four federated sources used to sync end-to-end one after the other. Now they run in parallel, each in its own write window, and embedding catches up async via a background job instead of holding the whole pipeline. A 200K + 13K + 5K + 88K brain that took 12 minutes serial now finishes in ~3.
|
||||
|
||||
The same change unlocks push-triggered sync. Wire a GitHub webhook at `POST /webhooks/github`, run `gbrain sources webhook set <source-id> --github-repo owner/name` to register the secret, and a `git push` lands in your brain within ~5 seconds instead of waiting for the next autopilot tick. Federation flip (`gbrain sources federate zion-brain`) now auto-submits an embed-backfill job for the missing chunks so search quality catches up without you having to remember `gbrain embed --stale`.
|
||||
|
||||
For the autopilot, the "brain looks healthy → sleep" shortcut no longer prevents source sync. A healthy brain score says nothing about whether GitHub has new commits; v0.40 decouples the two so per-source freshness checks fire independently of the score gate. Polling-only deployments (no webhook configured) still get freshness within the autopilot interval; webhook-driven deployments get sub-second.
|
||||
|
||||
### How to turn it on
|
||||
|
||||
`gbrain upgrade` handles the schema migration and the new file. Then:
|
||||
|
||||
```bash
|
||||
# See per-source health (lag, embed coverage, queue depth, failures)
|
||||
gbrain sources status
|
||||
|
||||
# Run doctor to surface federation health (lag/embed warnings + paste-ready fixes)
|
||||
gbrain doctor
|
||||
|
||||
# Onboard a new source — embed-backfill runs async
|
||||
gbrain sources add zion-brain --path ~/git/zion-brain --federated
|
||||
|
||||
# Set up push-trigger for a source
|
||||
gbrain sources webhook set zion-brain --github-repo Garry-s-List/zion-brain
|
||||
# (paste the displayed secret + URL into the GitHub webhook UI)
|
||||
|
||||
# Manually trigger a sync from a script
|
||||
gbrain sync trigger --source zion-brain
|
||||
```
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on a 4-source brain (default 197K pages, zion-brain 13K, media-corpus 5K, straylight 88K):
|
||||
|
||||
| Operation | v0.39 | v0.40 | Speedup |
|
||||
|---|---|---|---|
|
||||
| `gbrain sync --all` (no embed) | 12m sequential | 3m parallel (max-sources=4) | ~4x |
|
||||
| Push → sync visible | ≤5min (autopilot interval) | ~5s (webhook) | ~60x |
|
||||
| New 50K-page source onboarding (block on embed) | inline ~$5 + ~8min wait | async background | non-blocking |
|
||||
| `gbrain sources status` on 4 sources, 300K chunks | N/A (didn't exist) | <2s (batched GROUP BY) | new |
|
||||
|
||||
Per-source isolation in numbers: two `gbrain sync` calls against different sources used to serialize on a global `gbrain-sync` lock. Now they take `gbrain-sync:<source>` so cross-source runs go full-parallel; only same-source contention still serializes.
|
||||
|
||||
### What's safe to know about
|
||||
|
||||
- **Feature flag for clean rollback.** `gbrain config set sync.federated_v2 false` + restart autopilot reverts to v0.39 sequential behavior without uninstalling. Per-source lock and migration v92 stay on regardless (correctness fixes, not features).
|
||||
- **Webhook secret storage.** Per-source plaintext in `sources.config.webhook_secret` (trust posture: the DB IS the boundary; same posture as `config.access_policy`). The new `redactSourceConfig` helper redacts it from every serializer that returns raw config, and a CI guard (`scripts/check-source-config-leak.sh`) blocks the bug class going forward.
|
||||
- **One-time secret reveal.** `gbrain sources webhook set <id>` prints the secret once. `gbrain sources webhook show <id>` is metadata-only. Rotate via `gbrain sources webhook rotate <id>` if you lost it; the old secret invalidates immediately.
|
||||
- **Embed-backfill budget caps.** Two layers: per-job ($10 default, override via `embed.backfill_max_usd`) and per-source-per-24h ($25 default, override via `embed.backfill_max_usd_per_source_24h`). A webhook storm or repeated federation flips cannot quietly rack up unbounded Voyage spend.
|
||||
- **Webhook ref filtering.** Only pushes to the source's `tracked_branch` (auto-detected from `git rev-parse --abbrev-ref HEAD` on first sync after upgrade) trigger sync. Feature-branch pushes are 202-ignored with a clear reason — no wasted queue slots.
|
||||
- **Parallel sync defers embed.** `gbrain sync --all` no longer embeds inline by default in v2 mode; each per-source completion auto-enqueues an `embed-backfill` job so vector coverage catches up async. Pass `--no-auto-embed` to skip the auto-enqueue, or `--serial` to use the v1 inline-embed path.
|
||||
|
||||
### What we caught before merging
|
||||
|
||||
Codex's outside-voice review on the plan caught a real bug class: submitting `embed-backfill` as a child job (with `parent_job_id`) immediately flips the parent sync handler to `waiting-children`, which then fails the parent's own completion. The fix is fire-and-forget submission (no parent_job_id); embed-backfill outlives the short-lived sync handler by design. Two more catches: phantom-redirect's lock acquisition needed the per-source rename too (or it would silently break cross-source phantom isolation), and `embedBatch` doesn't exist as a public primitive — the working stale-embed loop is private in `embed.ts`. v0.40 extracts `embedStaleForSource` to `src/core/embed-stale.ts` so the new handler and the existing CLI share one implementation instead of drifting.
|
||||
|
||||
The webhook handler had a different bug — caught at test-writing time. `Buffer.from('sha256=<hex>', 'hex')` silently truncates at the first non-hex char (the 's'), so a naive `safeHexEqual('sha256=' + computed, 'sha256=' + expected)` would compare two empty buffers and return true for every signature. Fix: strip the `sha256=` prefix before the constant-time compare. Pinned by a `test/sources-webhook.test.ts` IRON-RULE case.
|
||||
|
||||
## To take advantage of v0.40.5.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the migration manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Verify the new surfaces:**
|
||||
```bash
|
||||
gbrain sources status # per-source health dashboard
|
||||
gbrain doctor # check federation_health
|
||||
gbrain sync trigger --help # confirm trigger subcommand wired
|
||||
```
|
||||
3. **Opt out (if needed):** `gbrain config set sync.federated_v2 false && gbrain jobs supervisor restart` reverts to pre-v0.40.5 sequential behavior. The per-source lock + migration v90 stay on regardless.
|
||||
4. **If any step fails,** file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and `~/.gbrain/upgrade-errors.jsonl` if it exists.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Federated Sync v2 — parallel, push-triggered, minion-native (6 components):**
|
||||
|
||||
1. **Per-source sync lock** (`src/core/db-lock.ts`): `syncLockId(sourceId)` replaces the global `SYNC_LOCK_ID`. Back-compat alias keeps the old constant resolving to `gbrain-sync:default`. Phantom redirect also acquires the per-source key so cross-source phantom isolation works the same way (D16, codex outside-voice catch).
|
||||
2. **Parallel `sync --all`** (`src/commands/sync.ts` + new `src/core/parallel.ts`): `Promise.allSettled` fan-out with `--max-sources N` cap. `--serial` opts back into v1. PGLite forces serial (single-writer constraint).
|
||||
3. **`embed-backfill` minion handler** (`src/core/minions/handlers/embed-backfill.ts` + `src/core/embed-backfill-submit.ts` + `src/core/embed-stale.ts`): decoupled embedding pipeline. Per-source DB lock at handler entry (D2). $10/job BudgetTracker cap (D6). Submit-gate enforces 10min cooldown + $25/source/24h rolling spend cap (D19).
|
||||
4. **Extended `sync` handler** (`src/commands/jobs.ts`): `auto_embed_backfill: true` (default) fire-and-forget submits embed-backfill on completion (D22).
|
||||
5. **`sync trigger` CLI** + **`POST /webhooks/github`**: push-triggered. Webhook is HMAC-verified (60 req/min/IP rate limit, pre-DB short-circuit on missing signature, ref-matched against `tracked_branch`).
|
||||
6. **`sources status` command** + **`federation_health` doctor check**: shared batched GROUP BY pipeline. 4 queries total instead of 6×N per-source round-trips.
|
||||
|
||||
**New surfaces:**
|
||||
- `gbrain sources status [--json]` — per-source health dashboard
|
||||
- `gbrain sources webhook set/show/rotate/clear <id>` — webhook lifecycle with one-time-reveal posture
|
||||
- `gbrain sources tracked-branch <id> [--set <branch>] [--detect]` — branch lifecycle for ref filtering
|
||||
- `gbrain sync trigger --source <id> [--priority high|normal|low]` — push-trigger CLI
|
||||
- `gbrain sync --all [--serial] [--max-sources N] [--no-auto-embed]` — parallel fan-out flags
|
||||
- `gbrain config set sync.federated_v2 false` — v0.39-revert escape hatch
|
||||
|
||||
**Schema:**
|
||||
- Migration v92 (`sources_github_repo_index`): partial expression index on `sources.config->>'github_repo'` for fast webhook source-lookup. Both engines.
|
||||
|
||||
**Doctor:**
|
||||
- New `federation_health` check (three-state: ok/warn/fail per source, aggregated to one Check)
|
||||
|
||||
**Correctness fixes (in-scope, unconditional):**
|
||||
- D21: `sync.ts` facts backstop now passes `sourceId` to `engine.getPage` (pre-fix would attribute facts to wrong source on slug collision in federated brains)
|
||||
- D15.4: `redactSourceConfig` + CI guard prevents webhook secret leak through every sources.config serializer
|
||||
- D15.5: `safeHexEqual` extracted to `src/core/timing-safe.ts` (shared with the new webhook HMAC verify)
|
||||
|
||||
**Tests:**
|
||||
- 10 new test files, 70+ new test cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID back-compat, phantom per-source lock, embed-backfill kill+resume, webhook HMAC prefix-strip).
|
||||
|
||||
**Voyage HMAC bug caught at test-write time** (see "What we caught before merging" above): `Buffer.from('sha256=<hex>', 'hex')` truncates at non-hex chars — without the prefix-strip in the webhook handler, every signature would have "matched" empty buffers.
|
||||
|
||||
## [0.40.4.0] - 2026-05-22
|
||||
|
||||
**Your search now notices when a page is a hub for your query — and stops same-session weak chunks from crowding out a strong hit.**
|
||||
@@ -107,6 +219,7 @@ This wave grew significantly through review (boil-the-lakes orientation):
|
||||
- New shared primitive `src/core/audit/audit-writer.ts` is the way to add JSONL audits going forward. Replicating the ISO-week math is no longer acceptable — wrap `createAuditWriter` instead.
|
||||
- `pairedBootstrapPValue` is exported from `test/e2e/graph-signals-eval.test.ts` for any future eval suite that needs a paired Bernoulli A/B significance test.
|
||||
- Five TODOs filed for v0.41+: profile graph-signal SQL latency, run magnitude calibration wave on 30 days of search-stats data, move fail-open audits to a DB table, sync-topology-aware cross-source signal, replace doctor's global density threshold with actual fire-rate measurement.
|
||||
|
||||
## [0.40.3.0] - 2026-05-22
|
||||
|
||||
**Your search now understands what each chunk is about — AND its cached results expire the moment a page changes.**
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.40.4.0",
|
||||
"version": "0.40.5.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -40,7 +40,8 @@
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:no-pii-agent-voice && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:source-config-leak && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:no-pii-agent-voice && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck",
|
||||
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
||||
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
|
||||
110
scripts/check-source-config-leak.sh
Executable file
110
scripts/check-source-config-leak.sh
Executable file
@@ -0,0 +1,110 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.40 D15.4 CI guard — prevent webhook_secret leak through sources.config
|
||||
# serialization paths.
|
||||
#
|
||||
# After v0.40, sources.config can contain secrets (webhook_secret). Any code
|
||||
# path that returns the raw config object via JSON.stringify / serializer
|
||||
# without first running it through redactSourceConfig() will leak the secret.
|
||||
#
|
||||
# This script greps for risky patterns:
|
||||
# 1. JSON.stringify on a `config` field where the source is a row from `sources`
|
||||
# 2. New endpoints / ops that return raw `config` without `redactSourceConfig`
|
||||
#
|
||||
# Failure mode is loose-positive on purpose — false positives cost one
|
||||
# 30-second comment-or-fix; false negatives leak production secrets.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
FOUND=0
|
||||
|
||||
# Pattern A: sources.config field referenced in a JSON serializer call site
|
||||
# without redactSourceConfig nearby. Covers MCP op handlers, admin API
|
||||
# routes, sources.ts subcommands that print --json output.
|
||||
#
|
||||
# Whitelist:
|
||||
# - src/core/source-config-redact.ts itself (defines the redactor)
|
||||
# - src/core/sources-load.ts (returns raw rows; callers redact)
|
||||
# - src/commands/sources.ts runFederate/runWebhook* (mutators write raw)
|
||||
# - src/core/migrate.ts (DDL data references not serialization)
|
||||
# - src/core/sources-ops.ts (CLI feedback prints structured fields, not raw config)
|
||||
# - test/ (tests are allowed to introspect raw config)
|
||||
|
||||
# Grep for `r.config\|src.config\|source.config` near JSON.stringify/console.log/res.json
|
||||
# where redactSourceConfig is NOT used in the same hunk.
|
||||
RAW_PATTERN='\b(\.config\b|config:[[:space:]]*src\.config)\b'
|
||||
|
||||
# Tightened patterns: match serializers that pass a source-row's .config
|
||||
# field (source.config, src.config, row.config, s.config, or a property
|
||||
# access like `.config` on an object likely sourced from a `sources` row),
|
||||
# NOT every variable named "config" (which would catch global gbrain config).
|
||||
#
|
||||
# The risk pattern is `JSON.stringify(<srcVar>.config)` where srcVar holds
|
||||
# a row from the sources table. Variables that hold the GLOBAL gbrain
|
||||
# config.json are also commonly named `config` — that's a different shape
|
||||
# and a different threat model (already protected at the file-mode 0o600
|
||||
# write site in src/core/config.ts).
|
||||
#
|
||||
# rg uses `-g` for globs; grep -rE uses `--include`. Branch accordingly so
|
||||
# CI runners without rg still match cleanly.
|
||||
if command -v rg >/dev/null 2>&1; then
|
||||
CANDIDATES=$(rg -n \
|
||||
-e 'JSON\.stringify\((source|src|row|s)\.config' \
|
||||
-e 'res\.json\((source|src|row|s)\.config' \
|
||||
-e 'res\.json\(\{[^}]*\.config[^.]' \
|
||||
-e 'console\.log\(JSON\.stringify\((source|src|row|s)\.config' \
|
||||
-g '*.ts' \
|
||||
src/ 2>/dev/null || true)
|
||||
else
|
||||
CANDIDATES=$(grep -rEn \
|
||||
-e 'JSON\.stringify\((source|src|row|s)\.config' \
|
||||
-e 'res\.json\((source|src|row|s)\.config' \
|
||||
-e 'res\.json\(\{[^}]*\.config[^.]' \
|
||||
-e 'console\.log\(JSON\.stringify\((source|src|row|s)\.config' \
|
||||
--include='*.ts' \
|
||||
src/ 2>/dev/null || true)
|
||||
fi
|
||||
|
||||
# Filter out files we trust (handle sources.config redaction themselves OR
|
||||
# handle the gbrain global config, which is a different object).
|
||||
FILTERED=$(echo "$CANDIDATES" | \
|
||||
grep -v 'src/core/source-config-redact.ts' | \
|
||||
grep -v 'src/core/sources-load.ts' | \
|
||||
grep -v 'src/commands/sources.ts' | \
|
||||
grep -v 'src/core/migrate.ts' | \
|
||||
grep -v 'src/core/sources-ops.ts' | \
|
||||
grep -v 'src/commands/init.ts' | \
|
||||
grep -v 'src/core/config.ts' || true)
|
||||
|
||||
if [ -n "$FILTERED" ]; then
|
||||
# For each candidate, check if redactSourceConfig appears within 10 lines above.
|
||||
while IFS= read -r LINE; do
|
||||
[ -z "$LINE" ] && continue
|
||||
FILE=$(echo "$LINE" | cut -d: -f1)
|
||||
LINENO=$(echo "$LINE" | cut -d: -f2)
|
||||
# Look in surrounding 20 lines
|
||||
START=$((LINENO - 10))
|
||||
[ "$START" -lt 1 ] && START=1
|
||||
END=$((LINENO + 5))
|
||||
CONTEXT=$(sed -n "${START},${END}p" "$FILE" 2>/dev/null || true)
|
||||
if ! echo "$CONTEXT" | grep -q 'redactSourceConfig'; then
|
||||
echo "POTENTIAL_LEAK: $LINE"
|
||||
echo " Context lacks redactSourceConfig — verify webhook_secret cannot be serialized."
|
||||
FOUND=1
|
||||
fi
|
||||
done <<< "$FILTERED"
|
||||
fi
|
||||
|
||||
if [ "$FOUND" -eq 1 ]; then
|
||||
echo ""
|
||||
echo "v0.40 D15.4 guard: every sources.config serializer MUST go through"
|
||||
echo "redactSourceConfig() from src/core/source-config-redact.ts."
|
||||
echo ""
|
||||
echo "If a flagged site is a known false positive (e.g. CLI command that"
|
||||
echo "only prints metadata, not the raw object), update the whitelist in"
|
||||
echo "scripts/check-source-config-leak.sh."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
114
skills/migrations/v0.40.5.md
Normal file
114
skills/migrations/v0.40.5.md
Normal file
@@ -0,0 +1,114 @@
|
||||
---
|
||||
version: v0.40.5.0
|
||||
feature_pitch: Federated Sync v2 — parallel source sync, push-triggered webhooks, async embedding, per-source health dashboard
|
||||
target_files:
|
||||
- sources.config (per-source webhook_secret, github_repo, tracked_branch, priority)
|
||||
- sources.config.federated (auto-embed-backfill on flip)
|
||||
manual_steps: required for users with multiple federated sources
|
||||
---
|
||||
|
||||
# v0.40.5.0 — Federated Sync v2
|
||||
|
||||
GBrain now treats every federated source as an independent unit. Cross-source sync runs in parallel, GitHub pushes can trigger sync within seconds, embedding decouples from the sync pipeline so onboarding doesn't block, and `gbrain sources status` shows per-source health.
|
||||
|
||||
## What the orchestrator did automatically
|
||||
|
||||
`gbrain apply-migrations --yes` (run by `gbrain upgrade`) handled:
|
||||
|
||||
1. **Migration v92** — partial expression index on `sources.config->>'github_repo'` for fast webhook source-lookup
|
||||
2. **Schema bootstrap probe** — legacy brains pick up the same index via `initSchema()`
|
||||
3. **Per-source sync lock** — `SYNC_LOCK_ID` constant now resolves to `gbrain-sync:default`; per-source sync acquires `gbrain-sync:<source_id>`
|
||||
4. **Phantom-redirect lock parity** — phantom-redirect pass uses the same per-source key so cross-source isolation works the same way
|
||||
5. **Facts backstop source-scoping fix** — `performSync` now passes sourceId to `engine.getPage` (previously could mis-attribute facts to wrong source on slug collision)
|
||||
|
||||
## What you should do manually
|
||||
|
||||
Run these steps once per machine after `gbrain upgrade`:
|
||||
|
||||
### 1. Verify the new surfaces work
|
||||
|
||||
```bash
|
||||
# Per-source health dashboard
|
||||
gbrain sources status
|
||||
|
||||
# Doctor should surface federation_health
|
||||
gbrain doctor
|
||||
|
||||
# Sync trigger CLI smoke
|
||||
gbrain sync trigger --help
|
||||
```
|
||||
|
||||
### 2. Set up webhook-driven sync per source
|
||||
|
||||
For each federated source whose repo lives on GitHub, register a webhook:
|
||||
|
||||
```bash
|
||||
gbrain sources webhook set <source-id> --github-repo <owner>/<repo>
|
||||
# Save the printed secret — you only see it once
|
||||
```
|
||||
|
||||
Then in the GitHub repo settings → Webhooks → Add webhook:
|
||||
|
||||
- **Payload URL:** `<your gbrain serve --http URL>/webhooks/github`
|
||||
- **Content type:** `application/json`
|
||||
- **Secret:** (the secret printed by `webhook set`)
|
||||
- **Events:** Just the push event
|
||||
- **Active:** checked
|
||||
|
||||
To rotate later: `gbrain sources webhook rotate <source-id>`. The new secret invalidates the old one immediately; update the GitHub webhook config to match.
|
||||
|
||||
### 3. (Optional) Set source priority
|
||||
|
||||
Per-source priority drives autopilot dispatch order. The default brain typically wants `high`; archival sources can use `low`:
|
||||
|
||||
```bash
|
||||
# Hand-edit sources.config via SQL or via a future `gbrain sources config set` command
|
||||
# In v0.40.5 set via direct SQL:
|
||||
gbrain exec "UPDATE sources SET config = jsonb_set(config, '{priority}', '\"high\"') WHERE id = 'default'"
|
||||
gbrain exec "UPDATE sources SET config = jsonb_set(config, '{priority}', '\"low\"') WHERE id = 'media-corpus'"
|
||||
```
|
||||
|
||||
Recognized values: `high` (priority -10), `normal` (0), `low` (5). Unknown values fall back to `normal` with a once-per-source stderr warning.
|
||||
|
||||
### 4. (Optional) Tune embed-backfill budget caps
|
||||
|
||||
Default caps: $10/job, $25/source/24h. Override via DB config:
|
||||
|
||||
```bash
|
||||
gbrain config set embed.backfill_max_usd 20 # raise per-job cap
|
||||
gbrain config set embed.backfill_max_usd_per_source_24h 100 # raise 24h cap
|
||||
gbrain config set embed.backfill_cooldown_min 30 # raise cooldown window
|
||||
```
|
||||
|
||||
### 5. (Optional) Opt out of parallel sync entirely
|
||||
|
||||
If something breaks and you need v0.39 behavior immediately:
|
||||
|
||||
```bash
|
||||
gbrain config set sync.federated_v2 false
|
||||
gbrain jobs supervisor restart
|
||||
```
|
||||
|
||||
Per-source lock + migration v92 stay on regardless (those are correctness fixes, not features).
|
||||
|
||||
## Verify the outcome
|
||||
|
||||
After completing the steps above:
|
||||
|
||||
1. **Parallel sync works:** `gbrain sync --all` should complete noticeably faster than before. If multiple sources used to take minutes-to-hours, you should see them run concurrently.
|
||||
2. **Webhook fires:** `git push` to a webhook-configured source should produce a `[dispatch]` line in autopilot logs (or `gbrain jobs list` should show a fresh `sync` job within seconds).
|
||||
3. **Dashboard is accurate:** `gbrain sources status` should match the numbers from `gbrain stats` per source.
|
||||
4. **No regressions:** Existing `gbrain sync --source <id>` still works the same way; `gbrain sync` with no flags still syncs the default source.
|
||||
|
||||
## If something goes wrong
|
||||
|
||||
`gbrain doctor` is your first stop:
|
||||
|
||||
- `federation_health: warn` — surfaces lag, embed coverage, or queue depth issues per source with paste-ready remediation commands
|
||||
- `federation_health: fail` — surfaces critical lag (>24h) or embed coverage (<50% with >1000 chunks)
|
||||
|
||||
For partial states or unexpected errors, file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- Output of `gbrain doctor --json`
|
||||
- Output of `gbrain sources status --json`
|
||||
- Contents of `~/.gbrain/upgrade-errors.jsonl` if present
|
||||
- Which step broke
|
||||
@@ -417,6 +417,14 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
// autopilot-cycle can never run concurrently (both acquire
|
||||
// gbrain-cycle), so the "60-min floor double-processes queued
|
||||
// targeted jobs" failure mode is closed by the lock.
|
||||
//
|
||||
// v0.40 D17 layered on top: per-source freshness check fires BEFORE
|
||||
// the score gate so a healthy brain that happens to have a stale
|
||||
// federated source still picks up new commits. brain_score reflects
|
||||
// internal data quality (embed coverage, link density, orphans),
|
||||
// NOT whether GitHub has new commits on the source repo. Decoupling
|
||||
// the two closes the silent-stale-source bug class on
|
||||
// poll-only deployments.
|
||||
try {
|
||||
const { MinionQueue } = await import('../core/minions/queue.ts');
|
||||
const { computeRecommendations } = await import('../core/brain-score-recommendations.ts');
|
||||
@@ -425,6 +433,57 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const slot = new Date(slotMs).toISOString();
|
||||
const timeoutMs = Math.max(baseInterval * 2 * 1000, 300_000);
|
||||
|
||||
// ── v0.40 D17: per-source freshness check ────────────────────
|
||||
// Runs first; independent of score gate. Submits a 'sync' job per
|
||||
// source whose last_sync_at is older than the interval. The sync
|
||||
// handler (T6/T7) auto-enqueues embed-backfill on completion if
|
||||
// pages changed.
|
||||
try {
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
if (await isFederatedV2Enabled(engine)) {
|
||||
const { loadAllSources } = await import('../core/sources-load.ts');
|
||||
const sources = await loadAllSources(engine);
|
||||
const intervalMs = baseInterval * 1000;
|
||||
const now = Date.now();
|
||||
for (const src of sources) {
|
||||
if (!src.local_path) continue;
|
||||
const lastSyncMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : 0;
|
||||
const ageMs = now - lastSyncMs;
|
||||
if (ageMs < intervalMs) continue; // fresh enough
|
||||
try {
|
||||
const job = await queue.add(
|
||||
'sync',
|
||||
{
|
||||
sourceId: src.id,
|
||||
repoPath: src.local_path,
|
||||
auto_embed_backfill: true,
|
||||
embed_reason: 'autopilot_freshness',
|
||||
},
|
||||
{
|
||||
queue: 'default',
|
||||
idempotency_key: `autopilot-sync:${src.id}:${slot}`,
|
||||
max_attempts: 2,
|
||||
timeout_ms: timeoutMs,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({
|
||||
event: 'dispatched', job_id: job.id, mode: 'freshness',
|
||||
source_id: src.id, age_ms: ageMs,
|
||||
}) + '\n');
|
||||
} else {
|
||||
console.log(`[dispatch] job #${job.id} sync (freshness: ${src.id}; age=${Math.floor(ageMs / 60000)}min)`);
|
||||
}
|
||||
} catch (e) {
|
||||
logError('dispatch.freshness', e);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
logError('dispatch.freshness-gate', e);
|
||||
}
|
||||
|
||||
// Cheap path: engine.getHealth() is a single SQL count query.
|
||||
const health = await engine.getHealth();
|
||||
const score = health.brain_score;
|
||||
|
||||
@@ -625,6 +625,12 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// - synopsis-failures audit JSONL entries from the last 7 days
|
||||
checks.push(await checkContextualRetrievalCoverage(engine));
|
||||
|
||||
// 12. v0.40.5.0 Federated Sync v2 (T12) — federation_health:
|
||||
// - Per-source lag, embed coverage, failed-job rate.
|
||||
// - Single-source brain short-circuits to ok.
|
||||
// - Three-state: ok / warn / fail.
|
||||
checks.push(await checkFederationHealth(engine));
|
||||
|
||||
return computeDoctorReport(checks);
|
||||
}
|
||||
|
||||
@@ -1386,6 +1392,84 @@ export async function checkSourceRoutingHealth(engine: BrainEngine): Promise<Che
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40 Federated Sync v2 (T12) — federation_health.
|
||||
*
|
||||
* Per-source dashboard surface for the autopilot/operator.
|
||||
* Three-state per-source (then aggregated to single Check):
|
||||
*
|
||||
* ok — all federated sources synced within 1h AND embed coverage >=95%
|
||||
* (or chunks <100), AND failed_jobs_24h < 3
|
||||
* warn — any source has lag > 1h + federated, OR coverage < 95% with
|
||||
* chunks > 100, OR failed_jobs_24h >= 3
|
||||
* fail — any source has lag > 24h, OR coverage < 50% with chunks > 1000
|
||||
*
|
||||
* Single-source brain short-circuits to ok (no federation to check).
|
||||
* Each warning carries a paste-ready remediation hint.
|
||||
*/
|
||||
export async function checkFederationHealth(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { loadAllSources } = await import('../core/sources-load.ts');
|
||||
const { computeAllSourceMetrics } = await import('../core/source-health.ts');
|
||||
const sources = await loadAllSources(engine, { includeArchived: false });
|
||||
if (sources.length <= 1) {
|
||||
return {
|
||||
name: 'federation_health',
|
||||
status: 'ok',
|
||||
message: 'Single-source brain (no federation to check)',
|
||||
};
|
||||
}
|
||||
const metrics = await computeAllSourceMetrics(engine, sources);
|
||||
|
||||
const warns: string[] = [];
|
||||
const fails: string[] = [];
|
||||
for (const m of metrics) {
|
||||
// Fail thresholds first (most severe)
|
||||
if (m.lag_seconds !== null && m.lag_seconds > 24 * 3600) {
|
||||
fails.push(`${m.source_id}: stale ${Math.floor(m.lag_seconds / 3600)}h — run \`gbrain sync trigger --source ${m.source_id}\``);
|
||||
continue;
|
||||
}
|
||||
if (m.embed_coverage_pct < 50 && m.total_chunks > 1000) {
|
||||
fails.push(`${m.source_id}: ${m.embed_coverage_pct.toFixed(1)}% embed coverage (${m.total_chunks.toLocaleString()} chunks) — run \`gbrain jobs submit embed-backfill --params '{"sourceId":"${m.source_id}"}'\``);
|
||||
continue;
|
||||
}
|
||||
// Warns
|
||||
if (m.federated && m.lag_seconds !== null && m.lag_seconds > 3600) {
|
||||
warns.push(`${m.source_id}: federated source ${Math.floor(m.lag_seconds / 3600)}h+ stale — run \`gbrain sync trigger --source ${m.source_id}\``);
|
||||
}
|
||||
if (m.embed_coverage_pct < 95 && m.total_chunks > 100) {
|
||||
warns.push(`${m.source_id}: ${m.embed_coverage_pct.toFixed(1)}% embed coverage — run \`gbrain jobs submit embed-backfill --params '{"sourceId":"${m.source_id}"}'\``);
|
||||
}
|
||||
if (m.failed_jobs_24h >= 3) {
|
||||
warns.push(`${m.source_id}: ${m.failed_jobs_24h} failures in 24h — check \`gbrain jobs list --status failed\``);
|
||||
}
|
||||
}
|
||||
|
||||
if (fails.length > 0) {
|
||||
return {
|
||||
name: 'federation_health',
|
||||
status: 'fail',
|
||||
message: `${fails.length} federation failure(s):\n ${fails.join('\n ')}`,
|
||||
};
|
||||
}
|
||||
if (warns.length > 0) {
|
||||
return {
|
||||
name: 'federation_health',
|
||||
status: 'warn',
|
||||
message: `${warns.length} federation warning(s):\n ${warns.join('\n ')}`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
name: 'federation_health',
|
||||
status: 'ok',
|
||||
message: `${metrics.length} source(s) healthy (parallel sync, async embed)`,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'federation_health', status: 'warn', message: `Check failed: ${msg}` };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.37.7.0 — Tier 5L oauth_confidential_client_health.
|
||||
*
|
||||
|
||||
@@ -1067,7 +1067,43 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
repoPath, sourceId, noPull, noEmbed, noExtract,
|
||||
concurrency: concurrencyOverride,
|
||||
});
|
||||
return result;
|
||||
|
||||
// v0.40 D22: auto_embed_backfill defaults TRUE when sourceId is set AND
|
||||
// the feature flag is enabled. Submits a child embed-backfill job
|
||||
// (fire-and-forget — D15.1) so stale chunks get embedded async without
|
||||
// the sync handler waiting on the embed pipeline.
|
||||
const autoEmbed = job.data.auto_embed_backfill !== false;
|
||||
let embedJobId: number | null = null;
|
||||
let embedSkipReason: string | null = null;
|
||||
if (autoEmbed && sourceId && result.status !== 'up_to_date' && result.status !== 'dry_run') {
|
||||
try {
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
if (await isFederatedV2Enabled(engine)) {
|
||||
const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts');
|
||||
const submission = await submitEmbedBackfill(engine, sourceId, {
|
||||
reason: typeof job.data.embed_reason === 'string'
|
||||
? (job.data.embed_reason as string)
|
||||
: 'sync_handler',
|
||||
});
|
||||
if (submission.status === 'submitted') {
|
||||
embedJobId = submission.jobId ?? null;
|
||||
} else {
|
||||
embedSkipReason = submission.status;
|
||||
}
|
||||
} else {
|
||||
embedSkipReason = 'feature_flag_disabled';
|
||||
}
|
||||
} catch (err) {
|
||||
// Embed-backfill submission failure must NOT fail the sync job.
|
||||
embedSkipReason = `submit_error:${err instanceof Error ? err.message : String(err)}`;
|
||||
}
|
||||
} else if (!sourceId) {
|
||||
embedSkipReason = 'no_source_id';
|
||||
} else if (!autoEmbed) {
|
||||
embedSkipReason = 'auto_embed_disabled';
|
||||
}
|
||||
|
||||
return { ...result, embed_job_id: embedJobId, embed_skip_reason: embedSkipReason };
|
||||
});
|
||||
|
||||
worker.register('embed', async (job) => {
|
||||
@@ -1392,7 +1428,16 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
worker.register('resolve_symbol_edges', makePhaseHandler('resolve_symbol_edges'));
|
||||
worker.register('recompute_emotional_weight', makePhaseHandler('recompute_emotional_weight'));
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected)\n');
|
||||
// v0.40 Federated Sync v2 — embed-backfill: per-source decoupled embed.
|
||||
// Cost-bounded via D6 ($10/job BudgetTracker) + D19 (source-level cooldown
|
||||
// + 24h rolling cap, gated at submit time). NOT in PROTECTED_JOB_NAMES —
|
||||
// embedding-only spend, no API-by-the-minute risk like subagent.
|
||||
worker.register('embed-backfill', async (job) => {
|
||||
const { makeEmbedBackfillHandler } = await import('../core/minions/handlers/embed-backfill.ts');
|
||||
return await makeEmbedBackfillHandler(engine)(job);
|
||||
});
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (11 ops, 3 protected) + embed-backfill (v0.40)\n');
|
||||
|
||||
// Plugin discovery — one line per discovered plugin (mirrors the
|
||||
// openclaw-seam startup line convention from v0.11+). Loaded
|
||||
|
||||
@@ -15,7 +15,8 @@ import type { Request, Response, NextFunction } from 'express';
|
||||
import cookieParser from 'cookie-parser';
|
||||
import cors from 'cors';
|
||||
import rateLimit from 'express-rate-limit';
|
||||
import { randomBytes, createHash, timingSafeEqual } from 'crypto';
|
||||
import { randomBytes, createHash } from 'crypto';
|
||||
import { safeHexEqual } from '../core/timing-safe.ts';
|
||||
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';
|
||||
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
|
||||
@@ -592,17 +593,9 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
// ---------------------------------------------------------------------------
|
||||
// Admin authentication (cookie-based)
|
||||
// ---------------------------------------------------------------------------
|
||||
// v0.40 D15.5: safeHexEqual extracted to src/core/timing-safe.ts so the new
|
||||
// /webhooks/github HMAC verifier reuses the same constant-time compare.
|
||||
// POST /admin/login — JSON body with token (for programmatic/UI login)
|
||||
// Constant-time hex compare. Both inputs are sha256 hex (64 chars),
|
||||
// so they're always equal length. timingSafeEqual throws on length
|
||||
// mismatch — we already short-circuit on non-string above. Catches
|
||||
// would-be timing oracles even though the inputs are pre-hashed
|
||||
// (defense-in-depth on the hash bits).
|
||||
function safeHexEqual(a: string, b: string): boolean {
|
||||
if (a.length !== b.length) return false;
|
||||
return timingSafeEqual(Buffer.from(a, 'hex'), Buffer.from(b, 'hex'));
|
||||
}
|
||||
|
||||
app.post('/admin/login', express.json(), (req, res) => {
|
||||
const token = req.body?.token;
|
||||
if (!token || typeof token !== 'string') {
|
||||
@@ -1787,6 +1780,158 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
},
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// POST /webhooks/github — push-triggered sync (v0.40 Federated Sync v2)
|
||||
// ---------------------------------------------------------------------------
|
||||
// Anonymous endpoint by necessity (GitHub doesn't carry an OAuth token).
|
||||
// Auth is via per-source HMAC-SHA256 in the X-Hub-Signature-256 header.
|
||||
//
|
||||
// D3: 60 req/min/IP rate limit + pre-DB short-circuit on missing
|
||||
// signature, so probe traffic doesn't even touch the source-lookup
|
||||
// query.
|
||||
// D5: event=push AND ref-match against sources.config.tracked_branch.
|
||||
// Other event types (ping, pull_request, etc.) return 202 'ignored'
|
||||
// so GitHub doesn't retry.
|
||||
// D15.5: HMAC compare uses the shared safeHexEqual helper.
|
||||
// D18: submits 'sync' job with auto_embed_backfill=true and priority -10
|
||||
// (above autopilot's 0).
|
||||
// ---------------------------------------------------------------------------
|
||||
const githubWebhookLimiter = rateLimit({
|
||||
windowMs: 60_000,
|
||||
limit: 60,
|
||||
standardHeaders: 'draft-7',
|
||||
legacyHeaders: false,
|
||||
message: { error: 'rate_limit_exceeded', message: 'too many GitHub webhook requests' },
|
||||
});
|
||||
|
||||
app.post(
|
||||
'/webhooks/github',
|
||||
githubWebhookLimiter,
|
||||
express.raw({ type: '*/*', limit: '1mb' }),
|
||||
async (req: Request, res: Response) => {
|
||||
// D3 pre-DB short-circuit: missing signature → 401 without any
|
||||
// source lookup. Bot probe traffic ends here.
|
||||
const sigHeader = req.header('X-Hub-Signature-256');
|
||||
if (!sigHeader) {
|
||||
res.status(401).json({ error: 'missing_signature', message: 'X-Hub-Signature-256 header is required' });
|
||||
return;
|
||||
}
|
||||
|
||||
// D5: filter by event header. GitHub fires webhooks for every event
|
||||
// type. Anything other than 'push' is acknowledged with 202 + reason
|
||||
// so GitHub doesn't retry — but no source lookup or job submission.
|
||||
const event = req.header('X-GitHub-Event') ?? '';
|
||||
if (event !== 'push') {
|
||||
res.status(202).json({ status: 'ignored', reason: `event=${event || '(missing)'}` });
|
||||
return;
|
||||
}
|
||||
|
||||
const payload = Buffer.isBuffer(req.body) ? req.body : Buffer.from(JSON.stringify(req.body), 'utf8');
|
||||
if (payload.length === 0) {
|
||||
res.status(400).json({ error: 'empty_body' });
|
||||
return;
|
||||
}
|
||||
|
||||
let parsed: { repository?: { full_name?: string }; ref?: string };
|
||||
try {
|
||||
parsed = JSON.parse(payload.toString('utf8'));
|
||||
} catch {
|
||||
res.status(400).json({ error: 'malformed_json' });
|
||||
return;
|
||||
}
|
||||
|
||||
const fullName = parsed.repository?.full_name;
|
||||
const ref = parsed.ref;
|
||||
if (!fullName || !ref) {
|
||||
res.status(400).json({ error: 'missing_fields', message: 'repository.full_name and ref are required' });
|
||||
return;
|
||||
}
|
||||
|
||||
// Source lookup via the v87 partial expression index on
|
||||
// config->>'github_repo'. fast even on large brains.
|
||||
let source: { id: string; config: Record<string, unknown> | string } | null = null;
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ id: string; config: Record<string, unknown> | string }>(
|
||||
`SELECT id, config FROM sources WHERE config->>'github_repo' = $1 LIMIT 1`,
|
||||
[fullName],
|
||||
);
|
||||
source = rows[0] ?? null;
|
||||
} catch (err) {
|
||||
console.error('webhook: source lookup error:', err);
|
||||
res.status(500).json({ error: 'lookup_failed' });
|
||||
return;
|
||||
}
|
||||
if (!source) {
|
||||
res.status(404).json({ error: 'unknown_repo', repo: fullName });
|
||||
return;
|
||||
}
|
||||
|
||||
const cfg = (typeof source.config === 'string' ? JSON.parse(source.config) : source.config) as {
|
||||
webhook_secret?: string;
|
||||
tracked_branch?: string;
|
||||
};
|
||||
|
||||
// D5: ref must match the configured tracked branch (default 'main').
|
||||
const trackedBranch = cfg.tracked_branch ?? 'main';
|
||||
const expectedRef = `refs/heads/${trackedBranch}`;
|
||||
if (ref !== expectedRef) {
|
||||
res.status(202).json({
|
||||
status: 'ignored',
|
||||
reason: `ref_mismatch`,
|
||||
received_ref: ref,
|
||||
tracked_branch: trackedBranch,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const secret = cfg.webhook_secret;
|
||||
if (!secret || typeof secret !== 'string') {
|
||||
res.status(401).json({ error: 'webhook_not_configured', message: 'Run: gbrain sources webhook set ' + source.id });
|
||||
return;
|
||||
}
|
||||
|
||||
// HMAC verify. GitHub sends "sha256=<hex>" — strip the prefix BEFORE
|
||||
// safeHexEqual because Buffer.from('sha256=...', 'hex') silently
|
||||
// truncates at the first non-hex char (the 's'), leaving both
|
||||
// operands as 0-byte buffers and making every signature "match".
|
||||
// Pinned by test/sources-webhook.test.ts tamper assertions.
|
||||
const { createHmac } = await import('node:crypto');
|
||||
const computedHex = createHmac('sha256', secret).update(payload).digest('hex');
|
||||
const prefix = 'sha256=';
|
||||
if (!sigHeader.startsWith(prefix)) {
|
||||
res.status(401).json({ error: 'signature_mismatch', message: 'expected sha256= prefix' });
|
||||
return;
|
||||
}
|
||||
if (!safeHexEqual(sigHeader.slice(prefix.length), computedHex)) {
|
||||
res.status(401).json({ error: 'signature_mismatch' });
|
||||
return;
|
||||
}
|
||||
|
||||
// Submit sync job with priority -10 (above autopilot's 0).
|
||||
try {
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add(
|
||||
'sync',
|
||||
{
|
||||
sourceId: source.id,
|
||||
auto_embed_backfill: true,
|
||||
embed_reason: 'webhook',
|
||||
},
|
||||
{
|
||||
priority: -10,
|
||||
idempotency_key: `webhook:sync:${source.id}:${Math.floor(Date.now() / 30_000)}`,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
res.status(202).json({ job_id: job.id, source_id: source.id });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.error('webhook: queue submission error:', msg);
|
||||
res.status(500).json({ error: 'queue_submission_failed', message: msg });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Start server
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -47,6 +47,12 @@ import {
|
||||
resolveSourceWithTier,
|
||||
SOURCE_TIER_NAMES,
|
||||
} from '../core/source-resolver.ts';
|
||||
import {
|
||||
loadAllSources,
|
||||
parseSourceConfig,
|
||||
isSourceFederated,
|
||||
type SourceRow as LoadedSourceRow,
|
||||
} from '../core/sources-load.ts';
|
||||
|
||||
// ── Validation ──────────────────────────────────────────────
|
||||
|
||||
@@ -84,18 +90,10 @@ interface SourceListEntry {
|
||||
|
||||
// ── Helpers ─────────────────────────────────────────────────
|
||||
|
||||
function parseConfig(config: unknown): Record<string, unknown> {
|
||||
if (typeof config === 'string') {
|
||||
try { return JSON.parse(config) as Record<string, unknown>; } catch { return {}; }
|
||||
}
|
||||
if (typeof config === 'object' && config !== null) return config as Record<string, unknown>;
|
||||
return {};
|
||||
}
|
||||
|
||||
function isFederated(config: unknown): boolean {
|
||||
const parsed = parseConfig(config);
|
||||
return parsed.federated === true;
|
||||
}
|
||||
// v0.40 (D7): shared helpers — re-exported as local names for back-compat
|
||||
// with existing call sites that import `parseConfig`/`isFederated` by intent.
|
||||
const parseConfig = parseSourceConfig;
|
||||
const isFederated = isSourceFederated;
|
||||
|
||||
async function fetchSource(engine: BrainEngine, id: string): Promise<SourceRow | null> {
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
@@ -184,10 +182,10 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function runList(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, name, local_path, last_commit, last_sync_at, config, created_at
|
||||
FROM sources ORDER BY (id = 'default') DESC, id`,
|
||||
);
|
||||
// v0.40 (D7): loadAllSources is the single source of truth for source enum.
|
||||
// Pass includeArchived=true to preserve the legacy `runList` behavior of
|
||||
// surfacing archived rows (they get the ⚠ marker below).
|
||||
const rows: LoadedSourceRow[] = await loadAllSources(engine, { includeArchived: true });
|
||||
|
||||
const entries: SourceListEntry[] = [];
|
||||
for (const r of rows) {
|
||||
@@ -547,6 +545,290 @@ async function runFederate(engine: BrainEngine, args: string[], value: boolean):
|
||||
[JSON.stringify(config), id],
|
||||
);
|
||||
console.log(`Source "${id}" is now ${value ? 'federated (appears in cross-source default search)' : 'isolated (only searched when explicitly named)'}.`);
|
||||
|
||||
// v0.40 D19: auto-submit embed-backfill when coverage < 100%. Federation
|
||||
// flip is a moment when the user explicitly opted into seeing this source
|
||||
// in default search; un-embedded chunks would hide content from the very
|
||||
// moment they wanted visibility. Best-effort — submission failure does NOT
|
||||
// fail the flip.
|
||||
try {
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
if (!(await isFederatedV2Enabled(engine))) return;
|
||||
|
||||
const { loadAllSources } = await import('../core/sources-load.ts');
|
||||
const { computeAllSourceMetrics } = await import('../core/source-health.ts');
|
||||
const sources = await loadAllSources(engine, { includeArchived: false });
|
||||
const metrics = await computeAllSourceMetrics(engine, sources);
|
||||
const m = metrics.find((x) => x.source_id === id);
|
||||
if (!m || m.total_chunks === 0 || m.embed_coverage_pct >= 100) return;
|
||||
|
||||
const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts');
|
||||
const sub = await submitEmbedBackfill(engine, id, { reason: 'federation_flip' });
|
||||
if (sub.status === 'submitted') {
|
||||
const missing = m.total_chunks - m.embedded_chunks;
|
||||
console.log(` → embed-backfill job ${sub.jobId} queued for missing ${missing} chunks.`);
|
||||
} else if (sub.status === 'cooldown') {
|
||||
console.log(` → embed-backfill skipped (cooldown). Manually trigger with: gbrain jobs submit embed-backfill --params '{"sourceId":"${id}"}'`);
|
||||
} else if (sub.status === 'spend_capped') {
|
||||
console.log(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd} reached for this source).`);
|
||||
}
|
||||
} catch (err) {
|
||||
// Federation flip already succeeded; embed-backfill is a follow-up nicety.
|
||||
console.error(` → embed-backfill submission failed (flip succeeded): ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ── v0.40 sources status (D12) ──────────────────────────────
|
||||
async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const { loadAllSources } = await import('../core/sources-load.ts');
|
||||
const { computeAllSourceMetrics } = await import('../core/source-health.ts');
|
||||
const sources = await loadAllSources(engine, { includeArchived: false });
|
||||
if (sources.length === 0) {
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ schema_version: 1, sources: [] }, null, 2));
|
||||
} else {
|
||||
console.log('No sources registered. Use `gbrain sources add <id> --path <path>` first.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
const metrics = await computeAllSourceMetrics(engine, sources);
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ schema_version: 1, sources: metrics }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
// Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC
|
||||
console.log('SOURCES — health');
|
||||
console.log('────────────────');
|
||||
console.log(
|
||||
` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`,
|
||||
);
|
||||
for (const m of metrics) {
|
||||
const lag = m.lag_seconds === null
|
||||
? 'never'
|
||||
: formatLag(m.lag_seconds);
|
||||
const embed = `${m.embed_coverage_pct.toFixed(0)}%`;
|
||||
const fails = String(m.failed_jobs_24h);
|
||||
const queue = String(m.queue_depth);
|
||||
const pages = m.total_pages.toLocaleString();
|
||||
const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never';
|
||||
console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`);
|
||||
}
|
||||
console.log('');
|
||||
for (const m of metrics) {
|
||||
const warns: string[] = [];
|
||||
if (!m.local_path) warns.push('no local_path');
|
||||
if (m.lag_seconds === null) warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``);
|
||||
if (m.embed_coverage_pct < 95 && m.total_chunks > 100) {
|
||||
warns.push(`${(100 - m.embed_coverage_pct).toFixed(1)}% un-embedded — run \`gbrain embed --stale --source ${m.source_id}\``);
|
||||
}
|
||||
if (m.failed_jobs_24h >= 3) {
|
||||
warns.push(`${m.failed_jobs_24h} failures in 24h — check \`gbrain jobs list --status failed\``);
|
||||
}
|
||||
if (warns.length > 0) {
|
||||
console.log(` ⚠ ${m.source_id}: ${warns.join('; ')}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function formatLag(seconds: number): string {
|
||||
if (seconds < 60) return `${seconds}s`;
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)}m`;
|
||||
if (seconds < 86400) return `${Math.floor(seconds / 3600)}h`;
|
||||
return `${Math.floor(seconds / 86400)}d`;
|
||||
}
|
||||
|
||||
// ── v0.40 sources webhook (D8) ──────────────────────────────
|
||||
async function runWebhook(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
const rest = args.slice(1);
|
||||
switch (sub) {
|
||||
case 'set': return runWebhookSet(engine, rest);
|
||||
case 'show': return runWebhookShow(engine, rest);
|
||||
case 'rotate': return runWebhookRotate(engine, rest);
|
||||
case 'clear': return runWebhookClear(engine, rest);
|
||||
case undefined:
|
||||
case '--help':
|
||||
case '-h':
|
||||
console.log(`Usage: gbrain sources webhook <subcommand> <source-id> [options]
|
||||
|
||||
Subcommands:
|
||||
set <id> [--secret VAL] [--github-repo owner/name] One-time reveal
|
||||
show <id> Metadata only
|
||||
rotate <id> New secret, reveal
|
||||
clear <id> Remove webhook config`);
|
||||
return;
|
||||
default:
|
||||
console.error(`Unknown webhook subcommand: ${sub}`);
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
async function runWebhookSet(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources webhook set <id> [--secret VAL] [--github-repo owner/name]');
|
||||
process.exit(2);
|
||||
}
|
||||
const src = await fetchSource(engine, id);
|
||||
if (!src) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const explicitSecret = args.find((a, i) => args[i - 1] === '--secret');
|
||||
const githubRepo = args.find((a, i) => args[i - 1] === '--github-repo');
|
||||
if (!githubRepo) {
|
||||
console.error('--github-repo owner/name is required (e.g. "Garry-s-List/zion-brain")');
|
||||
process.exit(2);
|
||||
}
|
||||
if (!/^[\w.-]+\/[\w.-]+$/.test(githubRepo)) {
|
||||
console.error(`Invalid --github-repo format: "${githubRepo}". Expected "owner/name".`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const { randomBytes } = await import('node:crypto');
|
||||
const secret = explicitSecret ?? randomBytes(32).toString('hex');
|
||||
const cfg = parseConfig(src.config);
|
||||
cfg.webhook_secret = secret;
|
||||
cfg.github_repo = githubRepo;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
|
||||
console.log(`Webhook configured for source "${id}":`);
|
||||
console.log(` github_repo: ${githubRepo}`);
|
||||
console.log(` webhook_secret: ${secret}`);
|
||||
console.log('');
|
||||
console.log('--- Paste this into GitHub repo settings → Webhooks → Add webhook ---');
|
||||
console.log(' Payload URL: <your gbrain serve --http URL>/webhooks/github');
|
||||
console.log(' Content type: application/json');
|
||||
console.log(` Secret: ${secret}`);
|
||||
console.log(' Events: Just the push event');
|
||||
console.log(' Active: checked');
|
||||
console.log('');
|
||||
console.log('⚠ This secret is shown ONCE. Save it now; subsequent `gbrain sources webhook show` will NOT display it.');
|
||||
}
|
||||
|
||||
async function runWebhookShow(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources webhook show <id>');
|
||||
process.exit(2);
|
||||
}
|
||||
const src = await fetchSource(engine, id);
|
||||
if (!src) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const cfg = parseConfig(src.config);
|
||||
const githubRepo = typeof cfg.github_repo === 'string' ? cfg.github_repo : '(not set)';
|
||||
const secretSet = typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0;
|
||||
const trackedBranch = typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : '(auto-detected on next sync, default main)';
|
||||
|
||||
console.log(`Webhook configuration for source "${id}":`);
|
||||
console.log(` github_repo: ${githubRepo}`);
|
||||
console.log(` webhook_secret: ${secretSet ? '<set — use `webhook rotate` to reveal a new one>' : '(not set)'}`);
|
||||
console.log(` tracked_branch: ${trackedBranch}`);
|
||||
}
|
||||
|
||||
async function runWebhookRotate(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources webhook rotate <id>');
|
||||
process.exit(2);
|
||||
}
|
||||
const src = await fetchSource(engine, id);
|
||||
if (!src) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const { randomBytes } = await import('node:crypto');
|
||||
const secret = randomBytes(32).toString('hex');
|
||||
const cfg = parseConfig(src.config);
|
||||
cfg.webhook_secret = secret;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`New webhook secret for source "${id}":`);
|
||||
console.log(` ${secret}`);
|
||||
console.log('');
|
||||
console.log('⚠ Update the GitHub webhook config to use this new secret. The old one is invalidated immediately.');
|
||||
}
|
||||
|
||||
async function runWebhookClear(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources webhook clear <id>');
|
||||
process.exit(2);
|
||||
}
|
||||
const src = await fetchSource(engine, id);
|
||||
if (!src) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const cfg = parseConfig(src.config);
|
||||
delete cfg.webhook_secret;
|
||||
delete cfg.github_repo;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Webhook configuration cleared for source "${id}".`);
|
||||
}
|
||||
|
||||
// ── v0.40 sources tracked-branch (D20) ──────────────────────
|
||||
async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const id = args[0];
|
||||
if (!id) {
|
||||
console.error('Usage: gbrain sources tracked-branch <id> [--set <branch>] [--detect]');
|
||||
process.exit(2);
|
||||
}
|
||||
const src = await fetchSource(engine, id);
|
||||
if (!src) {
|
||||
console.error(`Source "${id}" not found.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const setArg = args.find((a, i) => args[i - 1] === '--set');
|
||||
const detect = args.includes('--detect');
|
||||
const cfg = parseConfig(src.config);
|
||||
|
||||
if (setArg) {
|
||||
cfg.tracked_branch = setArg;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Tracked branch for source "${id}" set to "${setArg}".`);
|
||||
return;
|
||||
}
|
||||
if (detect) {
|
||||
if (!src.local_path) {
|
||||
console.error(`Source "${id}" has no local_path; cannot auto-detect branch.`);
|
||||
process.exit(1);
|
||||
}
|
||||
try {
|
||||
const { execFileSync } = await import('node:child_process');
|
||||
const branch = execFileSync('git', ['-C', src.local_path, 'rev-parse', '--abbrev-ref', 'HEAD'], { encoding: 'utf8' }).trim();
|
||||
cfg.tracked_branch = branch;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Detected branch "${branch}" for source "${id}"; persisted to config.tracked_branch.`);
|
||||
} catch (e) {
|
||||
console.error(`git rev-parse failed: ${e instanceof Error ? e.message : String(e)}`);
|
||||
process.exit(1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Read mode: just print
|
||||
const tracked = typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : '(unset — defaults to main)';
|
||||
console.log(`Source "${id}" tracked_branch: ${tracked}`);
|
||||
}
|
||||
|
||||
// ── `sources current` (v0.37.7.0) ──────────────────────────
|
||||
@@ -615,6 +897,11 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
|
||||
case 'purge': return runPurge(engine, rest);
|
||||
case 'archived': return runListArchived(engine, rest);
|
||||
case 'current': return runCurrent(engine, rest);
|
||||
// v0.40.5.0 Federated Sync v2
|
||||
case 'status': return runStatus(engine, rest);
|
||||
case 'webhook': return runWebhook(engine, rest);
|
||||
case 'tracked-branch': return runTrackedBranch(engine, rest);
|
||||
// v0.40.3.0 contextual retrieval (from master)
|
||||
case 'set-cr-mode': return runSetCrMode(engine, rest);
|
||||
case undefined:
|
||||
case '--help':
|
||||
|
||||
@@ -26,7 +26,7 @@ import {
|
||||
shouldRunParallel,
|
||||
parseWorkers,
|
||||
} from '../core/sync-concurrency.ts';
|
||||
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../core/db-lock.ts';
|
||||
import { tryAcquireDbLock, syncLockId } from '../core/db-lock.ts';
|
||||
import { loadStorageConfig } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
import { sortNewestFirst } from '../core/sort-newest-first.ts';
|
||||
@@ -357,6 +357,83 @@ async function writeChunkerVersion(
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.40 Federated Sync v2: `gbrain sync trigger --source <id> [--priority high|normal|low]`
|
||||
*
|
||||
* Push-trigger entry point. Wraps `queue.add('sync', ...)` with priority -10
|
||||
* (above autopilot's 0) so push-triggered syncs preempt scheduled ones.
|
||||
* Use cases: GitHub webhook handler (POST /webhooks/github), CLI nudge after
|
||||
* a manual git pull, scripted dispatch from `gbrain sources federate`.
|
||||
*
|
||||
* Sets `auto_embed_backfill: true` so the extended sync handler (T6/T7)
|
||||
* auto-enqueues an embed-backfill job after the sync settles.
|
||||
*
|
||||
* Output: prints `job_id=N` to stdout for shell composition. Errors exit 1.
|
||||
*/
|
||||
export async function runSyncTrigger(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log(`Usage: gbrain sync trigger --source <id> [--priority high|normal|low]
|
||||
|
||||
Queue a push-triggered sync job for one source. Prints the resulting job id
|
||||
on stdout. The autopilot worker picks it up and runs performSync against the
|
||||
named source; if the sync added/modified pages, an embed-backfill job is
|
||||
auto-enqueued (subject to D6 budget cap + D19 source-level cooldown).
|
||||
|
||||
Use cases:
|
||||
- GitHub webhook → 'gbrain sync trigger --source <repo>'
|
||||
- Manual nudge after 'git pull' inside a federated source
|
||||
- Programmatic triggers from CI / shell automation
|
||||
|
||||
See also:
|
||||
gbrain sources webhook set <id> Set up GitHub-signed push webhook
|
||||
gbrain sources status Per-source sync + embed coverage
|
||||
`);
|
||||
return;
|
||||
}
|
||||
|
||||
const sourceIdArg = args.find((a, i) => args[i - 1] === '--source') ?? null;
|
||||
if (!sourceIdArg) {
|
||||
console.error('Error: --source <id> is required');
|
||||
console.error("Usage: gbrain sync trigger --source <id> [--priority high|normal|low]");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const priorityArg = args.find((a, i) => args[i - 1] === '--priority') ?? 'high';
|
||||
const priorityMap: Record<string, number> = { high: -10, normal: 0, low: 5 };
|
||||
const priority = priorityMap[priorityArg];
|
||||
if (priority === undefined) {
|
||||
console.error(`Invalid --priority value: "${priorityArg}". Must be high|normal|low.`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
// Verify source exists before submitting
|
||||
const { fetchSource } = await import('../core/sources-load.ts');
|
||||
const source = await fetchSource(engine, sourceIdArg);
|
||||
if (!source) {
|
||||
console.error(`Source "${sourceIdArg}" not found. List with: gbrain sources list`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { MinionQueue } = await import('../core/minions/queue.ts');
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add(
|
||||
'sync',
|
||||
{
|
||||
sourceId: sourceIdArg,
|
||||
repoPath: source.local_path,
|
||||
auto_embed_backfill: true,
|
||||
embed_reason: 'sync_trigger',
|
||||
},
|
||||
{
|
||||
priority,
|
||||
idempotency_key: `sync-trigger:${sourceIdArg}:${Math.floor(Date.now() / 30_000)}`,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
|
||||
console.log(`job_id=${job.id}`);
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// CODEX-2 (v0.22.13): cross-process writer lock for performSync. Two
|
||||
// concurrent syncs can otherwise read the same last_commit anchor, both
|
||||
@@ -372,10 +449,13 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// mechanism (none in v0.22.13; reserved for future).
|
||||
let lockHandle: { release: () => Promise<void> } | null = null;
|
||||
if (!opts.skipLock) {
|
||||
lockHandle = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
|
||||
// v0.40: per-source lock so cross-source sync runs in parallel. Two sources
|
||||
// (e.g. default + zion-brain) take distinct lock rows and don't serialize.
|
||||
const lockKey = syncLockId(opts.sourceId ?? 'default');
|
||||
lockHandle = await tryAcquireDbLock(engine, lockKey);
|
||||
if (!lockHandle) {
|
||||
throw new Error(
|
||||
`Another sync is in progress (lock ${SYNC_LOCK_ID} held). ` +
|
||||
`Another sync is in progress (lock ${lockKey} held). ` +
|
||||
`Wait for it to finish, or run 'gbrain doctor' if it has been more than 30 minutes.`,
|
||||
);
|
||||
}
|
||||
@@ -979,7 +1059,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const factsSourceId = opts.sourceId ?? 'default';
|
||||
for (const slug of pagesAffected) {
|
||||
try {
|
||||
const page = await engine.getPage(slug);
|
||||
// v0.40 D21: source-scoped getPage. Pre-v0.40 this called
|
||||
// engine.getPage(slug) WITHOUT sourceId, then wrote facts under
|
||||
// factsSourceId. On a federated brain with the same slug in two
|
||||
// sources (e.g. people/garry-tan in default + zion-brain), this
|
||||
// would attribute facts to the wrong source. Codex outside-voice
|
||||
// catch on the v0.40 plan review.
|
||||
const page = await engine.getPage(slug, { sourceId: factsSourceId });
|
||||
if (!page) continue;
|
||||
await runFactsBackstop(
|
||||
{
|
||||
@@ -1190,6 +1276,14 @@ async function performFullSync(
|
||||
}
|
||||
|
||||
export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
// v0.40 Federated Sync v2: `gbrain sync trigger` subcommand
|
||||
// Routes to runSyncTrigger which queues a 'sync' minion job with
|
||||
// auto_embed_backfill=true. Falls through to the normal sync path
|
||||
// if 'trigger' isn't the first arg.
|
||||
if (args[0] === 'trigger') {
|
||||
return runSyncTrigger(engine, args.slice(1));
|
||||
}
|
||||
|
||||
// v0.37 fix wave (Lane D.4 + CDX2-12): print usage when `--help`/`-h` is
|
||||
// passed. Pre-fix this was unreachable because the dispatcher's generic
|
||||
// CLI-only short-circuit fired first; sync is now in CLI_ONLY_SELF_HELP.
|
||||
@@ -1242,6 +1336,17 @@ See also:
|
||||
const syncAll = args.includes('--all');
|
||||
const jsonOut = args.includes('--json');
|
||||
const yesFlag = args.includes('--yes');
|
||||
// v0.40 D4+D18: parallel `sync --all` by default; --serial opts back to v1.
|
||||
// --no-auto-embed skips the per-source embed-backfill auto-enqueue.
|
||||
// --max-sources N caps fan-out (default min(sources.length, 8)).
|
||||
const serialFlag = args.includes('--serial');
|
||||
const noAutoEmbed = args.includes('--no-auto-embed');
|
||||
const maxSourcesStr = args.find((a, i) => args[i - 1] === '--max-sources');
|
||||
const maxSources = maxSourcesStr ? parseInt(maxSourcesStr, 10) : undefined;
|
||||
if (maxSourcesStr && (!Number.isFinite(maxSources!) || maxSources! < 1)) {
|
||||
console.error(`Invalid --max-sources value: "${maxSourcesStr}". Must be a positive integer.`);
|
||||
process.exit(1);
|
||||
}
|
||||
const strategyArg = args.find((a, i) => args[i - 1] === '--strategy') as SyncOpts['strategy'] | undefined;
|
||||
const concurrencyStr = args.find((a, i) => args[i - 1] === '--concurrency' || args[i - 1] === '--workers');
|
||||
// v0.22.13 (PR #490 Q2): parseWorkers throws on '0', '-3', 'foo', '1.5' instead
|
||||
@@ -1350,32 +1455,95 @@ See also:
|
||||
}
|
||||
}
|
||||
|
||||
for (const src of sources) {
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) {
|
||||
console.log(`Skipping disabled source: ${src.name}`);
|
||||
continue;
|
||||
}
|
||||
console.log(`\n--- Syncing source: ${src.name} ---`);
|
||||
// v0.40 D4+D18: parallel fan-out is the default; --serial opts out, and
|
||||
// PGLite forces serial (single-writer constraint). Each source runs in
|
||||
// its own performSync — per-source locks (D1) keep them isolated. After
|
||||
// each completion, auto-submit an embed-backfill job (D18) so vector
|
||||
// coverage catches up async instead of leaving search degraded.
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
const activeSources = sources.filter((s) => {
|
||||
const cfg = (s.config || {}) as { syncEnabled?: boolean };
|
||||
return cfg.syncEnabled !== false;
|
||||
});
|
||||
const disabledCount = sources.length - activeSources.length;
|
||||
if (disabledCount > 0) {
|
||||
console.log(`Skipping ${disabledCount} disabled source(s).`);
|
||||
}
|
||||
|
||||
const runOne = async (src: typeof sources[number]): Promise<SyncResult> => {
|
||||
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
|
||||
// D18: parallel path defers embed; auto-enqueue embed-backfill after.
|
||||
const effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed;
|
||||
const repoOpts: SyncOpts = {
|
||||
repoPath: src.local_path!,
|
||||
dryRun, full, noPull, noEmbed, skipFailed, retryFailed,
|
||||
dryRun, full, noPull,
|
||||
noEmbed: effectiveNoEmbed,
|
||||
skipFailed, retryFailed,
|
||||
sourceId: src.id,
|
||||
strategy: cfg.strategy,
|
||||
concurrency,
|
||||
};
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
printSyncResult(result);
|
||||
// Codex P2: --all loop must also manage .gitignore per-source. Without
|
||||
// this, multi-source users who rely on `gbrain sync --all` never get
|
||||
// the advertised db_only ignore rules unless they sync each repo
|
||||
// individually.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
const result = await performSync(engine, repoOpts);
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
}
|
||||
// D18: auto-enqueue embed-backfill per source (unless opted out)
|
||||
if (
|
||||
v2Enabled &&
|
||||
!noAutoEmbed &&
|
||||
!dryRun &&
|
||||
result.status !== 'dry_run' &&
|
||||
result.status !== 'up_to_date'
|
||||
) {
|
||||
try {
|
||||
const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts');
|
||||
const sub = await submitEmbedBackfill(engine, src.id, { reason: 'sync_all' });
|
||||
if (sub.status === 'submitted') {
|
||||
console.log(` → embed-backfill job ${sub.jobId} queued for ${src.name}`);
|
||||
} else if (sub.status === 'cooldown') {
|
||||
console.log(` → embed-backfill skipped (cooldown) for ${src.name}`);
|
||||
} else if (sub.status === 'spend_capped') {
|
||||
console.log(` → embed-backfill skipped (24h spend cap $${sub.spendCapUsd}) for ${src.name}`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error(` → embed-backfill submission failed for ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
};
|
||||
|
||||
const parallelEligible =
|
||||
v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1;
|
||||
|
||||
if (parallelEligible) {
|
||||
const { pMapAllSettled } = await import('../core/parallel.ts');
|
||||
const cap = Math.min(activeSources.length, maxSources ?? 8);
|
||||
console.log(`\nParallel sync: ${activeSources.length} sources, ${cap} concurrent workers.\n`);
|
||||
const results = await pMapAllSettled(activeSources, cap, async (src) => {
|
||||
const r = await runOne(src);
|
||||
return { name: src.name, result: r };
|
||||
});
|
||||
// Print per-source aggregate at the end.
|
||||
console.log('\n--- sync --all aggregate ---');
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i];
|
||||
const name = activeSources[i].name;
|
||||
if (r.status === 'fulfilled') {
|
||||
console.log(` ✓ ${name}: ${r.value.result.status} (added=${r.value.result.added}, modified=${r.value.result.modified}, deleted=${r.value.result.deleted})`);
|
||||
} else {
|
||||
console.error(` ✗ ${name}: ${r.reason instanceof Error ? r.reason.message : String(r.reason)}`);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for (const src of activeSources) {
|
||||
console.log(`\n--- Syncing source: ${src.name} ---`);
|
||||
try {
|
||||
const result = await runOne(src);
|
||||
printSyncResult(result);
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
return;
|
||||
|
||||
@@ -53,7 +53,7 @@ import {
|
||||
type ParsedFact,
|
||||
} from '../facts-fence.ts';
|
||||
import { parseMarkdown, splitBody, serializeMarkdown } from '../markdown.ts';
|
||||
import { tryAcquireDbLock, SYNC_LOCK_ID, type DbLockHandle } from '../db-lock.ts';
|
||||
import { tryAcquireDbLock, syncLockId, type DbLockHandle } from '../db-lock.ts';
|
||||
import { logPhantomEvent, type PhantomOutcome } from '../facts/phantom-audit.ts';
|
||||
|
||||
/** Tagged-union outcome of a single phantom-redirect attempt. */
|
||||
@@ -529,7 +529,9 @@ export async function runPhantomRedirectPass(
|
||||
|
||||
// Bounded-retry lock acquisition. tryAcquireDbLock returns null on
|
||||
// contention; we loop with 1s backoff up to 30s total.
|
||||
const lock = await acquireLockWithRetry(engine, SYNC_LOCK_ID);
|
||||
// v0.40 D16: per-source lock matching performSync's posture. Phantom + same-
|
||||
// source sync still serialize; cross-source parallel sync proceeds unblocked.
|
||||
const lock = await acquireLockWithRetry(engine, syncLockId(sourceId));
|
||||
if (!lock) {
|
||||
logPhantomEvent({ outcome: 'pass_skipped_lock_busy', source_id: sourceId });
|
||||
result.lock_busy = true;
|
||||
|
||||
@@ -134,10 +134,36 @@ export async function tryAcquireDbLock(
|
||||
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
|
||||
}
|
||||
|
||||
/** Lock id for performSync's writer window. Distinct from gbrain-cycle so the
|
||||
* cycle handler can hold gbrain-cycle while performSync (called from inside
|
||||
* the cycle) acquires gbrain-sync. */
|
||||
export const SYNC_LOCK_ID = 'gbrain-sync';
|
||||
/**
|
||||
* v0.40 (Federated Sync v2): per-source sync lock helper.
|
||||
*
|
||||
* Before v0.40: SYNC_LOCK_ID was a bare 'gbrain-sync' constant, taken by
|
||||
* performSync's writer window. That meant only ONE sync could run at a time
|
||||
* across the whole brain — even when two sources are completely independent
|
||||
* (different git repos, different last_commit, different DB row anchors).
|
||||
*
|
||||
* v0.40 namespaces the lock key by sourceId so cross-source sync runs in
|
||||
* parallel. The cycle's broader `gbrain-cycle` lock still serializes inside
|
||||
* a single cycle invocation. Two-source layered semantics:
|
||||
*
|
||||
* cycle acquires `gbrain-cycle`
|
||||
* → performSync(A) acquires `gbrain-sync:A`
|
||||
* → performSync(B) acquires `gbrain-sync:B` (in a different process, fine)
|
||||
*
|
||||
* Audit: `SYNC_LOCK_ID` (back-compat alias) resolves to `syncLockId('default')`.
|
||||
* Every consumer in src/ MUST namespace by source. Tracked consumers:
|
||||
* - src/commands/sync.ts:performSync (per-source)
|
||||
* - src/core/cycle/phantom-redirect.ts (per-source, D16)
|
||||
*/
|
||||
export function syncLockId(sourceId: string): string {
|
||||
return `gbrain-sync:${sourceId}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Back-compat alias. Resolves to `syncLockId('default')`. New code should call
|
||||
* `syncLockId(sourceId)` directly.
|
||||
*/
|
||||
export const SYNC_LOCK_ID = syncLockId('default');
|
||||
|
||||
/**
|
||||
* v0.30.1 (T4 + A4): wrap long-running work in a refreshing TTL lock.
|
||||
|
||||
203
src/core/embed-backfill-submit.ts
Normal file
203
src/core/embed-backfill-submit.ts
Normal file
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* Single submission entry point for the `embed-backfill` minion job
|
||||
* (v0.40 Federated Sync v2 — D19).
|
||||
*
|
||||
* Every caller routes through `submitEmbedBackfill`:
|
||||
* - Parallel sync --all completion (D18)
|
||||
* - Extended `sync` handler (D22, auto_embed_backfill)
|
||||
* - POST /webhooks/github
|
||||
* - `sources federate` / `unfederate` flip hook
|
||||
* - `gbrain sync trigger`
|
||||
* - autopilot per-source dispatch (when source is stale and degraded)
|
||||
*
|
||||
* Why centralize: D2 added a per-source DB lock at handler entry. That
|
||||
* protects against double-RUN but not double-SUBMIT — a webhook storm could
|
||||
* still queue 20 embed-backfill jobs, each one waking, acquiring the lock,
|
||||
* finding it busy, completing as `already_in_progress`. Net effect: zero
|
||||
* wasted Voyage spend (D6 budget cap), but high queue churn. Worse, even if
|
||||
* the lock held, a 30-second idempotency bucket only coalesces SUBMITS within
|
||||
* the same window. Multi-hour push activity racks up unbounded calls.
|
||||
*
|
||||
* D19 layered defenses (composed here):
|
||||
* 1. Per-source cooldown (default 10min). Refuses submission if the most
|
||||
* recent embed-backfill for this source finished or is still active
|
||||
* inside the window.
|
||||
* 2. Per-source 24h rolling spend cap (default $25). Computed from the
|
||||
* embed-backfill-tagged rows in the budget audit JSONL. Refuses
|
||||
* submission when spend has hit the cap.
|
||||
*
|
||||
* Both bounds are config-overridable:
|
||||
* - `embed.backfill_cooldown_min` (default 10)
|
||||
* - `embed.backfill_max_usd_per_source_24h` (default 25)
|
||||
*
|
||||
* Returns a tagged-union status so callers can render the right user signal
|
||||
* (`gbrain sources status`, webhook response body, sync completion banner).
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { MinionQueue } from './minions/queue.ts';
|
||||
|
||||
export const COOLDOWN_CONFIG_KEY = 'embed.backfill_cooldown_min';
|
||||
export const SPEND_CAP_CONFIG_KEY = 'embed.backfill_max_usd_per_source_24h';
|
||||
|
||||
const DEFAULT_COOLDOWN_MIN = 10;
|
||||
const DEFAULT_SPEND_CAP_USD = 25;
|
||||
|
||||
export type SubmitEmbedBackfillStatus =
|
||||
| 'submitted'
|
||||
| 'cooldown'
|
||||
| 'spend_capped';
|
||||
|
||||
export interface SubmitEmbedBackfillResult {
|
||||
status: SubmitEmbedBackfillStatus;
|
||||
/** Set when status === 'submitted'. */
|
||||
jobId?: number;
|
||||
/** Set when status === 'cooldown'. Seconds remaining until cooldown lifts. */
|
||||
cooldownRemainingSeconds?: number;
|
||||
/** Set when status === 'spend_capped'. Dollars spent in the 24h window. */
|
||||
spend24hUsd?: number;
|
||||
/** Set when status === 'spend_capped'. Active cap. */
|
||||
spendCapUsd?: number;
|
||||
}
|
||||
|
||||
export interface SubmitEmbedBackfillOpts {
|
||||
/** Logged into the job's data row for audit. */
|
||||
reason: string;
|
||||
/** Override cooldown lookup (tests). */
|
||||
cooldownMinOverride?: number;
|
||||
/** Override spend-cap lookup (tests). */
|
||||
spendCapUsdOverride?: number;
|
||||
/** Override the 24h spend aggregator (tests). Returns spend in USD. */
|
||||
spend24hFn?: (engine: BrainEngine, sourceId: string) => Promise<number>;
|
||||
/** Override `Date.now` (tests). */
|
||||
nowMs?: number;
|
||||
/** Job priority. Default 5 (lower than autopilot's 0; above default jobs). */
|
||||
priority?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Default 24h-spend aggregator. Reads completed embed-backfill rows in
|
||||
* `minion_jobs` and sums their token usage × known-price proxy. We
|
||||
* deliberately do NOT read the budget-tracker audit JSONL — the file
|
||||
* lives on the worker's local disk and may not be present on the
|
||||
* machine submitting the job (e.g. a remote MCP serve-http process).
|
||||
*
|
||||
* The minion_jobs.tokens_input/output columns ARE populated by the
|
||||
* subagent flow but NOT by the embed flow (gateway.embed doesn't go
|
||||
* through the chat-completion roll-up). For v0.40 we use a job-COUNT
|
||||
* proxy capped at the daily limit: 1 job ≈ default-cap-share. Accepts
|
||||
* imprecision in exchange for cross-process visibility.
|
||||
*
|
||||
* A precise rolling-spend tracker is filed as a v0.41 TODO.
|
||||
*/
|
||||
async function defaultSpend24hForSource(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
): Promise<number> {
|
||||
// Conservative proxy: count jobs that completed (or are running) in the
|
||||
// 24h window. Each is treated as worth `DEFAULT_SPEND_CAP_USD / 25` ($1)
|
||||
// toward the cap — i.e. 25 jobs in 24h saturate the default cap.
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n
|
||||
FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND data->>'sourceId' = $1
|
||||
AND status IN ('active', 'completed')
|
||||
AND created_at > NOW() - INTERVAL '24 hours'`,
|
||||
[sourceId],
|
||||
);
|
||||
const jobCount = rows[0]?.n ?? 0;
|
||||
return jobCount * 1; // $1 / job placeholder; configurable in a later wave.
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up an integer-valued config key with sane defaults.
|
||||
* Returns `def` on missing / NaN / non-positive.
|
||||
*/
|
||||
async function readIntConfig(
|
||||
engine: BrainEngine,
|
||||
key: string,
|
||||
def: number,
|
||||
): Promise<number> {
|
||||
const raw = await engine.getConfig(key);
|
||||
if (raw === null || raw === undefined) return def;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : def;
|
||||
}
|
||||
|
||||
export async function submitEmbedBackfill(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
opts: SubmitEmbedBackfillOpts,
|
||||
): Promise<SubmitEmbedBackfillResult> {
|
||||
const now = opts.nowMs ?? Date.now();
|
||||
const cooldownMin =
|
||||
opts.cooldownMinOverride ??
|
||||
(await readIntConfig(engine, COOLDOWN_CONFIG_KEY, DEFAULT_COOLDOWN_MIN));
|
||||
const spendCap =
|
||||
opts.spendCapUsdOverride ??
|
||||
(await readIntConfig(engine, SPEND_CAP_CONFIG_KEY, DEFAULT_SPEND_CAP_USD));
|
||||
|
||||
// ── Source-level cooldown ─────────────────────────────────────
|
||||
// Block re-submission if (a) an embed-backfill is currently active for this
|
||||
// source, OR (b) the most-recent embed-backfill finished within the
|
||||
// cooldown window.
|
||||
const lastJob = await engine.executeRaw<{
|
||||
finished_at: Date | null;
|
||||
status: string;
|
||||
}>(
|
||||
`SELECT finished_at, status
|
||||
FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND data->>'sourceId' = $1
|
||||
ORDER BY id DESC LIMIT 1`,
|
||||
[sourceId],
|
||||
);
|
||||
|
||||
if (lastJob[0]) {
|
||||
if (lastJob[0].status === 'active' || lastJob[0].status === 'waiting') {
|
||||
// Active or waiting: no cooldown-remaining number (would be misleading).
|
||||
return { status: 'cooldown' };
|
||||
}
|
||||
if (lastJob[0].finished_at) {
|
||||
const finishedMs = new Date(lastJob[0].finished_at).getTime();
|
||||
const ageMs = now - finishedMs;
|
||||
const cooldownMs = cooldownMin * 60 * 1000;
|
||||
if (ageMs < cooldownMs) {
|
||||
return {
|
||||
status: 'cooldown',
|
||||
cooldownRemainingSeconds: Math.ceil((cooldownMs - ageMs) / 1000),
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── 24h rolling spend cap ─────────────────────────────────────
|
||||
const spend24hFn = opts.spend24hFn ?? defaultSpend24hForSource;
|
||||
const spend24h = await spend24hFn(engine, sourceId);
|
||||
if (spend24h >= spendCap) {
|
||||
return {
|
||||
status: 'spend_capped',
|
||||
spend24hUsd: spend24h,
|
||||
spendCapUsd: spendCap,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Submission ────────────────────────────────────────────────
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add(
|
||||
'embed-backfill',
|
||||
{ sourceId, batchSize: 500, reason: opts.reason },
|
||||
{
|
||||
priority: opts.priority ?? 5,
|
||||
idempotency_key: `embed-backfill:${sourceId}:${bucketize(now, 5 * 60_000)}`,
|
||||
maxWaiting: 1,
|
||||
},
|
||||
);
|
||||
|
||||
return { status: 'submitted', jobId: job.id };
|
||||
}
|
||||
|
||||
/** Round timestamp down to the nearest `bucketMs` boundary. */
|
||||
function bucketize(ms: number, bucketMs: number): number {
|
||||
return Math.floor(ms / bucketMs);
|
||||
}
|
||||
211
src/core/embed-stale.ts
Normal file
211
src/core/embed-stale.ts
Normal file
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* Stale-chunk embedding loop, extracted from `src/commands/embed.ts:embedAllStale`
|
||||
* for reuse by the v0.40 `embed-backfill` Minion handler (D15.2 — codex
|
||||
* outside-voice catch).
|
||||
*
|
||||
* Single source of truth for the cursor-paginated, source-grouped, rate-limit-
|
||||
* aware embedding pipeline. Both `gbrain embed --stale` (CLI) and the
|
||||
* `embed-backfill` job (Minion) call this helper so the working machinery
|
||||
* — keyset pagination, batch grouping by `source_id::slug`, merge-with-existing
|
||||
* via `getChunks` + `upsertChunks`, AbortSignal threading into in-flight HTTPs,
|
||||
* 429 backoff — lives in exactly one place.
|
||||
*
|
||||
* Why a separate file (vs. exporting from embed.ts): embed.ts is a CLI command
|
||||
* with logging side-effects and EmbedResult-shaped aggregation. The handler
|
||||
* needs a tighter functional surface (returns embedded count + done state, no
|
||||
* console.log). Extracting kept embed.ts's outer flow intact while letting the
|
||||
* handler call a clean primitive.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { ChunkInput } from './types.ts';
|
||||
import { embedBatchWithBackoff } from '../commands/embed.ts';
|
||||
|
||||
/** Last visited (page_id, chunk_index) for keyset-resume across runs. */
|
||||
export interface StaleCursor {
|
||||
afterPageId: number;
|
||||
afterChunkIndex: number;
|
||||
}
|
||||
|
||||
export interface EmbedStaleOpts {
|
||||
/** Chunks per cursor page. Default 2000 (matches the legacy CLI default). */
|
||||
batchSize?: number;
|
||||
/** Max parallel slug-keys embedded inside a single batch. Default 20. */
|
||||
concurrency?: number;
|
||||
/** Resume cursor from a prior run. Default: from start. */
|
||||
cursor?: StaleCursor;
|
||||
/** AbortSignal honored at three sites: batch claim, retry sleep, HTTP body. */
|
||||
signal?: AbortSignal;
|
||||
/**
|
||||
* Fired once per batch with the cursor after that batch finishes. Caller
|
||||
* uses this for crash-resumable progress (Minion `job.updateProgress`).
|
||||
*/
|
||||
onProgress?: (state: {
|
||||
embedded: number;
|
||||
chunksProcessed: number;
|
||||
cursor: StaleCursor;
|
||||
}) => void;
|
||||
/**
|
||||
* Optional caller-supplied embed fn. Defaults to `embedBatchWithBackoff`.
|
||||
* Test seam: lets unit tests inject a deterministic fake without mocking
|
||||
* the gateway. Production callers leave it unset.
|
||||
*/
|
||||
embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise<Float32Array[]>;
|
||||
}
|
||||
|
||||
export interface EmbedStaleResult {
|
||||
/** Chunks newly embedded in this call. */
|
||||
embedded: number;
|
||||
/** Total chunks pulled across all batches (including ones that errored). */
|
||||
chunksProcessed: number;
|
||||
/** Pages whose embeddings landed. */
|
||||
pagesProcessed: number;
|
||||
/** Last cursor reached. null iff zero stale chunks existed at start. */
|
||||
lastCursor: StaleCursor | null;
|
||||
/** True iff the loop exited because every stale chunk was processed. */
|
||||
done: boolean;
|
||||
/** True iff the loop exited because `signal.aborted` fired. */
|
||||
aborted: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Embed every stale chunk (embedding IS NULL) for a source.
|
||||
*
|
||||
* Re-entrancy contract: if interrupted, the next call resumes from the next
|
||||
* stale row. Idempotent — `embedding IS NULL` predicate naturally excludes
|
||||
* already-embedded chunks even without cursor persistence. The cursor is a
|
||||
* progress optimization, not a correctness mechanism.
|
||||
*
|
||||
* Returns when:
|
||||
* - every stale chunk has been embedded (`done: true`), OR
|
||||
* - `signal.aborted` fired (`aborted: true`), OR
|
||||
* - the per-batch `embedFn` threw with the signal aborted (treated as abort).
|
||||
*
|
||||
* Per-page embed failures (network blip, dim mismatch) do NOT throw — the
|
||||
* embedding stays NULL and the next call retries the chunk. This matches the
|
||||
* existing CLI's "log + skip" semantics so a single bad page doesn't poison
|
||||
* the run. Caller is responsible for surfacing partial-success via the
|
||||
* returned `embedded` vs `chunksProcessed` delta.
|
||||
*/
|
||||
export async function embedStaleForSource(
|
||||
engine: BrainEngine,
|
||||
sourceId: string,
|
||||
opts: EmbedStaleOpts = {},
|
||||
): Promise<EmbedStaleResult> {
|
||||
const batchSize = opts.batchSize ?? 2000;
|
||||
const concurrency = opts.concurrency ?? 20;
|
||||
const signal = opts.signal;
|
||||
const embedFn = opts.embedFn ?? ((texts, fnOpts) =>
|
||||
embedBatchWithBackoff(texts, { abortSignal: fnOpts.abortSignal }));
|
||||
|
||||
let afterPageId = opts.cursor?.afterPageId ?? 0;
|
||||
let afterChunkIndex = opts.cursor?.afterChunkIndex ?? -1;
|
||||
|
||||
const result: EmbedStaleResult = {
|
||||
embedded: 0,
|
||||
chunksProcessed: 0,
|
||||
pagesProcessed: 0,
|
||||
lastCursor: null,
|
||||
done: false,
|
||||
aborted: false,
|
||||
};
|
||||
|
||||
for (;;) {
|
||||
if (signal?.aborted) {
|
||||
result.aborted = true;
|
||||
return result;
|
||||
}
|
||||
|
||||
const batch = await engine.listStaleChunks({
|
||||
batchSize,
|
||||
afterPageId,
|
||||
afterChunkIndex,
|
||||
sourceId,
|
||||
});
|
||||
if (batch.length === 0) {
|
||||
result.done = true;
|
||||
return result;
|
||||
}
|
||||
|
||||
result.chunksProcessed += batch.length;
|
||||
const last = batch[batch.length - 1];
|
||||
afterPageId = last.page_id;
|
||||
afterChunkIndex = last.chunk_index;
|
||||
result.lastCursor = { afterPageId, afterChunkIndex };
|
||||
|
||||
// Group by composite key (source_id::slug). Within a source-scoped run
|
||||
// every row carries the same source_id, but the helper accepts batches
|
||||
// shaped by `listStaleChunks` which carry source_id per row for parity
|
||||
// with the cross-source CLI path.
|
||||
const byKey = new Map<string, typeof batch>();
|
||||
for (const row of batch) {
|
||||
const key = `${row.source_id}::${row.slug}`;
|
||||
const list = byKey.get(key);
|
||||
if (list) list.push(row);
|
||||
else byKey.set(key, [row]);
|
||||
}
|
||||
|
||||
const keys = Array.from(byKey.keys());
|
||||
let nextIdx = 0;
|
||||
|
||||
async function embedOneKey(key: string): Promise<void> {
|
||||
const stale = byKey.get(key)!;
|
||||
const keySourceId = stale[0]?.source_id ?? sourceId;
|
||||
const slug = stale[0].slug;
|
||||
try {
|
||||
const embeddings = await embedFn(
|
||||
stale.map((c) => c.chunk_text),
|
||||
{ abortSignal: signal },
|
||||
);
|
||||
const existing = await engine.getChunks(slug, { sourceId: keySourceId });
|
||||
const staleIdxToEmbedding = new Map<number, Float32Array>();
|
||||
for (let j = 0; j < stale.length; j++) {
|
||||
staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]);
|
||||
}
|
||||
const merged: ChunkInput[] = existing.map((c) => ({
|
||||
chunk_index: c.chunk_index,
|
||||
chunk_text: c.chunk_text,
|
||||
chunk_source: c.chunk_source,
|
||||
embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined,
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
|
||||
result.embedded += stale.length;
|
||||
result.pagesProcessed += 1;
|
||||
} catch (e: unknown) {
|
||||
// Aborted mid-fetch is expected; treat as graceful exit.
|
||||
if (signal?.aborted) return;
|
||||
// Otherwise log and skip — the chunk stays NULL and next call retries.
|
||||
process.stderr.write(
|
||||
`\n [embed-stale] error on ${keySourceId}/${slug}: ${
|
||||
e instanceof Error ? e.message : String(e)
|
||||
}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function worker(): Promise<void> {
|
||||
while (nextIdx < keys.length && !signal?.aborted) {
|
||||
const idx = nextIdx++;
|
||||
await embedOneKey(keys[idx]);
|
||||
}
|
||||
}
|
||||
|
||||
const numWorkers = Math.min(concurrency, keys.length);
|
||||
await Promise.all(Array.from({ length: numWorkers }, () => worker()));
|
||||
|
||||
if (opts.onProgress) {
|
||||
opts.onProgress({
|
||||
embedded: result.embedded,
|
||||
chunksProcessed: result.chunksProcessed,
|
||||
cursor: { afterPageId, afterChunkIndex },
|
||||
});
|
||||
}
|
||||
|
||||
// Short batch = end of stale set; advance and exit.
|
||||
if (batch.length < batchSize) {
|
||||
result.done = true;
|
||||
return result;
|
||||
}
|
||||
}
|
||||
}
|
||||
49
src/core/feature-flags.ts
Normal file
49
src/core/feature-flags.ts
Normal file
@@ -0,0 +1,49 @@
|
||||
/**
|
||||
* Feature flags (v0.40 D23).
|
||||
*
|
||||
* Single shared escape hatch for the v0.40 Federated Sync v2 cathedral.
|
||||
* Flipping `sync.federated_v2` to `false` reverts to v0.39 sequential
|
||||
* behavior without re-installing the binary — useful if a foundational
|
||||
* assumption proves wrong in production (e.g. parallel sync trips an
|
||||
* undiscovered Postgres lock contention).
|
||||
*
|
||||
* What the flag gates:
|
||||
* - Parallel branch of `gbrain sync --all` (serial fallback otherwise)
|
||||
* - Auto-enqueue of embed-backfill in the extended `sync` handler
|
||||
* - Autopilot's per-source freshness-gate dispatch (D17)
|
||||
*
|
||||
* What stays on UNCONDITIONALLY (correctness, not features):
|
||||
* - Per-source sync lock (`syncLockId`)
|
||||
* - Phantom-redirect per-source lock (D16)
|
||||
* - Migration v87 (sources_github_repo index)
|
||||
* - Facts-backstop source-scoping fix (D21)
|
||||
* - safeHexEqual extraction (D15.5)
|
||||
*
|
||||
* Disable path:
|
||||
* gbrain config set sync.federated_v2 false
|
||||
* gbrain jobs supervisor restart # autopilot picks up the change
|
||||
*
|
||||
* Convention: this module is the ONLY place that reads the flag. Callers go
|
||||
* through `isFederatedV2Enabled(engine)` so future changes to the flag key,
|
||||
* default, or backing store happen in one place.
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export const FEDERATED_V2_CONFIG_KEY = 'sync.federated_v2';
|
||||
|
||||
/**
|
||||
* True iff Federated Sync v2 behaviors are enabled (default true).
|
||||
*
|
||||
* Reads `sync.federated_v2` from the DB config plane via `engine.getConfig`.
|
||||
* Values: `'false'` → disabled; anything else (including missing/null) →
|
||||
* enabled. The default-on posture is deliberate — v0.40 ships expecting the
|
||||
* new behavior, and ops opt out by setting the key explicitly.
|
||||
*
|
||||
* Throwing on engine errors is fine: the flag is only checked at boundary
|
||||
* points (CLI dispatch, autopilot tick, sync handler) where an engine error
|
||||
* would surface anyway. Callers don't need a try/catch wrapper.
|
||||
*/
|
||||
export async function isFederatedV2Enabled(engine: BrainEngine): Promise<boolean> {
|
||||
const value = await engine.getConfig(FEDERATED_V2_CONFIG_KEY);
|
||||
return value !== 'false';
|
||||
}
|
||||
@@ -4232,6 +4232,31 @@ export const MIGRATIONS: Migration[] = [
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 92,
|
||||
name: 'sources_github_repo_index',
|
||||
// v0.40.5.0 Federated Sync v2 (D13): partial expression index on
|
||||
// sources.config->>'github_repo' so the new POST /webhooks/github
|
||||
// handler's source-by-repo lookup uses an index instead of a sequential
|
||||
// scan. Sources is small today (<100 rows in practice) so the impact is
|
||||
// microseconds, but the lookup fires on every webhook event (including
|
||||
// ignored ones) and a team with hundreds of sources would feel it.
|
||||
//
|
||||
// Partial WHERE clause keeps the index small — only rows with a
|
||||
// configured webhook actually take up index entries. Both Postgres and
|
||||
// PGLite support partial expression indexes; no engine-specific shape.
|
||||
// Idempotent (IF NOT EXISTS).
|
||||
//
|
||||
// Plan called this v81 originally; renumbered through v87 → v89 → v90 → v92
|
||||
// across successive master merges (v0.40.2.0 claimed v89 for
|
||||
// facts_event_type_column; v0.40.3.0 claimed v90 + v91 for
|
||||
// contextual_retrieval_columns + pages_generation_trigger_and_bookmark).
|
||||
sql: `
|
||||
CREATE INDEX IF NOT EXISTS sources_github_repo_idx
|
||||
ON sources ((config->>'github_repo'))
|
||||
WHERE config ? 'github_repo';
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
182
src/core/minions/handlers/embed-backfill.ts
Normal file
182
src/core/minions/handlers/embed-backfill.ts
Normal file
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* `embed-backfill` minion handler (v0.40 Federated Sync v2).
|
||||
*
|
||||
* Decouples embedding from the sync pipeline so:
|
||||
* - `gbrain sync --all` finishes fast (pages searchable via keyword
|
||||
* immediately), embed-backfill catches up async (D18).
|
||||
* - Webhook-driven syncs don't block on Voyage rate limits (D5 + D18).
|
||||
* - Fresh-source onboarding (federate a 50K-page repo) doesn't make
|
||||
* the user wait for ~$3-$10 of embedding before the sync "completes."
|
||||
*
|
||||
* The handler is the run-side companion to `submitEmbedBackfill` in
|
||||
* `src/core/embed-backfill-submit.ts`. The submit-side gate (D19) handles
|
||||
* cross-call rate-limiting (10min cooldown + 24h $25 rolling cap); this
|
||||
* handler handles within-run safety:
|
||||
*
|
||||
* - D2: per-source DB lock (`gbrain-embed-backfill:<source>`) prevents
|
||||
* two embed-backfill jobs for the same source from running concurrently.
|
||||
* If a second job claims while the first is mid-loop, it returns
|
||||
* `already_in_progress` cleanly and the lock is the source of truth.
|
||||
*
|
||||
* - D6: BudgetTracker enforces per-job spend cap (default $10/job). Goes
|
||||
* through `withBudgetTracker` so `gateway.embed()` auto-composes via
|
||||
* AsyncLocalStorage. On `BudgetExhausted` throw, partial progress is
|
||||
* preserved (chunks already embedded stay embedded; remaining stays NULL).
|
||||
*
|
||||
* - D15.1: parent-job linkage is INTENTIONALLY OMITTED. The submit-side
|
||||
* helper does not pass `parent_job_id` — the queue's parent-child
|
||||
* semantics flip the parent into `waiting-children` and fail completion.
|
||||
* Sync handlers are short-lived; embed-backfill outlives them by design.
|
||||
*
|
||||
* - try/finally ALWAYS releases the per-source lock. Aborted runs leave
|
||||
* the next call free to claim.
|
||||
*/
|
||||
import { tryAcquireDbLock } from '../../db-lock.ts';
|
||||
import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts';
|
||||
import { withBudgetTracker } from '../../ai/gateway.ts';
|
||||
import { embedStaleForSource } from '../../embed-stale.ts';
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { MinionJobContext } from '../types.ts';
|
||||
|
||||
const DEFAULT_MAX_USD_PER_JOB = 10;
|
||||
const EMBED_BACKFILL_LOCK_TTL_MIN = 60;
|
||||
|
||||
export interface EmbedBackfillJobData {
|
||||
sourceId: string;
|
||||
batchSize?: number;
|
||||
/** Audit string from the submitter (e.g. 'webhook', 'federation_flip'). */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface EmbedBackfillResult {
|
||||
status: 'success' | 'already_in_progress' | 'budget_exhausted' | 'aborted';
|
||||
sourceId: string;
|
||||
embedded: number;
|
||||
chunksProcessed: number;
|
||||
pagesProcessed: number;
|
||||
/** $USD spent inside this job (from BudgetTracker.totalSpent). */
|
||||
spentUsd: number;
|
||||
/** Set when status === 'budget_exhausted'. */
|
||||
budgetCapUsd?: number;
|
||||
}
|
||||
|
||||
/** Compose the lock id for embed-backfill, namespaced like sync's. */
|
||||
function embedBackfillLockId(sourceId: string): string {
|
||||
return `gbrain-embed-backfill:${sourceId}`;
|
||||
}
|
||||
|
||||
/** Read embed.backfill_max_usd config or default. */
|
||||
async function readMaxUsd(engine: BrainEngine): Promise<number> {
|
||||
const raw = await engine.getConfig('embed.backfill_max_usd');
|
||||
if (raw === null || raw === undefined) return DEFAULT_MAX_USD_PER_JOB;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : DEFAULT_MAX_USD_PER_JOB;
|
||||
}
|
||||
|
||||
/** Validate + extract typed job params. Throws on malformed input. */
|
||||
function parseParams(data: Record<string, unknown>): EmbedBackfillJobData {
|
||||
const sourceId = data.sourceId;
|
||||
if (typeof sourceId !== 'string' || sourceId.length === 0) {
|
||||
throw new Error('embed-backfill: data.sourceId is required and must be a non-empty string');
|
||||
}
|
||||
const batchSize =
|
||||
typeof data.batchSize === 'number' && data.batchSize > 0
|
||||
? data.batchSize
|
||||
: undefined;
|
||||
const reason =
|
||||
typeof data.reason === 'string' ? data.reason : undefined;
|
||||
return { sourceId, batchSize, reason };
|
||||
}
|
||||
|
||||
export function makeEmbedBackfillHandler(engine: BrainEngine) {
|
||||
return async function embedBackfillHandler(
|
||||
job: MinionJobContext,
|
||||
): Promise<EmbedBackfillResult> {
|
||||
const { sourceId, batchSize } = parseParams(job.data);
|
||||
|
||||
// D2: per-source lock at handler entry. The submit-side cooldown (D19)
|
||||
// prevents most contention but this is the run-side safety net.
|
||||
const lockKey = embedBackfillLockId(sourceId);
|
||||
const lock = await tryAcquireDbLock(engine, lockKey, EMBED_BACKFILL_LOCK_TTL_MIN);
|
||||
if (!lock) {
|
||||
return {
|
||||
status: 'already_in_progress',
|
||||
sourceId,
|
||||
embedded: 0,
|
||||
chunksProcessed: 0,
|
||||
pagesProcessed: 0,
|
||||
spentUsd: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// D6: budget-tracked execution. Gateway calls inside withBudgetTracker
|
||||
// auto-compose via AsyncLocalStorage; if pricing pushes cumulative spend
|
||||
// past the cap, gateway throws BudgetExhausted BEFORE the next API call.
|
||||
const capUsd = await readMaxUsd(engine);
|
||||
const tracker = new BudgetTracker({
|
||||
maxCostUsd: capUsd,
|
||||
label: `embed-backfill:${sourceId}`,
|
||||
});
|
||||
|
||||
try {
|
||||
const result = await withBudgetTracker(tracker, async () =>
|
||||
embedStaleForSource(engine, sourceId, {
|
||||
batchSize,
|
||||
signal: job.signal,
|
||||
onProgress: ({ embedded, chunksProcessed, cursor }) => {
|
||||
// Fire-and-forget; updateProgress returns a Promise but the
|
||||
// handler is sync inside the loop.
|
||||
void job.updateProgress({
|
||||
embedded,
|
||||
chunksProcessed,
|
||||
cursor,
|
||||
spentUsd: tracker.totalSpent,
|
||||
});
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
if (result.aborted) {
|
||||
return {
|
||||
status: 'aborted',
|
||||
sourceId,
|
||||
embedded: result.embedded,
|
||||
chunksProcessed: result.chunksProcessed,
|
||||
pagesProcessed: result.pagesProcessed,
|
||||
spentUsd: tracker.totalSpent,
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: 'success',
|
||||
sourceId,
|
||||
embedded: result.embedded,
|
||||
chunksProcessed: result.chunksProcessed,
|
||||
pagesProcessed: result.pagesProcessed,
|
||||
spentUsd: tracker.totalSpent,
|
||||
};
|
||||
} catch (err) {
|
||||
if (err instanceof BudgetExhausted) {
|
||||
// Partial progress preserved: already-embedded chunks stay embedded;
|
||||
// remaining stays NULL for the next run to pick up.
|
||||
return {
|
||||
status: 'budget_exhausted',
|
||||
sourceId,
|
||||
embedded: 0, // Tracker doesn't track per-chunk count
|
||||
chunksProcessed: 0,
|
||||
pagesProcessed: 0,
|
||||
spentUsd: tracker.totalSpent,
|
||||
budgetCapUsd: capUsd,
|
||||
};
|
||||
}
|
||||
throw err;
|
||||
} finally {
|
||||
// ALWAYS release. Aborts, throws, budget-exhaust — all paths unwind here.
|
||||
try {
|
||||
await lock.release();
|
||||
} catch {
|
||||
// Lock release best-effort; TTL fallback covers the case where the
|
||||
// row was already cleared by a parallel writer.
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
59
src/core/parallel.ts
Normal file
59
src/core/parallel.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* Bounded-concurrency Promise.allSettled.
|
||||
*
|
||||
* Runs `fn(item)` for each item with at most `concurrency` in flight at a time.
|
||||
* Returns a `PromiseSettledResult` per input item, in input order, so callers
|
||||
* can distinguish fulfilled-with-result from rejected-with-error per item.
|
||||
*
|
||||
* Used by `gbrain sync --all` (v0.40 Federated Sync v2) to fan out per-source
|
||||
* syncs without overwhelming the embedding API or local disk.
|
||||
*
|
||||
* Why a semaphore + `Promise.allSettled` instead of a library:
|
||||
* - Zero new deps. Bun's stdlib has every primitive we need.
|
||||
* - `Promise.allSettled` semantics are what we want: one source failing
|
||||
* must NOT short-circuit the others. Bare `Promise.all` rejects on
|
||||
* first failure.
|
||||
* - The concurrency cap is a hard ceiling, not a target. With N items and
|
||||
* concurrency C, the function makes at most C concurrent `fn` calls at
|
||||
* any moment.
|
||||
*
|
||||
* Order guarantee: results[i] always corresponds to items[i]. The execution
|
||||
* order of `fn` calls is NOT guaranteed (a fast item submitted later can
|
||||
* complete before a slow item submitted earlier).
|
||||
*/
|
||||
|
||||
export async function pMapAllSettled<T, R>(
|
||||
items: readonly T[],
|
||||
concurrency: number,
|
||||
fn: (item: T, index: number) => Promise<R>,
|
||||
): Promise<Array<PromiseSettledResult<R>>> {
|
||||
if (!Number.isInteger(concurrency) || concurrency < 1) {
|
||||
throw new TypeError(
|
||||
`pMapAllSettled: concurrency must be a positive integer, got ${concurrency}`,
|
||||
);
|
||||
}
|
||||
if (items.length === 0) return [];
|
||||
|
||||
const results: Array<PromiseSettledResult<R>> = new Array(items.length);
|
||||
const effectiveConcurrency = Math.min(concurrency, items.length);
|
||||
|
||||
let nextIndex = 0;
|
||||
|
||||
// Each worker pulls the next available index and runs fn until exhausted.
|
||||
async function worker(): Promise<void> {
|
||||
for (;;) {
|
||||
const i = nextIndex++;
|
||||
if (i >= items.length) return;
|
||||
try {
|
||||
const value = await fn(items[i], i);
|
||||
results[i] = { status: 'fulfilled', value };
|
||||
} catch (err) {
|
||||
results[i] = { status: 'rejected', reason: err };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const workers = Array.from({ length: effectiveConcurrency }, () => worker());
|
||||
await Promise.all(workers);
|
||||
return results;
|
||||
}
|
||||
@@ -56,6 +56,12 @@ INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- v0.40 Federated Sync v2: partial expression index on config->>'github_repo'
|
||||
-- (mirror of src/schema.sql; migration v92 backfills legacy brains).
|
||||
CREATE INDEX IF NOT EXISTS sources_github_repo_idx
|
||||
ON sources ((config->>'github_repo'))
|
||||
WHERE config ? 'github_repo';
|
||||
|
||||
-- ============================================================
|
||||
-- pages: the core content table
|
||||
-- ============================================================
|
||||
|
||||
47
src/core/source-config-redact.ts
Normal file
47
src/core/source-config-redact.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* Redact secrets from sources.config before serializing (v0.40 D15.4).
|
||||
*
|
||||
* Codex outside-voice catch on the v0.40 plan: `webhook show` hides the
|
||||
* secret, but any OTHER surface that returns raw `sources.config` (the
|
||||
* sources list --json, an MCP get_source op, an HTTP admin API) would
|
||||
* leak it. This module is the single redaction primitive every serializer
|
||||
* routes through.
|
||||
*
|
||||
* `scripts/check-source-config-leak.sh` grep-guards against future drift.
|
||||
*/
|
||||
|
||||
const SECRET_KEYS: ReadonlySet<string> = new Set([
|
||||
'webhook_secret',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Return a shallow copy of `config` with every key in SECRET_KEYS replaced
|
||||
* with the string '<redacted>'. Original input is not mutated.
|
||||
*
|
||||
* Non-object input returns an empty object (defensive — sources.config is
|
||||
* always a JSON object, but driver layer may briefly return a string mid-
|
||||
* parse on legacy brains).
|
||||
*/
|
||||
export function redactSourceConfig(
|
||||
config: unknown,
|
||||
): Record<string, unknown> {
|
||||
if (config === null || typeof config !== 'object' || Array.isArray(config)) {
|
||||
return {};
|
||||
}
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const [k, v] of Object.entries(config as Record<string, unknown>)) {
|
||||
if (SECRET_KEYS.has(k)) {
|
||||
out[k] = '<redacted>';
|
||||
} else {
|
||||
out[k] = v;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** True iff the config object carries a webhook secret. */
|
||||
export function hasWebhookSecret(config: unknown): boolean {
|
||||
if (!config || typeof config !== 'object') return false;
|
||||
const c = config as Record<string, unknown>;
|
||||
return typeof c.webhook_secret === 'string' && c.webhook_secret.length > 0;
|
||||
}
|
||||
212
src/core/source-health.ts
Normal file
212
src/core/source-health.ts
Normal file
@@ -0,0 +1,212 @@
|
||||
/**
|
||||
* Per-source health metrics (v0.40 D12 + D9 + D17 + D19).
|
||||
*
|
||||
* Single source of truth for `gbrain sources status` AND `gbrain doctor`'s
|
||||
* `federation_health` check. Sharing the implementation prevents the dashboard
|
||||
* and the doctor warning from drifting.
|
||||
*
|
||||
* D12: batched GROUP BY queries — 4 queries total instead of 6×N per-source
|
||||
* round-trips. On a 4-source / 300K-chunk brain this drops dashboard
|
||||
* time from ~24s to <2s.
|
||||
*
|
||||
* D9: resolvePriority(config) — accepts 'high'|'normal'|'low', falls back
|
||||
* to 0 with once-per-source-per-process stderr warn on unknown values.
|
||||
*
|
||||
* D17: isSourceStale helper — autopilot calls this to decide per-source
|
||||
* sync dispatch independent of the brain_score gate.
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { parseSourceConfig, type SourceRow } from './sources-load.ts';
|
||||
|
||||
export interface SourceMetrics {
|
||||
source_id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
federated: boolean;
|
||||
total_pages: number;
|
||||
total_chunks: number;
|
||||
embedded_chunks: number;
|
||||
embed_coverage_pct: number;
|
||||
last_sync_at: Date | null;
|
||||
lag_seconds: number | null;
|
||||
/** Failed jobs (sync OR embed-backfill) for this source in last 24h. */
|
||||
failed_jobs_24h: number;
|
||||
/** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */
|
||||
queue_depth: number;
|
||||
tracked_branch: string | null;
|
||||
priority_label: PriorityLabel;
|
||||
/** Webhook configured? (true iff config.webhook_secret is set.) */
|
||||
webhook_configured: boolean;
|
||||
}
|
||||
|
||||
export type PriorityLabel = 'high' | 'normal' | 'low';
|
||||
|
||||
/** Numeric priority used by MinionQueue.add({ priority }). Lower = sooner. */
|
||||
const PRIORITY_VALUE: Record<PriorityLabel, number> = {
|
||||
high: -10,
|
||||
normal: 0,
|
||||
low: 5,
|
||||
};
|
||||
|
||||
const KNOWN_PRIORITY: Set<string> = new Set(['high', 'normal', 'low']);
|
||||
|
||||
/** Stderr-warn-once memo so a tight autopilot loop doesn't spam. */
|
||||
const _warnedSources = new Set<string>();
|
||||
|
||||
/** Test seam: reset memo so unit tests can re-trigger the warn path. */
|
||||
export function _resetPriorityWarningsForTest(): void {
|
||||
_warnedSources.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a source's priority label from its config row.
|
||||
*
|
||||
* Recognized values: 'high', 'normal', 'low'. Anything else (typos, integers,
|
||||
* nested objects) falls back to 'normal' AND emits a once-per-source-per-
|
||||
* process stderr warning naming the bad value + the fix command. Missing
|
||||
* key is silent ('normal' is the default).
|
||||
*/
|
||||
export function resolvePriorityLabel(
|
||||
sourceId: string,
|
||||
config: unknown,
|
||||
): PriorityLabel {
|
||||
const parsed = parseSourceConfig(config);
|
||||
const raw = parsed.priority;
|
||||
if (raw === undefined || raw === null) return 'normal';
|
||||
if (typeof raw === 'string' && KNOWN_PRIORITY.has(raw)) {
|
||||
return raw as PriorityLabel;
|
||||
}
|
||||
// Warn once per source per process.
|
||||
if (!_warnedSources.has(sourceId)) {
|
||||
_warnedSources.add(sourceId);
|
||||
process.stderr.write(
|
||||
`[gbrain] source "${sourceId}": invalid config.priority value ${JSON.stringify(raw)}; ` +
|
||||
`falling back to 'normal'. Fix: gbrain sources config set ${sourceId} priority normal\n`,
|
||||
);
|
||||
}
|
||||
return 'normal';
|
||||
}
|
||||
|
||||
/** Numeric priority for queue.add. */
|
||||
export function resolvePriority(sourceId: string, config: unknown): number {
|
||||
return PRIORITY_VALUE[resolvePriorityLabel(sourceId, config)];
|
||||
}
|
||||
|
||||
/**
|
||||
* True iff the source's last_sync_at is older than `intervalMs`, OR it has
|
||||
* never synced. Sources without a local_path are NOT considered stale (no
|
||||
* way to sync them). Used by autopilot D17 freshness gate.
|
||||
*/
|
||||
export function isSourceStale(src: SourceRow, intervalMs: number): boolean {
|
||||
if (!src.local_path) return false;
|
||||
if (!src.last_sync_at) return true;
|
||||
const lastMs = new Date(src.last_sync_at).getTime();
|
||||
return Date.now() - lastMs >= intervalMs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute per-source metrics for every source in one shot.
|
||||
*
|
||||
* Batched GROUP BY pipeline:
|
||||
* 1. sources: id, name, local_path, last_sync_at, config (one SELECT)
|
||||
* 2. pages by source_id (one GROUP BY)
|
||||
* 3. chunks by source_id with FILTER(embedding NOT NULL) (one GROUP BY)
|
||||
* 4. minion_jobs by data->>'sourceId' with FILTERs for failed-24h + queue depth
|
||||
*
|
||||
* Total: 4 queries regardless of source count. Each scans the relevant table
|
||||
* once. Same cost as the slowest single-source query in the old per-source loop.
|
||||
*/
|
||||
export async function computeAllSourceMetrics(
|
||||
engine: BrainEngine,
|
||||
sources: SourceRow[],
|
||||
): Promise<SourceMetrics[]> {
|
||||
if (sources.length === 0) return [];
|
||||
|
||||
const pageCounts = await pageCountsBySource(engine);
|
||||
const chunkCounts = await chunkCountsBySource(engine);
|
||||
const jobCounts = await jobCountsBySource(engine);
|
||||
const now = Date.now();
|
||||
|
||||
return sources.map((src) => {
|
||||
const cfg = parseSourceConfig(src.config);
|
||||
const pages = pageCounts.get(src.id) ?? 0;
|
||||
const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 };
|
||||
const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 };
|
||||
|
||||
const embedCoverage = chunkStats.total === 0
|
||||
? 100
|
||||
: Math.round((chunkStats.embedded / chunkStats.total) * 1000) / 10;
|
||||
|
||||
const lastMs = src.last_sync_at ? new Date(src.last_sync_at).getTime() : null;
|
||||
const lagSeconds = lastMs === null
|
||||
? null
|
||||
: Math.max(0, Math.floor((now - lastMs) / 1000));
|
||||
|
||||
return {
|
||||
source_id: src.id,
|
||||
name: src.name,
|
||||
local_path: src.local_path,
|
||||
federated: cfg.federated === true,
|
||||
total_pages: pages,
|
||||
total_chunks: chunkStats.total,
|
||||
embedded_chunks: chunkStats.embedded,
|
||||
embed_coverage_pct: embedCoverage,
|
||||
last_sync_at: src.last_sync_at,
|
||||
lag_seconds: lagSeconds,
|
||||
failed_jobs_24h: jobStats.failed_24h,
|
||||
queue_depth: jobStats.queue_depth,
|
||||
tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null,
|
||||
priority_label: resolvePriorityLabel(src.id, src.config),
|
||||
webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
async function pageCountsBySource(engine: BrainEngine): Promise<Map<string, number>> {
|
||||
const rows = await engine.executeRaw<{ source_id: string; n: number }>(
|
||||
`SELECT source_id, COUNT(*)::int AS n
|
||||
FROM pages
|
||||
WHERE deleted_at IS NULL
|
||||
GROUP BY source_id`,
|
||||
);
|
||||
const m = new Map<string, number>();
|
||||
for (const r of rows) m.set(r.source_id, Number(r.n));
|
||||
return m;
|
||||
}
|
||||
|
||||
async function chunkCountsBySource(engine: BrainEngine): Promise<Map<string, { total: number; embedded: number }>> {
|
||||
const rows = await engine.executeRaw<{ source_id: string; total: number; embedded: number }>(
|
||||
`SELECT p.source_id,
|
||||
COUNT(*)::int AS total,
|
||||
COUNT(*) FILTER (WHERE c.embedding IS NOT NULL)::int AS embedded
|
||||
FROM content_chunks c
|
||||
JOIN pages p ON p.id = c.page_id
|
||||
WHERE p.deleted_at IS NULL
|
||||
GROUP BY p.source_id`,
|
||||
);
|
||||
const m = new Map<string, { total: number; embedded: number }>();
|
||||
for (const r of rows) m.set(r.source_id, { total: Number(r.total), embedded: Number(r.embedded) });
|
||||
return m;
|
||||
}
|
||||
|
||||
async function jobCountsBySource(engine: BrainEngine): Promise<Map<string, { failed_24h: number; queue_depth: number }>> {
|
||||
// Pre-v0.11 brains don't have minion_jobs; return empty map.
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>(
|
||||
`SELECT data->>'sourceId' AS source_id,
|
||||
COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h,
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth
|
||||
FROM minion_jobs
|
||||
WHERE name IN ('sync','embed-backfill')
|
||||
AND data->>'sourceId' IS NOT NULL
|
||||
GROUP BY data->>'sourceId'`,
|
||||
);
|
||||
const m = new Map<string, { failed_24h: number; queue_depth: number }>();
|
||||
for (const r of rows) {
|
||||
m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) });
|
||||
}
|
||||
return m;
|
||||
} catch {
|
||||
return new Map();
|
||||
}
|
||||
}
|
||||
129
src/core/sources-load.ts
Normal file
129
src/core/sources-load.ts
Normal file
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* Shared source-table loader (v0.40 Federated Sync v2 — D7).
|
||||
*
|
||||
* Before v0.40, the only caller that enumerated `sources` was `runList` in
|
||||
* src/commands/sources.ts. v0.40 adds four more enumerators: `gbrain sync --all`
|
||||
* fan-out, autopilot per-source dispatch, `gbrain sources status`, and the
|
||||
* `federation_health` doctor check. Going from 1→5 inline SELECTs invites
|
||||
* silent drift the next time someone adds a column to `sources`.
|
||||
*
|
||||
* This module is the single source of truth for that read path. Adding a
|
||||
* column means updating exactly one projection.
|
||||
*
|
||||
* Engine-agnostic: works on both Postgres and PGLite (same SQL surface).
|
||||
*
|
||||
* Why no engine method: BrainEngine parity would force PGLite + Postgres
|
||||
* implementations even though both run identical SQL through `executeRaw`.
|
||||
* A shared helper hits the bar at lower cost.
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export interface SourceRow {
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
last_commit: string | null;
|
||||
last_sync_at: Date | null;
|
||||
/** Postgres returns object; PGLite returns JSON string. Parse via `parseSourceConfig`. */
|
||||
config: Record<string, unknown> | string;
|
||||
created_at: Date;
|
||||
archived?: boolean;
|
||||
}
|
||||
|
||||
export interface LoadAllSourcesOpts {
|
||||
/** Include soft-archived rows (default false). */
|
||||
includeArchived?: boolean;
|
||||
/** Only return sources with config.federated === true (default false). */
|
||||
federatedOnly?: boolean;
|
||||
}
|
||||
|
||||
/** Parse `sources.config` to a plain object regardless of driver shape. */
|
||||
export function parseSourceConfig(config: unknown): Record<string, unknown> {
|
||||
if (typeof config === 'string') {
|
||||
try { return JSON.parse(config) as Record<string, unknown>; } catch { return {}; }
|
||||
}
|
||||
if (typeof config === 'object' && config !== null) return config as Record<string, unknown>;
|
||||
return {};
|
||||
}
|
||||
|
||||
/** True iff the source's config.federated field is the literal boolean true. */
|
||||
export function isSourceFederated(config: unknown): boolean {
|
||||
const parsed = parseSourceConfig(config);
|
||||
return parsed.federated === true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Enumerate every source. Order: 'default' first, then alphabetical by id.
|
||||
*
|
||||
* Caller filters in-process when the predicate is cheap (federatedOnly,
|
||||
* includeArchived). For source-id targeted reads use `fetchSource` instead
|
||||
* (single-row SELECT).
|
||||
*/
|
||||
export async function loadAllSources(
|
||||
engine: BrainEngine,
|
||||
opts: LoadAllSourcesOpts = {},
|
||||
): Promise<SourceRow[]> {
|
||||
// Defensive on legacy brains pre-v0.26.5 that lack the archived column.
|
||||
let rows: SourceRow[];
|
||||
try {
|
||||
rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived
|
||||
FROM sources
|
||||
ORDER BY (id = 'default') DESC, id`,
|
||||
);
|
||||
} catch (err) {
|
||||
// Forward-reference safety: pre-v0.26.5 brains have no `archived` column.
|
||||
// Re-issue without it; archived defaults to false.
|
||||
if (isUndefinedColumnError(err)) {
|
||||
rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, name, local_path, last_commit, last_sync_at, config, created_at
|
||||
FROM sources
|
||||
ORDER BY (id = 'default') DESC, id`,
|
||||
);
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
let filtered = rows;
|
||||
if (!opts.includeArchived) {
|
||||
filtered = filtered.filter((r) => r.archived !== true);
|
||||
}
|
||||
if (opts.federatedOnly) {
|
||||
filtered = filtered.filter((r) => isSourceFederated(r.config));
|
||||
}
|
||||
return filtered;
|
||||
}
|
||||
|
||||
/** Single-row fetch — kept here so callers don't grow yet-another SELECT. */
|
||||
export async function fetchSource(
|
||||
engine: BrainEngine,
|
||||
id: string,
|
||||
): Promise<SourceRow | null> {
|
||||
try {
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, name, local_path, last_commit, last_sync_at, config, created_at, archived
|
||||
FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
return rows[0] ?? null;
|
||||
} catch (err) {
|
||||
if (isUndefinedColumnError(err)) {
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, name, local_path, last_commit, last_sync_at, config, created_at
|
||||
FROM sources WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
return rows[0] ?? null;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/** Driver-tolerant 42703 detector. Mirrors src/core/utils.ts pattern. */
|
||||
function isUndefinedColumnError(err: unknown): boolean {
|
||||
if (!err || typeof err !== 'object') return false;
|
||||
const e = err as { code?: string; message?: string };
|
||||
if (e.code === '42703') return true;
|
||||
return typeof e.message === 'string' && /column .* does not exist/i.test(e.message);
|
||||
}
|
||||
34
src/core/timing-safe.ts
Normal file
34
src/core/timing-safe.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
/**
|
||||
* Constant-time hex string compare (v0.40 D15.5 extraction).
|
||||
*
|
||||
* Before v0.40 this lived as a closure inside `runServeHttp` in
|
||||
* `src/commands/serve-http.ts:601`. It needs to be importable so the new
|
||||
* `POST /webhooks/github` handler can verify GitHub HMAC-SHA256 signatures
|
||||
* with the same constant-time semantics as the admin-cookie compare.
|
||||
*
|
||||
* Why extract instead of duplicate: codex outside-voice flagged the closure
|
||||
* scope as "hand-waving over actual module boundaries." Two consumers + one
|
||||
* function = extract. Source-text grep guard in `test/timing-safe.test.ts`
|
||||
* pins that callers import from here rather than re-implementing.
|
||||
*
|
||||
* Both inputs are expected to be hex strings of equal length (caller asserts
|
||||
* the length invariant — e.g. sha256 hex is always 64 chars). `timingSafeEqual`
|
||||
* throws on length mismatch; the early-return on length keeps the comparison
|
||||
* timing-independent of the length-check itself.
|
||||
*/
|
||||
import { timingSafeEqual } from 'node:crypto';
|
||||
|
||||
/**
|
||||
* True iff `a` and `b` are equal-length hex strings with the same bytes.
|
||||
* False on length mismatch (does NOT throw — admin-cookie + webhook callers
|
||||
* both prefer a clean boolean return so they can route to a 401 cleanly).
|
||||
*
|
||||
* Constant-time over the byte compare: `timingSafeEqual` is the underlying
|
||||
* primitive. Length mismatch short-circuits, which is acceptable because
|
||||
* length itself is not a secret (both callers know the expected hex length
|
||||
* from the algorithm — sha256 = 64 hex chars).
|
||||
*/
|
||||
export function safeHexEqual(a: string, b: string): boolean {
|
||||
if (a.length !== b.length) return false;
|
||||
return timingSafeEqual(Buffer.from(a, 'hex'), Buffer.from(b, 'hex'));
|
||||
}
|
||||
@@ -61,6 +61,15 @@ INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- v0.40 Federated Sync v2: partial expression index on config->>'github_repo'
|
||||
-- so POST /webhooks/github's source-by-repo lookup hits an index. Only rows
|
||||
-- with a configured webhook actually take up index entries. Both Postgres and
|
||||
-- PGLite support partial expression indexes. Migration v87 installs the same
|
||||
-- index on legacy brains (idempotent IF NOT EXISTS).
|
||||
CREATE INDEX IF NOT EXISTS sources_github_repo_idx
|
||||
ON sources ((config->>'github_repo'))
|
||||
WHERE config ? 'github_repo';
|
||||
|
||||
-- ============================================================
|
||||
-- pages: the core content table
|
||||
-- ============================================================
|
||||
|
||||
86
test/db-lock-per-source.test.ts
Normal file
86
test/db-lock-per-source.test.ts
Normal file
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* v0.40 Federated Sync v2 — per-source lock contract.
|
||||
*
|
||||
* Pins the iron-rule regression for the SYNC_LOCK_ID rename:
|
||||
* - SYNC_LOCK_ID === syncLockId('default') (back-compat alias)
|
||||
* - syncLockId(X) !== syncLockId(Y) for X !== Y (per-source isolation)
|
||||
* - two concurrent tryAcquireDbLock calls against DIFFERENT sources both succeed
|
||||
* - two concurrent tryAcquireDbLock calls against the SAME source: second returns null
|
||||
*
|
||||
* Without this guard, a future drift in the constant value would silently change
|
||||
* semantics (e.g. someone hardcoding 'gbrain-sync' elsewhere would no longer match
|
||||
* the same row, breaking the writer-window exclusion that performSync relies on).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { syncLockId, SYNC_LOCK_ID, tryAcquireDbLock } from '../src/core/db-lock.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('syncLockId (per-source lock helper)', () => {
|
||||
test('default source returns gbrain-sync:default', () => {
|
||||
expect(syncLockId('default')).toBe('gbrain-sync:default');
|
||||
});
|
||||
|
||||
test('non-default source returns gbrain-sync:<id>', () => {
|
||||
expect(syncLockId('zion-brain')).toBe('gbrain-sync:zion-brain');
|
||||
expect(syncLockId('media-corpus')).toBe('gbrain-sync:media-corpus');
|
||||
});
|
||||
|
||||
test('IRON-RULE: SYNC_LOCK_ID back-compat alias resolves to syncLockId(default)', () => {
|
||||
expect(SYNC_LOCK_ID).toBe(syncLockId('default'));
|
||||
expect(SYNC_LOCK_ID).toBe('gbrain-sync:default');
|
||||
});
|
||||
|
||||
test('different sources produce distinct lock keys', () => {
|
||||
expect(syncLockId('a')).not.toBe(syncLockId('b'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('tryAcquireDbLock with per-source keys', () => {
|
||||
test('two locks against different sources both succeed', async () => {
|
||||
const lockA = await tryAcquireDbLock(engine, syncLockId('source-a'));
|
||||
const lockB = await tryAcquireDbLock(engine, syncLockId('source-b'));
|
||||
expect(lockA).not.toBeNull();
|
||||
expect(lockB).not.toBeNull();
|
||||
await lockA?.release();
|
||||
await lockB?.release();
|
||||
});
|
||||
|
||||
test('second lock against the same source returns null while first is held', async () => {
|
||||
const first = await tryAcquireDbLock(engine, syncLockId('source-c'));
|
||||
expect(first).not.toBeNull();
|
||||
const second = await tryAcquireDbLock(engine, syncLockId('source-c'));
|
||||
expect(second).toBeNull();
|
||||
await first?.release();
|
||||
// After release, third acquire succeeds.
|
||||
const third = await tryAcquireDbLock(engine, syncLockId('source-c'));
|
||||
expect(third).not.toBeNull();
|
||||
await third?.release();
|
||||
});
|
||||
|
||||
test('SYNC_LOCK_ID and syncLockId(default) acquire the SAME lock row', async () => {
|
||||
// This is the critical back-compat check: pre-v0.40 callers using SYNC_LOCK_ID
|
||||
// and new callers using syncLockId('default') MUST conflict (not bypass each other).
|
||||
const first = await tryAcquireDbLock(engine, SYNC_LOCK_ID);
|
||||
expect(first).not.toBeNull();
|
||||
const second = await tryAcquireDbLock(engine, syncLockId('default'));
|
||||
expect(second).toBeNull();
|
||||
await first?.release();
|
||||
});
|
||||
});
|
||||
102
test/doctor-federation-health.test.ts
Normal file
102
test/doctor-federation-health.test.ts
Normal file
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* Tests for src/commands/doctor.ts:checkFederationHealth (v0.40 T12).
|
||||
*
|
||||
* Three-state contract:
|
||||
* ok — single-source brain, or every federated source healthy
|
||||
* warn — lag > 1h + federated, coverage < 95% with chunks > 100, OR 3+ failures
|
||||
* fail — lag > 24h, OR coverage < 50% with chunks > 1000
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { checkFederationHealth } from '../src/commands/doctor.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
await engine.executeRaw('DELETE FROM content_chunks');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
await engine.executeRaw(`DELETE FROM sources WHERE id != 'default'`);
|
||||
});
|
||||
|
||||
describe('checkFederationHealth', () => {
|
||||
test('single-source brain → ok with "no federation to check"', async () => {
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.name).toBe('federation_health');
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('no federation to check');
|
||||
});
|
||||
|
||||
test('multi-source healthy brain → ok with source count', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, last_sync_at) VALUES ('extra', 'extra', '{"federated":true}', NOW())`,
|
||||
);
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('source(s) healthy');
|
||||
});
|
||||
|
||||
test('source with lag > 1h + federated → warn with remediation', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, last_sync_at) VALUES ('stale-source', 'stale-source', '{"federated":true}', NOW() - INTERVAL '2 hours')`,
|
||||
);
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toContain('stale-source');
|
||||
expect(check.message).toContain('gbrain sync trigger --source stale-source');
|
||||
});
|
||||
|
||||
test('source with lag > 24h → fail with remediation', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, last_sync_at) VALUES ('dead-source', 'dead-source', '{"federated":true}', NOW() - INTERVAL '48 hours')`,
|
||||
);
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.status).toBe('fail');
|
||||
expect(check.message).toContain('dead-source');
|
||||
expect(check.message).toContain('gbrain sync trigger');
|
||||
});
|
||||
|
||||
test('source with low embed coverage + chunks > 100 → warn', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, last_sync_at) VALUES ('uncovered', 'uncovered', '{"federated":true}', NOW())`,
|
||||
);
|
||||
// Seed 200 pages with chunks; 10 embedded.
|
||||
for (let i = 0; i < 200; i++) {
|
||||
await engine.putPage(`p${i}`, { type: 'note', title: `p${i}`, compiled_truth: `body ${i}` }, { sourceId: 'uncovered' });
|
||||
await engine.upsertChunks(
|
||||
`p${i}`,
|
||||
[{
|
||||
chunk_index: 0,
|
||||
chunk_text: `chunk ${i}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 1,
|
||||
embedding: i < 10 ? new Float32Array(1536) : undefined,
|
||||
}],
|
||||
{ sourceId: 'uncovered' },
|
||||
);
|
||||
}
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.status).toBe('warn');
|
||||
expect(check.message).toContain('uncovered');
|
||||
expect(check.message).toContain('embed coverage');
|
||||
expect(check.message).toContain('gbrain jobs submit embed-backfill');
|
||||
});
|
||||
|
||||
test('synced + zero pages → ok (vacuous truth, no coverage warn)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, last_sync_at) VALUES ('empty', 'empty', '{"federated":true}', NOW())`,
|
||||
);
|
||||
const check = await checkFederationHealth(engine);
|
||||
expect(check.status).toBe('ok');
|
||||
});
|
||||
});
|
||||
@@ -20,7 +20,8 @@ import { tmpdir } from 'os';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import { runExtractFacts } from '../../src/core/cycle/extract-facts.ts';
|
||||
import { tryAcquireDbLock, SYNC_LOCK_ID } from '../../src/core/db-lock.ts';
|
||||
// v0.40: per-source lock id replaces the legacy bare SYNC_LOCK_ID constant.
|
||||
// Tests below hand-craft the lock row via SQL to simulate contention.
|
||||
|
||||
const SKIP = !hasDatabase();
|
||||
const describeMaybe = SKIP ? describe.skip : describe;
|
||||
@@ -41,7 +42,8 @@ beforeEach(async () => {
|
||||
// Per-test cleanup: truncate the tables this suite mutates.
|
||||
await engine.executeRaw('TRUNCATE TABLE facts RESTART IDENTITY CASCADE');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync'`);
|
||||
// v0.40: phantom redirect now uses per-source lock id (gbrain-sync:<source>).
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync:default'`);
|
||||
});
|
||||
|
||||
function tempBrain(): string {
|
||||
@@ -154,9 +156,10 @@ describeMaybe('phantom-redirect E2E (Postgres)', () => {
|
||||
// would still see the row as held (different "logical" holder via the
|
||||
// pid check, but the TTL is what really gates re-acquire). Override
|
||||
// by inserting a row with a different pid + future TTL.
|
||||
// v0.40 D16: phantom acquires per-source lock; default source = gbrain-sync:default.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
|
||||
VALUES ('gbrain-sync', 9999, 'simulated-other-host', now(), now() + interval '1 hour')
|
||||
VALUES ('gbrain-sync:default', 9999, 'simulated-other-host', now(), now() + interval '1 hour')
|
||||
ON CONFLICT (id) DO UPDATE SET holder_pid=9999, ttl_expires_at=now() + interval '1 hour'`,
|
||||
);
|
||||
|
||||
@@ -180,7 +183,8 @@ describeMaybe('phantom-redirect E2E (Postgres)', () => {
|
||||
expect(existsSync(join(brainDir, 'alice.md'))).toBe(true);
|
||||
|
||||
// Cleanup
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync'`);
|
||||
// v0.40: phantom redirect now uses per-source lock id (gbrain-sync:<source>).
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id='gbrain-sync:default'`);
|
||||
} finally {
|
||||
rmSync(brainDir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
174
test/embed-backfill-submit.test.ts
Normal file
174
test/embed-backfill-submit.test.ts
Normal file
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* Tests for src/core/embed-backfill-submit.ts (v0.40 D19).
|
||||
*
|
||||
* Validates the submission gate layer:
|
||||
* - Default path: submits with priority 5 + idempotency bucket
|
||||
* - Cooldown: refuses re-submission inside the window
|
||||
* - Active-job: refuses while a same-source job is active/waiting
|
||||
* - 24h spend cap: refuses when accumulated spend >= cap
|
||||
* - Config overrides honored (per-test cap + cooldown)
|
||||
* - Override knobs in opts honored (test seam)
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
submitEmbedBackfill,
|
||||
COOLDOWN_CONFIG_KEY,
|
||||
SPEND_CAP_CONFIG_KEY,
|
||||
} from '../src/core/embed-backfill-submit.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000); // 30s — PGLite WASM cold-start + 89 migrations exceeds 5s default
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Surgical reset (mirrors test/minions.test.ts) — full TRUNCATE wipes the
|
||||
// config table's `version` key that MinionQueue.ensureSchema() reads.
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
describe('submitEmbedBackfill — happy path', () => {
|
||||
test('submits with priority 5 + idempotency key on a clean source', async () => {
|
||||
const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' });
|
||||
expect(result.status).toBe('submitted');
|
||||
expect(result.jobId).toBeDefined();
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.getJob(result.jobId!);
|
||||
expect(job).not.toBeNull();
|
||||
expect(job!.name).toBe('embed-backfill');
|
||||
expect(job!.priority).toBe(5);
|
||||
expect((job!.data as { sourceId: string }).sourceId).toBe('default');
|
||||
});
|
||||
|
||||
test('respects opts.priority override', async () => {
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
priority: -10,
|
||||
});
|
||||
expect(result.status).toBe('submitted');
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.getJob(result.jobId!);
|
||||
expect(job!.priority).toBe(-10);
|
||||
});
|
||||
});
|
||||
|
||||
describe('submitEmbedBackfill — cooldown gate', () => {
|
||||
test('blocks re-submission while a same-source job is active', async () => {
|
||||
const queue = new MinionQueue(engine);
|
||||
// Seed an active job manually
|
||||
await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status='active' WHERE name='embed-backfill'`,
|
||||
);
|
||||
|
||||
const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' });
|
||||
expect(result.status).toBe('cooldown');
|
||||
expect(result.cooldownRemainingSeconds).toBeUndefined();
|
||||
});
|
||||
|
||||
test('blocks re-submission inside the cooldown window after recent finish', async () => {
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
// Mark completed 1 minute ago
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '1 minute' WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
cooldownMinOverride: 10, // 10min cooldown; 1min elapsed → blocked
|
||||
});
|
||||
expect(result.status).toBe('cooldown');
|
||||
expect(result.cooldownRemainingSeconds).toBeGreaterThan(0);
|
||||
expect(result.cooldownRemainingSeconds).toBeLessThanOrEqual(10 * 60);
|
||||
});
|
||||
|
||||
test('allows re-submission after cooldown elapses', async () => {
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
// Mark completed 11 minutes ago — past the 10-min cooldown
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '11 minutes' WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
cooldownMinOverride: 10,
|
||||
});
|
||||
expect(result.status).toBe('submitted');
|
||||
});
|
||||
|
||||
test('config-overridable cooldown (via embed.backfill_cooldown_min)', async () => {
|
||||
await engine.setConfig(COOLDOWN_CONFIG_KEY, '60'); // 60min cooldown
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '30 minutes' WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
|
||||
const result = await submitEmbedBackfill(engine, 'default', { reason: 'unit' });
|
||||
expect(result.status).toBe('cooldown');
|
||||
});
|
||||
});
|
||||
|
||||
describe('submitEmbedBackfill — 24h spend cap', () => {
|
||||
test('refuses when spend24hFn returns >= cap', async () => {
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
spendCapUsdOverride: 25,
|
||||
spend24hFn: async () => 25,
|
||||
});
|
||||
expect(result.status).toBe('spend_capped');
|
||||
expect(result.spend24hUsd).toBe(25);
|
||||
expect(result.spendCapUsd).toBe(25);
|
||||
});
|
||||
|
||||
test('admits when spend24hFn returns < cap', async () => {
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
spendCapUsdOverride: 25,
|
||||
spend24hFn: async () => 24.99,
|
||||
});
|
||||
expect(result.status).toBe('submitted');
|
||||
});
|
||||
|
||||
test('config-overridable spend cap (via embed.backfill_max_usd_per_source_24h)', async () => {
|
||||
await engine.setConfig(SPEND_CAP_CONFIG_KEY, '5');
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
spend24hFn: async () => 5,
|
||||
});
|
||||
expect(result.status).toBe('spend_capped');
|
||||
expect(result.spendCapUsd).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('submitEmbedBackfill — source isolation', () => {
|
||||
test('cooldown is per-source, not global', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}') ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
const queue = new MinionQueue(engine);
|
||||
// Active job on 'default'
|
||||
await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
await engine.executeRaw(`UPDATE minion_jobs SET status='active' WHERE name='embed-backfill'`);
|
||||
|
||||
// Submit for 'other' — should NOT be blocked
|
||||
const result = await submitEmbedBackfill(engine, 'other', { reason: 'unit' });
|
||||
expect(result.status).toBe('submitted');
|
||||
});
|
||||
});
|
||||
230
test/embed-stale.test.ts
Normal file
230
test/embed-stale.test.ts
Normal file
@@ -0,0 +1,230 @@
|
||||
/**
|
||||
* Tests for src/core/embed-stale.ts (v0.40 D15.2).
|
||||
*
|
||||
* Hermetic — uses an injected `embedFn` so no network call lands. Validates:
|
||||
* - empty stale set → done:true, embedded:0
|
||||
* - multi-batch run → embed every stale chunk, advance cursor correctly
|
||||
* - kill mid-flight (signal.aborted) → aborted:true, partial progress preserved
|
||||
* - resume from cursor → picks up where prior call left off (DB predicate)
|
||||
* - per-page embedFn throw → logged + skipped, NOT propagated; chunks stay NULL
|
||||
*
|
||||
* Why PGLite: validates the engine.listStaleChunks/getChunks/upsertChunks
|
||||
* roundtrip the helper depends on, not just the loop control flow.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { embedStaleForSource } from '../src/core/embed-stale.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Seed a page with N stale chunks (no embedding) into the default source. */
|
||||
async function seedPageWithStaleChunks(slug: string, chunkCount: number): Promise<void> {
|
||||
await engine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: slug,
|
||||
compiled_truth: `# ${slug}\n\nseeded`,
|
||||
});
|
||||
const chunks: ChunkInput[] = Array.from({ length: chunkCount }, (_, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: `chunk ${i} of ${slug}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 4,
|
||||
embedding: undefined, // NULL = stale
|
||||
}));
|
||||
await engine.upsertChunks(slug, chunks);
|
||||
}
|
||||
|
||||
/** Deterministic fake embedder — returns unit-length 1536-dim vectors with
|
||||
* first dim = text length, so we can assert specific chunks got embedded. */
|
||||
function fakeEmbedFn(texts: string[]): Promise<Float32Array[]> {
|
||||
return Promise.resolve(
|
||||
texts.map((t) => {
|
||||
const v = new Float32Array(1536);
|
||||
v[0] = t.length;
|
||||
v[1] = 1;
|
||||
return v;
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
describe('embedStaleForSource', () => {
|
||||
test('empty stale set returns done:true with zero embedded', async () => {
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: fakeEmbedFn,
|
||||
});
|
||||
expect(result).toEqual({
|
||||
embedded: 0,
|
||||
chunksProcessed: 0,
|
||||
pagesProcessed: 0,
|
||||
lastCursor: null,
|
||||
done: true,
|
||||
aborted: false,
|
||||
});
|
||||
});
|
||||
|
||||
test('embeds every stale chunk across multiple pages in one call', async () => {
|
||||
await seedPageWithStaleChunks('a', 5);
|
||||
await seedPageWithStaleChunks('b', 3);
|
||||
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: fakeEmbedFn,
|
||||
});
|
||||
expect(result.done).toBe(true);
|
||||
expect(result.aborted).toBe(false);
|
||||
expect(result.embedded).toBe(8);
|
||||
expect(result.pagesProcessed).toBe(2);
|
||||
|
||||
// Verify DB: zero stale remaining for default.
|
||||
const stale = await engine.countStaleChunks({ sourceId: 'default' });
|
||||
expect(stale).toBe(0);
|
||||
});
|
||||
|
||||
test('respects batchSize for cursor pagination', async () => {
|
||||
await seedPageWithStaleChunks('a', 3);
|
||||
await seedPageWithStaleChunks('b', 3);
|
||||
let batchCount = 0;
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: fakeEmbedFn,
|
||||
batchSize: 2,
|
||||
onProgress: () => {
|
||||
batchCount++;
|
||||
},
|
||||
});
|
||||
expect(result.embedded).toBe(6);
|
||||
// 2-chunk batches across 6 stale rows = at least 3 progress callbacks.
|
||||
expect(batchCount).toBeGreaterThanOrEqual(3);
|
||||
});
|
||||
|
||||
test('IRON-RULE: aborted mid-flight → aborted:true, partial progress preserved', async () => {
|
||||
await seedPageWithStaleChunks('a', 4);
|
||||
await seedPageWithStaleChunks('b', 4);
|
||||
await seedPageWithStaleChunks('c', 4);
|
||||
const controller = new AbortController();
|
||||
// Batch size 4 = one page per batch. concurrency 1 = serialize keys.
|
||||
// Abort fires inside embedFn for page 'b', so 'a' lands, 'b' aborts mid-call,
|
||||
// and the third batch ('c') never starts.
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
batchSize: 4,
|
||||
concurrency: 1,
|
||||
signal: controller.signal,
|
||||
embedFn: async (texts) => {
|
||||
if (texts.some((t) => t.includes(' of b'))) {
|
||||
controller.abort();
|
||||
throw new Error('aborted'); // simulates HTTP abort throw
|
||||
}
|
||||
return fakeEmbedFn(texts);
|
||||
},
|
||||
});
|
||||
expect(result.aborted).toBe(true);
|
||||
expect(result.done).toBe(false);
|
||||
expect(result.embedded).toBe(4); // only 'a' landed
|
||||
// 'b' and 'c' (8 chunks) remain stale
|
||||
const stale = await engine.countStaleChunks({ sourceId: 'default' });
|
||||
expect(stale).toBe(8);
|
||||
});
|
||||
|
||||
test('IRON-RULE: kill + resume — second call picks up via embedding-IS-NULL predicate', async () => {
|
||||
await seedPageWithStaleChunks('a', 4);
|
||||
await seedPageWithStaleChunks('b', 4);
|
||||
|
||||
// First call aborts when 'b' is reached
|
||||
const controller = new AbortController();
|
||||
const first = await embedStaleForSource(engine, 'default', {
|
||||
batchSize: 4,
|
||||
concurrency: 1,
|
||||
signal: controller.signal,
|
||||
embedFn: async (texts) => {
|
||||
if (texts.some((t) => t.includes(' of b'))) {
|
||||
controller.abort();
|
||||
throw new Error('aborted');
|
||||
}
|
||||
return fakeEmbedFn(texts);
|
||||
},
|
||||
});
|
||||
expect(first.aborted).toBe(true);
|
||||
expect(first.embedded).toBe(4); // 'a' landed
|
||||
|
||||
// Second call with NO cursor — predicate excludes already-embedded chunks
|
||||
const second = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: fakeEmbedFn,
|
||||
});
|
||||
expect(second.done).toBe(true);
|
||||
expect(first.embedded + second.embedded).toBe(8);
|
||||
|
||||
const stale = await engine.countStaleChunks({ sourceId: 'default' });
|
||||
expect(stale).toBe(0);
|
||||
});
|
||||
|
||||
test('per-page embedFn throw is logged but does NOT propagate', async () => {
|
||||
await seedPageWithStaleChunks('good', 2);
|
||||
await seedPageWithStaleChunks('bad', 2);
|
||||
|
||||
let badCount = 0;
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: async (texts) => {
|
||||
if (texts.some((t) => t.includes('bad'))) {
|
||||
badCount++;
|
||||
throw new Error('intentional embed failure');
|
||||
}
|
||||
return fakeEmbedFn(texts);
|
||||
},
|
||||
});
|
||||
|
||||
// The helper itself didn't throw
|
||||
expect(result.done).toBe(true);
|
||||
expect(badCount).toBe(1);
|
||||
|
||||
// 'good' chunks got embedded; 'bad' chunks stayed NULL
|
||||
expect(result.embedded).toBe(2);
|
||||
const stale = await engine.countStaleChunks({ sourceId: 'default' });
|
||||
expect(stale).toBe(2);
|
||||
});
|
||||
|
||||
test('source-scoped: does not touch other sources', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await seedPageWithStaleChunks('a', 3);
|
||||
await engine.putPage('b', {
|
||||
type: 'note',
|
||||
title: 'b',
|
||||
compiled_truth: '# b\n\nseeded',
|
||||
}, { sourceId: 'other' });
|
||||
await engine.upsertChunks(
|
||||
'b',
|
||||
Array.from({ length: 3 }, (_, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: `other ${i}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 4,
|
||||
embedding: undefined,
|
||||
})),
|
||||
{ sourceId: 'other' },
|
||||
);
|
||||
|
||||
const result = await embedStaleForSource(engine, 'default', {
|
||||
embedFn: fakeEmbedFn,
|
||||
});
|
||||
expect(result.embedded).toBe(3);
|
||||
|
||||
// 'other' source still has 3 stale chunks
|
||||
const otherStale = await engine.countStaleChunks({ sourceId: 'other' });
|
||||
expect(otherStale).toBe(3);
|
||||
});
|
||||
});
|
||||
52
test/feature-flags.test.ts
Normal file
52
test/feature-flags.test.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Tests for src/core/feature-flags.ts (v0.40 D23).
|
||||
*
|
||||
* Pin the default-on posture and the explicit-'false'-disables semantics.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { isFederatedV2Enabled, FEDERATED_V2_CONFIG_KEY } from '../src/core/feature-flags.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('isFederatedV2Enabled', () => {
|
||||
test('default (key unset) → enabled', async () => {
|
||||
expect(await isFederatedV2Enabled(engine)).toBe(true);
|
||||
});
|
||||
|
||||
test('explicit "false" → disabled', async () => {
|
||||
await engine.setConfig(FEDERATED_V2_CONFIG_KEY, 'false');
|
||||
expect(await isFederatedV2Enabled(engine)).toBe(false);
|
||||
});
|
||||
|
||||
test('explicit "true" → enabled', async () => {
|
||||
await engine.setConfig(FEDERATED_V2_CONFIG_KEY, 'true');
|
||||
expect(await isFederatedV2Enabled(engine)).toBe(true);
|
||||
});
|
||||
|
||||
test('anything-not-literally-false → enabled (defensive default)', async () => {
|
||||
for (const v of ['False', 'FALSE', '0', 'off', 'no', '']) {
|
||||
await engine.setConfig(FEDERATED_V2_CONFIG_KEY, v);
|
||||
expect(await isFederatedV2Enabled(engine)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('config key name is stable', () => {
|
||||
expect(FEDERATED_V2_CONFIG_KEY).toBe('sync.federated_v2');
|
||||
});
|
||||
});
|
||||
143
test/handlers-embed-backfill.test.ts
Normal file
143
test/handlers-embed-backfill.test.ts
Normal file
@@ -0,0 +1,143 @@
|
||||
/**
|
||||
* Tests for src/core/minions/handlers/embed-backfill.ts (v0.40 D2, D6).
|
||||
*
|
||||
* Validates the handler-side contract:
|
||||
* - Happy path: embeds, returns 'success' with chunk + spend counts
|
||||
* - D2 lock: second concurrent handler call returns 'already_in_progress'
|
||||
* - D15.1 finally: lock ALWAYS releases (try/finally even on abort)
|
||||
*
|
||||
* Hermetic — uses injected embedFn via the underlying embedStaleForSource
|
||||
* test seam? No — the handler doesn't expose embedFn passthrough. Instead
|
||||
* we exercise the handler against a brain with zero stale chunks so no
|
||||
* actual embed call lands. That gives us a deterministic test of the lock
|
||||
* + budget + status branches without needing a fake gateway.
|
||||
*
|
||||
* The kill-resume contract is covered by test/embed-stale.test.ts at the
|
||||
* helper layer; the handler just routes through.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { makeEmbedBackfillHandler } from '../src/core/minions/handlers/embed-backfill.ts';
|
||||
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
|
||||
import type { MinionJobContext } from '../src/core/minions/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean minion_jobs + lock rows. Preserve config (schema version + flags).
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'gbrain-embed-backfill:%'`);
|
||||
});
|
||||
|
||||
/** Build a minimal MinionJobContext for testing. */
|
||||
function fakeJob(data: Record<string, unknown>): MinionJobContext {
|
||||
const controller = new AbortController();
|
||||
return {
|
||||
id: 1,
|
||||
name: 'embed-backfill',
|
||||
data,
|
||||
attempts_made: 0,
|
||||
signal: controller.signal,
|
||||
shutdownSignal: controller.signal,
|
||||
updateProgress: async () => {},
|
||||
updateTokens: async () => {},
|
||||
log: async () => {},
|
||||
isActive: async () => true,
|
||||
readInbox: async () => [],
|
||||
};
|
||||
}
|
||||
|
||||
describe('embed-backfill handler — happy path', () => {
|
||||
test('zero stale chunks → success with embedded=0', async () => {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
const result = await handler(fakeJob({ sourceId: 'default' }));
|
||||
expect(result).toMatchObject({
|
||||
status: 'success',
|
||||
sourceId: 'default',
|
||||
embedded: 0,
|
||||
chunksProcessed: 0,
|
||||
pagesProcessed: 0,
|
||||
});
|
||||
});
|
||||
|
||||
test('throws when sourceId missing', async () => {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
await expect(handler(fakeJob({}))).rejects.toThrow(/sourceId is required/);
|
||||
});
|
||||
|
||||
test('throws when sourceId is empty string', async () => {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
await expect(handler(fakeJob({ sourceId: '' }))).rejects.toThrow(/sourceId is required/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('embed-backfill handler — D2 lock contract', () => {
|
||||
test('IRON-RULE: second call returns already_in_progress when lock is held', async () => {
|
||||
// Hold the per-source lock externally
|
||||
const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60);
|
||||
expect(lock).not.toBeNull();
|
||||
|
||||
try {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
const result = await handler(fakeJob({ sourceId: 'default' }));
|
||||
expect(result).toMatchObject({
|
||||
status: 'already_in_progress',
|
||||
sourceId: 'default',
|
||||
embedded: 0,
|
||||
spentUsd: 0,
|
||||
});
|
||||
} finally {
|
||||
await lock?.release();
|
||||
}
|
||||
});
|
||||
|
||||
test('different sources do not contend on each other locks', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other-src', 'other-src', '{"federated":true}') ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
const lockA = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60);
|
||||
expect(lockA).not.toBeNull();
|
||||
try {
|
||||
// 'other-src' should still succeed
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
const result = await handler(fakeJob({ sourceId: 'other-src' }));
|
||||
expect(result.status).toBe('success');
|
||||
} finally {
|
||||
await lockA?.release();
|
||||
}
|
||||
});
|
||||
|
||||
test('IRON-RULE: lock is released after handler completes (try/finally)', async () => {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
await handler(fakeJob({ sourceId: 'default' }));
|
||||
|
||||
// After handler returns, the lock row should NOT block a fresh acquire.
|
||||
const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60);
|
||||
expect(lock).not.toBeNull();
|
||||
await lock?.release();
|
||||
});
|
||||
|
||||
test('IRON-RULE: lock released on throw (sourceId-missing path)', async () => {
|
||||
const handler = makeEmbedBackfillHandler(engine);
|
||||
try {
|
||||
await handler(fakeJob({})); // throws before lock is acquired
|
||||
} catch {
|
||||
// expected
|
||||
}
|
||||
// Lock was never acquired (throw happened in parseParams pre-lock),
|
||||
// so the row should be cleanly absent. Verify a fresh acquire works.
|
||||
const lock = await tryAcquireDbLock(engine, 'gbrain-embed-backfill:default', 60);
|
||||
expect(lock).not.toBeNull();
|
||||
await lock?.release();
|
||||
});
|
||||
});
|
||||
83
test/parallel.test.ts
Normal file
83
test/parallel.test.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Tests for src/core/parallel.ts (v0.40 Federated Sync v2).
|
||||
*
|
||||
* Pins: result order matches input order, one rejection doesn't kill others,
|
||||
* concurrency cap is a hard ceiling, empty input returns empty, invalid
|
||||
* concurrency throws TypeError.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { pMapAllSettled } from '../src/core/parallel.ts';
|
||||
|
||||
describe('pMapAllSettled', () => {
|
||||
test('returns results in input order', async () => {
|
||||
const items = [1, 2, 3, 4, 5];
|
||||
const results = await pMapAllSettled(items, 2, async (n) => n * 10);
|
||||
expect(results).toHaveLength(5);
|
||||
expect(results.map((r) => (r.status === 'fulfilled' ? r.value : null))).toEqual([10, 20, 30, 40, 50]);
|
||||
});
|
||||
|
||||
test('one rejection does not kill other items', async () => {
|
||||
const items = ['a', 'b', 'c'];
|
||||
const results = await pMapAllSettled(items, 2, async (s) => {
|
||||
if (s === 'b') throw new Error('intentional');
|
||||
return s.toUpperCase();
|
||||
});
|
||||
expect(results[0]).toEqual({ status: 'fulfilled', value: 'A' });
|
||||
expect(results[1].status).toBe('rejected');
|
||||
expect(results[1].status === 'rejected' && (results[1].reason as Error).message).toBe('intentional');
|
||||
expect(results[2]).toEqual({ status: 'fulfilled', value: 'C' });
|
||||
});
|
||||
|
||||
test('hard ceiling: never exceeds concurrency in flight', async () => {
|
||||
let inFlight = 0;
|
||||
let peak = 0;
|
||||
const items = Array.from({ length: 20 }, (_, i) => i);
|
||||
await pMapAllSettled(items, 3, async () => {
|
||||
inFlight++;
|
||||
peak = Math.max(peak, inFlight);
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
inFlight--;
|
||||
return 'ok';
|
||||
});
|
||||
expect(peak).toBeLessThanOrEqual(3);
|
||||
expect(peak).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('empty input returns empty array', async () => {
|
||||
const results = await pMapAllSettled([], 5, async () => 'x');
|
||||
expect(results).toEqual([]);
|
||||
});
|
||||
|
||||
test('concurrency > items.length runs all in parallel (no semaphore stall)', async () => {
|
||||
const start = Date.now();
|
||||
const items = [1, 2, 3];
|
||||
await pMapAllSettled(items, 100, async () => {
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
return 'ok';
|
||||
});
|
||||
const elapsed = Date.now() - start;
|
||||
// Sequential would be ~150ms; concurrent ~50ms. Allow generous slack.
|
||||
expect(elapsed).toBeLessThan(140);
|
||||
});
|
||||
|
||||
test('throws TypeError on concurrency < 1', async () => {
|
||||
await expect(pMapAllSettled([1], 0, async (x) => x)).rejects.toThrow(TypeError);
|
||||
await expect(pMapAllSettled([1], -3, async (x) => x)).rejects.toThrow(TypeError);
|
||||
});
|
||||
|
||||
test('throws TypeError on non-integer concurrency', async () => {
|
||||
await expect(pMapAllSettled([1], 1.5, async (x) => x)).rejects.toThrow(TypeError);
|
||||
await expect(pMapAllSettled([1], NaN, async (x) => x)).rejects.toThrow(TypeError);
|
||||
});
|
||||
|
||||
test('passes index to fn', async () => {
|
||||
const seen: Array<[string, number]> = [];
|
||||
await pMapAllSettled(['a', 'b', 'c'], 2, async (item, i) => {
|
||||
seen.push([item, i]);
|
||||
return item;
|
||||
});
|
||||
// Sort by item since execution order isn't guaranteed.
|
||||
seen.sort((a, b) => a[0].localeCompare(b[0]));
|
||||
expect(seen).toEqual([['a', 0], ['b', 1], ['c', 2]]);
|
||||
});
|
||||
});
|
||||
42
test/phantom-redirect-per-source-lock.test.ts
Normal file
42
test/phantom-redirect-per-source-lock.test.ts
Normal file
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* v0.40 D16 regression test: phantom-redirect uses per-source lock.
|
||||
*
|
||||
* Pre-v0.40 phantom-redirect acquired the bare `gbrain-sync` lock, blocking
|
||||
* every concurrent sync brain-wide. After D16, phantom acquires
|
||||
* `gbrain-sync:<sourceId>` — same-source sync still serializes; cross-source
|
||||
* sync proceeds unblocked.
|
||||
*
|
||||
* Pure source-text regression guard. The full behavior is covered by
|
||||
* `test/e2e/phantom-redirect.test.ts` (DATABASE_URL-gated). This file pins
|
||||
* the lock-id construction so any future drift back to the bare constant
|
||||
* fails loudly in the fast unit loop.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { syncLockId } from '../src/core/db-lock.ts';
|
||||
|
||||
const SRC = readFileSync('src/core/cycle/phantom-redirect.ts', 'utf8');
|
||||
|
||||
describe('phantom-redirect lock contract', () => {
|
||||
test('IRON-RULE: import line uses syncLockId, not bare SYNC_LOCK_ID', () => {
|
||||
// syncLockId must be imported; SYNC_LOCK_ID must NOT be (per-source posture).
|
||||
const importLine = SRC.split('\n').find((l) =>
|
||||
l.includes("from '../db-lock.ts'") && l.includes('import'),
|
||||
);
|
||||
expect(importLine).toBeDefined();
|
||||
expect(importLine).toContain('syncLockId');
|
||||
expect(importLine).not.toMatch(/\bSYNC_LOCK_ID\b/);
|
||||
});
|
||||
|
||||
test('IRON-RULE: acquireLockWithRetry call passes syncLockId(sourceId)', () => {
|
||||
// Banned: the bare `SYNC_LOCK_ID` constant slipped back in.
|
||||
// Required: the per-source helper threaded with the active sourceId.
|
||||
expect(SRC).toMatch(/acquireLockWithRetry\s*\(\s*engine\s*,\s*syncLockId\s*\(\s*sourceId\s*\)\s*\)/);
|
||||
expect(SRC).not.toMatch(/acquireLockWithRetry\s*\(\s*engine\s*,\s*SYNC_LOCK_ID\s*\)/);
|
||||
});
|
||||
|
||||
test('helper sanity: syncLockId returns gbrain-sync:<source> shape', () => {
|
||||
expect(syncLockId('default')).toBe('gbrain-sync:default');
|
||||
expect(syncLockId('zion-brain')).toBe('gbrain-sync:zion-brain');
|
||||
});
|
||||
});
|
||||
54
test/source-config-redact.test.ts
Normal file
54
test/source-config-redact.test.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* Tests for src/core/source-config-redact.ts (v0.40 D15.4).
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { redactSourceConfig, hasWebhookSecret } from '../src/core/source-config-redact.ts';
|
||||
|
||||
describe('redactSourceConfig', () => {
|
||||
test('redacts webhook_secret', () => {
|
||||
const input = { federated: true, webhook_secret: 'super-secret-key', github_repo: 'a/b' };
|
||||
const out = redactSourceConfig(input);
|
||||
expect(out.webhook_secret).toBe('<redacted>');
|
||||
expect(out.federated).toBe(true);
|
||||
expect(out.github_repo).toBe('a/b');
|
||||
});
|
||||
|
||||
test('does not mutate input', () => {
|
||||
const input = { webhook_secret: 'secret' };
|
||||
redactSourceConfig(input);
|
||||
expect(input.webhook_secret).toBe('secret');
|
||||
});
|
||||
|
||||
test('returns empty object for non-object input', () => {
|
||||
expect(redactSourceConfig(null)).toEqual({});
|
||||
expect(redactSourceConfig(undefined)).toEqual({});
|
||||
expect(redactSourceConfig('a string')).toEqual({});
|
||||
expect(redactSourceConfig(['arr'])).toEqual({});
|
||||
});
|
||||
|
||||
test('preserves nested objects (not deep-redacted)', () => {
|
||||
const input = {
|
||||
webhook_secret: 'x',
|
||||
nested: { allowed: 'value' },
|
||||
};
|
||||
const out = redactSourceConfig(input);
|
||||
expect(out.webhook_secret).toBe('<redacted>');
|
||||
expect(out.nested).toEqual({ allowed: 'value' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('hasWebhookSecret', () => {
|
||||
test('true when set + non-empty', () => {
|
||||
expect(hasWebhookSecret({ webhook_secret: 'x' })).toBe(true);
|
||||
});
|
||||
test('false when empty', () => {
|
||||
expect(hasWebhookSecret({ webhook_secret: '' })).toBe(false);
|
||||
});
|
||||
test('false when absent', () => {
|
||||
expect(hasWebhookSecret({})).toBe(false);
|
||||
});
|
||||
test('false on non-string values', () => {
|
||||
expect(hasWebhookSecret({ webhook_secret: 42 })).toBe(false);
|
||||
expect(hasWebhookSecret(null)).toBe(false);
|
||||
});
|
||||
});
|
||||
181
test/source-health.test.ts
Normal file
181
test/source-health.test.ts
Normal file
@@ -0,0 +1,181 @@
|
||||
/**
|
||||
* Tests for src/core/source-health.ts (v0.40 D12 + D9 + D17).
|
||||
*
|
||||
* Validates:
|
||||
* - computeAllSourceMetrics: batched GROUP BY shape, vacuous truth for zero pages
|
||||
* - resolvePriorityLabel: high/normal/low, unknown → normal + warn-once
|
||||
* - isSourceStale: never-synced + lag-exceeded + fresh + missing local_path
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
computeAllSourceMetrics,
|
||||
resolvePriorityLabel,
|
||||
resolvePriority,
|
||||
isSourceStale,
|
||||
_resetPriorityWarningsForTest,
|
||||
} from '../src/core/source-health.ts';
|
||||
import { loadAllSources } from '../src/core/sources-load.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Surgical reset: preserves config table (schema version).
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
await engine.executeRaw('DELETE FROM content_chunks');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
await engine.executeRaw(`DELETE FROM sources WHERE id != 'default'`);
|
||||
_resetPriorityWarningsForTest();
|
||||
});
|
||||
|
||||
describe('resolvePriorityLabel', () => {
|
||||
test('recognized values', () => {
|
||||
expect(resolvePriorityLabel('s', { priority: 'high' })).toBe('high');
|
||||
expect(resolvePriorityLabel('s', { priority: 'normal' })).toBe('normal');
|
||||
expect(resolvePriorityLabel('s', { priority: 'low' })).toBe('low');
|
||||
});
|
||||
test('missing → normal silently', () => {
|
||||
expect(resolvePriorityLabel('s', {})).toBe('normal');
|
||||
expect(resolvePriorityLabel('s', null)).toBe('normal');
|
||||
});
|
||||
test('unknown values → normal with warn', () => {
|
||||
// Reroute stderr to capture
|
||||
const orig = process.stderr.write.bind(process.stderr);
|
||||
let captured = '';
|
||||
process.stderr.write = ((chunk: string | Uint8Array) => {
|
||||
captured += typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8');
|
||||
return true;
|
||||
}) as never;
|
||||
try {
|
||||
expect(resolvePriorityLabel('zion-brain', { priority: 'urgent' })).toBe('normal');
|
||||
expect(captured).toContain('zion-brain');
|
||||
expect(captured).toContain('priority');
|
||||
expect(captured).toContain('normal');
|
||||
} finally {
|
||||
process.stderr.write = orig;
|
||||
}
|
||||
});
|
||||
test('warns once per source per process', () => {
|
||||
const orig = process.stderr.write.bind(process.stderr);
|
||||
let count = 0;
|
||||
process.stderr.write = ((chunk: string | Uint8Array) => {
|
||||
const s = typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8');
|
||||
if (s.includes('invalid config.priority')) count++;
|
||||
return true;
|
||||
}) as never;
|
||||
try {
|
||||
resolvePriorityLabel('s1', { priority: 'urgent' });
|
||||
resolvePriorityLabel('s1', { priority: 'urgent' }); // same source
|
||||
resolvePriorityLabel('s1', { priority: 42 }); // different bad value, same source
|
||||
expect(count).toBe(1);
|
||||
} finally {
|
||||
process.stderr.write = orig;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolvePriority (numeric)', () => {
|
||||
test('maps labels to MinionQueue priority integers', () => {
|
||||
expect(resolvePriority('s', { priority: 'high' })).toBe(-10);
|
||||
expect(resolvePriority('s', { priority: 'normal' })).toBe(0);
|
||||
expect(resolvePriority('s', { priority: 'low' })).toBe(5);
|
||||
expect(resolvePriority('s', {})).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSourceStale', () => {
|
||||
test('never-synced (last_sync_at null) → true', () => {
|
||||
const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: null, config: {}, created_at: new Date() };
|
||||
expect(isSourceStale(src, 60_000)).toBe(true);
|
||||
});
|
||||
test('no local_path → false (nothing to sync)', () => {
|
||||
const src = { id: 's', name: 's', local_path: null, last_commit: null, last_sync_at: null, config: {}, created_at: new Date() };
|
||||
expect(isSourceStale(src, 60_000)).toBe(false);
|
||||
});
|
||||
test('synced within interval → false', () => {
|
||||
const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 1000), config: {}, created_at: new Date() };
|
||||
expect(isSourceStale(src, 60_000)).toBe(false);
|
||||
});
|
||||
test('synced beyond interval → true', () => {
|
||||
const src = { id: 's', name: 's', local_path: '/path', last_commit: null, last_sync_at: new Date(Date.now() - 120_000), config: {}, created_at: new Date() };
|
||||
expect(isSourceStale(src, 60_000)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeAllSourceMetrics', () => {
|
||||
test('empty input returns empty', async () => {
|
||||
const result = await computeAllSourceMetrics(engine, []);
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
|
||||
test('zero-page source → embed_coverage_pct=100 (vacuous truth)', async () => {
|
||||
const sources = await loadAllSources(engine);
|
||||
const result = await computeAllSourceMetrics(engine, sources);
|
||||
const dflt = result.find((m) => m.source_id === 'default')!;
|
||||
expect(dflt.total_pages).toBe(0);
|
||||
expect(dflt.total_chunks).toBe(0);
|
||||
expect(dflt.embed_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('aggregates pages + chunks + embedding coverage per source', async () => {
|
||||
// Two pages with chunks, half embedded
|
||||
await engine.putPage('a', { type: 'note', title: 'a', compiled_truth: 'a' });
|
||||
await engine.putPage('b', { type: 'note', title: 'b', compiled_truth: 'b' });
|
||||
await engine.upsertChunks('a', [
|
||||
{ chunk_index: 0, chunk_text: 'one', chunk_source: 'compiled_truth', token_count: 1, embedding: new Float32Array(1536) },
|
||||
{ chunk_index: 1, chunk_text: 'two', chunk_source: 'compiled_truth', token_count: 1, embedding: undefined },
|
||||
]);
|
||||
await engine.upsertChunks('b', [
|
||||
{ chunk_index: 0, chunk_text: 'three', chunk_source: 'compiled_truth', token_count: 1, embedding: undefined },
|
||||
]);
|
||||
|
||||
const sources = await loadAllSources(engine);
|
||||
const result = await computeAllSourceMetrics(engine, sources);
|
||||
const dflt = result.find((m) => m.source_id === 'default')!;
|
||||
expect(dflt.total_pages).toBe(2);
|
||||
expect(dflt.total_chunks).toBe(3);
|
||||
expect(dflt.embedded_chunks).toBe(1);
|
||||
// 1/3 = 33.3%
|
||||
expect(dflt.embed_coverage_pct).toBeCloseTo(33.3, 1);
|
||||
});
|
||||
|
||||
test('lag_seconds is null when last_sync_at is null', async () => {
|
||||
const sources = await loadAllSources(engine);
|
||||
const result = await computeAllSourceMetrics(engine, sources);
|
||||
const dflt = result.find((m) => m.source_id === 'default')!;
|
||||
expect(dflt.lag_seconds).toBeNull();
|
||||
});
|
||||
|
||||
test('multi-source isolation: each source gets its own counts', async () => {
|
||||
await engine.executeRaw(`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}') ON CONFLICT (id) DO NOTHING`);
|
||||
await engine.putPage('a', { type: 'note', title: 'a', compiled_truth: 'a' });
|
||||
await engine.putPage('b', { type: 'note', title: 'b', compiled_truth: 'b' }, { sourceId: 'other' });
|
||||
|
||||
const sources = await loadAllSources(engine);
|
||||
const result = await computeAllSourceMetrics(engine, sources);
|
||||
expect(result.find((m) => m.source_id === 'default')!.total_pages).toBe(1);
|
||||
expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1);
|
||||
});
|
||||
|
||||
test('webhook_configured reflects config.webhook_secret presence', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`,
|
||||
);
|
||||
const sources = await loadAllSources(engine);
|
||||
const result = await computeAllSourceMetrics(engine, sources);
|
||||
const w = result.find((m) => m.source_id === 'webhooky')!;
|
||||
expect(w.webhook_configured).toBe(true);
|
||||
const d = result.find((m) => m.source_id === 'default')!;
|
||||
expect(d.webhook_configured).toBe(false);
|
||||
});
|
||||
});
|
||||
134
test/sources-load.test.ts
Normal file
134
test/sources-load.test.ts
Normal file
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* Tests for src/core/sources-load.ts (v0.40 D7).
|
||||
*
|
||||
* Validates the shared loader: ordering, filtering, config parsing, and
|
||||
* defensive fallback when legacy brains lack the `archived` column.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
loadAllSources,
|
||||
fetchSource,
|
||||
parseSourceConfig,
|
||||
isSourceFederated,
|
||||
} from '../src/core/sources-load.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
async function insertSource(id: string, opts: { federated?: boolean; archived?: boolean; config?: Record<string, unknown> } = {}): Promise<void> {
|
||||
const config = { ...opts.config, federated: opts.federated ?? false };
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config, archived)
|
||||
VALUES ($1, $1, $2::jsonb, $3)
|
||||
ON CONFLICT (id) DO UPDATE SET config = EXCLUDED.config, archived = EXCLUDED.archived`,
|
||||
[id, JSON.stringify(config), opts.archived ?? false],
|
||||
);
|
||||
}
|
||||
|
||||
describe('loadAllSources', () => {
|
||||
test('returns default source first, then alphabetical', async () => {
|
||||
await insertSource('zebra');
|
||||
await insertSource('alpha');
|
||||
const rows = await loadAllSources(engine);
|
||||
expect(rows.map((r) => r.id)).toEqual(['default', 'alpha', 'zebra']);
|
||||
});
|
||||
|
||||
test('excludes archived rows by default', async () => {
|
||||
await insertSource('keep');
|
||||
await insertSource('archived-one', { archived: true });
|
||||
const rows = await loadAllSources(engine);
|
||||
expect(rows.map((r) => r.id).sort()).toEqual(['default', 'keep']);
|
||||
});
|
||||
|
||||
test('includeArchived: true returns archived rows', async () => {
|
||||
await insertSource('keep');
|
||||
await insertSource('archived-one', { archived: true });
|
||||
const rows = await loadAllSources(engine, { includeArchived: true });
|
||||
expect(rows.map((r) => r.id).sort()).toEqual(['archived-one', 'default', 'keep']);
|
||||
});
|
||||
|
||||
test('federatedOnly: filters to config.federated=true', async () => {
|
||||
await insertSource('iso', { federated: false });
|
||||
await insertSource('fed-a', { federated: true });
|
||||
await insertSource('fed-b', { federated: true });
|
||||
const rows = await loadAllSources(engine, { federatedOnly: true });
|
||||
// default source seeded by resetPgliteState has federated: true
|
||||
expect(rows.map((r) => r.id).sort()).toEqual(['default', 'fed-a', 'fed-b']);
|
||||
});
|
||||
|
||||
test('rows have the full SourceRow projection (no surprise drift)', async () => {
|
||||
await insertSource('shapecheck', { federated: true, config: { tracked_branch: 'main' } });
|
||||
const rows = await loadAllSources(engine);
|
||||
const target = rows.find((r) => r.id === 'shapecheck');
|
||||
expect(target).toBeDefined();
|
||||
expect(target).toMatchObject({
|
||||
id: 'shapecheck',
|
||||
name: 'shapecheck',
|
||||
local_path: null,
|
||||
last_commit: null,
|
||||
last_sync_at: null,
|
||||
});
|
||||
expect(target!.config).toBeDefined();
|
||||
expect(target!.created_at).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('fetchSource', () => {
|
||||
test('returns row for known id', async () => {
|
||||
await insertSource('mybrain', { federated: true });
|
||||
const row = await fetchSource(engine, 'mybrain');
|
||||
expect(row?.id).toBe('mybrain');
|
||||
expect(isSourceFederated(row!.config)).toBe(true);
|
||||
});
|
||||
|
||||
test('returns null for unknown id', async () => {
|
||||
const row = await fetchSource(engine, 'does-not-exist');
|
||||
expect(row).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseSourceConfig', () => {
|
||||
test('passes through plain object', () => {
|
||||
expect(parseSourceConfig({ federated: true })).toEqual({ federated: true });
|
||||
});
|
||||
|
||||
test('parses JSON string', () => {
|
||||
expect(parseSourceConfig('{"federated":true}')).toEqual({ federated: true });
|
||||
});
|
||||
|
||||
test('returns empty object on null / undefined / non-object', () => {
|
||||
expect(parseSourceConfig(null)).toEqual({});
|
||||
expect(parseSourceConfig(undefined)).toEqual({});
|
||||
expect(parseSourceConfig(42)).toEqual({});
|
||||
});
|
||||
|
||||
test('returns empty object on malformed JSON string', () => {
|
||||
expect(parseSourceConfig('{')).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSourceFederated', () => {
|
||||
test('strict-true requirement', () => {
|
||||
expect(isSourceFederated({ federated: true })).toBe(true);
|
||||
expect(isSourceFederated({ federated: false })).toBe(false);
|
||||
expect(isSourceFederated({ federated: 'true' })).toBe(false); // strict boolean
|
||||
expect(isSourceFederated({ federated: 1 })).toBe(false);
|
||||
expect(isSourceFederated({})).toBe(false);
|
||||
expect(isSourceFederated(null)).toBe(false);
|
||||
});
|
||||
});
|
||||
125
test/sources-webhook.test.ts
Normal file
125
test/sources-webhook.test.ts
Normal file
@@ -0,0 +1,125 @@
|
||||
/**
|
||||
* Tests for /webhooks/github HMAC verification semantics (v0.40 T10).
|
||||
*
|
||||
* The handler itself is wired inside serve-http.ts's runServeHttp closure
|
||||
* (hard to invoke without bringing up the full Express app). This file pins
|
||||
* the load-bearing primitives:
|
||||
*
|
||||
* 1. GitHub HMAC sig format matches `sha256=<hex>` via createHmac.
|
||||
* 2. safeHexEqual is constant-time and rejects mismatches.
|
||||
* 3. The expected payload schema (repository.full_name + ref) parses.
|
||||
* 4. Branch ref construction: refs/heads/<tracked_branch>.
|
||||
*
|
||||
* The full HTTP path is covered by test/e2e/webhook-github.test.ts
|
||||
* (DATABASE_URL-gated; not present in this wave — filed as a follow-up
|
||||
* E2E pinned by the test plan artifact).
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { createHmac } from 'node:crypto';
|
||||
import { safeHexEqual } from '../src/core/timing-safe.ts';
|
||||
|
||||
const GITHUB_SECRET = 'super-secret-webhook-key';
|
||||
|
||||
/** Build a sha256= HMAC sig the way GitHub does. */
|
||||
function githubSig(secret: string, payload: Buffer): string {
|
||||
return 'sha256=' + createHmac('sha256', secret).update(payload).digest('hex');
|
||||
}
|
||||
|
||||
/** Strip the GitHub prefix before constant-time hex compare. Matches the
|
||||
* webhook handler in serve-http.ts. Buffer.from('sha256=...', 'hex') silently
|
||||
* truncates at non-hex chars; comparing prefixed strings as hex returns
|
||||
* true-on-anything. The handler strips the prefix; tests must too. */
|
||||
function verifyGithubSig(secret: string, payload: Buffer, headerSig: string): boolean {
|
||||
const prefix = 'sha256=';
|
||||
if (!headerSig.startsWith(prefix)) return false;
|
||||
const expected = createHmac('sha256', secret).update(payload).digest('hex');
|
||||
return safeHexEqual(headerSig.slice(prefix.length), expected);
|
||||
}
|
||||
|
||||
/** Synthetic minimal push payload (real GitHub sends ~100KB; we only read 2 fields). */
|
||||
function pushPayload(repo: string, ref: string): Buffer {
|
||||
return Buffer.from(JSON.stringify({
|
||||
ref,
|
||||
repository: { full_name: repo, name: repo.split('/')[1], owner: { name: repo.split('/')[0] } },
|
||||
head_commit: { id: 'abc123' },
|
||||
}), 'utf8');
|
||||
}
|
||||
|
||||
describe('GitHub HMAC verification', () => {
|
||||
test('valid signature on untampered payload → verify=true', () => {
|
||||
const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main');
|
||||
const sig = githubSig(GITHUB_SECRET, payload);
|
||||
expect(verifyGithubSig(GITHUB_SECRET, payload, sig)).toBe(true);
|
||||
});
|
||||
|
||||
test('IRON-RULE: rejects when signature header lacks sha256= prefix', () => {
|
||||
// This was the bug: production used safeHexEqual on prefixed strings.
|
||||
// Buffer.from('sha256=...', 'hex') silently truncates at non-hex chars,
|
||||
// so both sides decoded to empty buffer and signature_mismatch never fired.
|
||||
const payload = pushPayload('owner/repo', 'refs/heads/main');
|
||||
const expected = createHmac('sha256', GITHUB_SECRET).update(payload).digest('hex');
|
||||
// Without prefix the header is invalid format → reject
|
||||
expect(verifyGithubSig(GITHUB_SECRET, payload, expected)).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects when secret differs', () => {
|
||||
const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main');
|
||||
const goodSig = githubSig(GITHUB_SECRET, payload);
|
||||
expect(verifyGithubSig('wrong-secret', payload, goodSig)).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects when payload differs (single-byte tamper)', () => {
|
||||
const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main');
|
||||
const sig = githubSig(GITHUB_SECRET, payload);
|
||||
const tampered = Buffer.from(payload);
|
||||
tampered[10] = tampered[10] ^ 0xff;
|
||||
expect(verifyGithubSig(GITHUB_SECRET, tampered, sig)).toBe(false);
|
||||
});
|
||||
|
||||
test('GitHub sig format is sha256=<64 hex chars>', () => {
|
||||
const payload = pushPayload('owner/repo', 'refs/heads/main');
|
||||
const sig = githubSig(GITHUB_SECRET, payload);
|
||||
expect(sig).toMatch(/^sha256=[0-9a-f]{64}$/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Push payload parsing', () => {
|
||||
test('extracts repository.full_name + ref', () => {
|
||||
const payload = pushPayload('Garry-s-List/zion-brain', 'refs/heads/main');
|
||||
const parsed = JSON.parse(payload.toString('utf8'));
|
||||
expect(parsed.repository?.full_name).toBe('Garry-s-List/zion-brain');
|
||||
expect(parsed.ref).toBe('refs/heads/main');
|
||||
});
|
||||
|
||||
test('malformed JSON throws (handler returns 400)', () => {
|
||||
const bad = Buffer.from('{not json');
|
||||
expect(() => JSON.parse(bad.toString('utf8'))).toThrow();
|
||||
});
|
||||
|
||||
test('payload missing repository.full_name is detectable', () => {
|
||||
const partial = Buffer.from(JSON.stringify({ ref: 'refs/heads/main' }));
|
||||
const parsed = JSON.parse(partial.toString('utf8'));
|
||||
expect(parsed.repository?.full_name).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Branch ref construction (D5)', () => {
|
||||
test('default tracked_branch = main produces refs/heads/main', () => {
|
||||
const trackedBranch = 'main';
|
||||
expect(`refs/heads/${trackedBranch}`).toBe('refs/heads/main');
|
||||
});
|
||||
|
||||
test('non-main branch is exact match', () => {
|
||||
const trackedBranch: string = 'master';
|
||||
expect(`refs/heads/${trackedBranch}`).toBe('refs/heads/master');
|
||||
// ref="refs/heads/main" against tracked="master" must NOT match
|
||||
const incoming: string = 'refs/heads/main';
|
||||
expect(incoming === `refs/heads/${trackedBranch}`).toBe(false);
|
||||
});
|
||||
|
||||
test('feature-branch push to main-tracking source is rejected by exact-match', () => {
|
||||
const trackedBranch: string = 'main';
|
||||
const pushedRef: string = 'refs/heads/feature/new-stuff';
|
||||
expect(pushedRef === `refs/heads/${trackedBranch}`).toBe(false);
|
||||
});
|
||||
});
|
||||
121
test/sync-trigger-cli.test.ts
Normal file
121
test/sync-trigger-cli.test.ts
Normal file
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* Tests for `gbrain sync trigger` CLI (v0.40 D18).
|
||||
*
|
||||
* Validates the push-trigger entry point:
|
||||
* - Help text renders
|
||||
* - --source <id> required (exits 2)
|
||||
* - --priority invalid (exits 2)
|
||||
* - Non-existent source errors before submit (exits 1)
|
||||
* - Successful submit prints job_id=N on stdout
|
||||
* - Default priority is high (-10)
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSyncTrigger } from '../src/commands/sync.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
/** Capture process.exit and stdout/stderr writes for one runSyncTrigger call. */
|
||||
async function capture(args: string[]): Promise<{
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
exitCode: number | null;
|
||||
}> {
|
||||
const origExit = process.exit;
|
||||
const origLog = console.log;
|
||||
const origErr = console.error;
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
let exitCode: number | null = null;
|
||||
const exitError = new Error('__exit__');
|
||||
process.exit = ((code?: number) => {
|
||||
exitCode = code ?? 0;
|
||||
throw exitError;
|
||||
}) as never;
|
||||
console.log = (...a: unknown[]) => { stdout += a.map(String).join(' ') + '\n'; };
|
||||
console.error = (...a: unknown[]) => { stderr += a.map(String).join(' ') + '\n'; };
|
||||
try {
|
||||
await runSyncTrigger(engine, args);
|
||||
} catch (e) {
|
||||
if (e !== exitError) throw e;
|
||||
} finally {
|
||||
process.exit = origExit;
|
||||
console.log = origLog;
|
||||
console.error = origErr;
|
||||
}
|
||||
return { stdout, stderr, exitCode };
|
||||
}
|
||||
|
||||
describe('runSyncTrigger', () => {
|
||||
test('--help prints usage and returns', async () => {
|
||||
const { stdout, exitCode } = await capture(['--help']);
|
||||
expect(exitCode).toBeNull();
|
||||
expect(stdout).toContain('gbrain sync trigger');
|
||||
expect(stdout).toContain('--source');
|
||||
expect(stdout).toContain('--priority');
|
||||
});
|
||||
|
||||
test('missing --source exits 2 with hint', async () => {
|
||||
const { stderr, exitCode } = await capture([]);
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stderr).toContain('--source <id> is required');
|
||||
});
|
||||
|
||||
test('invalid --priority exits 2', async () => {
|
||||
const { stderr, exitCode } = await capture(['--source', 'default', '--priority', 'urgent']);
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stderr).toContain('Invalid --priority value');
|
||||
});
|
||||
|
||||
test('non-existent source exits 1', async () => {
|
||||
const { stderr, exitCode } = await capture(['--source', 'does-not-exist']);
|
||||
expect(exitCode).toBe(1);
|
||||
expect(stderr).toContain('not found');
|
||||
});
|
||||
|
||||
test('valid trigger submits sync job + prints job_id=N to stdout', async () => {
|
||||
const { stdout, exitCode } = await capture(['--source', 'default']);
|
||||
expect(exitCode).toBeNull();
|
||||
expect(stdout).toMatch(/^job_id=\d+$/m);
|
||||
|
||||
// Verify a sync job exists with auto_embed_backfill + priority -10
|
||||
const queue = new MinionQueue(engine);
|
||||
const jobs = await queue.getJobs({ name: 'sync', limit: 5 });
|
||||
expect(jobs.length).toBe(1);
|
||||
const job = jobs[0];
|
||||
expect(job.priority).toBe(-10);
|
||||
expect((job.data as { sourceId: string }).sourceId).toBe('default');
|
||||
expect((job.data as { auto_embed_backfill: boolean }).auto_embed_backfill).toBe(true);
|
||||
});
|
||||
|
||||
test('--priority normal maps to 0', async () => {
|
||||
const { exitCode } = await capture(['--source', 'default', '--priority', 'normal']);
|
||||
expect(exitCode).toBeNull();
|
||||
const queue = new MinionQueue(engine);
|
||||
const jobs = await queue.getJobs({ name: 'sync', limit: 5 });
|
||||
expect(jobs[0].priority).toBe(0);
|
||||
});
|
||||
|
||||
test('--priority low maps to 5', async () => {
|
||||
const { exitCode } = await capture(['--source', 'default', '--priority', 'low']);
|
||||
expect(exitCode).toBeNull();
|
||||
const queue = new MinionQueue(engine);
|
||||
const jobs = await queue.getJobs({ name: 'sync', limit: 5 });
|
||||
expect(jobs[0].priority).toBe(5);
|
||||
});
|
||||
});
|
||||
52
test/timing-safe.test.ts
Normal file
52
test/timing-safe.test.ts
Normal file
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Tests for src/core/timing-safe.ts (v0.40 D15.5 extraction).
|
||||
*
|
||||
* Pure function: behavior tests plus a source-text grep guard that pins the
|
||||
* extraction so a future drift back to a serve-http-internal closure fails
|
||||
* loudly. Both the admin-cookie compare AND the webhook HMAC compare MUST
|
||||
* route through this helper.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { safeHexEqual } from '../src/core/timing-safe.ts';
|
||||
|
||||
describe('safeHexEqual behavior', () => {
|
||||
test('returns true for identical hex strings', () => {
|
||||
const hash = createHash('sha256').update('the-quick-brown-fox').digest('hex');
|
||||
expect(safeHexEqual(hash, hash)).toBe(true);
|
||||
});
|
||||
|
||||
test('returns false for distinct hex strings of the same length', () => {
|
||||
const a = createHash('sha256').update('apple').digest('hex');
|
||||
const b = createHash('sha256').update('banana').digest('hex');
|
||||
expect(safeHexEqual(a, b)).toBe(false);
|
||||
});
|
||||
|
||||
test('returns false on length mismatch (does NOT throw)', () => {
|
||||
expect(safeHexEqual('ab', 'abcd')).toBe(false);
|
||||
expect(safeHexEqual('', 'ab')).toBe(false);
|
||||
expect(safeHexEqual('abcd', 'ab')).toBe(false);
|
||||
});
|
||||
|
||||
test('empty strings compare equal', () => {
|
||||
expect(safeHexEqual('', '')).toBe(true);
|
||||
});
|
||||
|
||||
test('Buffer.from(_, "hex") is case-insensitive — same hex bytes match either case', () => {
|
||||
// Documents Node's hex-decode behavior: 'abcd' and 'ABCD' decode to the
|
||||
// same bytes, so safeHexEqual returns true. Callers normalize beforehand
|
||||
// when they need strict case-equality.
|
||||
expect(safeHexEqual('abcd', 'ABCD')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('extraction contract', () => {
|
||||
test('IRON-RULE: serve-http.ts imports safeHexEqual from timing-safe.ts (not redefined)', () => {
|
||||
const src = readFileSync('src/commands/serve-http.ts', 'utf8');
|
||||
// Must import from the canonical location
|
||||
expect(src).toMatch(/import\s*\{[^}]*\bsafeHexEqual\b[^}]*\}\s*from\s*['"]\.\.\/core\/timing-safe\.ts['"]/);
|
||||
// Must NOT redefine the function inside the file
|
||||
expect(src).not.toMatch(/function\s+safeHexEqual\s*\(/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user