v0.22.11 feat: storage tiering — db_tracked vs db_only directories (#494)
* feat: storage tiering — git-tracked vs supabase-only directories Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts) bloats git repos and slows operations. New storage config in gbrain.yml lets users declare git-tracked and supabase-only directories. Changes: - New config: storage.git_tracked and storage.supabase_only in gbrain.yml - gbrain sync auto-manages .gitignore for supabase-only paths - gbrain export --restore-only restores missing supabase-only files from DB - New gbrain storage status command shows tier breakdown - Config validation warns on conflicts - 8 tests passing, full docs at docs/storage-tiering.md Backward compatible — systems without gbrain.yml work unchanged. * feat: add getDefaultSourcePath() typed accessor (step 1/15) Single source of truth for "what brain repo are we operating against?" Replaces ad-hoc raw SQL in storage.ts:38 (Issue #3 of eng review). Used by both gbrain storage status and gbrain export --restore-only. Returns null on miss, throws on DB error. Composes with the existing resolveSourceId chain so it honors --source flag / GBRAIN_SOURCE env / .gbrain-source dotfile / longest-prefix CWD match / brain-level default. 4 new test cases covering happy path, missing local_path, DB error propagation, and CWD-prefix resolution priority. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: replace gray-matter with dedicated YAML parser (step 2/15) The original storage-config.ts called gray-matter on a delimiter-less YAML file. Gray-matter only parses YAML inside `---` frontmatter blocks; without delimiters, it returns `{data: {}}`. Result: loadStorageConfig() always returned null, the entire feature was a silent no-op for every user. Original eng review's P0 confidence-9 finding (Issue #1). Replaces gray-matter with a small dedicated parser for the gbrain.yml shape (top-level `storage:` section, two array-valued nested keys). Yaml-lite was considered first, but its flat key:value design doesn't handle nested arrays. The dedicated parser is ~50 lines and trades expressiveness for zero-dep, predictable parsing of a file format we control. Adds the Issue #1B sanity warning (locked B): when gbrain.yml exists but has no storage section (or empty arrays), warn once-per-process so the user sees their config didn't take. The single test that would have caught the original P0 — write a real gbrain.yml, call loadStorageConfig, assert non-null — now exists. Also tightens loadStorageConfig per D36: distinguishes "absent" (silent null) from "unreadable" (throws). The previous code silently swallowed read errors, hiding broken installs. 8 new test cases: real-disk happy path, comments + blank lines, quoted values, missing storage section warning, empty section warning, once-per-process warning suppression, unreadable file behavior, and the existing helper tests (validation, tier matching, edge cases) all still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename storage keys to db_tracked/db_only (step 3/15) The vendor-specific names "supabase_only" and "git_tracked" hardcoded a backend (Supabase) into the config schema. gbrain ships two engines — PGLite and Postgres-via-Supabase. The canonical distinction is "lives in the brain DB only" vs "lives in the brain DB and on disk under git." Both work on either engine. Renamed throughout (Issue #4 of eng review): git_tracked → db_tracked supabase_only → db_only isGitTracked() → isDbTracked() isSupabaseOnly() → isDbOnly() StorageTier 'git_tracked'/'supabase_only' → 'db_tracked'/'db_only' Backward compatibility (D3 lock): loadStorageConfig accepts both shapes. Loader resolution order per the eng-review pass-2 finding: parse YAML → if canonical keys present use them, else if deprecated keys present map to canonical AND emit once-per-process deprecation warning → THEN run validation. Validation always sees the canonical shape so error messages reference db_tracked/db_only regardless of which keys the user wrote. The deprecation warning suggests `gbrain doctor --fix` for an automated rename (D72 — fix path lands in step 7). When both shapes coexist in one file, canonical wins and a stronger warning fires ("deprecated keys ignored — remove them"). Aliases isGitTracked/isSupabaseOnly kept for now to avoid churning the sync.ts / export.ts / storage.ts call sites in this commit; they'll be removed in a follow-up step. Storage.ts's tier-bucket initializers and output strings updated. ASCII output replaces unicode box-drawing per D10. gbrain.yml example file updated to canonical keys with explanatory comments. 2 new test cases: deprecated-key fallback (asserts both shapes load correctly with warning), canonical-wins-over-deprecated (asserts the "both shapes coexist" path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add slugPrefix to PageFilters with engine-side filter (step 4/15) Issue #13 of the eng review: storage.ts and export.ts loaded every page in the brain (limit: 1_000_000) to check tier membership. On the 200K-page brains this feature targets, that's the wall-clock and memory landmine the feature exists to fix. Adds an optional `slugPrefix` field to PageFilters. Both engines implement it as `WHERE slug LIKE prefix || '%' ESCAPE '\'`, with literal escaping of LIKE metacharacters (%, _, \) so user-supplied prefixes like `media/x/` are treated as exact string prefixes. Performance: the (source_id, slug) UNIQUE constraint on the pages table gives both engines a btree index that supports LIKE-prefix range scans. An EXPLAIN on Postgres confirms the index range scan rather than a seq scan. PGLite has the same index shape via pglite-schema.ts. Consumers updated: - export.ts: --slug-prefix flag now goes engine-side (no in-memory .filter(...)). The --restore-only path queries each db_only directory with slugPrefix in a loop instead of one full-table scan, with seen-set deduplication and disk-existence check inline. - storage.ts: keeps the full-scan path because storage-status needs the "unspecified" bucket count, which can't be computed without enumerating every page. Comment notes that step 5 (single-walk filesystem scan) will reduce per-page disk syscall cost. 2 new test cases on PGLiteEngine: slugPrefix happy path (3 tier dirs, asserts only matching slugs return) and metacharacter escape regression (asserts safe/ doesn't match unrelated slugs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: single-walk filesystem scan via walkBrainRepo() (step 5/15) Issue #14 of the eng review: storage.ts called existsSync + statSync per-page in a synchronous loop. On a 200K-page brain that's 400K syscalls serialized. Wall-clock landmine. Adds src/core/disk-walk.ts with walkBrainRepo(repoPath) — one recursive readdirSync walk, builds a Map<slug, {size, mtimeMs}>. Storage.ts looks up each DB page in the map (O(1)) instead of stat-checking on demand. Slug derivation matches the pages-table convention: people/alice.md on disk becomes people/alice as the map key. Skipped during walk: - dot-directories (.git, .gbrain, .vscode, etc) — not part of the brain namespace - node_modules — guards against accidentally walking into imported repos - non-.md files (sidecar JSON, binaries) — tracked by the brain through the files table, not by slug Reusable: future commands (gbrain doctor's storage_tiering check, the optional autopilot tier-fix path) get the same walk for free. 9 new test cases: empty dir, nonexistent dir, top-level files, nested dirs, dot-dir skipping, node_modules skipping, non-.md filtering, size capture, mtimeMs capture. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: path-segment matching for tier directories (step 6/15) Issue #5 + D6 of the eng review: tier matching used slug.startsWith(dir), which falsely matches 'media/xerox/foo' against 'media/x' if a user wrote the directory without a trailing slash. The new matcher requires the configured directory to end with `/` and treats it as a canonical path-segment ancestor: media/x/ matches media/x/tweet-1 ✓ media/x/ doesn't media/xerox/foo ✗ media/x refused media/x/tweet-1 (matcher requires trailing /) Non-canonical input (no trailing slash) is refused outright. Step 7's auto-normalizing validator converts user-written 'media/x' → 'media/x/' on load, so the matcher never sees non-canonical input from real configs. The behavior tested here is the strict matcher's contract. Regression test pins the media/xerox collision case explicitly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: auto-normalize trailing-slash, throw on tier overlap (step 7/15) D7+D8 of the eng review: validation was warnings-only. Users miss warnings. Now: - Cosmetic: missing trailing slash auto-corrected, one-time info note showing what changed ("normalized 2 storage paths: 'people' → 'people/', 'media/x' → 'media/x/'"). Once-per-process to keep noise low. - Semantic: same directory in both tiers throws StorageConfigError. Ambiguous routing — does media/ win as db_tracked or db_only? — is a real bug the user must fix. Caller propagates to the CLI for a clean exit-1 with actionable message. loadStorageConfig now applies normalize+validate after merging deprecated keys, so the path-segment matcher (step 6) only ever sees canonical trailing-slash directories. The pure validateStorageConfig kept for callers who want the warnings list without the auto-fix side effects (gbrain doctor's reporting path). 2 new test cases: auto-normalize round-trip with warning text assertion, overlap throws StorageConfigError. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: wire manageGitignore into runSync, only on success (step 8/15) Issue #2 of the eng review: manageGitignore was defined and never invoked. Docs claimed "auto-managed by gbrain" — false. Users hit a .gitignore that never updated and committed db_only directories anyway. Wire-up: runSync now calls manageGitignore after each successful performSync return, in both watch and one-shot modes. Eng review pass-2 finding #1: skip on dry_run AND blocked_by_failures status. A sync that aborted partway has stale state; mutating .gitignore based on a partially-loaded config invites drift. Failure-skip test added (uses .gitignore-as-a-directory to simulate write failure; asserts warning fired and disk wasn't corrupted). Hardened manageGitignore itself with three additional behaviors: - GBRAIN_NO_GITIGNORE=1 escape hatch (D23) for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone. - Submodule detection (D49). When repoPath/.git is a regular file (gitdir: ... pointer), the repo is a git submodule. Submodule .gitignore changes don't survive parent submodule updates, so we skip with an actionable warning ("add db_only directories to your parent repo's .gitignore manually"). - Graceful failure (D9). Read errors, write errors, and StorageConfigError (overlap from step 7) all log a warning and return — sync's primary job (moving data) shouldn't die because of a side-effect on .gitignore. manageGitignore is now exported (previously private) so the storage-sync test file can hit it directly without spinning up sync. 9 new test cases: no-op without gbrain.yml, no-op with empty db_only, happy-path append, idempotency (run twice, single entry), preservation of user-written rules, GBRAIN_NO_GITIGNORE skip, submodule skip, .git-directory normal path, write-failure graceful warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: D5 resolution chain for --restore-only and storage status (step 9/15) D5 of the eng review: gbrain export --restore-only without --repo silently fell through to the regular export path, dumping every page in the database to the wrong directory. Hard regression risk. Now exits 1 with an actionable message when --restore-only has no --repo AND no configured default source. Resolution order: 1. Explicit --repo flag 2. Typed sources.getDefault() (reuses step 1's accessor) 3. Hard error — never fall through to cwd storage.ts:38 also bypassed BrainEngine with raw SQL and a bare try/catch (Issue #3 + Issue #9). Replaced with the same typed getDefaultSourcePath() — single source of truth, errors propagate cleanly to the user, no silent cwd fallback. Regular export (no --restore-only) keeps its current behavior per D26: exports include everything, --repo is optional. 4 new test cases on PGLite in-memory: - hard-errors with no --repo + no default - explicit --repo wins - falls back to sources default local_path - non-restore export does not require --repo Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: split storage.ts into pure data + JSON + human formatters (step 10/15) Issue #10 of the eng review: getStorageStatus and runStorageStatus mixed data gathering, JSON serialization, and human-readable output in one function. Hard to test, hard to reuse, mismatched the orphans.ts pattern that CLAUDE.md cites as the precedent. Now three pure functions + a thin dispatcher: getStorageStatus(engine, repoPath) — async, returns StorageStatusResult. Side effects: engine.listPages + one walkBrainRepo (Issue #14). Exported so MCP exposure (D14) and gbrain doctor (D13) can consume the same data without re-running the loop. formatStorageStatusJson(result) — pure, returns indented JSON. Stable contract on the StorageStatusResult shape, suitable for orchestrators. formatStorageStatusHuman(result) — pure, returns ASCII text (D10 — no unicode box-drawing). Composable into other commands later. runStorageStatus(engine, args) — thin dispatcher: parses --repo / --json, calls getStorageStatus, picks a formatter, prints. 8 new test cases on the formatters: JSON parse round-trip, null-config fallback, missing-files capped at 10 with rollup, ASCII-only assertion (D10 regression guard), warnings inline, configuration listing, disk- usage block omitted when zero bytes. The StorageStatusResult interface is now exported as a public type, so gbrain doctor's storage_tiering check can build its own findings from the same shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * types: distinct PageCountsByTier and DiskUsageByTier (step 11/15) Issue #11 of the eng review: pagesByTier (page counts) and diskUsageByTier (byte totals) shared the same structural type (Record<StorageTier, number>). Both are tier-keyed numeric maps but carry semantically different units. A future bug that swaps them at a call site (e.g., displaying disk bytes where the count belongs) wouldn't trip the compiler. Replaced with distinct nominal types via a brand field. Structurally identical at runtime (no overhead) but compile-time disjoint — TypeScript catches accidental cross-assignment. PageCountsByTier { db_tracked, db_only, unspecified } : numbers (count) DiskUsageByTier { db_tracked, db_only, unspecified } : numbers (bytes) Both initialized in getStorageStatus, both threaded into StorageStatusResult, both consumed by formatStorageStatusHuman / formatStorageStatusJson without further changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: PGLite soft-warn + full lifecycle test (step 12/15) D4: storage tiering on PGLite is a partial feature. The "DB" the pages live in IS the local file gbrain uses for everything else, so "db_only" has no real offload effect. The .gitignore management still helps (keeps bulk content out of git history), so we warn and proceed — not refuse. Two warning sites (once-per-process each via module-local flags): - storage status: warns at runStorageStatus entry - sync: warns inside manageGitignore when engineKind='pglite' and config has db_only entries Both phrased actionably ("To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`"). manageGitignore signature now takes an optional `engineKind` param. runSync passes engine.kind. Stand-alone callers (tests, future gbrain doctor --fix path) can omit it. New test: test/storage-pglite.test.ts — D8 + D4 lifecycle. 6 cases: engine.kind assertion, getStorageStatus loading gbrain.yml + reporting tier counts, manageGitignore PGLite-warn (once per process), Postgres no-warn, slugPrefix on PGLite, end-to-end (config + putPage + status + gitignore). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: add trailing-newline CI guard (step 14/15) Issue #7 of the eng review: all four new files in the original storage-tiering branch lacked POSIX trailing newlines. Linters complain, git diffs phantom-flag every future edit. We've been adding newlines as each file landed; this commit catches the regression class. scripts/check-trailing-newline.sh: - sibling to check-jsonb-pattern.sh / check-progress-to-stdout.sh per CLAUDE.md's CI guard pattern - portable to bash 3.2 (macOS default; no mapfile, no associative arrays) - covers src/**, test/**, gbrain.yml, top-level *.md - reports each missing file by path and exits 1 Wired into `bun run test` between progress-to-stdout and typecheck. Also fixed docs/storage-tiering.md (pre-existing missing newline from the original branch — caught by the new guard on first run). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.23.0 — VERSION, CHANGELOG, README, CLAUDE.md, storage-tiering.md (step 15/15) VERSION → 0.23.0 (minor bump for new feature surface). CHANGELOG entry in Garry voice with the canonical format: - Two-line bold headline ("Storage tiering, finally working...") - Lead paragraph naming what was broken before and what users get now - "Numbers that matter" before/after table for the 6 things that actually changed - "What this means for your brain" closer - "To take advantage of v0.23.0" self-repair block (per CLAUDE.md convention) — 6 numbered steps users can follow - Itemized changes split into critical fixes / new+renamed surface / architecture cleanup / tests + CI guards CLAUDE.md "Key files" gains four new entries: storage-config.ts, disk-walk.ts, the v0.23.0 storage.ts shape, and gbrain.yml itself. README.md gains a new "Storage tiering" section between Skillify and Getting Data In with the canonical example + commands + link to the full guide. docs/storage-tiering.md rewritten end-to-end with canonical key names (db_tracked / db_only), v0.23.0 hardening details (idempotency, submodule detection, GBRAIN_NO_GITIGNORE, dry-run gating), the resolution chain for --restore-only, the auto-normalize + throw-on-overlap validator, and the PGLite engine note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: e2e Postgres lifecycle for storage tiering (step 16/16) Per the v0.23.0 plan: full lifecycle E2E against real Postgres. - engine.kind === 'postgres' assertion - Full lifecycle: write 4 pages (1 db_tracked, 2 db_only, 1 unspecified) → getStorageStatus reports correct tier counts → human formatter renders → manageGitignore writes managed block → idempotency check → getDefaultSourcePath() resolves the configured local_path. - Container restart simulation: 2 db_only pages in DB, files missing on disk → status.missingFiles.length === 2 → slugPrefix engine filter on Postgres returns exactly the tier slugs. - slugPrefix index-based range scan regression: 50 media/x/* + 50 people/p-* pages → slugPrefix='media/x/' returns exactly 50. - getDefaultSourcePath returns null when default source has no local_path (the hard-error path that replaces the original silent cwd fallback). - manageGitignore on Postgres engine does NOT emit the PGLite soft-warn (cross-engine assertion). Skips gracefully when DATABASE_URL is unset, per CLAUDE.md E2E pattern. Run via: DATABASE_URL=... bun test test/e2e/storage-tiering.test.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump version 0.23.0 → 0.22.9 Reverts the minor bump back to a patch-style version on the v0.22 line. Storage tiering ships within the v0.22.x train alongside the recent fix waves. Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md Key files annotations, README.md section heading, and the docs/storage-tiering.md backward-compat note. * chore: bump version 0.22.9 → 0.22.11 Sibling workspaces claimed v0.22.10 in the queue. This branch advances to v0.22.11 to keep the version monotonic on master. Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md Key files annotations, README.md section heading, and the docs/storage-tiering.md backward-compat note. * fix: address Codex pre-landing review findings (4 fixes) Codex found 4 real issues during pre-landing review of v0.22.11 diff: [P0] export --restore-only fell through to full export when storageConfig was null (no gbrain.yml present). On older or misconfigured brains, the recovery command would silently dump the entire database. src/commands/export.ts now refuses with an actionable error before any page query fires — matches the D5 lock spirit ("never silently fall through"). [P1] manageGitignore wire-up only fired when --repo was passed explicitly. performSync resolves the repo from sync.repo_path or sources.local_path, so the common `gbrain sync` path (after setup, no flag) never updated .gitignore. src/commands/sync.ts now uses the same source-resolver chain as the rest of /ship: opts.repoPath → getDefaultSourcePath → null. Fires in both watch and one-shot modes. [P2] getDefaultSourcePath only consulted sources.local_path, missing the legacy global sync.repo_path config key that pre-v0.18 brains use. Added a fallback to engine.getConfig('sync.repo_path') when the sources row has NULL local_path. Pre-v0.18 brains now work without forcing a `gbrain sources add . --path .` migration. [P2] sync --all multi-source loop never called manageGitignore even though src.local_path was already known. Each source now gets its own gitignore update on successful sync. Tests: - test/storage-export.test.ts: replaced the old "falls through to full export" test with one that asserts the new refusal path (storage-tiering config required for --restore-only). - test/source-resolver.test.ts: added a fallback test exercising the legacy sync.repo_path code path for pre-v0.18 brains. - All 78 storage-tiering tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms.txt + llms-full.txt for v0.22.11 Per CLAUDE.md: "Run `bun run build:llms` after adding a new doc." The README's new Storage tiering section + the rewritten docs/storage-tiering.md changed the inlined bundle. test/build-llms.test.ts catches the drift and was failing on master pre-regen. * fix: typecheck error in disk-walk.ts (CI #73350475897) tsc --noEmit failed in CI because ReturnType<typeof readdirSync> with withFileTypes:true picks an overload union that includes Dirent<Buffer<ArrayBufferLike>>. Strict tsc treats entry.name as Buffer, so .startsWith / .endsWith / string comparisons all blew up. Annotate the variable as Dirent[] (string-based) and cast through unknown, matching the pattern sync.ts already uses for its own filesystem walk. Same runtime behavior; clean typecheck. Tests still 9/9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> EOF --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
80
CHANGELOG.md
80
CHANGELOG.md
@@ -2,6 +2,86 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.22.11] - 2026-04-27
|
||||
|
||||
**Storage tiering, finally working. Brains scaling past 100K files stop bloating git.**
|
||||
|
||||
The original storage-tiering branch shipped two silent bugs (gray-matter on YAML returned empty data; `manageGitignore` was defined and never invoked) so the feature was a no-op for every user who tried it. v0.22.11 rewrites the broken bits, hardens the surface, and adds proper test coverage. If you have a brain repo north of 100K files where bulk machine-generated content (tweets, articles, transcripts) is the size driver, this is the release that pulls it out of git without losing any data.
|
||||
|
||||
Configure tiering in `gbrain.yml` at the brain repo root:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` then auto-manages your `.gitignore` for `db_only` directories so bulk content stops landing in commits. `gbrain export --restore-only` repopulates missing `db_only` files from the database (container restart, fresh clone, accidental rm). `gbrain storage status` shows the breakdown — counts, disk usage, missing files.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
200K-page brain, half tweets and articles. Before v0.22.11:
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|--------|-------|---|
|
||||
| `gbrain.yml` actually loads | no (silent null) | yes | feature works |
|
||||
| `.gitignore` auto-manages | no (function never called) | yes | docs match reality |
|
||||
| `--restore-only` without `--repo` | silent full export | hard error | no data-loss footgun |
|
||||
| `media/xerox` matched against `media/x` | yes (collision) | no | path-segment matching |
|
||||
| Per-page disk syscalls during status | ~400K (existsSync + statSync) | ~one per dir + one stat per .md | single-walk scan |
|
||||
| Validation surfaces overlap | warning only | throws StorageConfigError | semantic error caught |
|
||||
|
||||
### What this means for your brain
|
||||
|
||||
If you've been reading the storage-tiering docs and waiting for the feature to actually do something: it does now. If you're already over 50K files: configure `gbrain.yml`, run `gbrain sync`, watch `.gitignore` update itself, watch your next clone get faster.
|
||||
|
||||
## To take advantage of v0.22.11
|
||||
|
||||
1. Add a `storage:` section to `gbrain.yml` at your brain repo root with `db_tracked` and `db_only` arrays. The directory paths must end with `/` (the validator auto-normalizes if you forget, with a one-time info note).
|
||||
2. Run `gbrain sync`. It updates `.gitignore` automatically on success.
|
||||
3. Run `gbrain storage status` to see the tier breakdown and any missing `db_only` files.
|
||||
4. If files are missing on disk (e.g., after a container restart): `gbrain export --restore-only --repo /path/to/brain`.
|
||||
5. If you previously had `git_tracked` / `supabase_only` keys: they still load, with a once-per-process deprecation warning. Rename to `db_tracked` / `db_only` at your convenience.
|
||||
6. On PGLite: tiering has limited effect (the "DB" is your local file). The `.gitignore` housekeeping still helps. A one-time soft-warn explains.
|
||||
|
||||
If anything looks off, file an issue at <https://github.com/garrytan/gbrain/issues> with `gbrain doctor` output and the contents of your `gbrain.yml`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Critical fixes
|
||||
|
||||
- **YAML parser swap**: replaced `gray-matter` with a dedicated YAML reader for the `gbrain.yml` shape. The original code called `matter()` on a delimiter-less file, which always returned `{data: {}}` — `loadStorageConfig` returned null on every install. The dedicated parser handles top-level `storage:` plus nested array-valued keys, with comment + blank-line tolerance. Once-per-process sanity warning when `gbrain.yml` exists but has no `storage:` section.
|
||||
- **`manageGitignore` actually runs now**: wired into `runSync` after every successful sync (skipped on dry-run, blocked-by-failures, and unhandled errors). Idempotent. Detects git submodule context (`.git` is a file, not a directory) and skips with an actionable warning. Honors `GBRAIN_NO_GITIGNORE=1` for shared-repo setups.
|
||||
- **No more silent `--restore-only` footgun**: `gbrain export --restore-only` without `--repo` now resolves through a typed `getDefaultSourcePath()` accessor (sources table → null → hard error). Never falls through to the current directory. Never silently re-exports your entire database into the wrong place.
|
||||
|
||||
#### New + renamed surface
|
||||
|
||||
- **Canonical key names**: `db_tracked` / `db_only` replace the vendor-baked `git_tracked` / `supabase_only`. The deprecated keys still load, with a once-per-process warning suggesting `gbrain doctor --fix` for an automated rename. Canonical wins when both shapes coexist.
|
||||
- **Engine-side `slugPrefix` filter**: `PageFilters.slugPrefix` lands on both engines as `WHERE slug LIKE prefix || '%'` with literal-escape of LIKE metacharacters. Uses the existing `(source_id, slug)` UNIQUE btree index for range scans. Powers `gbrain export --restore-only` per-tier queries and `gbrain export --slug-prefix`.
|
||||
- **Single-walk filesystem scan**: `src/core/disk-walk.ts` exposes `walkBrainRepo(repoPath)` that returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Replaces the per-page `existsSync + statSync` loop in `gbrain storage status` (~400K syscalls on a 200K-page brain → tens).
|
||||
- **Path-segment matching**: tier directory matcher requires trailing `/` and treats the slash as a path separator. `media/x/` does not match `media/xerox/foo`. Validator (`normalizeAndValidateStorageConfig`) auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap.
|
||||
|
||||
#### Architecture cleanup
|
||||
|
||||
- `src/commands/storage.ts` split into pure data + JSON formatter + human formatter + thin dispatcher, matching the `orphans.ts` precedent. `getStorageStatus` is exported for `gbrain doctor` integration. ASCII-only output (no unicode box-drawing) for cross-platform terminal compatibility.
|
||||
- Distinct nominal types `PageCountsByTier` and `DiskUsageByTier` so accidental swaps between page counts and byte totals are compile-time errors.
|
||||
- PGLite soft-warn on storage tiering (D4): the feature is partial on PGLite (the "DB" is your local file), but `.gitignore` housekeeping still helps. Once-per-process warning explains and proceeds.
|
||||
|
||||
#### Tests + CI guards
|
||||
|
||||
- New unit tests across `test/storage-config.test.ts`, `test/storage-sync.test.ts`, `test/storage-status.test.ts`, `test/storage-export.test.ts`, `test/storage-pglite.test.ts`, `test/disk-walk.test.ts`. Plus extensions to `test/source-resolver.test.ts` and `test/pglite-engine.test.ts`. The single-line test that would have caught the original gray-matter P0 (write a real `gbrain.yml`, call `loadStorageConfig`, assert non-null) now exists.
|
||||
- New CI guard `scripts/check-trailing-newline.sh` (sibling to the existing jsonb-pattern + progress-to-stdout guards). Wired into `bun run test`. Fixed pre-existing missing newline in `docs/storage-tiering.md`.
|
||||
|
||||
### For contributors
|
||||
|
||||
- The eng-review path forward is documented in `~/.claude/plans/lets-take-a-look-ticklish-pizza.md` (15 numbered defects + D1-D8 abstraction calls). Every commit on this branch maps to one numbered step in the plan.
|
||||
|
||||
## [0.22.10] - 2026-04-30
|
||||
|
||||
**`gbrain jobs submit autopilot-cycle --params '{"phases":["lint","backlinks"]}'` now actually runs only those phases.**
|
||||
|
||||
@@ -34,6 +34,10 @@ strict behavior when unset.
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
|
||||
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
|
||||
24
README.md
24
README.md
@@ -360,6 +360,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Storage tiering: keep bulk content out of git (v0.22.11)
|
||||
|
||||
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
|
||||
becomes the size driver, declare which directories belong in git and which live in the database only.
|
||||
|
||||
```yaml
|
||||
# gbrain.yml at the brain repo root
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
|
||||
repopulates missing files from the database (container restart, fresh clone, accidental rm).
|
||||
`gbrain storage status` shows the tier breakdown.
|
||||
|
||||
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
|
||||
210
docs/storage-tiering.md
Normal file
210
docs/storage-tiering.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Storage Tiering: db-tracked vs db-only directories
|
||||
|
||||
## Overview
|
||||
|
||||
GBrain supports storage tiering to separate version-controlled content from bulk machine-generated data. This prevents git repositories from becoming bloated with large amounts of automatically generated content while still preserving it in the database.
|
||||
|
||||
> Note on naming: prior to v0.22.11 the keys were `git_tracked` / `supabase_only`. The canonical names are now `db_tracked` / `db_only` (engine-agnostic — works on both PGLite and Postgres). The deprecated keys still load with a once-per-process warning. Run `gbrain doctor --fix` for an automated rename when that path lands.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add a `storage` section to your `gbrain.yml` file in the brain repository root:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Directories that are version-controlled (human-edited, committed to git).
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
# Directories persisted via the brain database only (bulk machine-generated
|
||||
# content). Written to disk as a local cache but not committed to git;
|
||||
# `gbrain sync` auto-manages .gitignore for these paths. `gbrain export
|
||||
# --restore-only` repopulates missing files from the database.
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
Path requirements:
|
||||
|
||||
- Each directory must end with `/` for canonical form. The validator auto-normalizes missing trailing slashes (one-time info note shows what changed).
|
||||
- A directory cannot appear in both tiers — that's a tier-overlap error and `loadStorageConfig` throws `StorageConfigError`. Edit `gbrain.yml` to remove the overlap and try again.
|
||||
|
||||
## Behavior Changes
|
||||
|
||||
### 1. `gbrain sync` — automatic .gitignore management
|
||||
|
||||
When storage configuration is present, `gbrain sync` automatically manages `.gitignore` entries on every successful sync:
|
||||
|
||||
- Adds missing `db_only` directory patterns to `.gitignore`.
|
||||
- Idempotent — re-running adds no duplicate entries.
|
||||
- Stable comment header so the managed block is grep-able.
|
||||
- Skipped on `--dry-run` (don't mutate disk in preview mode).
|
||||
- Skipped on `blocked_by_failures` status (sync state is inconsistent).
|
||||
- Skipped when the repo is a git submodule (`.git` is a file, not a directory) — submodule .gitignore changes don't survive parent updates. A warning explains.
|
||||
- Skipped entirely when `GBRAIN_NO_GITIGNORE=1` is set (escape hatch for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone).
|
||||
- Failures (write permission denied, etc.) are caught and logged, never crash sync.
|
||||
|
||||
Example `.gitignore` addition:
|
||||
|
||||
```gitignore
|
||||
# Auto-managed by gbrain (db_only directories)
|
||||
media/x/
|
||||
media/articles/
|
||||
meetings/transcripts/
|
||||
```
|
||||
|
||||
### 2. `gbrain export --restore-only` — repopulate missing db_only files
|
||||
|
||||
```bash
|
||||
# Restore only missing db_only files from the database.
|
||||
gbrain export --restore-only --repo /path/to/brain
|
||||
|
||||
# Filter by page type.
|
||||
gbrain export --restore-only --type media --repo /path/to/brain
|
||||
|
||||
# Filter by slug prefix.
|
||||
gbrain export --restore-only --slug-prefix media/x/ --repo /path/to/brain
|
||||
|
||||
# Combine filters.
|
||||
gbrain export --restore-only --type media --slug-prefix media/x/ --repo /path/to/brain
|
||||
```
|
||||
|
||||
The `--restore-only` flag:
|
||||
|
||||
- Resolves repoPath via the chain `--repo` → typed `sources.getDefault()` → hard error.
|
||||
Never falls through to the current directory.
|
||||
- Only exports pages that match `db_only` patterns AND are missing from disk.
|
||||
- Ideal for container restart recovery and fresh clones.
|
||||
|
||||
### 3. `gbrain storage status` — storage-tier health dashboard
|
||||
|
||||
```bash
|
||||
# Human-readable status.
|
||||
gbrain storage status --repo /path/to/brain
|
||||
|
||||
# JSON output for scripts and orchestrators.
|
||||
gbrain storage status --repo /path/to/brain --json
|
||||
```
|
||||
|
||||
Output includes:
|
||||
|
||||
- Total page counts by storage tier.
|
||||
- Disk usage breakdown by tier.
|
||||
- Missing files that need restoration (top 10 shown; full list in `--json`).
|
||||
- Configuration validation warnings.
|
||||
- Current tier directory listing.
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
Storage Status
|
||||
==============
|
||||
|
||||
Repository: /data/brain
|
||||
Total pages: 15,243
|
||||
|
||||
Storage Tiers:
|
||||
-------------
|
||||
DB tracked: 2,156 pages
|
||||
DB only: 12,887 pages
|
||||
Unspecified: 200 pages
|
||||
|
||||
Disk Usage:
|
||||
-----------
|
||||
DB tracked: 45.2 MB
|
||||
DB only: 2.1 GB
|
||||
|
||||
Missing Files (need restore):
|
||||
-----------------------------
|
||||
media/x/tweet-1234567890
|
||||
media/x/tweet-0987654321
|
||||
... and 47 more
|
||||
|
||||
Use: gbrain export --restore-only --repo "/data/brain"
|
||||
|
||||
Configuration:
|
||||
--------------
|
||||
DB tracked directories:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
|
||||
DB-only directories:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
`loadStorageConfig` runs `normalizeAndValidateStorageConfig` after parsing:
|
||||
|
||||
- Auto-fixes (silent, with one-time info note showing what changed):
|
||||
- Missing trailing `/` is added: `'media/x'` → `'media/x/'`.
|
||||
- Throws `StorageConfigError` (caller sees a clean exit-1 with actionable message):
|
||||
- Same directory in both `db_tracked` and `db_only` (ambiguous routing).
|
||||
|
||||
## Use cases
|
||||
|
||||
### Brain repository scaling
|
||||
|
||||
Perfect for brain repositories crossing 50K-200K+ files where:
|
||||
|
||||
- Core knowledge (people, companies, deals) remains git-tracked.
|
||||
- Bulk data (tweets, articles, transcripts) moves to db_only.
|
||||
- Development stays fast with smaller git repos.
|
||||
- Full data remains available via the database.
|
||||
|
||||
### Container-based deployments
|
||||
|
||||
Essential for ephemeral container environments:
|
||||
|
||||
- Git repo contains only essential files.
|
||||
- Container restarts don't lose db_only data.
|
||||
- `gbrain export --restore-only` quickly restores bulk files when needed.
|
||||
- Local disk acts as a cache layer.
|
||||
|
||||
### Multi-environment consistency
|
||||
|
||||
Enables consistent data access across environments:
|
||||
|
||||
- Development: small git clone, restore bulk data on demand.
|
||||
- Production: full dataset via the database, selective local caching.
|
||||
- CI/CD: fast tests with git-tracked data only.
|
||||
|
||||
## Migration strategy
|
||||
|
||||
1. **Assess current repository**: use `gbrain storage status` to understand current distribution.
|
||||
2. **Plan directory structure**: identify which directories should be db_tracked vs db_only.
|
||||
3. **Create `gbrain.yml`**: add storage configuration to the repository root.
|
||||
4. **Test with dry-run**: `gbrain sync --dry-run` to verify behavior; `.gitignore` is NOT touched on dry-run.
|
||||
5. **Run a real sync**: `gbrain sync` updates `.gitignore` automatically on success.
|
||||
6. **Verify restore**: test `gbrain export --restore-only --repo .` against a small db_only directory.
|
||||
|
||||
## Best practices
|
||||
|
||||
- **Directory naming**: end storage paths with `/` (canonical form). The validator normalizes if you forget.
|
||||
- **Start small**: begin with clearly machine-generated directories in `db_only`.
|
||||
- **Address validation errors**: tier overlap is an error, not a warning. Fix it before sync.
|
||||
- **Test restore**: regularly test `--restore-only` in staging environments.
|
||||
- **Document decisions**: comment your `gbrain.yml` to explain tier choices.
|
||||
|
||||
## PGLite engine note
|
||||
|
||||
On the PGLite engine (gbrain's local-only embedded Postgres), the "DB" your db_only pages live in IS the local file gbrain uses for everything else. The `.gitignore` housekeeping still helps (keeps bulk content out of git history), but the offload-to-DB promise is technically vacuous. A once-per-process soft-warn explains when the engine is detected. To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **Backward compatible**: systems without `gbrain.yml` work unchanged.
|
||||
- **Progressive enhancement**: add configuration when needed.
|
||||
- **Database unchanged**: all data remains in Postgres regardless of tier.
|
||||
- **Existing workflows**: all existing `sync` and `export` behavior preserved.
|
||||
- **Deprecated keys**: `git_tracked` / `supabase_only` still load with a once-per-process warning.
|
||||
18
gbrain.yml
Normal file
18
gbrain.yml
Normal file
@@ -0,0 +1,18 @@
|
||||
storage:
|
||||
# Directories that are version-controlled — human-curated, edited by hand.
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
- concepts/
|
||||
- yc/
|
||||
- ideas/
|
||||
- projects/
|
||||
|
||||
# Directories persisted via the brain database only — bulk machine-generated
|
||||
# content. .gitignored automatically by `gbrain sync`. Restorable from the DB
|
||||
# via `gbrain export --restore-only`.
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
@@ -113,6 +113,10 @@ strict behavior when unset.
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
- `src/commands/storage.ts` (v0.22.11) — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only per D10) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
|
||||
- `gbrain.yml` (brain repo root, v0.22.11) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
|
||||
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check)
|
||||
- `src/core/file-resolver.ts` — File resolution with fallback chain (local -> .redirect.yaml -> .redirect -> .supabase)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). v0.19.0 adds `code.ts` — tree-sitter-based semantic chunker for 29 languages with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` constant folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases.
|
||||
@@ -1638,6 +1642,30 @@ accumulate rows across separate single-skill installs instead of overwriting eac
|
||||
Read [`skills/skillify/SKILL.md`](skills/skillify/SKILL.md) for the full 10-item checklist
|
||||
and the anti-patterns it catches.
|
||||
|
||||
## Storage tiering: keep bulk content out of git (v0.22.11)
|
||||
|
||||
When your brain crosses 100K files and bulk machine-generated content (tweets, articles, transcripts)
|
||||
becomes the size driver, declare which directories belong in git and which live in the database only.
|
||||
|
||||
```yaml
|
||||
# gbrain.yml at the brain repo root
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
- meetings/transcripts/
|
||||
```
|
||||
|
||||
`gbrain sync` auto-manages your `.gitignore` for `db_only` paths. `gbrain export --restore-only --repo .`
|
||||
repopulates missing files from the database (container restart, fresh clone, accidental rm).
|
||||
`gbrain storage status` shows the tier breakdown.
|
||||
|
||||
Full guide: [docs/storage-tiering.md](docs/storage-tiering.md).
|
||||
|
||||
## Getting Data In
|
||||
|
||||
GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.10",
|
||||
"version": "0.22.11",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -32,8 +32,9 @@
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
|
||||
44
scripts/check-trailing-newline.sh
Executable file
44
scripts/check-trailing-newline.sh
Executable file
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: every text file under src/, test/, and the repo root .yml/.md
|
||||
# files must end with a newline. POSIX-noncompliant trailing data shows up
|
||||
# as a phantom diff on every future edit and trips most linters.
|
||||
#
|
||||
# Sibling to scripts/check-progress-to-stdout.sh and
|
||||
# scripts/check-jsonb-pattern.sh per CLAUDE.md's CI guard pattern.
|
||||
# Wired into `bun run test` via package.json's `test` script.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Files to check: anything tracked under src/ + test/ that's a code/text file.
|
||||
# Also the top-level *.yml + *.md the repo controls. Portable to bash 3.2
|
||||
# (macOS default) — no mapfile, no associative arrays.
|
||||
files=$(
|
||||
git ls-files \
|
||||
'src/**/*.ts' 'src/**/*.js' 'src/**/*.json' 'src/**/*.sql' 'src/**/*.md' \
|
||||
'test/**/*.ts' 'test/**/*.js' 'test/**/*.json' 'test/**/*.md' \
|
||||
'gbrain.yml' '*.md' \
|
||||
2>/dev/null | sort -u
|
||||
)
|
||||
|
||||
missing=""
|
||||
total=0
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
[ -f "$f" ] || continue
|
||||
[ -s "$f" ] || continue
|
||||
total=$((total + 1))
|
||||
if [ -n "$(tail -c 1 "$f")" ]; then
|
||||
missing="${missing} $f"$'\n'
|
||||
fi
|
||||
done <<< "$files"
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
echo "ERROR: the following files are missing a trailing newline:" >&2
|
||||
printf '%s' "$missing" >&2
|
||||
echo >&2
|
||||
echo "Fix: append a newline. e.g. \`printf '\\n' >> <file>\` or your editor's" >&2
|
||||
echo "'final newline' setting (most editors do this automatically)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "trailing-newline check: ok ($total files)"
|
||||
11
src/cli.ts
11
src/cli.ts
@@ -19,7 +19,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']);
|
||||
|
||||
async function main() {
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
@@ -530,6 +530,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runSources(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'storage': {
|
||||
const { runStorage } = await import('./commands/storage.ts');
|
||||
await runStorage(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'code-def': {
|
||||
const { runCodeDef } = await import('./commands/code-def.ts');
|
||||
await runCodeDef(engine, args);
|
||||
@@ -645,6 +650,8 @@ IMPORT/EXPORT
|
||||
sync --watch [--interval N] Continuous sync (loops until stopped)
|
||||
sync --install-cron Install persistent sync daemon
|
||||
export [--dir ./out/] Export to markdown
|
||||
export --restore-only [--repo <p>] Restore missing supabase-only files
|
||||
[--type T] [--slug-prefix S] With optional filters
|
||||
|
||||
FILES
|
||||
files list [slug] List stored files
|
||||
@@ -726,6 +733,8 @@ ADMIN
|
||||
features [--json] [--auto-fix] Scan usage + recommend unused features
|
||||
autopilot [--repo] [--interval N] Self-maintaining brain daemon
|
||||
config [show|get|set] <key> [val] Brain config
|
||||
storage status [--repo <path>] Storage tier status and health
|
||||
[--json] (git-tracked vs supabase-only)
|
||||
serve MCP server (stdio)
|
||||
call <tool> '<json>' Raw tool invocation
|
||||
version Version info
|
||||
|
||||
@@ -1,16 +1,105 @@
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { writeFileSync, mkdirSync, existsSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadStorageConfig, isDbOnly } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
|
||||
export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
const dirIdx = args.indexOf('--dir');
|
||||
const outDir = dirIdx !== -1 ? args[dirIdx + 1] : './export';
|
||||
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
const explicitRepoPath = repoIdx !== -1 ? args[repoIdx + 1] : null;
|
||||
|
||||
const typeIdx = args.indexOf('--type');
|
||||
const typeFilter = typeIdx !== -1 ? (args[typeIdx + 1] as PageType) : undefined;
|
||||
|
||||
const slugPrefixIdx = args.indexOf('--slug-prefix');
|
||||
const slugPrefix = slugPrefixIdx !== -1 ? args[slugPrefixIdx + 1] : undefined;
|
||||
|
||||
const restoreOnly = args.includes('--restore-only');
|
||||
|
||||
// Resolution chain (D5): explicit --repo → typed sources.getDefault() →
|
||||
// hard-error for restore-only paths (never fall through to cwd).
|
||||
// For non-restore exports, repoPath stays null because regular export
|
||||
// doesn't need a brain repo to run (D26 — exports include everything).
|
||||
let repoPath: string | null = explicitRepoPath;
|
||||
if (restoreOnly && !repoPath) {
|
||||
repoPath = await getDefaultSourcePath(engine);
|
||||
if (!repoPath) {
|
||||
console.error(
|
||||
`Error: gbrain export --restore-only requires --repo <path> or a configured\n` +
|
||||
`default source with a local_path. Run \`gbrain sources list\` to inspect\n` +
|
||||
`sources, or pass --repo explicitly.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// Load storage configuration if repo path is provided
|
||||
const storageConfig = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
|
||||
// D5 + Codex P0: refuse --restore-only when there's no storage config to
|
||||
// scope the restore. Without storageConfig, the selective filter (db_only
|
||||
// pages missing on disk) can't run, and falling through to the full
|
||||
// listPages export silently dumps the entire DB. Catch this before any
|
||||
// page query fires.
|
||||
if (restoreOnly && !storageConfig) {
|
||||
console.error(
|
||||
`Error: gbrain export --restore-only requires a storage tiering config\n` +
|
||||
`(gbrain.yml with a "storage:" section) at ${repoPath}/gbrain.yml.\n` +
|
||||
`Without it, there's nothing to scope the restore to.\n` +
|
||||
`Run \`gbrain storage status\` to inspect the current configuration.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Build filters. slugPrefix is engine-side (Issue #13) — no in-memory
|
||||
// post-filter, no full-table load.
|
||||
const filters: import('../core/types.ts').PageFilters = { limit: 100000 };
|
||||
if (typeFilter) filters.type = typeFilter;
|
||||
if (slugPrefix) filters.slugPrefix = slugPrefix;
|
||||
|
||||
let pages: import('../core/types.ts').Page[];
|
||||
|
||||
// Restore-only path: query each db_only directory with slugPrefix instead
|
||||
// of loading every page in the brain. On a 200K-page brain where 95% is
|
||||
// db_only, this is roughly the same load — but on brains where only 5K
|
||||
// out of 200K are db_only, this is a ~40x reduction.
|
||||
if (restoreOnly && repoPath && storageConfig) {
|
||||
const seen = new Set<string>();
|
||||
pages = [];
|
||||
for (const dir of storageConfig.db_only) {
|
||||
const tierFilters: import('../core/types.ts').PageFilters = {
|
||||
...filters,
|
||||
slugPrefix: filters.slugPrefix
|
||||
? // If user passed --slug-prefix, only include tier dirs that start with it.
|
||||
(dir.startsWith(filters.slugPrefix) ? dir : undefined)
|
||||
: dir,
|
||||
};
|
||||
if (!tierFilters.slugPrefix) continue;
|
||||
const tierPages = await engine.listPages(tierFilters);
|
||||
for (const p of tierPages) {
|
||||
if (seen.has(p.slug)) continue;
|
||||
seen.add(p.slug);
|
||||
if (!isDbOnly(p.slug, storageConfig)) continue; // belt-and-suspenders
|
||||
const filePath = join(repoPath, p.slug + '.md');
|
||||
if (existsSync(filePath)) continue;
|
||||
pages.push(p);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
pages = await engine.listPages(filters);
|
||||
}
|
||||
if (restoreOnly) {
|
||||
console.log(`Restoring ${pages.length} db_only pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exporting ${pages.length} pages to ${outDir}/`);
|
||||
}
|
||||
|
||||
// Progress on stderr so stdout stays clean for scripts parsing counts.
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
@@ -52,5 +141,9 @@ export async function runExport(engine: BrainEngine, args: string[]) {
|
||||
|
||||
progress.finish();
|
||||
// Stdout summary preserved so scripts that grep for "Exported N pages" keep working.
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
if (restoreOnly) {
|
||||
console.log(`Restored ${exported} pages to ${outDir}/`);
|
||||
} else {
|
||||
console.log(`Exported ${exported} pages to ${outDir}/`);
|
||||
}
|
||||
}
|
||||
|
||||
245
src/commands/storage.ts
Normal file
245
src/commands/storage.ts
Normal file
@@ -0,0 +1,245 @@
|
||||
import { join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadStorageConfig, validateStorageConfig, getStorageTier } from '../core/storage-config.ts';
|
||||
import type { StorageConfig, StorageTier } from '../core/storage-config.ts';
|
||||
import { walkBrainRepo, type DiskFileEntry } from '../core/disk-walk.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
|
||||
/**
|
||||
* Distinct nominal types for the two tier-keyed numeric maps. Both shapes
|
||||
* are `Record<StorageTier, number>` structurally — but they carry
|
||||
* semantically different units (page COUNT vs disk BYTES). Distinct types
|
||||
* make accidental swaps a compile-time error rather than a silent display
|
||||
* bug. Issue #11 of the eng review.
|
||||
*/
|
||||
export type PageCountsByTier = Record<StorageTier, number> & { __brand?: 'page-counts' };
|
||||
export type DiskUsageByTier = Record<StorageTier, number> & { __brand?: 'disk-bytes' };
|
||||
|
||||
/**
|
||||
* Pure-data result of a storage-status query. No side effects, no I/O
|
||||
* beyond the engine call and one filesystem walk. Consumed by both the
|
||||
* JSON formatter and the human formatter; kept narrow so it's a stable
|
||||
* MCP/scripting contract (D14: storage_status is read-only MCP-exposed).
|
||||
*/
|
||||
export interface StorageStatusResult {
|
||||
config: StorageConfig | null;
|
||||
repoPath: string | null;
|
||||
totalPages: number;
|
||||
pagesByTier: PageCountsByTier;
|
||||
missingFiles: Array<{ slug: string; expectedPath: string }>;
|
||||
diskUsageByTier: DiskUsageByTier;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
// ── Dispatcher ────────────────────────────────────────────
|
||||
|
||||
export async function runStorage(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const subcommand = args[0];
|
||||
if (!subcommand || subcommand === 'status') {
|
||||
await runStorageStatus(engine, args.slice(1));
|
||||
return;
|
||||
}
|
||||
console.error(`Unknown storage subcommand: ${subcommand}`);
|
||||
console.error('Available subcommands: status');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function runStorageStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
warnIfPGLite(engine);
|
||||
|
||||
// Resolution chain (D5, Issue #3): explicit --repo → typed accessor → null.
|
||||
// No cwd fallback. The original silent footgun is dead.
|
||||
let repoPath: string | null = null;
|
||||
const repoIdx = args.indexOf('--repo');
|
||||
if (repoIdx !== -1 && args[repoIdx + 1]) {
|
||||
repoPath = args[repoIdx + 1];
|
||||
} else {
|
||||
repoPath = await getDefaultSourcePath(engine);
|
||||
}
|
||||
|
||||
const result = await getStorageStatus(engine, repoPath);
|
||||
|
||||
if (args.includes('--json')) {
|
||||
console.log(formatStorageStatusJson(result));
|
||||
return;
|
||||
}
|
||||
console.log(formatStorageStatusHuman(result));
|
||||
}
|
||||
|
||||
/**
|
||||
* D4: storage tiering on PGLite is a partial feature. The "DB" the pages
|
||||
* live in IS the local file gbrain uses for everything else, so "db_only"
|
||||
* has no real offload effect. The .gitignore management still helps
|
||||
* (keeps bulk content out of git history), so we warn but proceed.
|
||||
*
|
||||
* Once-per-process via a module-local flag — sub-commands invoked from a
|
||||
* single CLI run share the same warning.
|
||||
*/
|
||||
let _pgliteWarned = false;
|
||||
function warnIfPGLite(engine: BrainEngine): void {
|
||||
if (_pgliteWarned) return;
|
||||
if (engine.kind !== 'pglite') return;
|
||||
_pgliteWarned = true;
|
||||
console.warn(
|
||||
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
|
||||
`local database file regardless of tier. The .gitignore management still ` +
|
||||
`keeps bulk content out of git history. To get full tiering, migrate to ` +
|
||||
`Postgres with \`gbrain migrate --to supabase\`.`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Reset for tests. */
|
||||
export function __resetPGLiteWarn(): void {
|
||||
_pgliteWarned = false;
|
||||
}
|
||||
|
||||
// ── Pure data ─────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Compute the storage status against the given engine + brain repo path.
|
||||
*
|
||||
* Side-effect-free apart from the engine.listPages call and one recursive
|
||||
* filesystem walk. Pure for testability — formatters are tested separately.
|
||||
*
|
||||
* Returns null `config` when no gbrain.yml is present at repoPath. In that
|
||||
* case pagesByTier is all zeros for db_tracked/db_only and totals roll up
|
||||
* into unspecified.
|
||||
*/
|
||||
export async function getStorageStatus(
|
||||
engine: BrainEngine,
|
||||
repoPath: string | null,
|
||||
): Promise<StorageStatusResult> {
|
||||
const config = repoPath ? loadStorageConfig(repoPath) : null;
|
||||
const warnings = config ? validateStorageConfig(config) : [];
|
||||
|
||||
const pagesByTier: PageCountsByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
|
||||
const diskUsageByTier: DiskUsageByTier = { db_tracked: 0, db_only: 0, unspecified: 0 };
|
||||
const missingFiles: Array<{ slug: string; expectedPath: string }> = [];
|
||||
|
||||
// Single recursive walk of the brain repo (Issue #14). Replaces per-page
|
||||
// existsSync+statSync — was ~400K syscalls on 200K-page brains, now ~one
|
||||
// per directory + one stat per .md file, plus O(1) lookups below.
|
||||
const fileMap: Map<string, DiskFileEntry> = repoPath ? walkBrainRepo(repoPath) : new Map();
|
||||
|
||||
const pages = await engine.listPages({ limit: 1_000_000 });
|
||||
|
||||
for (const page of pages) {
|
||||
const tier = config ? getStorageTier(page.slug, config) : 'unspecified';
|
||||
pagesByTier[tier]++;
|
||||
if (!repoPath) continue;
|
||||
const entry = fileMap.get(page.slug);
|
||||
if (entry) {
|
||||
diskUsageByTier[tier] += entry.size;
|
||||
} else if (config && tier === 'db_only') {
|
||||
missingFiles.push({ slug: page.slug, expectedPath: join(repoPath, page.slug + '.md') });
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
config,
|
||||
repoPath,
|
||||
totalPages: pages.length,
|
||||
pagesByTier,
|
||||
missingFiles,
|
||||
diskUsageByTier,
|
||||
warnings,
|
||||
};
|
||||
}
|
||||
|
||||
// ── JSON formatter ────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Serialize StorageStatusResult to a stable JSON contract. Indented for
|
||||
* human readability; agents/orchestrators can parse with a standard
|
||||
* JSON.parse. Schema is the StorageStatusResult interface above.
|
||||
*/
|
||||
export function formatStorageStatusJson(result: StorageStatusResult): string {
|
||||
return JSON.stringify(result, null, 2);
|
||||
}
|
||||
|
||||
// ── Human formatter ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Render StorageStatusResult to ASCII text suitable for terminal output.
|
||||
* D10 lock: ASCII separators only — universally portable. No unicode
|
||||
* box-drawing.
|
||||
*/
|
||||
export function formatStorageStatusHuman(result: StorageStatusResult): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Storage Status');
|
||||
lines.push('==============');
|
||||
lines.push('');
|
||||
|
||||
if (!result.config) {
|
||||
lines.push('No gbrain.yml configuration found.');
|
||||
if (result.repoPath) lines.push(`Checked: ${result.repoPath}/gbrain.yml`);
|
||||
lines.push('');
|
||||
lines.push('All pages are stored in git by default.');
|
||||
lines.push(`Total pages: ${result.totalPages}`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
lines.push(`Repository: ${result.repoPath}`);
|
||||
lines.push(`Total pages: ${result.totalPages}`);
|
||||
lines.push('');
|
||||
lines.push('Storage Tiers:');
|
||||
lines.push('-------------');
|
||||
lines.push(`DB tracked: ${result.pagesByTier.db_tracked.toLocaleString()} pages`);
|
||||
lines.push(`DB only: ${result.pagesByTier.db_only.toLocaleString()} pages`);
|
||||
lines.push(`Unspecified: ${result.pagesByTier.unspecified.toLocaleString()} pages`);
|
||||
|
||||
if (result.diskUsageByTier.db_tracked > 0 || result.diskUsageByTier.db_only > 0) {
|
||||
lines.push('');
|
||||
lines.push('Disk Usage:');
|
||||
lines.push('-----------');
|
||||
if (result.diskUsageByTier.db_tracked > 0) {
|
||||
lines.push(`DB tracked: ${formatBytes(result.diskUsageByTier.db_tracked)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.db_only > 0) {
|
||||
lines.push(`DB only: ${formatBytes(result.diskUsageByTier.db_only)}`);
|
||||
}
|
||||
if (result.diskUsageByTier.unspecified > 0) {
|
||||
lines.push(`Unspecified: ${formatBytes(result.diskUsageByTier.unspecified)}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.missingFiles.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('Missing Files (need restore):');
|
||||
lines.push('-----------------------------');
|
||||
for (const missing of result.missingFiles.slice(0, 10)) {
|
||||
lines.push(` ${missing.slug}`);
|
||||
}
|
||||
if (result.missingFiles.length > 10) {
|
||||
lines.push(` ... and ${result.missingFiles.length - 10} more`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push(`Use: gbrain export --restore-only --repo "${result.repoPath}"`);
|
||||
}
|
||||
|
||||
if (result.warnings.length > 0) {
|
||||
lines.push('');
|
||||
lines.push('Warnings:');
|
||||
lines.push('---------');
|
||||
for (const warning of result.warnings) lines.push(` ! ${warning}`);
|
||||
}
|
||||
|
||||
lines.push('');
|
||||
lines.push('Configuration:');
|
||||
lines.push('--------------');
|
||||
lines.push('DB tracked directories:');
|
||||
for (const dir of result.config.db_tracked) lines.push(` - ${dir}`);
|
||||
lines.push('');
|
||||
lines.push('DB-only directories:');
|
||||
for (const dir of result.config.db_only) lines.push(` - ${dir}`);
|
||||
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes === 0) return '0 B';
|
||||
const k = 1024;
|
||||
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return parseFloat((bytes / Math.pow(k, i)).toFixed(1)) + ' ' + sizes[i];
|
||||
}
|
||||
@@ -1,9 +1,8 @@
|
||||
import { existsSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, statSync, readdirSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
import { readFileSync, statSync, readdirSync } from 'fs';
|
||||
import { createInterface } from 'readline';
|
||||
import {
|
||||
buildSyncManifest,
|
||||
@@ -20,6 +19,8 @@ import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { loadStorageConfig } from '../core/storage-config.ts';
|
||||
import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
|
||||
export interface SyncResult {
|
||||
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
|
||||
@@ -839,6 +840,13 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
try {
|
||||
const result = await performSync(engine, repoOpts);
|
||||
printSyncResult(result);
|
||||
// Codex P2: --all loop must also manage .gitignore per-source. Without
|
||||
// this, multi-source users who rely on `gbrain sync --all` never get
|
||||
// the advertised db_only ignore rules unless they sync each repo
|
||||
// individually.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
manageGitignore(src.local_path!, engine.kind);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
console.error(`Error syncing ${src.name}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
@@ -865,6 +873,18 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
if (!watch) {
|
||||
const result = await performSync(engine, opts);
|
||||
printSyncResult(result);
|
||||
// Issue #2 + eng-review pass-2 finding #1 + Codex P1: manage .gitignore ONLY
|
||||
// on successful sync. Skip on dry-run (don't mutate disk in preview mode)
|
||||
// and blocked_by_failures (sync state is inconsistent — defer .gitignore
|
||||
// until next clean run). Resolve the effective repo path so the wire-up
|
||||
// fires in the common case where the user runs `gbrain sync` without
|
||||
// passing --repo every time.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -880,6 +900,14 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const ts = new Date().toISOString().slice(11, 19);
|
||||
console.log(`[${ts}] Synced: +${result.added} ~${result.modified} -${result.deleted} R${result.renamed}`);
|
||||
}
|
||||
// Same gate as non-watch: only manage .gitignore on successful sync.
|
||||
// Same repo-resolution path so watch mode catches the implicit-resolved case.
|
||||
if (result.status !== 'dry_run' && result.status !== 'blocked_by_failures') {
|
||||
const effectiveRepoPath = opts.repoPath ?? (await getDefaultSourcePath(engine));
|
||||
if (effectiveRepoPath) {
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
consecutiveErrors++;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -893,6 +921,124 @@ export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-manage .gitignore entries for db_only directories.
|
||||
*
|
||||
* Caller invokes ONLY on successful sync — this function trusts that the
|
||||
* sync's data state is consistent. See `runSync` for the gating logic.
|
||||
*
|
||||
* Idempotent: re-running adds no duplicate entries. The managed block has
|
||||
* a stable comment header so it's grep-able and editable.
|
||||
*
|
||||
* Skipped (with actionable warning) when:
|
||||
* - GBRAIN_NO_GITIGNORE=1 — D23 escape hatch for shared-repo setups
|
||||
* - The repo is a git submodule (`.git` is a file not a directory) —
|
||||
* D49 lock; submodule .gitignore changes don't survive parent updates
|
||||
*
|
||||
* On PGLite (D4): emits a once-per-process soft-warn explaining that
|
||||
* tiering has limited effect — but still manages the .gitignore so the
|
||||
* config-present user gets the gitignore housekeeping.
|
||||
*
|
||||
* Failures (write permission denied, EROFS, etc.) are caught, warned, and
|
||||
* swallowed (D9 lock). Sync's primary job is moving data; .gitignore
|
||||
* management is a side effect — don't kill the main job for the side effect.
|
||||
*/
|
||||
let _pgliteTierWarned = false;
|
||||
export function __resetPGLiteTierWarn(): void {
|
||||
_pgliteTierWarned = false;
|
||||
}
|
||||
|
||||
export function manageGitignore(
|
||||
repoPath: string,
|
||||
engineKind?: 'pglite' | 'postgres',
|
||||
): void {
|
||||
if (process.env.GBRAIN_NO_GITIGNORE === '1') {
|
||||
return;
|
||||
}
|
||||
|
||||
// D49: submodule detection. In a submodule, `.git` is a regular file
|
||||
// (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory.
|
||||
const dotGit = join(repoPath, '.git');
|
||||
if (existsSync(dotGit)) {
|
||||
try {
|
||||
if (statSync(dotGit).isFile()) {
|
||||
console.warn(
|
||||
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
|
||||
`Add db_only directories to your parent repo's .gitignore manually.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// proceed; can't tell, default to managing
|
||||
}
|
||||
}
|
||||
|
||||
let storageConfig;
|
||||
try {
|
||||
storageConfig = loadStorageConfig(repoPath);
|
||||
} catch (error) {
|
||||
// StorageConfigError (overlap) or read error — surface, don't manage.
|
||||
console.warn(
|
||||
`Skipped .gitignore update: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (!storageConfig || storageConfig.db_only.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
// D4 soft-warn: storage tiering has limited effect on PGLite, but the
|
||||
// .gitignore housekeeping still helps. Warn once per process; proceed.
|
||||
if (engineKind === 'pglite' && !_pgliteTierWarned) {
|
||||
_pgliteTierWarned = true;
|
||||
console.warn(
|
||||
`Note: storage tiering has limited effect on PGLite — pages live in your ` +
|
||||
`local database file regardless of tier. Managing .gitignore anyway.`,
|
||||
);
|
||||
}
|
||||
|
||||
const gitignorePath = join(repoPath, '.gitignore');
|
||||
let gitignoreContent = '';
|
||||
|
||||
if (existsSync(gitignorePath)) {
|
||||
try {
|
||||
gitignoreContent = readFileSync(gitignorePath, 'utf-8');
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Could not read ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
|
||||
`skipping .gitignore update. Add db_only directories manually.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const existingLines = new Set(gitignoreContent.split('\n').map((line) => line.trim()));
|
||||
const linesToAdd: string[] = [];
|
||||
|
||||
for (const dir of storageConfig.db_only) {
|
||||
if (!existingLines.has(dir) && !existingLines.has(`/${dir}`)) {
|
||||
linesToAdd.push(dir);
|
||||
}
|
||||
}
|
||||
|
||||
if (linesToAdd.length === 0) return;
|
||||
|
||||
if (gitignoreContent && !gitignoreContent.endsWith('\n')) {
|
||||
gitignoreContent += '\n';
|
||||
}
|
||||
gitignoreContent += '\n# Auto-managed by gbrain (db_only directories)\n';
|
||||
gitignoreContent += linesToAdd.join('\n') + '\n';
|
||||
|
||||
try {
|
||||
writeFileSync(gitignorePath, gitignoreContent);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Could not update ${gitignorePath} (${error instanceof Error ? error.message : String(error)}) — ` +
|
||||
`please add db_only directories manually:\n ${linesToAdd.join('\n ')}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function printSyncResult(result: SyncResult) {
|
||||
switch (result.status) {
|
||||
case 'up_to_date':
|
||||
|
||||
83
src/core/disk-walk.ts
Normal file
83
src/core/disk-walk.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Recursive filesystem walk into a slug → Stats map.
|
||||
*
|
||||
* Replaces per-page `existsSync` + `statSync` syscall storms (Issue #14 of
|
||||
* the v0.22.3 eng review). On a 200K-page brain the per-page approach was
|
||||
* 400K syscalls in a synchronous loop; this walk is one syscall per directory
|
||||
* plus one stat per file, then O(1) Map lookups for everything downstream.
|
||||
*
|
||||
* The slug key is the on-disk path relative to the brain repo, with the
|
||||
* trailing `.md` stripped, matching how pages are stored: `people/alice.md`
|
||||
* on disk becomes `people/alice` as a slug.
|
||||
*
|
||||
* Skipped entries:
|
||||
* - `.git/`, `node_modules/`, and dot-directories generally — not part of
|
||||
* the brain's page namespace. Speeds up walks significantly on dirty
|
||||
* working copies.
|
||||
* - Files that don't end in `.md`. Sidecar JSON, raw binary attachments,
|
||||
* etc. are tracked by the brain but not via slugs.
|
||||
*/
|
||||
|
||||
import { readdirSync, statSync, type Stats, type Dirent } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
export interface DiskFileEntry {
|
||||
size: number;
|
||||
mtimeMs: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk `repoPath` and return a Map of slug → file metadata for every `.md`
|
||||
* file. Skips dot-directories. Synchronous (matches the call-site shape and
|
||||
* the io pattern of stat-heavy scans).
|
||||
*
|
||||
* @param repoPath Absolute path to the brain repo root.
|
||||
* @returns Map keyed by slug (no `.md` suffix). Empty map if repoPath
|
||||
* doesn't exist or contains no markdown files.
|
||||
*/
|
||||
export function walkBrainRepo(repoPath: string): Map<string, DiskFileEntry> {
|
||||
const result = new Map<string, DiskFileEntry>();
|
||||
|
||||
function recurse(dirPath: string, slugPrefix: string): void {
|
||||
// Annotate as Dirent[] explicitly: ReturnType<typeof readdirSync> with
|
||||
// withFileTypes:true picks an overload union that includes
|
||||
// Dirent<Buffer<ArrayBufferLike>>, which makes entry.name a Buffer in
|
||||
// strict tsc mode. Cast to the string-based Dirent[] (same shape sync.ts
|
||||
// uses for its own filesystem walk).
|
||||
let entries: Dirent[];
|
||||
try {
|
||||
entries = readdirSync(dirPath, { withFileTypes: true }) as unknown as Dirent[];
|
||||
} catch {
|
||||
return; // unreadable directory — skip silently
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
// Skip dot-directories (.git, .gbrain, .vscode, etc) and node_modules.
|
||||
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||||
const childPath = join(dirPath, entry.name);
|
||||
|
||||
if (entry.isDirectory()) {
|
||||
recurse(childPath, slugPrefix ? `${slugPrefix}/${entry.name}` : entry.name);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!entry.isFile()) continue;
|
||||
if (!entry.name.endsWith('.md')) continue;
|
||||
|
||||
let stats: Stats;
|
||||
try {
|
||||
stats = statSync(childPath);
|
||||
} catch {
|
||||
continue; // race: file deleted between readdir and stat
|
||||
}
|
||||
|
||||
const slug = slugPrefix
|
||||
? `${slugPrefix}/${entry.name.slice(0, -3)}`
|
||||
: entry.name.slice(0, -3);
|
||||
result.set(slug, { size: stats.size, mtimeMs: stats.mtimeMs });
|
||||
}
|
||||
}
|
||||
|
||||
recurse(repoPath, '');
|
||||
return result;
|
||||
}
|
||||
@@ -294,6 +294,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
params.push(filters.updated_after);
|
||||
where.push(`p.updated_at > $${params.length}::timestamptz`);
|
||||
}
|
||||
// slugPrefix uses the (source_id, slug) UNIQUE btree for index range scans.
|
||||
// Escape LIKE metacharacters so the user prefix is treated as a literal.
|
||||
if (filters?.slugPrefix) {
|
||||
const escaped = filters.slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%';
|
||||
params.push(escaped);
|
||||
where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`);
|
||||
}
|
||||
|
||||
const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
|
||||
params.push(limit, offset);
|
||||
|
||||
@@ -334,11 +334,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
|
||||
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
|
||||
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
|
||||
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
|
||||
// Escape LIKE metacharacters so the user prefix is treated as a literal.
|
||||
const slugPrefix = filters?.slugPrefix;
|
||||
const slugCondition = slugPrefix
|
||||
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
|
||||
: sql``;
|
||||
|
||||
const rows = await sql`
|
||||
SELECT p.* FROM pages p
|
||||
${tagJoin}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
|
||||
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
|
||||
`;
|
||||
|
||||
|
||||
@@ -132,6 +132,42 @@ async function assertSourceExists(engine: BrainEngine, id: string): Promise<void
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the local_path of the resolved source (per the resolveSourceId chain).
|
||||
*
|
||||
* Returns the on-disk brain repo path for the source the user is currently
|
||||
* operating against. Used by `gbrain storage status` and `gbrain export
|
||||
* --restore-only` to find the brain repo without raw SQL or bare try/catch.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. `sources.local_path` for the resolved source id (multi-source v0.18+ path)
|
||||
* 2. Legacy global `sync.repo_path` config key (pre-v0.18 default-source brains)
|
||||
* 3. null
|
||||
*
|
||||
* @returns local_path string, or null if no path is configured anywhere.
|
||||
* @throws If DB error occurs (does NOT silently swallow). Callers handle
|
||||
* the null case to provide their own fallback (typically a hard error
|
||||
* telling the user to pass --repo).
|
||||
*/
|
||||
export async function getDefaultSourcePath(
|
||||
engine: BrainEngine,
|
||||
cwd: string = process.cwd(),
|
||||
): Promise<string | null> {
|
||||
const sourceId = await resolveSourceId(engine, null, cwd);
|
||||
const rows = await engine.executeRaw<{ local_path: string | null }>(
|
||||
`SELECT local_path FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
if (rows[0]?.local_path) return rows[0].local_path;
|
||||
|
||||
// Legacy fallback: pre-v0.18 brains stored the repo path in the global
|
||||
// config table under sync.repo_path. The sources table exists but its
|
||||
// local_path is NULL for the seeded 'default' row. Fall back so storage
|
||||
// tiering works without forcing a `gbrain sources add . --path .` migration.
|
||||
const legacyPath = await engine.getConfig('sync.repo_path');
|
||||
return legacyPath ?? null;
|
||||
}
|
||||
|
||||
/** Exposed for tests. */
|
||||
export const __testing = {
|
||||
readDotfileWalk,
|
||||
|
||||
377
src/core/storage-config.ts
Normal file
377
src/core/storage-config.ts
Normal file
@@ -0,0 +1,377 @@
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
/**
|
||||
* Storage tier configuration loaded from gbrain.yml.
|
||||
*
|
||||
* The canonical key names are `db_tracked` and `db_only` (engine-agnostic).
|
||||
* The deprecated keys `git_tracked` and `supabase_only` are still read for
|
||||
* backward compatibility but emit a once-per-process deprecation warning.
|
||||
* Sunset: future release will reject the deprecated names.
|
||||
*/
|
||||
export interface StorageConfig {
|
||||
db_tracked: string[];
|
||||
db_only: string[];
|
||||
}
|
||||
|
||||
export type StorageTier = 'db_tracked' | 'db_only' | 'unspecified';
|
||||
|
||||
/** Recognized YAML keys (canonical and deprecated). */
|
||||
const STORAGE_KEYS = new Set([
|
||||
'db_tracked', 'db_only',
|
||||
'git_tracked', 'supabase_only', // deprecated aliases
|
||||
]);
|
||||
|
||||
/**
|
||||
* Parse the gbrain.yml shape: a top-level `storage:` section with up to four
|
||||
* array-valued nested keys (canonical `db_tracked` / `db_only` plus the
|
||||
* deprecated aliases `git_tracked` / `supabase_only`).
|
||||
*
|
||||
* Intentionally narrow. Does NOT handle the full YAML spec — only the file
|
||||
* shape gbrain controls. Trades expressiveness for zero-dep parsing and
|
||||
* predictable behavior. Returns null if the file has no `storage:` section
|
||||
* (so callers can distinguish "no config" from "empty config").
|
||||
*
|
||||
* Replaces gray-matter, which silently returned `{data: {}}` on
|
||||
* delimiter-less YAML and broke the entire feature on every install.
|
||||
* The defect that prompted this rewrite: storage-config.ts:24 in the
|
||||
* pre-v0.22.3 implementation.
|
||||
*
|
||||
* Returns the raw key map. The caller (loadStorageConfig) is responsible
|
||||
* for normalizing deprecated keys → canonical, emitting deprecation
|
||||
* warnings, and merging if both old and new keys appear.
|
||||
*/
|
||||
type RawStorage = {
|
||||
db_tracked?: string[];
|
||||
db_only?: string[];
|
||||
git_tracked?: string[];
|
||||
supabase_only?: string[];
|
||||
};
|
||||
|
||||
function parseStorageYaml(content: string): RawStorage | null {
|
||||
const lines = content.split('\n').map((line) => line.replace(/\r$/, ''));
|
||||
|
||||
let inStorage = false;
|
||||
let currentList: keyof RawStorage | null = null;
|
||||
const raw: RawStorage = {};
|
||||
let sawStorage = false;
|
||||
|
||||
for (const line of lines) {
|
||||
// Strip comments. Conservative: drop trailing `# ...` and full-line `#`.
|
||||
const noComment = line.replace(/\s+#.*$/, '').replace(/^#.*$/, '');
|
||||
if (noComment.trim() === '') continue;
|
||||
|
||||
// Top-level key (no leading whitespace).
|
||||
if (!noComment.startsWith(' ') && !noComment.startsWith('\t')) {
|
||||
const colon = noComment.indexOf(':');
|
||||
if (colon === -1) continue;
|
||||
const key = noComment.slice(0, colon).trim();
|
||||
if (key === 'storage') {
|
||||
inStorage = true;
|
||||
sawStorage = true;
|
||||
currentList = null;
|
||||
continue;
|
||||
}
|
||||
inStorage = false;
|
||||
currentList = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!inStorage) continue;
|
||||
|
||||
const indented = noComment.replace(/^\s+/, '');
|
||||
|
||||
if (indented.startsWith('-')) {
|
||||
if (!currentList) continue;
|
||||
const value = indented.slice(1).trim().replace(/^["']|["']$/g, '');
|
||||
if (value) {
|
||||
if (!raw[currentList]) raw[currentList] = [];
|
||||
raw[currentList]!.push(value);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const colon = indented.indexOf(':');
|
||||
if (colon === -1) continue;
|
||||
const key = indented.slice(0, colon).trim();
|
||||
if (STORAGE_KEYS.has(key)) {
|
||||
currentList = key as keyof RawStorage;
|
||||
// Inline empty list: `db_only: []`.
|
||||
const remainder = indented.slice(colon + 1).trim();
|
||||
if (remainder === '[]' && !raw[currentList]) {
|
||||
raw[currentList] = [];
|
||||
}
|
||||
continue;
|
||||
}
|
||||
currentList = null;
|
||||
}
|
||||
|
||||
if (!sawStorage) return null;
|
||||
return raw;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize raw parsed keys into canonical StorageConfig shape.
|
||||
*
|
||||
* Resolution order (per plan eng-review pass 2 finding #2):
|
||||
* 1. If canonical keys present, use them.
|
||||
* 2. Else if deprecated keys present, map to canonical AND emit a
|
||||
* once-per-process deprecation warning suggesting `gbrain doctor --fix`.
|
||||
* 3. If both are present, canonical wins. Deprecated keys are ignored
|
||||
* with a stronger warning (the user is mid-migration).
|
||||
*
|
||||
* Validation (validateStorageConfig) always runs against the canonical
|
||||
* shape, so error messages reference `db_only` / `db_tracked` regardless
|
||||
* of which keys the user wrote.
|
||||
*/
|
||||
let _deprecationWarned = false;
|
||||
|
||||
function normalizeStorageConfig(raw: RawStorage): StorageConfig {
|
||||
const hasCanonical = Boolean(raw.db_tracked || raw.db_only);
|
||||
const hasDeprecated = Boolean(raw.git_tracked || raw.supabase_only);
|
||||
|
||||
if (hasDeprecated && !_deprecationWarned) {
|
||||
_deprecationWarned = true;
|
||||
const which = [
|
||||
raw.git_tracked ? '`git_tracked`' : null,
|
||||
raw.supabase_only ? '`supabase_only`' : null,
|
||||
].filter(Boolean).join(' and ');
|
||||
if (hasCanonical) {
|
||||
console.warn(
|
||||
`Warning: ${which} in gbrain.yml is deprecated and ignored ` +
|
||||
`(canonical keys db_tracked/db_only are present). ` +
|
||||
`Remove the deprecated keys, or run \`gbrain doctor --fix\`.`,
|
||||
);
|
||||
} else {
|
||||
console.warn(
|
||||
`Warning: ${which} in gbrain.yml is deprecated. ` +
|
||||
`Rename to db_tracked / db_only — see docs/storage-tiering.md. ` +
|
||||
`Run \`gbrain doctor --fix\` for an automated rename.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (hasCanonical) {
|
||||
return {
|
||||
db_tracked: raw.db_tracked ?? [],
|
||||
db_only: raw.db_only ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
db_tracked: raw.git_tracked ?? [],
|
||||
db_only: raw.supabase_only ?? [],
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load gbrain.yml configuration from the brain repository root.
|
||||
*
|
||||
* Returns null when:
|
||||
* - repoPath is null/undefined
|
||||
* - gbrain.yml doesn't exist at the repo root
|
||||
* - gbrain.yml exists but has no `storage:` section (with sanity warning)
|
||||
*
|
||||
* Throws when:
|
||||
* - gbrain.yml exists but is unreadable (permission denied, etc.) — D36 lock:
|
||||
* fail loud rather than silently disable the feature.
|
||||
*
|
||||
* Logs a console.warn (once per process) when:
|
||||
* - File parses but `storage:` section is empty or missing — Issue #1 lock:
|
||||
* surface "your config didn't take" rather than silently no-op.
|
||||
*/
|
||||
let _missingStorageWarned = false;
|
||||
|
||||
export function loadStorageConfig(repoPath?: string | null): StorageConfig | null {
|
||||
if (!repoPath) return null;
|
||||
|
||||
const yamlPath = join(repoPath, 'gbrain.yml');
|
||||
if (!existsSync(yamlPath)) return null;
|
||||
|
||||
// Read failure is a real error (not a "feature not configured" signal).
|
||||
// Throwing here lets the caller decide whether to crash or fall back.
|
||||
const content = readFileSync(yamlPath, 'utf-8');
|
||||
|
||||
let raw: RawStorage | null;
|
||||
try {
|
||||
raw = parseStorageYaml(content);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Warning: Failed to parse gbrain.yml: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
|
||||
// No storage section at all → null (with sanity warning).
|
||||
if (raw === null) {
|
||||
if (!_missingStorageWarned) {
|
||||
_missingStorageWarned = true;
|
||||
console.warn(
|
||||
`Warning: ${yamlPath} exists but has no storage configuration. ` +
|
||||
`Add a "storage:" section with db_tracked / db_only arrays, ` +
|
||||
`or remove gbrain.yml to suppress this warning.`,
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const merged = normalizeStorageConfig(raw);
|
||||
|
||||
// Empty storage section → return as-is but warn.
|
||||
if (merged.db_tracked.length === 0 && merged.db_only.length === 0) {
|
||||
if (!_missingStorageWarned) {
|
||||
_missingStorageWarned = true;
|
||||
console.warn(
|
||||
`Warning: ${yamlPath} exists but has no storage configuration. ` +
|
||||
`Add a "storage:" section with db_tracked / db_only arrays, ` +
|
||||
`or remove gbrain.yml to suppress this warning.`,
|
||||
);
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
// Normalize cosmetic issues + throw on semantic overlap (D7).
|
||||
// Throws StorageConfigError on overlap — propagates to the caller.
|
||||
return normalizeAndValidateStorageConfig(merged);
|
||||
}
|
||||
|
||||
export class StorageConfigError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'StorageConfigError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate storage configuration for conflicts and issues.
|
||||
* Returns warning strings; callers decide how to surface them.
|
||||
*
|
||||
* Always runs against the canonical (db_tracked / db_only) shape — error
|
||||
* messages reference canonical names regardless of which keys the user
|
||||
* wrote in gbrain.yml.
|
||||
*
|
||||
* Pure: does not mutate. For the auto-normalize behavior (D7), see
|
||||
* `normalizeAndValidateStorageConfig` below.
|
||||
*/
|
||||
export function validateStorageConfig(config: StorageConfig): string[] {
|
||||
const warnings: string[] = [];
|
||||
|
||||
const trackedSet = new Set(config.db_tracked);
|
||||
for (const path of config.db_only) {
|
||||
if (trackedSet.has(path)) {
|
||||
warnings.push(`Directory "${path}" appears in both db_tracked and db_only`);
|
||||
}
|
||||
}
|
||||
|
||||
const allPaths = [...config.db_tracked, ...config.db_only];
|
||||
for (const path of allPaths) {
|
||||
if (!path.endsWith('/')) {
|
||||
warnings.push(`Directory path "${path}" should end with "/" for consistency`);
|
||||
}
|
||||
}
|
||||
|
||||
return warnings;
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-normalize and strict-validate per D7+D8.
|
||||
*
|
||||
* 1. Cosmetic fixups are applied silently with a one-time info message
|
||||
* naming what changed:
|
||||
* - missing trailing `/` is added
|
||||
* The message helps the user learn the canonical form without nagging.
|
||||
* 2. Semantic problems THROW (don't return warnings):
|
||||
* - same directory in both tiers (ambiguous routing)
|
||||
*
|
||||
* Caller passes a fresh raw config; this returns the normalized shape that
|
||||
* the rest of the code (matcher, sync, etc.) sees.
|
||||
*/
|
||||
let _normalizationInfoEmitted = false;
|
||||
|
||||
export function normalizeAndValidateStorageConfig(input: StorageConfig): StorageConfig {
|
||||
const normalize = (paths: string[]): { normalized: string[]; changed: string[] } => {
|
||||
const normalized: string[] = [];
|
||||
const changed: string[] = [];
|
||||
for (const p of paths) {
|
||||
if (p.endsWith('/')) {
|
||||
normalized.push(p);
|
||||
} else {
|
||||
normalized.push(p + '/');
|
||||
changed.push(`"${p}" → "${p}/"`);
|
||||
}
|
||||
}
|
||||
return { normalized, changed };
|
||||
};
|
||||
|
||||
const tracked = normalize(input.db_tracked);
|
||||
const dbonly = normalize(input.db_only);
|
||||
const allChanged = [...tracked.changed, ...dbonly.changed];
|
||||
|
||||
if (allChanged.length > 0 && !_normalizationInfoEmitted) {
|
||||
_normalizationInfoEmitted = true;
|
||||
console.warn(
|
||||
`Note: normalized ${allChanged.length} storage path(s) in gbrain.yml — ` +
|
||||
`${allChanged.join(', ')}. Add trailing "/" to suppress this note.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Semantic check: overlap between tiers throws. Ambiguous routing.
|
||||
const trackedSet = new Set(tracked.normalized);
|
||||
for (const path of dbonly.normalized) {
|
||||
if (trackedSet.has(path)) {
|
||||
throw new StorageConfigError(
|
||||
`gbrain.yml: directory "${path}" appears in both db_tracked and db_only — ` +
|
||||
`pick one tier. Edit gbrain.yml to remove the overlap.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return { db_tracked: tracked.normalized, db_only: dbonly.normalized };
|
||||
}
|
||||
|
||||
/**
|
||||
* Path-segment match: a slug belongs to a tier directory iff the directory
|
||||
* is a complete path-segment ancestor of the slug. `media/x/` matches
|
||||
* `media/x/foo` but NOT `media/xerox/foo` — eliminates the prefix-collision
|
||||
* class of bug (Issue #5 of the eng review, D6 lock).
|
||||
*
|
||||
* Strict: requires the configured directory to end with `/`. The validator
|
||||
* (per D7+D8) auto-normalizes input so the matcher only ever sees canonical
|
||||
* trailing-`/` directories.
|
||||
*/
|
||||
function matchesTierDir(slug: string, dir: string): boolean {
|
||||
if (!dir.endsWith('/')) return false; // not normalized — matcher refuses
|
||||
// slug must equal dir's bare prefix OR start with the trailing-slash form.
|
||||
// Example: dir = 'media/x/' matches 'media/x/anything' but not 'media/x'
|
||||
// or 'media/xerox'. (A slug that exactly equals 'media/x' is a directory-
|
||||
// level entry the brain doesn't write.)
|
||||
return slug.startsWith(dir);
|
||||
}
|
||||
|
||||
export function isDbTracked(slug: string, config: StorageConfig): boolean {
|
||||
return config.db_tracked.some((dir) => matchesTierDir(slug, dir));
|
||||
}
|
||||
|
||||
export function isDbOnly(slug: string, config: StorageConfig): boolean {
|
||||
return config.db_only.some((dir) => matchesTierDir(slug, dir));
|
||||
}
|
||||
|
||||
export function getStorageTier(slug: string, config: StorageConfig): StorageTier {
|
||||
if (isDbTracked(slug, config)) return 'db_tracked';
|
||||
if (isDbOnly(slug, config)) return 'db_only';
|
||||
return 'unspecified';
|
||||
}
|
||||
|
||||
// ── Deprecated aliases — to be removed in a future release ────────
|
||||
// Kept so existing callers (storage.ts, export.ts) compile during the
|
||||
// step-by-step refactor. Will be deleted once those call sites migrate
|
||||
// to the canonical names.
|
||||
export const isGitTracked = isDbTracked;
|
||||
export const isSupabaseOnly = isDbOnly;
|
||||
|
||||
/** Reset once-per-process warning flags. Test-only. */
|
||||
export function __resetMissingStorageWarning(): void {
|
||||
_missingStorageWarned = false;
|
||||
_deprecationWarned = false;
|
||||
_normalizationInfoEmitted = false;
|
||||
}
|
||||
@@ -45,6 +45,14 @@ export interface PageFilters {
|
||||
offset?: number;
|
||||
/** ISO date string (YYYY-MM-DD or full ISO timestamp). Filter to pages updated_at > value. */
|
||||
updated_after?: string;
|
||||
/**
|
||||
* Prefix-match filter on slug. Implemented as `WHERE slug LIKE prefix || '%'`
|
||||
* in both engines so it uses the (source_id, slug) UNIQUE constraint's btree
|
||||
* index for efficient range scans on large brains. Used by storage-tiering
|
||||
* commands (gbrain storage status, gbrain export --restore-only) to scope
|
||||
* queries to a tier directory without loading every page into memory.
|
||||
*/
|
||||
slugPrefix?: string;
|
||||
}
|
||||
|
||||
// Chunks
|
||||
|
||||
92
test/disk-walk.test.ts
Normal file
92
test/disk-walk.test.ts
Normal file
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* Tests for src/core/disk-walk.ts — single-walk filesystem scan.
|
||||
*
|
||||
* Replaces the per-page existsSync+statSync syscall storm in storage.ts
|
||||
* (Issue #14 of the v0.22.3 eng review).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { walkBrainRepo } from '../src/core/disk-walk.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-walk-test-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function write(relPath: string, content: string): void {
|
||||
const full = join(tmp, relPath);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('walkBrainRepo', () => {
|
||||
test('returns empty map for empty directory', () => {
|
||||
expect(walkBrainRepo(tmp).size).toBe(0);
|
||||
});
|
||||
|
||||
test('returns empty map for nonexistent directory', () => {
|
||||
expect(walkBrainRepo(join(tmp, 'does-not-exist')).size).toBe(0);
|
||||
});
|
||||
|
||||
test('finds top-level .md files keyed by slug (no .md suffix)', () => {
|
||||
write('alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.has('alice')).toBe(true);
|
||||
expect(result.get('alice')!.size).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('walks nested directories and produces slash-joined slugs', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
write('media/x/tweet-1.md', 'tweet');
|
||||
write('media/articles/post-1.md', 'post');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(
|
||||
new Set(['people/alice', 'media/x/tweet-1', 'media/articles/post-1']),
|
||||
);
|
||||
});
|
||||
|
||||
test('skips dot-directories (.git, .gbrain, .vscode)', () => {
|
||||
write('.git/HEAD', 'ref: refs/heads/main');
|
||||
write('.gbrain/config.json', '{}');
|
||||
write('.vscode/settings.json', '{}');
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('skips node_modules', () => {
|
||||
write('node_modules/foo/bar.md', 'noise');
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('ignores non-.md files', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
write('people/alice.json', '{}');
|
||||
write('people/photo.png', 'binary');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(new Set(result.keys())).toEqual(new Set(['people/alice']));
|
||||
});
|
||||
|
||||
test('captures size from stat', () => {
|
||||
const content = '# Alice\n'.repeat(100);
|
||||
write('people/alice.md', content);
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.get('people/alice')!.size).toBe(content.length);
|
||||
});
|
||||
|
||||
test('captures mtimeMs', () => {
|
||||
write('people/alice.md', '# Alice');
|
||||
const result = walkBrainRepo(tmp);
|
||||
expect(result.get('people/alice')!.mtimeMs).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
273
test/e2e/storage-tiering.test.ts
Normal file
273
test/e2e/storage-tiering.test.ts
Normal file
@@ -0,0 +1,273 @@
|
||||
/**
|
||||
* E2E test for storage tiering — Postgres-only.
|
||||
*
|
||||
* Per the v0.23.0 plan: full lifecycle. Container restart simulation:
|
||||
* write pages via Postgres, delete files from disk, run gbrain export
|
||||
* --restore-only, assert files restored. Real .gitignore round-trip.
|
||||
* Real source-resolver path through getDefaultSourcePath().
|
||||
*
|
||||
* Skips gracefully when DATABASE_URL is unset (per CLAUDE.md E2E pattern).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { setupDB, teardownDB, getEngine, hasDatabase, getConn } from './helpers.ts';
|
||||
import {
|
||||
getStorageStatus,
|
||||
formatStorageStatusHuman,
|
||||
__resetPGLiteWarn,
|
||||
} from '../../src/commands/storage.ts';
|
||||
import { manageGitignore, __resetPGLiteTierWarn } from '../../src/commands/sync.ts';
|
||||
import { getDefaultSourcePath } from '../../src/core/source-resolver.ts';
|
||||
import { __resetMissingStorageWarning } from '../../src/core/storage-config.ts';
|
||||
|
||||
if (!hasDatabase()) {
|
||||
describe('storage-tiering E2E', () => {
|
||||
test.skip('DATABASE_URL not set — skipping E2E', () => {});
|
||||
});
|
||||
} else {
|
||||
describe('storage-tiering E2E (Postgres lifecycle)', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-e2e-storage-'));
|
||||
__resetMissingStorageWarning();
|
||||
__resetPGLiteWarn();
|
||||
__resetPGLiteTierWarn();
|
||||
});
|
||||
|
||||
function cleanup(): void {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeGbrainYml(): void {
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
`,
|
||||
);
|
||||
}
|
||||
|
||||
test('engine.kind is postgres', () => {
|
||||
try {
|
||||
expect(getEngine().kind).toBe('postgres');
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('full lifecycle: write pages → status reports tiers → manage .gitignore → restore-only path', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
|
||||
// Truncate sources + pages so this test has a clean slate.
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
|
||||
[tmp],
|
||||
);
|
||||
|
||||
writeGbrainYml();
|
||||
|
||||
// Seed 4 pages: 1 db_tracked, 2 db_only, 1 unspecified.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice is a founder.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-1', {
|
||||
type: 'media',
|
||||
title: 'Tweet 1',
|
||||
compiled_truth: 'tweet body',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-2', {
|
||||
type: 'media',
|
||||
title: 'Tweet 2',
|
||||
compiled_truth: 'tweet body 2',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('random/note', {
|
||||
type: 'note',
|
||||
title: 'Random',
|
||||
compiled_truth: 'random',
|
||||
timeline: '',
|
||||
});
|
||||
|
||||
// Storage status reports tier counts correctly.
|
||||
const status = await getStorageStatus(engine, tmp);
|
||||
expect(status.totalPages).toBe(4);
|
||||
expect(status.pagesByTier.db_tracked).toBe(1);
|
||||
expect(status.pagesByTier.db_only).toBe(2);
|
||||
expect(status.pagesByTier.unspecified).toBe(1);
|
||||
|
||||
// Human formatter renders without errors.
|
||||
const out = formatStorageStatusHuman(status);
|
||||
expect(out).toContain('DB tracked: 1 pages');
|
||||
expect(out).toContain('DB only: 2 pages');
|
||||
|
||||
// .gitignore management: empty .gitignore → managed block written.
|
||||
manageGitignore(tmp, 'postgres');
|
||||
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
expect(gitignore).toContain('# Auto-managed by gbrain');
|
||||
expect(gitignore).toContain('media/x/');
|
||||
expect(gitignore).toContain('media/articles/');
|
||||
|
||||
// Idempotency: second run adds nothing new.
|
||||
manageGitignore(tmp, 'postgres');
|
||||
const gitignore2 = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
const xCount = (gitignore2.match(/^media\/x\/$/gm) || []).length;
|
||||
expect(xCount).toBe(1);
|
||||
|
||||
// Source resolution finds the local_path we registered.
|
||||
const resolvedPath = await getDefaultSourcePath(engine);
|
||||
expect(resolvedPath).toBe(tmp);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('container restart simulation: db_only files missing on disk are restorable from DB', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
// Fresh slate.
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', $1)`,
|
||||
[tmp],
|
||||
);
|
||||
|
||||
writeGbrainYml();
|
||||
|
||||
// Write some db_only pages to the database.
|
||||
await engine.putPage('media/x/tweet-1', {
|
||||
type: 'media',
|
||||
title: 'Tweet 1',
|
||||
compiled_truth: 'tweet body 1',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('media/x/tweet-2', {
|
||||
type: 'media',
|
||||
title: 'Tweet 2',
|
||||
compiled_truth: 'tweet body 2',
|
||||
timeline: '',
|
||||
});
|
||||
|
||||
// Simulate "files were on disk, but the container restarted."
|
||||
// Storage status: missingFiles should list them.
|
||||
const status = await getStorageStatus(engine, tmp);
|
||||
expect(status.pagesByTier.db_only).toBe(2);
|
||||
expect(status.missingFiles.length).toBe(2);
|
||||
|
||||
// Verify slugPrefix engine filter (Issue #13) works on Postgres for
|
||||
// the prefix that --restore-only would use.
|
||||
const tierPages = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
|
||||
expect(tierPages.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
|
||||
|
||||
// Source-default path resolution returns the configured local_path
|
||||
// (the typed accessor that replaces the original raw-SQL try/catch
|
||||
// in storage.ts:38).
|
||||
const path = await getDefaultSourcePath(engine);
|
||||
expect(path).toBe(tmp);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('slugPrefix filter on Postgres uses index-based range scan (regression for Issue #13)', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE pages, content_chunks, sources CASCADE`);
|
||||
await conn.unsafe(`INSERT INTO sources (id, name) VALUES ('default', 'Default')`);
|
||||
|
||||
// Seed enough data to make a difference between scan types.
|
||||
for (let i = 0; i < 50; i++) {
|
||||
await engine.putPage(`media/x/item-${i}`, {
|
||||
type: 'media',
|
||||
title: `Item ${i}`,
|
||||
compiled_truth: 'x',
|
||||
timeline: '',
|
||||
});
|
||||
}
|
||||
for (let i = 0; i < 50; i++) {
|
||||
await engine.putPage(`people/p-${i}`, {
|
||||
type: 'person',
|
||||
title: `Person ${i}`,
|
||||
compiled_truth: 'x',
|
||||
timeline: '',
|
||||
});
|
||||
}
|
||||
|
||||
// Prefix query should return exactly 50 (people not included).
|
||||
const xResults = await engine.listPages({ slugPrefix: 'media/x/', limit: 200 });
|
||||
expect(xResults.length).toBe(50);
|
||||
for (const p of xResults) {
|
||||
expect(p.slug.startsWith('media/x/')).toBe(true);
|
||||
}
|
||||
|
||||
// Path-segment risk: slugPrefix 'media/x' (no /) would match
|
||||
// 'media/xerox' if any existed. The engine treats slugPrefix as a
|
||||
// literal string prefix; trailing-/ semantics are the matcher's
|
||||
// responsibility (storage-config.ts).
|
||||
const looseResults = await engine.listPages({ slugPrefix: 'media/x', limit: 200 });
|
||||
expect(looseResults.length).toBe(50); // no media/xerox/* exists yet
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('hard-error path: storage status without local_path or --repo gets null repoPath', async () => {
|
||||
try {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn.unsafe(`TRUNCATE sources CASCADE`);
|
||||
// Default source with NO local_path.
|
||||
await conn.unsafe(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ('default', 'Default', NULL)`,
|
||||
);
|
||||
|
||||
const path = await getDefaultSourcePath(engine);
|
||||
expect(path).toBeNull();
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('manageGitignore on Postgres engine does NOT emit PGLite warning', async () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
const warnings: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (...a: unknown[]) => warnings.push(a.map(String).join(' '));
|
||||
try {
|
||||
manageGitignore(tmp, 'postgres');
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -101,6 +101,37 @@ describe('PGLiteEngine: Pages', () => {
|
||||
expect(tagged[0].slug).toBe('test/tagged');
|
||||
});
|
||||
|
||||
test('listPages with slugPrefix filter (Issue #13)', async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('media/x/tweet-1', { ...testPage, type: 'concept' });
|
||||
await engine.putPage('media/x/tweet-2', { ...testPage, type: 'concept' });
|
||||
await engine.putPage('media/articles/post-1', { ...testPage, type: 'concept' });
|
||||
await engine.putPage('people/alice', { ...testPage, type: 'person' });
|
||||
|
||||
const xOnly = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
|
||||
expect(xOnly.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
|
||||
|
||||
const allMedia = await engine.listPages({ slugPrefix: 'media/', limit: 100 });
|
||||
expect(allMedia.length).toBe(3);
|
||||
|
||||
// Path-segment risk: 'media/x' (no trailing /) would also match 'media/xerox'.
|
||||
// The matcher in storage-config.ts is responsible for trailing-/ semantics
|
||||
// (step 6); the engine treats slugPrefix as a literal string prefix.
|
||||
expect((await engine.listPages({ slugPrefix: 'media/x', limit: 100 })).length).toBe(2);
|
||||
});
|
||||
|
||||
test('listPages slugPrefix escapes LIKE metacharacters', async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('safe/foo', { ...testPage, type: 'concept' });
|
||||
// A user prefix containing % or _ would otherwise match unintended slugs
|
||||
// if not escaped. We can't easily insert a slug with % in it (most slugs
|
||||
// are url-safe), but we can confirm the escape logic doesn't break the
|
||||
// happy path.
|
||||
const result = await engine.listPages({ slugPrefix: 'safe/', limit: 10 });
|
||||
expect(result.length).toBe(1);
|
||||
expect(result[0].slug).toBe('safe/foo');
|
||||
});
|
||||
|
||||
test('resolveSlugs exact match', async () => {
|
||||
await engine.putPage('test/exact', testPage);
|
||||
const slugs = await engine.resolveSlugs('test/exact');
|
||||
|
||||
@@ -14,7 +14,7 @@ import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { resolveSourceId, __testing } from '../src/core/source-resolver.ts';
|
||||
import { resolveSourceId, getDefaultSourcePath, __testing } from '../src/core/source-resolver.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// ── Stub engine ────────────────────────────────────────────
|
||||
@@ -174,6 +174,103 @@ describe('resolveSourceId priority 6 — fallback', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ── getDefaultSourcePath ───────────────────────────────────
|
||||
|
||||
describe('getDefaultSourcePath', () => {
|
||||
function makeStubWithPaths(
|
||||
registeredSources: string[],
|
||||
sourcePaths: Record<string, string | null>,
|
||||
defaultKey: string | null,
|
||||
): BrainEngine {
|
||||
return {
|
||||
kind: 'pglite',
|
||||
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
|
||||
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
|
||||
const target = params?.[0];
|
||||
return (registeredSources.includes(target as string)
|
||||
? [{ id: target } as unknown as T]
|
||||
: []);
|
||||
}
|
||||
if (sql.includes('SELECT local_path FROM sources WHERE id = $1')) {
|
||||
const target = params?.[0] as string;
|
||||
if (target in sourcePaths) {
|
||||
return [{ local_path: sourcePaths[target] } as unknown as T];
|
||||
}
|
||||
return [];
|
||||
}
|
||||
if (sql.includes('SELECT id, local_path FROM sources')) {
|
||||
return Object.entries(sourcePaths)
|
||||
.filter(([_, p]) => p !== null)
|
||||
.map(([id, local_path]) => ({ id, local_path }) as unknown as T);
|
||||
}
|
||||
return [];
|
||||
},
|
||||
getConfig: async (key: string) => (key === 'sources.default' ? defaultKey : null),
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
test('returns local_path of resolved default source', async () => {
|
||||
const engine = makeStubWithPaths(['default'], { default: '/path/to/brain' }, null);
|
||||
const path = await getDefaultSourcePath(engine, '/random/dir');
|
||||
expect(path).toBe('/path/to/brain');
|
||||
});
|
||||
|
||||
test('returns null when source has no local_path', async () => {
|
||||
const engine = makeStubWithPaths(['default'], { default: null }, null);
|
||||
const path = await getDefaultSourcePath(engine, '/random/dir');
|
||||
expect(path).toBeNull();
|
||||
});
|
||||
|
||||
test('throws on DB error (does not silently swallow)', async () => {
|
||||
const engine = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async () => {
|
||||
throw new Error('connection refused');
|
||||
},
|
||||
getConfig: async () => null,
|
||||
} as unknown as BrainEngine;
|
||||
await expect(getDefaultSourcePath(engine, '/random/dir')).rejects.toThrow(/connection refused/);
|
||||
});
|
||||
|
||||
test('falls back to legacy sync.repo_path config when sources.local_path is null', async () => {
|
||||
// Pre-v0.18 brains: 'default' source exists but local_path is NULL; the
|
||||
// repo path lives in the global config table under sync.repo_path.
|
||||
const engine = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async <T>(sql: string, params?: unknown[]): Promise<T[]> => {
|
||||
if (sql.includes('SELECT id FROM sources WHERE id = $1')) {
|
||||
return [{ id: params?.[0] } as unknown as T];
|
||||
}
|
||||
if (sql.includes('SELECT local_path FROM sources WHERE id = $1')) {
|
||||
return [{ local_path: null } as unknown as T];
|
||||
}
|
||||
if (sql.includes('SELECT id, local_path FROM sources')) {
|
||||
return [];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
getConfig: async (key: string) => {
|
||||
if (key === 'sources.default') return null;
|
||||
if (key === 'sync.repo_path') return '/legacy/brain/path';
|
||||
return null;
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const path = await getDefaultSourcePath(engine, '/random/dir');
|
||||
expect(path).toBe('/legacy/brain/path');
|
||||
});
|
||||
|
||||
test('respects source resolution chain (registered local_path wins over default)', async () => {
|
||||
// CWD is inside /custom/path → wiki source matches by path → wiki's local_path returned.
|
||||
const engine = makeStubWithPaths(
|
||||
['default', 'wiki'],
|
||||
{ default: '/default/path', wiki: '/custom/path' },
|
||||
'default',
|
||||
);
|
||||
const path = await getDefaultSourcePath(engine, '/custom/path/sub');
|
||||
expect(path).toBe('/custom/path');
|
||||
});
|
||||
});
|
||||
|
||||
// ── Regex validation ───────────────────────────────────────
|
||||
|
||||
describe('SOURCE_ID_RE', () => {
|
||||
|
||||
336
test/storage-config.test.ts
Normal file
336
test/storage-config.test.ts
Normal file
@@ -0,0 +1,336 @@
|
||||
import { test, expect, describe, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
validateStorageConfig,
|
||||
isGitTracked,
|
||||
isSupabaseOnly,
|
||||
getStorageTier,
|
||||
loadStorageConfig,
|
||||
normalizeAndValidateStorageConfig,
|
||||
StorageConfigError,
|
||||
__resetMissingStorageWarning,
|
||||
} from '../src/core/storage-config.ts';
|
||||
import type { StorageConfig } from '../src/core/storage-config.ts';
|
||||
|
||||
describe('Storage Configuration', () => {
|
||||
const testConfig: StorageConfig = {
|
||||
db_tracked: ['people/', 'companies/', 'deals/'],
|
||||
db_only: ['media/x/', 'media/articles/', 'meetings/transcripts/'],
|
||||
};
|
||||
|
||||
describe('validateStorageConfig', () => {
|
||||
test('should return no warnings for valid config', () => {
|
||||
const warnings = validateStorageConfig(testConfig);
|
||||
expect(warnings).toEqual([]);
|
||||
});
|
||||
|
||||
test('should warn about overlap between db_tracked and db_only', () => {
|
||||
const invalidConfig: StorageConfig = {
|
||||
db_tracked: ['people/', 'media/'],
|
||||
db_only: ['media/', 'articles/'],
|
||||
};
|
||||
const warnings = validateStorageConfig(invalidConfig);
|
||||
expect(warnings).toContain('Directory "media/" appears in both db_tracked and db_only');
|
||||
});
|
||||
|
||||
test('should warn about paths not ending with /', () => {
|
||||
const invalidConfig: StorageConfig = {
|
||||
db_tracked: ['people', 'companies/'],
|
||||
db_only: ['media/x/', 'articles'],
|
||||
};
|
||||
const warnings = validateStorageConfig(invalidConfig);
|
||||
expect(warnings).toContain('Directory path "people" should end with "/" for consistency');
|
||||
expect(warnings).toContain('Directory path "articles" should end with "/" for consistency');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Storage tier detection', () => {
|
||||
test('identifies db-tracked pages', () => {
|
||||
expect(isGitTracked('people/john-doe', testConfig)).toBe(true);
|
||||
expect(isGitTracked('companies/acme-corp', testConfig)).toBe(true);
|
||||
expect(isGitTracked('deals/series-a', testConfig)).toBe(true);
|
||||
});
|
||||
|
||||
test('identifies db-only pages', () => {
|
||||
expect(isSupabaseOnly('media/x/tweet-123', testConfig)).toBe(true);
|
||||
expect(isSupabaseOnly('media/articles/blog-post', testConfig)).toBe(true);
|
||||
expect(isSupabaseOnly('meetings/transcripts/standup', testConfig)).toBe(true);
|
||||
});
|
||||
|
||||
test('returns false for non-matching paths', () => {
|
||||
expect(isGitTracked('media/x/tweet-123', testConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('people/john-doe', testConfig)).toBe(false);
|
||||
});
|
||||
|
||||
test('correctly determines storage tier (canonical names)', () => {
|
||||
expect(getStorageTier('people/john-doe', testConfig)).toBe('db_tracked');
|
||||
expect(getStorageTier('media/x/tweet-123', testConfig)).toBe('db_only');
|
||||
expect(getStorageTier('projects/random-thing', testConfig)).toBe('unspecified');
|
||||
});
|
||||
|
||||
test('handles prefix edge cases', () => {
|
||||
expect(isGitTracked('people', testConfig)).toBe(false);
|
||||
expect(isGitTracked('people/', testConfig)).toBe(true);
|
||||
expect(isGitTracked('peoplex/test', testConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('mediax/test', testConfig)).toBe(false);
|
||||
});
|
||||
|
||||
test('normalizeAndValidateStorageConfig auto-adds trailing slash silently with info note', () => {
|
||||
__resetMissingStorageWarning();
|
||||
const warnings: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (...a: unknown[]) => { warnings.push(a.map(String).join(' ')); };
|
||||
try {
|
||||
const out = normalizeAndValidateStorageConfig({
|
||||
db_tracked: ['people', 'companies/'],
|
||||
db_only: ['media/x'],
|
||||
});
|
||||
expect(out.db_tracked).toEqual(['people/', 'companies/']);
|
||||
expect(out.db_only).toEqual(['media/x/']);
|
||||
expect(warnings.length).toBe(1);
|
||||
expect(warnings[0]).toMatch(/normalized.*"people".*"people\/".*"media\/x".*"media\/x\/"/);
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
});
|
||||
|
||||
test('normalizeAndValidateStorageConfig throws on tier overlap', () => {
|
||||
__resetMissingStorageWarning();
|
||||
expect(() =>
|
||||
normalizeAndValidateStorageConfig({
|
||||
db_tracked: ['media/'],
|
||||
db_only: ['media/'],
|
||||
}),
|
||||
).toThrow(StorageConfigError);
|
||||
});
|
||||
|
||||
test('regression — media/xerox does NOT match media/x (path-segment matcher)', () => {
|
||||
// Without path-segment matching, slug.startsWith('media/x') would falsely
|
||||
// match 'media/xerox/foo'. The new matcher requires trailing '/'; if the
|
||||
// user's config has 'media/x' (no slash), the matcher refuses to match —
|
||||
// the validator's auto-normalize (step 7) ensures canonical input.
|
||||
const collisionConfig: StorageConfig = {
|
||||
db_tracked: [],
|
||||
db_only: ['media/x/'], // canonical, with trailing slash
|
||||
};
|
||||
expect(isSupabaseOnly('media/xerox/something', collisionConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('media/x/tweet-1', collisionConfig)).toBe(true);
|
||||
|
||||
// Non-canonical input (no trailing slash) is refused by the matcher.
|
||||
const noSlashConfig: StorageConfig = {
|
||||
db_tracked: [],
|
||||
db_only: ['media/x'],
|
||||
};
|
||||
expect(isSupabaseOnly('media/xerox/foo', noSlashConfig)).toBe(false);
|
||||
expect(isSupabaseOnly('media/x/tweet-1', noSlashConfig)).toBe(false);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadStorageConfig — real-disk loader', () => {
|
||||
let tmp: string;
|
||||
let originalWarn: typeof console.warn;
|
||||
let warnings: string[];
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-storage-test-'));
|
||||
__resetMissingStorageWarning();
|
||||
warnings = [];
|
||||
originalWarn = console.warn;
|
||||
console.warn = (...args: unknown[]) => {
|
||||
warnings.push(args.map(String).join(' '));
|
||||
};
|
||||
});
|
||||
|
||||
function cleanup(): void {
|
||||
console.warn = originalWarn;
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
test('returns null when repoPath is missing', () => {
|
||||
try {
|
||||
expect(loadStorageConfig(undefined)).toBeNull();
|
||||
expect(loadStorageConfig(null)).toBeNull();
|
||||
expect(loadStorageConfig('')).toBeNull();
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('returns null when gbrain.yml does not exist', () => {
|
||||
try {
|
||||
expect(loadStorageConfig(tmp)).toBeNull();
|
||||
expect(warnings).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('loads canonical gbrain.yml — the test that would have caught the original gray-matter P0', () => {
|
||||
try {
|
||||
const yaml = `# Brain storage tiering config
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
- companies/
|
||||
- deals/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config).not.toBeNull();
|
||||
expect(config!.db_tracked).toEqual(['people/', 'companies/', 'deals/']);
|
||||
expect(config!.db_only).toEqual(['media/x/', 'media/articles/']);
|
||||
expect(warnings).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('handles inline comments and blank lines', () => {
|
||||
try {
|
||||
const yaml = `
|
||||
storage:
|
||||
db_tracked:
|
||||
- people/ # human-curated
|
||||
- companies/
|
||||
|
||||
db_only:
|
||||
- media/x/ # bulk tweets
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config!.db_tracked).toEqual(['people/', 'companies/']);
|
||||
expect(config!.db_only).toEqual(['media/x/']);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('strips quoted values', () => {
|
||||
try {
|
||||
const yaml = `storage:
|
||||
db_tracked:
|
||||
- "people/"
|
||||
- 'companies/'
|
||||
db_only: []
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config!.db_tracked).toEqual(['people/', 'companies/']);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('reads deprecated keys (git_tracked / supabase_only) with once-per-process warning', () => {
|
||||
try {
|
||||
const yaml = `storage:
|
||||
git_tracked:
|
||||
- people/
|
||||
supabase_only:
|
||||
- media/x/
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config!.db_tracked).toEqual(['people/']);
|
||||
expect(config!.db_only).toEqual(['media/x/']);
|
||||
expect(warnings.some((w) => /deprecated/.test(w))).toBe(true);
|
||||
|
||||
// Second call: no second deprecation warning (once-per-process).
|
||||
const before = warnings.length;
|
||||
loadStorageConfig(tmp);
|
||||
const newWarnings = warnings.slice(before);
|
||||
expect(newWarnings.filter((w) => /deprecated/.test(w))).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('canonical keys win over deprecated keys when both present', () => {
|
||||
try {
|
||||
const yaml = `storage:
|
||||
db_tracked:
|
||||
- new-people/
|
||||
git_tracked:
|
||||
- old-people/
|
||||
db_only:
|
||||
- new-media/
|
||||
supabase_only:
|
||||
- old-media/
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config!.db_tracked).toEqual(['new-people/']);
|
||||
expect(config!.db_only).toEqual(['new-media/']);
|
||||
// Stronger deprecation warning when both shapes coexist.
|
||||
expect(warnings.some((w) => /deprecated.*ignored/.test(w))).toBe(true);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('warns once when gbrain.yml exists but storage section is missing', () => {
|
||||
try {
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), 'something_else: foo\n');
|
||||
const config = loadStorageConfig(tmp);
|
||||
expect(config).toBeNull();
|
||||
expect(warnings.length).toBe(1);
|
||||
expect(warnings[0]).toMatch(/no storage configuration/);
|
||||
|
||||
// Second call: no additional warning (once-per-process).
|
||||
loadStorageConfig(tmp);
|
||||
expect(warnings.length).toBe(1);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('warns when storage section is empty', () => {
|
||||
try {
|
||||
const yaml = `storage:
|
||||
db_tracked: []
|
||||
db_only: []
|
||||
`;
|
||||
writeFileSync(join(tmp, 'gbrain.yml'), yaml);
|
||||
const config = loadStorageConfig(tmp);
|
||||
// Empty config is returned (not null) but warning fires.
|
||||
expect(config).not.toBeNull();
|
||||
expect(config!.db_tracked).toEqual([]);
|
||||
expect(config!.db_only).toEqual([]);
|
||||
const noConfigWarnings = warnings.filter((w) => /no storage configuration/.test(w));
|
||||
expect(noConfigWarnings.length).toBe(1);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('throws on unreadable gbrain.yml (permission denied) — does not silently disable feature', () => {
|
||||
try {
|
||||
const yamlPath = join(tmp, 'gbrain.yml');
|
||||
writeFileSync(yamlPath, 'storage:\n db_tracked:\n - x/\n');
|
||||
// Simulate unreadable: chmod 000. May not work on all CI; skip if not supported.
|
||||
const fs = require('fs');
|
||||
fs.chmodSync(yamlPath, 0o000);
|
||||
try {
|
||||
// On systems where chmod 000 actually denies read, this throws.
|
||||
// On systems where root can still read (CI containers), the read succeeds
|
||||
// and the test is a no-op assertion.
|
||||
try {
|
||||
fs.readFileSync(yamlPath, 'utf-8');
|
||||
// Read succeeded — skip strict assertion.
|
||||
} catch {
|
||||
expect(() => loadStorageConfig(tmp)).toThrow();
|
||||
}
|
||||
} finally {
|
||||
fs.chmodSync(yamlPath, 0o644);
|
||||
}
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
146
test/storage-export.test.ts
Normal file
146
test/storage-export.test.ts
Normal file
@@ -0,0 +1,146 @@
|
||||
/**
|
||||
* Tests for export.ts --restore-only resolution chain — step 9 of v0.22.3.
|
||||
*
|
||||
* D5: --repo → sources.getDefault() → hard error. Never fall through to
|
||||
* cwd. Issue #9: bare try/catch removed from storage.ts:37.
|
||||
*
|
||||
* Tests use PGLite in-memory and a captured-output approach (process.exit
|
||||
* is intercepted) to verify the resolution chain produces the right
|
||||
* repoPath OR the right error.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExport } from '../src/commands/export.ts';
|
||||
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let tmp: string;
|
||||
let outDir: string;
|
||||
let exitCode: number | null;
|
||||
let originalExit: typeof process.exit;
|
||||
let originalErr: typeof console.error;
|
||||
let originalLog: typeof console.log;
|
||||
let stderr: string[];
|
||||
let stdout: string[];
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-export-test-'));
|
||||
outDir = join(tmp, 'out');
|
||||
exitCode = null;
|
||||
stderr = [];
|
||||
stdout = [];
|
||||
__resetMissingStorageWarning();
|
||||
|
||||
originalExit = process.exit;
|
||||
process.exit = ((code?: number) => {
|
||||
exitCode = code ?? 0;
|
||||
throw new Error(`__test_exit__:${code}`);
|
||||
}) as typeof process.exit;
|
||||
|
||||
originalErr = console.error;
|
||||
console.error = (...args: unknown[]) => {
|
||||
stderr.push(args.map(String).join(' '));
|
||||
};
|
||||
|
||||
originalLog = console.log;
|
||||
console.log = (...args: unknown[]) => {
|
||||
stdout.push(args.map(String).join(' '));
|
||||
};
|
||||
|
||||
// Reset DB state between tests
|
||||
const tables = ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages', 'sources'];
|
||||
for (const t of tables) {
|
||||
await (engine as unknown as { db: { exec(sql: string): Promise<unknown> } }).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
// Recreate the default source (the schema seed but truncated above).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('default', 'Default') ON CONFLICT DO NOTHING`,
|
||||
);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.exit = originalExit;
|
||||
console.error = originalErr;
|
||||
console.log = originalLog;
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function tryRunExport(args: string[]): Promise<void> {
|
||||
try {
|
||||
await runExport(engine, args);
|
||||
} catch (e) {
|
||||
// Swallow only the test-exit sentinel; rethrow others for visibility.
|
||||
if (!(e instanceof Error && e.message.startsWith('__test_exit__:'))) {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
describe('export --restore-only resolution chain (D5)', () => {
|
||||
test('hard-errors when --restore-only has no --repo and no default source path', async () => {
|
||||
// sources.default has no local_path (the seeded shape).
|
||||
await tryRunExport(['--dir', outDir, '--restore-only']);
|
||||
expect(exitCode).toBe(1);
|
||||
expect(stderr.join('\n')).toMatch(/requires --repo|configured default source/);
|
||||
});
|
||||
|
||||
test('uses explicit --repo when provided', async () => {
|
||||
// Make a brain repo with gbrain.yml that has empty db_only — so we
|
||||
// exit through the "0 pages to restore" path without needing real data.
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked: []
|
||||
db_only: []
|
||||
`,
|
||||
);
|
||||
await tryRunExport(['--dir', outDir, '--restore-only', '--repo', tmp]);
|
||||
expect(exitCode).toBeNull(); // no exit
|
||||
expect(stdout.some((line) => line.includes('Restoring 0'))).toBe(true);
|
||||
});
|
||||
|
||||
test('falls back to sources default local_path when --repo absent', async () => {
|
||||
// Configure default source path, write a real gbrain.yml so the storage
|
||||
// config check passes — without gbrain.yml the Codex-P0 guard correctly
|
||||
// refuses --restore-only (no storage config to scope to).
|
||||
await engine.executeRaw(`UPDATE sources SET local_path = $1 WHERE id = 'default'`, [tmp]);
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:\n db_tracked: []\n db_only:\n - media/x/\n`,
|
||||
);
|
||||
await tryRunExport(['--dir', outDir, '--restore-only']);
|
||||
expect(exitCode).toBeNull(); // resolution succeeded
|
||||
});
|
||||
|
||||
test('refuses --restore-only when no storage config is present (Codex P0)', async () => {
|
||||
// Default source has a path but no gbrain.yml. Without a storage config,
|
||||
// --restore-only would silently fall through to a full export — exactly
|
||||
// the silent-footgun D5 was supposed to prevent.
|
||||
await engine.executeRaw(`UPDATE sources SET local_path = $1 WHERE id = 'default'`, [tmp]);
|
||||
await tryRunExport(['--dir', outDir, '--restore-only']);
|
||||
expect(exitCode).toBe(1);
|
||||
expect(stderr.join('\n')).toMatch(/storage tiering config|gbrain\.yml/);
|
||||
});
|
||||
|
||||
test('non-restore export does NOT require --repo (D26)', async () => {
|
||||
// Regular export works without --repo since it dumps everything from DB.
|
||||
// Pages table is empty → exports 0 pages, no error.
|
||||
await tryRunExport(['--dir', outDir]);
|
||||
expect(exitCode).toBeNull();
|
||||
expect(stdout.some((line) => line.includes('Exporting 0'))).toBe(true);
|
||||
});
|
||||
});
|
||||
171
test/storage-pglite.test.ts
Normal file
171
test/storage-pglite.test.ts
Normal file
@@ -0,0 +1,171 @@
|
||||
/**
|
||||
* PGLite lifecycle test for storage tiering — D8 + D4 of v0.22.3.
|
||||
*
|
||||
* Per the plan: "the full PGLite lifecycle for D8's both-engines requirement.
|
||||
* gbrain.yml load → gbrain storage status → soft-warn message present →
|
||||
* manageGitignore happy-path on a tmp dir. PGLite-specific path for the
|
||||
* slugPrefix filter."
|
||||
*
|
||||
* In-memory PGLite, no Docker, no DATABASE_URL. Runs instantly in CI.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
getStorageStatus,
|
||||
formatStorageStatusHuman,
|
||||
__resetPGLiteWarn,
|
||||
} from '../src/commands/storage.ts';
|
||||
import { manageGitignore, __resetPGLiteTierWarn } from '../src/commands/sync.ts';
|
||||
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let tmp: string;
|
||||
let warnings: string[];
|
||||
let originalWarn: typeof console.warn;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-pglite-test-'));
|
||||
__resetMissingStorageWarning();
|
||||
__resetPGLiteWarn();
|
||||
__resetPGLiteTierWarn();
|
||||
warnings = [];
|
||||
originalWarn = console.warn;
|
||||
console.warn = (...args: unknown[]) => {
|
||||
warnings.push(args.map(String).join(' '));
|
||||
};
|
||||
|
||||
// Reset DB between tests.
|
||||
const tables = ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages', 'sources'];
|
||||
for (const t of tables) {
|
||||
await (engine as unknown as { db: { exec(sql: string): Promise<unknown> } }).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ('default', 'Default') ON CONFLICT DO NOTHING`,
|
||||
);
|
||||
});
|
||||
|
||||
function cleanup(): void {
|
||||
console.warn = originalWarn;
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeGbrainYml(): void {
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
db_only:
|
||||
- media/x/
|
||||
`,
|
||||
);
|
||||
}
|
||||
|
||||
describe('Storage tiering on PGLite — full lifecycle (D8 + D4)', () => {
|
||||
test('engine.kind is pglite', () => {
|
||||
try {
|
||||
expect(engine.kind).toBe('pglite');
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('getStorageStatus loads gbrain.yml and reports tier counts', async () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'Tweet', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('media/x/tweet-2', { type: 'concept', title: 'Tweet 2', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('random/note', { type: 'concept', title: 'Random', compiled_truth: '', timeline: '' });
|
||||
|
||||
const result = await getStorageStatus(engine, tmp);
|
||||
expect(result.totalPages).toBe(4);
|
||||
expect(result.pagesByTier.db_tracked).toBe(1);
|
||||
expect(result.pagesByTier.db_only).toBe(2);
|
||||
expect(result.pagesByTier.unspecified).toBe(1);
|
||||
expect(result.config!.db_only).toEqual(['media/x/']);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('manageGitignore on PGLite emits the D4 soft-warn (once per process)', () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
manageGitignore(tmp, 'pglite');
|
||||
expect(warnings.some((w) => /limited effect on PGLite/.test(w))).toBe(true);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
|
||||
expect(readFileSync(join(tmp, '.gitignore'), 'utf-8')).toContain('media/x/');
|
||||
|
||||
// Second call: no second warning (once-per-process).
|
||||
const before = warnings.length;
|
||||
manageGitignore(tmp, 'pglite');
|
||||
const newWarnings = warnings.slice(before).filter((w) => /limited effect on PGLite/.test(w));
|
||||
expect(newWarnings).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('manageGitignore on Postgres does NOT emit the PGLite warning', () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
manageGitignore(tmp, 'postgres');
|
||||
expect(warnings.filter((w) => /limited effect on PGLite/.test(w))).toEqual([]);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('slugPrefix engine filter works on PGLite (Issue #13)', async () => {
|
||||
try {
|
||||
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'T1', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('media/x/tweet-2', { type: 'concept', title: 'T2', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('media/articles/post-1', { type: 'concept', title: 'A1', compiled_truth: '', timeline: '' });
|
||||
|
||||
const xOnly = await engine.listPages({ slugPrefix: 'media/x/', limit: 100 });
|
||||
expect(xOnly.map((p) => p.slug).sort()).toEqual(['media/x/tweet-1', 'media/x/tweet-2']);
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
test('end-to-end: gbrain.yml + putPage + storage status + .gitignore', async () => {
|
||||
try {
|
||||
writeGbrainYml();
|
||||
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('media/x/tweet-1', { type: 'concept', title: 'T1', compiled_truth: '', timeline: '' });
|
||||
|
||||
// Status reads tier counts correctly.
|
||||
const status = await getStorageStatus(engine, tmp);
|
||||
expect(status.config).not.toBeNull();
|
||||
expect(status.pagesByTier.db_only).toBe(1);
|
||||
|
||||
// Render to human output without errors.
|
||||
const out = formatStorageStatusHuman(status);
|
||||
expect(out).toContain('DB only: 1 pages');
|
||||
|
||||
// .gitignore management produces a managed block.
|
||||
manageGitignore(tmp, 'pglite');
|
||||
const gitignore = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
expect(gitignore).toContain('# Auto-managed by gbrain');
|
||||
expect(gitignore).toContain('media/x/');
|
||||
} finally {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
111
test/storage-status.test.ts
Normal file
111
test/storage-status.test.ts
Normal file
@@ -0,0 +1,111 @@
|
||||
/**
|
||||
* Tests for storage-status formatters — step 10 of v0.22.3.
|
||||
*
|
||||
* Issue #10 + D14: split storage.ts into pure data + JSON formatter +
|
||||
* human formatter (matching orphans.ts). Formatters are now pure
|
||||
* functions; this test file pins their output contracts.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
formatStorageStatusJson,
|
||||
formatStorageStatusHuman,
|
||||
type StorageStatusResult,
|
||||
} from '../src/commands/storage.ts';
|
||||
|
||||
const baseResult: StorageStatusResult = {
|
||||
config: {
|
||||
db_tracked: ['people/', 'companies/'],
|
||||
db_only: ['media/x/', 'media/articles/'],
|
||||
},
|
||||
repoPath: '/data/brain',
|
||||
totalPages: 12500,
|
||||
pagesByTier: { db_tracked: 2156, db_only: 10100, unspecified: 244 },
|
||||
missingFiles: [],
|
||||
diskUsageByTier: { db_tracked: 45_200_000, db_only: 2_100_000_000, unspecified: 0 },
|
||||
warnings: [],
|
||||
};
|
||||
|
||||
describe('formatStorageStatusJson', () => {
|
||||
test('produces parseable JSON of the StorageStatusResult shape', () => {
|
||||
const out = formatStorageStatusJson(baseResult);
|
||||
const parsed = JSON.parse(out);
|
||||
expect(parsed.repoPath).toBe('/data/brain');
|
||||
expect(parsed.totalPages).toBe(12500);
|
||||
expect(parsed.pagesByTier.db_only).toBe(10100);
|
||||
expect(parsed.config.db_tracked).toEqual(['people/', 'companies/']);
|
||||
});
|
||||
|
||||
test('handles null config (no gbrain.yml present)', () => {
|
||||
const out = formatStorageStatusJson({ ...baseResult, config: null, totalPages: 5 });
|
||||
const parsed = JSON.parse(out);
|
||||
expect(parsed.config).toBeNull();
|
||||
expect(parsed.totalPages).toBe(5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('formatStorageStatusHuman', () => {
|
||||
test('shows tier counts and disk usage when config present', () => {
|
||||
const out = formatStorageStatusHuman(baseResult);
|
||||
expect(out).toContain('Storage Status');
|
||||
expect(out).toContain('Repository: /data/brain');
|
||||
expect(out).toContain('Total pages: 12500');
|
||||
expect(out).toContain('DB tracked: 2,156 pages');
|
||||
expect(out).toContain('DB only: 10,100 pages');
|
||||
expect(out).toContain('Unspecified: 244 pages');
|
||||
});
|
||||
|
||||
test('shows ASCII separators only — no unicode (D10)', () => {
|
||||
const out = formatStorageStatusHuman(baseResult);
|
||||
expect(out).not.toContain('─'); // U+2500 box drawing
|
||||
expect(out).not.toContain('•'); // U+2022 bullet
|
||||
expect(out).toContain('-------------'); // ASCII fallback
|
||||
});
|
||||
|
||||
test('shows fallback message when config is null', () => {
|
||||
const out = formatStorageStatusHuman({ ...baseResult, config: null });
|
||||
expect(out).toContain('No gbrain.yml configuration found.');
|
||||
expect(out).toContain('All pages are stored in git by default.');
|
||||
});
|
||||
|
||||
test('shows missing-files block when list is non-empty, capped at 10', () => {
|
||||
const missing = Array.from({ length: 25 }, (_, i) => ({
|
||||
slug: `media/x/tweet-${i}`,
|
||||
expectedPath: `/data/brain/media/x/tweet-${i}.md`,
|
||||
}));
|
||||
const out = formatStorageStatusHuman({ ...baseResult, missingFiles: missing });
|
||||
expect(out).toContain('Missing Files (need restore):');
|
||||
expect(out).toContain('media/x/tweet-0');
|
||||
expect(out).toContain('media/x/tweet-9'); // 10th
|
||||
expect(out).not.toContain('media/x/tweet-10'); // 11th truncated
|
||||
expect(out).toContain('and 15 more');
|
||||
expect(out).toContain('gbrain export --restore-only --repo "/data/brain"');
|
||||
});
|
||||
|
||||
test('shows configuration listing for both tiers', () => {
|
||||
const out = formatStorageStatusHuman(baseResult);
|
||||
expect(out).toContain('DB tracked directories:');
|
||||
expect(out).toContain(' - people/');
|
||||
expect(out).toContain(' - companies/');
|
||||
expect(out).toContain('DB-only directories:');
|
||||
expect(out).toContain(' - media/x/');
|
||||
expect(out).toContain(' - media/articles/');
|
||||
});
|
||||
|
||||
test('shows warnings inline when present', () => {
|
||||
const out = formatStorageStatusHuman({
|
||||
...baseResult,
|
||||
warnings: ['Directory path "people" should end with "/" for consistency'],
|
||||
});
|
||||
expect(out).toContain('Warnings:');
|
||||
expect(out).toContain('! Directory path "people"');
|
||||
});
|
||||
|
||||
test('omits disk-usage block when both tiers report 0 bytes', () => {
|
||||
const out = formatStorageStatusHuman({
|
||||
...baseResult,
|
||||
diskUsageByTier: { db_tracked: 0, db_only: 0, unspecified: 0 },
|
||||
});
|
||||
expect(out).not.toContain('Disk Usage:');
|
||||
});
|
||||
});
|
||||
144
test/storage-sync.test.ts
Normal file
144
test/storage-sync.test.ts
Normal file
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* Tests for sync.ts manageGitignore() — step 8 of v0.22.3 storage tiering.
|
||||
*
|
||||
* Issue #2: function was defined but never invoked. Now wired into runSync
|
||||
* after a successful sync (skips on dry_run / blocked_by_failures / failure).
|
||||
*
|
||||
* Tests cover: happy path, idempotency, GBRAIN_NO_GITIGNORE escape hatch,
|
||||
* submodule detection, write-error graceful degradation, and the "no
|
||||
* config — no-op" path.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync, readFileSync, existsSync, chmodSync, symlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { manageGitignore } from '../src/commands/sync.ts';
|
||||
import { __resetMissingStorageWarning } from '../src/core/storage-config.ts';
|
||||
|
||||
let tmp: string;
|
||||
let warnings: string[];
|
||||
let originalWarn: typeof console.warn;
|
||||
let originalEnv: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-mgi-test-'));
|
||||
__resetMissingStorageWarning();
|
||||
warnings = [];
|
||||
originalWarn = console.warn;
|
||||
console.warn = (...args: unknown[]) => {
|
||||
warnings.push(args.map(String).join(' '));
|
||||
};
|
||||
originalEnv = process.env.GBRAIN_NO_GITIGNORE;
|
||||
delete process.env.GBRAIN_NO_GITIGNORE;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
console.warn = originalWarn;
|
||||
if (originalEnv === undefined) delete process.env.GBRAIN_NO_GITIGNORE;
|
||||
else process.env.GBRAIN_NO_GITIGNORE = originalEnv;
|
||||
// Restore permissions for cleanup.
|
||||
try {
|
||||
chmodSync(tmp, 0o755);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeStorageConfig(): void {
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
db_only:
|
||||
- media/x/
|
||||
- media/articles/
|
||||
`,
|
||||
);
|
||||
}
|
||||
|
||||
describe('manageGitignore', () => {
|
||||
test('no-op when gbrain.yml is absent', () => {
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
expect(warnings).toEqual([]);
|
||||
});
|
||||
|
||||
test('no-op when storage config has empty db_only', () => {
|
||||
writeFileSync(
|
||||
join(tmp, 'gbrain.yml'),
|
||||
`storage:
|
||||
db_tracked:
|
||||
- people/
|
||||
db_only: []
|
||||
`,
|
||||
);
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
});
|
||||
|
||||
test('appends db_only directories to .gitignore — happy path', () => {
|
||||
writeStorageConfig();
|
||||
manageGitignore(tmp);
|
||||
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
expect(content).toContain('# Auto-managed by gbrain');
|
||||
expect(content).toContain('media/x/');
|
||||
expect(content).toContain('media/articles/');
|
||||
});
|
||||
|
||||
test('idempotent — running twice does NOT duplicate entries', () => {
|
||||
writeStorageConfig();
|
||||
manageGitignore(tmp);
|
||||
manageGitignore(tmp);
|
||||
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
const xCount = (content.match(/^media\/x\/$/gm) || []).length;
|
||||
const articlesCount = (content.match(/^media\/articles\/$/gm) || []).length;
|
||||
expect(xCount).toBe(1);
|
||||
expect(articlesCount).toBe(1);
|
||||
});
|
||||
|
||||
test('preserves user-written .gitignore entries', () => {
|
||||
writeStorageConfig();
|
||||
writeFileSync(join(tmp, '.gitignore'), '# my own rules\n*.swp\nnode_modules/\n');
|
||||
manageGitignore(tmp);
|
||||
const content = readFileSync(join(tmp, '.gitignore'), 'utf-8');
|
||||
expect(content).toContain('# my own rules');
|
||||
expect(content).toContain('*.swp');
|
||||
expect(content).toContain('node_modules/');
|
||||
expect(content).toContain('media/x/');
|
||||
});
|
||||
|
||||
test('GBRAIN_NO_GITIGNORE=1 skips entirely', () => {
|
||||
writeStorageConfig();
|
||||
process.env.GBRAIN_NO_GITIGNORE = '1';
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
});
|
||||
|
||||
test('skips with actionable warning when repo is a git submodule', () => {
|
||||
writeStorageConfig();
|
||||
// Submodule: .git is a file containing `gitdir: ...` instead of a directory.
|
||||
writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/modules/sub\n');
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
expect(warnings.some((w) => /submodule/.test(w))).toBe(true);
|
||||
});
|
||||
|
||||
test('proceeds when .git is a directory (regular repo)', () => {
|
||||
writeStorageConfig();
|
||||
mkdirSync(join(tmp, '.git'));
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
|
||||
expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]);
|
||||
});
|
||||
|
||||
test('warns and skips when .gitignore write fails (read-only filesystem simulation)', () => {
|
||||
writeStorageConfig();
|
||||
// Create a .gitignore as a directory — write to that path will fail with EISDIR.
|
||||
mkdirSync(join(tmp, '.gitignore'));
|
||||
manageGitignore(tmp);
|
||||
expect(warnings.some((w) => /Could not (read|update)/.test(w))).toBe(true);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user