feat: GBrain v0.2.0 — incremental sync, file storage, install skill (#2)
* refactor: extract importFile from import.ts + add tag reconciliation Shared single-file import function used by both import and sync. Adds tag reconciliation (removes stale tags on reimport), >1MB file skip, and import->sync checkpoint continuity (writes git HEAD to config table after import so sync picks up seamlessly). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add sync pure functions, updateSlug engine method, and sync tests - buildSyncManifest: parses git diff --name-status -M output - isSyncable: filters to .md pages, excludes hidden/ops/.raw/skip-list - pathToSlug: converts file paths to page slugs with optional prefix - updateSlug: renames page slug in-place (preserves page_id, chunks, embeddings) - rewriteLinks: stub for v0.2 (FKs use page_id, already correct) - 20 new tests, all passing (39 total across 3 files) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain sync command with CLI, MCP, and watch mode 18-step sync protocol: read config, git pull, ancestry validation, git diff --name-status -M for net changes, isSyncable filter, process deletes/renames/adds/modifies via importFile, batch optimization, sync state checkpoint in Postgres config table. Watch mode with polling and consecutive error counter. MCP sync_brain tool returns structured SyncResult. Stale page deletion for un-syncable files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add files table, gbrain files commands, and config show redaction - files table: page_slug FK with ON DELETE SET NULL + ON UPDATE CASCADE, storage_path, storage_url, mime_type, content_hash for dedup - gbrain files list/upload/sync/verify commands for Supabase Storage - gbrain config show redacts postgresql:// passwords and secret keys - CLI help updated with FILES section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add install skill for GBrain onboarding 6-phase install workflow: environment discovery, Supabase setup (magic path via CLI OAuth or fallback 2-copy-paste), init + import, ongoing sync cron, optional file migration with mandatory verification, and agent teaching (AGENTS.md rules). Every error gets what + why + fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.2.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add v0.2 features to README (sync, files, install skill) README.md: added sync command to IMPORT/EXPORT section, added FILES section with 4 commands, added files table to schema diagram, added install skill to skills table, updated MCP tools count from 20 to 21 (sync_brain added). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: OpenClaw DX improvements (skill count, upgrade docs, config show help) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: consolidate version to single source of truth Create src/version.ts that reads from package.json via static import (safe for bun compiled binaries). Update mcp/server.ts from hardcoded '0.1.0' to use shared VERSION. Bump skills/manifest.json to 0.2.0. * fix: upgrade detection order, npm→bun naming, clawhub false positives Reorder detection: node_modules first, binary second, clawhub last. Rename 'npm' install method to 'bun'. Use 'clawhub --version' instead of 'which clawhub' to avoid false positives from dangling symlinks. Add 120s timeout to execSync calls to prevent hanging. Add --help flag. * feat: per-command --help, unknown command check before DB connection Add COMMAND_HELP map covering all 28 commands. Check --help before init/upgrade dispatch and before connectEngine() so help works without a database. Use COMMAND_HELP keys as known-command set to catch unknown commands before wasting a DB round-trip. * docs: standardize npm references to bun, add Upgrade section to README Fix init.ts: npx→bunx, npm→bun for supabase CLI guidance. Fix README: npm install→bun add for standalone CLI install. Add ## Upgrade section to README with all three install methods. Update install skill Upgrading section to list bun, ClawHub, and binary. * test: full coverage audit — CLI dispatch, upgrade detection, config, edge cases New test files: - test/cli.test.ts: COMMAND_HELP ↔ switch consistency, version from package.json, per-command --help, unknown command handling, global help - test/upgrade.test.ts: detection order verification, npm→bun naming, clawhub --version (not which), timeout presence - test/config.test.ts: redactUrl for postgresql URLs, edge cases Extended existing tests: - test/sync.test.ts: empty string pathToSlug, uppercase .MD rejection, deeply nested files, multiple renames, unknown status codes - test/markdown.test.ts: multiple --- separators, missing frontmatter, no frontmatter at all, empty string, type inference from paths Tests: 39 → 83 (+44 new). All pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 100% coverage — import-file mock engine, files utils, chunker edge cases New test files: - test/import-file.test.ts (9 tests): mock BrainEngine to test importFile without DB — MAX_FILE_SIZE skip, content_hash dedup, tag reconciliation (remove stale + add new), compiled_truth/timeline chunking, noEmbed flag, sequential chunk_index - test/files.test.ts (22 tests): getMimeType for all extensions + uppercase + unknown + no-extension, fileHash consistency + different content + empty, collectFiles pattern (skip .md, skip hidden dirs, recurse, sorted output) Extended: - test/chunkers/recursive.test.ts (+6 tests): single newline splits, word-only text, clause delimiters, lossless preservation, default options, mixed delimiter hierarchy Tests: 83 → 118 (+35 new). All pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
15
CHANGELOG.md
15
CHANGELOG.md
@@ -2,6 +2,21 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.2.0] - 2026-04-05
|
||||
|
||||
### Added
|
||||
|
||||
- You can now keep your brain current with `gbrain sync`, which uses git's own diff machinery to process only what changed. No more 30-second full directory walks when 3 files changed.
|
||||
- Watch mode (`gbrain sync --watch`) polls for changes and syncs automatically. Set it and forget it.
|
||||
- Binary file management with `gbrain files` commands (list, upload, sync, verify). Store images, PDFs, and audio in Supabase Storage instead of clogging your git repo.
|
||||
- Install skill (`skills/install/SKILL.md`) that walks you through setup from scratch, including Supabase CLI magic path for zero-copy-paste onboarding.
|
||||
- Import and sync now share a checkpoint. Run `gbrain import`, then `gbrain sync`, and it picks up right where import left off. Zero gap.
|
||||
- Tag reconciliation on reimport. If you remove a tag from your markdown, it actually gets removed from the database now.
|
||||
- `gbrain config show` redacts database passwords so you can safely share your config.
|
||||
- `updateSlug` engine method preserves page identity (page_id, chunks, embeddings) across renames. Zero re-embedding cost.
|
||||
- `sync_brain` MCP tool returns structured results so agents know exactly what changed.
|
||||
- 20 new sync tests (39 total across 3 test files)
|
||||
|
||||
## [0.1.0] - 2026-04-05
|
||||
|
||||
### Added
|
||||
|
||||
16
CLAUDE.md
16
CLAUDE.md
@@ -14,11 +14,13 @@ use the tools — ingest meetings, answer queries, maintain the brain, enrich fr
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/core/import-file.ts` — Shared single-file import (used by import + sync)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion)
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided)
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/mcp/server.ts` — MCP stdio server exposing all tools
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL (includes files table)
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -26,15 +28,17 @@ Run `gbrain --help` or `gbrain --tools-json` for full command reference.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. Tests: `test/markdown.test.ts` (frontmatter parsing,
|
||||
round-trip serialization), `test/chunkers/recursive.test.ts` (delimiter splitting,
|
||||
overlap, chunk sizing). Future: `test/import.test.ts` for full import/export round-trip.
|
||||
`bun test` runs all tests (39 tests across 3 files). Tests: `test/markdown.test.ts`
|
||||
(frontmatter parsing, round-trip serialization), `test/chunkers/recursive.test.ts`
|
||||
(delimiter splitting, overlap, chunk sizing), `test/sync.test.ts` (manifest parsing,
|
||||
isSyncable filtering, pathToSlug conversion).
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. They contain the
|
||||
workflows, heuristics, and quality rules for ingestion, querying, maintenance, and
|
||||
enrichment.
|
||||
workflows, heuristics, and quality rules for ingestion, querying, maintenance,
|
||||
enrichment, and installation. 7 skills: ingest, query, maintain, enrich, briefing,
|
||||
migrate, install.
|
||||
|
||||
## Build
|
||||
|
||||
|
||||
41
README.md
41
README.md
@@ -96,7 +96,7 @@ You: "Install gbrain and set up my knowledge brain.
|
||||
4. Read the skill files in skills/ so you know how to use the brain"
|
||||
```
|
||||
|
||||
OpenClaw will install the package, walk through the Supabase connection wizard, import demo data, and learn the 6 brain skills (ingest, query, maintain, enrich, briefing, migrate).
|
||||
OpenClaw will install the package, walk through the Supabase connection wizard, import demo data, and learn the 7 brain skills (ingest, query, maintain, enrich, briefing, migrate, install).
|
||||
|
||||
After setup, you talk to your brain through OpenClaw:
|
||||
|
||||
@@ -109,18 +109,20 @@ You: "Import my Obsidian vault into the brain"
|
||||
|
||||
OpenClaw reads the skill files in `skills/`, figures out which gbrain commands to run, and does the work. You never touch the CLI directly unless you want to.
|
||||
|
||||
GBrain keeps your brain current automatically. After setup, `gbrain sync --watch` polls your git repo and imports only what changed. Binary files (images, PDFs, audio) can be moved to Supabase Storage with `gbrain files sync` to slim down your git repo.
|
||||
|
||||
### With ClawHub
|
||||
|
||||
```bash
|
||||
clawhub install gbrain
|
||||
```
|
||||
|
||||
This installs the npm package, copies the skill files, and runs `gbrain init --supabase` on first use.
|
||||
This installs the package, copies the skill files, and runs `gbrain init --supabase` on first use.
|
||||
|
||||
### Standalone CLI
|
||||
|
||||
```bash
|
||||
npm install -g gbrain
|
||||
bun add -g gbrain
|
||||
```
|
||||
|
||||
### As a library
|
||||
@@ -135,6 +137,23 @@ import { PostgresEngine } from 'gbrain';
|
||||
|
||||
All paths require a Postgres database with pgvector. Supabase Pro ($25/mo) is the recommended zero-ops option.
|
||||
|
||||
## Upgrade
|
||||
|
||||
Upgrade depends on how you installed:
|
||||
|
||||
```bash
|
||||
# Installed via bun (standalone or library)
|
||||
bun update gbrain
|
||||
|
||||
# Installed via ClawHub
|
||||
clawhub update gbrain
|
||||
|
||||
# Compiled binary
|
||||
# Download the latest from https://github.com/garrytan/gbrain/releases
|
||||
```
|
||||
|
||||
After upgrading, run `gbrain init` again to apply any schema migrations (idempotent, safe to re-run).
|
||||
|
||||
## Setup
|
||||
|
||||
After installing via CLI or library path, run the setup wizard:
|
||||
@@ -269,9 +288,13 @@ page_versions Snapshot history for compiled_truth
|
||||
raw_data Sidecar JSON from external APIs
|
||||
page_id, source, data (JSONB)
|
||||
|
||||
files Binary attachments in Supabase Storage
|
||||
page_slug (FK) Links to pages (ON UPDATE CASCADE)
|
||||
storage_path, storage_url, content_hash, mime_type, metadata (JSONB)
|
||||
|
||||
ingest_log Audit trail of import/ingest operations
|
||||
|
||||
config Brain-level settings (embedding model, chunk strategy)
|
||||
config Brain-level settings (embedding model, chunk strategy, sync state)
|
||||
```
|
||||
|
||||
Indexes: B-tree on slug/type, GIN on frontmatter/search_vector, HNSW on embeddings, pg_trgm on title for fuzzy slug resolution.
|
||||
@@ -305,8 +328,15 @@ SEARCH
|
||||
|
||||
IMPORT/EXPORT
|
||||
gbrain import <dir> [--no-embed] Import markdown directory (idempotent)
|
||||
gbrain sync [--repo <path>] [flags] Git-to-brain incremental sync
|
||||
gbrain export [--dir ./out/] Export to markdown (round-trip)
|
||||
|
||||
FILES
|
||||
gbrain files list [slug] List stored files
|
||||
gbrain files upload <file> --page <slug> Upload file to storage
|
||||
gbrain files sync <dir> Bulk upload directory
|
||||
gbrain files verify Verify all uploads
|
||||
|
||||
EMBEDDINGS
|
||||
gbrain embed [<slug>|--all|--stale] Generate/refresh embeddings
|
||||
|
||||
@@ -386,7 +416,7 @@ Add to your Claude Code or Cursor MCP config:
|
||||
}
|
||||
```
|
||||
|
||||
20 tools: get_page, put_page, delete_page, list_pages, search, query, add_tag, remove_tag, get_tags, add_link, remove_link, get_links, get_backlinks, traverse_graph, add_timeline_entry, get_timeline, get_stats, get_health, get_versions, revert_version.
|
||||
21 tools: get_page, put_page, delete_page, list_pages, search, query, add_tag, remove_tag, get_tags, add_link, remove_link, get_links, get_backlinks, traverse_graph, add_timeline_entry, get_timeline, get_stats, get_health, get_versions, revert_version, sync_brain.
|
||||
|
||||
Every tool mirrors a CLI command. Drift tests verify identical behavior.
|
||||
|
||||
@@ -402,6 +432,7 @@ Fat markdown files that tell AI agents HOW to use gbrain. No skill logic in the
|
||||
| **enrich** | Enrich pages from external APIs. Raw data stored separately, distilled highlights go to compiled truth. |
|
||||
| **briefing** | Daily briefing: today's meetings with participant context, active deals with deadlines, time-sensitive threads, recent changes. |
|
||||
| **migrate** | Universal migration from Obsidian (wikilinks to gbrain links), Notion (stripped UUIDs), Logseq (block refs), plain markdown, CSV, JSON, Roam. |
|
||||
| **install** | Set up GBrain from scratch: Supabase setup (magic path via CLI or 2-copy-paste fallback), import, sync cron, optional file migration, agent teaching. |
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
210
skills/install/SKILL.md
Normal file
210
skills/install/SKILL.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Install GBrain
|
||||
|
||||
Set up GBrain from scratch. The agent drives the process, the human provides secrets and approvals.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Supabase account (Pro tier recommended: $25/mo for 8GB DB + 100GB storage)
|
||||
- An OpenAI API key (for semantic search embeddings, ~$4-5 for 7,500 pages)
|
||||
- A git-backed markdown knowledge base (or start fresh)
|
||||
|
||||
## Phase 1: Environment Discovery
|
||||
|
||||
Scan the environment to understand what we're working with.
|
||||
|
||||
```bash
|
||||
# Find all git repos with markdown content
|
||||
echo "=== GBrain Environment Discovery ==="
|
||||
for dir in /data/* ~/git/* ~/Documents/* 2>/dev/null; do
|
||||
if [ -d "$dir/.git" ]; then
|
||||
md_count=$(find "$dir" -name "*.md" -not -path "*/node_modules/*" -not -path "*/.git/*" 2>/dev/null | wc -l | tr -d ' ')
|
||||
if [ "$md_count" -gt 10 ]; then
|
||||
total_size=$(du -sh "$dir" 2>/dev/null | cut -f1)
|
||||
binary_count=$(find "$dir" -not -name "*.md" -not -path "*/node_modules/*" -not -path "*/.git/*" -type f \( -name "*.jpg" -o -name "*.png" -o -name "*.pdf" -o -name "*.mp4" -o -name "*.m4a" -o -name "*.heic" -o -name "*.tiff" -o -name "*.dng" \) 2>/dev/null | wc -l | tr -d ' ')
|
||||
echo ""
|
||||
echo " $dir ($total_size, $md_count .md files, $binary_count binary files)"
|
||||
# Detect knowledge base type
|
||||
if [ -d "$dir/.obsidian" ]; then
|
||||
echo " Type: Obsidian vault (detected, wikilink conversion needed in future release)"
|
||||
elif [ -d "$dir/logseq" ]; then
|
||||
echo " Type: Logseq (detected, block-ref conversion needed in future release)"
|
||||
else
|
||||
echo " Type: Plain markdown (ready for import)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
echo "=== Discovery Complete ==="
|
||||
```
|
||||
|
||||
Present findings to the human. Recommend which repos to import.
|
||||
|
||||
## Phase 2: Supabase Setup
|
||||
|
||||
### Magic Path (zero copy-pastes)
|
||||
|
||||
Check if the Supabase CLI is available:
|
||||
|
||||
```bash
|
||||
which supabase 2>/dev/null || npx supabase --version 2>/dev/null
|
||||
```
|
||||
|
||||
If available, use the magic path:
|
||||
|
||||
1. Tell the human: "I'll set up Supabase for you. Click 'Authorize' when your browser opens."
|
||||
2. Run `supabase login` (opens browser for OAuth)
|
||||
3. Run `supabase projects create --name gbrain --region us-east-1`
|
||||
4. Extract credentials from `supabase projects api-keys`
|
||||
5. Proceed to Phase 3 automatically
|
||||
|
||||
### Fallback Path (2 copy-pastes)
|
||||
|
||||
If the Supabase CLI is not available, tell the human exactly what to do:
|
||||
|
||||
1. "Log into Supabase and add a credit card: https://supabase.com/dashboard/account/billing"
|
||||
2. "Create a new project: https://supabase.com/dashboard/new/_"
|
||||
- Name: gbrain
|
||||
- Region: closest to you
|
||||
- Generate a strong password
|
||||
3. "Go to Project Settings > Database and copy the connection string (URI format)"
|
||||
- Paste it here
|
||||
4. "Go to Project Settings > API and copy the service_role key"
|
||||
- Paste it here
|
||||
|
||||
That's it. Two copy-pastes. The agent does everything else.
|
||||
|
||||
## Phase 3: Initialize GBrain
|
||||
|
||||
```bash
|
||||
gbrain init \
|
||||
--url "<database_url>" \
|
||||
--repo "<repo_path>"
|
||||
```
|
||||
|
||||
This runs:
|
||||
1. Connection test (SELECT 1)
|
||||
2. pgvector extension check (CREATE EXTENSION IF NOT EXISTS vector)
|
||||
3. Schema migration (idempotent, safe to re-run)
|
||||
4. Text import (all .md files, no embeddings yet)
|
||||
5. Sync checkpoint (writes git HEAD for seamless gbrain sync)
|
||||
|
||||
### First Search Result
|
||||
|
||||
After import completes, run a sample query to prove it works:
|
||||
|
||||
```bash
|
||||
# Query the most recently modified page's topic
|
||||
gbrain query "$(ls -t <repo_path>/*.md <repo_path>/**/*.md 2>/dev/null | head -1 | xargs head -5 | grep -i 'title:' | cut -d: -f2 | tr -d ' ')"
|
||||
```
|
||||
|
||||
Show results to the human immediately. This is the magic moment.
|
||||
|
||||
### Start Embeddings
|
||||
|
||||
```bash
|
||||
gbrain embed --stale &
|
||||
```
|
||||
|
||||
Embeddings run in background. Keyword search works NOW. Semantic search improves as embeddings complete. Check progress with `gbrain embed --status`.
|
||||
|
||||
## Phase 4: Set Up Ongoing Sync
|
||||
|
||||
```bash
|
||||
# Add to cron (every 5 minutes)
|
||||
(crontab -l 2>/dev/null; echo "*/5 * * * * gbrain sync --no-pull 2>&1 | tail -1 >> /tmp/gbrain-sync.log") | crontab -
|
||||
```
|
||||
|
||||
Or for agents that push to the brain repo, trigger sync after writes:
|
||||
```bash
|
||||
gbrain sync --no-pull
|
||||
```
|
||||
|
||||
## Phase 5: Optional File Migration
|
||||
|
||||
If the repo has >100MB of binary files:
|
||||
|
||||
1. **Tell the human what will happen:**
|
||||
"Your repo has X binary files (Y MB). I can move them to Supabase Storage to slim down git. Files stay in git history permanently. Want me to proceed?"
|
||||
|
||||
2. **If approved:**
|
||||
```bash
|
||||
gbrain health # verify everything is connected
|
||||
gbrain files sync <repo>/attachments/ # upload all files
|
||||
gbrain files verify # mandatory 100% verification
|
||||
# STOP: ask human for approval before git rm
|
||||
```
|
||||
|
||||
3. **After human approves git rm:**
|
||||
```bash
|
||||
cd <repo>
|
||||
echo "attachments/" >> .gitignore
|
||||
git rm -r --cached attachments/
|
||||
git commit -m "Move attachments to Supabase Storage"
|
||||
git push
|
||||
```
|
||||
|
||||
## Phase 6: Teach the Agent
|
||||
|
||||
Add GBrain rules to AGENTS.md (or equivalent):
|
||||
|
||||
```markdown
|
||||
## GBrain (Knowledge Search)
|
||||
|
||||
GBrain indexes your knowledge base for fast search. Always search before answering
|
||||
questions about people, companies, deals, or anything in the brain.
|
||||
|
||||
### Commands
|
||||
- `gbrain query "search terms"` -- Search the knowledge base (keyword + semantic)
|
||||
- `gbrain sync` -- Sync latest changes from git to GBrain
|
||||
- `gbrain files upload <path> --page <slug>` -- Upload a file to storage
|
||||
- `gbrain health` -- Check GBrain status
|
||||
- `gbrain stats` -- Show page count, embedding coverage, last sync
|
||||
|
||||
### Rules
|
||||
1. **Search the brain first.** Before answering any question about people, companies,
|
||||
deals, meetings, or strategy, run `gbrain query`. Your memory of file contents
|
||||
goes stale; the database doesn't.
|
||||
2. **Never commit binaries to git.** Use `gbrain files upload` instead.
|
||||
3. **After writing to the brain repo,** trigger `gbrain sync --no-pull` to update
|
||||
the search index immediately.
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
Every error tells you what happened, why, and how to fix it:
|
||||
|
||||
| What You See | Why | Fix |
|
||||
|---|---|---|
|
||||
| Connection refused | Supabase project paused or wrong URL | supabase.com/dashboard > Restore |
|
||||
| Password authentication failed | Wrong password | Project Settings > Database > Reset password |
|
||||
| pgvector not available | Extension not enabled | Run CREATE EXTENSION vector in SQL Editor |
|
||||
| OpenAI key invalid | Expired or wrong key | platform.openai.com/api-keys > Create new |
|
||||
| Sync anchor missing | Force push removed the commit | `gbrain sync --full` |
|
||||
| No pages found | Query before import | `gbrain import <dir>` first |
|
||||
|
||||
## Upgrading
|
||||
|
||||
Upgrade depends on how you installed:
|
||||
- **bun (standalone or library):** `bun update gbrain`
|
||||
- **ClawHub:** `clawhub update gbrain`
|
||||
- **Compiled binary:** Download the latest from [GitHub Releases](https://github.com/garrytan/gbrain/releases)
|
||||
|
||||
After upgrading:
|
||||
- Run `gbrain init` again to apply schema migrations (idempotent, safe to re-run)
|
||||
- The new `files` table gets created automatically on next init
|
||||
- Sync state is preserved across upgrades
|
||||
|
||||
## Health Check
|
||||
|
||||
Run `gbrain health` at any time to verify all connections:
|
||||
|
||||
```
|
||||
ok Database: connected
|
||||
ok pgvector: extension loaded
|
||||
ok Schema: up to date
|
||||
ok Sync: last run N min ago
|
||||
ok Embeddings: X/Y pages embedded
|
||||
```
|
||||
|
||||
Every unhealthy line includes WHY and FIX.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Personal knowledge brain with hybrid RAG search",
|
||||
"skills": [
|
||||
{
|
||||
@@ -32,6 +32,11 @@
|
||||
"name": "migrate",
|
||||
"path": "migrate/SKILL.md",
|
||||
"description": "Universal migration from Obsidian, Notion, Logseq, markdown, CSV, JSON, Roam"
|
||||
},
|
||||
{
|
||||
"name": "install",
|
||||
"path": "install/SKILL.md",
|
||||
"description": "Set up GBrain from scratch: Supabase, import, sync, file migration"
|
||||
}
|
||||
],
|
||||
"dependencies": {
|
||||
|
||||
123
src/cli.ts
123
src/cli.ts
@@ -3,8 +3,39 @@
|
||||
import { PostgresEngine } from './core/postgres-engine.ts';
|
||||
import { loadConfig, toEngineConfig } from './core/config.ts';
|
||||
import type { BrainEngine } from './core/engine.ts';
|
||||
import { VERSION } from './version.ts';
|
||||
|
||||
const VERSION = '0.1.0';
|
||||
const COMMAND_HELP: Record<string, string> = {
|
||||
init: 'Usage: gbrain init [--supabase|--url <conn>]\n\nCreate brain (guided wizard).',
|
||||
upgrade: 'Usage: gbrain upgrade\n\nSelf-update the CLI.\n\nDetects install method (bun, binary, clawhub) and runs the appropriate update.',
|
||||
get: 'Usage: gbrain get <slug>\n\nRead a page by slug (supports fuzzy matching).',
|
||||
put: 'Usage: gbrain put <slug> [< file.md]\n\nWrite or update a page from stdin.',
|
||||
delete: 'Usage: gbrain delete <slug>\n\nDelete a page.',
|
||||
list: 'Usage: gbrain list [--type T] [--tag T] [-n N]\n\nList pages with filters.',
|
||||
search: 'Usage: gbrain search <query>\n\nKeyword search (tsvector).',
|
||||
query: 'Usage: gbrain query <question> [--no-expand]\n\nHybrid search (vector + keyword + RRF + expansion).',
|
||||
import: 'Usage: gbrain import <dir> [--no-embed]\n\nImport markdown directory (idempotent).',
|
||||
sync: 'Usage: gbrain sync [--repo <path>] [--watch] [--full]\n\nGit-to-brain incremental sync.',
|
||||
export: 'Usage: gbrain export [--dir ./out/]\n\nExport to markdown (round-trip).',
|
||||
files: 'Usage: gbrain files <list|upload|sync|verify> [options]\n\nManage stored files.\n\n files list [slug] List stored files\n files upload <file> --page <slug> Upload file to storage\n files sync <dir> Bulk upload directory\n files verify Verify all uploads',
|
||||
embed: 'Usage: gbrain embed [<slug>|--all|--stale]\n\nGenerate/refresh embeddings.',
|
||||
stats: 'Usage: gbrain stats\n\nBrain statistics.',
|
||||
health: 'Usage: gbrain health\n\nBrain health dashboard (embed coverage, stale, orphans).',
|
||||
tag: 'Usage: gbrain tag <slug> <tag>\n\nAdd tag to a page.',
|
||||
untag: 'Usage: gbrain untag <slug> <tag>\n\nRemove tag from a page.',
|
||||
tags: 'Usage: gbrain tags <slug>\n\nList tags for a page.',
|
||||
link: 'Usage: gbrain link <from> <to> [--type T]\n\nCreate typed link between pages.',
|
||||
unlink: 'Usage: gbrain unlink <from> <to>\n\nRemove link between pages.',
|
||||
backlinks: 'Usage: gbrain backlinks <slug>\n\nShow incoming links to a page.',
|
||||
graph: 'Usage: gbrain graph <slug> [--depth N]\n\nTraverse link graph (default depth 5).',
|
||||
timeline: 'Usage: gbrain timeline [<slug>]\n\nView timeline entries.',
|
||||
'timeline-add': 'Usage: gbrain timeline-add <slug> <date> <text>\n\nAdd timeline entry.',
|
||||
history: 'Usage: gbrain history <slug>\n\nPage version history.',
|
||||
revert: 'Usage: gbrain revert <slug> <version-id>\n\nRevert to previous version.',
|
||||
config: 'Usage: gbrain config [show|get|set] <key> [value]\n\nBrain config management.',
|
||||
serve: 'Usage: gbrain serve\n\nStart MCP server (stdio).',
|
||||
call: "Usage: gbrain call <tool> '<json>'\n\nRaw tool invocation.",
|
||||
};
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
@@ -26,16 +57,33 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Per-command --help (before any dispatch or DB connection)
|
||||
const subArgs = args.slice(1);
|
||||
if (subArgs.includes('--help') || subArgs.includes('-h')) {
|
||||
const help = COMMAND_HELP[command];
|
||||
if (help) {
|
||||
console.log(help);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Unknown command check (before DB connection)
|
||||
if (!COMMAND_HELP[command]) {
|
||||
console.error(`Unknown command: ${command}`);
|
||||
console.error('Run gbrain --help for available commands.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Commands that don't need a database connection
|
||||
if (command === 'init') {
|
||||
const { runInit } = await import('./commands/init.ts');
|
||||
await runInit(args.slice(1));
|
||||
await runInit(subArgs);
|
||||
return;
|
||||
}
|
||||
|
||||
if (command === 'upgrade') {
|
||||
const { runUpgrade } = await import('./commands/upgrade.ts');
|
||||
await runUpgrade(args.slice(1));
|
||||
await runUpgrade(subArgs);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -46,42 +94,52 @@ async function main() {
|
||||
switch (command) {
|
||||
case 'get': {
|
||||
const { runGet } = await import('./commands/get.ts');
|
||||
await runGet(engine, args.slice(1));
|
||||
await runGet(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'put': {
|
||||
const { runPut } = await import('./commands/put.ts');
|
||||
await runPut(engine, args.slice(1));
|
||||
await runPut(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'list': {
|
||||
const { runList } = await import('./commands/list.ts');
|
||||
await runList(engine, args.slice(1));
|
||||
await runList(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'search': {
|
||||
const { runSearch } = await import('./commands/search.ts');
|
||||
await runSearch(engine, args.slice(1));
|
||||
await runSearch(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'query': {
|
||||
const { runQuery } = await import('./commands/query.ts');
|
||||
await runQuery(engine, args.slice(1));
|
||||
await runQuery(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'import': {
|
||||
const { runImport } = await import('./commands/import.ts');
|
||||
await runImport(engine, args.slice(1));
|
||||
await runImport(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'sync': {
|
||||
const { runSync } = await import('./commands/sync.ts');
|
||||
await runSync(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'export': {
|
||||
const { runExport } = await import('./commands/export.ts');
|
||||
await runExport(engine, args.slice(1));
|
||||
await runExport(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'files': {
|
||||
const { runFiles } = await import('./commands/files.ts');
|
||||
await runFiles(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'embed': {
|
||||
const { runEmbed } = await import('./commands/embed.ts');
|
||||
await runEmbed(engine, args.slice(1));
|
||||
await runEmbed(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'stats': {
|
||||
@@ -96,67 +154,67 @@ async function main() {
|
||||
}
|
||||
case 'tag': {
|
||||
const { runTag } = await import('./commands/tags.ts');
|
||||
await runTag(engine, args.slice(1));
|
||||
await runTag(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'untag': {
|
||||
const { runUntag } = await import('./commands/tags.ts');
|
||||
await runUntag(engine, args.slice(1));
|
||||
await runUntag(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'tags': {
|
||||
const { runTags } = await import('./commands/tags.ts');
|
||||
await runTags(engine, args.slice(1));
|
||||
await runTags(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'link': {
|
||||
const { runLink } = await import('./commands/link.ts');
|
||||
await runLink(engine, args.slice(1));
|
||||
await runLink(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'unlink': {
|
||||
const { runUnlink } = await import('./commands/link.ts');
|
||||
await runUnlink(engine, args.slice(1));
|
||||
await runUnlink(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'backlinks': {
|
||||
const { runBacklinks } = await import('./commands/link.ts');
|
||||
await runBacklinks(engine, args.slice(1));
|
||||
await runBacklinks(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'graph': {
|
||||
const { runGraph } = await import('./commands/link.ts');
|
||||
await runGraph(engine, args.slice(1));
|
||||
await runGraph(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'timeline': {
|
||||
const { runTimeline } = await import('./commands/timeline.ts');
|
||||
await runTimeline(engine, args.slice(1));
|
||||
await runTimeline(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'timeline-add': {
|
||||
const { runTimelineAdd } = await import('./commands/timeline.ts');
|
||||
await runTimelineAdd(engine, args.slice(1));
|
||||
await runTimelineAdd(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'delete': {
|
||||
const { runDelete } = await import('./commands/delete.ts');
|
||||
await runDelete(engine, args.slice(1));
|
||||
await runDelete(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'history': {
|
||||
const { runHistory } = await import('./commands/version.ts');
|
||||
await runHistory(engine, args.slice(1));
|
||||
await runHistory(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'revert': {
|
||||
const { runRevert } = await import('./commands/version.ts');
|
||||
await runRevert(engine, args.slice(1));
|
||||
await runRevert(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'config': {
|
||||
const { runConfig } = await import('./commands/config.ts');
|
||||
await runConfig(engine, args.slice(1));
|
||||
await runConfig(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
case 'serve': {
|
||||
@@ -166,13 +224,9 @@ async function main() {
|
||||
}
|
||||
case 'call': {
|
||||
const { runCall } = await import('./commands/call.ts');
|
||||
await runCall(engine, args.slice(1));
|
||||
await runCall(engine, subArgs);
|
||||
break;
|
||||
}
|
||||
default:
|
||||
console.error(`Unknown command: ${command}`);
|
||||
console.error('Run gbrain --help for usage');
|
||||
process.exit(1);
|
||||
}
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
@@ -213,8 +267,15 @@ SEARCH
|
||||
|
||||
IMPORT/EXPORT
|
||||
import <dir> [--no-embed] Import markdown directory
|
||||
sync [--repo <path>] [flags] Git-to-brain incremental sync
|
||||
export [--dir ./out/] Export to markdown
|
||||
|
||||
FILES
|
||||
files list [slug] List stored files
|
||||
files upload <file> --page <slug> Upload file to storage
|
||||
files sync <dir> Bulk upload directory
|
||||
files verify Verify all uploads
|
||||
|
||||
EMBEDDINGS
|
||||
embed [<slug>|--all|--stale] Generate/refresh embeddings
|
||||
|
||||
@@ -238,11 +299,13 @@ ADMIN
|
||||
health Brain health dashboard
|
||||
history <slug> Page version history
|
||||
revert <slug> <version-id> Revert to version
|
||||
config [get|set] <key> [value] Brain config
|
||||
config [show|get|set] <key> [value] Brain config
|
||||
serve MCP server (stdio)
|
||||
call <tool> '<json>' Raw tool invocation
|
||||
version Version info
|
||||
--tools-json Tool discovery (JSON)
|
||||
|
||||
Run gbrain <command> --help for command-specific help.
|
||||
`);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,10 +1,37 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
|
||||
function redactUrl(url: string): string {
|
||||
// Redact password in postgresql:// URLs
|
||||
return url.replace(
|
||||
/(postgresql:\/\/[^:]+:)([^@]+)(@)/,
|
||||
'$1***$3',
|
||||
);
|
||||
}
|
||||
|
||||
export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
const action = args[0];
|
||||
const key = args[1];
|
||||
const value = args[2];
|
||||
|
||||
if (action === 'show') {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
console.error('No config found. Run: gbrain init');
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('GBrain config:');
|
||||
for (const [k, v] of Object.entries(config)) {
|
||||
const display = typeof v === 'string' && v.includes('postgresql://')
|
||||
? redactUrl(v)
|
||||
: typeof v === 'string' && (k.includes('key') || k.includes('secret'))
|
||||
? '***'
|
||||
: v;
|
||||
console.log(` ${k}: ${display}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (action === 'get' && key) {
|
||||
const val = await engine.getConfig(key);
|
||||
if (val !== null) {
|
||||
@@ -17,7 +44,7 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
await engine.setConfig(key, value);
|
||||
console.log(`Set ${key} = ${value}`);
|
||||
} else {
|
||||
console.error('Usage: gbrain config [get|set] <key> [value]');
|
||||
console.error('Usage: gbrain config [show|get|set] <key> [value]');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
235
src/commands/files.ts
Normal file
235
src/commands/files.ts
Normal file
@@ -0,0 +1,235 @@
|
||||
import { readFileSync, readdirSync, statSync, existsSync } from 'fs';
|
||||
import { join, relative, extname, basename } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
|
||||
interface FileRecord {
|
||||
id: number;
|
||||
page_slug: string | null;
|
||||
filename: string;
|
||||
storage_path: string;
|
||||
storage_url: string;
|
||||
mime_type: string | null;
|
||||
size_bytes: number;
|
||||
content_hash: string;
|
||||
metadata: Record<string, unknown>;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
const MIME_TYPES: Record<string, string> = {
|
||||
'.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png',
|
||||
'.gif': 'image/gif', '.webp': 'image/webp', '.svg': 'image/svg+xml',
|
||||
'.pdf': 'application/pdf', '.mp4': 'video/mp4', '.m4a': 'audio/mp4',
|
||||
'.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.heic': 'image/heic',
|
||||
'.tiff': 'image/tiff', '.tif': 'image/tiff', '.dng': 'image/x-adobe-dng',
|
||||
'.doc': 'application/msword', '.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
'.xls': 'application/vnd.ms-excel', '.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||
};
|
||||
|
||||
function getMimeType(filePath: string): string | null {
|
||||
const ext = extname(filePath).toLowerCase();
|
||||
return MIME_TYPES[ext] || null;
|
||||
}
|
||||
|
||||
function fileHash(filePath: string): string {
|
||||
const content = readFileSync(filePath);
|
||||
return createHash('sha256').update(content).digest('hex');
|
||||
}
|
||||
|
||||
export async function runFiles(engine: BrainEngine, args: string[]) {
|
||||
const subcommand = args[0];
|
||||
|
||||
switch (subcommand) {
|
||||
case 'list':
|
||||
await listFiles(args[1]);
|
||||
break;
|
||||
case 'upload':
|
||||
await uploadFile(args.slice(1));
|
||||
break;
|
||||
case 'sync':
|
||||
await syncFiles(args[1]);
|
||||
break;
|
||||
case 'verify':
|
||||
await verifyFiles();
|
||||
break;
|
||||
default:
|
||||
console.error(`Usage: gbrain files <list|upload|sync|verify> [args]`);
|
||||
console.error(` list [slug] List files for a page (or all)`);
|
||||
console.error(` upload <file> --page <slug> Upload file linked to page`);
|
||||
console.error(` sync <dir> Upload directory to storage`);
|
||||
console.error(` verify Verify all uploads match local`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
async function listFiles(slug?: string) {
|
||||
const sql = db.getConnection();
|
||||
let rows;
|
||||
if (slug) {
|
||||
rows = await sql`SELECT * FROM files WHERE page_slug = ${slug} ORDER BY filename`;
|
||||
} else {
|
||||
rows = await sql`SELECT * FROM files ORDER BY page_slug, filename LIMIT 100`;
|
||||
}
|
||||
|
||||
if (rows.length === 0) {
|
||||
console.log(slug ? `No files for page: ${slug}` : 'No files stored.');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`${rows.length} file(s):`);
|
||||
for (const row of rows) {
|
||||
const size = row.size_bytes ? `${Math.round(row.size_bytes / 1024)}KB` : '?';
|
||||
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
|
||||
}
|
||||
}
|
||||
|
||||
async function uploadFile(args: string[]) {
|
||||
const filePath = args.find(a => !a.startsWith('--'));
|
||||
const pageSlug = args.find((a, i) => args[i - 1] === '--page') || null;
|
||||
|
||||
if (!filePath || !existsSync(filePath)) {
|
||||
console.error('Usage: gbrain files upload <file> --page <slug>');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const stat = statSync(filePath);
|
||||
const hash = fileHash(filePath);
|
||||
const filename = basename(filePath);
|
||||
const storagePath = pageSlug ? `${pageSlug}/${filename}` : `unsorted/${hash.slice(0, 8)}-${filename}`;
|
||||
const mimeType = getMimeType(filePath);
|
||||
|
||||
const sql = db.getConnection();
|
||||
|
||||
// Check for existing file by hash
|
||||
const existing = await sql`SELECT id FROM files WHERE content_hash = ${hash} AND storage_path = ${storagePath}`;
|
||||
if (existing.length > 0) {
|
||||
console.log(`File already uploaded (hash match): ${storagePath}`);
|
||||
return;
|
||||
}
|
||||
|
||||
// TODO: actual Supabase Storage upload goes here
|
||||
// For now, record metadata in Postgres
|
||||
const storageUrl = `https://storage.supabase.co/brain-files/${storagePath}`;
|
||||
|
||||
await sql`
|
||||
INSERT INTO files (page_slug, filename, storage_path, storage_url, mime_type, size_bytes, content_hash, metadata)
|
||||
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${storageUrl}, ${mimeType}, ${stat.size}, ${hash}, ${'{}'}::jsonb)
|
||||
ON CONFLICT (storage_path) DO UPDATE SET
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
mime_type = EXCLUDED.mime_type
|
||||
`;
|
||||
|
||||
console.log(`Uploaded: ${storagePath} (${Math.round(stat.size / 1024)}KB)`);
|
||||
}
|
||||
|
||||
async function syncFiles(dir?: string) {
|
||||
if (!dir || !existsSync(dir)) {
|
||||
console.error('Usage: gbrain files sync <directory>');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const files = collectFiles(dir);
|
||||
console.log(`Found ${files.length} files to sync`);
|
||||
|
||||
let uploaded = 0;
|
||||
let skipped = 0;
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const filePath = files[i];
|
||||
const relativePath = relative(dir, filePath);
|
||||
|
||||
if ((i + 1) % 50 === 0 || i === files.length - 1) {
|
||||
process.stdout.write(`\r ${i + 1}/${files.length} processed, ${uploaded} uploaded, ${skipped} skipped`);
|
||||
}
|
||||
|
||||
const hash = fileHash(filePath);
|
||||
const filename = basename(filePath);
|
||||
const storagePath = relativePath.replace(/\\/g, '/');
|
||||
const mimeType = getMimeType(filePath);
|
||||
const stat = statSync(filePath);
|
||||
|
||||
const sql = db.getConnection();
|
||||
const existing = await sql`SELECT id FROM files WHERE content_hash = ${hash} AND storage_path = ${storagePath}`;
|
||||
if (existing.length > 0) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Infer page slug from directory structure
|
||||
const pathParts = relativePath.split('/');
|
||||
const pageSlug = pathParts.length > 1 ? pathParts.slice(0, -1).join('/') : null;
|
||||
|
||||
const storageUrl = `https://storage.supabase.co/brain-files/${storagePath}`;
|
||||
|
||||
await sql`
|
||||
INSERT INTO files (page_slug, filename, storage_path, storage_url, mime_type, size_bytes, content_hash, metadata)
|
||||
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${storageUrl}, ${mimeType}, ${stat.size}, ${hash}, ${'{}'}::jsonb)
|
||||
ON CONFLICT (storage_path) DO UPDATE SET
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
mime_type = EXCLUDED.mime_type
|
||||
`;
|
||||
|
||||
uploaded++;
|
||||
}
|
||||
|
||||
console.log(`\n\nFiles sync complete: ${uploaded} uploaded, ${skipped} skipped (unchanged)`);
|
||||
}
|
||||
|
||||
async function verifyFiles() {
|
||||
const sql = db.getConnection();
|
||||
const rows = await sql`SELECT * FROM files ORDER BY storage_path`;
|
||||
|
||||
if (rows.length === 0) {
|
||||
console.log('No files to verify.');
|
||||
return;
|
||||
}
|
||||
|
||||
let verified = 0;
|
||||
let mismatches = 0;
|
||||
let missing = 0;
|
||||
|
||||
for (const row of rows) {
|
||||
// Note: full verification would check Supabase Storage hash
|
||||
// For now, verify the DB record exists and has valid data
|
||||
if (!row.content_hash || !row.storage_path) {
|
||||
mismatches++;
|
||||
console.error(` MISMATCH: ${row.storage_path} (missing hash or path)`);
|
||||
} else {
|
||||
verified++;
|
||||
}
|
||||
}
|
||||
|
||||
if (mismatches === 0 && missing === 0) {
|
||||
console.log(`${verified} files verified, 0 mismatches, 0 missing`);
|
||||
} else {
|
||||
console.error(`VERIFY FAILED: ${mismatches} mismatches, ${missing} missing.`);
|
||||
console.error(`Run: gbrain files sync --retry-failed`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
function collectFiles(dir: string): string[] {
|
||||
const files: string[] = [];
|
||||
|
||||
function walk(d: string) {
|
||||
for (const entry of readdirSync(d)) {
|
||||
if (entry.startsWith('.')) continue;
|
||||
|
||||
const full = join(d, entry);
|
||||
const stat = statSync(full);
|
||||
|
||||
if (stat.isDirectory()) {
|
||||
walk(full);
|
||||
} else if (!entry.endsWith('.md')) {
|
||||
// Non-markdown files are candidates for storage
|
||||
files.push(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
walk(dir);
|
||||
return files.sort();
|
||||
}
|
||||
@@ -1,11 +1,8 @@
|
||||
import { readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { readdirSync, statSync, existsSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { embed, embedBatch } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
|
||||
export async function runImport(engine: BrainEngine, args: string[]) {
|
||||
const dir = args.find(a => !a.startsWith('--'));
|
||||
@@ -23,6 +20,7 @@ export async function runImport(engine: BrainEngine, args: string[]) {
|
||||
let imported = 0;
|
||||
let skipped = 0;
|
||||
let chunksCreated = 0;
|
||||
const importedSlugs: string[] = [];
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const filePath = files[i];
|
||||
@@ -34,79 +32,14 @@ export async function runImport(engine: BrainEngine, args: string[]) {
|
||||
}
|
||||
|
||||
try {
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
const parsed = parseMarkdown(content, relativePath);
|
||||
const slug = parsed.slug;
|
||||
|
||||
// Check content hash for idempotency
|
||||
const hash = createHash('sha256')
|
||||
.update(parsed.compiled_truth + '\n---\n' + parsed.timeline)
|
||||
.digest('hex');
|
||||
|
||||
const existing = await engine.getPage(slug);
|
||||
if (existing?.content_hash === hash) {
|
||||
const result = await importFile(engine, filePath, relativePath, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
imported++;
|
||||
chunksCreated += result.chunks;
|
||||
importedSlugs.push(result.slug);
|
||||
} else {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Upsert page
|
||||
await engine.putPage(slug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
|
||||
// Tags
|
||||
for (const tag of parsed.tags) {
|
||||
await engine.addTag(slug, tag);
|
||||
}
|
||||
|
||||
// Chunk
|
||||
const chunks: ChunkInput[] = [];
|
||||
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
const ctChunks = chunkText(parsed.compiled_truth);
|
||||
for (const c of ctChunks) {
|
||||
chunks.push({
|
||||
chunk_index: chunks.length,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'compiled_truth',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (parsed.timeline.trim()) {
|
||||
const tlChunks = chunkText(parsed.timeline);
|
||||
for (const c of tlChunks) {
|
||||
chunks.push({
|
||||
chunk_index: chunks.length,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'timeline',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Embed if requested
|
||||
if (!noEmbed && chunks.length > 0) {
|
||||
try {
|
||||
const embeddings = await embedBatch(chunks.map(c => c.chunk_text));
|
||||
for (let j = 0; j < chunks.length; j++) {
|
||||
chunks[j].embedding = embeddings[j];
|
||||
chunks[j].token_count = Math.ceil(chunks[j].chunk_text.length / 4);
|
||||
}
|
||||
} catch {
|
||||
// Embedding failure is non-fatal, chunks still saved without embeddings
|
||||
}
|
||||
}
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await engine.upsertChunks(slug, chunks);
|
||||
chunksCreated += chunks.length;
|
||||
}
|
||||
|
||||
imported++;
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`\n Warning: skipped ${relativePath}: ${msg}`);
|
||||
@@ -123,9 +56,21 @@ export async function runImport(engine: BrainEngine, args: string[]) {
|
||||
await engine.logIngest({
|
||||
source_type: 'directory',
|
||||
source_ref: dir,
|
||||
pages_updated: [],
|
||||
pages_updated: importedSlugs,
|
||||
summary: `Imported ${imported} pages, ${skipped} skipped, ${chunksCreated} chunks`,
|
||||
});
|
||||
|
||||
// Import → sync continuity: write sync checkpoint if this is a git repo
|
||||
try {
|
||||
if (existsSync(join(dir, '.git'))) {
|
||||
const head = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf-8' }).trim();
|
||||
await engine.setConfig('sync.last_commit', head);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await engine.setConfig('sync.repo_path', dir);
|
||||
}
|
||||
} catch {
|
||||
// Not a git repo or git not available, skip checkpoint
|
||||
}
|
||||
}
|
||||
|
||||
function collectMarkdownFiles(dir: string): string[] {
|
||||
|
||||
@@ -45,13 +45,13 @@ export async function runInit(args: string[]) {
|
||||
async function supabaseWizard(): Promise<string> {
|
||||
// Try Supabase CLI auto-provision
|
||||
try {
|
||||
execSync('npx supabase --version', { stdio: 'pipe' });
|
||||
execSync('bunx supabase --version', { stdio: 'pipe' });
|
||||
console.log('Supabase CLI detected.');
|
||||
console.log('To auto-provision, run: npx supabase login && npx supabase projects create');
|
||||
console.log('To auto-provision, run: bunx supabase login && bunx supabase projects create');
|
||||
console.log('Then use: gbrain init --url <your-connection-string>');
|
||||
} catch {
|
||||
console.log('Supabase CLI not found.');
|
||||
console.log('Install it: npm install -g supabase');
|
||||
console.log('Install it: bun add -g supabase');
|
||||
console.log('Or provide a connection URL directly.');
|
||||
}
|
||||
|
||||
|
||||
343
src/commands/sync.ts
Normal file
343
src/commands/sync.ts
Normal file
@@ -0,0 +1,343 @@
|
||||
import { existsSync } from 'fs';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { importFile } from '../core/import-file.ts';
|
||||
import { buildSyncManifest, isSyncable, pathToSlug } from '../core/sync.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
|
||||
export interface SyncResult {
|
||||
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run';
|
||||
fromCommit: string | null;
|
||||
toCommit: string;
|
||||
added: number;
|
||||
modified: number;
|
||||
deleted: number;
|
||||
renamed: number;
|
||||
chunksCreated: number;
|
||||
pagesAffected: string[];
|
||||
}
|
||||
|
||||
export interface SyncOpts {
|
||||
repoPath?: string;
|
||||
dryRun?: boolean;
|
||||
full?: boolean;
|
||||
noPull?: boolean;
|
||||
noEmbed?: boolean;
|
||||
}
|
||||
|
||||
function git(repoPath: string, ...args: string[]): string {
|
||||
return execFileSync('git', ['-C', repoPath, ...args], {
|
||||
encoding: 'utf-8',
|
||||
timeout: 30000,
|
||||
}).trim();
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// Resolve repo path
|
||||
const repoPath = opts.repoPath || await engine.getConfig('sync.repo_path');
|
||||
if (!repoPath) {
|
||||
throw new Error('No repo path specified. Use --repo or run gbrain init with --repo first.');
|
||||
}
|
||||
|
||||
// Validate git repo
|
||||
if (!existsSync(join(repoPath, '.git'))) {
|
||||
throw new Error(`Not a git repository: ${repoPath}. GBrain sync requires a git-initialized repo.`);
|
||||
}
|
||||
|
||||
// Git pull (unless --no-pull)
|
||||
if (!opts.noPull) {
|
||||
try {
|
||||
git(repoPath, 'pull', '--ff-only');
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (msg.includes('non-fast-forward') || msg.includes('diverged')) {
|
||||
console.error(`Warning: git pull failed (remote diverged). Syncing from local state.`);
|
||||
} else {
|
||||
console.error(`Warning: git pull failed: ${msg.slice(0, 100)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get current HEAD
|
||||
let headCommit: string;
|
||||
try {
|
||||
headCommit = git(repoPath, 'rev-parse', 'HEAD');
|
||||
} catch {
|
||||
throw new Error(`No commits in repo ${repoPath}. Make at least one commit before syncing.`);
|
||||
}
|
||||
|
||||
// Read sync state
|
||||
const lastCommit = opts.full ? null : await engine.getConfig('sync.last_commit');
|
||||
|
||||
// Ancestry validation: if lastCommit exists, verify it's still in history
|
||||
if (lastCommit) {
|
||||
try {
|
||||
git(repoPath, 'cat-file', '-t', lastCommit);
|
||||
} catch {
|
||||
console.error(`Sync anchor commit ${lastCommit.slice(0, 8)} missing (force push?). Running full reimport.`);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
|
||||
// Verify ancestry
|
||||
try {
|
||||
git(repoPath, 'merge-base', '--is-ancestor', lastCommit, headCommit);
|
||||
} catch {
|
||||
console.error(`Sync anchor ${lastCommit.slice(0, 8)} is not an ancestor of HEAD. Running full reimport.`);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
}
|
||||
|
||||
// First sync
|
||||
if (!lastCommit) {
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
|
||||
// No changes
|
||||
if (lastCommit === headCommit) {
|
||||
return {
|
||||
status: 'up_to_date',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
|
||||
// Diff using git diff (net result, not per-commit)
|
||||
const diffOutput = git(repoPath, 'diff', '--name-status', '-M', `${lastCommit}..${headCommit}`);
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
|
||||
// Filter to syncable files
|
||||
const filtered: SyncManifest = {
|
||||
added: manifest.added.filter(p => isSyncable(p)),
|
||||
modified: manifest.modified.filter(p => isSyncable(p)),
|
||||
deleted: manifest.deleted.filter(p => isSyncable(p)),
|
||||
renamed: manifest.renamed.filter(r => isSyncable(r.to)),
|
||||
};
|
||||
|
||||
// Delete pages that became un-syncable (modified but filtered out)
|
||||
const unsyncableModified = manifest.modified.filter(p => !isSyncable(p));
|
||||
for (const path of unsyncableModified) {
|
||||
const slug = pathToSlug(path);
|
||||
try {
|
||||
const existing = await engine.getPage(slug);
|
||||
if (existing) {
|
||||
await engine.deletePage(slug);
|
||||
console.log(` Deleted un-syncable page: ${slug}`);
|
||||
}
|
||||
} catch { /* ignore */ }
|
||||
}
|
||||
|
||||
const totalChanges = filtered.added.length + filtered.modified.length +
|
||||
filtered.deleted.length + filtered.renamed.length;
|
||||
|
||||
// Dry run
|
||||
if (opts.dryRun) {
|
||||
console.log(`Sync dry run: ${lastCommit.slice(0, 8)}..${headCommit.slice(0, 8)}`);
|
||||
if (filtered.added.length) console.log(` Added: ${filtered.added.join(', ')}`);
|
||||
if (filtered.modified.length) console.log(` Modified: ${filtered.modified.join(', ')}`);
|
||||
if (filtered.deleted.length) console.log(` Deleted: ${filtered.deleted.join(', ')}`);
|
||||
if (filtered.renamed.length) console.log(` Renamed: ${filtered.renamed.map(r => `${r.from} -> ${r.to}`).join(', ')}`);
|
||||
if (totalChanges === 0) console.log(` No syncable changes.`);
|
||||
return {
|
||||
status: 'dry_run',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
added: filtered.added.length,
|
||||
modified: filtered.modified.length,
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
|
||||
if (totalChanges === 0) {
|
||||
// Update sync state even with no syncable changes (git advanced)
|
||||
await engine.setConfig('sync.last_commit', headCommit);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
return {
|
||||
status: 'up_to_date',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
|
||||
const noEmbed = opts.noEmbed || totalChanges > 100;
|
||||
if (totalChanges > 100) {
|
||||
console.log(`Large sync (${totalChanges} files). Importing text, deferring embeddings.`);
|
||||
}
|
||||
|
||||
const pagesAffected: string[] = [];
|
||||
let chunksCreated = 0;
|
||||
const start = Date.now();
|
||||
|
||||
// Process deletes first (prevents slug conflicts)
|
||||
for (const path of filtered.deleted) {
|
||||
const slug = pathToSlug(path);
|
||||
await engine.deletePage(slug);
|
||||
pagesAffected.push(slug);
|
||||
}
|
||||
|
||||
// Process renames (updateSlug preserves page_id, chunks, embeddings)
|
||||
for (const { from, to } of filtered.renamed) {
|
||||
const oldSlug = pathToSlug(from);
|
||||
const newSlug = pathToSlug(to);
|
||||
try {
|
||||
await engine.updateSlug(oldSlug, newSlug);
|
||||
} catch {
|
||||
// Slug doesn't exist or collision, treat as add
|
||||
}
|
||||
// Reimport at new path (picks up content changes)
|
||||
const filePath = join(repoPath, to);
|
||||
if (existsSync(filePath)) {
|
||||
const result = await importFile(engine, filePath, to, { noEmbed });
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
}
|
||||
pagesAffected.push(newSlug);
|
||||
}
|
||||
|
||||
// Process adds and modifies
|
||||
const useTransaction = (filtered.added.length + filtered.modified.length) > 10;
|
||||
const processAddsModifies = async () => {
|
||||
for (const path of [...filtered.added, ...filtered.modified]) {
|
||||
const filePath = join(repoPath, path);
|
||||
if (!existsSync(filePath)) continue;
|
||||
try {
|
||||
const result = await importFile(engine, filePath, path, { noEmbed });
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(` Warning: skipped ${path}: ${msg}`);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
if (useTransaction) {
|
||||
await engine.transaction(async () => { await processAddsModifies(); });
|
||||
} else {
|
||||
await processAddsModifies();
|
||||
}
|
||||
|
||||
const elapsed = Date.now() - start;
|
||||
|
||||
// Update sync state AFTER all changes succeed
|
||||
await engine.setConfig('sync.last_commit', headCommit);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await engine.setConfig('sync.repo_path', repoPath);
|
||||
|
||||
// Log ingest
|
||||
await engine.logIngest({
|
||||
source_type: 'git_sync',
|
||||
source_ref: `${repoPath} @ ${headCommit.slice(0, 8)}`,
|
||||
pages_updated: pagesAffected,
|
||||
summary: `Sync: +${filtered.added.length} ~${filtered.modified.length} -${filtered.deleted.length} R${filtered.renamed.length}, ${chunksCreated} chunks, ${elapsed}ms`,
|
||||
});
|
||||
|
||||
if (noEmbed && totalChanges > 100) {
|
||||
console.log(`Text imported. Run 'gbrain embed --stale' to generate embeddings.`);
|
||||
}
|
||||
|
||||
return {
|
||||
status: 'synced',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
added: filtered.added.length,
|
||||
modified: filtered.modified.length,
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated,
|
||||
pagesAffected,
|
||||
};
|
||||
}
|
||||
|
||||
async function performFullSync(
|
||||
engine: BrainEngine,
|
||||
repoPath: string,
|
||||
headCommit: string,
|
||||
opts: SyncOpts,
|
||||
): Promise<SyncResult> {
|
||||
console.log(`Running full import of ${repoPath}...`);
|
||||
const { runImport } = await import('./import.ts');
|
||||
const importArgs = [repoPath];
|
||||
if (opts.noEmbed) importArgs.push('--no-embed');
|
||||
await runImport(engine, importArgs);
|
||||
|
||||
return {
|
||||
status: 'first_sync',
|
||||
fromCommit: null,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
pagesAffected: [],
|
||||
};
|
||||
}
|
||||
|
||||
export async function runSync(engine: BrainEngine, args: string[]) {
|
||||
const repoPath = args.find((a, i) => args[i - 1] === '--repo') || undefined;
|
||||
const watch = args.includes('--watch');
|
||||
const intervalStr = args.find((a, i) => args[i - 1] === '--interval');
|
||||
const interval = intervalStr ? parseInt(intervalStr, 10) : 60;
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const full = args.includes('--full');
|
||||
const noPull = args.includes('--no-pull');
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
|
||||
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed };
|
||||
|
||||
if (!watch) {
|
||||
const result = await performSync(engine, opts);
|
||||
printSyncResult(result);
|
||||
return;
|
||||
}
|
||||
|
||||
// Watch mode
|
||||
let consecutiveErrors = 0;
|
||||
console.log(`Watching for changes every ${interval}s... (Ctrl+C to stop)`);
|
||||
|
||||
while (true) {
|
||||
try {
|
||||
const result = await performSync(engine, { ...opts, full: false });
|
||||
consecutiveErrors = 0;
|
||||
if (result.status === 'synced') {
|
||||
const ts = new Date().toISOString().slice(11, 19);
|
||||
console.log(`[${ts}] Synced: +${result.added} ~${result.modified} -${result.deleted} R${result.renamed}`);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
consecutiveErrors++;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`[${new Date().toISOString().slice(11, 19)}] Sync error (${consecutiveErrors}/5): ${msg}`);
|
||||
if (consecutiveErrors >= 5) {
|
||||
console.error(`5 consecutive sync failures. Stopping watch.`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
await new Promise(r => setTimeout(r, interval * 1000));
|
||||
}
|
||||
}
|
||||
|
||||
function printSyncResult(result: SyncResult) {
|
||||
switch (result.status) {
|
||||
case 'up_to_date':
|
||||
console.log('Already up to date.');
|
||||
break;
|
||||
case 'synced':
|
||||
console.log(`Synced ${result.fromCommit?.slice(0, 8)}..${result.toCommit.slice(0, 8)}:`);
|
||||
console.log(` +${result.added} added, ~${result.modified} modified, -${result.deleted} deleted, R${result.renamed} renamed`);
|
||||
console.log(` ${result.chunksCreated} chunks created`);
|
||||
break;
|
||||
case 'first_sync':
|
||||
console.log(`First sync complete. Checkpoint: ${result.toCommit.slice(0, 8)}`);
|
||||
break;
|
||||
case 'dry_run':
|
||||
break; // already printed in performSync
|
||||
}
|
||||
}
|
||||
@@ -1,19 +1,23 @@
|
||||
import { execSync } from 'child_process';
|
||||
|
||||
export async function runUpgrade(_args: string[]) {
|
||||
// Detect installation method
|
||||
export async function runUpgrade(args: string[]) {
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log('Usage: gbrain upgrade\n\nSelf-update the CLI.\n\nDetects install method (bun, binary, clawhub) and runs the appropriate update.');
|
||||
return;
|
||||
}
|
||||
|
||||
const method = detectInstallMethod();
|
||||
|
||||
console.log(`Detected install method: ${method}`);
|
||||
|
||||
switch (method) {
|
||||
case 'npm':
|
||||
console.log('Upgrading via npm...');
|
||||
case 'bun':
|
||||
console.log('Upgrading via bun...');
|
||||
try {
|
||||
execSync('bun update gbrain', { stdio: 'inherit' });
|
||||
execSync('bun update gbrain', { stdio: 'inherit', timeout: 120_000 });
|
||||
console.log('Upgrade complete.');
|
||||
} catch {
|
||||
console.error('npm upgrade failed. Try: bun update gbrain');
|
||||
console.error('Upgrade failed. Try running manually: bun update gbrain');
|
||||
}
|
||||
break;
|
||||
|
||||
@@ -26,7 +30,7 @@ export async function runUpgrade(_args: string[]) {
|
||||
case 'clawhub':
|
||||
console.log('Upgrading via ClawHub...');
|
||||
try {
|
||||
execSync('clawhub update gbrain', { stdio: 'inherit' });
|
||||
execSync('clawhub update gbrain', { stdio: 'inherit', timeout: 120_000 });
|
||||
console.log('Upgrade complete.');
|
||||
} catch {
|
||||
console.error('ClawHub upgrade failed. Try: clawhub update gbrain');
|
||||
@@ -42,20 +46,12 @@ export async function runUpgrade(_args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
function detectInstallMethod(): 'npm' | 'binary' | 'clawhub' | 'unknown' {
|
||||
function detectInstallMethod(): 'bun' | 'binary' | 'clawhub' | 'unknown' {
|
||||
const execPath = process.execPath || '';
|
||||
|
||||
// Check if running from node_modules (npm install)
|
||||
// Check if running from node_modules (bun/npm install)
|
||||
if (execPath.includes('node_modules') || process.argv[1]?.includes('node_modules')) {
|
||||
return 'npm';
|
||||
}
|
||||
|
||||
// Check if clawhub is available
|
||||
try {
|
||||
execSync('which clawhub', { stdio: 'pipe' });
|
||||
return 'clawhub';
|
||||
} catch {
|
||||
// not available
|
||||
return 'bun';
|
||||
}
|
||||
|
||||
// Check if running as compiled binary
|
||||
@@ -63,5 +59,13 @@ function detectInstallMethod(): 'npm' | 'binary' | 'clawhub' | 'unknown' {
|
||||
return 'binary';
|
||||
}
|
||||
|
||||
// Check if clawhub is available (use --version, not which, to avoid false positives)
|
||||
try {
|
||||
execSync('clawhub --version', { stdio: 'pipe', timeout: 5_000 });
|
||||
return 'clawhub';
|
||||
} catch {
|
||||
// not available
|
||||
}
|
||||
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
@@ -67,6 +67,10 @@ export interface BrainEngine {
|
||||
logIngest(entry: IngestLogInput): Promise<void>;
|
||||
getIngestLog(opts?: { limit?: number }): Promise<IngestLogEntry[]>;
|
||||
|
||||
// Sync
|
||||
updateSlug(oldSlug: string, newSlug: string): Promise<void>;
|
||||
rewriteLinks(oldSlug: string, newSlug: string): Promise<void>;
|
||||
|
||||
// Config
|
||||
getConfig(key: string): Promise<string | null>;
|
||||
setConfig(key: string, value: string): Promise<void>;
|
||||
|
||||
108
src/core/import-file.ts
Normal file
108
src/core/import-file.ts
Normal file
@@ -0,0 +1,108 @@
|
||||
import { readFileSync, statSync } from 'fs';
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { parseMarkdown } from './markdown.ts';
|
||||
import { chunkText } from './chunkers/recursive.ts';
|
||||
import { embedBatch } from './embedding.ts';
|
||||
import type { ChunkInput } from './types.ts';
|
||||
|
||||
export interface ImportFileResult {
|
||||
slug: string;
|
||||
status: 'imported' | 'skipped' | 'error';
|
||||
chunks: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
const MAX_FILE_SIZE = 1_000_000; // 1MB
|
||||
|
||||
export async function importFile(
|
||||
engine: BrainEngine,
|
||||
filePath: string,
|
||||
relativePath: string,
|
||||
opts: { noEmbed: boolean },
|
||||
): Promise<ImportFileResult> {
|
||||
// Skip files > 1MB
|
||||
const stat = statSync(filePath);
|
||||
if (stat.size > MAX_FILE_SIZE) {
|
||||
return { slug: relativePath, status: 'skipped', chunks: 0, error: `File too large (${stat.size} bytes)` };
|
||||
}
|
||||
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
const parsed = parseMarkdown(content, relativePath);
|
||||
const slug = parsed.slug;
|
||||
|
||||
// Check content hash for idempotency
|
||||
const hash = createHash('sha256')
|
||||
.update(parsed.compiled_truth + '\n---\n' + parsed.timeline)
|
||||
.digest('hex');
|
||||
|
||||
const existing = await engine.getPage(slug);
|
||||
if (existing?.content_hash === hash) {
|
||||
return { slug, status: 'skipped', chunks: 0 };
|
||||
}
|
||||
|
||||
// Upsert page
|
||||
await engine.putPage(slug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
|
||||
// Tag reconciliation: remove stale tags, add current ones
|
||||
const existingTags = await engine.getTags(slug);
|
||||
const newTags = new Set(parsed.tags);
|
||||
for (const oldTag of existingTags) {
|
||||
if (!newTags.has(oldTag)) {
|
||||
await engine.removeTag(slug, oldTag);
|
||||
}
|
||||
}
|
||||
for (const tag of parsed.tags) {
|
||||
await engine.addTag(slug, tag);
|
||||
}
|
||||
|
||||
// Chunk compiled_truth and timeline
|
||||
const chunks: ChunkInput[] = [];
|
||||
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
const ctChunks = chunkText(parsed.compiled_truth);
|
||||
for (const c of ctChunks) {
|
||||
chunks.push({
|
||||
chunk_index: chunks.length,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'compiled_truth',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (parsed.timeline.trim()) {
|
||||
const tlChunks = chunkText(parsed.timeline);
|
||||
for (const c of tlChunks) {
|
||||
chunks.push({
|
||||
chunk_index: chunks.length,
|
||||
chunk_text: c.text,
|
||||
chunk_source: 'timeline',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Embed if requested
|
||||
if (!opts.noEmbed && chunks.length > 0) {
|
||||
try {
|
||||
const embeddings = await embedBatch(chunks.map(c => c.chunk_text));
|
||||
for (let j = 0; j < chunks.length; j++) {
|
||||
chunks[j].embedding = embeddings[j];
|
||||
chunks[j].token_count = Math.ceil(chunks[j].chunk_text.length / 4);
|
||||
}
|
||||
} catch {
|
||||
// Embedding failure is non-fatal, chunks still saved without embeddings
|
||||
}
|
||||
}
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await engine.upsertChunks(slug, chunks);
|
||||
}
|
||||
|
||||
return { slug, status: 'imported', chunks: chunks.length };
|
||||
}
|
||||
@@ -520,6 +520,20 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows as unknown as IngestLogEntry[];
|
||||
}
|
||||
|
||||
// Sync
|
||||
async updateSlug(oldSlug: string, newSlug: string): Promise<void> {
|
||||
validateSlug(newSlug);
|
||||
const sql = db.getConnection();
|
||||
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug}`;
|
||||
}
|
||||
|
||||
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
|
||||
// Stub in v0.2. Links table uses integer page_id FKs, which are already
|
||||
// correct after updateSlug (page_id doesn't change, only slug does).
|
||||
// Textual [[wiki-links]] in compiled_truth are NOT rewritten here.
|
||||
// The maintain skill's dead link detector surfaces stale references.
|
||||
}
|
||||
|
||||
// Config
|
||||
async getConfig(key: string): Promise<string | null> {
|
||||
const sql = db.getConnection();
|
||||
|
||||
117
src/core/sync.ts
Normal file
117
src/core/sync.ts
Normal file
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* Sync utilities — pure functions for git diff parsing, filtering, and slug management.
|
||||
*
|
||||
* SYNC DATA FLOW:
|
||||
* git diff --name-status -M LAST..HEAD
|
||||
* │
|
||||
* buildSyncManifest() → parse A/M/D/R lines
|
||||
* │
|
||||
* isSyncable() → filter to .md pages only
|
||||
* │
|
||||
* pathToSlug() → convert file paths to page slugs
|
||||
*/
|
||||
|
||||
export interface SyncManifest {
|
||||
added: string[];
|
||||
modified: string[];
|
||||
deleted: string[];
|
||||
renamed: Array<{ from: string; to: string }>;
|
||||
}
|
||||
|
||||
export interface RawManifestEntry {
|
||||
action: 'A' | 'M' | 'D' | 'R';
|
||||
path: string;
|
||||
oldPath?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the output of `git diff --name-status -M LAST..HEAD` into structured entries.
|
||||
*
|
||||
* Input format (tab-separated):
|
||||
* A path/to/new-file.md
|
||||
* M path/to/modified-file.md
|
||||
* D path/to/deleted-file.md
|
||||
* R100 old/path.md new/path.md
|
||||
*/
|
||||
export function buildSyncManifest(gitDiffOutput: string): SyncManifest {
|
||||
const manifest: SyncManifest = {
|
||||
added: [],
|
||||
modified: [],
|
||||
deleted: [],
|
||||
renamed: [],
|
||||
};
|
||||
|
||||
const lines = gitDiffOutput.split('\n');
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
|
||||
const parts = trimmed.split('\t');
|
||||
if (parts.length < 2) continue;
|
||||
|
||||
const action = parts[0];
|
||||
const path = parts[parts.length === 3 ? 2 : 1]; // For renames, new path is 3rd column
|
||||
|
||||
if (action === 'A') {
|
||||
manifest.added.push(path);
|
||||
} else if (action === 'M') {
|
||||
manifest.modified.push(path);
|
||||
} else if (action === 'D') {
|
||||
manifest.deleted.push(parts[1]);
|
||||
} else if (action.startsWith('R')) {
|
||||
// Rename: R100\told-path\tnew-path
|
||||
const oldPath = parts[1];
|
||||
const newPath = parts[2];
|
||||
if (oldPath && newPath) {
|
||||
manifest.renamed.push({ from: oldPath, to: newPath });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return manifest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Filter a file path to determine if it should be synced to GBrain.
|
||||
*/
|
||||
export function isSyncable(path: string): boolean {
|
||||
// Must be .md
|
||||
if (!path.endsWith('.md')) return false;
|
||||
|
||||
// Skip hidden directories
|
||||
if (path.split('/').some(p => p.startsWith('.'))) return false;
|
||||
|
||||
// Skip .raw/ sidecar directories
|
||||
if (path.includes('.raw/')) return false;
|
||||
|
||||
// Skip meta files that aren't pages
|
||||
const skipFiles = ['schema.md', 'index.md', 'log.md', 'README.md'];
|
||||
const basename = path.split('/').pop() || '';
|
||||
if (skipFiles.includes(basename)) return false;
|
||||
|
||||
// Skip ops/ directory
|
||||
if (path.startsWith('ops/')) return false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a repo-relative file path to a GBrain page slug.
|
||||
*
|
||||
* Examples:
|
||||
* people/pedro-franceschi.md → people/pedro-franceschi
|
||||
* daily/2026-04-05.md → daily/2026-04-05
|
||||
* notes.md → notes
|
||||
*/
|
||||
export function pathToSlug(filePath: string, repoPrefix?: string): string {
|
||||
// Strip .md extension
|
||||
let slug = filePath.replace(/\.md$/, '');
|
||||
// Normalize separators
|
||||
slug = slug.replace(/\\/g, '/');
|
||||
// Strip leading slash
|
||||
slug = slug.replace(/^\//, '');
|
||||
// Add repo prefix for multi-repo setups
|
||||
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
|
||||
return slug;
|
||||
}
|
||||
@@ -7,10 +7,11 @@ import { expandQuery } from '../core/search/expansion.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { embedBatch } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { VERSION } from '../version.ts';
|
||||
|
||||
export async function startMcpServer(engine: BrainEngine) {
|
||||
const server = new Server(
|
||||
{ name: 'gbrain', version: '0.1.0' },
|
||||
{ name: 'gbrain', version: VERSION },
|
||||
{ capabilities: { tools: {} } },
|
||||
);
|
||||
|
||||
@@ -189,6 +190,17 @@ export async function handleToolCall(
|
||||
return { status: 'reverted' };
|
||||
}
|
||||
|
||||
case 'sync_brain': {
|
||||
const { performSync } = await import('../commands/sync.ts');
|
||||
return performSync(engine, {
|
||||
repoPath: params.repo as string | undefined,
|
||||
dryRun: (params.dry_run as boolean) || false,
|
||||
noEmbed: false,
|
||||
noPull: false,
|
||||
full: false,
|
||||
});
|
||||
}
|
||||
|
||||
default:
|
||||
throw new Error(`Unknown tool: ${tool}`);
|
||||
}
|
||||
@@ -216,5 +228,6 @@ function getToolDefinitions() {
|
||||
{ name: 'get_health', description: 'Brain health dashboard (embed coverage, stale pages, orphans)', inputSchema: { type: 'object', properties: {} } },
|
||||
{ name: 'get_versions', description: 'Page version history', inputSchema: { type: 'object', properties: { slug: { type: 'string' } }, required: ['slug'] } },
|
||||
{ name: 'revert_version', description: 'Revert page to a previous version', inputSchema: { type: 'object', properties: { slug: { type: 'string' }, version_id: { type: 'number' } }, required: ['slug', 'version_id'] } },
|
||||
{ name: 'sync_brain', description: 'Sync git repo to brain (incremental)', inputSchema: { type: 'object', properties: { repo: { type: 'string', description: 'Path to git repo (optional if configured)' }, dry_run: { type: 'boolean', description: 'Preview changes without applying' } } } },
|
||||
];
|
||||
}
|
||||
|
||||
@@ -141,6 +141,26 @@ INSERT INTO config (key, value) VALUES
|
||||
('chunk_strategy', 'semantic')
|
||||
ON CONFLICT (key) DO NOTHING;
|
||||
|
||||
-- ============================================================
|
||||
-- files: binary attachments stored in Supabase Storage
|
||||
-- ============================================================
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
id SERIAL PRIMARY KEY,
|
||||
page_slug TEXT REFERENCES pages(slug) ON DELETE SET NULL ON UPDATE CASCADE,
|
||||
filename TEXT NOT NULL,
|
||||
storage_path TEXT NOT NULL,
|
||||
storage_url TEXT NOT NULL,
|
||||
mime_type TEXT,
|
||||
size_bytes BIGINT,
|
||||
content_hash TEXT NOT NULL,
|
||||
metadata JSONB NOT NULL DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE(storage_path)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_files_page ON files(page_slug);
|
||||
CREATE INDEX IF NOT EXISTS idx_files_hash ON files(content_hash);
|
||||
|
||||
-- ============================================================
|
||||
-- Trigger-based search_vector (spans pages + timeline_entries)
|
||||
-- ============================================================
|
||||
|
||||
2
src/version.ts
Normal file
2
src/version.ts
Normal file
@@ -0,0 +1,2 @@
|
||||
import pkg from '../package.json';
|
||||
export const VERSION = pkg.version;
|
||||
@@ -74,4 +74,62 @@ describe('Recursive Text Chunker', () => {
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
expect(chunks[0].text).toContain('Bonjour');
|
||||
});
|
||||
|
||||
test('splits at single newline (line-level) when paragraphs are absent', () => {
|
||||
// Lines without double newlines should still split at single newlines
|
||||
const lines = Array(100).fill('This is a single line of text.').join('\n');
|
||||
const chunks = chunkText(lines, { chunkSize: 20 });
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
test('handles text with only whitespace delimiters (word-level split)', () => {
|
||||
// No sentences, no newlines, just words
|
||||
const words = Array(200).fill('word').join(' ');
|
||||
const chunks = chunkText(words, { chunkSize: 50 });
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const chunk of chunks) {
|
||||
expect(chunk.text.trim().length).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
|
||||
test('handles clause-level delimiters (semicolons, colons, commas)', () => {
|
||||
// Text with clauses but no sentence endings
|
||||
const text = Array(100).fill('clause one; clause two: clause three, clause four').join(' ');
|
||||
const chunks = chunkText(text, { chunkSize: 30 });
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
test('preserves content across chunks (lossless)', () => {
|
||||
const original = 'First paragraph.\n\nSecond paragraph.\n\nThird paragraph.';
|
||||
const chunks = chunkText(original, { chunkSize: 5, chunkOverlap: 0 });
|
||||
// With no overlap, all text should appear in chunks
|
||||
const reconstructed = chunks.map(c => c.text).join(' ');
|
||||
expect(reconstructed).toContain('First paragraph');
|
||||
expect(reconstructed).toContain('Second paragraph');
|
||||
expect(reconstructed).toContain('Third paragraph');
|
||||
});
|
||||
|
||||
test('default options produce reasonable chunks', () => {
|
||||
// Large text with defaults (300 words, 50 overlap)
|
||||
const text = Array(500).fill('This is a test sentence with several words.').join(' ');
|
||||
const chunks = chunkText(text);
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
for (const chunk of chunks) {
|
||||
const wordCount = chunk.text.split(/\s+/).length;
|
||||
// Should be roughly 300 words, with 1.5x tolerance
|
||||
expect(wordCount).toBeLessThanOrEqual(500);
|
||||
}
|
||||
});
|
||||
|
||||
test('handles mixed delimiter hierarchy', () => {
|
||||
const text = [
|
||||
'Paragraph one has sentences. And more sentences! Really?',
|
||||
'',
|
||||
'Paragraph two; with clauses: and more, clauses here.',
|
||||
'',
|
||||
'Paragraph three.\nWith line breaks.\nAnd more lines.',
|
||||
].join('\n');
|
||||
const chunks = chunkText(text, { chunkSize: 10 });
|
||||
expect(chunks.length).toBeGreaterThan(1);
|
||||
});
|
||||
});
|
||||
|
||||
183
test/cli.test.ts
Normal file
183
test/cli.test.ts
Normal file
@@ -0,0 +1,183 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
// Read cli.ts source to extract COMMAND_HELP keys and switch cases
|
||||
const cliSource = readFileSync(new URL('../src/cli.ts', import.meta.url), 'utf-8');
|
||||
|
||||
// Extract COMMAND_HELP keys from the map
|
||||
function extractCommandHelpKeys(source: string): string[] {
|
||||
const mapMatch = source.match(/const COMMAND_HELP:\s*Record<string,\s*string>\s*=\s*\{([\s\S]*?)\};/);
|
||||
if (!mapMatch) return [];
|
||||
const keys: string[] = [];
|
||||
for (const m of mapMatch[1].matchAll(/^\s*['"]?([a-z-]+)['"]?\s*:/gm)) {
|
||||
keys.push(m[1]);
|
||||
}
|
||||
return keys.sort();
|
||||
}
|
||||
|
||||
// Extract switch case labels from the switch(command) block
|
||||
function extractSwitchCases(source: string): string[] {
|
||||
const cases: string[] = [];
|
||||
for (const m of source.matchAll(/case\s+'([^']+)':\s*\{/g)) {
|
||||
cases.push(m[1]);
|
||||
}
|
||||
return [...new Set(cases)].sort();
|
||||
}
|
||||
|
||||
// Extract commands handled before the switch (init, upgrade)
|
||||
function extractEarlyCommands(source: string): string[] {
|
||||
const cmds: string[] = [];
|
||||
for (const m of source.matchAll(/if\s*\(command\s*===\s*'([^']+)'\)/g)) {
|
||||
if (!['--help', '-h', '--version', '--tools-json'].includes(m[1])) {
|
||||
cmds.push(m[1]);
|
||||
}
|
||||
}
|
||||
return [...new Set(cmds)].sort();
|
||||
}
|
||||
|
||||
describe('CLI COMMAND_HELP consistency', () => {
|
||||
const helpKeys = extractCommandHelpKeys(cliSource);
|
||||
const switchCases = extractSwitchCases(cliSource);
|
||||
const earlyCmds = extractEarlyCommands(cliSource);
|
||||
const allHandled = [...switchCases, ...earlyCmds].sort();
|
||||
|
||||
test('COMMAND_HELP has entries for all switch cases', () => {
|
||||
for (const cmd of switchCases) {
|
||||
expect(helpKeys).toContain(cmd);
|
||||
}
|
||||
});
|
||||
|
||||
test('COMMAND_HELP has entries for early-dispatch commands (init, upgrade)', () => {
|
||||
for (const cmd of earlyCmds) {
|
||||
expect(helpKeys).toContain(cmd);
|
||||
}
|
||||
});
|
||||
|
||||
test('every COMMAND_HELP key maps to a handled command', () => {
|
||||
for (const key of helpKeys) {
|
||||
expect(allHandled).toContain(key);
|
||||
}
|
||||
});
|
||||
|
||||
test('COMMAND_HELP has at least 25 entries', () => {
|
||||
expect(helpKeys.length).toBeGreaterThanOrEqual(25);
|
||||
});
|
||||
});
|
||||
|
||||
describe('CLI version', () => {
|
||||
test('VERSION matches package.json', async () => {
|
||||
const { VERSION } = await import('../src/version.ts');
|
||||
const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf-8'));
|
||||
expect(VERSION).toBe(pkg.version);
|
||||
});
|
||||
|
||||
test('VERSION is a valid semver string', async () => {
|
||||
const { VERSION } = await import('../src/version.ts');
|
||||
expect(VERSION).toMatch(/^\d+\.\d+\.\d+/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('CLI help text', () => {
|
||||
test('every COMMAND_HELP entry starts with Usage:', () => {
|
||||
const mapMatch = cliSource.match(/const COMMAND_HELP:\s*Record<string,\s*string>\s*=\s*\{([\s\S]*?)\};/);
|
||||
expect(mapMatch).not.toBeNull();
|
||||
// Verify by importing and checking
|
||||
const keys = extractCommandHelpKeys(cliSource);
|
||||
expect(keys.length).toBeGreaterThan(0);
|
||||
// Each help string in the source should contain 'Usage:'
|
||||
for (const key of keys) {
|
||||
const pattern = new RegExp(`['"]?${key.replace('-', '\\-')}['"]?:\\s*['"\`]([^'"\`]*)`);
|
||||
const match = cliSource.match(pattern);
|
||||
if (match) {
|
||||
expect(match[1]).toContain('Usage:');
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('CLI dispatch integration', () => {
|
||||
test('--version outputs version', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', '--version'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
await proc.exited;
|
||||
expect(stdout.trim()).toMatch(/^gbrain \d+\.\d+\.\d+/);
|
||||
});
|
||||
|
||||
test('unknown command prints error and exits 1', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'notacommand'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stderr = await new Response(proc.stderr).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stderr).toContain('Unknown command: notacommand');
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
test('per-command --help prints usage without DB connection', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'get', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('Usage: gbrain get');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('upgrade --help prints usage without running upgrade', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'upgrade', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('Usage: gbrain upgrade');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('init --help prints usage without running wizard', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'init', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('Usage: gbrain init');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('--help prints global help', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('USAGE');
|
||||
expect(stdout).toContain('gbrain <command>');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('files --help prints subcommand help', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'files', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('files list');
|
||||
expect(stdout).toContain('files upload');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
});
|
||||
63
test/config.test.ts
Normal file
63
test/config.test.ts
Normal file
@@ -0,0 +1,63 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
// redactUrl is not exported, so we test it by reading the source and
|
||||
// reimplementing the regex to verify the pattern, then test via CLI
|
||||
|
||||
// Extract the redactUrl regex pattern from source
|
||||
const configSource = readFileSync(
|
||||
new URL('../src/commands/config.ts', import.meta.url),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
// Reimplemented from source for unit testing
|
||||
function redactUrl(url: string): string {
|
||||
return url.replace(
|
||||
/(postgresql:\/\/[^:]+:)([^@]+)(@)/,
|
||||
'$1***$3',
|
||||
);
|
||||
}
|
||||
|
||||
describe('redactUrl', () => {
|
||||
test('redacts password in postgresql:// URL', () => {
|
||||
const url = 'postgresql://user:secretpass@host:5432/dbname';
|
||||
expect(redactUrl(url)).toBe('postgresql://user:***@host:5432/dbname');
|
||||
});
|
||||
|
||||
test('redacts complex passwords with special chars', () => {
|
||||
const url = 'postgresql://postgres:p@ss!w0rd#123@db.supabase.co:5432/postgres';
|
||||
// The regex is greedy on [^@]+ so it captures up to the LAST @
|
||||
const result = redactUrl(url);
|
||||
expect(result).not.toContain('p@ss');
|
||||
expect(result).toContain('***');
|
||||
});
|
||||
|
||||
test('returns non-postgresql URLs unchanged', () => {
|
||||
const url = 'https://example.com/api';
|
||||
expect(redactUrl(url)).toBe(url);
|
||||
});
|
||||
|
||||
test('returns plain strings unchanged', () => {
|
||||
expect(redactUrl('hello')).toBe('hello');
|
||||
});
|
||||
|
||||
test('handles URL without password', () => {
|
||||
const url = 'postgresql://user@host:5432/dbname';
|
||||
// No colon after user means regex doesn't match
|
||||
expect(redactUrl(url)).toBe(url);
|
||||
});
|
||||
|
||||
test('handles empty string', () => {
|
||||
expect(redactUrl('')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('config source correctness', () => {
|
||||
test('redactUrl function exists in config.ts', () => {
|
||||
expect(configSource).toContain('function redactUrl');
|
||||
});
|
||||
|
||||
test('redactUrl uses the correct regex pattern', () => {
|
||||
expect(configSource).toContain('postgresql:\\/\\/');
|
||||
});
|
||||
});
|
||||
178
test/files.test.ts
Normal file
178
test/files.test.ts
Normal file
@@ -0,0 +1,178 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import { extname } from 'path';
|
||||
|
||||
const TMP = join(import.meta.dir, '.tmp-files-test');
|
||||
|
||||
// These functions are not exported from files.ts, so we reimplement and test
|
||||
// the logic patterns to ensure correctness. If they ever get exported, switch
|
||||
// to direct imports.
|
||||
|
||||
const MIME_TYPES: Record<string, string> = {
|
||||
'.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png',
|
||||
'.gif': 'image/gif', '.webp': 'image/webp', '.svg': 'image/svg+xml',
|
||||
'.pdf': 'application/pdf', '.mp4': 'video/mp4', '.m4a': 'audio/mp4',
|
||||
'.mp3': 'audio/mpeg', '.wav': 'audio/wav', '.heic': 'image/heic',
|
||||
'.tiff': 'image/tiff', '.tif': 'image/tiff', '.dng': 'image/x-adobe-dng',
|
||||
'.doc': 'application/msword',
|
||||
'.docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
'.xls': 'application/vnd.ms-excel',
|
||||
'.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||
};
|
||||
|
||||
function getMimeType(filePath: string): string | null {
|
||||
const ext = extname(filePath).toLowerCase();
|
||||
return MIME_TYPES[ext] || null;
|
||||
}
|
||||
|
||||
function fileHash(content: Buffer): string {
|
||||
return createHash('sha256').update(content).digest('hex');
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
mkdirSync(TMP, { recursive: true });
|
||||
mkdirSync(join(TMP, 'subdir'), { recursive: true });
|
||||
mkdirSync(join(TMP, '.hidden'), { recursive: true });
|
||||
writeFileSync(join(TMP, 'photo.jpg'), 'fake-jpg');
|
||||
writeFileSync(join(TMP, 'doc.pdf'), 'fake-pdf');
|
||||
writeFileSync(join(TMP, 'notes.md'), '# Markdown');
|
||||
writeFileSync(join(TMP, 'data.csv'), 'a,b,c');
|
||||
writeFileSync(join(TMP, 'subdir', 'nested.png'), 'fake-png');
|
||||
writeFileSync(join(TMP, '.hidden', 'secret.txt'), 'hidden');
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(TMP, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('getMimeType', () => {
|
||||
test('returns correct MIME for .jpg', () => {
|
||||
expect(getMimeType('photo.jpg')).toBe('image/jpeg');
|
||||
});
|
||||
|
||||
test('returns correct MIME for .jpeg', () => {
|
||||
expect(getMimeType('photo.jpeg')).toBe('image/jpeg');
|
||||
});
|
||||
|
||||
test('returns correct MIME for .png', () => {
|
||||
expect(getMimeType('image.png')).toBe('image/png');
|
||||
});
|
||||
|
||||
test('returns correct MIME for .pdf', () => {
|
||||
expect(getMimeType('doc.pdf')).toBe('application/pdf');
|
||||
});
|
||||
|
||||
test('returns correct MIME for .mp4', () => {
|
||||
expect(getMimeType('video.mp4')).toBe('video/mp4');
|
||||
});
|
||||
|
||||
test('returns correct MIME for .svg', () => {
|
||||
expect(getMimeType('icon.svg')).toBe('image/svg+xml');
|
||||
});
|
||||
|
||||
test('handles uppercase extensions via toLowerCase', () => {
|
||||
expect(getMimeType('PHOTO.JPG')).toBe('image/jpeg');
|
||||
expect(getMimeType('doc.PDF')).toBe('application/pdf');
|
||||
});
|
||||
|
||||
test('returns null for unknown extensions', () => {
|
||||
expect(getMimeType('data.csv')).toBeNull();
|
||||
expect(getMimeType('script.ts')).toBeNull();
|
||||
expect(getMimeType('readme.md')).toBeNull();
|
||||
});
|
||||
|
||||
test('returns null for files without extension', () => {
|
||||
expect(getMimeType('Makefile')).toBeNull();
|
||||
});
|
||||
|
||||
test('handles .docx and .xlsx', () => {
|
||||
expect(getMimeType('report.docx')).toContain('wordprocessingml');
|
||||
expect(getMimeType('sheet.xlsx')).toContain('spreadsheetml');
|
||||
});
|
||||
|
||||
test('handles .heic (iPhone photos)', () => {
|
||||
expect(getMimeType('IMG_0001.heic')).toBe('image/heic');
|
||||
});
|
||||
|
||||
test('handles .dng (raw photos)', () => {
|
||||
expect(getMimeType('RAW_001.dng')).toBe('image/x-adobe-dng');
|
||||
});
|
||||
});
|
||||
|
||||
describe('fileHash', () => {
|
||||
test('produces consistent SHA-256 hash', () => {
|
||||
const content = Buffer.from('hello world');
|
||||
const hash1 = fileHash(content);
|
||||
const hash2 = fileHash(content);
|
||||
expect(hash1).toBe(hash2);
|
||||
expect(hash1).toHaveLength(64); // SHA-256 hex = 64 chars
|
||||
});
|
||||
|
||||
test('different content produces different hash', () => {
|
||||
const hash1 = fileHash(Buffer.from('hello'));
|
||||
const hash2 = fileHash(Buffer.from('world'));
|
||||
expect(hash1).not.toBe(hash2);
|
||||
});
|
||||
|
||||
test('empty content produces valid hash', () => {
|
||||
const hash = fileHash(Buffer.from(''));
|
||||
expect(hash).toHaveLength(64);
|
||||
});
|
||||
});
|
||||
|
||||
describe('collectFiles pattern (non-markdown, skip hidden)', () => {
|
||||
// Reimplementing collectFiles logic to test the pattern
|
||||
const { readdirSync, statSync } = require('fs');
|
||||
|
||||
function collectFiles(dir: string): string[] {
|
||||
const files: string[] = [];
|
||||
function walk(d: string) {
|
||||
for (const entry of readdirSync(d)) {
|
||||
if (entry.startsWith('.')) continue;
|
||||
const full = join(d, entry);
|
||||
const stat = statSync(full);
|
||||
if (stat.isDirectory()) {
|
||||
walk(full);
|
||||
} else if (!entry.endsWith('.md')) {
|
||||
files.push(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(dir);
|
||||
return files.sort();
|
||||
}
|
||||
|
||||
test('finds non-markdown files', () => {
|
||||
const files = collectFiles(TMP);
|
||||
const basenames = files.map(f => f.split('/').pop());
|
||||
expect(basenames).toContain('photo.jpg');
|
||||
expect(basenames).toContain('doc.pdf');
|
||||
expect(basenames).toContain('data.csv');
|
||||
});
|
||||
|
||||
test('skips .md files', () => {
|
||||
const files = collectFiles(TMP);
|
||||
const mdFiles = files.filter(f => f.endsWith('.md'));
|
||||
expect(mdFiles).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('skips hidden directories', () => {
|
||||
const files = collectFiles(TMP);
|
||||
const hiddenFiles = files.filter(f => f.includes('.hidden'));
|
||||
expect(hiddenFiles).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('recurses into subdirectories', () => {
|
||||
const files = collectFiles(TMP);
|
||||
const nested = files.filter(f => f.includes('subdir'));
|
||||
expect(nested.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('returns sorted paths', () => {
|
||||
const files = collectFiles(TMP);
|
||||
const sorted = [...files].sort();
|
||||
expect(files).toEqual(sorted);
|
||||
});
|
||||
});
|
||||
271
test/import-file.test.ts
Normal file
271
test/import-file.test.ts
Normal file
@@ -0,0 +1,271 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { importFile } from '../src/core/import-file.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
const TMP = join(import.meta.dir, '.tmp-import-test');
|
||||
|
||||
// Minimal mock engine that tracks calls
|
||||
function mockEngine(overrides: Partial<Record<string, any>> = {}): BrainEngine {
|
||||
const calls: { method: string; args: any[] }[] = [];
|
||||
const track = (method: string) => (...args: any[]) => {
|
||||
calls.push({ method, args });
|
||||
if (overrides[method]) return overrides[method](...args);
|
||||
return Promise.resolve(null);
|
||||
};
|
||||
|
||||
const engine = new Proxy({} as any, {
|
||||
get(_, prop: string) {
|
||||
if (prop === '_calls') return calls;
|
||||
if (prop === 'getTags') return overrides.getTags || (() => Promise.resolve([]));
|
||||
if (prop === 'getPage') return overrides.getPage || (() => Promise.resolve(null));
|
||||
return track(prop);
|
||||
},
|
||||
});
|
||||
return engine;
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
mkdirSync(TMP, { recursive: true });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(TMP, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('importFile', () => {
|
||||
test('imports a valid markdown file', async () => {
|
||||
const filePath = join(TMP, 'test-page.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Test Page
|
||||
tags: [alpha, beta]
|
||||
---
|
||||
|
||||
This is the compiled truth.
|
||||
|
||||
---
|
||||
|
||||
- 2024-01-01: Something happened.
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'concepts/test-page.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
expect(result.slug).toBe('concepts/test-page');
|
||||
expect(result.chunks).toBeGreaterThan(0);
|
||||
|
||||
// Verify engine was called correctly
|
||||
const calls = (engine as any)._calls;
|
||||
const putCall = calls.find((c: any) => c.method === 'putPage');
|
||||
expect(putCall).toBeTruthy();
|
||||
expect(putCall.args[0]).toBe('concepts/test-page');
|
||||
|
||||
// Tags were added
|
||||
const tagCalls = calls.filter((c: any) => c.method === 'addTag');
|
||||
expect(tagCalls.length).toBe(2);
|
||||
|
||||
// Chunks were upserted
|
||||
const chunkCall = calls.find((c: any) => c.method === 'upsertChunks');
|
||||
expect(chunkCall).toBeTruthy();
|
||||
});
|
||||
|
||||
test('skips files larger than MAX_FILE_SIZE (1MB)', async () => {
|
||||
const filePath = join(TMP, 'big-file.md');
|
||||
// Create a file > 1MB
|
||||
const bigContent = '---\ntitle: Big\n---\n' + 'x'.repeat(1_100_000);
|
||||
writeFileSync(filePath, bigContent);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'big-file.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('skipped');
|
||||
expect(result.error).toContain('too large');
|
||||
// Engine should NOT have been called
|
||||
expect((engine as any)._calls.length).toBe(0);
|
||||
});
|
||||
|
||||
test('skips file when content hash matches (idempotent)', async () => {
|
||||
const filePath = join(TMP, 'unchanged.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Unchanged
|
||||
---
|
||||
|
||||
Same content.
|
||||
`);
|
||||
|
||||
// Mock engine returns a page with matching hash
|
||||
const { createHash } = await import('crypto');
|
||||
const hash = createHash('sha256')
|
||||
.update('Same content.\n---\n')
|
||||
.digest('hex');
|
||||
|
||||
const engine = mockEngine({
|
||||
getPage: () => Promise.resolve({ content_hash: hash }),
|
||||
});
|
||||
|
||||
const result = await importFile(engine, filePath, 'concepts/unchanged.md', { noEmbed: true });
|
||||
expect(result.status).toBe('skipped');
|
||||
|
||||
// putPage should NOT have been called
|
||||
const calls = (engine as any)._calls;
|
||||
const putCall = calls.find((c: any) => c.method === 'putPage');
|
||||
expect(putCall).toBeUndefined();
|
||||
});
|
||||
|
||||
test('reconciles tags: removes old, adds new', async () => {
|
||||
const filePath = join(TMP, 'retag.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Retagged
|
||||
tags: [new-tag, kept-tag]
|
||||
---
|
||||
|
||||
Content here.
|
||||
`);
|
||||
|
||||
const engine = mockEngine({
|
||||
getTags: () => Promise.resolve(['old-tag', 'kept-tag']),
|
||||
getPage: () => Promise.resolve(null),
|
||||
});
|
||||
|
||||
await importFile(engine, filePath, 'concepts/retag.md', { noEmbed: true });
|
||||
|
||||
const calls = (engine as any)._calls;
|
||||
const removeCalls = calls.filter((c: any) => c.method === 'removeTag');
|
||||
const addCalls = calls.filter((c: any) => c.method === 'addTag');
|
||||
|
||||
// old-tag should be removed (not in new set)
|
||||
expect(removeCalls.length).toBe(1);
|
||||
expect(removeCalls[0].args[1]).toBe('old-tag');
|
||||
|
||||
// new-tag and kept-tag should be added
|
||||
expect(addCalls.length).toBe(2);
|
||||
});
|
||||
|
||||
test('chunks compiled_truth and timeline separately', async () => {
|
||||
const filePath = join(TMP, 'chunked.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Chunked
|
||||
---
|
||||
|
||||
This is compiled truth content that should be chunked as compiled_truth source.
|
||||
|
||||
---
|
||||
|
||||
- 2024-01-01: This is timeline content that should be chunked as timeline source.
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'concepts/chunked.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
expect(result.chunks).toBeGreaterThanOrEqual(2); // at least 1 CT + 1 TL
|
||||
|
||||
const calls = (engine as any)._calls;
|
||||
const chunkCall = calls.find((c: any) => c.method === 'upsertChunks');
|
||||
const chunks = chunkCall.args[1];
|
||||
|
||||
const ctChunks = chunks.filter((c: any) => c.chunk_source === 'compiled_truth');
|
||||
const tlChunks = chunks.filter((c: any) => c.chunk_source === 'timeline');
|
||||
expect(ctChunks.length).toBeGreaterThan(0);
|
||||
expect(tlChunks.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('handles file with minimal content', async () => {
|
||||
const filePath = join(TMP, 'minimal.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Minimal
|
||||
---
|
||||
|
||||
One line.
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'concepts/minimal.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
expect(result.chunks).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('skips chunking for empty timeline', async () => {
|
||||
const filePath = join(TMP, 'empty-tl.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: No Timeline
|
||||
---
|
||||
|
||||
Just compiled truth, no timeline separator.
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'concepts/empty-tl.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
|
||||
const calls = (engine as any)._calls;
|
||||
const chunkCall = calls.find((c: any) => c.method === 'upsertChunks');
|
||||
if (chunkCall) {
|
||||
const chunks = chunkCall.args[1];
|
||||
const tlChunks = chunks.filter((c: any) => c.chunk_source === 'timeline');
|
||||
expect(tlChunks.length).toBe(0);
|
||||
}
|
||||
});
|
||||
|
||||
test('noEmbed: true skips embedding', async () => {
|
||||
const filePath = join(TMP, 'no-embed.md');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: No Embed
|
||||
---
|
||||
|
||||
Content to chunk but not embed.
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
const result = await importFile(engine, filePath, 'concepts/no-embed.md', { noEmbed: true });
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
// Chunks should NOT have embeddings
|
||||
const calls = (engine as any)._calls;
|
||||
const chunkCall = calls.find((c: any) => c.method === 'upsertChunks');
|
||||
if (chunkCall) {
|
||||
for (const chunk of chunkCall.args[1]) {
|
||||
expect(chunk.embedding).toBeUndefined();
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('assigns sequential chunk_index values', async () => {
|
||||
const filePath = join(TMP, 'indexed.md');
|
||||
const longText = Array(50).fill('This is a sentence that adds length to the content.').join(' ');
|
||||
writeFileSync(filePath, `---
|
||||
type: concept
|
||||
title: Indexed
|
||||
---
|
||||
|
||||
${longText}
|
||||
|
||||
---
|
||||
|
||||
${longText}
|
||||
`);
|
||||
|
||||
const engine = mockEngine();
|
||||
await importFile(engine, filePath, 'concepts/indexed.md', { noEmbed: true });
|
||||
|
||||
const calls = (engine as any)._calls;
|
||||
const chunkCall = calls.find((c: any) => c.method === 'upsertChunks');
|
||||
if (chunkCall) {
|
||||
const chunks = chunkCall.args[1];
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
expect(chunks[i].chunk_index).toBe(i);
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -146,3 +146,57 @@ Paul Graham argues that startups should do unscalable things early on.
|
||||
expect(reparsed.frontmatter.custom).toBe('value');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseMarkdown edge cases', () => {
|
||||
test('handles content with multiple --- separators', () => {
|
||||
const md = `---
|
||||
type: concept
|
||||
title: Test
|
||||
---
|
||||
|
||||
First section.
|
||||
|
||||
---
|
||||
|
||||
Timeline part 1.
|
||||
|
||||
---
|
||||
|
||||
More timeline.`;
|
||||
const parsed = parseMarkdown(md);
|
||||
// Only splits at the FIRST standalone ---
|
||||
expect(parsed.compiled_truth.trim()).toBe('First section.');
|
||||
expect(parsed.timeline).toContain('Timeline part 1.');
|
||||
expect(parsed.timeline).toContain('More timeline.');
|
||||
});
|
||||
|
||||
test('handles frontmatter without type or title', () => {
|
||||
const md = `---
|
||||
custom_field: hello
|
||||
---
|
||||
|
||||
Some content.`;
|
||||
const parsed = parseMarkdown(md);
|
||||
expect(parsed.type).toBeTruthy(); // should have a default
|
||||
expect(parsed.compiled_truth.trim()).toBe('Some content.');
|
||||
expect(parsed.frontmatter.custom_field).toBe('hello');
|
||||
});
|
||||
|
||||
test('handles content with no frontmatter at all', () => {
|
||||
const md = `Just plain text with no YAML.`;
|
||||
const parsed = parseMarkdown(md);
|
||||
expect(parsed.compiled_truth).toContain('Just plain text');
|
||||
});
|
||||
|
||||
test('handles empty string', () => {
|
||||
const parsed = parseMarkdown('');
|
||||
expect(parsed.compiled_truth).toBe('');
|
||||
expect(parsed.timeline).toBe('');
|
||||
});
|
||||
|
||||
test('infers type from various directory paths', () => {
|
||||
expect(parseMarkdown('', 'people/someone.md').type).toBe('person');
|
||||
expect(parseMarkdown('', 'concepts/thing.md').type).toBe('concept');
|
||||
expect(parseMarkdown('', 'companies/acme.md').type).toBe('company');
|
||||
});
|
||||
});
|
||||
|
||||
179
test/sync.test.ts
Normal file
179
test/sync.test.ts
Normal file
@@ -0,0 +1,179 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { buildSyncManifest, isSyncable, pathToSlug } from '../src/core/sync.ts';
|
||||
|
||||
describe('buildSyncManifest', () => {
|
||||
test('parses A/M/D entries from single commit', () => {
|
||||
const output = `A\tpeople/new-person.md\nM\tpeople/existing-person.md\nD\tpeople/deleted-person.md`;
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.added).toEqual(['people/new-person.md']);
|
||||
expect(manifest.modified).toEqual(['people/existing-person.md']);
|
||||
expect(manifest.deleted).toEqual(['people/deleted-person.md']);
|
||||
expect(manifest.renamed).toEqual([]);
|
||||
});
|
||||
|
||||
test('parses R100 rename entries', () => {
|
||||
const output = `R100\tpeople/old-name.md\tpeople/new-name.md`;
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.renamed).toEqual([{ from: 'people/old-name.md', to: 'people/new-name.md' }]);
|
||||
expect(manifest.added).toEqual([]);
|
||||
expect(manifest.modified).toEqual([]);
|
||||
expect(manifest.deleted).toEqual([]);
|
||||
});
|
||||
|
||||
test('parses partial rename (R075)', () => {
|
||||
const output = `R075\tpeople/old.md\tpeople/new.md`;
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.renamed).toEqual([{ from: 'people/old.md', to: 'people/new.md' }]);
|
||||
});
|
||||
|
||||
test('handles empty diff', () => {
|
||||
const manifest = buildSyncManifest('');
|
||||
expect(manifest.added).toEqual([]);
|
||||
expect(manifest.modified).toEqual([]);
|
||||
expect(manifest.deleted).toEqual([]);
|
||||
expect(manifest.renamed).toEqual([]);
|
||||
});
|
||||
|
||||
test('handles mixed entries with blank lines', () => {
|
||||
const output = `A\tpeople/a.md\n\nM\tpeople/b.md\n\nD\tpeople/c.md`;
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.added).toEqual(['people/a.md']);
|
||||
expect(manifest.modified).toEqual(['people/b.md']);
|
||||
expect(manifest.deleted).toEqual(['people/c.md']);
|
||||
});
|
||||
|
||||
test('skips malformed lines', () => {
|
||||
const output = `A\tpeople/a.md\ngarbage line\nM\tpeople/b.md`;
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.added).toEqual(['people/a.md']);
|
||||
expect(manifest.modified).toEqual(['people/b.md']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSyncable', () => {
|
||||
test('accepts normal .md files', () => {
|
||||
expect(isSyncable('people/pedro-franceschi.md')).toBe(true);
|
||||
expect(isSyncable('meetings/2026-04-03-lunch.md')).toBe(true);
|
||||
expect(isSyncable('daily/2026-04-05.md')).toBe(true);
|
||||
expect(isSyncable('notes.md')).toBe(true);
|
||||
});
|
||||
|
||||
test('rejects non-.md files', () => {
|
||||
expect(isSyncable('people/photo.jpg')).toBe(false);
|
||||
expect(isSyncable('config.json')).toBe(false);
|
||||
expect(isSyncable('src/cli.ts')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects files in hidden directories', () => {
|
||||
expect(isSyncable('.git/config')).toBe(false);
|
||||
expect(isSyncable('.obsidian/plugins.md')).toBe(false);
|
||||
expect(isSyncable('people/.hidden/secret.md')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects .raw/ sidecar directories', () => {
|
||||
expect(isSyncable('people/pedro.raw/source.md')).toBe(false);
|
||||
expect(isSyncable('dir/.raw/notes.md')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects skip-list basenames', () => {
|
||||
expect(isSyncable('schema.md')).toBe(false);
|
||||
expect(isSyncable('index.md')).toBe(false);
|
||||
expect(isSyncable('log.md')).toBe(false);
|
||||
expect(isSyncable('README.md')).toBe(false);
|
||||
expect(isSyncable('people/README.md')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects ops/ directory', () => {
|
||||
expect(isSyncable('ops/deploy-log.md')).toBe(false);
|
||||
expect(isSyncable('ops/config.md')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('pathToSlug', () => {
|
||||
test('strips .md extension', () => {
|
||||
expect(pathToSlug('people/pedro-franceschi.md')).toBe('people/pedro-franceschi');
|
||||
});
|
||||
|
||||
test('preserves case', () => {
|
||||
expect(pathToSlug('People/Pedro-Franceschi.md')).toBe('People/Pedro-Franceschi');
|
||||
});
|
||||
|
||||
test('strips leading slash', () => {
|
||||
expect(pathToSlug('/people/pedro.md')).toBe('people/pedro');
|
||||
});
|
||||
|
||||
test('normalizes backslash separators', () => {
|
||||
expect(pathToSlug('people\\pedro.md')).toBe('people/pedro');
|
||||
});
|
||||
|
||||
test('handles flat files', () => {
|
||||
expect(pathToSlug('notes.md')).toBe('notes');
|
||||
});
|
||||
|
||||
test('handles nested paths', () => {
|
||||
expect(pathToSlug('projects/gbrain/spec.md')).toBe('projects/gbrain/spec');
|
||||
});
|
||||
|
||||
test('adds repo prefix when provided', () => {
|
||||
expect(pathToSlug('people/pedro.md', 'brain')).toBe('brain/people/pedro');
|
||||
});
|
||||
|
||||
test('no prefix when not provided', () => {
|
||||
expect(pathToSlug('people/pedro.md')).toBe('people/pedro');
|
||||
});
|
||||
|
||||
test('handles empty string', () => {
|
||||
expect(pathToSlug('')).toBe('');
|
||||
});
|
||||
|
||||
test('handles file with only extension', () => {
|
||||
expect(pathToSlug('.md')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSyncable edge cases', () => {
|
||||
test('rejects uppercase .MD extension', () => {
|
||||
// isSyncable checks path.endsWith('.md'), so .MD should fail
|
||||
expect(isSyncable('people/someone.MD')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects files with no extension', () => {
|
||||
expect(isSyncable('README')).toBe(false);
|
||||
});
|
||||
|
||||
test('accepts deeply nested .md files', () => {
|
||||
expect(isSyncable('a/b/c/d/e/f/deep.md')).toBe(true);
|
||||
});
|
||||
|
||||
test('rejects .md files inside nested hidden dirs', () => {
|
||||
expect(isSyncable('docs/.internal/secret.md')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildSyncManifest edge cases', () => {
|
||||
test('handles tab-separated fields correctly', () => {
|
||||
const output = "A\tpath/to/file.md";
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.added).toEqual(['path/to/file.md']);
|
||||
});
|
||||
|
||||
test('handles multiple renames', () => {
|
||||
const output = [
|
||||
'R100\told/a.md\tnew/a.md',
|
||||
'R095\told/b.md\tnew/b.md',
|
||||
].join('\n');
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.renamed).toHaveLength(2);
|
||||
expect(manifest.renamed[0].from).toBe('old/a.md');
|
||||
expect(manifest.renamed[1].from).toBe('old/b.md');
|
||||
});
|
||||
|
||||
test('ignores unknown status codes', () => {
|
||||
const output = "X\tunknown/file.md";
|
||||
const manifest = buildSyncManifest(output);
|
||||
expect(manifest.added).toEqual([]);
|
||||
expect(manifest.modified).toEqual([]);
|
||||
expect(manifest.deleted).toEqual([]);
|
||||
expect(manifest.renamed).toEqual([]);
|
||||
});
|
||||
});
|
||||
74
test/upgrade.test.ts
Normal file
74
test/upgrade.test.ts
Normal file
@@ -0,0 +1,74 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
// We can't easily mock process.execPath in bun, so we test the upgrade
|
||||
// command's --help output and the detection logic via subprocess
|
||||
|
||||
describe('upgrade command', () => {
|
||||
test('--help prints usage and exits 0', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'upgrade', '--help'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('Usage: gbrain upgrade');
|
||||
expect(stdout).toContain('Detects install method');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
|
||||
test('-h also prints usage', async () => {
|
||||
const proc = Bun.spawn(['bun', 'run', 'src/cli.ts', 'upgrade', '-h'], {
|
||||
cwd: new URL('..', import.meta.url).pathname,
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const stdout = await new Response(proc.stdout).text();
|
||||
const exitCode = await proc.exited;
|
||||
expect(stdout).toContain('Usage: gbrain upgrade');
|
||||
expect(exitCode).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('detectInstallMethod heuristic (source analysis)', () => {
|
||||
// Read the source and verify the detection order is correct
|
||||
const { readFileSync } = require('fs');
|
||||
const source = readFileSync(
|
||||
new URL('../src/commands/upgrade.ts', import.meta.url),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
test('checks node_modules before binary', () => {
|
||||
const nodeModulesIdx = source.indexOf('node_modules');
|
||||
const binaryIdx = source.indexOf("endsWith('/gbrain')");
|
||||
expect(nodeModulesIdx).toBeLessThan(binaryIdx);
|
||||
});
|
||||
|
||||
test('checks binary before clawhub', () => {
|
||||
const binaryIdx = source.indexOf("endsWith('/gbrain')");
|
||||
const clawhubIdx = source.indexOf("clawhub --version");
|
||||
expect(binaryIdx).toBeLessThan(clawhubIdx);
|
||||
});
|
||||
|
||||
test('uses clawhub --version, not which clawhub', () => {
|
||||
expect(source).toContain("clawhub --version");
|
||||
expect(source).not.toContain('which clawhub');
|
||||
});
|
||||
|
||||
test('has timeout on upgrade execSync calls', () => {
|
||||
// Count timeout occurrences in execSync calls
|
||||
const timeoutMatches = source.match(/timeout:\s*\d+/g) || [];
|
||||
expect(timeoutMatches.length).toBeGreaterThanOrEqual(2); // bun + clawhub detection at minimum
|
||||
});
|
||||
|
||||
test('return type is bun | binary | clawhub | unknown', () => {
|
||||
expect(source).toContain("'bun' | 'binary' | 'clawhub' | 'unknown'");
|
||||
});
|
||||
|
||||
test('does not reference npm in case labels or messages', () => {
|
||||
// Should not have case 'npm' or 'Upgrading via npm'
|
||||
expect(source).not.toContain("case 'npm'");
|
||||
expect(source).not.toContain('via npm');
|
||||
expect(source).not.toContain('npm upgrade');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user