feat: GBrain v0.1.0 — Postgres-native personal knowledge brain (#1)
* chore: add CLAUDE.md with project context and gstack skill routing rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: initialize project with Bun + TypeScript package.json with dependencies (postgres, pgvector, openai, anthropic, MCP SDK, gray-matter). TypeScript config targeting ESNext with bundler module resolution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add foundation layer — engine interface, Postgres engine, schema BrainEngine pluggable interface with full PostgresEngine: CRUD, search (keyword + vector), links, tags, timeline, versions, stats, health, ingest log, config. Trigger-based tsvector spanning pages + timeline_entries. Markdown parser with frontmatter, compiled_truth / timeline splitting, and round-trip serialization. 19 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 3-tier chunking and embedding service Recursive delimiter-aware chunker (5-level hierarchy, 300-word chunks, 50-word overlap). Semantic chunker with Savitzky-Golay boundary detection and recursive fallback. LLM-guided chunker via Claude Haiku with sliding window topic detection. OpenAI embedding service with batch support, exponential backoff, and rate limit handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add hybrid search with RRF fusion, expansion, and 4-layer dedup Hybrid search merges vector (pgvector HNSW) + keyword (tsvector) via Reciprocal Rank Fusion. Multi-query expansion via Claude Haiku generates 2 alternative phrasings. 4-layer dedup pipeline: by source, cosine similarity, type diversity (60% cap), per-page cap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add GBRAIN_V0 spec, pluggable engine architecture, SQLite engine plan GBRAIN_V0.md: full product spec with architecture decisions, CLI commands, schema, search architecture, chunking strategies, first-time experience, and future plans. ENGINES.md: pluggable engine interface, capability matrix, how to add new backends. SQLITE_ENGINE.md: complete SQLite implementation plan with schema, FTS5 setup, vector search options, and contributor guide. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add CLI with all commands Full CLI dispatcher with 25+ commands: init (Supabase wizard), get, put, delete, list, search, query (hybrid RRF), import (bulk with progress bar), export (round-trip), embed, stats, health, tag/untag/tags, link/unlink/ backlinks/graph, timeline/timeline-add, history/revert, config, upgrade, serve, call. Smart slug resolution on reads. Version snapshots on updates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add MCP stdio server with all brain tools 20 MCP tools mirroring CLI operations: get/put/delete/list pages, search (keyword), query (hybrid RRF + expansion), tags, links with graph traversal, timeline, stats, health, version history, and revert. Auto-chunks and embeds on put_page. CLI and MCP share the same engine. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 6 skill files and ClawHub manifest Fat markdown skills for AI agents: ingest (meetings/docs/articles with timeline merge), query (3-layer search + synthesis + citations), maintain (health checks, stale detection, orphan audit), enrich (external API enrichment), briefing (daily briefing compilation), migrate (universal migration from Obsidian/Notion/Logseq/markdown/CSV/JSON/Roam). ClawHub manifest for skill distribution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add README, CONTRIBUTING, update CLAUDE.md test references README with quickstart, commands, architecture, library usage, MCP setup, and links to design docs. CONTRIBUTING with setup, project structure, and guides for adding commands and engines. CLAUDE.md updated to reference actual test files instead of planned-but-unwritten import test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address adversarial review findings — 5 critical/high fixes - revertToVersion: add page_id check to prevent cross-page data corruption - traverseGraph: use UNION instead of UNION ALL for cycle safety - embedAll: preserve all chunks when embedding stale subset only - embedding: throw on retry exhaustion instead of returning zero vectors - putPage: validate slugs to prevent path traversal on export Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.1.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: expand README with schema, install, search architecture, and motivation Why it exists, how search works (with ASCII diagram), full database schema with all 9 tables and index details, chunking strategies explained, storage estimates, setup wizard walkthrough, knowledge model with example page, library usage with more examples, expanded skills table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: add MIT license (Copyright 2026 Garry Tan) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add OpenClaw install flow as primary option in README OpenClaw users just say "install gbrain" and the orchestrator handles everything: package install, Supabase setup wizard, skill registration. Shows the conversational interface for querying, ingesting, and briefings. ClawHub and standalone CLI paths follow as alternatives. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add prerequisites and explicit OpenClaw install instructions Prerequisites table listing Supabase, OpenAI, and Anthropic dependencies with links. Environment variable setup. Explicit step-by-step prompt for OpenClaw users showing exactly what to tell the orchestrator. Note that search degrades gracefully without API keys (keyword-only without OpenAI, no expansion without Anthropic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: scrub named references, add PG essay demo section to README Replace all Pedro/Brex/Jensen Huang/River AI examples with Paul Graham essay examples using the kindling corpus. Add "Try it" section to README showing the power of hybrid search on PG essays in 90 seconds. Update test fixtures to use concept pages instead of person pages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
545
docs/GBRAIN_V0.md
Normal file
545
docs/GBRAIN_V0.md
Normal file
@@ -0,0 +1,545 @@
|
||||
# GBrain v0: Postgres-Native Personal Knowledge Brain
|
||||
|
||||
## What this is
|
||||
|
||||
GBrain is a compiled intelligence system. Not a note-taking app. Not "chat with your notes."
|
||||
|
||||
Every page is an intelligence assessment. Above the line: compiled truth (your current best understanding, rewritten when evidence changes). Below the line: timeline (append-only evidence trail). AI agents maintain the brain. MCP clients query it. The intelligence lives in fat markdown skills, not application code.
|
||||
|
||||
The core insight: personal knowledge at scale is an intelligence problem, not a storage problem.
|
||||
|
||||
## Why it exists
|
||||
|
||||
A 7,471-file / 2.3GB markdown wiki is choking git. Git doesn't scale past ~5K files for wiki-style use. The compiled truth + timeline model (Karpathy-style knowledge pages) is right, but it needs a real database underneath.
|
||||
|
||||
There's already a production-grade RAG system (Ruby on Rails, Postgres + pgvector) with 3-tier chunking, hybrid search with RRF, multi-query expansion, and 4-layer dedup. GBrain ports these proven patterns to a standalone Bun + TypeScript tool.
|
||||
|
||||
## The knowledge model
|
||||
|
||||
```
|
||||
+--------------------------------------------------+
|
||||
| Page: concepts/do-things-that-dont-scale |
|
||||
| |
|
||||
| --- frontmatter (YAML) --- |
|
||||
| type: concept |
|
||||
| tags: [startups, growth, pg-essay] |
|
||||
| |
|
||||
| === COMPILED TRUTH === |
|
||||
| Current best understanding. |
|
||||
| Rewritten on new evidence. |
|
||||
| This is the "what we know now" section. |
|
||||
| |
|
||||
| --- |
|
||||
| |
|
||||
| === TIMELINE === |
|
||||
| Append-only evidence trail. |
|
||||
| - 2013-07-01: Published on paulgraham.com |
|
||||
| - 2024-11-15: Referenced in batch kickoff talk |
|
||||
| Never edited, only appended. |
|
||||
+--------------------------------------------------+
|
||||
| |
|
||||
v v
|
||||
[Semantic chunks] [Recursive chunks]
|
||||
(best quality for (predictable format
|
||||
compiled truth) for timeline)
|
||||
| |
|
||||
v v
|
||||
[Embeddings: text-embedding-3-large, 1536 dims]
|
||||
|
|
||||
v
|
||||
[HNSW index + tsvector + pg_trgm]
|
||||
|
|
||||
v
|
||||
[Hybrid search: vector + keyword + RRF fusion]
|
||||
```
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
### v0 stack
|
||||
|
||||
| Layer | Choice | Why |
|
||||
|-------|--------|-----|
|
||||
| Database | Postgres + pgvector | Proven RAG patterns, production-tested. World-class hybrid search. |
|
||||
| Hosting | Supabase Pro ($25/mo) | Zero-ops. Managed Postgres, pgvector, connection pooling. 8GB storage. |
|
||||
| Runtime | Bun + TypeScript | Consistent with GStack ecosystem. Fast. Compiles to single binary. |
|
||||
| Embeddings | OpenAI text-embedding-3-large | 1536 dims (reduced from 3072 via dimensions API). ~$0.13/1M tokens. |
|
||||
| LLM (chunking/expansion) | Claude Haiku | Cheapest model for topic boundary detection and query expansion. |
|
||||
| Background jobs | Trigger.dev | Serverless. Embed backfill, stale detection, orphan audit, tag consistency. |
|
||||
| Distribution | npm package + compiled binary + MCP server | Library for OpenClaw, CLI for humans, MCP for agents. |
|
||||
|
||||
### What we chose and why
|
||||
|
||||
**Postgres over SQLite.** We have 3+ years of proven RAG patterns running on Postgres. tsvector for full-text search, pgvector HNSW for semantic search, pg_trgm for fuzzy slug matching. Porting these to SQLite would mean reimplementing search from scratch. SQLite is a future pluggable engine for lightweight open source users (see `docs/ENGINES.md`).
|
||||
|
||||
**Supabase over self-hosted.** Zero maintenance. The brain should be infrastructure that AI agents use, not something you administer. Free tier has pgvector but only 500MB (not enough for 7K+ pages with embeddings, which need ~750MB). Pro tier at $25/mo gives 8GB. No Docker, no self-hosted Postgres in v1.
|
||||
|
||||
**Full port over minimal viable.** The patterns are proven. The port is mechanical. Shipping the full 3-tier chunking + hybrid search + 4-layer dedup means world-class RAG from day one. "We'll add that later" means rebuilding everything later.
|
||||
|
||||
**Library-first distribution.** gbrain is an npm package. OpenClaw installs it as a dependency (`bun add gbrain`), imports the engine directly. Zero-overhead function calls, shared connection pool, TypeScript types. The CLI and MCP server are thin wrappers over the same engine.
|
||||
|
||||
**Trigger-based tsvector (not generated column).** To include timeline_entries content in full-text search, the tsvector needs to span multiple tables. Generated columns can't do cross-table references. A trigger on pages + timeline_entries updates the search_vector.
|
||||
|
||||
**Auto-embed during import.** No separate embed step. `gbrain import` chunks and embeds in one pass. Progress bar shows status. `--no-embed` flag for users who want to defer. `embedded_at` column enables `gbrain embed --stale` for backfill.
|
||||
|
||||
## Distribution model
|
||||
|
||||
```
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| npm package | | Compiled binary | | MCP server |
|
||||
| (library) | | (CLI) | | (stdio) |
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| | | | | |
|
||||
| bun add gbrain | | GitHub Releases | | gbrain serve |
|
||||
| import { Postgres | | npx gbrain | | in mcp.json |
|
||||
| Engine } | | | | |
|
||||
| | | | | |
|
||||
| WHO: OpenClaw, | | WHO: Humans | | WHO: Claude Code, |
|
||||
| AlphaClaw | | | | Cursor, etc. |
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| | |
|
||||
+-------------------------+-------------------------+
|
||||
|
|
||||
+--------v--------+
|
||||
| BrainEngine |
|
||||
| (pluggable |
|
||||
| interface) |
|
||||
+-----------------+
|
||||
|
|
||||
+-------------+-------------+
|
||||
| |
|
||||
+------v------+ +-------v-------+
|
||||
| Postgres | | SQLite |
|
||||
| Engine | | Engine |
|
||||
| (v0, ships) | | (future, see |
|
||||
+-------------+ | ENGINES.md) |
|
||||
+---------------+
|
||||
```
|
||||
|
||||
package.json exports:
|
||||
- Library: `src/core/index.ts` (BrainEngine interface, PostgresEngine, types)
|
||||
- CLI binary: `src/cli.ts`
|
||||
|
||||
## First-time experience
|
||||
|
||||
### Path 1: OpenClaw user (primary)
|
||||
|
||||
OpenClaw is the AI orchestrator that uses gbrain as its knowledge backend. This is the most common install path.
|
||||
|
||||
```bash
|
||||
# 1. Install gbrain as a ClawHub skill
|
||||
clawhub install gbrain
|
||||
|
||||
# 2. The skill runs guided setup on first use:
|
||||
# - Detects if Supabase CLI is available
|
||||
# - If yes: auto-provisions a new Supabase project
|
||||
# - If no: prompts for connection URL
|
||||
# - Runs schema migration
|
||||
# - Imports bundled kindling corpus (10 PG essays)
|
||||
# - Shows live entity/edge extraction animation
|
||||
# - Brain is ready
|
||||
|
||||
# 3. From OpenClaw, brain tools are now available:
|
||||
# "What essays do we have about startups?"
|
||||
# "Ingest my meeting notes from today"
|
||||
# "What does PG say about doing things that don't scale?"
|
||||
```
|
||||
|
||||
Behind the scenes, `clawhub install gbrain`:
|
||||
1. Installs the `gbrain` npm package
|
||||
2. Ships SKILL.md files (ingest, query, maintain, enrich, briefing, migrate)
|
||||
3. Registers brain tools with the orchestrator
|
||||
4. Runs `gbrain init --supabase` on first use (guided wizard)
|
||||
|
||||
### Path 2: CLI user (standalone)
|
||||
|
||||
```bash
|
||||
# 1. Install
|
||||
npm install -g gbrain
|
||||
# or: download binary from GitHub Releases
|
||||
|
||||
# 2. Initialize with Supabase
|
||||
gbrain init --supabase
|
||||
# Guided wizard:
|
||||
# Try 1: Supabase CLI auto-provision (npx supabase)
|
||||
# Try 2: If CLI not installed or not logged in, fallback:
|
||||
# "Enter your Supabase connection URL:"
|
||||
# Then: runs schema migration, verifies pgvector extension
|
||||
# Then: imports kindling corpus (10 PG essays as demo data)
|
||||
# Then: shows live entity extraction animation
|
||||
# Output: "Brain ready. 10 pages imported. Try: gbrain query 'what does PG say about startups?'"
|
||||
|
||||
# 3. Import your data
|
||||
gbrain import /path/to/markdown/wiki/
|
||||
# Progress bar: 7,471 files, auto-chunk, auto-embed
|
||||
# ~30s for text import, ~10-15 min for embedding
|
||||
|
||||
# 4. Query
|
||||
gbrain query "what does PG say about doing things that don't scale?"
|
||||
```
|
||||
|
||||
### Path 3: MCP user (Claude Code, Cursor)
|
||||
|
||||
```json
|
||||
// ~/.config/claude/mcp.json
|
||||
{
|
||||
"mcpServers": {
|
||||
"gbrain": {
|
||||
"command": "gbrain",
|
||||
"args": ["serve"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then in Claude Code: "Search my brain for people who know about robotics"
|
||||
|
||||
### The init wizard in detail
|
||||
|
||||
`gbrain init --supabase` runs through these steps:
|
||||
|
||||
```
|
||||
Step 1: Database Setup
|
||||
├── Check for Supabase CLI (npx supabase --version)
|
||||
│ ├── Found + logged in → auto-create project
|
||||
│ │ ├── Create project via supabase CLI
|
||||
│ │ ├── Wait for project to be ready
|
||||
│ │ └── Extract connection string
|
||||
│ ├── Found + not logged in →
|
||||
│ │ └── Error: "Supabase CLI found but not logged in."
|
||||
│ │ Cause: "You need to authenticate first."
|
||||
│ │ Fix: "Run: npx supabase login"
|
||||
│ │ Docs: "https://supabase.com/docs/guides/cli"
|
||||
│ └── Not found → fallback to manual
|
||||
│ └── Prompt: "Enter your Supabase connection URL:"
|
||||
│
|
||||
Step 2: Schema Migration
|
||||
├── Connect to database
|
||||
├── CREATE EXTENSION IF NOT EXISTS vector
|
||||
├── CREATE EXTENSION IF NOT EXISTS pg_trgm
|
||||
├── Run src/schema.sql (all tables, indexes, triggers)
|
||||
└── Verify: test insert + vector query
|
||||
|
||||
Step 3: Config
|
||||
├── Write ~/.gbrain/config.json (0600 permissions)
|
||||
│ { "database_url": "...", "service_role_key": "..." }
|
||||
└── Verify connection
|
||||
|
||||
Step 4: Kindling Import
|
||||
├── Import 10 bundled PG essays as demo data
|
||||
├── Chunk + embed each essay
|
||||
├── Show live entity/edge extraction animation:
|
||||
│ "Extracting entities... Paul Graham (person), Y Combinator (company)..."
|
||||
│ "Creating links... Paul Graham → Y Combinator (founded)..."
|
||||
└── Output: "Brain ready. 10 pages imported."
|
||||
|
||||
Step 5: First Query
|
||||
└── "Try: gbrain query 'what does PG say about doing things that don't scale?'"
|
||||
```
|
||||
|
||||
Every error follows the style guide: problem + cause + fix + docs link.
|
||||
|
||||
## CLI commands
|
||||
|
||||
```
|
||||
gbrain init [--supabase|--url <conn>] # create brain
|
||||
gbrain get <slug> # read a page
|
||||
gbrain put <slug> [< file.md] # write/update a page
|
||||
gbrain search <query> # keyword search (tsvector)
|
||||
gbrain query <question> # hybrid search (RRF + expansion)
|
||||
gbrain ingest <file> [--type ...] # ingest a source document
|
||||
gbrain link <from> <to> [--type <type>] # create typed link
|
||||
gbrain unlink <from> <to> # remove link
|
||||
gbrain graph <slug> [--depth 5] # traverse link graph (recursive CTE)
|
||||
gbrain backlinks <slug> # incoming links
|
||||
gbrain tags <slug> # list tags
|
||||
gbrain tag <slug> <tag> # add tag
|
||||
gbrain untag <slug> <tag> # remove tag
|
||||
gbrain timeline [<slug>] # view timeline
|
||||
gbrain timeline-add <slug> <date> <text> # add timeline entry
|
||||
gbrain list [--type] [--tag] [--limit] # list with filters
|
||||
gbrain stats # brain statistics
|
||||
gbrain health # brain health dashboard
|
||||
gbrain import <dir> [--no-embed] # import from markdown directory
|
||||
gbrain export [--dir ./export/] # export to markdown (round-trip)
|
||||
gbrain embed [<slug>|--all|--stale] # generate/refresh embeddings
|
||||
gbrain serve # MCP server (stdio)
|
||||
gbrain call <tool> '<json>' # raw tool invocation
|
||||
gbrain upgrade # self-update (npm, binary, ClawHub)
|
||||
gbrain version # version info
|
||||
gbrain config [get|set] <key> [value] # brain config
|
||||
```
|
||||
|
||||
CLI and MCP expose identical operations. Drift tests assert identical results for all operations across both interfaces.
|
||||
|
||||
## Database schema
|
||||
|
||||
9 tables in Postgres + pgvector:
|
||||
|
||||
```
|
||||
+------------------+ +-------------------+ +------------------+
|
||||
| pages |---->| content_chunks | | links |
|
||||
|------------------| |-------------------| |------------------|
|
||||
| id (PK) | | id (PK) | | id (PK) |
|
||||
| slug (UNIQUE) | | page_id (FK) | | from_page_id(FK) |
|
||||
| type | | chunk_index | | to_page_id (FK) |
|
||||
| title | | chunk_text | | link_type |
|
||||
| compiled_truth | | chunk_source | | context |
|
||||
| timeline | | embedding (1536) | +------------------+
|
||||
| frontmatter(JSONB)| | model |
|
||||
| search_vector | | token_count | +------------------+
|
||||
| created_at | | embedded_at | | tags |
|
||||
| updated_at | +-------------------+ |------------------|
|
||||
+------------------+ | id (PK) |
|
||||
| | page_id (FK) |
|
||||
+-----> +--------------------+ | tag |
|
||||
| | timeline_entries | +------------------+
|
||||
| |--------------------|
|
||||
| | id (PK) | +------------------+
|
||||
| | page_id (FK) | | page_versions |
|
||||
| | date | |------------------|
|
||||
| | source | | id (PK) |
|
||||
| | summary | | page_id (FK) |
|
||||
| | detail (markdown) | | compiled_truth |
|
||||
| +--------------------+ | frontmatter |
|
||||
| | snapshot_at |
|
||||
+-----> +--------------------+ +------------------+
|
||||
| | raw_data |
|
||||
| |--------------------| +------------------+
|
||||
| | id (PK) | | config |
|
||||
| | page_id (FK) | |------------------|
|
||||
| | source | | key (PK) |
|
||||
| | data (JSONB) | | value |
|
||||
| +--------------------+ +------------------+
|
||||
|
|
||||
+-----> +--------------------+
|
||||
| ingest_log |
|
||||
|--------------------|
|
||||
| id (PK) |
|
||||
| source_type |
|
||||
| source_ref |
|
||||
| pages_updated |
|
||||
| summary |
|
||||
+--------------------+
|
||||
```
|
||||
|
||||
Indexes:
|
||||
- `pages.slug`: UNIQUE constraint (implicit B-tree)
|
||||
- `pages.type`: B-tree
|
||||
- `pages.search_vector`: GIN (full-text search)
|
||||
- `pages.frontmatter`: GIN (JSONB queries)
|
||||
- `pages.title`: GIN with pg_trgm (fuzzy slug resolution)
|
||||
- `content_chunks.embedding`: HNSW with cosine ops (vector search)
|
||||
- `content_chunks.page_id`: B-tree
|
||||
- `links.from_page_id`, `links.to_page_id`: B-tree
|
||||
- `tags.tag`, `tags.page_id`: B-tree
|
||||
- `timeline_entries.page_id`, `timeline_entries.date`: B-tree
|
||||
|
||||
## Search architecture
|
||||
|
||||
```
|
||||
Query: "when should you ignore conventional wisdom?"
|
||||
|
|
||||
v
|
||||
+---------------------+
|
||||
| Multi-query expansion|
|
||||
| (Claude Haiku) |
|
||||
| "contrarian thinking"
|
||||
| "going against the crowd"
|
||||
+---------------------+
|
||||
| | |
|
||||
v v v
|
||||
[embed all 3 queries]
|
||||
| | |
|
||||
+---+---+
|
||||
|
|
||||
+----+----+
|
||||
| |
|
||||
v v
|
||||
+--------+ +--------+
|
||||
| Vector | | Keyword|
|
||||
| Search | | Search |
|
||||
| (HNSW | | (tsv + |
|
||||
| cosine)| | ts_rank)|
|
||||
+--------+ +--------+
|
||||
| |
|
||||
+----+----+
|
||||
|
|
||||
v
|
||||
+------------------+
|
||||
| RRF Fusion |
|
||||
| score = sum( |
|
||||
| 1/(60 + rank)) |
|
||||
+------------------+
|
||||
|
|
||||
v
|
||||
+------------------+
|
||||
| 4-Layer Dedup |
|
||||
| 1. By source |
|
||||
| 2. Cosine > 0.85 |
|
||||
| 3. Type cap 60% |
|
||||
| 4. Per-page max |
|
||||
+------------------+
|
||||
|
|
||||
v
|
||||
+------------------+
|
||||
| Stale alerts |
|
||||
| (compiled_truth |
|
||||
| older than |
|
||||
| latest timeline)|
|
||||
+------------------+
|
||||
|
|
||||
v
|
||||
[Results]
|
||||
```
|
||||
|
||||
## Chunking strategies
|
||||
|
||||
| Strategy | Input | Algorithm | When to use |
|
||||
|----------|-------|-----------|-------------|
|
||||
| Recursive | Any text | 5-level delimiter hierarchy (paragraphs > lines > sentences > clauses > whitespace). 300-word chunks, 50-word overlap. | Timeline (predictable format), bulk import |
|
||||
| Semantic | Quality text | Embed each sentence, Savitzky-Golay filter for topic boundaries, cosine similarity minima. Falls back to recursive. | Compiled truth (intelligence assessments) |
|
||||
| LLM-guided | High-value text | Pre-split to 128-word candidates, Claude Haiku finds topic shifts in sliding windows. 3 retries per window. | Explicitly requested via `--chunker llm` |
|
||||
|
||||
Dispatch: compiled_truth gets semantic chunker. Timeline gets recursive chunker. Override with `--chunker` flag or `chunk_strategy` in frontmatter.
|
||||
|
||||
## Skills (fat markdown, no code)
|
||||
|
||||
Each skill is a markdown file that AI agents (Claude Code, OpenClaw) read and follow. The skill contains the workflow, heuristics, and quality rules. No skill logic is in the binary.
|
||||
|
||||
| Skill | What it does |
|
||||
|-------|-------------|
|
||||
| `skills/ingest/SKILL.md` | Ingest meetings, docs, articles. Update compiled truth, append timeline, create links. |
|
||||
| `skills/query/SKILL.md` | 3-layer search (FTS + vector + structured). Synthesize answer with citations. |
|
||||
| `skills/maintain/SKILL.md` | Find contradictions, stale info, orphans, dead links, tag inconsistency. |
|
||||
| `skills/enrich/SKILL.md` | Enrich from external APIs (Crustdata, Happenstance, Exa). Store raw data, distill to compiled truth. |
|
||||
| `skills/briefing/SKILL.md` | Daily briefing: meetings with context, active deals, open threads. |
|
||||
| `skills/migrate/SKILL.md` | Universal migration from Obsidian, Notion, Logseq, plain markdown, CSV, JSON, Roam. |
|
||||
|
||||
## CEO scope expansions (accepted for v0)
|
||||
|
||||
1. **CLI/MCP parity with drift tests.** Both interfaces are thin wrappers over the engine. Tests assert identical output.
|
||||
2. **Smart slug resolution.** Fuzzy matching via pg_trgm for reads. Writes require exact slugs. `gbrain get "dont scale"` resolves to `concepts/do-things-that-dont-scale`.
|
||||
3. **Brain health dashboard.** `gbrain health` shows page count, embed coverage, stale pages, orphans, dead links.
|
||||
4. **Normalized timeline.** `timeline_entries` table only (no TEXT column). `detail` field supports markdown.
|
||||
5. **Page version control.** `page_versions` table stores full snapshots (compiled_truth + frontmatter + links + tags). `gbrain history`, `gbrain diff`, `gbrain revert` commands. Revert re-chunks and re-embeds.
|
||||
6. **Typed links + graph traversal.** `link_type` column (knows, invested_in, works_at, etc.). `gbrain graph` uses recursive CTE with max depth (default 5, configurable via `--depth`).
|
||||
7. **Trigger.dev data cleanup jobs.** Daily embed backfill, weekly stale detection + orphan audit + tag consistency.
|
||||
8. **Stale alert annotations.** Search results flag pages where compiled_truth is older than latest timeline entry.
|
||||
9. **Timeline merge on ingest.** Same event created across all mentioned entities.
|
||||
|
||||
## Security model (v0)
|
||||
|
||||
Single-user, local-only:
|
||||
- Supabase service role key in `~/.gbrain/config.json` (0600 permissions)
|
||||
- MCP stdio transport is inherently local (client spawns `gbrain serve` as subprocess)
|
||||
- No multi-user, no RLS, no OAuth in v0
|
||||
- Multi-user path (future): Supabase RLS + per-user API keys
|
||||
|
||||
## Upgrade mechanism
|
||||
|
||||
`gbrain upgrade` detects the installation method and updates accordingly:
|
||||
|
||||
| Path | How |
|
||||
|------|-----|
|
||||
| npm | `bun update gbrain` (or npm equivalent) |
|
||||
| Compiled binary | Download new binary to temp dir, atomic rename swap, exec new process |
|
||||
| ClawHub | `clawhub update gbrain` |
|
||||
|
||||
Version check: compare local version against latest GitHub release tag.
|
||||
|
||||
## Storage and cost estimates
|
||||
|
||||
### Storage (~750MB for 7,471 pages)
|
||||
|
||||
| Component | Size |
|
||||
|-----------|------|
|
||||
| Page text (compiled_truth + timeline) | ~150MB |
|
||||
| JSONB frontmatter | ~20MB |
|
||||
| tsvector + GIN indexes | ~50MB |
|
||||
| Content chunks (~22K, text) | ~80MB |
|
||||
| Embeddings (22K x 1536 floats x 4 bytes) | ~134MB |
|
||||
| HNSW index overhead (~2x embeddings) | ~270MB |
|
||||
| Links, tags, timeline, raw_data, versions | ~50MB |
|
||||
| **Total** | **~750MB** |
|
||||
|
||||
Supabase free tier (500MB) won't fit. Supabase Pro ($25/mo, 8GB) is the starting point.
|
||||
|
||||
### Embedding cost (~$4-5 for initial import)
|
||||
|
||||
| Step | Cost |
|
||||
|------|------|
|
||||
| Semantic chunker sentence embeddings (~374K sentences) | ~$1 |
|
||||
| Chunk embeddings (~22K chunks) | ~$0.30 |
|
||||
| Query expansion (per query, ~3 embeds) | negligible |
|
||||
| **Total initial import** | **~$4-5** |
|
||||
|
||||
Budget alternative: `gbrain import --chunker recursive` skips sentence-level embeddings, then `gbrain embed --rechunk --chunker semantic` upgrades later.
|
||||
|
||||
## Serverless operations stack
|
||||
|
||||
```
|
||||
+------------------+ +------------------+ +------------------+
|
||||
| Supabase | | Vercel | | Trigger.dev |
|
||||
| (Postgres + | | (web/API, | | (background |
|
||||
| pgvector) | | optional) | | jobs) |
|
||||
+------------------+ +------------------+ +------------------+
|
||||
| Database | | Future web UI | | Embed backfill |
|
||||
| Connection pool | | API endpoints | | Stale detection |
|
||||
| pgvector HNSW | | Edge functions | | Orphan audit |
|
||||
| tsvector FTS | | | | Tag consistency |
|
||||
| pg_trgm fuzzy | | | | Daily briefing |
|
||||
+------------------+ +------------------+ +------------------+
|
||||
```
|
||||
|
||||
The CLI connects directly to Supabase Postgres. Trigger.dev and Vercel are for async/scheduled work. The CLI works without them.
|
||||
|
||||
## Verification checklist
|
||||
|
||||
1. `gbrain import /data/brain/` migrates all 7,471 files losslessly
|
||||
2. `gbrain export` round-trips to semantically identical markdown
|
||||
3. `gbrain query "what does PG say about doing things that don't scale?"` returns relevant hybrid search results
|
||||
4. `gbrain serve` starts MCP server connectable by Claude Code
|
||||
5. All 3 chunkers produce correct output with test fixtures
|
||||
6. `gbrain init --supabase` works end-to-end
|
||||
7. `bun test` passes all tests
|
||||
8. `clawhub install gbrain` installs the skill and runs guided setup
|
||||
9. `bun add gbrain` + `import { PostgresEngine } from 'gbrain'` works in external project
|
||||
10. Drift tests pass: CLI and MCP produce identical results
|
||||
11. `gbrain health` outputs accurate brain health metrics
|
||||
12. Migration skill successfully imports an Obsidian vault
|
||||
|
||||
## Future plans
|
||||
|
||||
See `docs/ENGINES.md` for the pluggable engine architecture and future backend plans.
|
||||
|
||||
### v1 candidates (deferred from v0)
|
||||
|
||||
- **`gbrain ask` natural language CLI alias.** Trivial to add. P1 TODO.
|
||||
- **Intelligence compiler.** Treat every fact as a first-class claim with source span, entity links, validity window, confidence, and contradiction status. "What changed, why, and what evidence would flip it again?" From Codex review. Builds on compiled truth model.
|
||||
- **Active skills via Trigger.dev.** Application-specific briefings, meeting prep. Belongs in OpenClaw, not generic brain infra.
|
||||
- **Multi-user access.** Supabase RLS + per-user API keys. v0 is single-user.
|
||||
- **SQLite engine.** Community PRs welcome. See `docs/SQLITE_ENGINE.md`.
|
||||
- **Docker Compose for self-hosted Postgres.** Community PRs welcome.
|
||||
- **Web UI.** Optional Vercel-hosted dashboard for browsing brain pages.
|
||||
|
||||
### Interface abstraction principle
|
||||
|
||||
All operations go through `BrainEngine`. The engine interface is the contract. Postgres-specific features (tsvector, pgvector HNSW, pg_trgm, recursive CTEs) are implementation details inside `PostgresEngine`. The interface exposes capabilities, not SQL.
|
||||
|
||||
This means:
|
||||
- A SQLite engine can implement `searchKeyword` using FTS5 instead of tsvector
|
||||
- A SQLite engine can implement `searchVector` using sqlite-vss instead of pgvector
|
||||
- A future DuckDB engine could implement analytics-heavy workloads
|
||||
- The CLI, MCP server, and library consumers never know which engine runs underneath
|
||||
|
||||
See `docs/ENGINES.md` for the full interface spec and `docs/SQLITE_ENGINE.md` for the SQLite implementation plan.
|
||||
|
||||
## Review history
|
||||
|
||||
| Review | Runs | Status | Key findings |
|
||||
|--------|------|--------|-------------|
|
||||
| /office-hours | 1 | APPROVED | Builder mode. Full port approach chosen. |
|
||||
| /plan-ceo-review | 1 | CLEAR | 11 proposals, 10 accepted, 1 deferred. SCOPE EXPANSION mode. |
|
||||
| /codex review | 1 | issues_found | 24 points challenged, 3 accepted (fuzzy slug, revert spec, tsvector). |
|
||||
| /plan-eng-review | 2 | CLEAR | 3 issues (upgrade paths, import guardrails, init wizard), 0 critical gaps. |
|
||||
| /plan-devex-review | 1 | CLEAR | DX score 5/10 to 7/10. TTHW 25min to 90s. Champion tier. |
|
||||
Reference in New Issue
Block a user