# GBrain Your AI agent is smart but it doesn't know anything about your life. GBrain fixes that. Meetings, emails, tweets, calendar events, voice calls, original ideas... all of it flows into a searchable knowledge base that your agent reads before every response and writes to after every conversation. The agent gets smarter every day. > **Requires a frontier model.** Tested with **Claude Opus 4.6** and **GPT-5.4 Thinking**. Likely to break with smaller models. ## Start here: paste this into your agent Copy this block into [OpenClaw](https://openclaw.ai) or [Hermes Agent](https://github.com/NousResearch/hermes-agent). The agent will install GBrain, set up your database, import your files, and verify everything works. ``` GOAL: Install GBrain so I can search my own markdown files with hybrid search (keyword + vector). Do not skip steps. Verify each step before proceeding. If a step fails, stop and fix it. STEP 1 — INSTALL curl -fsSL https://bun.sh/install | bash && source ~/.bashrc bun add github:garrytan/gbrain Verify: gbrain --version STEP 2 — DATABASE Run: gbrain init --supabase Ask the user for their Supabase Shared Pooler connection string: "Go to https://supabase.com → your project → Get Connected → Direct Connection String → Session Pooler → copy Shared Pooler" Do NOT ask for the anon key or project URL. Those are wrong. Verify: gbrain doctor --json (all checks pass) STEP 3 — IMPORT Find markdown: find ~/git ~/Documents ~/notes -maxdepth 3 -name "*.md" | head -30 Tell user what you found. Import: gbrain import /path/ --no-embed Verify: gbrain stats (page count > 0) STEP 4 — SEARCH Run: gbrain embed --stale (generates vector embeddings, ~1 min per 1000 pages) Then: gbrain query "what are the key themes across these documents?" Show the user the results. This is the magic moment. Verify: results include semantic matches, not just keyword hits STEP 5 — GO LIVE a. Read docs/GBRAIN_SKILLPACK.md — adopt brain-first lookup, entity detection, and back-linking into your behavior immediately b. Set up live sync: gbrain sync --repo /path/ && gbrain embed --stale (see docs/guides/live-sync.md for cron/watch/webhook options) c. Set up auto-update: gbrain check-update --json on a daily cron (never auto-install — always ask the user first) d. Show available integrations: gbrain integrations e. Run verification: read and execute docs/GBRAIN_VERIFY.md DONE. Tell the user: "GBrain is live. You have [N] pages with hybrid search. I now check the brain before answering questions. Run 'gbrain integrations' to add voice, email, calendar, or Twitter." ``` ### Without an agent (standalone CLI) ```bash bun add -g github:garrytan/gbrain gbrain init --supabase # guided wizard gbrain import ~/git/brain/ # index your markdown gbrain query "what do we know about competitive dynamics?" ``` Run `gbrain --help` for all commands. See [MCP setup](docs/mcp/DEPLOY.md) for connecting Claude Desktop, Perplexity, etc. ## Getting Data In Once GBrain is installed, your agent needs data flowing in. GBrain ships integration recipes that your agent sets up for you. It reads the recipe, asks for API keys, validates each one, and runs a smoke test. [Markdown is code](docs/ethos/THIN_HARNESS_FAT_SKILLS.md)... the recipe IS the installer. | Recipe | Requires | What It Does | |--------|----------|-------------| | [Public Tunnel](recipes/ngrok-tunnel.md) | — | Fixed URL for MCP + voice (ngrok Hobby $8/mo) | | [Credential Gateway](recipes/credential-gateway.md) | — | Gmail + Calendar access (ClawVisor or Google OAuth) | | [Voice-to-Brain](recipes/twilio-voice-brain.md) | ngrok-tunnel | Phone calls → brain pages (Twilio + OpenAI Realtime) | | [Email-to-Brain](recipes/email-to-brain.md) | credential-gateway | Gmail → entity pages (deterministic collector) | | [X-to-Brain](recipes/x-to-brain.md) | — | Twitter → brain pages (timeline + mentions + deletions) | | [Calendar-to-Brain](recipes/calendar-to-brain.md) | credential-gateway | Google Calendar → searchable daily pages | | [Meeting Sync](recipes/meeting-sync.md) | — | Circleback transcripts → brain pages with attendees | Run `gbrain integrations` to see status. Dependencies resolve automatically. See [Getting Data In](docs/integrations/README.md) for the full guide. ## The Compounding Thesis Most tools help you find things. GBrain makes you smarter over time. ``` Signal arrives (meeting, email, tweet, link) → Agent detects entities (people, companies, ideas) → READ: check the brain first (gbrain search, gbrain get) → Respond with full context → WRITE: update brain pages with new information → Sync: gbrain indexes changes for next query ``` Every cycle through this loop adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context. You never start from zero. An agent without this loop answers from stale context. An agent with it gets smarter every conversation. The difference compounds daily. > "Who should I invite to dinner who knows both Pedro and Diana?" > — cross-references the social graph across 3,000+ people pages > "What have I said about the relationship between shame and founder performance?" > — searches YOUR thinking, not the internet > "Prep me for my meeting with Jordan in 30 minutes" > — pulls dossier, shared history, recent activity, open threads ## How this happened I was setting up my [OpenClaw](https://openclaw.ai) agent and started a markdown brain repo. One page per person, one page per company, compiled truth on top, append-only timeline on the bottom. The agent got smarter the more it knew, so I kept feeding it. Within a week I had 10,000+ markdown files, 3,000+ people with compiled dossiers, 13 years of calendar data, 280+ meeting transcripts, and 300+ captured original ideas. The agent runs while I sleep. The dream cycle scans every conversation, enriches missing entities, fixes broken citations, and consolidates memory. I wake up and the brain is smarter than when I went to sleep. See the [cron schedule guide](docs/guides/cron-schedule.md) for setup. **You don't need Postgres to start.** The knowledge model is just markdown files in a git repo. The [skills](docs/GBRAIN_SKILLPACK.md) and [schema](docs/GBRAIN_RECOMMENDED_SCHEMA.md) work with any AI agent that can read and write files. **When you need Postgres:** at 1,000+ files, `grep` stops working. GBrain adds hybrid search (keyword + vector + RRF fusion) on top of Postgres + pgvector. The CLI and MCP layer handle chunking, embedding, and incremental sync. Add Postgres when search speed matters, or when you want Claude Desktop, ChatGPT, Perplexity, or other MCP clients to connect to your brain remotely. ## Architecture ``` ┌──────────────────┐ ┌───────────────┐ ┌──────────────────┐ │ Brain Repo │ │ GBrain │ │ AI Agent │ │ (git) │ │ (retrieval) │ │ (read/write) │ │ │ │ │ │ │ │ markdown files │───>│ Postgres + │<──>│ skills define │ │ = source of │ │ pgvector │ │ HOW to use the │ │ truth │ │ │ │ brain │ │ │<───│ hybrid │ │ │ │ human can │ │ search │ │ entity detect │ │ always read │ │ (vector + │ │ enrich │ │ & edit │ │ keyword + │ │ ingest │ │ │ │ RRF) │ │ brief │ └──────────────────┘ └───────────────┘ └──────────────────┘ ``` The repo is the system of record. GBrain is the retrieval layer. The agent reads and writes through both. Human always wins — you can edit any markdown file directly and `gbrain sync` picks up the changes. ## What a Production Agent Looks Like The numbers above aren't theoretical. They come from a real deployment documented in [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) — a reference architecture for how a production AI agent uses gbrain as its knowledge backbone. **Read the skillpack.** It's the most important doc in this repo. It tells your agent HOW to use gbrain, not just what commands exist: - **The brain-agent loop** — the read-write cycle that makes knowledge compound - **Entity detection** — spawn on every message, capture people/companies/original ideas - **Enrichment pipeline** — 7-step protocol with tiered API spend - **Meeting ingestion** — transcript to brain pages with entity propagation - **Source attribution** — every fact traceable to where it came from - **Reference cron schedule** — 20+ recurring jobs that keep the brain alive Without the skillpack, your agent has tools but no playbook. With it, the agent knows when to read, when to write, how to enrich, and how to keep the brain alive autonomously. It's a pattern book, not a tutorial. "Here's what works, here's why." ## How gbrain fits with OpenClaw/Hermes GBrain is world knowledge — people, companies, deals, meetings, concepts, your original thinking. It's the long-term memory of what you know about the world. [OpenClaw](https://openclaw.ai) agent memory (`memory_search`) is operational state — preferences, decisions, session context, how the agent should behave. They're complementary: | Layer | What it stores | How to query | |-------|---------------|-------------| | **gbrain** | People, companies, meetings, ideas, media | `gbrain search`, `gbrain query`, `gbrain get` | | **Agent memory** | Preferences, decisions, operational config | `memory_search` | | **Session context** | Current conversation | (automatic) | All three should be checked. GBrain for facts about the world. Memory for agent config. Session for immediate context. Install via `openclaw skills install gbrain`. ## Try it: your files, searchable in 90 seconds GBrain doesn't ship with demo data. It finds YOUR markdown and makes it searchable. **Act 1: Discovery.** GBrain scans your machine for markdown repos. ``` === GBrain Environment Discovery === ~/git/brain (2.3GB, 342 .md files, 87 binary files) Type: Plain markdown (ready for import) ~/Documents/obsidian-vault (180MB, 1,203 .md files, 0 binary files) Type: Obsidian vault (wikilink conversion available) === Discovery Complete === ``` **Act 2: Import.** Your files move from the repo into Supabase. ```bash gbrain import ~/git/brain/ # Imported 342 files into Supabase (1,847 chunks). Embedding in background... gbrain stats # Pages: 342, Chunks: 1,847, Embedded: 0 (embedding...), Links: 0 ``` **Act 3: Search.** The agent picks a query from your actual content. ```bash # The agent reads your corpus and picks a relevant query gbrain query "what do we know about competitive dynamics?" # 3 results, scored by hybrid search (vector + keyword + RRF fusion) # 30 seconds later, embeddings finish: gbrain stats # Pages: 342, Chunks: 1,847, Embedded: 1,847, Links: 0 # Now semantic search is live too gbrain query "what are our biggest risks right now?" # Finds pages about moats, board prep, and strategy -- by meaning, not keywords ``` Your file count will be different. Your queries will be different. The agent picks them based on what it imported. That's the point: this is YOUR brain, not a demo. **The compounding effect.** Search for Pedro. The agent pulls his page, his relationship history, his company. Next time Brex comes up in conversation, the agent already knows Pedro co-founded it, what you discussed last, and what's on your open threads. You didn't do anything — the brain already had it. ## Upgrade Upgrade depends on how you installed: ```bash # Installed via bun (standalone or library) bun update gbrain # Installed via ClawHub clawhub update gbrain # Compiled binary # Download the latest from https://github.com/garrytan/gbrain/releases ``` After upgrading, run `gbrain init` again to apply any schema migrations (idempotent, safe to re-run). ## Setup After installing via CLI or library path, run the setup wizard: ```bash # Guided wizard: auto-provisions Supabase or accepts a connection URL gbrain init --supabase # Or connect to any Postgres with pgvector gbrain init --url postgresql://user:pass@host:5432/dbname ``` The init wizard: 1. Checks for Supabase CLI, offers auto-provisioning 2. Falls back to manual connection URL if CLI isn't available 3. Runs the full schema migration (tables, indexes, triggers, extensions) 4. Verifies the connection and confirms the database is ready for import Config is saved to `~/.gbrain/config.json` with 0600 permissions. OpenClaw users skip this step. The orchestrator runs the wizard for you during install. ## First import ```bash # Import your markdown wiki (auto-chunks and auto-embeds) gbrain import /path/to/brain/ # Skip embedding if you want to import fast and embed later gbrain import /path/to/brain/ --no-embed # Backfill embeddings for pages that don't have them gbrain embed --stale ``` Import is idempotent. Re-running it skips unchanged files (compared by SHA-256 content hash). Progress bar shows status. ~30s for text import of 7,000 files, ~10-15 min for embedding. ## File storage and migration Brain repos accumulate binary files: images, PDFs, audio recordings, raw API responses. A repo with 3,000 markdown pages might have 2GB of binaries making `git clone` painful. GBrain has a three-stage migration lifecycle that moves binaries to cloud storage while preserving every reference: ``` Local files in git repo │ ▼ gbrain files mirror Cloud copy exists, local files untouched │ ▼ gbrain files redirect Local files replaced with .redirect breadcrumbs (tiny YAML pointers) │ ▼ gbrain files clean Breadcrumbs removed, cloud is the only copy ``` Every stage is reversible until `clean`: ```bash # Stage 1: Copy to cloud (git repo unchanged) gbrain files mirror ~/git/brain/attachments/ --dry-run # preview first gbrain files mirror ~/git/brain/attachments/ # Stage 2: Replace local files with breadcrumbs gbrain files redirect ~/git/brain/attachments/ --dry-run gbrain files redirect ~/git/brain/attachments/ # Your git repo just dropped from 2GB to 50MB # Undo: download everything back from cloud gbrain files restore ~/git/brain/attachments/ # Stage 3: Remove breadcrumbs (irreversible, cloud is the only copy) gbrain files clean ~/git/brain/attachments/ --yes ``` **Storage backends:** S3-compatible (AWS S3, Cloudflare R2, MinIO), Supabase Storage, or local filesystem. Configured during `gbrain init`. Additional file commands: ```bash gbrain files list [slug] # list files for a page (or all) gbrain files upload --page # upload file linked to page gbrain files sync # bulk upload directory gbrain files verify # verify all uploads match local gbrain files status # show migration status of directories gbrain files unmirror # remove mirror marker (files stay in cloud) ``` The file resolver (`src/core/file-resolver.ts`) handles fallback automatically: if a local file is missing, it checks for a `.redirect` breadcrumb, then a `.supabase` marker, and resolves to the cloud URL. Code that references files by path keeps working after migration. ## The knowledge model Every page in the brain follows the compiled truth + timeline pattern: ```markdown --- type: concept title: Do Things That Don't Scale tags: [startups, growth, pg-essay] --- Paul Graham's argument that startups should do unscalable things early on. The most common: recruiting users manually, one at a time. Airbnb went door to door in New York photographing apartments. Stripe manually installed their payment integration for early users. The key insight: the unscalable effort teaches you what users actually want, which you can't learn any other way. --- - 2013-07-01: Published on paulgraham.com - 2024-11-15: Referenced in batch W25 kickoff talk - 2025-02-20: Cited in discussion about AI agent onboarding strategies ``` Above the `---` separator: **compiled truth**. Your current best understanding. Gets rewritten when new evidence changes the picture. Below: **timeline**. Append-only evidence trail. Never edited, only added to. The compiled truth is the answer. The timeline is the proof. ## How search works ``` Query: "when should you ignore conventional wisdom?" | Multi-query expansion (Claude Haiku) "contrarian thinking startups", "going against the crowd" | +----+----+ | | Vector Keyword (HNSW (tsvector + cosine) ts_rank) | | +----+----+ | RRF Fusion: score = sum(1/(60 + rank)) | 4-Layer Dedup 1. Best chunk per page 2. Cosine similarity > 0.85 3. Type diversity (60% cap) 4. Per-page chunk cap | Stale alerts (compiled truth older than latest timeline) | Results ``` Keyword search alone misses conceptual matches. "Ignore conventional wisdom" won't find an essay titled "The Bus Ticket Theory of Genius" even though it's exactly about that. Vector search alone misses exact phrases when the embedding is diluted by surrounding text. RRF fusion gets both right. Multi-query expansion catches phrasings you didn't think of. ## Database schema 10 tables in Postgres + pgvector: ``` pages The core content table slug (UNIQUE) e.g. "concepts/do-things-that-dont-scale" type person, company, deal, yc, civic, project, concept, source, media title, compiled_truth, timeline frontmatter (JSONB) Arbitrary metadata search_vector Trigger-based tsvector (title + compiled_truth + timeline + timeline_entries) content_hash SHA-256 for import idempotency content_chunks Chunked content with embeddings page_id (FK) Links to pages chunk_text The chunk content chunk_source 'compiled_truth' or 'timeline' embedding (vector) 1536-dim from text-embedding-3-large HNSW index Cosine similarity search links Cross-references between pages from_page_id, to_page_id link_type knows, invested_in, works_at, founded, references, etc. tags page_id + tag (many-to-many) timeline_entries Structured timeline events page_id, date, source, summary, detail (markdown) page_versions Snapshot history for compiled_truth compiled_truth, frontmatter, snapshot_at raw_data Sidecar JSON from external APIs page_id, source, data (JSONB) files Binary attachments in Supabase Storage page_slug (FK) Links to pages (ON UPDATE CASCADE) storage_path, content_hash, mime_type, metadata (JSONB) ingest_log Audit trail of import/ingest operations config Brain-level settings (embedding model, chunk strategy, sync state) ``` Indexes: B-tree on slug/type, GIN on frontmatter/search_vector, HNSW on embeddings, pg_trgm on title for fuzzy slug resolution. ## Chunking Three strategies, dispatched by content type: **Recursive** (timeline, bulk import): 5-level delimiter hierarchy (paragraphs, lines, sentences, clauses, words). 300-word chunks with 50-word sentence-aware overlap. Fast, predictable, lossless. **Semantic** (compiled truth): Embeds each sentence, computes adjacent cosine similarities, applies Savitzky-Golay smoothing to find topic boundaries. Falls back to recursive on failure. Best quality for intelligence assessments. **LLM-guided** (high-value content, on request): Pre-splits into 128-word candidates, asks Claude Haiku to identify topic shifts in sliding windows. 3 retries per window. Most expensive, best results. ## Commands ``` SETUP gbrain init [--supabase|--url ] Create brain (guided wizard) gbrain upgrade Self-update PAGES gbrain get Read a page (supports fuzzy slug matching) gbrain put [< file.md] Write/update a page (auto-versions) gbrain delete Delete a page gbrain list [--type T] [--tag T] [-n N] List pages with filters SEARCH gbrain search Keyword search (tsvector) gbrain query Hybrid search (vector + keyword + RRF + expansion) IMPORT/EXPORT gbrain import [--no-embed] Import markdown directory (idempotent) gbrain sync [--repo ] [flags] Git-to-brain incremental sync gbrain export [--dir ./out/] Export to markdown (round-trip) FILES gbrain files list [slug] List stored files gbrain files upload --page Upload file to storage gbrain files sync Bulk upload directory gbrain files verify Verify all uploads EMBEDDINGS gbrain embed [|--all|--stale] Generate/refresh embeddings LINKS + GRAPH gbrain link [--type T] Create typed link gbrain unlink Remove link gbrain backlinks Incoming links gbrain graph [--depth N] Traverse link graph (recursive CTE, default depth 5) TAGS gbrain tags List tags gbrain tag Add tag gbrain untag Remove tag TIMELINE gbrain timeline [] View timeline entries gbrain timeline-add Add timeline entry ADMIN gbrain doctor [--json] Health checks (pgvector, RLS, schema, embeddings) gbrain stats Brain statistics gbrain health Health dashboard (embed coverage, stale, orphans) gbrain history Page version history gbrain revert Revert to previous version gbrain config [get|set] [value] Brain config gbrain serve MCP server (stdio, local) scripts/deploy-remote.sh Deploy remote MCP server (Supabase Edge Functions) bun run src/commands/auth.ts Token management (create/list/revoke/test) gbrain call '' Raw tool invocation gbrain --tools-json Tool discovery (JSON) ``` ## Library and MCP details See [GBrain without OpenClaw](#gbrain-without-openclaw) above for library usage examples, MCP server config, and skill file loading. The `BrainEngine` interface is pluggable. See `docs/ENGINES.md` for how to add backends. 30 MCP tools are generated from the contract-first `operations.ts`. Parity tests verify structural identity between CLI, MCP, and tools-json. ## Skills Fat markdown files that tell AI agents HOW to use gbrain. No skill logic in the binary. | Skill | What it does | |-------|-------------| | **ingest** | Ingest meetings, docs, articles. Updates compiled truth (rewrite, not append), appends timeline, creates cross-reference links across all mentioned entities. | | **query** | 3-layer search (keyword + vector + structured) with synthesis and citations. Says "the brain doesn't have info on X" rather than hallucinating. | | **maintain** | Periodic health: find contradictions, stale compiled truth, orphan pages, dead links, tag inconsistency, missing embeddings, overdue threads. | | **enrich** | Enrich pages from external APIs. Raw data stored separately, distilled highlights go to compiled truth. | | **briefing** | Daily briefing: today's meetings with participant context, active deals with deadlines, time-sensitive threads, recent changes. | | **migrate** | Universal migration from Obsidian (wikilinks to gbrain links), Notion (stripped UUIDs), Logseq (block refs), plain markdown, CSV, JSON, Roam. | | **setup** | Set up GBrain from scratch: auto-provision Supabase via CLI, AGENTS.md injection, import, sync. Target TTHW < 2 min. | ## Engine Architecture ``` CLI / MCP Server (thin wrappers, identical operations) | BrainEngine interface (pluggable backend) | +--------+--------+ | | PostgresEngine SQLiteEngine (ships v0) (designed, community PRs welcome) | Supabase Pro ($25/mo) Postgres + pgvector + pg_trgm connection pooling via Supavisor ``` Embedding, chunking, and search fusion are engine-agnostic. Only raw keyword search (`searchKeyword`) and raw vector search (`searchVector`) are engine-specific. RRF fusion, multi-query expansion, and 4-layer dedup run above the engine on `SearchResult[]` arrays. ## Storage estimates For a brain with ~7,500 pages: | Component | Size | |-----------|------| | Page text (compiled_truth + timeline) | ~150MB | | JSONB frontmatter + indexes | ~70MB | | Content chunks (~22K, text) | ~80MB | | Embeddings (22K x 1536 floats) | ~134MB | | HNSW index overhead | ~270MB | | Links, tags, timeline, versions | ~50MB | | **Total** | **~750MB** | Supabase free tier (500MB) won't fit a large brain. Supabase Pro ($25/mo, 8GB) is the starting point. Initial embedding cost: ~$4-5 for 7,500 pages via OpenAI text-embedding-3-large. ## Docs **For agents:** - **[GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md)** -- **Start here.** Index of all patterns, skills, and integrations - [Individual guides](docs/guides/) -- 17 standalone guides broken out from the skillpack - [Getting Data In](docs/integrations/README.md) -- Integration recipes, credential setup, data flow patterns - [GBRAIN_VERIFY.md](docs/GBRAIN_VERIFY.md) -- Installation verification runbook **For humans:** - [GBRAIN_RECOMMENDED_SCHEMA.md](docs/GBRAIN_RECOMMENDED_SCHEMA.md) -- Brain repo directory structure - [Infrastructure Layer](docs/architecture/infra-layer.md) -- How import, chunking, embedding, and search work - [Thin Harness, Fat Skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md) -- Architecture philosophy - [Homebrew for Personal AI](docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md) -- Why markdown is code **Reference:** - [GBRAIN_V0.md](docs/GBRAIN_V0.md) -- Full product spec, all architecture decisions - [ENGINES.md](docs/ENGINES.md) -- Pluggable engine interface, how to add backends - [SQLITE_ENGINE.md](docs/SQLITE_ENGINE.md) -- SQLite engine plan (community PRs welcome) ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For E2E tests against real Postgres+pgvector: `docker compose -f docker-compose.test.yml up -d` then `DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run test:e2e`. Welcome PRs for: - SQLite engine implementation - New enrichment API integrations - Performance optimizations - Docker Compose for self-hosted Postgres ## License MIT