GBrain v0.4.0 — production agent documentation + reference architecture (#10)
* fix: widen validateSlug to accept any filename characters Git is the system of record. Slugs are lowercased repo-relative paths. The restrictive regex rejected spaces, parens, and special chars, blocking 5,861 Apple Notes files from importing. Now only rejects empty slugs, path traversal (..), and leading slash. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: enable RLS on all tables with BYPASSRLS safety check Without RLS, the Supabase anon key gives full read access to the DB. Enable RLS on all 10 tables with no policies — the postgres role (used by gbrain via pooler) has BYPASSRLS and is unaffected. Only enables if the current role actually has BYPASSRLS privilege to avoid locking ourselves out on non-Supabase setups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: import resilience — 5MB limit, error suppression, structured progress Raise MAX_FILE_SIZE from 1MB to 5MB for Apple Notes with attachments. Track error patterns and suppress after 5 identical errors to prevent 5,861 identical warnings from killing the agent process. Replace \r progress bar with structured log lines (rate, ETA) for agent parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: init detects IPv6-only Supabase URLs, adds pgvector check Detect db.*.supabase.co direct URLs and warn about IPv6 failure. On ECONNREFUSED/ETIMEDOUT to Supabase, suggest the Session pooler connection string with exact dashboard click path. Check for pgvector extension after connecting and fail with clear instructions if missing. Update wizard hints to show pooler URL format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add pre-ship requirement for E2E tests E2E tests against real Postgres+pgvector must pass before /ship or /review. Adds the requirement to CLAUDE.md so all agents enforce it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: parallel import with per-worker engine instances Refactor PostgresEngine to support instance-level DB connections instead of only the module-global singleton. Each worker gets its own connection with poolSize:2 (vs 10 for the main engine), so 8 workers = 16 connections. Add --workers N flag to gbrain import. Workers pull from a shared queue and use independent engine instances — no transaction context corruption. The bottleneck is network round-trips to Supabase (one per page upsert). Parallel workers cut import time proportionally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: automatic schema migration runner Migrations are embedded as string constants in migrate.ts (survives Bun --compile). Each migration runs in a transaction for clean rollback on failure. Runs automatically on initSchema() — no manual step needed when a user updates the gbrain binary against an older DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: pluggable storage backend (S3 + Supabase Storage + local) Add StorageBackend interface with three implementations: - S3Storage: works with AWS S3, Cloudflare R2, MinIO (any S3-compatible) - SupabaseStorage: uses Supabase Storage REST API with service role key - LocalStorage: filesystem-based, for testing Add file-resolver.ts with fallback chain: local file → .redirect breadcrumb → .supabase marker → storage backend. Supports the three-stage migration (mirror → redirect → clean). Add yaml-lite.ts for parsing marker and breadcrumb files without adding a YAML dependency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: gbrain doctor command — health checks with --json output Checks: connection, pgvector extension, RLS on all tables, schema version, embedding coverage. Outputs structured JSON with --json flag for agent parsing. Exit code 0 if healthy, 1 if issues found. Agents should run gbrain doctor --json when any command fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite setup skill + README for agent-first DX Setup skill: add Why Supabase, step-by-step project creation, explicit agent instructions (nohup for large imports, doctor on failure, don't ask for anon key), available init flags, file migration offer after first import. Remove ClawHub references. README: simplify to single OpenClaw install path, remove ClawHub, fix squatted npm name to github:garrytan/gbrain, add Supabase settings note about Session pooler. Add Apple Notes test fixtures with spaces and parens in filenames for E2E testing of the slug fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add RLS verification, schema health, and nohup hints to maintain skill Maintenance skill now checks RLS status and schema version as part of periodic health checks. Adds nohup pattern for large embedding refreshes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: import resume checkpoint + Supabase smart URL parsing Import resume: saves checkpoint every 100 files to ~/.gbrain/import-checkpoint.json. On restart with same directory and file count, skips already-processed files. Use --fresh to ignore checkpoint and start over. Cleared on successful completion. Supabase admin: extractProjectRef() parses any Supabase URL format (dashboard, direct, pooler, project URL) to extract the project ref. discoverPoolerUrl() uses the Management API to find the correct pooler connection string (including the exact region prefix). checkRls() verifies RLS status via the API. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add 56 unit tests for all new code 8 new test files covering every feature added in this branch: - slug-validation.test.ts: spaces, parens, unicode, path traversal (10 tests) - yaml-lite.test.ts: parse + stringify, marker/redirect formats (9 tests) - supabase-admin.test.ts: extractProjectRef for 4 URL formats (7 tests) - migrate.test.ts: version export, runMigrations callable (2 tests) - storage.test.ts: LocalStorage CRUD + createStorage factory (14 tests) - file-resolver.test.ts: fallback chain, redirect, marker parsing (6 tests) - import-resume.test.ts: checkpoint save/load/resume/fresh (6 tests) - doctor.test.ts: module export, CLI registration (3 tests) Total: 184 pass, 0 fail (up from 128). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: bulk chunk INSERT + E2E tests for all new features Bulk INSERT: upsertChunks now builds a multi-row VALUES query instead of inserting chunks one-by-one. Reduces DB round-trips by ~50x per page. E2E tests added to mechanical.test.ts: - Slug with special chars: import Apple Notes fixtures with spaces/parens, verify search finds them, verify idempotency - RLS verification: check pg_tables.rowsecurity on all tables, verify current user has BYPASSRLS - Doctor command: verify exit 0 on healthy DB, --json produces valid JSON with check structure - Parallel import: --workers 2 produces same page count as sequential Unit tests added: - setup-branching.test.ts: IPv6 detection, defaultWorkers auto-tuning, smart URL parsing across all Supabase URL formats Fixtures added: - large/big-file.md (2.1MB) for testing raised file size limit - apple-notes/ fixtures already existed Total: 200 pass, 0 fail (up from 184). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: --json on init/import, file migration CLI, lifecycle tests --json flag: init and import now support --json for structured output. Agents get parseable JSON instead of human-readable text. File migration CLI: implement mirror, unmirror, redirect, restore, clean, and status subcommands for the three-stage file migration lifecycle (local → mirrored → redirected → cloud-only). File migration tests: full lifecycle test covering every transition in the state machine (LOCAL → MIRROR → UNMIRROR → REDIRECT → RESTORE → CLEAN), including edge cases and file resolver at each stage. Bulk chunk INSERT: upsertChunks now builds multi-row parameterized VALUES query, reducing round-trips per page from ~50 to 1. Total: 207 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: thorough E2E tests for parallel import concurrency Replace the weak single-comparison parallel import test with 7 tests: - Sequential baseline: capture page count, chunk count, and all slugs - --workers 2: verify page count matches sequential - Chunk count matches (no duplicates from concurrent writes) - Page slugs match exactly - No duplicate pages (SQL GROUP BY HAVING count > 1) - No duplicate chunks (SQL GROUP BY page_id, chunk_index) - --workers 4: also works correctly - Re-import with workers is idempotent These tests catch the exact bug Codex found (db.ts singleton causing concurrent transaction corruption) by verifying data integrity after parallel writes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add batch embedding queue as P1 TODO Deferred during eng review (per-worker embedding is good enough for now). Revisit after profiling real imports to confirm embedding is the bottleneck. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: E2E test failures — fixture counts, arg parsing, doctor exit code Fix fixture count assertions: 13 → 16 pages (added apple-notes + large file), companies 2 → 3 (ohmygreen), concepts 3 → 5 (notes, big-file). Fix --workers arg parsing: the worker count value (e.g. "2") was being picked up as the directory arg. Skip flag values when finding the dir. Fix doctor exit code: warnings (like missing embeddings) should exit 0, only actual failures exit 1. E2E tests import with --no-embed, so embeddings are always WARN. Fix E2E CLI tests: add initCli() before doctor and parallel import tests so ~/.gbrain/config.json exists for the subprocess. All E2E tests pass: 63 pass, 0 fail. All unit tests pass: 207 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.4.0 New CHANGELOG entry for all post-0.3.0 features (doctor, storage backends, parallel import, resume checkpoints, RLS, schema migrations, --json output). Version bumped 0.3.0 → 0.4.0 across all manifests. CLAUDE.md: test count 9→19, skill count 8→7, added key files. CONTRIBUTING.md: fixture count 13→16, added missing source files. README.md: added gbrain doctor to commands, fixed stale welcome PRs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add GBRAIN_SKILLPACK.md reference architecture Production agent patterns from a real deployment with 14,700+ brain files. Covers: entity detection on every message, brain-first lookup protocol, 7-step enrichment pipeline with tiered API spend, compiled truth + timeline, source attribution with mandatory citations, meeting ingestion with entity propagation, cron schedule with quiet hours and travel-aware timezone, YouTube/media ingestion via Diarize.io, integration guides for ClawVisor, Circleback webhooks, and Quo/OpenPhone SMS. Opens with the Vannevar Bush memex framing and the originals folder for capturing intellectual capital. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite README opener with memex pitch and production architecture Replace code-first opener with mimetic-desire pitch: Vannevar Bush memex tagline, production brain numbers (10K+ files, 3K+ people, 13 years of calendar), "ask it anything" examples, compounding thesis. New sections: The Compounding Thesis (read-write loop), Architecture (three-column diagram), What a Production Agent Looks Like (SKILLPACK reference), How gbrain fits with OpenClaw (three-layer complement). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update skills with brain-first lookup, entity detection, heartbeat setup: Phase D rewritten with brain-first lookup protocol (gbrain search → query → get → grep fallback), sync-after-write rule, memory_search complement table. query: token-budget awareness (chunks not full pages), source precedence hierarchy (user > compiled truth > timeline > external). ingest: entity detection on every message (scan, check brain, create or enrich, commit and sync). maintain: heartbeat integration (doctor, embed --stale, sync verification, stale compiled truth detection). briefing: gbrain-native context loading (search attendees before meetings, search sender before email, daily deal/meeting/commitment queries). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add OpenClaw positioning to README opener Make it clear up top that GBrain is built for OpenClaw agents and works with any OpenClaw deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: credit Karpathy's Knowledge LLM vision, add origin story GBrain started as Karpathy's LLM wiki idea built for real. Worked great until the brain hit thousands of files and grep fell apart. GBrain is the search layer that had to exist once the brain outgrew grep. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -22,6 +22,32 @@ Compile a daily briefing from brain context.
|
||||
6. **Stale alerts.** From gbrain health check:
|
||||
- Pages flagged as stale that are relevant to today's meetings
|
||||
|
||||
## GBrain-Native Context Loading
|
||||
|
||||
Before generating any briefing, load context from gbrain systematically.
|
||||
|
||||
### Before a meeting
|
||||
|
||||
For every attendee on the calendar invite:
|
||||
- `gbrain search "<attendee name>"` -- find their brain page
|
||||
- `gbrain get <slug>` -- load compiled truth, recent timeline, relationship context
|
||||
- If no page exists, note the gap ("No brain page for Sarah Chen -- consider enrichment")
|
||||
|
||||
### Before an email reply
|
||||
|
||||
Before drafting or triaging any email:
|
||||
- `gbrain search "<sender name>"` -- load sender context
|
||||
- Read their compiled truth to understand who they are, what they care about, and
|
||||
your relationship history. This turns a cold reply into an informed one.
|
||||
|
||||
### Daily briefing queries
|
||||
|
||||
Run these queries to populate the briefing sections:
|
||||
- `gbrain query "active deals status"` -- deal pipeline snapshot
|
||||
- `gbrain query "meetings this week"` -- recent meeting pages with insights
|
||||
- `gbrain query "pending commitments follow-ups"` -- open threads and action items
|
||||
- `gbrain search --type person --sort updated --limit 10` -- people in play
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
|
||||
@@ -13,6 +13,39 @@ Ingest meetings, articles, documents, and conversations into the brain.
|
||||
4. **Create cross-reference links.** Link entities in gbrain for every entity pair mentioned together, using the appropriate relationship type.
|
||||
5. **Timeline merge.** The same event appears on ALL mentioned entities' timelines. If Alice met Bob at Acme Corp, the event goes on Alice's page, Bob's page, and Acme Corp's page.
|
||||
|
||||
## Entity Detection on Every Message
|
||||
|
||||
Production agents should detect entity mentions on EVERY inbound message. This is
|
||||
the signal detection loop that makes the brain compound over time.
|
||||
|
||||
### Protocol
|
||||
|
||||
1. **Scan the message** for entity mentions: people, companies, concepts, original
|
||||
thinking. Fire on every message (no exceptions unless purely operational).
|
||||
2. **For each entity detected:**
|
||||
- `gbrain search "name"` -- does a page already exist?
|
||||
- **If yes:** load context with `gbrain get <slug>`. Use the compiled truth to
|
||||
inform your response. Update the page if the message contains new information.
|
||||
- **If no:** assess notability. If the entity is worth tracking (will come up
|
||||
again, is relevant to the user's world), create a new page with
|
||||
`gbrain put <type/slug>` and populate with what you know.
|
||||
3. **After creating or updating pages:** commit changes to the brain repo, then
|
||||
sync to gbrain:
|
||||
```bash
|
||||
git add brain/ && git commit -m "update entity pages"
|
||||
gbrain sync --no-pull --no-embed
|
||||
```
|
||||
4. **Don't block the conversation.** Entity detection and enrichment should happen
|
||||
alongside the response, not before it. The user shouldn't wait for brain writes
|
||||
to get an answer.
|
||||
|
||||
### What counts as notable
|
||||
|
||||
- People the user interacts with or discusses (not random mentions)
|
||||
- Companies relevant to the user's work, investments, or interests
|
||||
- Concepts or frameworks the user references or creates
|
||||
- The user's own original thinking (ideas, theses, observations) -- highest value
|
||||
|
||||
## Quality Rules
|
||||
|
||||
- Executive summary in compiled_truth must be updated, not just timeline appended
|
||||
|
||||
@@ -31,12 +31,53 @@ Inconsistent tagging (e.g., "vc" vs "venture-capital", "ai" vs "artificial-intel
|
||||
|
||||
### Embedding freshness
|
||||
Chunks without embeddings, or chunks embedded with an old model.
|
||||
- Refresh stale embeddings in gbrain
|
||||
- For large embedding refreshes (>1000 chunks), use nohup:
|
||||
`nohup gbrain embed refresh > /tmp/gbrain-embed.log 2>&1 &`
|
||||
- Then check progress: `tail -1 /tmp/gbrain-embed.log`
|
||||
|
||||
### Security (RLS verification)
|
||||
Run `gbrain doctor --json` and check the RLS status.
|
||||
All tables should show RLS enabled. If not, run `gbrain init` again.
|
||||
|
||||
### Schema health
|
||||
Check that the schema version is up to date. `gbrain doctor --json` reports
|
||||
the current version vs expected. If behind, `gbrain init` runs migrations
|
||||
automatically.
|
||||
|
||||
### Open threads
|
||||
Timeline items older than 30 days with unresolved action items.
|
||||
- Flag for review
|
||||
|
||||
## Heartbeat Integration
|
||||
|
||||
For production agents running on a schedule, integrate gbrain health checks into
|
||||
your operational heartbeat.
|
||||
|
||||
### On every heartbeat (hourly or per-session)
|
||||
|
||||
Run `gbrain doctor --json` and check for degradation. Report any failing checks
|
||||
to the user. Key signals: connection health, schema version, RLS status, embedding
|
||||
staleness.
|
||||
|
||||
### Weekly maintenance
|
||||
|
||||
Run `gbrain embed --stale` to refresh embeddings for pages that have changed since
|
||||
their last embedding. For large brains (>5000 pages), run this with nohup:
|
||||
```bash
|
||||
nohup gbrain embed --stale > /tmp/gbrain-embed.log 2>&1 &
|
||||
```
|
||||
|
||||
### Daily verification
|
||||
|
||||
Verify sync is running: check `gbrain stats` and confirm `last_sync` is within
|
||||
the last 24 hours. If sync has stopped, the brain is drifting from the repo.
|
||||
|
||||
### Stale compiled truth detection
|
||||
|
||||
Flag pages where compiled truth is >30 days old but the timeline has recent entries.
|
||||
This means new evidence exists that hasn't been synthesized. These pages need a
|
||||
compiled truth rewrite (see the maintain workflow above).
|
||||
|
||||
## Quality Rules
|
||||
|
||||
- Never delete pages without confirmation
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.3.0",
|
||||
"version": "0.4.0",
|
||||
"description": "Personal knowledge brain with hybrid RAG search",
|
||||
"skills": [
|
||||
{
|
||||
|
||||
@@ -25,6 +25,30 @@ Answer questions using the brain's knowledge with 3-layer search and synthesis.
|
||||
- For "what happened" questions, use timeline entries
|
||||
- For "what do we know" questions, read compiled_truth directly
|
||||
|
||||
## Token-Budget Awareness
|
||||
|
||||
Search returns **chunks**, not full pages. Read the excerpts first before deciding
|
||||
whether to load a full page.
|
||||
|
||||
- `gbrain search` / `gbrain query` return ranked chunks with context snippets.
|
||||
These are often enough to answer the question directly.
|
||||
- Only use `gbrain get <slug>` to load the full page when a chunk confirms the
|
||||
page is relevant and you need more context (e.g., compiled truth, timeline).
|
||||
- **"Tell me about X"** -- get the full page (the user wants the complete picture).
|
||||
- **"Did anyone mention Y?"** -- search results are enough (the user wants a yes/no with evidence).
|
||||
|
||||
### Source precedence
|
||||
|
||||
When multiple sources provide conflicting information, follow this precedence:
|
||||
|
||||
1. **User's direct statements** (highest authority -- what the user told you directly)
|
||||
2. **Compiled truth** (the brain's synthesized, cited understanding)
|
||||
3. **Timeline entries** (raw evidence, reverse-chronological)
|
||||
4. **External sources** (web search, API enrichment -- lowest authority)
|
||||
|
||||
When sources conflict, note the contradiction with both citations. Don't silently
|
||||
pick one.
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Keyword search gbrain (search)
|
||||
|
||||
@@ -1,38 +1,83 @@
|
||||
# Setup GBrain
|
||||
|
||||
Set up GBrain from scratch. Target: working brain in under 2 minutes.
|
||||
Set up GBrain from scratch. Target: working brain in under 5 minutes.
|
||||
|
||||
## Install (if not already installed)
|
||||
|
||||
```bash
|
||||
bun add github:garrytan/gbrain
|
||||
```
|
||||
|
||||
## How GBrain connects
|
||||
|
||||
GBrain connects directly to Postgres over the wire protocol. NOT through the
|
||||
Supabase REST API. You need the **database connection string** (a `postgresql://` URI),
|
||||
not the project URL or anon key. The password is embedded in the connection string.
|
||||
|
||||
Use the **Session pooler** connection string (port 6543), not the direct connection
|
||||
(port 5432). The direct hostname resolves to IPv6 only, which many environments
|
||||
can't reach.
|
||||
|
||||
**Do NOT ask for the Supabase anon key.** GBrain doesn't use it.
|
||||
|
||||
## Why Supabase
|
||||
|
||||
Supabase gives you managed Postgres + pgvector (vector search built in) for $25/mo:
|
||||
- 8GB database + 100GB storage on Pro tier
|
||||
- No server to manage, automatic backups, dashboard for debugging
|
||||
- pgvector pre-installed, just works
|
||||
- Alternative: any Postgres with pgvector extension (self-hosted, Neon, Railway, etc.)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Supabase account (Pro tier recommended: $25/mo for 8GB DB + 100GB storage)
|
||||
- A Supabase account (Pro tier recommended, $25/mo) OR any Postgres with pgvector
|
||||
- An OpenAI API key (for semantic search embeddings, ~$4-5 for 7,500 pages)
|
||||
- A git-backed markdown knowledge base (or start fresh)
|
||||
|
||||
## Phase A: Auto-Provision (Supabase CLI)
|
||||
## Available init options
|
||||
|
||||
Check if the Supabase CLI is available. If it is, use the fast path:
|
||||
- `gbrain init --supabase` -- interactive wizard (prompts for connection string)
|
||||
- `gbrain init --url <connection_string>` -- direct, no prompts
|
||||
- `gbrain init --non-interactive --url <connection_string>` -- for scripts/agents
|
||||
- `gbrain doctor --json` -- health check after init
|
||||
|
||||
1. Tell the user: "I'll set up Supabase for you. Click 'Authorize' when your browser opens."
|
||||
2. Run `supabase login` (opens browser for OAuth)
|
||||
3. Run `supabase projects create --name gbrain --region us-east-1`
|
||||
4. Extract the database connection URL from `supabase projects api-keys`
|
||||
5. Initialize gbrain with the connection URL in non-interactive mode
|
||||
6. Proceed to Phase C automatically
|
||||
There is no `--local`, `--sqlite`, or offline mode. GBrain requires Postgres + pgvector.
|
||||
|
||||
## Phase B: Manual Fallback
|
||||
## Phase A: Supabase Setup (recommended)
|
||||
|
||||
If the Supabase CLI is not available, guide the user:
|
||||
Guide the user through creating a Supabase project:
|
||||
|
||||
1. "Log into Supabase and add a credit card: https://supabase.com/dashboard/account/billing"
|
||||
2. "Create a new project: https://supabase.com/dashboard/new/_"
|
||||
- Name: gbrain
|
||||
- Region: closest to you
|
||||
- Generate a strong password
|
||||
3. "Go to Project Settings > Database and copy the connection string (URI format)"
|
||||
- Paste it here
|
||||
4. Initialize gbrain with the provided URL in non-interactive mode
|
||||
1. "Go to https://supabase.com and sign up or log in."
|
||||
2. "Click 'New Project' in the top left."
|
||||
- Name: `gbrain`
|
||||
- Region: pick the one closest to you
|
||||
- Database password: generate a strong one and save it
|
||||
3. "Wait about 2 minutes for the project to initialize."
|
||||
4. "Find the connection string: click the **gear icon** (Project Settings) in the
|
||||
left sidebar, then **Database**, scroll to **Connection string**, click the
|
||||
**URI** tab, change the dropdown from **Direct connection** to **Session pooler**
|
||||
(port 6543), and copy the full string."
|
||||
5. Initialize gbrain:
|
||||
```bash
|
||||
gbrain init --non-interactive --url "postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:6543/postgres"
|
||||
```
|
||||
6. Verify: `gbrain doctor --json`
|
||||
|
||||
That's it. One copy-paste. The agent does everything else.
|
||||
**OpenClaw agent instructions:** Store the Supabase access token in your persistent
|
||||
env as `SUPABASE_ACCESS_TOKEN`. gbrain doesn't store it, you need it for future
|
||||
`gbrain doctor` runs. Generate at: https://supabase.com/dashboard/account/tokens
|
||||
|
||||
## Phase B: BYO Postgres (alternative)
|
||||
|
||||
If the user already has Postgres with pgvector:
|
||||
|
||||
1. Get the connection string from the user.
|
||||
2. Run: `gbrain init --non-interactive --url "<connection_string>"`
|
||||
3. Verify: `gbrain doctor --json`
|
||||
|
||||
If the connection fails with ECONNREFUSED and the URL contains `supabase.co`,
|
||||
the user probably pasted the direct connection (IPv6 only). Guide them to the
|
||||
Session pooler string instead (see Phase A step 4).
|
||||
|
||||
## Phase C: First Import
|
||||
|
||||
@@ -52,60 +97,103 @@ done
|
||||
echo "=== Discovery Complete ==="
|
||||
```
|
||||
|
||||
2. **Import the best candidate.** Import the recommended directory into gbrain.
|
||||
3. **Prove search works.** Search gbrain for a topic from the imported data. Show results immediately.
|
||||
4. **Start embeddings.** Refresh stale embeddings in gbrain (runs in background). Keyword search works NOW, semantic search improves as embeddings complete.
|
||||
2. **Import the best candidate.** For large imports (>1000 files), use nohup to
|
||||
survive session timeouts:
|
||||
```bash
|
||||
nohup gbrain import <dir> --no-embed --workers 4 > /tmp/gbrain-import.log 2>&1 &
|
||||
```
|
||||
Then check progress: `tail -1 /tmp/gbrain-import.log`
|
||||
|
||||
## Phase D: AGENTS.md Injection
|
||||
For smaller imports, run directly:
|
||||
```bash
|
||||
gbrain import <dir> --no-embed
|
||||
```
|
||||
|
||||
Auto-inject gbrain instructions into the project's AGENTS.md (or equivalent). Use a delimited managed block that's upgrade-safe:
|
||||
3. **Prove search works.** Pick a semantic query based on what you imported:
|
||||
```bash
|
||||
gbrain search "<topic from the imported data>"
|
||||
```
|
||||
This is the magical moment: the user sees search finding things grep couldn't.
|
||||
|
||||
```markdown
|
||||
<!-- gbrain:start -->
|
||||
## GBrain (Knowledge Search)
|
||||
4. **Start embeddings.** Refresh stale embeddings (runs in background). Keyword
|
||||
search works NOW, semantic search improves as embeddings complete.
|
||||
|
||||
GBrain indexes your knowledge base for fast search. Always search before answering
|
||||
questions about people, companies, deals, or anything in the brain.
|
||||
5. **Offer file migration.** If the repo has binary files (.raw/ directories with
|
||||
images, PDFs, audio):
|
||||
> "You have N binary files (X GB) in your brain repo. Want to move them to cloud
|
||||
> storage? Your git repo will drop from X GB to Y MB. All links keep working."
|
||||
|
||||
### How to use
|
||||
- Search gbrain for any topic before answering questions
|
||||
- After writing new content, sync the repository to gbrain
|
||||
- Upload binary files to gbrain storage instead of committing to git
|
||||
- Check gbrain health periodically
|
||||
If no markdown repos are found, create a starter brain with a few template pages
|
||||
(a person page, a company page, a concept page) from docs/GBRAIN_RECOMMENDED_SCHEMA.md.
|
||||
|
||||
### Rules
|
||||
1. **Search the brain first.** Before answering any question about people, companies,
|
||||
deals, meetings, or strategy, search gbrain. Your memory of file contents goes
|
||||
stale; the database doesn't.
|
||||
2. **Never commit binaries to git.** Upload to gbrain file storage instead.
|
||||
3. **After writing to the brain repo,** sync to gbrain immediately.
|
||||
<!-- gbrain:end -->
|
||||
## Phase D: Brain-First Lookup Protocol
|
||||
|
||||
Inject the brain-first lookup protocol into the project's AGENTS.md (or equivalent).
|
||||
This replaces grep-based knowledge lookups with structured gbrain queries.
|
||||
|
||||
### BEFORE (grep) vs AFTER (gbrain)
|
||||
|
||||
| Task | Before (grep) | After (gbrain) |
|
||||
|------|---------------|-----------------|
|
||||
| Find a person | `grep -r "Pedro" brain/` | `gbrain search "Pedro"` |
|
||||
| Understand a topic | `grep -rl "deal" brain/ \| head -5 && cat ...` | `gbrain query "what's the status of the deal"` |
|
||||
| Read a known page | `cat brain/people/pedro.md` | `gbrain get people/pedro` |
|
||||
| Find connections | `grep -rl "Brex" brain/ \| xargs grep "Pedro"` | `gbrain query "Pedro Brex relationship"` |
|
||||
|
||||
### Lookup sequence (MANDATORY for every entity question)
|
||||
|
||||
1. `gbrain search "name"` -- keyword match, fast, works without embeddings
|
||||
2. `gbrain query "what do we know about name"` -- hybrid search, needs embeddings
|
||||
3. `gbrain get <slug>` -- direct page read when you know the slug from steps 1-2
|
||||
4. `grep` fallback -- only if gbrain returns zero results AND the file may exist outside the indexed brain
|
||||
|
||||
Stop at the first step that gives you what you need. Most lookups resolve at step 1.
|
||||
|
||||
### Sync-after-write rule
|
||||
|
||||
After creating or updating any brain page in the repo, sync immediately so the
|
||||
index stays current:
|
||||
|
||||
```bash
|
||||
gbrain sync --no-pull --no-embed
|
||||
```
|
||||
|
||||
This indexes new/changed files without pulling from git or regenerating embeddings.
|
||||
Embeddings can be refreshed later in batch (`gbrain embed --stale`).
|
||||
|
||||
### gbrain vs memory_search
|
||||
|
||||
| Layer | What it stores | When to use |
|
||||
|-------|---------------|-------------|
|
||||
| **gbrain** | World knowledge: people, companies, deals, meetings, concepts, media | "Who is Pedro?", "What happened at the board meeting?" |
|
||||
| **memory_search** | Agent operational state: preferences, decisions, session context | "How does the user like formatting?", "What did we decide about X?" |
|
||||
|
||||
Both should be checked. gbrain for facts about the world. memory_search for how
|
||||
the agent should behave.
|
||||
|
||||
## Phase E: Health Check
|
||||
|
||||
After setup is complete, check gbrain health. Every dimension should be healthy.
|
||||
Report the final state to the user:
|
||||
- Page count and statistics
|
||||
- Embedding coverage
|
||||
- Search verification (run a sample query)
|
||||
Run `gbrain doctor --json` and report the results. Every check should be OK.
|
||||
If any check fails, the doctor output tells you exactly what's wrong and how to fix it.
|
||||
|
||||
## Error Handling
|
||||
## Error Recovery
|
||||
|
||||
Every error tells you what happened, why, and how to fix it:
|
||||
**If any gbrain command fails, run `gbrain doctor --json` first.** Report the full
|
||||
output. It checks connection, pgvector, RLS, schema version, and embeddings.
|
||||
|
||||
| What You See | Why | Fix |
|
||||
|---|---|---|
|
||||
| Connection refused | Supabase project paused or wrong URL | supabase.com/dashboard > Restore |
|
||||
| Connection refused | Supabase project paused, IPv6, or wrong URL | Use Session pooler (port 6543), or supabase.com/dashboard > Restore |
|
||||
| Password authentication failed | Wrong password | Project Settings > Database > Reset password |
|
||||
| pgvector not available | Extension not enabled | Run CREATE EXTENSION vector in SQL Editor |
|
||||
| pgvector not available | Extension not enabled | Run `CREATE EXTENSION vector;` in SQL Editor |
|
||||
| OpenAI key invalid | Expired or wrong key | platform.openai.com/api-keys > Create new |
|
||||
| No pages found | Query before import | Import files into gbrain first |
|
||||
| RLS not enabled | Security gap | Run `gbrain init` again (auto-enables RLS) |
|
||||
|
||||
## Tools Used
|
||||
|
||||
- Initialize gbrain (via CLI: gbrain init --non-interactive --url ...)
|
||||
- Import files into gbrain (via CLI: gbrain import)
|
||||
- Search gbrain (query)
|
||||
- Check gbrain health (get_health)
|
||||
- Get gbrain statistics (get_stats)
|
||||
- `gbrain init --non-interactive --url ...` -- create brain
|
||||
- `gbrain import <dir> --no-embed [--workers N]` -- import files
|
||||
- `gbrain search <query>` -- search brain
|
||||
- `gbrain doctor --json` -- health check
|
||||
- `gbrain embed refresh` -- generate embeddings
|
||||
|
||||
Reference in New Issue
Block a user