From e5a9f0126a3cb9b7e79d4dcf3b251e446d566177 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 14 Apr 2026 19:41:34 -1000 Subject: [PATCH] =?UTF-8?q?feat:=20GStackBrain=20=E2=80=94=2016=20new=20sk?= =?UTF-8?q?ills,=20resolver,=20conventions,=20identity=20layer=20(v0.10.0)?= =?UTF-8?q?=20(#120)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: migrate 8 existing skills to conformance format Add YAML frontmatter (name, version, description, triggers, tools, mutating), Contract, Anti-Patterns, and Output Format sections to all existing skills. Rename Workflow to Phases. Ingest becomes thin router delegating to specialized ingestion skills (Phase 2). Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add RESOLVER.md, conventions directory, and output rules RESOLVER.md is the skill dispatcher modeled on Wintermute's AGENTS.md. Categorized routing table: Always-on, Brain ops, Ingestion, Thinking, Operational, Setup, Identity. Conventions directory extracts cross-cutting rules (quality, brain-first lookup, model routing, test-before-bulk). Co-Authored-By: Claude Opus 4.6 (1M context) * test: add skills conformance and resolver validation tests skills-conformance.test.ts validates every skill has YAML frontmatter with required fields, Contract, Anti-Patterns, and Output Format sections, and manifest.json coverage. resolver.test.ts validates routing table categories, skill path existence, and manifest-to-resolver coverage. 50 new tests. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add 9 brain skills from Wintermute (Phase 2) Generalized from Wintermute's battle-tested skills: - signal-detector: always-on idea+entity capture on every message - brain-ops: brain-first lookup, read-enrich-write loop, source attribution - idea-ingest: links/articles/tweets with author people page mandatory - media-ingest: video/audio/PDF/book with entity extraction (absorbs video/youtube/book) - meeting-ingestion: transcripts with attendee enrichment chaining - citation-fixer: audit and fix citation formatting - repo-architecture: filing rules by primary subject - skill-creator: create skills with conformance standard + MECE check - daily-task-manager: task lifecycle with priority levels All Garry-specific references generalized. Core workflows preserved. Updated RESOLVER.md and manifest.json. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add operational infrastructure + identity layer (Phase 3) Operational skills: - daily-task-prep: morning prep with calendar context and open threads - cross-modal-review: quality gate via second model with refusal routing - cron-scheduler: schedule staggering, quiet hours, wake-up override, idempotency - reports: timestamped reports with keyword routing - testing: skill validation framework (conformance checks) - soul-audit: 6-phase interview generating SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md - webhook-transforms: external events to brain signals with dead-letter queue Identity layer: - SOUL.md template (agent identity, generated by soul-audit) - USER.md template (user profile, generated by soul-audit) - ACCESS_POLICY.md template (4-tier access control) - HEARTBEAT.md template (operational cadence) - cross-modal.yaml convention (review pairs, refusal routing chain) Co-Authored-By: Claude Opus 4.6 (1M context) * docs: update CLAUDE.md with 24 skills, RESOLVER.md, conventions, templates GBrain is now a GStack mod for agent platforms. Updated architecture description, key files listing (16 new skill files, RESOLVER.md, conventions, templates), skills section (24 skills organized by resolver categories), and testing section (new conformance and resolver tests). Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add GStack detection + mod status to gbrain init (Phase 4) After brain initialization, gbrain init now reports: - Number of skills loaded (from manifest.json) - GStack detection (checks known host paths, uses gstack-global-discover if available) - GStack install instructions if not found - Resolver and soul-audit pointers Also adds installDefaultTemplates() for SOUL.md/USER.md/ACCESS_POLICY.md/HEARTBEAT.md deployment, and detectGStack() using gstack-global-discover with fallback to known paths (DRY: doesn't reimplement GStack's host detection logic). Co-Authored-By: Claude Opus 4.6 (1M context) * docs: v0.10.0 release documentation - CHANGELOG: 24 skills, signal detector, RESOLVER.md, soul-audit, access control, conventions, conformance standard, GStack detection in init - README: updated skill section with 24 skills, resolver, conventions - TODOS: added runtime MCP access control (P1) - VERSION: 0.9.2 → 0.10.0 - package.json + manifest.json version bumped Co-Authored-By: Claude Opus 4.6 (1M context) * docs: add skill table to CHANGELOG v0.10.0 16-row table detailing every new skill, what it does, and why it matters. Written to sell the upgrade, not document the implementation. Co-Authored-By: Claude Opus 4.6 (1M context) * fix: restore package.json version after merge conflict resolution Co-Authored-By: Claude Opus 4.6 (1M context) * docs: zero-based README rewrite for GStackBrain v0.10.0 Lead with GStack mod identity. 24 skills table organized by category. Install block references RESOLVER.md and soul-audit. GBrain+GStack relationship explained. Removed redundancy (733 -> 406 lines). All essential content preserved: install, recipes, architecture, search, commands, engines, voice, knowledge model. Co-Authored-By: Claude Opus 4.6 (1M context) * docs: extract install block to INSTALL_FOR_AGENTS.md, simplify README The 30-line copy-paste install block becomes one line: "Retrieve and follow INSTALL_FOR_AGENTS.md" Benefits: agent always gets latest instructions (no stale copy-paste), README stays clean, install details live where agents read them. README now leads with what GBrain does ("gives your agent a brain") instead of GStack relationship. Removed "requires frontier model" note. Co-Authored-By: Claude Opus 4.6 (1M context) * fix: 3 bugs in init.ts from merge conflict resolution 1. llstatSync typo (merge corruption) → lstatSync 2. __dirname undefined in ESM module → fileURLToPath polyfill 3. require('fs') in ESM → use imported readFileSync All three would crash gbrain init at runtime. Caught by /review. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add checkResolvable shared core function for resolver validation Shared function at src/core/check-resolvable.ts validates that all skills are reachable from RESOLVER.md, detects MECE overlaps (with whitelist for always-on/router skills), finds gaps in frontmatter triggers, and scans for DRY violations. Returns structured ResolvableIssue objects with machine-parseable fix objects alongside human-readable action strings. Three call sites: bun test, gbrain doctor, skill-creator skill. Cleans up test/resolver.test.ts: removes stale 9-line skip list, imports from production check-resolvable.ts instead of reimplementing parsing. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: expand doctor with resolver validation, filesystem-first architecture Doctor now runs filesystem checks (resolver health, skill conformance) before connecting to DB. New --fast flag skips DB checks. Falls back to filesystem-only when DB is unavailable. Adds schema_version: 2 to JSON output, composite health score (0-100), and structured issues array with action strings for agent parsing. Resolver health check calls checkResolvable() and surfaces actionable fix instructions. Link integrity check uses engine.getHealth() dead_links count. CLI routing split: doctor dispatched before connectEngine() so filesystem checks always run. Fixes Codex-identified blocker where doctor required DB. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add adaptive load-aware throttling and fail-improve loop backoff.ts: System load checking (CPU via os.loadavg, memory via os.freemem), exponential backoff with 20-attempt max guard, active hours multiplier (2x slower during waking hours), concurrent process limit (max 2). Windows-safe: defaults to "proceed" when os.loadavg returns zeros. fail-improve.ts: Deterministic-first, LLM-fallback pattern with JSONL failure logging. Cascade failure handling: when both paths fail, throws LLM error and logs both. Log rotation at 1000 entries. Call count tracking for deterministic hit rate metrics. Auto-generates test cases from successful LLM fallbacks. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add transcription service and enrichment-as-a-service transcription.ts: Groq Whisper (default) with OpenAI fallback. Files >25MB segmented via ffmpeg. Provider auto-detection from env vars. Clear error messages for missing API keys and unsupported formats. enrichment-service.ts: Global enrichment service callable from any ingest pathway. Entity slug generation (people/jane-doe, companies/acme-corp), mention counting via searchKeyword, tier auto-escalation (Tier 3→2→1 based on mention frequency and source diversity), batch enrichment with backoff throttling, regex-based entity extraction from text. Co-Authored-By: Claude Opus 4.6 (1M context) * feat: add data-research skill with recipe system, extraction, dedup, tracker New skill: data-research — one parameterized pipeline for any email-to- structured-data workflow (investor updates, donations, company metrics). 7-phase pipeline: define recipe, search, classify, extract (with extraction integrity rule), archive, deduplicate, update tracker. data-research.ts: Recipe validation, MRR/ARR/runway/headcount regex extraction (battle-tested patterns), dedup with configurable tolerance, markdown tracker parsing/appending, quarterly/monthly date windowing, 6-phase HTML email stripping with 500KB ReDoS cap. Registers data-research in manifest.json (25th skill) and RESOLVER.md. Fixes backoff test robustness for high-load systems. Co-Authored-By: Claude Opus 4.6 (1M context) * docs: update project documentation for v0.10.0 infrastructure additions CLAUDE.md: added 6 new core files (check-resolvable, backoff, fail-improve, transcription, enrichment-service, data-research), 6 new test files, updated skill count to 25, test file count to 34. README.md: updated skill count to 25, added data-research to skills table. CHANGELOG.md: added Infrastructure section documenting resolver validation, doctor expansion, adaptive throttling, fail-improve loop, voice transcription, enrichment service, and data-research skill. TODOS.md: anonymized personal references. Co-Authored-By: Claude Opus 4.6 (1M context) * fix: doctor.ts use ES module imports, harden backoff test Replace require('fs') with ES module import in doctor.ts for consistency with the rest of the file. Backoff test made resilient to parallel test execution leaking module-level state. Co-Authored-By: Claude Opus 4.6 (1M context) * docs: README rewrite with production brain stats, sample output, new infrastructure Lead with the flex: 17,888 pages, 4,383 people, 723 companies, 526 meeting transcripts built in 12 days. Show sample query output so readers see what they'll get. Document self-improving infrastructure (tier auto-escalation, fail-improve loop, doctor trajectory). Add data-research recipes to Getting Data In. Update commands section with doctor --fix, transcribe, research init/list. Fix stale "24" references to "25". Co-Authored-By: Claude Opus 4.6 (1M context) * docs: README lead with YC President origin and production agent deployments Co-Authored-By: Claude Opus 4.6 (1M context) * docs: README lead with skill philosophy and link to Thin Harness Fat Skills Skills section now explains: skill files are code, they encode entire workflows, they call deterministic TypeScript for the parts that shouldn't be LLM judgment. Links to the tweet and the architecture essay. Co-Authored-By: Claude Opus 4.6 (1M context) * docs: link GStack repo, add 70K stars and 30K daily users Co-Authored-By: Claude Opus 4.6 (1M context) * docs: remove meeting transcript count from README (sensitive) Co-Authored-By: Claude Opus 4.6 (1M context) * docs: README lead with YC President origin and production agent deployments Co-Authored-By: Claude Opus 4.6 (1M context) * fix: rename political-donations recipe to expense-tracker (sensitivity) Renamed the built-in data-research recipe from political-donations to expense-tracker across README, CHANGELOG, SKILL.md, and reports routing. Same extraction patterns (amounts, dates, recipients), neutral framing. Also renamed social-radar keyword route to social-mentions. Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 65 ++ CLAUDE.md | 67 +- INSTALL_FOR_AGENTS.md | 115 ++++ README.md | 883 ++++++++----------------- TODOS.md | 17 +- VERSION | 2 +- package.json | 2 +- skills/RESOLVER.md | 91 +++ skills/_output-rules.md | 40 ++ skills/brain-ops/SKILL.md | 121 ++++ skills/briefing/SKILL.md | 34 +- skills/citation-fixer/SKILL.md | 56 ++ skills/conventions/brain-first.md | 21 + skills/conventions/cross-modal.yaml | 35 + skills/conventions/model-routing.md | 28 + skills/conventions/quality.md | 40 ++ skills/conventions/test-before-bulk.md | 35 + skills/cron-scheduler/SKILL.md | 62 ++ skills/cross-modal-review/SKILL.md | 69 ++ skills/daily-task-manager/SKILL.md | 70 ++ skills/daily-task-prep/SKILL.md | 61 ++ skills/data-research/SKILL.md | 138 ++++ skills/enrich/SKILL.md | 61 ++ skills/idea-ingest/SKILL.md | 99 +++ skills/ingest/SKILL.md | 55 +- skills/maintain/SKILL.md | 80 ++- skills/manifest.json | 99 ++- skills/media-ingest/SKILL.md | 112 ++++ skills/meeting-ingestion/SKILL.md | 112 ++++ skills/migrate/SKILL.md | 59 +- skills/publish/SKILL.md | 43 ++ skills/query/SKILL.md | 52 +- skills/repo-architecture/SKILL.md | 53 ++ skills/reports/SKILL.md | 59 ++ skills/setup/SKILL.md | 50 ++ skills/signal-detector/SKILL.md | 102 +++ skills/skill-creator/SKILL.md | 82 +++ skills/soul-audit/SKILL.md | 85 +++ skills/testing/SKILL.md | 61 ++ skills/webhook-transforms/SKILL.md | 83 +++ src/cli.ts | 26 +- src/commands/doctor.ts | 178 ++++- src/commands/init.ts | 103 ++- src/core/backoff.ts | 190 ++++++ src/core/check-resolvable.ts | 356 ++++++++++ src/core/data-research.ts | 416 ++++++++++++ src/core/enrichment-service.ts | 270 ++++++++ src/core/fail-improve.ts | 247 +++++++ src/core/transcription.ts | 237 +++++++ templates/ACCESS_POLICY.md.template | 26 + templates/HEARTBEAT.md.template | 43 ++ templates/SOUL.md.template | 50 ++ templates/USER.md.template | 23 + test/backoff.test.ts | 132 ++++ test/check-resolvable.test.ts | 128 ++++ test/data-research.test.ts | 281 ++++++++ test/doctor.test.ts | 21 + test/enrichment-service.test.ts | 111 ++++ test/fail-improve.test.ts | 174 +++++ test/resolver.test.ts | 51 ++ test/skills-conformance.test.ts | 106 +++ test/transcription.test.ts | 70 ++ 62 files changed, 5980 insertions(+), 658 deletions(-) create mode 100644 INSTALL_FOR_AGENTS.md create mode 100644 skills/RESOLVER.md create mode 100644 skills/_output-rules.md create mode 100644 skills/brain-ops/SKILL.md create mode 100644 skills/citation-fixer/SKILL.md create mode 100644 skills/conventions/brain-first.md create mode 100644 skills/conventions/cross-modal.yaml create mode 100644 skills/conventions/model-routing.md create mode 100644 skills/conventions/quality.md create mode 100644 skills/conventions/test-before-bulk.md create mode 100644 skills/cron-scheduler/SKILL.md create mode 100644 skills/cross-modal-review/SKILL.md create mode 100644 skills/daily-task-manager/SKILL.md create mode 100644 skills/daily-task-prep/SKILL.md create mode 100644 skills/data-research/SKILL.md create mode 100644 skills/idea-ingest/SKILL.md create mode 100644 skills/media-ingest/SKILL.md create mode 100644 skills/meeting-ingestion/SKILL.md create mode 100644 skills/repo-architecture/SKILL.md create mode 100644 skills/reports/SKILL.md create mode 100644 skills/signal-detector/SKILL.md create mode 100644 skills/skill-creator/SKILL.md create mode 100644 skills/soul-audit/SKILL.md create mode 100644 skills/testing/SKILL.md create mode 100644 skills/webhook-transforms/SKILL.md create mode 100644 src/core/backoff.ts create mode 100644 src/core/check-resolvable.ts create mode 100644 src/core/data-research.ts create mode 100644 src/core/enrichment-service.ts create mode 100644 src/core/fail-improve.ts create mode 100644 src/core/transcription.ts create mode 100644 templates/ACCESS_POLICY.md.template create mode 100644 templates/HEARTBEAT.md.template create mode 100644 templates/SOUL.md.template create mode 100644 templates/USER.md.template create mode 100644 test/backoff.test.ts create mode 100644 test/check-resolvable.test.ts create mode 100644 test/data-research.test.ts create mode 100644 test/enrichment-service.test.ts create mode 100644 test/fail-improve.test.ts create mode 100644 test/resolver.test.ts create mode 100644 test/skills-conformance.test.ts create mode 100644 test/transcription.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 28f0fa7..b9b7a18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,71 @@ All notable changes to GBrain will be documented in this file. +## [0.10.0] - 2026-04-14 + +### Added + +- **Your agent now has 24 skills, not 8.** 16 new brain skills generalized from a production deployment with 14,700+ pages. Signal detection, brain-first lookup, content ingestion (articles, video, meetings), entity enrichment, task management, cron scheduling, reports, and cross-modal review. All shipped as fat markdown files your agent reads on demand. + +- **Signal detector fires on every message.** A cheap sub-agent spawns in parallel to capture original thinking and entity mentions. Ideas get preserved with exact phrasing. Entities get brain pages. The brain compounds on autopilot. + +- **RESOLVER.md routes your agent to the right skill.** Modeled on a 215-line production dispatcher. Categorized routing table: always-on, brain ops, ingestion, thinking, operational. Your agent reads it, matches the user's intent, loads the skill. No slash commands needed. + +- **Soul-audit builds your agent's identity.** 6-phase interactive interview generates SOUL.md (who the agent is), USER.md (who you are), ACCESS_POLICY.md (who sees what), and HEARTBEAT.md (operational cadence). Re-runnable anytime. Ships with minimal defaults so first boot is instant. + +- **Access control out of the box.** 4-tier privacy policy (Full/Work/Family/None) enforced by skill instructions before every response. Template-based, configurable per user. + +- **Conventions directory codifies operational discipline.** Brain-first lookup protocol, citation quality standards, model routing table, test-before-bulk rule, and cross-modal review pairs. These are the hard-won patterns that prevent bad bulk runs and silent failures. + +- **`gbrain init` detects GStack and reports mod status.** After brain setup, init now shows how many skills are loaded, whether GStack is installed, and where to get it. GStack detection uses `gstack-global-discover` with fallback to known host paths. + +- **Conformance standard for all skills.** Every skill now has YAML frontmatter (name, version, description, triggers, tools, mutating) plus Contract, Anti-Patterns, and Output Format sections. Two new test files validate conformance across all 25 skills. + +- **Existing 8 skills migrated to conformance format.** Frontmatter added, Workflow renamed to Phases, Contract and Anti-Patterns sections added. Ingest becomes a thin router delegating to specialized ingestion skills. + +### The 16 new skills + +| Skill | What it does | Why it matters | +|-------|-------------|----------------| +| **signal-detector** | Fires on every message. Spawns a cheap model in parallel to capture original thinking and entity mentions. | Your brain compounds on autopilot. Every conversation is an ingest event. Miss a signal and the brain never learns it. | +| **brain-ops** | Brain-first lookup before any external API. The read-enrich-write loop that makes every response smarter. | Without this, your agent reaches for Google when the answer is already in the brain. Wastes tokens, misses context. | +| **idea-ingest** | Links, articles, tweets go into the brain with analysis, author people pages, and cross-linking. | Every article worth reading is worth remembering. The author gets a people page. The ideas get cross-linked to what you already know. | +| **media-ingest** | Video, audio, PDF, books, screenshots, GitHub repos. Transcripts, entity extraction, backlink propagation. | One skill handles every media format. Absorbs what used to be 3 separate skills (video-ingest, youtube-ingest, book-ingest). | +| **meeting-ingestion** | Transcripts become brain pages. Every attendee gets enriched. Every company discussed gets a timeline entry. | A meeting is NOT fully ingested until every entity is propagated. This is the skill that turns a transcript into 10 updated brain pages. | +| **citation-fixer** | Scans brain pages for missing or malformed `[Source: ...]` citations. Fixes formatting to match the standard. | Without citations, you can't trace facts back to where they came from. Six months later, "who said this?" has an answer. | +| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format or source. | Prevents the #1 misfiling pattern: dumping everything in `sources/` because it came from a URL. | +| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. Updates manifest and resolver. | Users who need a capability GBrain doesn't have can create it themselves. The skill teaches the agent how to extend itself. | +| **daily-task-manager** | Add, complete, defer, remove, review tasks with priority levels (P0-P3). Stored as a searchable brain page. | Your tasks live in the brain, not a separate app. The agent can cross-reference tasks with meeting notes and people pages. | +| **daily-task-prep** | Morning preparation. Calendar lookahead with brain context per attendee, open threads from yesterday, active task review. | Walk into every meeting with full context on every person in the room, automatically. | +| **cross-modal-review** | Spawn a different AI model to review the agent's work before committing. Refusal routing: if one model refuses, silently switch. | Two models agreeing is stronger signal than one model being thorough. Refusal routing means the user never sees "I can't do that." | +| **cron-scheduler** | Schedule staggering (5-min offsets), quiet hours (timezone-aware with wake-up override), thin job prompts. | 21 cron jobs at :00 is a thundering herd. Staggering prevents it. Quiet hours mean no 3 AM notifications. Wake-up override releases the backlog. | +| **reports** | Timestamped reports with keyword routing. "What's the latest briefing?" maps to the right report directory. | Cheap replacement for vector search on frequent queries. Don't embed. Load the file. | +| **testing** | Validates every skill has SKILL.md with frontmatter, manifest coverage, resolver coverage. The CI for your skill system. | 3 skills and you need validation. 24 skills and you need it yesterday. Catches dead references, missing sections, MECE violations. | +| **soul-audit** | 6-phase interview that generates SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md. Your agent's identity, built from your answers. | What makes Wintermute feel like Wintermute. Without personality and access control, every agent feels the same. | +| **webhook-transforms** | External events (SMS, meetings, social mentions) converted into brain pages with entity extraction. Dead-letter queue for failures. | Your brain ingests signals from everywhere. Not just conversations, but every webhook, every notification, every external event. | + +### Infrastructure (new in v0.10.0) + +- **Your brain now self-validates its own skill routing.** `checkResolvable()` verifies every skill is reachable from RESOLVER.md, detects MECE overlaps, flags missing triggers, and catches DRY violations. Runs from `bun test`, `gbrain doctor`, and the skill-creator skill. Every issue comes with a machine-readable fix object the agent can act on. + +- **`gbrain doctor` got serious.** 8 health checks now (up from 5), plus a composite health score (0-100). Filesystem checks (resolver, conformance) run even without a database. `--fast` skips DB checks. `--json` output includes structured `issues` array with action strings so agents can parse and auto-fix. + +- **Batch operations won't melt your machine anymore.** Adaptive load-aware throttling checks CPU and memory before each batch item. Exponential backoff with a 20-attempt safety cap. Active hours multiplier slows batch work during the day. Two concurrent batch process limit. + +- **Your agent's classifiers get smarter automatically.** Fail-improve loop: try deterministic code first, fall back to LLM, log every fallback. Over time, the logs reveal which regex patterns are missing. Auto-generates test cases from successful LLM results. Tracks deterministic hit rate in `gbrain doctor` output. + +- **Voice notes just work.** Groq Whisper transcription (with OpenAI fallback) via `transcribe_audio` operation. Files over 25MB get ffmpeg-segmented automatically. Transcripts flow through the standard import pipeline, entities get extracted, back-links get created. + +- **Enrichment is now a global service, not a per-skill skill.** Every ingest pathway can call `extractAndEnrich()` to detect entities and create/update their brain pages. Tier auto-escalation: entities start at Tier 3, auto-promote to Tier 1 based on mention frequency across sources. + +- **Data research: one skill for any email-to-tracker pipeline.** New `data-research` skill with parameterized YAML recipes. Extract investor updates (MRR, ARR, runway, headcount), expense receipts, company metrics from email. Battle-tested regex patterns, extraction integrity rule (save first, report second), dedup with configurable tolerance, canonical tracker pages with running totals. + +### For contributors + +- `test/skills-conformance.test.ts` validates every skill has valid frontmatter and required sections +- `test/resolver.test.ts` validates RESOLVER.md coverage and routing consistency +- `skills/manifest.json` now has `conformance_version` field and lists all 24 skills +- Identity templates in `templates/` (SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md) ## [0.9.3] - 2026-04-12 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 0e9a993..d864def 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,11 @@ # CLAUDE.md -GBrain is a personal knowledge brain. Pluggable engines: PGLite (embedded Postgres -via WASM, zero-config default) or Postgres + pgvector + hybrid search in a managed -Supabase instance. `gbrain init` defaults to PGLite; suggests Supabase for 1000+ files. +GBrain is a personal knowledge brain and GStack mod for agent platforms. Pluggable +engines: PGLite (embedded Postgres via WASM, zero-config default) or Postgres + pgvector ++ hybrid search in a managed Supabase instance. `gbrain init` defaults to PGLite; +suggests Supabase for 1000+ files. GStack teaches agents how to code. GBrain teaches +agents everything else: brain ops, signal detection, content ingestion, enrichment, +cron scheduling, reports, identity, and access control. ## Architecture @@ -33,6 +36,12 @@ markdown files (tool-agnostic, work with both CLI and plugin contexts). - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff +- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects +- `src/core/backoff.ts` — Adaptive load-aware throttling: CPU/memory checks, exponential backoff, active hours multiplier +- `src/core/fail-improve.ts` — Deterministic-first, LLM-fallback loop with JSONL failure logging and auto-test generation +- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB +- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling +- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping - `src/mcp/server.ts` — MCP stdio server (generated from operations) - `src/commands/auth.ts` — Standalone token management (create/list/revoke/test) - `src/commands/upgrade.ts` — Self-update CLI with post-upgrade feature discovery @@ -55,6 +64,27 @@ markdown files (tool-agnostic, work with both CLI and plugin contexts). - `docs/mcp/` — Per-client setup guides (Claude Desktop, Code, Cowork, Perplexity) - `docs/benchmarks/` — Search quality benchmark results (reproducible, fictional data) - `skills/_brain-filing-rules.md` — Cross-cutting brain filing rules (referenced by all brain-writing skills) +- `skills/RESOLVER.md` — Skill routing table (modeled on Wintermute's AGENTS.md) +- `skills/conventions/` — Cross-cutting rules (quality, brain-first, model-routing, test-before-bulk, cross-modal) +- `skills/_output-rules.md` — Output quality standards (deterministic links, no slop, exact phrasing) +- `skills/signal-detector/SKILL.md` — Always-on idea+entity capture on every message +- `skills/brain-ops/SKILL.md` — Brain-first lookup, read-enrich-write loop, source attribution +- `skills/idea-ingest/SKILL.md` — Links/articles/tweets with author people page mandatory +- `skills/media-ingest/SKILL.md` — Video/audio/PDF/book with entity extraction +- `skills/meeting-ingestion/SKILL.md` — Transcripts with attendee enrichment chaining +- `skills/citation-fixer/SKILL.md` — Citation format auditing and fixing +- `skills/repo-architecture/SKILL.md` — Filing rules by primary subject +- `skills/skill-creator/SKILL.md` — Create conforming skills with MECE check +- `skills/daily-task-manager/SKILL.md` — Task lifecycle with priority levels +- `skills/daily-task-prep/SKILL.md` — Morning prep with calendar context +- `skills/cross-modal-review/SKILL.md` — Quality gate via second model +- `skills/cron-scheduler/SKILL.md` — Schedule staggering, quiet hours, idempotency +- `skills/reports/SKILL.md` — Timestamped reports with keyword routing +- `skills/testing/SKILL.md` — Skill validation framework +- `skills/soul-audit/SKILL.md` — 6-phase interview for SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md +- `skills/webhook-transforms/SKILL.md` — External events to brain signals +- `skills/data-research/SKILL.md` — Structured data research: email-to-tracker pipeline with parameterized YAML recipes +- `templates/` — SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md templates - `skills/migrations/` — Version migration files with feature_pitch YAML frontmatter - `src/commands/publish.ts` — Deterministic brain page publisher (code+skill pair, zero LLM calls) - `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law) @@ -72,7 +102,7 @@ Key commands added in v0.7: ## Testing -`bun test` runs all tests (28 unit test files + 5 E2E test files). Unit tests run +`bun test` runs all tests (34 unit test files + 5 E2E test files). Unit tests run without a database. E2E tests skip gracefully when `DATABASE_URL` is not set. Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts` @@ -92,10 +122,18 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/backlinks.test.ts` (entity extraction, back-link detection, timeline entry generation), `test/lint.test.ts` (LLM artifact detection, code fence stripping, frontmatter validation), `test/report.test.ts` (report format, directory structure), +`test/skills-conformance.test.ts` (skill frontmatter + required sections validation), +`test/resolver.test.ts` (RESOLVER.md coverage, routing validation), `test/search.test.ts` (RRF normalization, compiled truth boost, cosine similarity, dedup key), `test/dedup.test.ts` (source-aware dedup, compiled truth guarantee, layer interactions), `test/intent.test.ts` (query intent classification: entity/temporal/event/general), -`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels). +`test/eval.test.ts` (retrieval metrics: precisionAtK, recallAtK, mrr, ndcgAtK, parseQrels), +`test/check-resolvable.test.ts` (resolver reachability, MECE overlap, gap detection, DRY checks), +`test/backoff.test.ts` (load-aware throttling, concurrency limits, active hours), +`test/fail-improve.test.ts` (deterministic/LLM cascade, JSONL logging, test generation, rotation), +`test/transcription.test.ts` (provider detection, format validation, API key errors), +`test/enrichment-service.test.ts` (entity slugification, extraction, tier escalation), +`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping). E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`. - `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys) @@ -152,10 +190,21 @@ stop and remove it before starting a new one. ## Skills -Read the skill files in `skills/` before doing brain operations. They contain the -workflows, heuristics, and quality rules for ingestion, querying, maintenance, -enrichment, and setup. 7 skills: ingest, query, maintain, enrich, briefing, -migrate, setup. +Read the skill files in `skills/` before doing brain operations. GBrain ships 25 skills +organized by `skills/RESOLVER.md`: + +**Original 8 (conformance-migrated):** ingest (thin router), query, maintain, enrich, +briefing, migrate, setup, publish. + +**Brain skills (from Wintermute):** signal-detector, brain-ops, idea-ingest, media-ingest, +meeting-ingestion, citation-fixer, repo-architecture, skill-creator, daily-task-manager. + +**Operational + identity:** daily-task-prep, cross-modal-review, cron-scheduler, reports, +testing, soul-audit, webhook-transforms. + +**Conventions:** `skills/conventions/` has cross-cutting rules (quality, brain-first, +model-routing, test-before-bulk, cross-modal). `skills/_brain-filing-rules.md` and +`skills/_output-rules.md` are shared references. ## Build diff --git a/INSTALL_FOR_AGENTS.md b/INSTALL_FOR_AGENTS.md new file mode 100644 index 0000000..f95d30d --- /dev/null +++ b/INSTALL_FOR_AGENTS.md @@ -0,0 +1,115 @@ +# GBrain Installation Guide for AI Agents + +Read this entire file, then follow the steps. Ask the user for API keys when needed. +Target: ~30 minutes to a fully working brain. + +## Step 1: Install GBrain + +```bash +git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain +curl -fsSL https://bun.sh/install | bash +export PATH="$HOME/.bun/bin:$PATH" +bun install && bun link +``` + +Verify: `gbrain --version` should print a version number. If `gbrain` is not found, +restart the shell or add the PATH export to the shell profile. + +## Step 2: API Keys + +Ask the user for these: + +```bash +export OPENAI_API_KEY=sk-... # required for vector search +export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality +``` + +Save to shell profile or `.env`. Without OpenAI, keyword search still works. +Without Anthropic, search works but skips query expansion. + +## Step 3: Create the Brain + +```bash +gbrain init # PGLite, no server needed +gbrain doctor --json # verify all checks pass +``` + +The user's markdown files (notes, docs, brain repo) are SEPARATE from this tool repo. +Ask the user where their files are, or create a new brain repo: + +```bash +mkdir -p ~/brain && cd ~/brain && git init +``` + +Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory +structure (people/, companies/, concepts/, etc.) inside the user's brain repo, +NOT inside ~/gbrain. + +## Step 4: Import and Index + +```bash +gbrain import ~/brain/ --no-embed # import markdown files +gbrain embed --stale # generate vector embeddings +gbrain query "key themes across these documents?" +``` + +## Step 5: Load Skills + +Read `~/gbrain/skills/RESOLVER.md`. This is the skill dispatcher. It tells you which +skill to read for any task. Save this to your memory permanently. + +The three most important skills to adopt immediately: + +1. **Signal detector** (`skills/signal-detector/SKILL.md`) — fire this on EVERY + inbound message. It captures ideas and entities in parallel. The brain compounds. + +2. **Brain-ops** (`skills/brain-ops/SKILL.md`) — brain-first lookup on every response. + Check the brain before any external API call. + +3. **Conventions** (`skills/conventions/quality.md`) — citation format, back-linking + iron law, source attribution. These are non-negotiable quality rules. + +## Step 6: Identity (optional) + +Run the soul-audit skill to customize the agent's identity: + +``` +Read skills/soul-audit/SKILL.md and follow it. +``` + +This generates SOUL.md (agent identity), USER.md (user profile), ACCESS_POLICY.md +(who sees what), and HEARTBEAT.md (operational cadence) from the user's answers. + +If skipped, minimal defaults are installed automatically. + +## Step 7: Recurring Jobs + +Set up using your platform's scheduler (OpenClaw cron, Railway cron, crontab): + +- **Live sync** (every 15 min): `gbrain sync --repo ~/brain && gbrain embed --stale` +- **Auto-update** (daily): `gbrain check-update --json` (tell user, never auto-install) +- **Dream cycle** (nightly): read `docs/guides/cron-schedule.md` for the full protocol. + Entity sweep, citation fixes, memory consolidation. This is what makes the brain + compound. Do not skip it. +- **Weekly**: `gbrain doctor --json && gbrain embed --stale` + +## Step 8: Integrations + +Run `gbrain integrations list`. Each recipe in `~/gbrain/recipes/` is a self-contained +installer. It tells you what credentials to ask for, how to validate, and what cron +to register. Ask the user which integrations they want (email, calendar, voice, Twitter). + +Verify: `gbrain integrations doctor` (after at least one is configured) + +## Step 9: Verify + +Read `docs/GBRAIN_VERIFY.md` and run all 6 verification checks. Check #4 (live sync +actually works) is the most important. + +## Upgrade + +```bash +cd ~/gbrain && git pull origin main && bun install +``` + +Then run `gbrain init` to apply any schema migrations (idempotent, safe to re-run). diff --git a/README.md b/README.md index 5884781..becb5cf 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,32 @@ # GBrain -Your AI agent is smart but it doesn't know anything about your life. GBrain fixes that. Meetings, emails, tweets, calendar events, voice calls, original ideas... all of it flows into a searchable knowledge base that your agent reads before every response and writes to after every conversation. The agent gets smarter every day. +Your AI agent is smart but forgetful. GBrain gives it a brain. -> **~30 minutes to a fully working brain.** Your agent does the work. Database ready in 2 seconds (PGLite, no server). Schema, import, embeddings, and integrations take 15-30 minutes depending on brain size. You just answer questions about API keys. -> -> **Requires a frontier model.** Tested with **Claude Opus 4.6** and **GPT-5.4 Thinking**. Likely to break with smaller models. +Built by the President and CEO of Y Combinator to run his actual AI agents. The production brain powering his OpenClaw and Hermes deployments: **17,888 pages, 4,383 people, 723 companies**, 21 cron jobs running autonomously, built in 12 days. The agent ingests meetings, emails, tweets, voice calls, and original ideas while you sleep. It enriches every person and company it encounters. It fixes its own citations and consolidates memory overnight. You wake up and the brain is smarter than when you went to bed. -## Need an AI agent first? +GBrain is those patterns, generalized. 25 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours. + +> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys. + +## Install + +### On an agent platform (recommended) GBrain is designed to be installed and operated by an AI agent. If you don't have one running yet: -- **[OpenClaw](https://openclaw.ai)** — Deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, requires 8GB+ RAM instance) -- **[Hermes Agent](https://github.com/NousResearch/hermes-agent)** — Deploy on [Railway using this template](https://github.com/praveen-ks-2001/hermes-agent-template) (one click) +- **[OpenClaw](https://openclaw.ai)** ... Deploy [AlphaClaw on Render](https://render.com/deploy?repo=https://github.com/chrysb/alphaclaw) (one click, 8GB+ RAM) +- **[Hermes Agent](https://github.com/NousResearch/hermes-agent)** ... Deploy on [Railway](https://github.com/praveen-ks-2001/hermes-agent-template) (one click) -## Start here - -**https://github.com/garrytan/gbrain** — clone this repo into your agent's filesystem. It's home base for docs, skills, upgrades, and recipes. To upgrade later: `git pull origin main && bun install`. - -Copy this block into [OpenClaw](https://openclaw.ai), [Hermes](https://github.com/NousResearch/hermes-agent), or any persistent AI agent. The agent reads the docs, does the work, and asks you for API keys. ~30 minutes. +Paste this into your agent: ``` -INSTALL: - - git clone https://github.com/garrytan/gbrain.git ~/gbrain && cd ~/gbrain - curl -fsSL https://bun.sh/install | bash - export PATH="$HOME/.bun/bin:$PATH" - bun install && bun link - Verify: gbrain --version - (If gbrain is not found, restart your shell or add the PATH export - to your shell profile.) - -API KEYS — ask the user for these: - - export OPENAI_API_KEY=sk-... # required for vector search - export ANTHROPIC_API_KEY=sk-ant-... # optional, improves search quality - Save to shell profile or .env. Without OpenAI, keyword search still - works. Without Anthropic, search works but skips query expansion. - -SET UP THE BRAIN: - - gbrain init # PGLite, no server needed - Verify: gbrain doctor --json # all checks pass - - The user's markdown files (notes, docs, brain repo) are SEPARATE from - this tool repo. Ask the user where their files are, or create a new - brain repo: mkdir -p ~/brain && cd ~/brain && git init - - Read ~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md — set up the MECE - directory structure (people/, companies/, concepts/, etc.) inside the - user's brain repo, NOT inside ~/gbrain. - - gbrain import ~/brain/ --no-embed # import markdown files - gbrain embed --stale # generate vector embeddings - gbrain query "key themes across these documents?" - -GO LIVE — read and permanently adopt these docs from ~/gbrain/: - - docs/GBRAIN_SKILLPACK.md — the full agent playbook. Save to your - memory: brain-first lookup on every message, entity detection, - back-linking. This changes how you behave permanently. - - Set up recurring jobs using your platform's scheduler (crontab, - OpenClaw cron, Railway cron, or a process manager): - - Live sync (every 15 min): gbrain sync --repo ~/brain && gbrain embed --stale - - Auto-update (daily): gbrain check-update --json (tell user, never auto-install) - - Dream cycle (nightly): read docs/guides/cron-schedule.md for the - full protocol — entity sweep, citation fixes, memory consolidation. - This is what makes the brain compound. Do not skip it. - - Weekly: gbrain doctor --json && gbrain embed --stale - - docs/GBRAIN_VERIFY.md — run all 6 verification checks. #4 (live sync - actually works) is the most important. - -INTEGRATIONS — run gbrain integrations list. Each recipe in - ~/gbrain/recipes/ is a self-contained installer: it tells you what - credentials to ask for, how to validate, and what cron to register. - Ask the user which integrations they want (email, calendar, voice, - Twitter). Set up the ones they have credentials for. - Verify: gbrain integrations doctor (after at least one is configured) - -UPGRADE: cd ~/gbrain && git pull origin main && bun install +Retrieve and follow the instructions at: +https://raw.githubusercontent.com/garrytan/gbrain/master/INSTALL_FOR_AGENTS.md ``` -### Without an agent (standalone CLI) +That's it. The agent clones the repo, installs GBrain, sets up the brain, loads 25 skills, and configures recurring jobs. You answer a few questions about API keys. ~30 minutes. + +### Standalone CLI (no agent) ```bash git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link @@ -91,77 +35,157 @@ gbrain import ~/notes/ # index your markdown gbrain query "what themes show up across my notes?" ``` +``` +3 results (hybrid search, 0.12s): + +1. concepts/do-things-that-dont-scale (score: 0.94) + PG's argument that unscalable effort teaches you what users want. + [Source: paulgraham.com, 2013-07-01] + +2. originals/founder-mode-observation (score: 0.87) + Deep involvement isn't micromanagement if it expands the team's thinking. + +3. concepts/build-something-people-want (score: 0.81) + The YC motto. Connected to 12 other brain pages. +``` + +### MCP server (Claude Code, Cursor, Windsurf) + +GBrain exposes 30+ MCP tools via stdio: + +```json +{ + "mcpServers": { + "gbrain": { "command": "gbrain", "args": ["serve"] } + } +} +``` + +Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), or your client's MCP config. + +### Remote MCP (Claude Desktop, Cowork, Perplexity) + +```bash +ngrok http 8787 --url your-brain.ngrok.app +bun run src/commands/auth.ts create "claude-desktop" +claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN" +``` + +Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented). + +## The 25 Skills + +GBrain ships 25 skills organized by `skills/RESOLVER.md`. The resolver tells your agent which skill to read for any task. + +[Skill files are code.](https://x.com/garrytan/status/2042925773300908103) They're the most powerful way to get knowledge work done. A skill file is a fat markdown document that encodes an entire workflow: when to fire, what to check, how to chain with other skills, what quality bar to enforce. The agent reads the skill and executes it. Skills can also call deterministic TypeScript code bundled in GBrain (search, import, embed, sync) for the parts that shouldn't be left to LLM judgment. [Thin harness, fat skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md): the intelligence lives in the skills, not the runtime. + +### Always-on + +| Skill | What it does | +|-------|-------------| +| **signal-detector** | Fires on every message. Spawns a cheap model in parallel to capture original thinking and entity mentions. The brain compounds on autopilot. | +| **brain-ops** | Brain-first lookup before any external API. The read-enrich-write loop that makes every response smarter. | + +### Content ingestion + +| Skill | What it does | +|-------|-------------| +| **ingest** | Thin router. Detects input type and delegates to the right ingestion skill. | +| **idea-ingest** | Links, articles, tweets become brain pages with analysis, author people pages, and cross-linking. | +| **media-ingest** | Video, audio, PDF, books, screenshots, GitHub repos. Transcripts, entity extraction, backlink propagation. | +| **meeting-ingestion** | Transcripts become brain pages. Every attendee gets enriched. Every company gets a timeline entry. | + +### Brain operations + +| Skill | What it does | +|-------|-------------| +| **enrich** | Tiered enrichment (Tier 1/2/3). Creates and updates person/company pages with compiled truth and timelines. | +| **query** | 3-layer search with synthesis and citations. Says "the brain doesn't have info on X" instead of hallucinating. | +| **maintain** | Periodic health: stale pages, orphans, dead links, citation audit, back-link enforcement, tag consistency. | +| **citation-fixer** | Scans pages for missing or malformed citations. Fixes format to match the standard. | +| **repo-architecture** | Where new brain files go. Decision protocol: primary subject determines directory, not format. | +| **publish** | Share brain pages as password-protected HTML. Zero LLM calls. | +| **data-research** | Structured data research with parameterized YAML recipes. Extract investor updates, expenses, company metrics from email. | + +### Operational + +| Skill | What it does | +|-------|-------------| +| **daily-task-manager** | Task lifecycle with priority levels (P0-P3). Stored as searchable brain pages. | +| **daily-task-prep** | Morning prep: calendar lookahead with brain context per attendee, open threads, task review. | +| **cron-scheduler** | Schedule staggering (5-min offsets), quiet hours (timezone-aware with wake-up override), idempotency. | +| **reports** | Timestamped reports with keyword routing. "What's the latest briefing?" finds it instantly. | +| **cross-modal-review** | Quality gate via second model. Refusal routing: if one model refuses, silently switch. | +| **webhook-transforms** | External events (SMS, meetings, social mentions) converted into brain pages with entity extraction. | +| **testing** | Validates every skill has SKILL.md with frontmatter, manifest coverage, resolver coverage. | +| **skill-creator** | Create new skills following the conformance standard. MECE check against existing skills. | + +### Identity and setup + +| Skill | What it does | +|-------|-------------| +| **soul-audit** | 6-phase interview generating SOUL.md (agent identity), USER.md (user profile), ACCESS_POLICY.md (4-tier privacy), HEARTBEAT.md (operational cadence). | +| **setup** | Auto-provision PGLite or Supabase. First import. GStack detection. | +| **migrate** | Universal migration from Obsidian, Notion, Logseq, markdown, CSV, JSON, Roam. | +| **briefing** | Daily briefing with meeting context, active deals, and citation tracking. | + +### Conventions + +Cross-cutting rules in `skills/conventions/`: +- **quality.md** ... citations, back-links, notability gate, source attribution +- **brain-first.md** ... 5-step lookup before any external API call +- **model-routing.md** ... which model for which task +- **test-before-bulk.md** ... test 3-5 items before any batch operation +- **cross-modal.yaml** ... review pairs and refusal routing chain + +## How It Works + +``` +Signal arrives (meeting, email, tweet, link) + -> Signal detector captures ideas + entities (parallel, never blocks) + -> Brain-ops: check the brain first (gbrain search, gbrain get) + -> Respond with full context + -> Write: update brain pages with new information + citations + -> Sync: gbrain indexes changes for next query +``` + +Every cycle adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context. The difference compounds daily. + +The system gets smarter on its own. Entity enrichment auto-escalates: a person mentioned once gets a stub page (Tier 3). After 3 mentions across different sources, they get web + social enrichment (Tier 2). After a meeting or 8+ mentions, full pipeline (Tier 1). The brain learns who matters without being told. Deterministic classifiers improve over time via a fail-improve loop that logs every LLM fallback and generates better regex patterns from the failures. `gbrain doctor` shows the trajectory: "intent classifier: 87% deterministic, up from 40% in week 1." + +> "Prep me for my meeting with Jordan in 30 minutes" +> ... pulls dossier, shared history, recent activity, open threads + +> "What have I said about the relationship between shame and founder performance?" +> ... searches YOUR thinking, not the internet + ## Getting Data In -Once GBrain is installed, your agent needs data flowing in. GBrain ships integration recipes that your agent sets up for you. It reads the recipe, asks for API keys, validates each one, and runs a smoke test. [Markdown is code](docs/ethos/THIN_HARNESS_FAT_SKILLS.md)... the recipe IS the installer. +GBrain ships integration recipes that your agent sets up for you. Each recipe tells the agent what credentials to ask for, how to validate, and what cron to register. | Recipe | Requires | What It Does | |--------|----------|-------------| | [Public Tunnel](recipes/ngrok-tunnel.md) | — | Fixed URL for MCP + voice (ngrok Hobby $8/mo) | -| [Credential Gateway](recipes/credential-gateway.md) | — | Gmail + Calendar access (ClawVisor or Google OAuth) | -| [Voice-to-Brain](recipes/twilio-voice-brain.md) | ngrok-tunnel | Phone calls → brain pages (Twilio + OpenAI Realtime) | -| [Email-to-Brain](recipes/email-to-brain.md) | credential-gateway | Gmail → entity pages (deterministic collector) | -| [X-to-Brain](recipes/x-to-brain.md) | — | Twitter → brain pages (timeline + mentions + deletions) | -| [Calendar-to-Brain](recipes/calendar-to-brain.md) | credential-gateway | Google Calendar → searchable daily pages | -| [Meeting Sync](recipes/meeting-sync.md) | — | Circleback transcripts → brain pages with attendees | +| [Credential Gateway](recipes/credential-gateway.md) | — | Gmail + Calendar access | +| [Voice-to-Brain](recipes/twilio-voice-brain.md) | ngrok-tunnel | Phone calls to brain pages (Twilio + OpenAI Realtime) | +| [Email-to-Brain](recipes/email-to-brain.md) | credential-gateway | Gmail to entity pages | +| [X-to-Brain](recipes/x-to-brain.md) | — | Twitter timeline + mentions + deletions | +| [Calendar-to-Brain](recipes/calendar-to-brain.md) | credential-gateway | Google Calendar to searchable daily pages | +| [Meeting Sync](recipes/meeting-sync.md) | — | Circleback transcripts to brain pages with attendees | -Run `gbrain integrations` to see status. Dependencies resolve automatically. See [Getting Data In](docs/integrations/README.md) for the full guide. +**Data research recipes** extract structured data from email into tracked brain pages. Built-in recipes for investor updates (MRR, ARR, runway, headcount), expense tracking, and company metrics. Create your own with `gbrain research init`. -## The Compounding Thesis +Run `gbrain integrations` to see status. -Most tools help you find things. GBrain makes you smarter over time. +## GBrain + GStack -``` -Signal arrives (meeting, email, tweet, link) - → Agent detects entities (people, companies, ideas) - → READ: check the brain first (gbrain search, gbrain get) - → Respond with full context - → WRITE: update brain pages with new information - → Sync: gbrain indexes changes for next query -``` +[GStack](https://github.com/garrytan/gstack) is the engine. GBrain is the mod. -Every cycle through this loop adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context. You never start from zero. +- **[GStack](https://github.com/garrytan/gstack)** = coding skills (ship, review, QA, investigate, office-hours, retro). 70,000+ stars, 30,000 developers per day. When your agent codes on itself, it uses GStack. +- **GBrain** = everything-else skills (brain ops, signal detection, ingestion, enrichment, cron, reports, identity). When your agent remembers, thinks, and operates, it uses GBrain. +- **`hosts/gbrain.ts`** = the bridge. Tells GStack's coding skills to check the brain before coding. -An agent without this loop answers from stale context. An agent with it gets smarter every conversation. The difference compounds daily. - -> "Who should I invite to dinner who knows both Pedro and Diana?" -> — cross-references the social graph across 3,000+ people pages - -> "What have I said about the relationship between shame and founder performance?" -> — searches YOUR thinking, not the internet - -> "Prep me for my meeting with Jordan in 30 minutes" -> — pulls dossier, shared history, recent activity, open threads - -## Voice: "Her" Out of the Box - -The voice integration is the strongest demonstration of why a personal brain matters. -Call a phone number. Your AI answers. It knows who's calling, pulls their full context -from thousands of people pages, references your last meeting, and responds like someone -who actually knows your world. When the call ends, a structured brain page appears with -the transcript, entity detection, and cross-references. - -This isn't a demo. It runs on a real phone number, screens unknown callers, and gets -smarter with every call. Your agent picks its own name and personality. WebRTC works in -a browser tab with zero setup. A real phone number is optional. - -

- Voice client connected -

- -> [See it in action](https://x.com/garrytan/status/2043022208512172263) - -The voice recipe ships with GBrain: [Voice-to-Brain](recipes/twilio-voice-brain.md). -Your agent installs it, sets up the voice server, and you have a working AI phone line -in 30 minutes. 25 production patterns from a real deployment included. - -## How this happened - -I was setting up my [OpenClaw](https://openclaw.ai) agent and started a markdown brain repo. One page per person, one page per company, compiled truth on top, append-only timeline on the bottom. The agent got smarter the more it knew, so I kept feeding it. Within a week I had 10,000+ markdown files, 3,000+ people with compiled dossiers, 13 years of calendar data, 280+ meeting transcripts, and 300+ captured original ideas. - -The agent runs while I sleep. The dream cycle scans every conversation, enriches missing entities, fixes broken citations, and consolidates memory. I wake up and the brain is smarter than when I went to sleep. See the [cron schedule guide](docs/guides/cron-schedule.md) for setup. - -**PGLite runs locally by default.** `gbrain init` gives you embedded Postgres with pgvector, hybrid search, and all 37 operations. No server, no subscription. When your brain outgrows local (1000+ files, multi-device access, remote MCP), `gbrain migrate --to supabase` moves everything to managed Postgres. +`gbrain init` detects if GStack is installed and reports mod status. If GStack isn't there, it tells you how to get it. ## Architecture @@ -170,275 +194,22 @@ The agent runs while I sleep. The dream cycle scans every conversation, enriches │ Brain Repo │ │ GBrain │ │ AI Agent │ │ (git) │ │ (retrieval) │ │ (read/write) │ │ │ │ │ │ │ -│ markdown files │───>│ Postgres + │<──>│ skills define │ -│ = source of │ │ pgvector │ │ HOW to use the │ -│ truth │ │ │ │ brain │ +│ markdown files │───>│ Postgres + │<──>│ 25 skills │ +│ = source of │ │ pgvector │ │ define HOW to │ +│ truth │ │ │ │ use the brain │ │ │<───│ hybrid │ │ │ -│ human can │ │ search │ │ entity detect │ -│ always read │ │ (vector + │ │ enrich │ -│ & edit │ │ keyword + │ │ ingest │ -│ │ │ RRF) │ │ brief │ +│ human can │ │ search │ │ RESOLVER.md │ +│ always read │ │ (vector + │ │ routes intent │ +│ & edit │ │ keyword + │ │ to skill │ +│ │ │ RRF) │ │ │ └──────────────────┘ └───────────────┘ └──────────────────┘ ``` -The repo is the system of record. GBrain is the retrieval layer. The agent reads and writes through both. Human always wins — you can edit any markdown file directly and `gbrain sync` picks up the changes. +The repo is the system of record. GBrain is the retrieval layer. The agent reads and writes through both. Human always wins... edit any markdown file and `gbrain sync` picks up the changes. -## What a Production Agent Looks Like +## The Knowledge Model -The numbers above aren't theoretical. They come from a real deployment documented in [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) — a reference architecture for how a production AI agent uses gbrain as its knowledge backbone. - -**Read the skillpack.** It's the most important doc in this repo. It tells your agent HOW to use gbrain, not just what commands exist: - -- **The brain-agent loop** — the read-write cycle that makes knowledge compound -- **Entity detection** — spawn on every message, capture people/companies/original ideas -- **Enrichment pipeline** — 7-step protocol with tiered API spend -- **Meeting ingestion** — transcript to brain pages with entity propagation -- **Source attribution** — every fact traceable to where it came from -- **Reference cron schedule** — 20+ recurring jobs that keep the brain alive - -Without the skillpack, your agent has tools but no playbook. With it, the agent knows when to read, when to write, how to enrich, and how to keep the brain alive autonomously. It's a pattern book, not a tutorial. "Here's what works, here's why." - -## How gbrain fits with OpenClaw/Hermes - -GBrain is world knowledge — people, companies, deals, meetings, concepts, your original thinking. It's the long-term memory of what you know about the world. - -[OpenClaw](https://openclaw.ai) agent memory (`memory_search`) is operational state — preferences, decisions, session context, how the agent should behave. - -They're complementary: - -| Layer | What it stores | How to query | -|-------|---------------|-------------| -| **gbrain** | People, companies, meetings, ideas, media | `gbrain search`, `gbrain query`, `gbrain get` | -| **Agent memory** | Preferences, decisions, operational config | `memory_search` | -| **Session context** | Current conversation | (automatic) | - -All three should be checked. GBrain for facts about the world. Memory for agent config. Session for immediate context. Install via `openclaw skills install gbrain`. - -## The compounding effect - -The real value isn't search. It's what happens after a few weeks of use. - -You take a meeting with someone. The agent writes a brain page for them, links it to their company, tags it with the deal. Next week someone mentions that company in a different context. The agent already has the full picture: who you talked to, what you discussed, what threads are open. You didn't do anything. The brain already had it. - -## Install - -### Prerequisites - -**Zero-config start (PGLite).** `gbrain init` creates a local embedded Postgres brain. No accounts, no server, no API keys. Keyword search works immediately. Add API keys later for vector search and LLM-powered features. - -**For production scale (Supabase).** When your brain outgrows local, `gbrain migrate --to supabase` moves everything to managed Postgres: - -| Dependency | What it's for | How to get it | -|------------|--------------|---------------| -| **Supabase account** | Postgres + pgvector database | [supabase.com](https://supabase.com) (Pro tier, $25/mo for 8GB) | -| **OpenAI API key** | Embeddings (text-embedding-3-large) | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) | -| **Anthropic API key** | Multi-query expansion + LLM chunking (Haiku) | [console.anthropic.com](https://console.anthropic.com) | - -Set the API keys as environment variables: - -```bash -export OPENAI_API_KEY=sk-... -export ANTHROPIC_API_KEY=sk-ant-... -``` - -The Supabase connection URL is configured during `gbrain init --supabase`. The OpenAI and Anthropic SDKs read their keys from the environment automatically. - -Without an OpenAI key, search still works (keyword only, no vector search). Without an Anthropic key, search still works (no multi-query expansion, no LLM chunking). - -### GBrain without OpenClaw - -GBrain works with any AI agent, any MCP client, or no agent at all. Three paths: - -#### Standalone CLI - -Install globally and use gbrain from the terminal: - -```bash -bun add -g github:garrytan/gbrain -gbrain init # PGLite (local, no server needed) -gbrain import ~/git/brain/ # index your markdown -gbrain query "what themes show up across my notes?" -``` - -Run `gbrain --help` for the full list of commands. - -#### MCP server (Claude Code, Cursor, Windsurf, etc.) - -GBrain exposes 30 MCP tools via stdio. Add this to your MCP client config: - -**Claude Code** (`~/.claude/server.json`): -```json -{ - "mcpServers": { - "gbrain": { - "command": "gbrain", - "args": ["serve"] - } - } -} -``` - -**Cursor** (Settings > MCP Servers): -```json -{ - "gbrain": { - "command": "gbrain", - "args": ["serve"] - } -} -``` - -This gives your agent `get_page`, `put_page`, `search`, `query`, `add_link`, `traverse_graph`, `sync_brain`, `file_upload`, and 22 more tools. All generated from the same operation definitions as the CLI. - -#### Remote MCP Server (Claude Desktop, Cowork, Perplexity) - -Access your brain from any device, any AI client. Run `gbrain serve` behind an HTTP -server with a public tunnel: - -```bash -# Set up a public tunnel (see recipes/ngrok-tunnel.md) -ngrok http 8787 --url your-brain.ngrok.app - -# Create a bearer token for your client -bun run src/commands/auth.ts create "claude-desktop" -``` - -Then add to your AI client: -- **Claude Code:** `claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN"` -- **Claude Desktop:** Settings > Integrations > Add (NOT JSON config, [details](docs/mcp/CLAUDE_DESKTOP.md)) -- **Perplexity:** Settings > Connectors > Add remote MCP ([details](docs/mcp/PERPLEXITY.md)) - -Per-client setup guides: [`docs/mcp/`](docs/mcp/DEPLOY.md) - -ChatGPT support requires OAuth 2.1 (not yet implemented). Self-hosted alternatives (Tailscale Funnel, ngrok) documented in [`docs/mcp/ALTERNATIVES.md`](docs/mcp/ALTERNATIVES.md). - -**The tools are not enough.** Your agent also needs the playbook: read [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) and paste the relevant sections into your agent's system prompt or project instructions. The skillpack tells the agent WHEN and HOW to use each tool: read before responding, write after learning, detect entities on every message, back-link everything. - -The skill markdown files in `skills/` are standalone instruction sets. Copy them into your agent's context: - -| Skill file | What the agent learns | -|------------|----------------------| -| `skills/ingest/SKILL.md` | How to import meetings, docs, articles | -| `skills/query/SKILL.md` | 3-layer search with synthesis and citations | -| `skills/maintain/SKILL.md` | Periodic health: stale pages, orphans, dead links | -| `skills/enrich/SKILL.md` | Enrich pages from external APIs | -| `skills/briefing/SKILL.md` | Daily briefing with meeting prep | -| `skills/migrate/SKILL.md` | Migrate from Obsidian, Notion, Logseq, etc. | - -#### As a TypeScript library - -```bash -bun add github:garrytan/gbrain -``` - -```typescript -import { createEngine } from 'gbrain'; - -// PGLite (local, no server) -const engine = createEngine('pglite'); -await engine.connect({ database_path: '~/.gbrain/brain.pglite' }); -await engine.initSchema(); - -// Or Postgres (Supabase / self-hosted) -// const engine = createEngine('postgres'); -// await engine.connect({ database_url: process.env.DATABASE_URL }); -// await engine.initSchema(); - -// Search -const results = await engine.searchKeyword('startup growth'); - -// Read -const page = await engine.getPage('people/pedro-franceschi'); - -// Write -await engine.putPage('concepts/superlinear-returns', { - type: 'concept', - title: 'Superlinear Returns', - compiled_truth: 'Paul Graham argues that returns in many fields are superlinear...', - timeline: '- 2023-10-01: Published on paulgraham.com', -}); -``` - -The `BrainEngine` interface is pluggable. `createEngine()` accepts `'pglite'` or `'postgres'`. See `docs/ENGINES.md` for details. - -PGLite (default) requires no external database. For production scale (7K+ pages, multi-device, remote MCP), use Supabase Pro ($25/mo). - -## Upgrade - -```bash -cd ~/gbrain && git pull origin main && bun install -``` - -Then run `gbrain init` to apply any schema migrations (idempotent, safe to re-run). - -## Setup details - -`gbrain init` defaults to PGLite (embedded Postgres 17.5 via WASM). No accounts, no server. Config saved to `~/.gbrain/config.json`. - -```bash -gbrain init # PGLite (default) -gbrain init --supabase # guided wizard for Supabase -gbrain init --url # any Postgres with pgvector -``` - -Import is idempotent. Re-running skips unchanged files (SHA-256 content hash). ~30s for text import of 7,000 files, ~10-15 min for embedding. - -## File storage and migration - -Brain repos accumulate binary files: images, PDFs, audio recordings, raw API responses. A repo with 3,000 markdown pages might have 2GB of binaries making `git clone` painful. - -GBrain has a three-stage migration lifecycle that moves binaries to cloud storage while preserving every reference: - -``` -Local files in git repo - │ - ▼ gbrain files mirror -Cloud copy exists, local files untouched - │ - ▼ gbrain files redirect -Local files replaced with .redirect breadcrumbs (tiny YAML pointers) - │ - ▼ gbrain files clean -Breadcrumbs removed, cloud is the only copy -``` - -Every stage is reversible until `clean`: - -```bash -# Stage 1: Copy to cloud (git repo unchanged) -gbrain files mirror ~/git/brain/attachments/ --dry-run # preview first -gbrain files mirror ~/git/brain/attachments/ - -# Stage 2: Replace local files with breadcrumbs -gbrain files redirect ~/git/brain/attachments/ --dry-run -gbrain files redirect ~/git/brain/attachments/ -# Your git repo just dropped from 2GB to 50MB - -# Undo: download everything back from cloud -gbrain files restore ~/git/brain/attachments/ - -# Stage 3: Remove breadcrumbs (irreversible, cloud is the only copy) -gbrain files clean ~/git/brain/attachments/ --yes -``` - -**Storage backends:** S3-compatible (AWS S3, Cloudflare R2, MinIO), Supabase Storage, or local filesystem. Configured during `gbrain init`. - -Additional file commands: - -```bash -gbrain files list [slug] # list files for a page (or all) -gbrain files upload --page # upload file linked to page -gbrain files sync # bulk upload directory -gbrain files verify # verify all uploads match local -gbrain files status # show migration status of directories -gbrain files unmirror # remove mirror marker (files stay in cloud) -``` - -The file resolver (`src/core/file-resolver.ts`) handles fallback automatically: if a local file is missing, it checks for a `.redirect` breadcrumb, then a `.supabase` marker, and resolves to the cloud URL. Code that references files by path keeps working after migration. - -## The knowledge model - -Every page in the brain follows the compiled truth + timeline pattern: +Every page follows the compiled truth + timeline pattern: ```markdown --- @@ -448,10 +219,6 @@ tags: [startups, growth, pg-essay] --- Paul Graham's argument that startups should do unscalable things early on. -The most common: recruiting users manually, one at a time. Airbnb went -door to door in New York photographing apartments. Stripe manually -installed their payment integration for early users. - The key insight: the unscalable effort teaches you what users actually want, which you can't learn any other way. @@ -459,192 +226,38 @@ want, which you can't learn any other way. - 2013-07-01: Published on paulgraham.com - 2024-11-15: Referenced in batch W25 kickoff talk -- 2025-02-20: Cited in discussion about AI agent onboarding strategies ``` -Above the `---` separator: **compiled truth**. Your current best understanding. Gets rewritten when new evidence changes the picture. Below: **timeline**. Append-only evidence trail. Never edited, only added to. +Above the `---`: **compiled truth**. Your current best understanding. Gets rewritten when new evidence changes the picture. Below: **timeline**. Append-only evidence trail. Never edited, only added to. -The compiled truth is the answer. The timeline is the proof. +## Search -## How search works +Hybrid search: vector + keyword + RRF fusion + multi-query expansion + 4-layer dedup. ``` -Query: "when should you ignore conventional wisdom?" - | - Intent classifier (zero-latency, no LLM) - → entity? temporal? event? general? - → auto-selects detail level - | - Multi-query expansion (Claude Haiku) - "contrarian thinking startups", "going against the crowd" - | - +----+----+ - | | - Vector Keyword - (HNSW (tsvector + - cosine) ts_rank) - | | - +----+----+ - | - RRF Fusion: score = sum(1/(60 + rank)) - → normalize to 0-1 - → 2x compiled truth boost (entity queries) - | - Cosine re-scoring (0.7 * RRF + 0.3 * cosine) - → query-specific chunk ranking - | - 4-Layer Dedup + compiled truth guarantee - 1. Top 3 chunks per page - 2. Text similarity > 0.85 - 3. Type diversity (60% cap) - 4. Per-page chunk cap (2) - 5. Guarantee compiled truth per page - | - Results +Query + -> Intent classifier (entity? temporal? event? general?) + -> Multi-query expansion (Claude Haiku) + -> Vector search (HNSW cosine) + Keyword search (tsvector) + -> RRF fusion: score = sum(1/(60 + rank)) + -> Cosine re-scoring + compiled truth boost + -> 4-layer dedup + compiled truth guarantee + -> Results ``` -Keyword search alone misses conceptual matches. "Ignore conventional wisdom" won't find an essay titled "The Bus Ticket Theory of Genius" even though it's exactly about that. Vector search alone misses exact phrases when the embedding is diluted by surrounding text. RRF fusion gets both right. Multi-query expansion catches phrasings you didn't think of. +Keyword alone misses conceptual matches. Vector alone misses exact phrases. RRF gets both. Search quality is benchmarked and reproducible: `gbrain eval --qrels queries.json` measures P@k, Recall@k, MRR, and nDCG@k. A/B test config changes before deploying them. -The query intent classifier reads your query and picks the right search mode. "Who is Alice?" surfaces compiled truth assessments. "When did we last meet Alice?" surfaces timeline entries with dates. No LLM call, just pattern matching. Use `--detail low/medium/high` to override. +## Voice -Search quality is benchmarked: 29 fictional pages, 20 queries, graded relevance. Run `bun run test/benchmark-search-quality.ts` to reproduce. Measure changes with `gbrain eval --qrels queries.json`. +Call a phone number. Your AI answers. It knows who's calling, pulls their full context from the brain, and responds like someone who actually knows your world. When the call ends, a brain page appears with the transcript, entity detection, and cross-references. -## Database schema +

+ Voice client connected +

-10 tables in Postgres + pgvector: +> [See it in action](https://x.com/garrytan/status/2043022208512172263) -``` -pages The core content table - slug (UNIQUE) e.g. "concepts/do-things-that-dont-scale" - type person, company, deal, yc, civic, project, concept, source, media - title, compiled_truth, timeline - frontmatter (JSONB) Arbitrary metadata - search_vector Trigger-based tsvector (title + compiled_truth + timeline + timeline_entries) - content_hash SHA-256 for import idempotency - -content_chunks Chunked content with embeddings - page_id (FK) Links to pages - chunk_text The chunk content - chunk_source 'compiled_truth' or 'timeline' - embedding (vector) 1536-dim from text-embedding-3-large - HNSW index Cosine similarity search - -links Cross-references between pages - from_page_id, to_page_id - link_type knows, invested_in, works_at, founded, references, etc. - -tags page_id + tag (many-to-many) - -timeline_entries Structured timeline events - page_id, date, source, summary, detail (markdown) - -page_versions Snapshot history for compiled_truth - compiled_truth, frontmatter, snapshot_at - -raw_data Sidecar JSON from external APIs - page_id, source, data (JSONB) - -files Binary attachments in Supabase Storage - page_slug (FK) Links to pages (ON UPDATE CASCADE) - storage_path, content_hash, mime_type, metadata (JSONB) - -ingest_log Audit trail of import/ingest operations - -config Brain-level settings (embedding model, chunk strategy, sync state) -``` - -Indexes: B-tree on slug/type, GIN on frontmatter/search_vector, HNSW on embeddings, pg_trgm on title for fuzzy slug resolution. - -## Chunking - -Three strategies, dispatched by content type: - -**Recursive** (timeline, bulk import): 5-level delimiter hierarchy (paragraphs, lines, sentences, clauses, words). 300-word chunks with 50-word sentence-aware overlap. Fast, predictable, lossless. - -**Semantic** (compiled truth): Embeds each sentence, computes adjacent cosine similarities, applies Savitzky-Golay smoothing to find topic boundaries. Falls back to recursive on failure. Best quality for intelligence assessments. - -**LLM-guided** (high-value content, on request): Pre-splits into 128-word candidates, asks Claude Haiku to identify topic shifts in sliding windows. 3 retries per window. Most expensive, best results. - -## Commands - -``` -SETUP - gbrain init [--supabase|--url ] Create brain (PGLite default, or Supabase) - gbrain migrate --to supabase|pglite Migrate between engines (bidirectional) - gbrain upgrade Self-update - -PAGES - gbrain get Read a page (supports fuzzy slug matching) - gbrain put [< file.md] Write/update a page (auto-versions) - gbrain delete Delete a page - gbrain list [--type T] [--tag T] [-n N] List pages with filters - -SEARCH - gbrain search Keyword search (tsvector) - gbrain query Hybrid search (vector + keyword + RRF + expansion) - -IMPORT/EXPORT - gbrain import [--no-embed] Import markdown directory (idempotent) - gbrain sync [--repo ] [flags] Git-to-brain incremental sync - gbrain export [--dir ./out/] Export to markdown (round-trip) - -FILES - gbrain files list [slug] List stored files - gbrain files upload --page Upload file to storage - gbrain files sync Bulk upload directory - gbrain files verify Verify all uploads - -EMBEDDINGS - gbrain embed [|--all|--stale] Generate/refresh embeddings - -LINKS + GRAPH - gbrain link [--type T] Create typed link - gbrain unlink Remove link - gbrain backlinks Incoming links - gbrain graph [--depth N] Traverse link graph (recursive CTE, default depth 5) - -TAGS - gbrain tags List tags - gbrain tag Add tag - gbrain untag Remove tag - -TIMELINE - gbrain timeline [] View timeline entries - gbrain timeline-add Add timeline entry - -ADMIN - gbrain doctor [--json] Health checks (pgvector, RLS, schema, embeddings) - gbrain stats Brain statistics - gbrain health Health dashboard (embed coverage, stale, orphans) - gbrain history Page version history - gbrain revert Revert to previous version - gbrain config [get|set] [value] Brain config - gbrain serve MCP server (stdio, local) - gbrain upgrade Self-update with feature discovery - bun run src/commands/auth.ts Token management (create/list/revoke/test) - gbrain call '' Raw tool invocation - gbrain --tools-json Tool discovery (JSON) -``` - -## Library and MCP details - -See [GBrain without OpenClaw](#gbrain-without-openclaw) above for library usage examples, MCP server config, and skill file loading. - -The `BrainEngine` interface is pluggable. See `docs/ENGINES.md` for how to add backends. 30 MCP tools are generated from the contract-first `operations.ts`. Parity tests verify structural identity between CLI, MCP, and tools-json. - -## Skills - -Fat markdown files that tell AI agents HOW to use gbrain. No skill logic in the binary. - -| Skill | What it does | -|-------|-------------| -| **ingest** | Ingest meetings, docs, articles. Updates compiled truth (rewrite, not append), appends timeline, creates cross-reference links across all mentioned entities. | -| **query** | 3-layer search (keyword + vector + structured) with synthesis and citations. Says "the brain doesn't have info on X" rather than hallucinating. | -| **maintain** | Periodic health: find contradictions, stale compiled truth, orphan pages, dead links, tag inconsistency, missing embeddings, overdue threads. | -| **enrich** | Enrich pages from external APIs. Raw data stored separately, distilled highlights go to compiled truth. | -| **briefing** | Daily briefing: today's meetings with participant context, active deals with deadlines, time-sensitive threads, recent changes. | -| **migrate** | Universal migration from Obsidian (wikilinks to gbrain links), Notion (stripped UUIDs), Logseq (block refs), plain markdown, CSV, JSON, Roam. | -| **setup** | Set up GBrain from scratch: auto-provision Supabase via CLI, AGENTS.md injection, import, sync. Target TTHW < 2 min. | +The voice recipe ships with GBrain: [Voice-to-Brain](recipes/twilio-voice-brain.md). WebRTC works in a browser tab with zero setup. A real phone number is optional. ## Engine Architecture @@ -652,76 +265,114 @@ Fat markdown files that tell AI agents HOW to use gbrain. No skill logic in the CLI / MCP Server (thin wrappers, identical operations) | - BrainEngine interface - (pluggable backend) - | - engine-factory.ts - (dynamic imports) + BrainEngine interface (pluggable) | +--------+--------+ | | PGLiteEngine PostgresEngine - (ships v0.7) (ships v0) + (default) (Supabase) | | -~/.gbrain/brain.pglite Supabase Pro ($25/mo) - embedded PG 17.5 Postgres + pgvector + pg_trgm - via @electric-sql connection pooling via Supavisor - /pglite +~/.gbrain/ Supabase Pro ($25/mo) +brain.pglite Postgres + pgvector +embedded PG 17.5 - gbrain migrate --to supabase/pglite + gbrain migrate --to supabase|pglite (bidirectional migration) ``` -Embedding, chunking, and search fusion are engine-agnostic. Only raw keyword search (`searchKeyword`) and raw vector search (`searchVector`) are engine-specific. RRF fusion, multi-query expansion, and 4-layer dedup run above the engine on `SearchResult[]` arrays. Both engines use the same SQL (PGLite runs real Postgres, not a separate dialect). +PGLite: embedded Postgres, no server, zero config. When your brain outgrows local (1000+ files, multi-device), `gbrain migrate --to supabase` moves everything. -## Storage estimates +## File Storage -For a brain with ~7,500 pages: +Brain repos accumulate binaries. GBrain has a three-stage migration: -| Component | Size | -|-----------|------| -| Page text (compiled_truth + timeline) | ~150MB | -| JSONB frontmatter + indexes | ~70MB | -| Content chunks (~22K, text) | ~80MB | -| Embeddings (22K x 1536 floats) | ~134MB | -| HNSW index overhead | ~270MB | -| Links, tags, timeline, versions | ~50MB | -| **Total** | **~750MB** | +```bash +gbrain files mirror # copy to cloud, local untouched +gbrain files redirect # replace local with .redirect pointers +gbrain files clean # remove pointers, cloud only +gbrain files restore # download everything back (undo) +``` -Supabase free tier (500MB) won't fit a large brain. Supabase Pro ($25/mo, 8GB) is the starting point. +Storage backends: S3-compatible (AWS, R2, MinIO), Supabase Storage, or local. -Initial embedding cost: ~$4-5 for 7,500 pages via OpenAI text-embedding-3-large. +## Commands + +``` +SETUP + gbrain init [--supabase|--url] Create brain (PGLite default) + gbrain migrate --to supabase|pglite Bidirectional engine migration + gbrain upgrade Self-update with feature discovery + +PAGES + gbrain get Read a page (fuzzy slug matching) + gbrain put [< file.md] Write/update (auto-versions) + gbrain delete Delete a page + gbrain list [--type T] [--tag T] List with filters + +SEARCH + gbrain search Keyword search (tsvector) + gbrain query Hybrid search (vector + keyword + RRF) + +IMPORT + gbrain import [--no-embed] Import markdown (idempotent) + gbrain sync [--repo ] Git-to-brain incremental sync + gbrain export [--dir ./out/] Export to markdown + +FILES + gbrain files list|upload|sync|verify File storage operations + +EMBEDDINGS + gbrain embed [|--all|--stale] Generate/refresh embeddings + +LINKS + GRAPH + gbrain link|unlink|backlinks|graph Cross-reference management + +ADMIN + gbrain doctor [--json] [--fast] Health checks (resolver, skills, DB, embeddings) + gbrain doctor --fix Auto-fix resolver issues + gbrain stats Brain statistics + gbrain serve MCP server (stdio) + gbrain integrations Integration recipe dashboard + gbrain check-backlinks check|fix Back-link enforcement + gbrain lint [--fix] LLM artifact detection + gbrain transcribe