feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39)

* docs: break SKILLPACK into 17 individual guides

The 1,281-line SKILLPACK monolith is now 17 individually linkable guides
in docs/guides/, organized by category: core patterns, data pipelines,
operations, search, and administration.

GBRAIN_SKILLPACK.md becomes a structured index with categorized tables
linking to each guide. The URL stays stable for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add integration guides, architecture docs, and ethos

New documentation directories:
- docs/integrations/ — "Getting Data In" landing page, credential gateway,
  meeting webhooks. Includes recipe format documentation.
- docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search)
- docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide
- docs/designs/ — "Homebrew for Personal AI" 10-star vision document

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add gbrain integrations command + voice-to-brain recipe

New CLI command: gbrain integrations (list/show/status/doctor/stats/test)
- Standalone command, no database connection needed
- Uses gray-matter directly for recipe parsing (not parseMarkdown)
- --json flag on every subcommand for agent-parseable output
- Bare command shows senses/reflexes dashboard
- Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl

First recipe: recipes/twilio-voice-brain.md
- Phone calls create brain pages via Twilio + OpenAI Realtime
- Opinionated defaults: caller screening, brain-first lookup, quiet hours
- Outbound call smoke test (GBrain calls the user to prove it works)
- Validate-as-you-go credential testing
- Twilio signature validation for webhook security

Migration file for v0.7.0 with agent-readable changelog.
13 unit tests covering parseRecipe, CLI routing, and recipe validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Getting Data In to README, update CLAUDE.md and manifest

README: voice calls in intro bullet list, new "Getting Data In" section
with integration table (voice, email, X, calendar) and recipe philosophy.

CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/,
docs/integrations/, docs/architecture/, docs/ethos/).

manifest.json: bump to v0.7.0, add recipes_dir field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: v0.7.0 CHANGELOG, TODOS, VERSION bump

CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain,
gbrain integrations command, SKILLPACK breakout, and new documentation.

TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL,
community recipe submission, always-on deployment recipes).

VERSION + package.json: bump to 0.7.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite voice recipe with agent instructions and verified links

Major improvements to recipes/twilio-voice-brain.md:

- Agent preamble: explains WHY sequential execution matters (each step
  depends on the previous), defines 4 stop points where the agent MUST
  pause and verify, tells agent to never say "something went wrong"
  but instead explain the exact error and fix

- User actions are now specific: exact URLs for every credential
  (Twilio console, OpenAI API keys page, ngrok dashboard), what
  buttons to click, what fields to copy, common failure modes

- All URLs verified via web search against current 2026 documentation:
  Twilio SID/token at twilio.com/console, OpenAI keys at
  platform.openai.com/api-keys, ngrok token at
  dashboard.ngrok.com/get-started/your-authtoken

- Cost estimate corrected: OpenAI Realtime is $0.06/min input +
  $0.24/min output (was understated), total ~$20-22/mo for 100 min

- Validate-as-you-go: each credential tested immediately with exact
  curl commands, failure messages explain what went wrong and how to fix

- Smoke test flow: tells user exactly what to say, verifies ALL
  three outputs (messaging notification + brain page + search result)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add "Homebrew for Personal AI" essay (markdown is code)

New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution
corollary to "Thin Harness, Fat Skills." Argues that markdown skill files
are simultaneously documentation, specification, package, and source code.
The agent is the package manager. The git repo is the app store.

Referenced from SKILLPACK index and CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite agent instructions as command language, promote skills

The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide.
Every step is an imperative command with exact verification criteria and
explicit stop-on-failure behavior. No FYI, no suggestions, just rails.

Key changes:
- 11-step setup with STOP points after each step
- Exact user instructions for Supabase connection string (what to click,
  what NOT to give the agent, what the string looks like)
- "Verify: run X. You must see Y. If not: Z" after every step
- Skills table now links to both skill files AND guide docs
- Integration recipes table simplified (no "coming soon" placeholders)
- Docs section reorganized: for agents / for humans / reference

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 4 codex findings + add email-to-brain recipe

Codex review found 4 issues, all fixed:

1. getStatus() returned "configured" if ANY secret was set (e.g. just
   OPENAI_API_KEY). Now requires ALL required secrets before marking
   configured. Prevents false "configured" status and spurious doctor runs.

2. Twilio health check hit unauthenticated endpoint (always 401). Now
   uses authenticated curl with SID:token, matching the setup validation.

3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after
   SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md.

4. Compiled binary can't find recipes/ via import.meta.dir. Added
   GBRAIN_RECIPES_DIR env var override + global bun install path fallback.

Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern
with ClawVisor credential gateway, validate-as-you-go, agent instructions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add email, X, calendar, and meeting sync recipes

Four new integration recipes extracted from production wintermute patterns:

- recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector
  pattern (code pulls emails with baked-in links, agent does judgment),
  noise filtering, signature detection, digest generation

- recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search,
  deletion detection (diffs previous run, verifies 404), engagement
  velocity tracking, rate limit awareness

- recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical
  backfill (years of data), daily markdown files with attendees + locations,
  attendee enrichment for brain pages

- recipes/meeting-sync.md: Circleback API, transcript import with speaker
  labels, attendee detection + filtering, entity propagation to people/
  company pages, action item extraction, idempotent by source_id

All recipes follow the same format: agent preamble with sequential execution
rules, validate-as-you-go credentials, exact URLs for API key setup,
stop-on-failure verification, and heartbeat logging.

Updated README, SKILLPACK index, and integrations landing page with all 5 recipes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes

Both recipes now offer two auth options:
- Option A: ClawVisor (recommended, handles OAuth + token refresh)
- Option B: Google OAuth2 directly (no extra service, you manage tokens)

Option B includes step-by-step instructions for Google Cloud Console:
exact URLs, which buttons to click, which scopes to add, how to enable
the API, and the OAuth flow for token exchange.

This removes ClawVisor as a hard dependency for getting started.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add implementation guides with pseudocode and test suggestions

Every recipe now includes an "Implementation Guide" section with:

- Production-tested pseudocode the agent can follow to build each collector
- Edge cases and failure modes discovered in real deployment
- Non-obvious implementation details (why the 48h staleness heuristic,
  why Gmail links need authuser, why SSE responses need double-parsing)
- Test suggestions: what the agent should verify after setup

email-to-brain: noise filtering algorithm, signature detection patterns,
  Gmail link generation (authuser is critical), sent-mail dedup

x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness,
  API verification), engagement velocity thresholds (50 min for 2x, 100
  absolute jump), atomic writes, stdout contract, rate limit handling

calendar-to-brain: smart chunking (monthly for sparse years, weekly for
  dense), attendee filtering (rooms, groups, distros), merge-with-existing
  (only replace ## Calendar section), date/time parsing edge cases

meeting-sync: SSE double-JSON parsing, idempotency double-check (grep +
  filename), auto-tagging from meeting names, git commit after sync

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: 6 new guides from production patterns (wintermute extraction)

New guides extracted and generalized from production deployment:

- repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge).
  Strict boundary rules, decision tree, hard rule: never write knowledge
  to the agent repo.

- sub-agent-routing.md: Model routing table by task type. Signal detector
  pattern (spawn Sonnet on every message). Research pipeline pattern
  (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization.

- skill-development.md: 5-step cycle (concept, prototype, evaluate, codify,
  cron). MECE discipline (no overlapping skills). Quality bar checklist.
  "If you ask twice, it should already be a skill."

- idea-capture.md: Originality distribution rating (0-100 across 4
  populations). Depth test ("could someone unfamiliar understand WHY?").
  Deep cross-linking mandate. Notability filtering.

- quiet-hours.md: Hold notifications 11pm-8am local time. Held messages
  directory pattern. Timezone-aware delivery. Morning briefing pickup.

- diligence-ingestion.md: 9-step pipeline for data room materials. Detection
  patterns (PDF filenames, spreadsheet tabs, user language). Index.md
  template with bull/bear case. Company page enrichment.

All PII scrubbed. Patterns generalized for any user.
SKILLPACK index updated with 6 new entries. CLAUDE.md references added.
All 37 SKILLPACK links verified.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: upgrade all guides to operational playbooks with pseudocode

Every guide now follows the playbook structure:
- Goal: one sentence, what this achieves
- What the User Gets: without this / with this
- Implementation: pseudocode with actual gbrain commands
- Tricky Spots: production-tested gotchas
- How to Verify: test steps the agent runs after setup

Guides upgraded (15 files):
- brain-agent-loop: on_message() loop with read/write/sync pseudocode
- brain-first-lookup: 4-step lookup cascade with exact commands
- brain-vs-memory: routing algorithm for 3 knowledge layers
- compiled-truth: page structure + rewrite vs append rules
- content-media: 3 ingest patterns (YouTube, social, PDFs)
- cron-schedule: full schedule table + dream cycle pseudocode
- enrichment-pipeline: 7-step protocol with tier classification
- entity-detection: spawn pattern + detection prompt + notability filter
- executive-assistant: 3 workflow algorithms (triage, prep, post-inbox)
- meeting-ingestion: 6-step transcript-to-brain flow
- operational-disciplines: 5 executable discipline blocks
- originals-folder: detection + exact-phrasing capture + cross-linking
- search-modes: decision tree for keyword vs hybrid vs direct
- source-attribution: citation format + hierarchy + conflict resolution
- Plus Goal/What User Gets headers on 6 newer guides

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add WebRTC to voice recipe + ngrok Hobby setup guide

Voice recipe updates:
- Added WebRTC endpoint (POST /session, GET /call, POST /tool) for
  browser-based calling with RNNoise noise suppression
- WebRTC pseudocode with the 4 non-obvious gotchas from production
  (voice under audio.output.voice, no turn_detection, no session.update
  on connect, trigger greeting via data channel)
- Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier
- Fixed domain means URLs never change, Twilio never breaks

New guide: docs/mcp/NGROK_SETUP.md
- How to set up ngrok Hobby for both MCP and voice agent
- Fixed domain setup, watchdog pattern, AI client configuration
- Claude Desktop requires Settings > Integrations (not JSON config)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add dependency graph + ngrok-tunnel + credential-gateway recipes

Recipes now have real dependencies via the `requires` field:
- voice-to-brain requires ngrok-tunnel (needs public URL for Twilio)
- email-to-brain requires credential-gateway (needs Gmail access)
- calendar-to-brain requires credential-gateway (needs Calendar access)
- x-to-brain and meeting-sync are standalone (direct API keys)

Two new infrastructure recipes:
- ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby
  ($8/mo) for a domain that never changes. Includes watchdog pattern.
- credential-gateway: secure Google service access via ClawVisor
  (recommended) or direct OAuth2. One setup, all Google recipes use it.

Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure,
not MCP-specific.

README and integrations landing page show dependency chains.
When agent installs voice-to-brain, it sets up ngrok-tunnel first.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add infra category, fix dashboard alignment, show dependencies

DX audit found two bugs in gbrain integrations dashboard:

1. Column alignment broken — IDs > 18 chars ran into descriptions
   with no space. Fixed: pad to 22 chars.

2. ngrok-tunnel and credential-gateway showed as SENSES but they're
   infrastructure. Added 'infra' category. Dashboard now shows three
   sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES.

3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)"

Also added 'requires' field to JSON output for agent consumption.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add frontier model requirement disclaimer to README

GBrain's markdown-is-code approach requires models capable of
interpreting intent and implementing from architecture descriptions.
Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models
will struggle with the recipe format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add PGLite → Supabase upgrade path to README

Clarify the database progression: start with PGLite (Postgres as WASM,
zero infrastructure, pgvector built in, nothing to install). Graduate
to Supabase or self-hosted Postgres when you need connection pooling,
concurrency, and remote MCP access from Claude Desktop, Cowork,
ChatGPT, Perplexity Computer, or any MCP-compatible agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: revert PGLite mention (coming in next branch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify)

Every guide now has exactly these sections in this order:
- ## Goal (one sentence)
- ## What the User Gets (without this / with this)
- ## Implementation (pseudocode with gbrain commands)
- ## Tricky Spots (3-5 numbered gotchas)
- ## How to Verify (3-5 numbered test steps)

11 guides restructured from non-standard headings:
- deterministic-collectors, live-sync, upgrades-auto-update (full rewrites)
- entity-detection, diligence-ingestion, idea-capture, quiet-hours,
  repo-architecture, skill-development, sub-agent-routing (restructured)

23/23 guides now pass consistency audit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: restructure README around the #1 blocker (getting data in)

The README was leading with Postgres and database architecture. Most
users are stuck at step zero: "I have an agent but it doesn't know
anything about my life."

New structure:
1. The Problem — your agent doesn't know your life
2. Getting Data In — integration recipes, front and center
3. The Compounding Thesis — why this matters
4. How this happened — credibility, origin story
5. When you need Postgres — scale, not starting point

Postgres is de-emphasized from a full section to two paragraphs:
"You don't need Postgres to start" and "When you need Postgres"
(1,000+ files, remote MCP access, multiple AI clients).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move Install to top of README, remove duplicate section

Install now appears right after Getting Data In (line 38), not buried
at line 295. The user sees: Problem → Getting Data In → Install.

Removed the duplicate Install section (262 lines) that was lower in
the README. The agent instructions block, CLI quickstart, and all
content is now in the single Install section near the top.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move agent install block to first thing in README

"Start here: paste this into your agent" is now the first section,
right after the one-line pitch. No scrolling, no context, no preamble.
User opens the README, sees the paste block, copies it into OpenClaw
or Hermes, and the agent takes over.

Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: compress install block from 11 steps to 5

The agent install block was 102 lines and 11 steps. Now it's 40 lines
and 5 steps. Same coverage, half the text.

Changes:
- Merged "prove keyword search" + "embed" + "prove hybrid search"
  into one SEARCH step (the user doesn't care about the intermediate)
- Merged skillpack, sync, auto-update, integrations, verification
  into one GO LIVE step with sub-items (post-install polish, not install)
- Shortened database instructions (one line instead of 5 sub-steps)
- Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now
  just "Do not skip steps. Verify each step.")

The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: gitignore all .env files, not just specific ones

CSO audit found .gitignore covered .env.testing and .env.production
but not bare .env. A user creating .env with database credentials
could accidentally commit it.

Fix: .env and .env.* are now gitignored. .env.*.example files are
explicitly un-ignored so templates remain tracked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: scrub PII from essay and recipe examples

- 510-MY-GARRY phone mnemonic → "Your Phone Number"
- "Garry → Authenticated Mode" → "Owner → Authenticated Mode"
- "Telegram" → "secure channel" in auth example
- @garrytan → @yourhandle in X recipe example

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-10 23:39:06 -10:00
committed by GitHub
parent 8de04d3827
commit ce15062694
50 changed files with 7428 additions and 1494 deletions

5
.gitignore vendored
View File

@@ -2,8 +2,9 @@ node_modules/
bin/ bin/
.DS_Store .DS_Store
*.log *.log
.env.testing .env
.env.production .env.*
!.env.*.example
.18a49dfd730ff378-00000000.bun-build .18a49dfd730ff378-00000000.bun-build
.18a49f9dfb996f70-00000000.bun-build .18a49f9dfb996f70-00000000.bun-build
.gstack/ .gstack/

View File

@@ -2,6 +2,18 @@
All notable changes to GBrain will be documented in this file. All notable changes to GBrain will be documented in this file.
## [0.7.0] - 2026-04-11
### Added
- **Your brain gets new senses automatically.** Integration recipes teach your agent how to wire up voice calls, email, Twitter, and calendar into your brain. Run `gbrain integrations` to see what's available. Your agent reads the recipe, asks for API keys, validates each one, and sets everything up. Markdown is code — the recipe IS the installer.
- **Voice-to-brain: phone calls create brain pages.** The first recipe: Twilio + OpenAI Realtime voice agent. Call a number, talk, and a structured brain page appears with entity detection, cross-references, and a summary posted to your messaging app. Opinionated defaults: caller screening, brain-first lookup, quiet hours, thinking sounds. The smoke test calls YOU (outbound) so you experience the magic immediately.
- **`gbrain integrations` command.** Six subcommands for managing integration recipes: `list` (dashboard of senses + reflexes), `show` (recipe details), `status` (credential checks with direct links to get missing keys), `doctor` (health checks), `stats` (signal analytics), `test` (recipe validation). `--json` on every subcommand for agent-parseable output. No database connection needed.
- **Health heartbeat.** Integrations log events to `~/.gbrain/integrations/<id>/heartbeat.jsonl`. Status checks detect stale integrations and include diagnostic steps.
- **17 individually linkable SKILLPACK guides.** The 1,281-line monolith is now broken into standalone guides at `docs/guides/`, organized by category. Each guide is individually searchable and linkable. The SKILLPACK index stays at the same URL (backward compatible).
- **"Getting Data In" documentation.** New `docs/integrations/` with a landing page, recipe format documentation, credential gateway guide, and meeting webhook guide. Explains the deterministic collector pattern: code for data, LLMs for judgment.
- **Architecture and philosophy docs.** `docs/architecture/infra-layer.md` documents the shared foundation (import, chunk, embed, search). `docs/ethos/THIN_HARNESS_FAT_SKILLS.md` is Garry's essay on the architecture philosophy with an agent decision guide. `docs/designs/HOMEBREW_FOR_PERSONAL_AI.md` maps the 10-star vision.
## [0.6.1] - 2026-04-10 ## [0.6.1] - 2026-04-10
### Fixed ### Fixed

View File

@@ -28,6 +28,20 @@ server are both generated from this single source. Skills are fat markdown files
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test) - `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`) - `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts) - `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
- `src/commands/integrations.ts` — Standalone integration recipe management (no DB needed)
- `recipes/` — Integration recipe files (YAML frontmatter + markdown setup instructions)
- `docs/guides/` — Individual SKILLPACK guides (broken out from monolith)
- `docs/integrations/` — "Getting Data In" guides and integration docs
- `docs/architecture/infra-layer.md` — Shared infrastructure documentation
- `docs/ethos/THIN_HARNESS_FAT_SKILLS.md` — Architecture philosophy essay
- `docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md` — "Homebrew for Personal AI" essay
- `docs/guides/repo-architecture.md` — Two-repo pattern (agent vs brain)
- `docs/guides/sub-agent-routing.md` — Model routing table for sub-agents
- `docs/guides/skill-development.md` — 5-step skill development cycle + MECE
- `docs/guides/idea-capture.md` — Originality distribution, depth test, cross-linking
- `docs/guides/quiet-hours.md` — Notification hold + timezone-aware delivery
- `docs/guides/diligence-ingestion.md` — Data room to brain pages pipeline
- `docs/designs/HOMEBREW_FOR_PERSONAL_AI.md` — 10-star vision for integration system
- `scripts/deploy-remote.sh` — One-script remote MCP deployment - `scripts/deploy-remote.sh` — One-script remote MCP deployment
- `docs/mcp/` — Per-client setup guides (Claude Desktop, Code, Cowork, Perplexity, ChatGPT) - `docs/mcp/` — Per-client setup guides (Claude Desktop, Code, Cowork, Perplexity, ChatGPT)
- `openclaw.plugin.json` — ClawHub bundle plugin manifest - `openclaw.plugin.json` — ClawHub bundle plugin manifest
@@ -50,7 +64,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration), `test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration),
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI). `test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
`test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation).
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`. E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys) - `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys)

332
README.md
View File

@@ -1,56 +1,88 @@
# GBrain # GBrain
The memex Vannevar Bush imagined, built for people who think for a living. Your AI agent is smart but it doesn't know anything about your life. GBrain fixes that. Meetings, emails, tweets, calendar events, voice calls, original ideas... all of it flows into a searchable knowledge base that your agent reads before every response and writes to after every conversation. The agent gets smarter every day.
## How this happened > **Requires a frontier model.** Tested with **Claude Opus 4.6** and **GPT-5.4 Thinking**. Likely to break with smaller models.
I was setting up my [OpenClaw](https://openclaw.ai) agent and started a markdown brain repo. One page per person, one page per company, compiled truth on top, append-only timeline on the bottom. The agent got smarter the more it knew, so I kept feeding it. Meetings, emails, tweets, Apple Notes, calendar data, original ideas. One thing led to another. Within a week I had: ## Start here: paste this into your agent
- **10,000+ markdown files** indexed and searchable Copy this block into [OpenClaw](https://openclaw.ai) or [Hermes Agent](https://github.com/NousResearch/hermes-agent). The agent will install GBrain, set up your database, import your files, and verify everything works.
- **3,000+ people** with compiled dossiers and relationship history
- **13 years of calendar data** (21,000+ events)
- **5,800+ Apple Notes** going back to 2009
- **280+ meeting transcripts** with AI analysis
- **300+ captured original ideas** organized by thesis
- **500+ media pages** (video transcripts, books, articles)
- Company profiles, food guides, travel logs
This is what I actually use day to day. The agent runs while I sleep... literally. The dream cycle scans every conversation from the day, enriches missing entities, fixes broken citations, and consolidates memory. I wake up and the brain is smarter than when I went to sleep. OpenClaw ships this as DREAMS.md. Hermes Agent can do the same with a nightly cron job (see the [SKILLPACK](docs/GBRAIN_SKILLPACK.md#the-dream-cycle) for setup). ```
GOAL: Install GBrain so I can search my own markdown files with
hybrid search (keyword + vector). Do not skip steps. Verify each
step before proceeding. If a step fails, stop and fix it.
**You don't need Postgres to start.** The knowledge model is just markdown files in a git repo. The [skills](docs/GBRAIN_SKILLPACK.md) and [schema](docs/GBRAIN_RECOMMENDED_SCHEMA.md) work with any AI agent that can read and write files. Start there. STEP 1 — INSTALL
curl -fsSL https://bun.sh/install | bash && source ~/.bashrc
bun add github:garrytan/gbrain
Verify: gbrain --version
I added Postgres + pgvector later because at 1,000 to 10,000 long markdown docs, `grep` stops working. You need real chunking, real retrieval, real search. GBrain is the thin CLI and MCP layer I built on top of Postgres to solve that, optimized for OpenClaw and smart agents. STEP 2 — DATABASE
Run: gbrain init --supabase
Ask the user for their Supabase Shared Pooler connection string:
"Go to https://supabase.com → your project → Get Connected →
Direct Connection String → Session Pooler → copy Shared Pooler"
Do NOT ask for the anon key or project URL. Those are wrong.
Verify: gbrain doctor --json (all checks pass)
### Ask it anything STEP 3 — IMPORT
Find markdown: find ~/git ~/Documents ~/notes -maxdepth 3 -name "*.md" | head -30
Tell user what you found. Import: gbrain import /path/ --no-embed
Verify: gbrain stats (page count > 0)
> "Who should I invite to dinner who knows both Pedro and Diana?" STEP 4 — SEARCH
> — cross-references the social graph across 3,000+ people pages Run: gbrain embed --stale (generates vector embeddings, ~1 min per 1000 pages)
Then: gbrain query "what are the key themes across these documents?"
Show the user the results. This is the magic moment.
Verify: results include semantic matches, not just keyword hits
> "What have I said about the relationship between shame and founder performance?" STEP 5 — GO LIVE
> — searches YOUR thinking, not the internet a. Read docs/GBRAIN_SKILLPACK.md — adopt brain-first lookup, entity
detection, and back-linking into your behavior immediately
b. Set up live sync: gbrain sync --repo /path/ && gbrain embed --stale
(see docs/guides/live-sync.md for cron/watch/webhook options)
c. Set up auto-update: gbrain check-update --json on a daily cron
(never auto-install — always ask the user first)
d. Show available integrations: gbrain integrations
e. Run verification: read and execute docs/GBRAIN_VERIFY.md
> "What changed with the Series A since Tuesday?" DONE. Tell the user: "GBrain is live. You have [N] pages with hybrid
> — diffs timeline entries across deal and company pages search. I now check the brain before answering questions. Run
'gbrain integrations' to add voice, email, calendar, or Twitter."
```
> "Prep me for my meeting with Jordan in 30 minutes" ### Without an agent (standalone CLI)
> — pulls dossier, shared history, recent activity, open threads
Your markdown repo is the source of truth. GBrain makes it searchable. Your AI agent makes it live. ```bash
bun add -g github:garrytan/gbrain
gbrain init --supabase # guided wizard
gbrain import ~/git/brain/ # index your markdown
gbrain query "what do we know about competitive dynamics?"
```
## Why Postgres Run `gbrain --help` for all commands. See [MCP setup](docs/mcp/DEPLOY.md) for connecting Claude Desktop, Perplexity, etc.
At 500 files, `grep` is fine. At 3,000 people pages, 5,800 Apple Notes, and 13 years of calendar data, `grep` falls apart. You need keyword search for exact names, vector search for semantic meaning, and something that fuses both. You need an index that updates incrementally when one file changes, not a full directory walk. You need your agent to find "everyone who was at the board dinner last March" in milliseconds, not 30 seconds of grepping. ## Getting Data In
GBrain gives you hybrid search that combines keyword and vector approaches, plus a knowledge model that treats every page like an intelligence assessment: compiled truth on top (your current best understanding, rewritten when evidence changes), append-only timeline on the bottom (the evidence trail that never gets edited). Once GBrain is installed, your agent needs data flowing in. GBrain ships integration recipes that your agent sets up for you. It reads the recipe, asks for API keys, validates each one, and runs a smoke test. [Markdown is code](docs/ethos/THIN_HARNESS_FAT_SKILLS.md)... the recipe IS the installer.
AI agents maintain the brain. You ingest a document and the agent updates every entity mentioned, creates cross-reference links, and appends timeline entries. MCP clients query it. The intelligence lives in fat markdown skills, not application code. | Recipe | Requires | What It Does |
|--------|----------|-------------|
| [Public Tunnel](recipes/ngrok-tunnel.md) | — | Fixed URL for MCP + voice (ngrok Hobby $8/mo) |
| [Credential Gateway](recipes/credential-gateway.md) | — | Gmail + Calendar access (ClawVisor or Google OAuth) |
| [Voice-to-Brain](recipes/twilio-voice-brain.md) | ngrok-tunnel | Phone calls → brain pages (Twilio + OpenAI Realtime) |
| [Email-to-Brain](recipes/email-to-brain.md) | credential-gateway | Gmail → entity pages (deterministic collector) |
| [X-to-Brain](recipes/x-to-brain.md) | — | Twitter → brain pages (timeline + mentions + deletions) |
| [Calendar-to-Brain](recipes/calendar-to-brain.md) | credential-gateway | Google Calendar → searchable daily pages |
| [Meeting Sync](recipes/meeting-sync.md) | — | Circleback transcripts → brain pages with attendees |
Run `gbrain integrations` to see status. Dependencies resolve automatically. See [Getting Data In](docs/integrations/README.md) for the full guide.
## The Compounding Thesis ## The Compounding Thesis
Most tools help you find things. GBrain makes you smarter over time. Most tools help you find things. GBrain makes you smarter over time.
The core loop:
``` ```
Signal arrives (meeting, email, tweet, link) Signal arrives (meeting, email, tweet, link)
→ Agent detects entities (people, companies, ideas) → Agent detects entities (people, companies, ideas)
@@ -60,11 +92,28 @@ Signal arrives (meeting, email, tweet, link)
→ Sync: gbrain indexes changes for next query → Sync: gbrain indexes changes for next query
``` ```
Every cycle through this loop adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context — their role, your history, what they care about, what you discussed last time. You never start from zero. Every cycle through this loop adds knowledge. The agent enriches a person page after a meeting. Next time that person comes up, the agent already has context. You never start from zero.
An agent without this loop answers from stale context. An agent with it gets smarter every conversation. The difference compounds daily. An agent without this loop answers from stale context. An agent with it gets smarter every conversation. The difference compounds daily.
Never do anything twice. If you look someone up once, that lookup lives in the brain forever. If a pattern emerges across three meetings, the agent captures it. If you generate an original idea in conversation, it goes to `originals/` — your searchable intellectual archive. > "Who should I invite to dinner who knows both Pedro and Diana?"
> — cross-references the social graph across 3,000+ people pages
> "What have I said about the relationship between shame and founder performance?"
> — searches YOUR thinking, not the internet
> "Prep me for my meeting with Jordan in 30 minutes"
> — pulls dossier, shared history, recent activity, open threads
## How this happened
I was setting up my [OpenClaw](https://openclaw.ai) agent and started a markdown brain repo. One page per person, one page per company, compiled truth on top, append-only timeline on the bottom. The agent got smarter the more it knew, so I kept feeding it. Within a week I had 10,000+ markdown files, 3,000+ people with compiled dossiers, 13 years of calendar data, 280+ meeting transcripts, and 300+ captured original ideas.
The agent runs while I sleep. The dream cycle scans every conversation, enriches missing entities, fixes broken citations, and consolidates memory. I wake up and the brain is smarter than when I went to sleep. See the [cron schedule guide](docs/guides/cron-schedule.md) for setup.
**You don't need Postgres to start.** The knowledge model is just markdown files in a git repo. The [skills](docs/GBRAIN_SKILLPACK.md) and [schema](docs/GBRAIN_RECOMMENDED_SCHEMA.md) work with any AI agent that can read and write files.
**When you need Postgres:** at 1,000+ files, `grep` stops working. GBrain adds hybrid search (keyword + vector + RRF fusion) on top of Postgres + pgvector. The CLI and MCP layer handle chunking, embedding, and incremental sync. Add Postgres when search speed matters, or when you want Claude Desktop, ChatGPT, Perplexity, or other MCP clients to connect to your brain remotely.
## Architecture ## Architecture
@@ -165,203 +214,6 @@ Your file count will be different. Your queries will be different. The agent pic
**The compounding effect.** Search for Pedro. The agent pulls his page, his relationship history, his company. Next time Brex comes up in conversation, the agent already knows Pedro co-founded it, what you discussed last, and what's on your open threads. You didn't do anything — the brain already had it. **The compounding effect.** Search for Pedro. The agent pulls his page, his relationship history, his company. Next time Brex comes up in conversation, the agent already knows Pedro co-founded it, what you discussed last, and what's on your open threads. You didn't do anything — the brain already had it.
## Install
### Prerequisites
**Without Postgres**, you can use the GBrain knowledge model right now: the [skills](docs/GBRAIN_SKILLPACK.md), [schema](docs/GBRAIN_RECOMMENDED_SCHEMA.md), and compiled truth + timeline pattern work with any agent that reads and writes markdown files. Add Postgres when `grep` stops being enough.
**With Postgres**, GBrain needs three things:
| Dependency | What it's for | How to get it |
|------------|--------------|---------------|
| **Supabase account** | Postgres + pgvector database | [supabase.com](https://supabase.com) (Pro tier, $25/mo for 8GB) |
| **OpenAI API key** | Embeddings (text-embedding-3-large) | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) |
| **Anthropic API key** | Multi-query expansion + LLM chunking (Haiku) | [console.anthropic.com](https://console.anthropic.com) |
Set the API keys as environment variables:
```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
```
The Supabase connection URL is configured during `gbrain init`. The OpenAI and Anthropic SDKs read their keys from the environment automatically.
Without an OpenAI key, search still works (keyword only, no vector search). Without an Anthropic key, search still works (no multi-query expansion, no LLM chunking).
### With OpenClaw or Hermes Agent (recommended)
To install, paste this into [OpenClaw](https://openclaw.ai) or [Hermes Agent](https://github.com/NousResearch/hermes-agent) and we'll work with you to do the rest:
```
Set up gbrain (https://github.com/garrytan/gbrain) as my
knowledge brain.
1. Make sure bun is installed:
curl -fsSL https://bun.sh/install | bash
Then run: bun add github:garrytan/gbrain
2. Run: gbrain init --supabase
(follow the wizard to connect my Supabase database)
3. Scan ~/git/ and ~/Documents/ for markdown repos,
pick the best one, and run: gbrain import <path> --no-embed
4. Run a query against the imported data to prove search works
5. Read docs/GBRAIN_RECOMMENDED_SCHEMA.md and offer to
restructure my knowledge base
6. Read docs/GBRAIN_SKILLPACK.md and update all your skills
with the production agent patterns: brain-agent loop,
entity detection, source attribution, iron law back-linking,
and the enrichment pipeline
7. Set up a daily cron for `gbrain check-update`.
Notify me on new features only, not patches.
Never auto-install, just tell me what's new.
8. Set up automatic sync so the vector DB stays current
with the brain repo. Read GBRAIN_SKILLPACK.md Section 18
for approaches (cron, --watch, webhook). Pick what fits
your environment. Always chain:
gbrain sync --repo <path> && gbrain embed --stale
Verify: push a change, confirm it appears in search.
9. Run the verification runbook (docs/GBRAIN_VERIFY.md)
to confirm everything works: schema, sync, embeddings,
brain-first lookup.
```
OpenClaw will install gbrain, walk through Supabase setup, discover your markdown files, import them, and prove search works with a query from your data.
After setup, you talk to your brain through OpenClaw:
```
Search the brain for everything we know about [topic]
Ingest my meeting notes from today
Give me a briefing for my meetings tomorrow
How many pages are in the brain now?
```
GBrain keeps your brain current. After setup, `gbrain sync --watch` polls your git repo and imports only what changed. Binary files (images, PDFs, audio) can be moved to cloud storage with `gbrain files mirror` to slim down your git repo.
> **Supabase settings:** GBrain connects directly to Postgres (not the REST API).
> You need the **Shared Pooler connection string**, not the project URL or anon key.
> Find it: go to your project, click **Get Connected** next to the project URL,
> then **Direct Connection String** > **Session Pooler**, and copy the
> **Shared Pooler** connection string.
### GBrain without OpenClaw
GBrain works with any AI agent, any MCP client, or no agent at all. Three paths:
#### Standalone CLI
Install globally and use gbrain from the terminal:
```bash
bun add -g github:garrytan/gbrain
gbrain init --supabase # guided wizard, connects to your Postgres
gbrain import ~/git/brain/ # index your markdown
gbrain query "what do we know about competitive dynamics?"
```
The CLI gives you every operation: page CRUD, search, tags, links, timeline, graph traversal, file management, health checks. Run `gbrain --help` for the full list.
#### MCP server (Claude Code, Cursor, Windsurf, etc.)
GBrain exposes 30 MCP tools via stdio. Add this to your MCP client config:
**Claude Code** (`~/.claude/server.json`):
```json
{
"mcpServers": {
"gbrain": {
"command": "gbrain",
"args": ["serve"]
}
}
}
```
**Cursor** (Settings > MCP Servers):
```json
{
"gbrain": {
"command": "gbrain",
"args": ["serve"]
}
}
```
This gives your agent `get_page`, `put_page`, `search`, `query`, `add_link`, `traverse_graph`, `sync_brain`, `file_upload`, and 22 more tools. All generated from the same operation definitions as the CLI.
#### Remote MCP Server (Claude Desktop, Cowork, Perplexity, ChatGPT)
Access your brain from any device, any AI client. Deploy as a serverless endpoint on your existing Supabase instance:
```bash
cp .env.production.example .env.production # fill in 3 values
bash scripts/deploy-remote.sh # links, builds, deploys
bun run src/commands/auth.ts create "claude-desktop" # get a token
```
Then add to your AI client:
- **Claude Code:** `claude mcp add gbrain -t http https://YOUR_REF.supabase.co/functions/v1/gbrain-mcp/mcp -H "Authorization: Bearer TOKEN"`
- **Claude Desktop:** Settings > Integrations > Add (NOT JSON config)
- **Perplexity Computer:** Settings > Connectors > Add remote MCP
Per-client setup guides: [`docs/mcp/`](docs/mcp/DEPLOY.md)
ChatGPT support requires OAuth 2.1 and is coming in v0.7. Self-hosted alternatives (Tailscale Funnel, ngrok) documented in [`docs/mcp/ALTERNATIVES.md`](docs/mcp/ALTERNATIVES.md).
**The tools are not enough.** Your agent also needs the playbook: read [GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md) and paste the relevant sections into your agent's system prompt or project instructions. The skillpack tells the agent WHEN and HOW to use each tool: read before responding, write after learning, detect entities on every message, back-link everything.
The skill markdown files in `skills/` are standalone instruction sets. Copy them into your agent's context:
| Skill file | What the agent learns |
|------------|----------------------|
| `skills/ingest/SKILL.md` | How to import meetings, docs, articles |
| `skills/query/SKILL.md` | 3-layer search with synthesis and citations |
| `skills/maintain/SKILL.md` | Periodic health: stale pages, orphans, dead links |
| `skills/enrich/SKILL.md` | Enrich pages from external APIs |
| `skills/briefing/SKILL.md` | Daily briefing with meeting prep |
| `skills/migrate/SKILL.md` | Migrate from Obsidian, Notion, Logseq, etc. |
#### As a TypeScript library
```bash
bun add github:garrytan/gbrain
```
```typescript
import { PostgresEngine } from 'gbrain';
const engine = new PostgresEngine();
await engine.connect({ database_url: process.env.DATABASE_URL });
await engine.initSchema();
// Search
const results = await engine.searchKeyword('startup growth');
// Read
const page = await engine.getPage('people/pedro-franceschi');
// Write
await engine.putPage('concepts/superlinear-returns', {
type: 'concept',
title: 'Superlinear Returns',
compiled_truth: 'Paul Graham argues that returns in many fields are superlinear...',
timeline: '- 2023-10-01: Published on paulgraham.com',
});
```
The `BrainEngine` interface is pluggable. See `docs/ENGINES.md` for how to add backends.
All paths require a Postgres database with pgvector. Supabase Pro ($25/mo) is the recommended zero-ops option.
## Upgrade ## Upgrade
Upgrade depends on how you installed: Upgrade depends on how you installed:
@@ -707,12 +559,22 @@ Initial embedding cost: ~$4-5 for 7,500 pages via OpenAI text-embedding-3-large.
## Docs ## Docs
- **[GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md)** -- **Start here for agents.** Reference architecture for production agents: brain-agent loop, entity detection, enrichment pipeline, meeting ingestion, cron schedule **For agents:**
- [GBRAIN_RECOMMENDED_SCHEMA.md](docs/GBRAIN_RECOMMENDED_SCHEMA.md) -- The recommended brain schema: MECE directories, compiled truth + timeline, enrichment pipelines, resolver decision tree - **[GBRAIN_SKILLPACK.md](docs/GBRAIN_SKILLPACK.md)** -- **Start here.** Index of all patterns, skills, and integrations
- [GBRAIN_V0.md](docs/GBRAIN_V0.md) -- Full product spec, all architecture decisions, every option considered - [Individual guides](docs/guides/) -- 17 standalone guides broken out from the skillpack
- [ENGINES.md](docs/ENGINES.md) -- Pluggable engine interface, capability matrix, how to add backends - [Getting Data In](docs/integrations/README.md) -- Integration recipes, credential setup, data flow patterns
- [SQLITE_ENGINE.md](docs/SQLITE_ENGINE.md) -- Complete SQLite engine plan with schema, FTS5, vector search options - [GBRAIN_VERIFY.md](docs/GBRAIN_VERIFY.md) -- Installation verification runbook
- [GBRAIN_VERIFY.md](docs/GBRAIN_VERIFY.md) -- Installation verification runbook: schema, live sync, embeddings, brain-first lookup
**For humans:**
- [GBRAIN_RECOMMENDED_SCHEMA.md](docs/GBRAIN_RECOMMENDED_SCHEMA.md) -- Brain repo directory structure
- [Infrastructure Layer](docs/architecture/infra-layer.md) -- How import, chunking, embedding, and search work
- [Thin Harness, Fat Skills](docs/ethos/THIN_HARNESS_FAT_SKILLS.md) -- Architecture philosophy
- [Homebrew for Personal AI](docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md) -- Why markdown is code
**Reference:**
- [GBRAIN_V0.md](docs/GBRAIN_V0.md) -- Full product spec, all architecture decisions
- [ENGINES.md](docs/ENGINES.md) -- Pluggable engine interface, how to add backends
- [SQLITE_ENGINE.md](docs/SQLITE_ENGINE.md) -- SQLite engine plan (community PRs welcome)
## Contributing ## Contributing

View File

@@ -32,8 +32,49 @@
**Depends on:** v0.6.0 remote MCP server (shipped). **Depends on:** v0.6.0 remote MCP server (shipped).
## P1 (new from v0.7.0)
### Constrained health_check DSL for third-party recipes
**What:** Replace shell command health_checks with a typed DSL: `{type: "env_exists", name: "KEY"}`, `{type: "url_responds", url: "..."}`, `{type: "heartbeat_fresh", max_age: "24h"}`.
**Why:** Shell commands in recipe frontmatter = arbitrary code execution from markdown. Currently trusted because recipes are first-party only. This DSL is the mandatory gate before opening community recipe submissions.
**Pros:** Eliminates RCE risk from third-party recipes. Health checks become machine-parseable.
**Cons:** Less flexible than shell commands for novel checks. Need to define enough check types to cover common cases.
**Context:** From CEO review + Codex outside voice (2026-04-11). User approved shell commands for first-party but explicitly requested constrained DSL before third-party recipes.
**Depends on:** v0.7.0 recipe format (shipped).
## P2 ## P2
### Community recipe submission (`gbrain integrations submit`)
**What:** Package a user's custom integration recipe as a PR to the GBrain repo. Validates frontmatter, checks constrained DSL health_checks, creates PR with template.
**Why:** Turns GBrain from "Garry's integrations" into a community ecosystem. The recipe format IS the contribution format.
**Pros:** Community-driven integration library. Users build Slack-to-brain, RSS-to-brain, Discord-to-brain.
**Cons:** Support burden. Need constrained DSL (P1) before accepting third-party recipes. Need review process for recipe quality.
**Context:** From CEO review (2026-04-11). User explicitly deferred due to bandwidth constraints. Target v0.9.0.
**Depends on:** Constrained health_check DSL (P1).
### Always-on deployment recipes (Fly.io, Railway)
**What:** Alternative deployment recipes for voice-to-brain and future integrations that run on cloud servers instead of local + ngrok.
**Why:** ngrok free URLs are ephemeral (change on restart). Always-on deployment eliminates the watchdog complexity and gives a stable webhook URL.
**Pros:** Stable URLs, no ngrok dependency, production-grade uptime.
**Cons:** Costs $5-10/mo per integration. Requires cloud account.
**Context:** From DX review (2026-04-11). v0.7.0 ships local+ngrok as v1 deployment path.
**Depends on:** v0.7.0 recipe format (shipped).
### Fly.io HTTP server as alternative deployment ### Fly.io HTTP server as alternative deployment
**What:** Add `gbrain serve --http` and a Dockerfile/fly.toml for users who prefer a traditional server over Edge Functions. **What:** Add `gbrain serve --http` and a Dockerfile/fly.toml for users who prefer a traditional server over Edge Functions.

View File

@@ -1 +1 @@
0.6.1 0.7.0

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,105 @@
# GBrain Infrastructure Layer
The shared foundation that all skills, recipes, and integrations build on.
## Data Pipeline
```
INPUT (markdown files, git repo)
FILE RESOLUTION (local → .redirect → .supabase → error)
MARKDOWN PARSER (gray-matter frontmatter + body)
→ compiled_truth + timeline separation
CONTENT HASH (SHA-256 idempotency check — skip if unchanged)
CHUNKING (3 strategies, configurable)
├── Recursive: 300-word chunks, 50-word overlap, 5-level delimiter hierarchy
├── Semantic: embed sentences, cosine similarity, Savitzky-Golay smoothing
└── LLM-guided: Claude Haiku identifies topic shifts in 128-word candidates
EMBEDDING (OpenAI text-embedding-3-large, 1536 dimensions)
→ batch 100, exponential backoff, non-fatal if fails
DATABASE TRANSACTION (atomic: page + chunks + tags + version)
SEARCH (hybrid, available immediately)
```
## Search Architecture
GBrain uses Reciprocal Rank Fusion (RRF) to merge vector and keyword search:
```
User Query
EXPANSION (optional: Claude Haiku generates 2 alternative phrasings)
├── VECTOR SEARCH (pgvector HNSW, cosine distance)
│ → 2x limit results per query variant
└── KEYWORD SEARCH (PostgreSQL tsvector, ts_rank)
→ 2x limit results
RRF MERGE (score = Σ(1/(60 + rank)), balances both fairly)
4-LAYER DEDUP
├── Best 3 chunks per page (source dedup)
├── Jaccard similarity > 0.85 (text dedup)
├── No type exceeds 60% (diversity)
└── Max 2 chunks per page (page cap)
TOP N RESULTS (default 20)
```
## Key Components
| File | Purpose |
|------|---------|
| `src/core/engine.ts` | Pluggable engine interface (BrainEngine) |
| `src/core/postgres-engine.ts` | Postgres + pgvector implementation |
| `src/core/import-file.ts` | importFromFile + importFromContent pipeline |
| `src/core/sync.ts` | Git-based incremental change detection |
| `src/core/markdown.ts` | YAML frontmatter + compiled_truth/timeline parsing |
| `src/core/embedding.ts` | OpenAI embedding with batch, retry, backoff |
| `src/core/chunkers/recursive.ts` | Base chunker (300w, 5-level delimiters) |
| `src/core/chunkers/semantic.ts` | Embedding-based topic boundary detection |
| `src/core/chunkers/llm.ts` | Claude Haiku guided chunking |
| `src/core/search/hybrid.ts` | RRF merge of vector + keyword |
| `src/core/search/dedup.ts` | 4-layer result deduplication |
| `src/core/search/expansion.ts` | Multi-query expansion via Claude Haiku |
| `src/core/storage.ts` | Pluggable storage (S3, Supabase, local) |
| `src/core/operations.ts` | Contract-first operation definitions (31 ops) |
| `src/schema.sql` | Full DDL (10 tables, RLS, tsvector, HNSW) |
## Schema Overview
10 tables in Postgres:
- **pages** — slug (unique), type, title, compiled_truth, timeline, frontmatter (JSONB)
- **content_chunks** — pgvector 1536-dim embedding, chunk_source (compiled_truth|timeline)
- **links** — typed edges (knows, works_at, invested_in, founded, etc.)
- **tags** — many-to-many page tagging
- **timeline_entries** — structured events (date, source, summary, detail)
- **page_versions** — snapshot history for diff/revert
- **raw_data** — sidecar JSON from external APIs (preserves provenance)
- **files** — binary attachments in storage backend
- **ingest_log** — audit trail of import operations
- **config** — brain-level settings (version, embedding model, chunk strategy)
Full-text search uses weighted tsvector: title (A), compiled_truth (B), timeline (C).
Vector search uses HNSW index with cosine distance on content_chunks.embedding.
## The Thin Harness Principle
GBrain is the deterministic layer. Skills and recipes are the latent space layer.
See [Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md) for the full
architecture philosophy.
- **GBrain CLI** = thin harness (same input → same output)
- **Skills** (ingest, query, maintain, enrich, briefing, migrate, setup) = fat skills
- **Recipes** (voice-to-brain, email-to-brain) = fat skills that install infrastructure
The agent reads the skill/recipe and uses GBrain's deterministic tools to do the work.

View File

@@ -0,0 +1,154 @@
# Homebrew for Personal AI Infrastructure
The 10-star vision for GBrain's integration system. Ship Approach B (v0.7.0),
build toward this over subsequent releases.
## The Vision
GBrain becomes a personal infrastructure operating system where every signal in
your life flows through the brain automatically. Integrations are **senses**
(data inputs) and **reflexes** (automated responses to patterns). Users subscribe
to the creator's actual operating system, then customize it.
```
$ gbrain integrations
SENSES (data inputs) STATUS
-------------------------------------------------------
voice-to-brain Phone calls -> brain pages ACTIVE last call: 2h ago
email-to-brain Gmail -> entity updates ACTIVE 47 emails today
x-to-brain Twitter -> media pages ACTIVE 312 tweets tracked
calendar-to-brain Google Cal -> meeting prep ACTIVE 3 meetings tomorrow
photos-to-brain Camera roll -> visual mem AVAILABLE
slack-to-brain Slack -> conversation index AVAILABLE
rss-to-brain RSS feeds -> media pages AVAILABLE
REFLEXES (automated responses) STATUS
-------------------------------------------------------
meeting-prep Brief me before meetings ACTIVE next: 9am tomorrow
entity-enrich Auto-enrich new contacts ACTIVE 12 enriched today
dream-cycle Overnight brain maintenance ACTIVE last run: 3am
deal-tracker Alert on deal changes AVAILABLE
follow-up-nudge Remind on stale threads AVAILABLE
This week: 1,247 signals ingested. Top: email (47%), voice (23%), X (18%).
34 new entity pages created. 7 calls transcribed.
Run 'gbrain integrations show <id>' for setup details.
```
The user feels: "My brain is alive. It's watching everything I care about, and
it's getting smarter every day. I didn't have to write any code. I just said yes
when the agent asked."
## Architecture: Senses & Reflexes
### Recipe Format (YAML frontmatter + markdown body)
```yaml
---
id: voice-to-brain
name: Voice-to-Brain
version: 0.7.0
description: Phone calls create brain pages via Twilio + OpenAI Realtime + GBrain MCP
category: sense
requires: [credential-gateway]
secrets:
- name: TWILIO_ACCOUNT_SID
description: Twilio account SID
where: https://console.twilio.com
- name: OPENAI_API_KEY
description: OpenAI API key (for Realtime voice)
where: https://platform.openai.com/api-keys
health_checks:
- curl -s https://api.twilio.com/2010-04-01 > /dev/null
- curl -s https://api.openai.com/v1/models > /dev/null
setup_time: 30 min
---
[Opinionated setup instructions the agent executes...]
```
### Dependency Graph
Recipes declare `requires` in frontmatter. The CLI resolves dependencies before
setup. If voice-to-brain requires credential-gateway, the agent sets up
credential-gateway first.
```
credential-gateway
├── voice-to-brain (requires credentials for Twilio)
├── email-to-brain (requires credentials for Gmail)
└── calendar-to-brain (requires credentials for Google Calendar)
x-to-brain (standalone, uses X API directly)
```
### Health Dashboard
`gbrain integrations doctor` runs health_checks from every configured recipe:
```
$ gbrain integrations doctor
voice-to-brain: ✓ Twilio reachable ✓ OpenAI key valid ✓ ngrok tunnel up
email-to-brain: ✓ Gmail auth valid ✗ No emails in 48h (check cron)
OVERALL: 1 warning
```
### Sense Analytics
`gbrain integrations stats` aggregates heartbeat data:
```
$ gbrain integrations stats
This week: 1,247 signals ingested
Top sources: email (47%), voice (23%), X (18%), calendar (12%)
34 new entity pages created
7 calls transcribed
Brain growth: 12,400 → 12,834 pages (+434)
```
### Reflex Rules Engine (future)
Reflexes are recipes that trigger on brain state changes:
```yaml
---
id: deal-tracker
category: reflex
triggers:
- type: page_updated
filter: {type: deal, field: status}
- type: timeline_entry
filter: {source: email, mentions: deal}
action: alert
---
When a deal page's status changes or a new email mentions a deal,
alert the user with context from the brain.
```
## Roadmap
| Version | What Ships | Key Recipe |
|---------|-----------|------------|
| v0.7.0 | Recipe format, CLI, SKILLPACK breakout | voice-to-brain |
| v0.8.0 | 3 more senses, reflex format | email, X, calendar |
| v0.9.0 | Community recipes, install executor | community submissions |
| v1.0.0 | Full senses/reflexes, health dashboard | meeting-prep, dream-cycle |
## Key Design Decisions
1. **GBrain is deterministic infrastructure.** Cross-sense correlation, pattern
detection, and intelligent responses are the agent's job (OpenClaw/Hermes).
GBrain provides the plumbing.
2. **Agents ARE the runtime.** No npm packages, Docker images, or deterministic
scripts. The recipe markdown IS the installer. The agent reads it and does
the work.
3. **Very opinionated defaults.** Ship the creator's exact production setup as
the default. Users customize from there. Unknown callers get screened. Quiet
hours are enforced. Brain-first lookup happens on every call.
4. **Agent-readable outputs.** All CLI output must be parseable by agents (--json
flag). Migration files include agent instructions. The agent is the primary
consumer, not the human.

View File

@@ -0,0 +1,188 @@
---
type: essay
title: "Homebrew for Personal AI"
subtitle: "Why Markdown is Code and Your Agent is a Package Manager"
author: Garry Tan
created: 2026-04-11
updated: 2026-04-11
tags: [ai, gbrain, gstack, markdown-is-code, open-source, software-distribution, agents, openclaw]
status: draft-v2
prior: "Thin Harness, Fat Skills"
---
# Homebrew for Personal AI
`brew install` gives you someone else's binary. `npm install` gives you someone else's source code. Both require you to understand the tool, configure it, integrate it, maintain it.
What if software distribution worked differently? What if you could describe a capability in plain English, hand that description to an AI agent, and the agent built a native implementation tailored to your setup?
That's what happens when markdown is code.
## Markdown is code
Here's a real skill file. This one teaches an AI agent to screen phone calls:
```markdown
# Voice Agent — Your Phone Number
Caller → Twilio → <Stream> WebSocket → Voice Server (port 8765)
↕ audio
OpenAI Realtime API
↓ tool calls
Brain / Calendar / Telegram
## Call Routing
Every inbound call routes based on caller phone number + brain lookup:
### Owner → Authenticated Mode
- Send crypto-random 6-digit code to secure channel
- Caller reads it back
- Match → full assistant mode (brain, calendar, scheduling)
- No match → treated as unknown caller
### Known Person, Inner Circle (brain score ≥ 4) → Forward
- Greet by name with brain context
- Transfer to cell
- If no answer (30s timeout), take message
- Text Telegram with who called and context
### Unknown Caller → Screen
- Get their name, look them up in brain
- If inner circle → offer to transfer
- Otherwise → take message
- Create brain entry with phone number (marked UNVERIFIED)
```
That's not pseudocode. That's not documentation. That's a working specification that a model like Claude Opus 4.6 with a million-token context window can read and implement. The architecture diagram tells it the components. The routing table tells it the logic. The security model tells it the constraints. The agent reads this file, understands it, and builds the Twilio integration, the WebSocket server, the Telegram bot hooks, the brain lookup, all of it, shaped to whatever infrastructure the user already has.
A skill file is a method call. It takes parameters (your phone number, your brain, your preferred messaging app). Same skill, different arguments, different implementation. The procedure is the package. The model is the runtime.
## The distribution mechanism
Traditional package managers distribute artifacts: compiled binaries, source tarballs, container images. The consumer runs someone else's code.
GBrain distributes recipes: markdown files that describe capabilities with enough specificity that an AI agent can implement them from scratch. The consumer gets a native implementation. No dependency hell. No version conflicts. No transitive vulnerability chains. Because there is no upstream code. There's a description of what to build and why.
Here's how it works:
1. **Build a feature.** Implement a voice agent, meeting ingestion pipeline, email triage system, investment diligence workflow, whatever.
2. **GBrain captures the recipe.** Not just the code. The architecture, the integration points, the failure modes, the judgment calls. A markdown file that encodes the full capability.
3. **Push to the repo.** Open source. Anyone can read it.
4. **Someone else's agent pulls the recipe.** Reads the markdown. Says: "New recipe available: AI voice agent with caller screening. Want it?" User says yes. The agent reads the spec and builds it.
No installation. No configuration wizard. No README. The agent read a document and figured it out.
## Why this works now
This didn't work two years ago. Two things changed.
**Context windows hit a million tokens.** A real skill file for meeting ingestion is 200+ lines. The enrichment skill that calls it references a brain schema, a resolver, a citation standard, five external APIs, and a cross-linking protocol. An agent implementing this recipe needs to hold all of that in working memory simultaneously while also understanding the user's existing setup. At 8K tokens, impossible. At 128K, marginal. At 1M, comfortable.
**Models crossed the judgment threshold.** Here's a snippet from a real enrichment recipe:
```markdown
## Philosophy
A brain page should read like an intelligence dossier crossed
with a therapist's notes, not a LinkedIn scrape. We want:
- What they believe — ideology, worldview, first principles
- What they're building — current projects, what's next
- What motivates them — ambition drivers, career arc
- What makes them emotional — angry, excited, defensive, proud
- Their trajectory — ascending, plateauing, pivoting, declining?
- Hard facts — role, company, funding, location, contact info
Facts are table stakes. Texture is the value.
```
A model implementing this recipe has to understand the difference between a LinkedIn scrape and an intelligence dossier. That's a judgment call about what information is worth capturing and how to weight it. GPT-3 couldn't do this. GPT-4 could sort of do it. Opus 4.6 does it well. The enabling technology is models that are smart enough to interpret intent, not just follow instructions.
## What a recipe actually contains
A good recipe has five sections:
**Architecture.** The component diagram. What talks to what, over what protocol, with what data flow. This is the skeleton the agent builds first.
**Routing logic.** The decision tree. When X happens, do Y. When Z fails, fall back to W. This is where domain knowledge lives. A voice agent recipe encodes call routing. A diligence recipe encodes how to process pitch decks vs. financial models vs. cap tables. A meeting ingestion recipe encodes how to turn a raw transcript into actionable intelligence.
**Integration points.** What external systems does this touch? Twilio, Telegram, Gmail, Circleback, Slack, GitHub, Supabase, whatever. The recipe names the integrations; the agent figures out how to connect them given what the user already has configured.
**Judgment calls.** The hard part. Not "send an email" but "decide whether this email is worth surfacing to the user based on sender importance, time sensitivity, and whether it requires a decision." Recipes that skip the judgment calls produce shallow implementations. The judgment calls are the actual value.
**Failure modes.** What goes wrong and what to do about it. "If Circleback token expires, message the user and ask them to reconnect. Don't silently skip." "If caller ID is spoofed, never trust it for authentication. Use a challenge-response code via a separate channel." Recipes without failure modes produce brittle systems.
Here's a real example. This is the diligence recipe's detection logic:
```markdown
## Detection
Recognize data room materials by:
- PDF filenames: "Data Deck", "Intro Deck", "Cap Table",
"Financial Model", "Pitch Deck", "Series [A-D]"
- Spreadsheets with tabs: Revenue, Retention, Cohorts,
CAC, Gross Margin, Unit Economics, ARR
- User saying: "data room", "diligence", "deck", "pitch"
- Context: shared in the Diligence topic
```
That's a pattern matcher expressed in English. An agent reads this and knows how to classify incoming documents. No regex. No file type configuration. Just a description of the pattern and the model's judgment about whether a given document matches.
## Pick and choose
GBrain is not monolithic. Recipes are independent. Take what you want:
- **Voice agent** — phone screening, caller ID, brain lookup, message routing
- **Meeting ingestion** — transcript processing, entity extraction, action item capture, timeline updates
- **Email triage** — inbox sweep, priority classification, draft replies, scheduling extraction
- **Enrichment pipeline** — people and company research from multiple data sources, diarized into brain pages
- **Diligence processing** — data room ingestion, PDF extraction, financial model analysis
- **Social monitoring** — X/Twitter timeline analysis, mention tracking, narrative detection
- **Content pipeline** — idea capture, link ingestion, article summarization
Each recipe is self-contained. Your agent knows what you already have. GBrain pings daily: "Three new recipes since last sync. Want any?" You pick. It builds.
And because the source code is English, forking is trivial. Don't like how the voice agent handles unknown callers? Edit the markdown. Change "take a message" to "ask three screening questions first." The behavior changes because the spec changed.
## The thin harness, fat skills connection
This essay is a sequel. The prequel was "Thin Harness, Fat Skills," which argued that the secret to 100x AI productivity isn't better models but better context management. Keep the harness thin (the program running the model). Make the skills fat (markdown procedures encoding judgment and process).
"Markdown is code" is the distribution corollary. If the skills are fat markdown files, and if models are smart enough to implement from markdown, then the skills are distributable software. The skill file is simultaneously:
- **Documentation** for humans reading it
- **Specification** for the implementing agent
- **Package** for the distribution system
- **Source code** for the resulting capability
Four artifacts collapsed into one. That's why this is different from every previous package manager. `brew install` separates the formula from the binary from the docs from the source. GBrain collapses them. The markdown is all four.
## The architecture underneath
Three layers, same as the talk:
**Fat skills** on top. Markdown recipes encoding judgment, process, failure modes, and domain knowledge. This is where 90% of the value lives. This is what gets distributed.
**Thin harness** in the middle. The program running the model. File operations, tool dispatch, context management, safety enforcement. About 200 lines. OpenClaw or any equivalent. The less the harness constrains, the more the recipes can express.
**Deterministic foundation** on the bottom. Databases, APIs, CLIs. Same input, same output, every time. SQL queries, HTTP calls, file reads. The skills describe WHEN to call these; the harness executes them.
Push intelligence UP into skills. Push execution DOWN into deterministic tooling. Distribute the skills. That's the whole system.
## What this means
When implementation cost approaches zero, the bottleneck shifts. It's no longer "can we build this?" It's "should we build this?" and "what exactly should it do?"
Taste, vision, and domain knowledge become the scarce resources. The person who deeply understands call screening and writes a precise recipe creates more value than the person who can implement a Twilio integration from scratch. The recipe IS the implementation.
This also means the best AI agent setups will be open source by default. Closed, proprietary agent configurations are competing against a world where someone publishes a recipe and a thousand agents implement it overnight. The recipe propagates at the speed of a git push. The moat is taste, not code.
Software distribution reimagined: the package is a markdown file, the runtime is a sufficiently smart model, the package manager is your AI agent, and the app store is a git repo.
`gbrain install voice-agent`
That's it.

View File

@@ -0,0 +1,208 @@
---
type: essay
title: "Thin Harness, Fat Skills"
subtitle: "How to Make AI Agents Actually Understand Your Data"
author: Garry Tan
created: 2026-04-09
updated: 2026-04-09
tags: [ai, agents, gstack, harness-engineering, skills, architecture]
status: draft-v4
talk: "YC Spring 2026 — Thin Harness, Fat Skills"
---
# Thin Harness, Fat Skills
Steve Yegge says people using AI coding agents are "10x to 100x as productive as engineers using Cursor and chat today, and roughly 1000x as productive as Googlers were back in 2005."
That's a real number. I've seen it. I've lived it. But when people hear 100x, they think: better models. Smarter Claude. More parameters.
That's the wrong frame entirely. The 2x people and the 100x people are using the same models. The difference is five concepts that fit on an index card.
## The harness is the secret sauce
On March 31, 2026, Anthropic accidentally shipped the entire source code for Claude Code to the npm registry. 512,000 lines. When I read it, it confirmed everything I'd been teaching at YC. The secret sauce isn't the model. It's the thing wrapping the model: the harness. Live repo context. Prompt caching. Purpose-built tools. Context bloat minimization. Structured session memory. Parallel sub-agents.
None of that is about making the model smarter. All of it is about giving the model the right context, at the right time, without drowning it in noise.
That's the only question that matters. And the answer has a specific shape. I call it **thin harness, fat skills**.
## Five definitions
The bottleneck is never the model's intelligence. The bottleneck is whether the model understands your schema. Models already know how to reason, synthesize, and write code. They fail because they don't know your data. Five definitions fix this.
### Definition 1: Skill File
A skill file is a reusable markdown procedure that teaches the model HOW to do something. Not WHAT to do. The user supplies the specifics. The skill supplies the process.
**Markdown is actually code.** A skill file is a more perfect encapsulation of capability than rigid source code, because it describes process, judgment, and context in the language the model already thinks in.
On the left is a skill called `/investigate`. Seven steps: scope the dataset, build a timeline, diarize every document, synthesize, argue both sides, cite sources. It takes three parameters: TARGET, QUESTION, and DATASET.
On the right are two completely different invocations of the same skill. One points at Dr. Sarah Chen and 2.1 million discovery emails, asking whether a safety scientist was silenced. The other points at Pacific Corporate Services and FEC filings, asking whether shell companies are coordinating campaign donations.
Same skill. Same seven steps. Same markdown file. In one case it's a medical research analyst. In the other it's a forensic investigator. The skill describes a process of judgment. The invocation supplies the world.
**This is the key insight most people miss: a skill file works like a method call.** It takes parameters. You invoke it with different arguments. The same procedure produces radically different capabilities depending on what you pass in. This is not prompt engineering. This is software design, using markdown as the programming language and human judgment as the runtime.
### Definition 2: Harness
The harness is the program that runs the LLM. It does four things: runs the model in a loop, reads and writes your files, manages context, and enforces safety. That's the "thin."
The anti-pattern is a fat harness with thin skills: 40+ tool definitions eating half the context window. God tools with 2 to 5 second MCP round-trips. REST API wrappers that turn every endpoint into a tool. 3x the tokens, 3x the latency, 3x the failure rate.
What you should build instead: a Playwright CLI that does each browser operation in 100 milliseconds. Compare: Chrome MCP takes 15 seconds for screenshot + find + click + wait + read. Playwright CLI takes 200 milliseconds for screenshot + assert. 75x faster. Software doesn't have to be precious anymore. Build exactly what you need.
### Definition 3: Resolver
A resolver is a routing table for context. When task type X appears, load document Y first.
Skills say HOW. Resolvers say WHAT to load WHEN. A developer changes a prompt. Without the resolver, they ship it. With the resolver, the model reads `docs/EVALS.md` first, which says: run the eval suite, compare scores, if accuracy drops more than 2%, revert and investigate. The developer didn't know the eval suite existed. The resolver loaded the right context at the right moment.
Claude Code has a built-in resolver. Every skill has a description field, and the model matches user intent to skill descriptions automatically. You never have to remember `/ship` exists. The description IS the resolver. It's like Clippy. Except it actually works.
A confession: my CLAUDE.md was 20,000 lines. Every single thing I ran across went in there. Every quirk, every pattern, every lesson. Completely ridiculous. The model's attention degraded. Claude Code literally told me to cut it back. The fix: about 200 lines. Just pointers to documents. The resolver loads the right one when it matters.
### Definition 4: Latent vs. Deterministic
Every step in your system is one or the other.
**Latent space** is where intelligence lives. The model reads, interprets, decides. Judgment. Synthesis. Pattern recognition.
**Deterministic** is where trust lives. Same input, same output. Every time. SQL. Code. Numbers.
An LLM can seat 8 people at a dinner table. Ask it to seat 800 and it will hallucinate a seating chart that looks plausible but is completely wrong. That's a deterministic problem forced into latent space. The worst systems put the wrong work on the wrong side.
### Definition 5: Diarization
The model reads everything about a subject and writes a structured profile. Read 50 documents, produce 1 page of judgment.
No SQL query produces this. No RAG pipeline produces this. The model has to actually read, hold contradictions in mind, notice what changed and when, and write structured intelligence. This is what makes AI useful for real knowledge work.
## The architecture
Three layers:
**Fat skills** on top. Markdown procedures that encode judgment, process, and domain knowledge. This is where 90% of the value lives.
**Thin CLI harness** in the middle. About 200 lines. JSON in, text out. Read-only by default. CLI first, add MCP later.
**Your app** on the bottom. QueryDB. ReadDoc. Search. Timeline. The deterministic foundation.
Push intelligence UP into skills. Push execution DOWN into deterministic tooling. Keep the harness THIN.
## The system that learns: YC Startup School
Let me show you all five definitions working together. Not in theory. In an actual system we're building at YC.
Chase Center. July 2026. 6,000 founders. Each one has a structured application, questionnaire answers, transcripts from 1:1 advisor chats, and public signals: X posts, GitHub commits, Claude Code transcripts showing how fast they ship.
The traditional approach: a program team of 15 reads applications, makes gut calls, updates a spreadsheet. It works at 200 founders. It breaks at 6,000.
No human can hold 6,000 profiles in working memory and notice that the three best candidates for the infrastructure-for-AI-agents cohort are a dev tools founder in Lagos, a compliance founder in Singapore, and a CLI-tooling founder in Brooklyn who all described the same pain point in different words during their 1:1 chats.
The model can.
**Step 1: Enrich every founder.**
The `/enrich-founder` skill: pull all sources, run enrichments, diarize, highlight what they SAY vs what they're ACTUALLY BUILDING. On the right, the deterministic calls: SQL to find stale profiles, GitHub stats, browser test on the demo URL, social signal pulls, CrustData for company intel.
Cron runs nightly at 2am. 6,000 profiles, every night, always fresh.
The diarization output catches things no keyword search would find:
```
FOUNDER: Maria Santos
COMPANY: Contrail (contrail.dev)
SAYS: "Datadog for AI agents"
ACTUALLY BUILDING: 80% of commits are in billing module.
She's building a FinOps tool disguised as observability.
```
"SAYS" vs "ACTUALLY BUILDING." That requires reading the GitHub commit history, the application, and the advisor transcript and holding all three in mind at once.
**Step 2: Match 6,000 founders. Make judgment calls.**
This is where skill-as-method-call really shines. Three invocations:
`/match-breakout`: 1,200 founders, cluster by sector affinity, 30 per room. Embed + deterministic assign.
`/match-lunch`: 600 founders, serendipity matching (cross-sector), 8 per table, no repeats. The LLM invents the themes, then assigns.
`/match-live`: whoever is in the zone, nearest-neighbor embedding, real-time at 200ms, 1:1 pairs, not already met.
Same skill. Three invocations. Three completely different matching strategies. Different parameters, different strategies, different group sizes. The skill describes the process. The arguments shape the output.
And the model's judgment calls: "Santos and Oram are both AI infra, but they're not competitors. Santos is cost attribution, Oram is orchestration. Put them in the same group." And: "Kim applied as 'developer tools' but his 1:1 transcript reveals he's building compliance automation for SOC2. Move him to FinTech/RegTech."
No embedding captures the Kim reclassification. No algorithm can do it. The model has to read the entire profile.
**Step 3: The self-learning loop.**
After the event, the `/improve` skill reads NPS surveys, diarizes the "OK" responses (not the bad ones, the mediocre ones), and extracts patterns. Then it proposes new rules and writes them back into the matching skills:
```
When attendee says "AI infrastructure"
but startup is 80%+ billing code:
-> Classify as FinTech, not AI Infra.
When two attendees in same group
already know each other:
-> Penalize proximity.
Prioritize novel introductions.
```
These rules get written back into the skill file. Next run uses them automatically. The skill rewrites itself.
July event: 12% "OK" ratings. Next event: 4%. The skill file learned what "OK" actually meant.
Same pattern as every other domain: retrieve, read, diarize, count, synthesize. Then: survey, investigate, diarize, rewrite the skill. It transfers everywhere.
## OpenClaw: where the skills live
I want to tell you about one more harness. Not for coding. For everything else.
I run a personal AI agent on OpenClaw. It has a persona, knows who I am, and maintains a knowledge base of thousands of interconnected files. But the thing that makes it work is the exact same principle. Thin harness, fat skills.
I tweeted about this a few days ago:
> *You are not allowed to do one-off work. If I ask you to do something and it's the kind of thing that will need to happen again, you must: do it manually the first time on 3 to 10 items. Show me the output. If I approve, codify it into a skill file. If it should run automatically, put it on a cron.*
> *The test: if I have to ask you for something twice, you failed.*
That resonated: a thousand likes, twenty-five hundred bookmarks. People thought it was a prompt engineering trick. It's not. It's the same architecture.
Claude Code is the best harness for coding. OpenClaw is the best harness for everything else: email, calendar, meetings, people, research, alerts. Same principle. Thin harness. Fat skills. The skills are method calls. The parameters change. The process stays.
Every skill I write is a permanent upgrade. It never degrades. It never forgets. It runs at 3 AM while I sleep. And when the next model drops, every skill instantly gets better, because the judgment in the latent steps improves while the deterministic steps stay perfectly reliable.
That's how you get Yegge's 10x to 100x. Not a smarter model. Fat skills, thin harness, and the discipline to codify everything.
The system compounds. Build it once. It runs forever.
Five definitions. Three layers. One principle. Thin harness, fat skills.
---
## Agent Decision Guide: Skill or Code?
When building GBrain features, use this decision guide:
| Question | If YES | If NO |
|----------|--------|-------|
| Does the agent need to think, adapt, or ask questions? | **Skill** (recipe markdown) | Code |
| Same input always produces same output? | **Code** (CLI command) | Skill |
| Does it require judgment about the user's environment? | **Skill** | Code |
| Is it a lookup, list, or status check? | **Code** | Probably skill |
| Does it change behavior based on conversation context? | **Skill** | Code |
**GBrain examples:**
- `gbrain integrations list` = **Code** (reads files, checks env vars, deterministic)
- `gbrain integrations status` = **Code** (checks env vars + heartbeat, deterministic)
- `gbrain integrations doctor` = **Code** (runs health checks, deterministic)
- `gbrain integrations stats` = **Code** (aggregates JSONL, deterministic)
- Recipe setup flow = **Skill** (asks for API keys, adapts to environment, validates)
- Recipe changelog surfacing = **Skill** (agent describes changes conversationally)
- Entity detection = **Skill** (reads message, decides what's important, creates pages)
- Meeting ingestion = **Skill** (reads transcript, extracts entities, updates pages)
**The rule:** If it's a lookup table, it's code. If the agent needs to think, it's a skill.

View File

@@ -0,0 +1,129 @@
# The Brain-Agent Loop
## Goal
Every conversation makes the brain smarter. Every brain lookup makes responses
better. The loop compounds daily.
## What the User Gets
Without this: the agent answers from stale context. You discuss a deal on Monday,
and by Friday the agent has forgotten. Every conversation starts from zero.
With this: six months in, the agent knows more about your world than you can hold
in working memory. It never forgets. It never stops indexing.
## The Loop
```
Signal arrives (message, meeting, email, tweet, link)
DETECT entities (people, companies, concepts, original thinking)
│ → spawn sub-agent (see entity-detection.md)
READ: check brain FIRST (before responding)
│ → gbrain search "{entity name}"
│ → gbrain get {slug} (if you know it)
│ → gbrain query "what do we know about {topic}"
RESPOND with brain context (every answer is better with context)
WRITE: update brain pages (new info → compiled truth + timeline)
│ → gbrain put {slug} (update page)
│ → add_timeline_entry (append to timeline)
│ → add_link (cross-reference to other entities)
SYNC: gbrain indexes changes
│ → gbrain sync --no-pull --no-embed
(next signal arrives — agent is now smarter)
```
## Implementation
### On Every Inbound Message
```
on_message(text):
// 1. DETECT (async, don't block)
spawn_entity_detector(text)
// 2. READ (before composing response)
entities = extract_entity_names(text) // quick regex/NER
context = []
for name in entities:
results = gbrain_search(name)
if results:
page = gbrain_get(results[0].slug)
context.append(page.compiled_truth)
// 3. RESPOND (with brain context injected)
response = compose_response(text, context)
// 4. WRITE (after responding, if new info emerged)
if response_contains_new_info(response):
for entity in mentioned_entities:
gbrain_add_timeline_entry(entity.slug, {
date: today,
summary: "Discussed {topic}",
source: "[Source: User, conversation, {date}]"
})
// 5. SYNC
gbrain_sync()
```
### The Two Invariants
1. **Every READ improves the response.** If you answered a question about a
person without checking their brain page first, you gave a worse answer
than you could have. The brain almost always has something. External APIs
fill gaps, they don't start from scratch.
2. **Every WRITE improves future reads.** If a meeting transcript mentioned
new information about a company and you didn't update the company page,
you created a gap that will bite you later.
## Tricky Spots
1. **Read BEFORE responding, not after.** The temptation is to respond first
and update the brain later. But the brain context makes the response better.
Read first.
2. **Don't skip the write step.** "I'll update the brain later" means never.
Write immediately after the conversation, while the context is fresh.
3. **Sync after every write batch.** Without sync, the brain search index is
stale. The next query won't find what you just wrote.
4. **External APIs are fallback, not primary.** `gbrain search` before
Brave Search. `gbrain get` before Crustdata. The brain has relationship
history, your own assessments, meeting transcripts, cross-references.
No external API can provide that.
## How to Verify It Works
1. **Mention a person the brain knows.** Ask "what do we know about {name}?"
The agent should search the brain and return compiled truth, not hallucinate
or do a web search.
2. **Discuss something new about a known entity.** Say "I heard Acme Corp
just raised Series B." After the conversation, check: does Acme Corp's
brain page have a new timeline entry?
3. **Ask about the same person a day later.** The agent should immediately
pull brain context without you asking. If it doesn't reference the brain
page, the loop isn't running.
4. **Check the sync.** After a conversation, run `gbrain search "{topic}"`
from the CLI. The new information should be searchable.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Entity Detection](entity-detection.md), [Brain-First Lookup](brain-first-lookup.md)*

View File

@@ -0,0 +1,85 @@
# Brain-First Lookup Protocol
## Goal
Check the brain before calling ANY external API. The brain almost always has
something. External APIs fill gaps, they don't start from scratch.
## What the User Gets
Without this: the agent calls Brave Search for someone you've had 12 meetings with.
You get a LinkedIn summary instead of your relationship history.
With this: the agent pulls your compiled truth, recent timeline entries, and
shared context before doing anything else. External APIs only fill gaps.
## Implementation
```
lookup(name_or_topic):
// STEP 1: Keyword search (fast, works day one, no embeddings needed)
results = gbrain search "{name_or_topic}"
if results.length > 0:
page = gbrain get {results[0].slug}
return page // done, brain had it
// STEP 2: Hybrid search (needs embeddings, finds semantic matches)
results = gbrain query "what do we know about {name_or_topic}"
if results.length > 0:
page = gbrain get {results[0].slug}
return page
// STEP 3: Direct slug (if you know or can guess the slug)
page = gbrain get "people/{slugify(name_or_topic)}"
if page: return page
// STEP 4: External API (FALLBACK ONLY)
// Only reach here if brain has nothing
return external_search(name_or_topic)
```
**This is mandatory.** An agent that calls Brave Search before checking the brain
is wasting money and giving worse answers.
## Why Brain First
The brain has context no external API can provide:
- Relationship history (how you know them, what you discussed)
- Your own assessments (what you think of them, not their LinkedIn bio)
- Meeting transcripts (what was said, what was decided)
- Cross-references (who they know, what companies they're connected to)
- Timeline (what changed recently, what's trending)
A LinkedIn scrape gives you their job title. The brain gives you: "co-founded
Brex, you had coffee with him 3 times, last discussed the payments infrastructure
thesis, he's interested in your take on AI agents."
## Tricky Spots
1. **Try keyword first, then hybrid.** Keyword search works without embeddings
(day one). Hybrid search needs embeddings but finds semantic matches. Try
both in sequence.
2. **Fuzzy slug matching.** `gbrain get` supports fuzzy matching. If the exact
slug doesn't exist, it suggests alternatives. Use this for name variants
("Pedro" → "pedro-franceschi").
3. **Don't skip for "simple" questions.** Even "what's Acme Corp's address?"
should check the brain first. The brain might have it, and the lookup adds
no latency (< 100ms for keyword search).
4. **Load compiled truth + recent timeline.** The compiled truth gives you the
state of play in 30 seconds. The timeline gives you what changed recently.
Both together = full context.
## How to Verify
1. Ask about someone in the brain. Verify the agent searched the brain FIRST
(check tool call order in the response).
2. Ask about someone NOT in the brain. Verify the agent searched the brain,
found nothing, THEN fell back to external search.
3. Ask the same question twice. Second time should be instant (brain has it).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Brain-Agent Loop](brain-agent-loop.md), [Search Modes](search-modes.md)*

View File

@@ -0,0 +1,75 @@
# Brain vs Memory vs Session
## Goal
Know what goes in GBrain, what goes in agent memory, and what stays in session context -- so every piece of information lands in the right layer.
## What the User Gets
Without this: people dossiers get stored in agent memory (lost on agent reset), user preferences get stored in GBrain (cluttering knowledge pages), and the agent re-asks questions it already knows the answer to. With this: world knowledge persists in the brain, operational state persists in agent memory, and the agent never puts information in the wrong layer.
## Implementation
```
on new_information(info):
# Three layers, three purposes -- route to the right one
if info.is_about_the_world:
# GBRAIN: people, companies, deals, meetings, concepts, ideas
# This is world knowledge -- facts about entities external to the agent
gbrain put <slug> --content "..."
# Examples:
# "Pedro is CEO of Brex" -> gbrain (person page)
# "Brex raised Series D at $12B" -> gbrain (company page)
# "Tuesday's meeting covered Q2" -> gbrain (meeting page)
# "The meatsuit maintenance tax" -> gbrain (originals page)
elif info.is_about_operations:
# AGENT MEMORY: preferences, decisions, tool config, session continuity
# This is how the agent operates -- not facts about the world
memory_write(info)
# Examples:
# "User prefers concise formatting" -> agent memory
# "Deploy to staging before prod" -> agent memory
# "Use dark mode in code blocks" -> agent memory
# "API key for Crustdata goes in .env" -> agent memory
elif info.is_current_conversation:
# SESSION CONTEXT: what was just said, current task, immediate state
# This is automatic -- already in the conversation window
# No storage action needed
# Examples:
# "We were just discussing the board deck" -> session
# "You asked me to review this PR" -> session
# "The file I just shared" -> session
# Lookup routing:
on user_asks(question):
if question.about_person or question.about_company or question.about_meeting:
gbrain search "{entity}" # -> world knowledge
gbrain get <slug>
elif question.about_preference or question.about_how_to_operate:
memory_search("{topic}") # -> operational state
elif question.about_current_context:
# Already in session -- just reference conversation history
pass
```
## Tricky Spots
1. **Don't store people in agent memory.** "Pedro prefers email over Slack" feels like a preference, but it's a fact about Pedro -- it goes in GBrain on Pedro's page. Agent memory is for the agent's own operational state, not facts about people in the world.
2. **Don't store user preferences in GBrain.** "User likes bullet points over paragraphs" is about how the agent should behave, not about the world. It goes in agent memory. GBrain pages are for entities, not for agent configuration.
3. **Synthesis of external ideas goes in GBrain.** "User's take on Peter Thiel's zero-to-one framework" is the user's original thinking -- it goes in GBrain under originals/, not in agent memory.
4. **Agent memory doesn't survive agent resets on some platforms.** Critical world knowledge MUST be in GBrain, which is durable. If the agent loses memory, the brain still has everything.
5. **When in doubt, ask: is this about the world or about how to operate?** World -> GBrain. Operations -> agent memory. Current conversation -> session.
## How to Verify
1. Ask the agent "Who is Pedro?" -- confirm it runs `gbrain search` or `gbrain get`, not `memory_search`. Person lookup should hit GBrain.
2. Ask the agent "How should I format responses?" -- confirm it checks agent memory, not GBrain. Preferences are operational state.
3. Check that no person or company pages exist in agent memory storage. Run `memory_search "person"` -- it should return preferences, not dossiers.
4. Check that GBrain doesn't contain pages about agent behavior. Run `gbrain search "user prefers"` -- it should return nothing (preferences belong in agent memory).
5. After an agent reset, confirm GBrain knowledge is still accessible. Run `gbrain get <any_slug>` -- world knowledge should survive the reset.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,137 @@
# Compiled Truth + Timeline Pattern
## Goal
Every brain page has two zones: compiled truth (current synthesis, rewritten as
evidence changes) and timeline (append-only evidence trail, never edited).
## What the User Gets
Without this: brain pages are append-only logs. To understand a person, you read
200 timeline entries. The answer is buried in entry #147.
With this: the compiled truth gives you the state of play in 30 seconds. The
timeline is the proof. Six months of entries compress into a one-paragraph
assessment that's always current.
## Implementation
### Page Structure
```markdown
---
type: person
title: Sarah Chen
tags: [engineering, acme-corp]
---
## Executive Summary
One paragraph. How you know them, why they matter.
## State
VP Engineering at Acme Corp. Managing 45-person team. Reports to CEO.
## What They Believe
Strong opinions on test coverage. "Ship it when the tests pass, not before."
## What They're Building
Leading the API migration from REST to GraphQL. Target: Q3 completion.
## Assessment
Sharp technical leader. Under-appreciated internally. Watch for signs of burnout.
## Trajectory
Ascending. Likely CTO track if the migration succeeds.
## Relationship
Met through Pedro. Had coffee 3x. Last: discussed API architecture thesis.
## Contact
sarah@acmecorp.com | @sarahchen | linkedin.com/in/sarahchen
---
## Timeline
- **2026-04-07** | Met at team sync. Discussed API migration timeline.
Seemed energized about GraphQL pivot.
[Source: Meeting notes, 2026-04-07 2:00 PM PT]
- **2026-04-03** | Mentioned in email re Q2 planning. Taking lead on ops.
[Source: Gmail, sarah@acmecorp.com, 2026-04-03 10:30 AM PT]
- **2026-03-15** | First meeting. Intro from Pedro. Strong technical background.
[Source: User, direct conversation, 2026-03-15 3:00 PM PT]
```
### Updating a Page
```
update_brain_page(slug, new_info, source):
page = gbrain get {slug}
// TIMELINE: always APPEND (never edit existing entries)
gbrain add_timeline_entry {slug} {
date: today,
summary: new_info.summary,
detail: new_info.detail,
source: format_source(source) // [Source: who, channel, date time tz]
}
// COMPILED TRUTH: REWRITE (not append)
// Read the existing compiled truth
// Integrate new information
// Write the updated synthesis
updated_truth = rewrite_compiled_truth(page.compiled_truth, new_info)
gbrain put {slug} {
compiled_truth: updated_truth,
// timeline is NOT passed — it's managed by add_timeline_entry
}
```
### The Rules
| Zone | Action | Explanation |
|------|--------|-------------|
| Compiled truth | **REWRITE** | Current synthesis. Changes when evidence changes. |
| Timeline | **APPEND** | Evidence trail. Never edited, only added to. |
**Every compiled truth claim must trace to timeline entries.** If the Assessment
says "under-appreciated internally," there should be timeline entries that
support that claim.
## Tricky Spots
1. **REWRITE means rewrite, not append.** Don't add a new paragraph to compiled
truth. Rewrite the entire section with the new information integrated. Old
assessments that are no longer accurate should be updated, not kept alongside
contradictory new ones.
2. **Timeline entries are immutable.** Never edit a timeline entry. If information
turns out to be wrong, add a NEW entry correcting it:
`- 2026-04-10 | Correction: Sarah is VP Eng, not CTO. Previous entry was wrong.`
3. **GBrain search weights compiled truth higher.** `gbrain query` returns compiled
truth chunks with higher relevance than timeline chunks. This means the freshest
synthesis surfaces first in search results.
4. **The --- separator matters.** GBrain uses the first standalone `---` after
frontmatter to split compiled_truth from timeline. Everything above is compiled
truth, everything below is timeline.
5. **Don't skip the Assessment section.** The assessment is the value. "Strong
technical leader" is something no API can provide. It's YOUR read on this
person. That's what makes the brain page better than LinkedIn.
## How to Verify
1. **Update a person page.** Add new meeting info. Check: compiled truth was
REWRITTEN (not appended), timeline has new entry at the top.
2. **Search for the person.** `gbrain query "Sarah Chen"`. The compiled truth
(current synthesis) should appear first, not a random timeline entry.
3. **Check traceability.** Every claim in compiled truth should have a
corresponding timeline entry. Read both sections and verify.
4. **Check immutability.** After update, old timeline entries should be unchanged.
Dates, sources, and content should match the originals exactly.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Source Attribution](source-attribution.md), [Entity Detection](entity-detection.md)*

View File

@@ -0,0 +1,136 @@
# Content and Media Ingestion
## Goal
YouTube videos, social media, PDFs, and documents become searchable brain pages with the agent's own analysis and full cross-references to every entity mentioned.
## What the User Gets
Without this: media links are bookmarks that decay -- you remember watching a video but can't find what was said, who said it, or why it mattered. With this: every piece of media is a permanent brain page with the agent's analysis layered on top, every mentioned entity gets a back-link, and the full content is searchable forever.
## Implementation
```
on user_shares_media(url_or_file):
# PATTERN 1: YouTube Video Ingestion
if media.type == "youtube":
# Step 1: Get FULL transcript with speaker diarization
# WHO said WHAT -- not just a wall of text
# Use Diarize.io or equivalent service
transcript = diarize(video_url) # speaker-attributed transcript
# NEVER use YouTube's auto-generated summary or AI summary
# Step 2: Agent writes OWN analysis (this is the value)
# NOT a summary. NOT regurgitation. The agent's TAKE:
# - What matters and why (given the user's worldview)
# - Key quotes attributed to specific speakers
# - Connections to existing brain pages
# - Implications and follow-up angles
analysis = agent_analyze(transcript, user_context)
# Step 3: Create brain page
slug = f"media/youtube/{video_slug}"
gbrain put <slug> --content """
# {title}
**Channel:** {channel} | **Date:** {date} | **Link:** {url}
## Analysis
{agent_analysis}
## Key Quotes
- **{Speaker}** ({timestamp}): "{quote}" -- {why_it_matters}
---
## Full Transcript
{diarized_transcript}
"""
# Step 4: Extract and cross-reference entities
for person in transcript.mentioned_people:
gbrain add_link <slug> <person_slug>
gbrain add_link <person_slug> <slug>
gbrain add_timeline_entry <person_slug> \
--entry "Discussed in {video_title}: {what_was_said}" \
--source "YouTube: {url}"
# PATTERN 2: Social Media Bundles
elif media.type == "tweet" or media.type == "social":
# Don't just save a tweet -- reconstruct FULL context
bundle = {
"original": fetch_tweet(url),
"thread": reconstruct_thread(url), # quoted tweets, replies
"linked_articles": fetch_linked_urls(), # fetch and summarize
"engagement": get_engagement_data(), # what resonated
}
slug = f"media/social/{platform}-{author}-{date}"
gbrain put <slug> --content """
# {author}: {topic}
{agent_analysis_of_full_bundle}
## Thread
{reconstructed_thread}
## Linked Articles
{article_summaries}
---
## Raw
{original_tweet_text}
"""
# Extract entities and cross-reference
for entity in bundle.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
# PATTERN 3: PDFs and Documents
elif media.type == "pdf" or media.type == "document":
# OCR if needed (scanned PDFs)
content = ocr_if_needed(file) or extract_text(file)
# For books and long-form:
slug = f"sources/{document_slug}"
gbrain put <slug> --content """
# {title}
**Author:** {author} | **Date:** {date}
## Chapter Summaries
{per_chapter_summary}
## Key Quotes
- p.{page}: "{quote}" -- {why_it_matters}
## Cross-References
{links_to_brain_pages_for_people_and_concepts}
---
## Source
{full_text_or_key_sections}
"""
for entity in document.mentioned_entities:
gbrain add_link <slug> <entity_slug>
gbrain add_link <entity_slug> <slug>
# Always sync after ingestion
gbrain sync
```
## Tricky Spots
1. **Always FULL transcript, never AI summary.** YouTube's auto-summary and AI-generated summaries lose the texture: who said what, exact phrasing, tone, what was left unsaid. The full diarized transcript is the evidence base. The agent's analysis goes above it.
2. **The agent's OWN analysis is the value, not regurgitation.** "The video discussed AI safety" is worthless. "Dario made a specific claim about compute scaling that contradicts what Ilya said in the NeurIPS talk -- see media/youtube/ilya-neurips-2025" is useful. The analysis connects the new media to the existing brain.
3. **Social media is a bundle, not a single tweet.** A tweet without its thread, quoted tweets, linked articles, and engagement context is a fragment. Reconstruct the full context before creating the brain page.
4. **Cross-references make media pages alive.** A YouTube page without back-links to the people and companies mentioned is a dead archive. Every mentioned entity gets a link and a timeline entry.
5. **Over time, `media/` becomes a searchable archive.** Every video, podcast, talk, interview, article, and tweet the user has consumed, with the agent's commentary layered on top. This is the memex at full power.
## How to Verify
1. Ingest a YouTube video. Run `gbrain get media/youtube/{slug}`. Confirm the page has: the agent's analysis (not just a summary), key quotes with speaker attribution, and the full diarized transcript.
2. Run `gbrain get_links media/youtube/{slug}`. Confirm back-links exist to brain pages for every person and company mentioned in the video.
3. Pick a person mentioned in the video. Run `gbrain get <person_slug>`. Confirm their timeline has a new entry referencing the video with specific context.
4. Ingest a tweet. Confirm the brain page includes the thread context, linked article summaries, and entity cross-references -- not just the tweet text.
5. Run `gbrain search "{topic_from_video}"`. Confirm the media page appears in search results (verifies the content is indexed and searchable).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,193 @@
# Reference Cron Schedule
## Goal
A production brain runs 20+ recurring jobs that keep it alive, current, and
compounding. This guide shows the schedule, the patterns, and how to set it up.
## What the User Gets
Without this: the brain only updates when you manually ingest data. Pages go
stale, entities are thin, citations break, and the agent answers from old context.
With this: the brain maintains itself. Email, social, calendar, and meetings
flow in automatically. Thin pages get enriched overnight. Broken citations get
fixed. You wake up and the brain is smarter than when you went to sleep.
## The Schedule
| Frequency | Job | Brain Interaction | Recipe |
|-----------|-----|-------------------|--------|
| Every 30 min | Email monitoring | Search sender, update people pages | [email-to-brain](../../recipes/email-to-brain.md) |
| Every 30 min | X/Twitter collection | Create/update media pages, entity extraction | [x-to-brain](../../recipes/x-to-brain.md) |
| 3x/day (weekdays) | Meeting sync | Full ingestion + attendee propagation | [meeting-sync](../../recipes/meeting-sync.md) |
| Weekly | Calendar sync | Daily files + attendee enrichment | [calendar-to-brain](../../recipes/calendar-to-brain.md) |
| Daily AM | Morning briefing | Search calendar attendees, deal status, active threads | [briefing skill](../../skills/briefing/SKILL.md) |
| Weekly | Brain maintenance | `gbrain doctor`, embed stale, orphan detection | [maintain skill](../../skills/maintain/SKILL.md) |
| Nightly | Dream cycle | Entity sweep, enrich thin spots, fix citations | See below |
## Implementation: Setting Up Cron Jobs
```bash
# Email collector — every 30 minutes
*/30 * * * * cd /path/to/email-collector && node email-collector.mjs collect && node email-collector.mjs digest
# X/Twitter collector — every 30 minutes
*/30 * * * * cd /path/to/x-collector && node x-collector.mjs collect >> /tmp/x-collector.log 2>&1
# Meeting sync — 10 AM, 4 PM, 9 PM on weekdays
0 10,16,21 * * 1-5 cd /path/to/meeting-sync && node meeting-sync.mjs >> /tmp/meeting-sync.log 2>&1
# Calendar sync — Sundays at 10 AM
0 10 * * 0 cd /path/to/calendar-sync && node calendar-sync.mjs --start $(date -v-7d +%Y-%m-%d) --end $(date +%Y-%m-%d)
# Brain health — weekly Mondays at 6 AM
0 6 * * 1 gbrain doctor --json >> /tmp/gbrain-health.log 2>&1 && gbrain embed --stale
# Dream cycle — nightly at 2 AM
0 2 * * * /path/to/dream-cycle.sh
```
### Quiet Hours Gate (MANDATORY)
Every cron job that sends notifications MUST check quiet hours first.
See [Quiet Hours](quiet-hours.md) for the full pattern.
```bash
# In every cron script:
if ! bash scripts/quiet-hours-gate.sh; then
mkdir -p /tmp/cron-held
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
exit 0
fi
# Not quiet hours — send normally
```
### Travel-Aware Timezone Handling
The agent reads your calendar for flights, hotels, and out-of-office blocks to
infer your current location and timezone. All times shown in YOUR local timezone.
```
// Example: user flew to Tokyo
// 2 PM Pacific = 3 AM Tokyo = quiet hours
// Hold the notification, fold into morning briefing
get_user_timezone():
calendar = gbrain search "flight" --type calendar --recent 7d
if recent_flight:
return infer_timezone(flight.destination)
return config.default_timezone // fallback: US/Pacific
```
When you travel: cron jobs that would fire during your waking hours at home but
hit your sleeping hours at the destination get held and folded into the next
morning briefing. Zero config change needed.
## The Dream Cycle
The most important cron job. Runs while you sleep.
### What It Does
```
dream_cycle():
// Phase 1: Entity Sweep
conversations = get_todays_conversations()
for message in conversations:
entities = detect_entities(message)
for entity in entities:
page = gbrain search "{entity.name}"
if not page:
create_page(entity) // new entity, create + enrich
elif page.is_thin():
enrich_page(entity) // thin page, fill it out
else:
update_timeline(entity) // existing page, add today's mentions
// Phase 2: Fix Broken Citations
pages = gbrain list --type person --limit 100
for page in pages:
for entry in page.timeline:
if not entry.has_source_attribution():
fix_citation(entry) // add [Source: ...] where missing
if entry.has_tweet_url() and not entry.url_is_valid():
fix_url(entry) // broken tweet links
// Phase 3: Consolidate Memory
patterns = detect_patterns_across_conversations()
for pattern in patterns:
promote_to_memory(pattern) // ephemeral → durable knowledge
// Phase 4: Sync
gbrain sync --no-pull --no-embed
gbrain embed --stale
```
### Setting Up the Dream Cycle
**OpenClaw:** Ships with DREAMS.md as a default skill. Three phases (light,
deep, REM) run automatically during quiet hours.
**Hermes Agent:**
```bash
/cron add "0 2 * * *" "Dream cycle: search today's sessions for
entities I mentioned. For each person, company, or idea: check
if a brain page exists (gbrain search), create or update it if
thin. Fix any broken citations. Then consolidate: read MEMORY.md,
promote important signals, remove stale entries."
--name "nightly-dream-cycle"
```
**Claude Code / Custom agents:** Create a script:
```bash
#!/bin/bash
# dream-cycle.sh
# Check quiet hours (should be quiet — that's when we run)
echo "Dream cycle starting at $(date)"
# Phase 1: Entity sweep (spawn sub-agent)
# Read today's conversation logs, extract entities, update brain
# Phase 2: Citation hygiene
gbrain doctor --json | jq '.checks[] | select(.status=="warn")'
# Phase 3: Embed any stale content
gbrain embed --stale
echo "Dream cycle complete at $(date)"
```
## Tricky Spots
1. **The dream cycle is NOT optional.** Without it, signal leaks out of every
conversation. With it, nothing is lost. This is the difference between an
agent that forgets and one that remembers.
2. **Quiet hours gate on EVERY notification job.** If you skip it, the user
gets pinged at 3 AM. One 3 AM ping and they'll disable the whole system.
3. **Don't over-cron.** 20+ jobs sounds like a lot. Start with: email (30 min),
dream cycle (nightly), brain health (weekly). Add more as you add
integration recipes.
4. **Timezone changes are automatic.** Don't make the user reconfigure cron
when they travel. Read the calendar, infer the timezone, adjust delivery.
5. **Held messages MUST be picked up.** If quiet hours hold a notification,
the morning briefing MUST include it. Otherwise information is lost.
## How to Verify
1. **Quiet hours:** Set quiet hours to current hour. Run a notification cron.
Verify output went to `/tmp/cron-held/`, not to messaging.
2. **Dream cycle:** Run the dream cycle manually. Check that thin entity pages
got enriched and broken citations were fixed.
3. **Email collector cron:** Wait 30 minutes. Check `data/digests/` for new digest.
4. **Morning briefing:** Check that held messages appear in the briefing.
5. **Health check:** Run `gbrain doctor --json`. All checks should pass.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Quiet Hours](quiet-hours.md), [Operational Disciplines](operational-disciplines.md)*

View File

@@ -0,0 +1,146 @@
# Deterministic Collectors: Code for Data, LLMs for Judgment
## Goal
Separate mechanical work (100% reliable code) from analytical work (LLM judgment) so that deterministic tasks never fail probabilistically.
## What the User Gets
Without this: the LLM generates Gmail links, formats tables, and tracks state.
It follows the rule for the first 10 items, then drops a link on item 11. You
write "NO EXCEPTIONS" in the prompt. It still fails. 90% reliability over 20
items means visible failures twice per day. Trust is destroyed.
With this: code handles URLs, formatting, and state (100% reliable). The LLM
reads pre-formatted data and adds judgment, classification, and enrichment.
Links are never wrong because the LLM never generates them.
## Implementation
```
// The pattern: code collects, LLM analyzes
// STEP 1: Deterministic collector (script, no LLM calls)
collector_run():
messages = gmail_api.fetch_unread()
for msg in messages:
structured = {
id: msg.id,
from: msg.sender,
subject: msg.subject,
snippet: msg.snippet,
gmail_link: f"https://mail.google.com/mail/u/?authuser={account}#inbox/{msg.id}",
gmail_markdown: f"[Open in Gmail]({gmail_link})",
is_signature: regex_match(msg, DOCUSIGN_PATTERNS),
is_noise: regex_match(msg, NOISE_PATTERNS),
is_new: msg.id not in state.seen_ids
}
store(structured)
state.seen_ids.add(msg.id)
generate_markdown_digest(structured_messages)
// STEP 2: LLM reads the pre-formatted digest
llm_analyze():
digest = read("data/digests/today.md") // links already baked in
classify_urgency(digest) // judgment call
add_commentary(digest) // contextual analysis
run_brain_enrichment(notable_entities) // gbrain search + update
draft_replies(urgent_items) // creative work
surface_to_user(final_output) // delivery
// STEP 3: Wire into cron
cron_job():
collector_run() // fast, cheap, deterministic
llm_analyze() // slower, expensive, creative
```
### The Architecture
```
+-----------------------------+ +------------------------------+
| Deterministic Collector |---->| LLM Agent |
| (Node.js / Python script) | | |
| | | - Read the pre-formatted |
| - Pull data from API | | digest |
| - Store structured JSON | | - Classify items |
| - Generate links/URLs | | - Add commentary |
| - Detect patterns (regex) | | - Run brain enrichment |
| - Track state (seen/new) | | - Draft replies |
| - Output markdown digest | | - Surface to user |
| | | |
| CODE — deterministic, | | AI — judgment, context, |
| never forgets | | creativity |
+-----------------------------+ +------------------------------+
```
### File Structure
```
scripts/email-collector/
├── email-collector.mjs # No LLM calls, no external deps
├── data/
│ ├── state.json # Last pull timestamp, known IDs, pending signatures
│ ├── messages/ # Structured JSON per day
│ │ └── 2026-04-09.json
│ └── digests/ # Pre-formatted markdown
│ └── 2026-04-09.md
```
### Where the Pattern Applies
| Signal Source | Collector Generates | LLM Adds |
|--------------|-------------------|----------|
| **Email** | Gmail links, sender metadata, signature detection | Urgency classification, enrichment, reply drafts |
| **X/Twitter** | Tweet links, engagement metrics, deletion detection | Sentiment analysis, narrative detection, content ideas |
| **Calendar** | Event links, attendee lists, conflict detection | Prep briefings, meeting context from brain |
| **Slack** | Channel links, thread links, mention detection | Priority classification, action item extraction |
| **GitHub** | PR/issue links, diff stats, CI status | Code review context, priority assessment |
### The Principle
If a piece of output MUST be present and MUST be formatted correctly every
time, generate it in code. If a piece of output requires judgment, context,
or creativity, generate it with the LLM. Don't ask the LLM to do both in
the same pass.
## Tricky Spots
1. **LLMs forget links -- bake them in code.** The LLM will follow the
"include a Gmail link" rule for the first 10 items, then silently drop
it on item 11. No amount of prompt engineering fixes probabilistic
formatting over long outputs. The fix: generate every link in the
collector script. The LLM reads pre-formatted markdown where links are
already embedded. It can't forget what it didn't generate.
2. **Noise filtering must be deterministic.** Regex-based noise detection
(newsletters, automated receipts, marketing) belongs in the collector,
not the LLM. The LLM might classify a newsletter as "possibly important"
on one run and "noise" on the next. Code classifies the same input the
same way every time.
3. **Atomic writes prevent corruption.** The collector writes to a state
file (`state.json`) that tracks which messages have been seen. If the
script crashes mid-write, the state file can be corrupted. Write to a
temp file first, then rename atomically. This also prevents the LLM
from reading a partial digest if the cron fires during a collection run.
## How to Verify
1. **Run the collector and check every link.** Execute the collector script
manually. Open the generated digest. Click every `[Open in Gmail]` link
(or equivalent). Every single link must resolve to the correct item. If
any link is broken or missing, the collector has a bug.
2. **Verify noise filtering is consistent.** Run the collector twice on the
same input data. The noise classification (is_noise field) must be
identical both times. If it varies, a probabilistic element leaked into
the deterministic layer.
3. **Verify the LLM reads structured output.** Run the full pipeline
(collector then LLM). Check that the LLM's analysis references data
from the structured digest, not from its own generation. The links in
the final output should be identical to the links in the digest file.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,151 @@
# Diligence Ingestion: Data Room to Brain Pages
## Goal
Turn pitch decks, financial models, and data room materials into searchable, cross-referenced brain pages with bull/bear analysis.
## What the User Gets
Without this: pitch decks sit in email attachments. Financial models in Google
Drive. No cross-reference to the company brain page. You can't search "what
were the key metrics from Acme Corp's Series A deck?"
With this: every data room document is extracted, diarized, cross-referenced to
the company page, and searchable. Index.md gives you the bull/bear case at a
glance. `gbrain query "Acme Corp revenue growth"` finds the exact chart.
## Implementation
Recognize data room materials by PDF filenames containing "Data Deck", "Intro
Deck", "Data Room", "Cap Table", "Financial Model", "Investor Memo", "Pitch
Deck", or series round names. Spreadsheet tabs with Revenue, Retention, Cohorts,
CAC, Gross Margin, Unit Economics, ARR. User language like "data room",
"diligence", "deck", "pitch", "fundraise materials".
### The 9-Step Pipeline
**Step 1: Identify the Company.**
From the document content or filename, identify the company name.
Check if `brain/companies/{slug}.md` exists.
**Step 2: Create Diligence Directory.**
```bash
mkdir -p brain/diligence/{company-slug}/.raw
```
**Step 3: Extract Content.**
- **PDFs:** Use PDF extraction tool. For scanned/image-heavy PDFs,
use OCR (e.g., Mistral OCR or similar).
- **Spreadsheets:** Export each sheet as CSV. For Google Sheets:
```
https://docs.google.com/spreadsheets/d/{ID}/gviz/tq?tqx=out:csv&sheet={Sheet Name}
```
**Step 4: Diarize and Save.**
Write extracted content to `brain/diligence/{company}/{doc-name}.md`:
- Document title and type
- Section-by-section breakdown with key metrics
- Notable footnotes or caveats
- Raw data tables where relevant
**Step 5: Save Raw Files.**
Copy original PDFs/files to `brain/diligence/{company}/.raw/`
Preserve originals for reference. The diarized version is for search.
**Step 6: Create or Update index.md.**
Every diligence directory needs an `index.md`:
```markdown
# {Company Name} — Diligence
## Round Details
- Stage: Series A
- Amount: $10M
- Date: 2026-04
## Document Inventory
- [Pitch Deck](pitch-deck.md) — 25 slides, company overview + traction
- [Financial Model](financial-model.md) — 5 tabs, 3-year projections
- [Cap Table](cap-table.md) — current ownership + option pool
## Key Findings
- Revenue growing 30% MoM for last 6 months
- CAC payback period: 4 months
- Net retention: 135%
## Bull Case
- Strong product-market fit signal (NPS 72)
- Expanding into adjacent vertical
## Bear Case
- Single customer represents 40% of revenue
- Burn rate increased 3x last quarter
## Open Questions
- What's the path to profitability?
- How defensible is the moat?
```
**Step 7: Enrich Company Brain Page.**
Update `brain/companies/{slug}.md`:
- Add document sources to frontmatter
- Update compiled truth with key findings
- Add "See Also" link to diligence directory
- If no company page exists, create one via the enrich skill
**Step 8: Commit.**
```bash
cd brain/ && git add -A && git commit -m "diligence: {Company} — {doc type} ingestion" && git push
```
**Step 9: Publish (if asked).**
When the user wants a shareable brief, create a password-protected
published version. Strip internal notes and raw assessment language.
### Quality Bar
A good diligence page reads like an intelligence assessment:
- **What they say** vs **what the data shows** (the gap is the insight)
- Explicit bull/bear case (not just a summary)
- Key metrics highlighted, not buried
- Open questions that need answers before decision
## Tricky Spots
1. **PDF extraction is lossy.** Scanned decks and image-heavy PDFs lose
tables and charts during extraction. Always check the diarized output
against the original `.raw/` file. If key metrics are missing, re-extract
with OCR or transcribe manually.
2. **Idempotency on re-ingestion.** If the user sends an updated deck for
the same company, don't create a duplicate directory. Check for an existing
`brain/diligence/{company-slug}/` and update in place. Append a version
suffix to the document file if the old version should be preserved.
3. **index.md completeness.** The index.md is the entry point for the entire
diligence package. If it's missing the bull/bear case or open questions,
the diligence is incomplete. Always generate all sections even if some
require judgment calls -- flag uncertain assessments explicitly.
## How to Verify
1. **Search for key metrics.** After ingestion, run
`gbrain search "revenue growth"` or `gbrain search "{company name} CAC"`.
The diarized content should appear in results. If it doesn't, the sync
or embedding step was missed.
2. **Check the company page cross-reference.** Open
`brain/companies/{slug}.md` and verify it links to the diligence directory.
The compiled truth section should include key findings from the deck.
3. **Verify index.md has all sections.** Open
`brain/diligence/{company}/index.md` and confirm it has Round Details,
Document Inventory, Key Findings, Bull Case, Bear Case, and Open Questions.
Missing sections mean the pipeline stopped early.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,103 @@
# Enrichment Pipeline
## Goal
Enrich brain pages from external APIs with tiered spend -- full pipeline for key people, light touch for passing mentions, raw data preserved for auditability.
## What the User Gets
Without this: brain pages are thin shells with only what the user manually typed, API calls are wasted on nobodies, and enrichment data vanishes after the agent session ends. With this: key people have rich, multi-source portraits; spend scales to importance; raw API responses are preserved for re-processing; and cross-references connect the entire graph.
## Implementation
```
on enrich(entity, trigger):
# trigger: meeting mention, email thread, social interaction, user request
# Step 1: Identify entities from the incoming signal
entities = extract_entities(signal)
# people names, company names, associations
# Step 2: Check brain state -- UPDATE or CREATE path?
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
page = gbrain get <entity_slug>
path = "UPDATE"
else:
path = "CREATE"
# Step 3: Determine tier -- scale spend to importance
tier = classify_tier(entity):
# Tier 1 (10-15 API calls): key people, inner circle, business partners,
# portfolio companies. Full pipeline, ALL data sources.
# Tier 2 (3-5 API calls): notable people, occasional interactions.
# Web search + social + brain cross-reference.
# Tier 3 (1-2 API calls): minor mentions, everyone else worth tracking.
# Brain cross-reference + social lookup if handle known.
# Step 4: Run external lookups (priority order, stop when enough signal)
data = {}
data["brain"] = gbrain search "{entity.name}" # Always first (free)
if tier <= 2:
data["web"] = brave_search("{entity.name}") # Background, press, talks
if tier <= 2:
data["twitter"] = twitter_lookup(entity.handle) # Beliefs, building, network
if tier == 1:
data["linkedin"] = crustdata_enrich(entity.name) # Career, connections
data["research"] = happenstance_research(entity) # Career arcs, web presence
data["funding"] = captain_api(entity.company) # Funding, valuation, team
data["meetings"] = circleback_search(entity.name) # Transcript search
data["contacts"] = google_contacts(entity.email) # Contact data
# Step 5: Store raw data (auditable, re-processable)
gbrain put_raw_data <entity_slug> \
--data '{"sources": {"crustdata": {"fetched_at": "...", "data": {...}}, ...}}'
# Overwrite on re-enrichment, don't append
# Step 6: Write to brain page
if path == "CREATE":
gbrain put <entity_slug> --content "<compiled_truth_from_all_sources>"
gbrain add_timeline_entry <entity_slug> --entry "Page created via enrichment"
elif path == "UPDATE":
# Append timeline, update compiled truth ONLY if materially new
gbrain add_timeline_entry <entity_slug> --entry "Enriched: {new_signal}"
# Flag contradictions -- don't silently resolve them
# Step 7: Cross-reference the graph
gbrain add_link <person_slug> <company_slug> # person -> company
gbrain add_link <company_slug> <person_slug> # company -> person
gbrain add_link <person_slug> <deal_slug> # person -> deal
# Every entity page links to every other entity page that references it
# People page sections (not a LinkedIn profile -- a living portrait):
# Executive Summary, State, What They Believe, What They're Building,
# What Motivates Them, Assessment, Trajectory, Relationship, Contact, Timeline
# Facts are table stakes. TEXTURE is the value.
# Extract texture, not just facts:
# Opinion expressed? -> What They Believe
# Building or shipping? -> What They're Building
# Emotion expressed? -> What Makes Them Tick
# Who did they engage with? -> Network / Relationship
# Recurring topic? -> Hobby Horses
# Committed to something? -> Open Threads
# Energy level? -> Trajectory
```
## Tricky Spots
1. **Don't overwrite human-written assessments.** If the user wrote an Assessment section with their own read on someone, API enrichment NEVER overwrites it. API data goes into State, Contact, Timeline. The user's assessment is sacrosanct.
2. **Don't re-enrich the same page more than once per week.** Check `put_raw_data` timestamps before running the pipeline again. Enrichment is expensive and data doesn't change that fast.
3. **LinkedIn connection count < 20 means wrong person.** Crustdata sometimes returns a different person with the same name. If the LinkedIn profile has fewer than 20 connections, it's almost certainly a false match. Discard it.
4. **X/Twitter is the most underrated data source.** When you have someone's handle, their tweets reveal beliefs, what they're building, hobby horses, network (reply patterns), and trajectory (posting frequency, tone shifts). This is richer than LinkedIn for "What They Believe" and "What Makes Them Tick."
5. **Cross-references are not optional.** After enriching a person, update their company page. After enriching a company, update founder pages. An enriched page without cross-links is a dead end in the graph.
## How to Verify
1. Enrich a Tier 1 person. Run `gbrain get <slug>` and confirm the page has Executive Summary, State, What They Believe, Contact, and Timeline sections populated from multiple sources.
2. Run `gbrain get_raw_data <slug>`. Confirm raw API responses are stored with `sources.{provider}.fetched_at` timestamps.
3. Run `gbrain get_links <slug>`. Confirm cross-reference links exist to the person's company page, deal pages, and related entities.
4. Check a page that was enriched AND has a user-written Assessment. Confirm the Assessment section was preserved, not overwritten by API data.
5. Try to re-enrich the same person. Confirm the system checks the `fetched_at` timestamp and skips if less than a week old.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,190 @@
# Entity Detection: Run It on Every Message
## Goal
Every inbound message gets scanned for original thinking AND entity mentions so the brain grows on every conversation, automatically.
## What the User Gets
Without this: the agent answers questions but forgets everything. You mention
Pedro in a meeting, and next week the agent doesn't know who Pedro is.
With this: every person, company, and idea mentioned in conversation gets a
brain page. Next time Pedro comes up, the agent already has context. The
brain compounds.
## Implementation
Spawn a lightweight sub-agent on EVERY inbound message. Do NOT wait for it
to finish before responding. It runs in parallel.
```
on_every_message(message_text, source_context):
// 1. SPAWN ASYNC — don't block the response
spawn_subagent({
model: "sonnet-class", // cheap + fast, not opus
timeout: 120, // seconds
task: build_detection_prompt(message_text, source_context)
})
// 2. RESPOND TO USER NORMALLY
// The sub-agent runs in the background
```
### The Detection Prompt
```
build_detection_prompt(text, source):
return `
SIGNAL DETECTION — scan this message for ideas AND entities:
Message: "${text}"
Source: [Source: User, ${source.topic}, ${source.platform}, ${source.timestamp}]
STEP 1 — IDEAS FIRST (highest priority):
Is the user expressing an original thought, observation, thesis, or framework?
If yes:
- Create or update brain/originals/{slug}.md
- Use the user's EXACT phrasing (the language IS the insight)
- "The ambition-to-lifespan ratio has never been more broken" is better
than "tension between ambition and mortality"
- Include [Source: ...] citation with full context
If the idea references a world concept: brain/concepts/{slug}.md
If it's a product/business idea: brain/ideas/{slug}.md
STEP 2 — ENTITIES:
Extract all person names, company names, media titles.
For each entity:
a. Run: gbrain search "{name}"
b. If page exists AND new info: append timeline entry
Format: - YYYY-MM-DD | {what happened} [Source: {who}, {context}, {date}]
c. If no page AND entity is notable: create page with web enrichment
d. If page is thin (< 5 lines compiled truth): spawn background enrichment
STEP 3 — BACK-LINKING (mandatory):
For every entity mentioned, add a back-link FROM their page TO this source.
An unlinked mention is a broken brain.
Format: - **YYYY-MM-DD** | Referenced in [{page title}]({path}) — {context}
STEP 4 — SYNC:
Run: gbrain sync --no-pull --no-embed
If nothing to capture, reply "No signals detected" and exit.
`
```
### Notability Filtering
Before creating a new entity page, check notability:
```
is_notable(entity):
// CREATE a page for:
- People the user knows or discusses with specificity
- Companies the user is evaluating, working with, or investing in
- Media the user mentions with personal reaction
- Anyone the user has explicitly engaged with
// DON'T create a page for:
- Generic references or passing examples
- Low-engagement accounts who mentioned the user once
- Pure metaphors ("like the Roman Empire...")
- One-off encounters with no follow-up
// If notable AND no page: create FULL page (not a stub)
// If not notable: skip silently
```
### What Counts as Original Thinking
| Capture | Don't Capture |
|---------|---------------|
| Original observations about how the world works | "ok", "do it", "sure" |
| Novel connections between disparate things | Pure questions without observations |
| Frameworks and mental models | Echoing back what the agent said |
| Pattern recognition ("I keep seeing X in every Y") | Acknowledgments and reactions |
| Hot takes with reasoning | Routine operational messages |
| Metaphors that reveal new angles | Requests without embedded insight |
### Filing Rules
| Signal | Destination |
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| Person mentioned | `brain/people/{slug}.md` |
| Company mentioned | `brain/companies/{slug}.md` |
| Media referenced | `brain/media/{type}/{slug}.md` |
### The Iron Law of Back-Linking
Every entity mention MUST create a back-link FROM the entity page TO the
source. This is not optional.
```
// When message mentions "Pedro" and creates a meeting page:
// 1. Update the meeting page (normal)
brain/meetings/2026-04-10-board-sync.md:
- Pedro presented Q1 numbers
// 2. ALSO update Pedro's page (back-link)
brain/people/pedro-franceschi.md:
## Timeline
- **2026-04-10** | Presented Q1 numbers at board sync
[Source: User, board meeting, 2026-04-10]
```
Without back-links, you can't traverse the graph. "Show me everything related
to Pedro" only works if Pedro's page links back to every mention.
## Tricky Spots
1. **Don't block the conversation.** Entity detection runs async. The user
should see a response immediately, not wait 2 minutes while the sub-agent
enriches 5 entity pages.
2. **Sonnet, not Opus.** Entity detection is pattern matching, not deep
reasoning. Sonnet is 5-10x cheaper and fast enough. Use Opus for the
main conversation.
3. **Exact phrasing matters.** "Markdown is actually code" is an insight.
"Markdown can be used as code" is a summary. Capture the first version.
4. **Don't create stubs.** If you create a page, make it good. Run a web
search, build out the compiled truth, add context. A stub page with just
a name is worse than no page (it gives false confidence).
5. **Dedup before creating.** Always `gbrain search` before creating a page.
Variant spellings, nicknames, and company abbreviations cause duplicates.
"Pedro Franceschi" and "Pedro" might be the same person.
## How to Verify
1. **Send a message mentioning a person.** Say "I had coffee with Sarah Chen
from Acme Corp today." Verify: brain/people/sarah-chen.md was created or
updated, brain/companies/acme-corp.md was created or updated, both have
timeline entries with today's date.
2. **Send a message with an original idea.** Say "What if we could distribute
software as markdown files that agents execute?" Verify:
brain/originals/{slug}.md was created with your exact phrasing.
3. **Check back-links.** Open Sarah Chen's page. It should have a timeline
entry linking back to today's conversation. Open Acme Corp's page. Same.
4. **Send a boring message.** Say "ok sounds good." Verify: nothing was
created. The detector should report "No signals detected."
5. **Check for duplicates.** Mention "Pedro" then later "Pedro Franceschi."
Verify: one page, not two.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,109 @@
# Executive Assistant Pattern
## Goal
Email triage, meeting prep, and scheduling powered by brain context -- so every interaction is informed by the full history of the relationship.
## What the User Gets
Without this: the agent triages email mechanically ("you have 12 unread"), preps for meetings with generic LinkedIn bios, and schedules without relationship context. With this: the agent knows who every sender is before reading their email, surfaces shared history before every meeting, and nudges scheduling based on relationship temperature and open threads.
## Implementation
```
# WORKFLOW 1: Email Triage
on email_batch(emails):
for email in emails:
# Step 1: Search sender BEFORE reading the email body
# Brain context makes triage 10x better
sender_page = gbrain search "{email.sender_name}"
if sender_page:
context = gbrain get <sender_slug>
# Now you know: who they are, relationship history,
# what they care about, open threads
# Step 2: Read the email WITH brain context loaded
# Classification is now informed, not mechanical
# Step 3: Classify with context
if context.relationship == "inner_circle" or context.has_open_threads:
priority = "urgent"
elif context.is_known_entity:
priority = "normal"
else:
priority = "noise" # unknown sender, no brain page
# Step 4: Draft reply with relationship context
if needs_reply(email):
draft = compose_reply(
email,
context=context, # their brain page
open_threads=context.open_threads, # what you're working on together
relationship=context.relationship # tone calibration
)
# WORKFLOW 2: Meeting Prep
on upcoming_meeting(meeting):
briefing = {}
for attendee in meeting.attendees:
# Search brain for each attendee
results = gbrain search "{attendee.name}"
if results:
page = gbrain get <attendee_slug>
briefing[attendee] = {
"compiled_truth": page.compiled_truth,
"last_interaction": page.timeline[0], # most recent
"open_threads": page.open_threads,
"relationship_temperature": page.relationship,
"relevant_deals": gbrain get_links <attendee_slug>,
}
else:
briefing[attendee] = "No brain page -- consider enriching"
# Surface: shared history, what to follow up on, what to watch for
# "Last time you discussed the Series B timeline. Pedro was concerned
# about burn rate. Here's the latest from his company page."
# WORKFLOW 3: Post-Inbox Brain Updates
on inbox_cleared():
for email in processed_emails:
if email.contained_new_information:
# Update the sender's brain page with new signal
gbrain add_timeline_entry <sender_slug> \
--entry "Email re: {subject}. Key info: {extracted_signal}" \
--source "email from {sender} re {subject}, {date}"
# Update any mentioned entity pages too
for entity in email.mentioned_entities:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said_about_them}" \
--source "email from {sender}, {date}"
# WORKFLOW 4: Scheduling Nudges
on schedule_request(meeting):
for attendee in meeting.attendees:
page = gbrain get <attendee_slug>
if page.last_interaction > 6_weeks_ago:
nudge("You haven't met with {attendee} in {weeks} weeks")
if page.has_open_threads:
nudge("{attendee} has an open thread about {topic}")
if page.relationship_temperature == "cooling":
nudge("Relationship with {attendee} may need attention")
```
## Tricky Spots
1. **Search sender BEFORE reading the email.** This is counterintuitive but critical. Loading brain context first means you know who they are, what you're working on together, and what they care about -- before you even see the subject line. The triage is informed, not mechanical.
2. **Unknown senders with no brain page are almost always noise.** If `gbrain search` returns nothing for a sender, they're probably not important. Classify as low priority unless the email content signals otherwise.
3. **Meeting prep is the highest-leverage EA workflow.** The user walks into every meeting already briefed on each attendee: last interaction, open threads, relationship history. This is the difference between "you have a meeting at 3" and "you have a meeting at 3 with Pedro -- last time you discussed the Series B, he was concerned about burn rate."
4. **Post-inbox brain updates are where the brain compounds.** Every email is signal. If you clear the inbox without updating brain pages, the information is lost. This is the step most agents skip.
5. **Scheduling nudges require timeline data.** "You haven't met with Diana in 6 weeks" only works if meeting pages have been ingested with proper entity propagation (see meeting-ingestion guide).
## How to Verify
1. Run meeting prep for tomorrow's calendar. For each attendee, confirm the agent ran `gbrain search` and loaded their brain page before generating the briefing.
2. Triage 5 emails. Confirm the agent searched for each sender in the brain before classifying the email.
3. After clearing an inbox, check 2 sender brain pages with `gbrain get <slug>`. Confirm new timeline entries were added with information from the emails.
4. Check a scheduling suggestion. Confirm the agent referenced the attendee's brain page (last interaction date, open threads) in the nudge.
5. Send a test email from someone with a brain page. Confirm the triage response references their relationship context, not just the email content.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

190
docs/guides/idea-capture.md Normal file
View File

@@ -0,0 +1,190 @@
# Idea Capture: Originals, Depth, and Distribution
## Goal
Capture the user's original thinking with exact phrasing, deep context, and cross-links so the originals folder becomes the highest-value content in the brain.
## What the User Gets
Without this: brilliant ideas said in conversation disappear. The agent heard
"the ambition-to-lifespan ratio has never been more broken" and forgot it.
With this: every original observation is captured verbatim, cross-linked to
the people and ideas that shaped it, and rated for publishing potential. Your
intellectual archive grows with every conversation.
## Implementation
```
capture_idea(message_text, source_context):
// 1. AUTHORSHIP TEST — where does this idea belong?
if user_generated_the_idea(message_text):
destination = "brain/originals/{slug}.md"
elif user_synthesis_of_others(message_text):
destination = "brain/originals/{slug}.md" // synthesis IS original
elif world_concept(message_text):
destination = "brain/concepts/{slug}.md"
elif product_or_business_idea(message_text):
destination = "brain/ideas/{slug}.md"
elif ghostwritten_by_user(message_text):
destination = "brain/originals/{slug}.md" // note ghostwriter in metadata
elif article_about_user(message_text):
destination = "brain/media/writings/{slug}.md"
// 2. CAPTURE WITH EXACT PHRASING — never paraphrase
page = create_or_update(destination, {
content: message_text, // verbatim, not summarized
source: source_context, // conversation, meeting, moment
reasoning_path: influences, // what led to the insight
depth_context: emotional_nuance // the WHY behind the WHAT
})
// 3. ORIGINALITY RATING (for notable ideas)
if is_notable(message_text):
rate_originality(page, populations=[
"general_population", "tech_industry",
"intellectual_media", "political_establishment"
])
// 4. CROSS-LINK (mandatory — an original without links is dead)
link_to_people(page, mentioned_people)
link_to_companies(page, mentioned_companies)
link_to_meetings(page, source_meeting)
link_to_media(page, influences)
link_to_other_originals(page, related_ideas)
link_to_concepts(page, referenced_concepts)
// 5. SYNC
gbrain sync --no-pull --no-embed
```
### The Authorship Test
| Signal | Destination |
|--------|-------------|
| User generated the idea | `brain/originals/{slug}.md` |
| User's unique synthesis of others' ideas | `brain/originals/` (the synthesis is original) |
| World concept someone else coined | `brain/concepts/{slug}.md` |
| Product or business idea | `brain/ideas/{slug}.md` |
| User's ghostwritten book/essay | `brain/originals/` (note ghostwriter in metadata) |
| Article ABOUT user | `brain/media/writings/` |
### Capture Standards
**Use the user's EXACT phrasing.** The language IS the insight.
"The ambition-to-lifespan ratio has never been more broken" captures something that
"tension between ambition and mortality" doesn't. Don't clean it up. Don't paraphrase.
The vivid version is the real version.
**What counts as worth capturing:**
- Original observations about how the world works
- Novel connections between disparate things
- Frameworks and mental models
- Pattern recognition moments ("I keep seeing X in every Y")
- Hot takes with reasoning behind them
- Metaphors that reveal new angles
- Emotional/psychological insights about self or others
**What does NOT count:**
- Routine operational messages ("ok", "do it")
- Pure questions without embedded observations
- Echoing back something the agent said
- Acknowledgments and reactions
### The Depth Test
**Could someone unfamiliar with the user read this page and understand not
just WHAT they think but WHY and HOW they got there?**
If the answer is no, it needs more depth. Include:
- The reasoning path (what led to the insight)
- The influences (what they were reading/watching/experiencing)
- The context (conversation, meeting, moment)
- The emotional or psychological nuance
### Originality Distribution Rating
For notable ideas, rate originality 0-100 across different populations:
```markdown
## Originality Distribution
- **General population:** 72/100 — most people haven't encountered this framework
- **Tech industry:** 45/100 — common in startup circles but novel to most
- **Intellectual/media class:** 68/100 — would resonate, not yet articulated
- **Political establishment:** 82/100 — completely foreign to policy thinking
**Publish signal:** Strong essay candidate. Best audience: founders, builders.
```
This tells the user which ideas are worth turning into essays, talks, or videos,
and which audience would find them most novel.
### Deep Cross-Linking Mandate
**An original without cross-links is a dead original.** The connections ARE
the intelligence.
Every original MUST link to:
- **People** who shaped the thinking
- **Companies** where the idea played out
- **Meetings** where it was discussed
- **Books and media** that influenced it
- **Other originals** it connects to (ideas form clusters)
- **Concepts** it builds on or challenges
### Notability Filtering
Before creating any entity page, check notability:
**Create a page for:**
- People you know or discuss with specificity
- Companies you're evaluating, working with, or investing in
- Media you mention with personal reaction
- Anyone you've explicitly engaged with
**Don't create pages for:**
- Generic references or passing examples
- Low-engagement accounts who mentioned you once
- Pure metaphors ("like the Roman Empire...")
- One-off encounters with no follow-up
**Decision:** If notable AND no page exists, create a full page with web
search enrichment. No stubs. If you make a page, make it good.
## Tricky Spots
1. **Synthesis IS original.** When the user connects two existing ideas in a
new way, that synthesis belongs in `brain/originals/`, not `brain/concepts/`.
The novel combination is the insight, even if the component ideas aren't new.
2. **Exact phrasing is non-negotiable.** Never paraphrase, summarize, or
"clean up" the user's language. "The ambition-to-lifespan ratio has never
been more broken" is the insight. "Tension between ambition and mortality"
is a corpse. Capture the first version.
3. **Cross-links are mandatory, not optional.** An original without links to
the people, companies, meetings, and concepts that shaped it is a dead
original. The connections ARE the intelligence. Check every original for
at least 2 cross-links before considering it captured.
## How to Verify
1. **Generate an idea and check the page.** Say something original in
conversation (e.g., "What if markdown files are actually distributed
software?"). Verify that `brain/originals/{slug}.md` was created with
your exact phrasing, not a paraphrase.
2. **Check cross-links exist.** Open the newly created original page. It
should link to at least the people or concepts mentioned. Open those
linked pages and verify they back-link to the original.
3. **Verify the depth test passes.** Read the captured page as if you were
a stranger. Can you understand not just WHAT the user thinks but WHY?
If the reasoning path and context are missing, the capture is incomplete.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

138
docs/guides/live-sync.md Normal file
View File

@@ -0,0 +1,138 @@
# Live Sync: Keep the Index Current
## Goal
Every markdown change in the brain repo is searchable within minutes, automatically, with no manual intervention.
## What the User Gets
Without this: you correct a hallucination in a brain page, but the vector DB
keeps serving the old text because nobody ran `gbrain sync`. Stale search
results erode trust. The brain becomes unreliable.
With this: edits show up in search within minutes. The vector DB stays current
with the brain repo automatically. You never have to remember to run sync.
## Implementation
### Prerequisite: Session Mode Pooler
Sync uses `engine.transaction()` on every import. If `DATABASE_URL` points to
Supabase's **Transaction mode** pooler, sync will throw `.begin() is not a
function` and **silently skip most pages**. This is the number one cause of
"sync ran but nothing happened."
Fix: use the **Session mode** pooler string (port 6543, Session mode) or the
direct connection (port 5432, IPv6-only). Verify by running `gbrain sync` and
checking that the page count in `gbrain stats` matches the syncable file count
in the repo.
### The Primitives
Always chain sync + embed:
```bash
gbrain sync --repo /path/to/brain && gbrain embed --stale
```
- `gbrain sync --repo <path>` -- one-shot incremental sync. Detects changes via
`git diff`, imports only what changed. For small changesets (<= 100 files),
embeddings are generated inline during import.
- `gbrain embed --stale` -- backfill embeddings for any chunks that don't have
them. Safety net for large syncs (>100 files) or prior `--no-embed` runs.
- `gbrain sync --watch --repo <path>` -- foreground polling loop, every 60s
(configurable with `--interval N`). Embeds inline for small changesets. Exits
after 5 consecutive failures, so run under a process manager or pair with a
cron fallback.
### Approach 1: Cron Job (recommended)
Run every 5-30 minutes. Works with any cron scheduler.
```bash
gbrain sync --repo /data/brain && gbrain embed --stale
```
**OpenClaw:**
```
Name: gbrain-auto-sync
Schedule: */15 * * * *
Prompt: "Run: gbrain sync --repo /data/brain && gbrain embed --stale
Log the result. If sync fails with .begin() is not a function,
the DATABASE_URL is using Transaction mode pooler."
```
**Hermes:**
```
/cron add "*/15 * * * *" "Run gbrain sync --repo /data/brain &&
gbrain embed --stale. Log the result." --name "gbrain-auto-sync"
```
### Approach 2: Long-Lived Watcher
For near-instant sync (60s polling). Run under a process manager that
auto-restarts on exit. Pair with a cron fallback since `--watch` exits
on repeated failures.
```bash
gbrain sync --watch --repo /data/brain
```
### Approach 3: Git Hook / Webhook
Triggers sync on push events for instant sync (<5s).
- **GitHub webhook:** Set up the webhook to call
`gbrain sync --repo /data/brain && gbrain embed --stale`.
Verify `X-Hub-Signature-256` against a shared secret.
- **Git post-receive hook:** If the brain repo is on the same machine.
### What Gets Synced
Sync only indexes "syncable" markdown files. These are excluded by design:
- Hidden paths (`.git/`, `.raw/`, etc.)
- The `ops/` directory
- Meta files: `README.md`, `index.md`, `schema.md`, `log.md`
### Sync is Idempotent
Concurrent runs are safe. Two syncs on the same commit no-op because content
hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
## Tricky Spots
1. **Always chain sync + embed.** Running `gbrain sync` without
`gbrain embed --stale` leaves new chunks without embeddings. They exist
in the database but are invisible to vector search. Always run both
commands together. The `&&` ensures embed only runs if sync succeeds.
2. **--watch polls, it doesn't stream.** The `--watch` flag polls every 60s
(configurable). It is not a filesystem watcher or git hook. It exits after
5 consecutive failures, so it needs a process manager (systemd, pm2) or a
cron fallback to stay alive. Don't assume it runs forever.
3. **Webhook needs the server running.** If you use a GitHub webhook for
instant sync, the receiving server must be running and reachable. If the
server is down when a push happens, that sync is missed. Pair webhooks
with a cron fallback that catches anything the webhook missed.
## How to Verify
1. **Edit a file and search for the change.** Edit a brain markdown file,
commit, and push. Wait for the next sync cycle (cron interval or `--watch`
poll). Run `gbrain search "<text from the edit>"`. The updated content
should appear in results. If it returns old content, sync failed.
2. **Compare page count to file count.** Run `gbrain stats` and count the
syncable markdown files in the brain repo. The page count in the database
should match. If they diverge, files are being silently skipped (likely
a Transaction mode pooler issue).
3. **Check embedded chunk count.** In `gbrain stats`, the embedded chunk
count should be close to the total chunk count. A large gap means
`gbrain embed --stale` isn't running after sync, leaving chunks invisible
to vector search.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,80 @@
# Meeting Ingestion
## Goal
Meeting transcripts become brain pages that update every mentioned entity -- attendees, companies, deals, and action items all propagated in one pass.
## What the User Gets
Without this: meetings vanish into memory, action items are forgotten, and the agent has no idea what was discussed last time you met someone. With this: every meeting is a permanent record that enriches every person and company page it touches, and the user walks into every follow-up already briefed.
## Implementation
```
on new_meeting_transcript(meeting):
# Step 1: Pull the COMPLETE transcript -- NOT the AI summary
# AI summaries hallucinate framing ("it was agreed that...")
# The transcript is ground truth
transcript = fetch_full_transcript(meeting.id) # e.g., Circleback API
# Must have speaker diarization: WHO said WHAT
# Step 2: Create the meeting page
slug = f"meetings/{meeting.date}-{short_description}"
compiled_truth = agent_analysis(transcript):
# Above the bar: agent's OWN analysis, not a generic recap
# - Reframe through the user's priorities
# - Flag surprises, contradictions, implications
# - Name real decisions (not performative ones)
# - Call out what was left unsaid or unresolved
timeline = format_diarized_transcript(transcript)
# Below the bar: full transcript, append-only
# Format: **Speaker** (HH:MM:SS): Words.
gbrain put <slug> --content "<compiled_truth>\n---\n<timeline>"
# Step 3: Propagate to ALL entity pages (MANDATORY -- most agents skip this)
for person in meeting.attendees + meeting.mentioned_people:
gbrain add_timeline_entry <person_slug> \
--entry "Met in '{meeting.title}' on {date}. Key points: ..." \
--source "Meeting notes '{meeting.title}', {date}"
# Update their State section if new information surfaced
# Update company pages for each person's company if relevant
for company in meeting.mentioned_companies:
gbrain add_timeline_entry <company_slug> \
--entry "Discussed in '{meeting.title}': {what_was_said}" \
--source "Meeting notes '{meeting.title}', {date}"
# Step 4: Extract action items
action_items = extract_action_items(transcript)
# Add to task list with owner attribution
# Step 5: Back-link everything (bidirectional graph)
for entity in all_entities_mentioned:
gbrain add_link <slug> <entity_slug> # meeting -> entity
gbrain add_link <entity_slug> <slug> # entity -> meeting
# Step 6: Sync so new pages are immediately searchable
gbrain sync
# Schedule: cron 3x/day (10 AM, 4 PM, 9 PM) to catch new meetings
# Source: Circleback (https://circleback.ai) or any service with
# speaker diarization + API/webhook access
```
## Tricky Spots
1. **Always pull the COMPLETE transcript, never the AI summary.** AI summaries hallucinate framing -- they editorialize what was "agreed" or "decided" when no such agreement happened. The diarized transcript is ground truth.
2. **Entity propagation is the step most agents skip.** A meeting is NOT fully ingested until every attendee's page, every mentioned person's page, and every company's page has a new timeline entry. The meeting page alone is useless without propagation.
3. **Mentioned people are not just attendees.** If the meeting discussed "Sarah's team at Brex," then Sarah's page AND Brex's page need updates -- even though Sarah wasn't in the room.
4. **The agent's analysis is the value, not a summary.** "They discussed Q2 targets" is worthless. "Pedro pushed back on the burn rate, Diana didn't commit to the timeline, and nobody addressed the pricing gap" is useful.
5. **Back-links must be bidirectional.** The meeting page links to attendee pages AND attendee pages link back to the meeting. The graph is bidirectional. Always.
## How to Verify
1. After ingesting a meeting, run `gbrain get meetings/{date}-{slug}`. Confirm the page has the agent's analysis above the bar and the full diarized transcript below it.
2. For each attendee, run `gbrain get <attendee_slug>`. Check that their timeline has a new entry referencing the meeting with specific insights (not just "attended meeting").
3. Pick a company mentioned in the meeting. Run `gbrain get <company_slug>`. Confirm a timeline entry exists referencing what was discussed about the company.
4. Run `gbrain get_links meetings/{date}-{slug}`. Verify back-links exist to all attendee and entity pages.
5. Run `gbrain search "{meeting_topic}"`. Confirm the meeting page appears in search results (verifies sync ran).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,120 @@
# Operational Disciplines
## Goal
Five non-negotiable rules that separate a production brain from a demo -- signal detection, brain-first lookup, sync after every write, daily heartbeat, and nightly dream cycle.
## What the User Gets
Without this: the agent misses signals in conversation, wastes money on external APIs when the brain already has the answer, leaves search results stale after writes, and lets the brain rot quietly. With this: every message is scanned for entities, the brain is always consulted first, search is always current, health is monitored daily, and the brain compounds overnight.
## Implementation
```
# DISCIPLINE 1: Signal Detection on Every Message (MANDATORY)
on every_inbound_message(message):
# No exceptions. If the user thinks out loud and the brain doesn't
# capture it, the system is broken. This is the #1 discipline.
entities = detect_entities(message)
# people, companies, deals, original ideas
for entity in entities:
existing = gbrain search "{entity.name}"
if existing:
gbrain add_timeline_entry <entity_slug> \
--entry "{what_was_said}" \
--source "User, direct message, {timestamp}"
# else: flag for enrichment if important enough
originals = detect_original_thinking(message)
for idea in originals:
gbrain put originals/{slug} --content "{user's exact phrasing}"
# DISCIPLINE 2: Brain-First Lookup Before External APIs (MANDATORY)
on information_needed(topic):
# ALWAYS check the brain before reaching for the web
brain_result = gbrain search "{topic}"
if brain_result:
page = gbrain get <slug>
# Use brain data first. External APIs FILL GAPS, not replace.
else:
# Brain has nothing -- now use external APIs
external_result = brave_search("{topic}")
# An agent that reaches for the web before checking its own brain
# is wasting money and giving worse answers.
# DISCIPLINE 3: Sync After Every Write (MANDATORY)
on brain_write_complete():
gbrain sync
# Without this, search results are stale.
# The page you just wrote won't appear in gbrain search or gbrain query
# until sync runs. Skipping this means the next lookup misses the
# most recent data.
# DISCIPLINE 4: Daily Heartbeat Check
on daily_schedule("09:00"):
gbrain doctor
# Checks: database connectivity, embedding health, sync status,
# page count, stale pages, broken links
# If doctor reports issues, fix them before doing anything else.
# DISCIPLINE 5: Nightly Dream Cycle
on nightly_schedule("02:00"):
# The dream cycle is the most important discipline.
# The brain COMPOUNDS overnight.
# 5a: Entity sweep -- find unlinked mentions
pages = gbrain list_pages
for page in pages:
mentions = extract_entity_mentions(page.content)
existing_links = gbrain get_links <page.slug>
for mention in mentions:
if mention not in existing_links:
gbrain add_link <page.slug> <mention_slug> # fix broken graph
# 5b: Citation audit -- find facts without sources
for page in pages:
facts_without_sources = audit_citations(page.content)
if facts_without_sources:
flag_for_remediation(page, facts_without_sources)
# 5c: Memory consolidation -- update compiled truth from timeline
for page in stale_pages(older_than="7d"):
timeline = gbrain get_timeline <page.slug>
if timeline.has_new_entries_since_last_consolidation:
# Re-synthesize compiled truth from accumulated timeline
updated_truth = consolidate(page.compiled_truth, timeline.new_entries)
gbrain put <page.slug> --content updated_truth
# 5d: Sync everything
gbrain sync
# BONUS: Durable Skills Over One-Off Work
# If you do something twice, make it a skill + cron.
# 1. Concept the process
# 2. Run it manually for 3-10 items
# 3. Revise -- iterate on quality
# 4. Codify into a skill
# 5. Add to cron -- automate it
# Each entity type and signal source has exactly one owner skill.
# Two skills creating the same page = coverage violation.
```
## Tricky Spots
1. **The dream cycle is the most important discipline.** Brains compound overnight. Entity sweeps fix broken graphs, citation audits catch sourceless facts, and memory consolidation keeps compiled truth current. Skip the dream cycle and the brain slowly rots.
2. **Skipping Discipline 3 (sync after write) means stale search results.** You write a page, then immediately search for it -- and get nothing back. The page exists but isn't indexed. Always sync after writes.
3. **Signal detection must fire on EVERY message.** Not just messages that look important. The user says "I talked to Pedro yesterday about the board seat" in passing -- that's a timeline entry on Pedro's page, a potential update to his State section, and a signal about the board. If the agent doesn't catch it, the system is broken.
4. **Brain-first saves money AND gives better answers.** The brain has context that external APIs don't: relationship history, meeting notes, the user's own assessment. An API lookup for "Pedro Franceschi" returns a LinkedIn profile. The brain returns the full picture including private context.
5. **`gbrain doctor` catches silent failures.** Embedding pipelines can stall, sync can fail silently, database connections can drop. The daily heartbeat catches these before they compound into data loss.
## How to Verify
1. Send a message mentioning a person with a brain page. Confirm the agent detects the entity and adds a timeline entry to their page (`gbrain get_timeline <slug>`).
2. Ask the agent about someone in the brain. Confirm it runs `gbrain search` or `gbrain get` BEFORE reaching for external APIs (check the tool call order).
3. Write a new page with `gbrain put`, then immediately run `gbrain search` for it. Confirm it appears in results (verifies sync ran).
4. Run `gbrain doctor`. Confirm it returns a health report with database status, page count, and any flagged issues.
5. After a dream cycle runs, check a page that had unlinked entity mentions. Confirm new links were added (`gbrain get_links <slug>`).
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,87 @@
# The Originals Folder
## Goal
Capture the user's original thinking with their exact phrasing, deep cross-links, and full provenance -- so intellectual capital compounds instead of evaporating.
## What the User Gets
Without this: the user generates a brilliant framework in conversation and it vanishes when the session ends. Six months later, they vaguely remember the idea but can't find it, can't recall the exact phrasing, and can't trace what influenced it. With this: every original observation, thesis, framework, and hot take is captured verbatim in `brain/originals/`, cross-linked to the people, companies, and media that shaped it, and searchable forever.
## Implementation
```
on user_message(message):
# Detect original thinking in every message
if contains_original_thinking(message):
# The authorship test:
# User generated the idea? -> originals/{slug}.md
# User's unique synthesis of someone else's? -> originals/ (synthesis IS original)
# World concept someone else coined? -> concepts/{slug}.md
# Product or business idea? -> ideas/{slug}.md
# Step 1: Use the user's EXACT phrasing for the slug
# "meatsuit-maintenance-tax"
# NOT "biological-needs-maintenance-overhead"
# The vividness IS the concept.
slug = slugify(user_exact_phrase)
# Step 2: Create the originals page
gbrain put originals/{slug} --content """
# {User's Exact Phrase}
## The Idea
{User's original thinking, captured in their own words.
Do NOT paraphrase. Do NOT clean up the language.
The raw phrasing is the intellectual artifact.}
## Context
{What triggered this thinking. Meeting? Article? Conversation?
Include the source that sparked it.}
[Source: User, {context}, {date} {time} {tz}]
## Connections
- Related to: [[{person_slug}]] -- {how they connect}
- Emerged from: [[{meeting_slug}]] -- {what was discussed}
- Influenced by: [[{book_or_media_slug}]] -- {what resonated}
- Builds on: [[{other_original_slug}]] -- {how ideas cluster}
"""
# Step 3: Cross-link to everything that shaped the thinking
for entity in idea.influences:
gbrain add_link originals/{slug} <entity_slug>
gbrain add_link <entity_slug> originals/{slug}
# Step 4: Sync
gbrain sync
# What counts as original thinking:
# - Novel frameworks ("the meatsuit maintenance tax")
# - Hot takes on someone else's work (synthesis IS original)
# - Pattern recognition across multiple entities
# - Predictions or bets about the future
# - Contrarian positions with reasoning
# What does NOT go in originals/:
# - Facts about the world (-> entity pages)
# - Concepts someone else coined (-> concepts/)
# - Product ideas (-> ideas/)
# - Preferences (-> agent memory)
```
## Tricky Spots
1. **Naming: the vividness IS the concept.** `meatsuit-maintenance-tax` not `biological-needs-maintenance-overhead`. `ambition-debt` not `deferred-career-risk-accumulation`. The user's colorful phrasing is the intellectual artifact. Never sanitize it into corporate-speak.
2. **Synthesis IS original.** The user's take on Peter Thiel's zero-to-one framework goes in `originals/`, not `concepts/`. The original part is the user's synthesis, interpretation, or disagreement -- even though the underlying ideas came from someone else.
3. **An original without cross-links is a dead original.** The connections ARE the intelligence. An idea about "ambition debt" that doesn't link to the people who exemplify it, the meeting where it was discussed, and the book that influenced it is just a note in a graveyard. Cross-link aggressively.
4. **Originals form clusters.** Over time, the user's ideas connect to each other. "Meatsuit maintenance tax" connects to "ambition debt" connects to "founder energy budget." Link originals to other originals. The cluster IS the user's worldview.
5. **Capture the trigger context.** What conversation, meeting, article, or moment sparked this idea? The context often matters as much as the idea itself for future retrieval. Include it in the page.
## How to Verify
1. Generate an original idea in conversation (e.g., "I call this the 'ambition debt' problem -- every year you delay going big, the compound interest works against you"). Confirm a new page appears at `brain/originals/ambition-debt` with `gbrain get originals/ambition-debt`.
2. Check that the page uses the user's exact phrasing for the title and slug -- not a sanitized version.
3. Run `gbrain get_links originals/ambition-debt`. Confirm cross-links exist to related people, meetings, or other originals.
4. Express a take on someone else's idea (e.g., "I think Thiel's contrarian question is wrong because..."). Confirm it goes to `originals/` (synthesis is original), not `concepts/`.
5. Run `gbrain search "ambition debt"`. Confirm the originals page appears in search results and is discoverable.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

165
docs/guides/quiet-hours.md Normal file
View File

@@ -0,0 +1,165 @@
# Quiet Hours and Timezone-Aware Delivery
## Goal
Hold all notifications during sleep hours, merge held messages into the morning briefing, and adjust automatically when the user travels.
## What the User Gets
Without this: 3 AM pings from cron jobs. One bad notification and the user
disables the entire system.
With this: the brain works overnight (dream cycle, collectors, enrichment)
but notifications are held until morning. Travel to Tokyo? The system adjusts
automatically from your calendar, no config change needed.
## Implementation
### Quiet Hours Gate
Every cron job that sends notifications must check quiet hours FIRST.
```
QUIET_START = 23 // 11 PM local time
QUIET_END = 8 // 8 AM local time
is_quiet(local_hour):
return local_hour >= QUIET_START OR local_hour < QUIET_END
```
**Before sending any notification:**
1. Determine user's current timezone (from config or heartbeat state)
2. Convert current UTC time to local time
3. If quiet hours: hold the message, don't send
### Held Messages
During quiet hours, output goes to a held directory instead of being sent:
```
if is_quiet():
mkdir -p /tmp/cron-held/
write("/tmp/cron-held/{job-name}.md", output)
exit // don't send
else:
send(output)
```
The morning briefing picks up held messages:
```
morning_briefing():
held_files = list("/tmp/cron-held/*.md")
if held_files:
briefing += "## Overnight Updates\n\n"
for file in held_files:
briefing += read(file)
delete(file)
```
This way nothing is lost. Overnight cron results get folded into the
first thing the user sees in the morning.
### Timezone Awareness
The agent should know what timezone the user is in. Store it in
the agent's operational state:
```json
{
"currentLocation": {
"timezone": "US/Pacific",
"city": "San Francisco"
}
}
```
**Update the timezone when:**
- Calendar shows the user flying somewhere (check for airline/hotel events)
- User mentions being in a different city
- User's active hours shift (they're responding at 3 AM PT = they're probably traveling)
**All times shown to the user should be in their LOCAL timezone.** Never
show UTC or a timezone the user isn't in.
### Shell Implementation
```bash
#!/bin/bash
# quiet-hours-gate.sh — run before any notification
TIMEZONE="${USER_TIMEZONE:-US/Pacific}"
LOCAL_HOUR=$(TZ="$TIMEZONE" date +%H)
if [ "$LOCAL_HOUR" -ge 23 ] || [ "$LOCAL_HOUR" -lt 8 ]; then
echo "QUIET_HOURS=true"
exit 1 # don't send
fi
echo "QUIET_HOURS=false"
exit 0 # ok to send
```
**In cron job scripts:**
```bash
# Check quiet hours first
if ! bash scripts/quiet-hours-gate.sh; then
mkdir -p /tmp/cron-held
echo "$OUTPUT" > /tmp/cron-held/$(basename "$0" .sh).md
exit 0
fi
# Not quiet hours — send normally
send_notification "$OUTPUT"
```
### Configurable Hours
Some users want different quiet hours. Store the config:
```json
{
"quiet_hours": {
"start": 23,
"end": 8,
"enabled": true
}
}
```
Set `enabled: false` to disable quiet hours entirely (e.g., for 24/7 monitoring).
## Tricky Spots
1. **Gate on EVERY job.** The quiet hours check must run before every single
cron job that produces notifications. If even one job skips the gate, the
user gets a 3 AM ping and loses trust in the entire system. No exceptions.
2. **Held messages MUST be picked up.** If the morning briefing doesn't read
`/tmp/cron-held/`, overnight results vanish silently. Verify the briefing
skill reads and clears the held directory. Orphaned held files mean the
pickup integration is broken.
3. **Timezone auto-detection is fragile.** Calendar-based timezone detection
relies on the user having airline/hotel events with location data. If the
user books travel without calendar entries, the system won't detect the
move. Fall back to activity-hour analysis (responding at 3 AM PT = probably
not in PT anymore) and ask the user if uncertain.
## How to Verify
1. **Set quiet hours to the current hour.** Temporarily set `QUIET_START` to
one hour before now and `QUIET_END` to one hour after. Trigger a cron job.
Verify the output goes to `/tmp/cron-held/` instead of being sent.
2. **Check held message pickup.** After step 1, run or simulate the morning
briefing. Verify the held message appears in the "Overnight Updates"
section and the file is deleted from `/tmp/cron-held/`.
3. **Verify timezone adjustment.** Change the timezone config to a zone where
it's currently quiet hours. Trigger a notification. Verify it's held. Change
back to your real timezone during active hours. Trigger again. Verify it sends.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,158 @@
# Two-Repo Architecture: Agent Behavior vs World Knowledge
## Goal
Separate agent behavior (replaceable) from world knowledge (permanent) into two repos with strict boundaries.
## What the User Gets
Without this: agent config and world knowledge are mixed together. Switch agents
and you lose your knowledge. Switch knowledge tools and you lose your agent setup.
With this: your brain (14,700+ files of people, companies, meetings, ideas)
survives any agent swap. Your agent config survives any knowledge tool swap.
## Implementation
### The Boundary Test
**"Is this about how the agent operates, or is this knowledge about the world?"**
| Question | If YES -> Agent Repo | If YES -> Brain Repo |
|----------|---------------------|---------------------|
| Would this file transfer if you switched AI agents? | YES | -- |
| Would this file transfer if you switched to a different person? | -- | YES |
| Is this about how the agent behaves? | YES | -- |
| Is this about a person, company, deal, meeting, or idea? | -- | YES |
### Quick Decision Tree
```
New file to create?
|-- About a person, company, deal, project, meeting, idea? -> brain/
|-- A spec, research doc, or strategic analysis? -> brain/
|-- An original idea or observation? -> brain/originals/
|-- A daily session log or heartbeat state? -> agent-repo/
|-- A skill, config, cron, or ops file? -> agent-repo/
|-- A task or todo? -> agent-repo/tasks/
```
### Agent Repo (operational config)
How the agent works. Identity, configuration, operational state.
```
agent-repo/
├── AGENTS.md # Agent identity + operational rules
├── SOUL.md # Persona, voice, values
├── USER.md # User preferences + context
├── HEARTBEAT.md # Daily ops flow
├── TOOLS.md # Available tools + credentials
├── MEMORY.md # Operational memory (preferences, decisions)
├── skills/ # Agent capabilities (SKILL.md files)
│ ├── ingest/SKILL.md
│ ├── query/SKILL.md
│ ├── enrich/SKILL.md
│ └── ...
├── cron/ # Scheduled jobs
│ └── jobs.json
├── tasks/ # Current task list
│ └── current.md
├── hooks/ # Event hooks + transforms
├── scripts/ # Operational scripts (collectors, gates)
└── memory/ # Session logs, state files
├── heartbeat-state.json
└── YYYY-MM-DD.md # Daily session logs
```
### Brain Repo (world knowledge)
What you know. People, companies, deals, meetings, ideas, media.
This is the repo GBrain indexes.
```
brain/
├── people/ # Person dossiers (compiled truth + timeline)
├── companies/ # Company profiles
├── deals/ # Deal tracking
├── meetings/ # Meeting transcripts + analysis
├── originals/ # YOUR original thinking (highest value)
├── concepts/ # World concepts and frameworks
├── ideas/ # Product and business ideas
├── media/ # Video transcripts, books, articles
│ ├── youtube/
│ ├── podcasts/
│ └── articles/
├── sources/ # Source material summaries
├── daily/ # Daily data (calendar, logs)
│ └── calendar/
│ └── YYYY/
│ └── YYYY-MM-DD.md
├── projects/ # Project specs and docs
├── writing/ # Essays, drafts, published work
├── diligence/ # Investment diligence materials
│ └── company-name/
│ ├── index.md
│ ├── pitch-deck.md
│ └── .raw/ # Original PDFs/files
└── Apple Notes/ # Imported Apple Notes archive
```
### The Hard Rule
**Never write knowledge to the agent repo.** If a skill, sub-agent, or cron
job needs to create a file about a person, company, deal, meeting, project,
or idea, it MUST write to the brain repo, never to the agent repo.
The brain is the permanent record. The agent repo is replaceable.
### Why Two Repos
**Independence.** You can switch AI agents (OpenClaw -> Hermes -> custom) without
losing your knowledge. You can switch knowledge tools (GBrain -> something else)
without losing your agent setup.
**Scale.** The brain grows large (10,000+ files). The agent repo stays small
(< 100 files). Different backup strategies, different sync cadences.
**Privacy.** The brain contains sensitive information (people, deals, personal
notes). The agent repo contains operational config. Different access controls.
**GBrain indexes the brain repo.** Run `gbrain sync --repo ~/brain/` to keep
the search index current. The agent repo is never indexed by GBrain.
## Tricky Spots
1. **Never write knowledge to the agent repo.** This is the most common
violation. A skill that creates a person page, a cron job that saves
meeting notes, a sub-agent that captures an idea -- all of these MUST
write to the brain repo. If it's about the world, it goes in the brain.
2. **The brain is the permanent record.** When in doubt, ask: "Would this
file survive switching to a completely different AI agent?" If yes, it
belongs in the brain. Agent configs, skills, cron jobs, and operational
state are replaceable. People, companies, ideas, and meetings are not.
3. **Don't index the agent repo.** GBrain indexes the brain repo only.
Running `gbrain sync` against the agent repo pollutes search results
with operational config instead of world knowledge.
## How to Verify
1. **Check file placement.** After any skill or cron job creates a file,
verify it landed in the correct repo. Person/company/idea/meeting files
should be in `brain/`. Skill/config/cron/state files should be in the
agent repo. Any knowledge file in the agent repo is a boundary violation.
2. **Run the boundary test.** Pick 5 recently created files and ask: "Would
this transfer if I switched AI agents?" and "Would this transfer if I
switched to a different person?" If the answers don't match the file's
location, it's in the wrong repo.
3. **Verify GBrain only indexes brain.** Run `gbrain stats` and check the
indexed paths. None should point to the agent repo directory. If agent
config files appear in search results, the sync target is misconfigured.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,78 @@
# Search Modes
## Goal
Know which search command to use and when -- keyword, hybrid, or direct -- so every lookup is fast and returns the right result.
## What the User Gets
Without this: the agent fumbles between search commands, returns chunks when full pages are needed, runs expensive semantic searches when a direct get would do, or misses results entirely. With this: every lookup uses the optimal mode, token budgets are respected, and the user gets the right information in the fewest calls.
## Implementation
```
on user_asks_about(topic):
# Decision tree: pick the right search mode
if know_exact_slug(topic):
# MODE 3: Direct get -- instant, no search overhead
result = gbrain get <slug>
# e.g., "Tell me about Pedro" -> gbrain get pedro-franceschi
# Returns the FULL page -- compiled truth + timeline
elif topic.is_exact_name or topic.is_keyword:
# MODE 1: Keyword search -- fast, no embeddings needed, day-one ready
results = gbrain search "{name_or_keyword}"
# e.g., "Find anything about Series A" -> gbrain search "Series A"
# Returns CHUNKS, not full pages
# IMPORTANT: keyword search returns chunks
# If the chunk confirms relevance, THEN load the full page:
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
elif topic.is_semantic_question:
# MODE 2: Hybrid search -- semantic + keyword, needs embeddings
results = gbrain query "{natural language question}"
# e.g., "Who do I know at fintech companies?" -> gbrain query "fintech contacts"
# Returns ranked chunks via vector + keyword + RRF
# Same rule: chunks first, then get full page if needed
if chunk.confirms_relevance:
full_page = gbrain get <slug_from_chunk>
# Quick reference:
# | Mode | Command | Needs Embeddings | Speed | Best For |
# |---------|----------------------|------------------|---------|---------------------------------|
# | Keyword | gbrain search "term" | No | Fastest | Known names, exact matches |
# | Hybrid | gbrain query "..." | Yes | Fast | Semantic questions, fuzzy match |
# | Direct | gbrain get <slug> | No | Instant | When you know the slug |
# Progression over time:
# Day 1: keyword search (works without embeddings)
# After first embed: hybrid search unlocked
# Once you know slugs: direct get for speed
# Precedence for conflicting information within a page:
# 1. User's direct statements (always wins)
# 2. Compiled truth sections (synthesized from evidence)
# 3. Timeline entries (raw signal, reverse chronological)
# 4. External sources (web search, APIs)
```
## Tricky Spots
1. **Search returns chunks, not full pages.** After `gbrain search` or `gbrain query`, you get excerpts. Always run `gbrain get <slug>` to load the full page when the chunk confirms relevance. Don't answer questions from chunks alone when the full context matters.
2. **Keyword search works without embeddings.** On day one before any embedding run, `gbrain search` still works. Don't tell the user "search isn't available yet" -- keyword search is always available.
3. **Don't use hybrid search for known names.** `gbrain query "Pedro Franceschi"` wastes embedding compute. Use `gbrain search "Pedro Franceschi"` or better yet `gbrain get pedro-franceschi` if you know the slug.
4. **Token budget awareness.** A full page via `gbrain get` can be large. Read the search chunks first to confirm relevance before pulling the full page. "Did anyone mention the Series A?" -- search results (chunks) are probably enough. "Tell me everything about Pedro" -- get the full page.
5. **Hybrid search needs embeddings to have been run.** If `gbrain query` returns nothing but `gbrain search` finds results, the embeddings haven't been generated yet. Run the embedding pipeline first.
## How to Verify
1. Run `gbrain search "Pedro"` -- confirm it returns chunks with matching text and slug references.
2. Run `gbrain query "who works at fintech companies"` -- confirm it returns semantically relevant results (not just keyword matches on "fintech").
3. Run `gbrain get pedro-franceschi` -- confirm it returns the full page with compiled truth and timeline.
4. Compare: search for the same entity using all three modes. Keyword should be fastest, hybrid should surface conceptual matches, direct should return the complete page.
5. After a search returns a chunk, run `gbrain get` on the slug from that chunk. Confirm the full page contains more context than the chunk alone.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,131 @@
# Skill Development Cycle
## Goal
Turn every repeating task into a durable, automated skill so that if you ask twice, it should already be running on a cron.
## What the User Gets
Without this: ad-hoc work that the agent forgets how to do. You ask "enrich
this person" and the agent invents a new process each time. Quality varies.
With this: every capability is codified, tested, and scheduled. Enrichment
runs the same way every time. New patterns get skill-ified within a day.
## Implementation
**The Rule:** If you have to ask your agent for something twice, it should
already be a skill running on a cron. First time is discovery. Second time
is system failure.
### The 5-Step Cycle
**Step 1: Concept the Process.**
Describe what needs to happen in plain language:
- What's the input? What's the output? What triggers it?
- What data sources does it touch?
- How often should it run?
**Step 2: Run Manually for 3-10 Items.**
Actually do the work by hand on a small batch. This is the prototype phase.
Do NOT write a SKILL.md yet. Just do the work and observe:
- What does the output actually look like?
- What edge cases appear?
- What quality bar is right?
**Step 3: Evaluate Output.**
Show the user the results. Get feedback.
- Does output look good? Is quality right?
- Did you miss anything? Over-engineer?
- Revise the process based on what you learned.
**Step 4: Codify into a Skill.**
Write the SKILL.md. Either:
- **New skill** -- genuinely new capability
- **Add to existing skill** -- variation of something that exists (parameterize it)
The skill must be:
- **Durable** -- works tomorrow, next week, next month without manual intervention
- **MECE** -- doesn't overlap with other skills (see below)
- **Parameterized** -- handles variations through parameters, not separate skills
**Step 5: Add to Cron (if recurring).**
If the process should run automatically:
- Add to existing cron job if it fits naturally
- Create new cron job if it has a distinct scheduling concern
- Monitor the first 2-3 automated runs for quality
- Fix issues that emerge at scale
### MECE Discipline
Skills should be **Mutually Exclusive, Collectively Exhaustive**:
- Each entity type has exactly ONE owner skill
- Each signal source has exactly ONE owner skill
- Two skills creating the same brain page = MECE violation
**Example ownership (no overlap):**
| Signal Source | Owner Skill | Creates |
|--------------|-------------|---------|
| Meeting transcripts | meeting-ingestion | brain/meetings/ pages |
| Email messages | executive-assistant | brain/people/ timeline entries |
| X/Twitter posts | x-collector | brain/media/ pages |
| Person enrichment | enrich | brain/people/ compiled truth |
| Calendar events | calendar-sync | brain/daily/calendar/ pages |
| Video/podcast content | media-ingest | brain/media/ pages |
### Quality Bar Checklist
A skill is ready when:
- [ ] Ran successfully on 3-10 real items with good output
- [ ] User reviewed output and approved
- [ ] SKILL.md is under 500 lines (use references for overflow)
- [ ] Checks notability before creating brain pages (don't create pages for nobodies)
- [ ] Has citation enforcement (every fact has a source)
- [ ] Doesn't overlap with existing skills (MECE)
- [ ] If recurring: on a cron with appropriate schedule
- [ ] If it creates brain pages: checks notability first
### What This Means in Practice
- Don't do ad-hoc brain enrichment, use the enrich skill
- Don't manually check social media, use an automated cron
- Don't manually ingest meeting notes, use the meeting-sync recipe
- Don't manually create entity pages, use the entity detector
- If a new pattern emerges, prototype it, skill-ify it, cron-ify it
## Tricky Spots
1. **MECE violations compound silently.** Two skills that both create
`brain/people/` pages will produce duplicates and conflicting data.
Before creating a new skill, check the ownership table. If an existing
skill already owns that entity type, extend it with parameters instead
of creating a new skill.
2. **The quality bar is real.** Don't ship a skill that hasn't been tested
on 3-10 real items with user approval. A skill that produces bad output
is worse than no skill -- it creates bad brain pages at scale on a cron.
3. **Don't create stubs.** A SKILL.md with "TODO: implement" is not a skill.
Every skill must be complete enough to run end-to-end on real data. If
you can't finish it, don't create the file. Keep it as manual work until
you can do it right.
## How to Verify
1. **Run the skill on 3 real items.** Execute the skill against live data
(not test data). Check that the output matches the quality bar: citations
present, notability checked, no stubs created.
2. **Check MECE against existing skills.** Review the ownership table. Does
this new skill create pages in a directory already owned by another skill?
If yes, it's a MECE violation. Merge or parameterize instead.
3. **Verify the quality bar checklist.** Walk through every item in the
Quality Bar Checklist above. If any item is unchecked, the skill isn't
ready for cron deployment.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,75 @@
# Source Attribution
## Goal
Every fact in the brain traces to where it came from -- who said it, in what context, and when.
## What the User Gets
Without this: six months from now, someone reads a brain page and has no idea if "Pedro co-founded Brex" came from Pedro himself, a LinkedIn scrape, or a hallucination. With this: every claim is auditable, conflicts are surfaced, and the brain is a court-admissible record of reality.
## Implementation
```
on brain_write(page, fact):
# EVERY fact gets a citation -- compiled truth AND timeline
citation = format_citation(source)
# format: [Source: {who}, {channel/context}, {date} {time} {tz}]
# Category-specific formats:
if source.type == "direct":
# [Source: User, direct message, 2026-04-07 12:33 PM PT]
elif source.type == "meeting":
# [Source: Meeting notes "Team Sync" #12345, 2026-04-03 12:11 PM PT]
elif source.type == "api_enrichment":
# [Source: Crustdata LinkedIn enrichment, 2026-04-07 12:35 PM PT]
elif source.type == "social_media":
# MUST include full URL -- not just @handle
# [Source: X/@pedroh96 tweet, product launch, 2026-04-07](https://x.com/pedroh96/status/...)
elif source.type == "email":
# [Source: email from Sarah Chen re Q2 board deck, 2026-04-05 2:30 PM PT]
elif source.type == "workspace":
# [Source: Slack #engineering, Keith re deploy schedule, 2026-04-06 11:45 AM PT]
elif source.type == "web":
# [Source: Happenstance research, 2026-04-07 12:35 PM PT]
elif source.type == "published":
# [Source: [Wall Street Journal, 2026-04-05](https://wsj.com/...)]
elif source.type == "funding":
# [Source: Captain API funding data, 2026-04-07 2:00 PM PT]
# Attach citation inline with the fact
gbrain put <slug> --content "...fact [Source: ...]..."
# When sources conflict, note BOTH -- never silently pick one
if conflicts_exist(fact, existing_page):
append_to_compiled_truth(
"Conflict: Source A says X, Source B says Y. "
"[Source: A] [Source: B]"
)
# Source hierarchy for conflict resolution (highest authority first):
SOURCE_PRIORITY = [
"User direct statements", # 1 -- always wins
"Primary sources", # 2 -- meetings, emails, direct conversations
"Enrichment APIs", # 3 -- Crustdata, Happenstance, Captain
"Web search results", # 4
"Social media posts", # 5
]
```
## Tricky Spots
1. **Compiled truth is NOT exempt from citations.** "Pedro co-founded Brex" in the synthesis section needs `[Source: ...]` just as much as a timeline entry does. Most agents skip citations above the bar.
2. **Tweet URLs are mandatory.** `[Source: X/@handle tweet, topic, date]` without a URL is a broken citation. Hundreds of brain pages end up with unreachable tweet references when the URL is omitted. Always: `[Source: X/@handle tweet, topic, date](https://x.com/handle/status/ID)`.
3. **"User said it" isn't enough.** WHERE, ABOUT WHAT, WHEN. `[Source: User, direct message, 2026-04-07 12:33 PM PT]` -- not just `[Source: User]`.
4. **Don't silently resolve conflicts.** When the user says one thing and an API says another, note the contradiction in compiled truth with both citations. Let the reader decide.
5. **Timeline entries need sources too.** Every append to the timeline carries provenance. A timeline entry without a source is an orphan fact.
## How to Verify
1. Open any brain page with `gbrain get <slug>`. Read the compiled truth section above the bar. Every factual claim should have an inline `[Source: ...]` citation.
2. Search for tweet references: `gbrain search "X/@"`. Every result should have a full URL, not just an @handle.
3. Find a page with data from multiple sources (e.g., a person enriched via API + mentioned in a meeting). Confirm both sources are cited independently.
4. Check timeline entries on 3 random pages. Each entry should have a source citation with date and context.
5. Look for a page where the user stated something that contradicts an API result. Confirm the contradiction is noted, not silently resolved.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,122 @@
# Sub-Agent Model Routing
## Goal
Route sub-agents to the cheapest model that can do the job, saving 10-40x on costs without sacrificing quality.
## What the User Gets
Without this: every sub-agent runs on Opus ($15/MTok). Entity detection on
every message costs $3-5/day. Research tasks cost $10+ each.
With this: entity detection runs on Sonnet ($3/MTok, 5x cheaper). Research
runs on DeepSeek ($0.50/MTok, 30x cheaper). Main session stays on Opus for
quality. Total cost drops 70-80%.
## Implementation
### Routing Table
| Task Type | Recommended Model | Why |
|-----------|------------------|-----|
| Main session / complex instructions | Opus-class (default) | Best reasoning and instruction following |
| Research / synthesis / analysis | DeepSeek V3 or equivalent | 25-40x cheaper, strong on exploratory work |
| Structured output / long context | Large context model (Qwen, Gemini) | 200K+ context, reliable JSON output |
| Fast lightweight sub-agents | Fast inference model (Groq) | 500 tok/s, cheap, good for quick tasks |
| Deep reasoning (use sparingly) | Reasoning model (DeepSeek-R1, o3) | Best for hard problems, expensive |
| Entity detection (signal detector) | Sonnet-class | Fast, cheap, sufficient quality for detection |
### The Signal Detector Pattern
Spawn a lightweight sub-agent on EVERY inbound message. This is mandatory.
```
on_every_message(text):
// Spawn async — don't block the response
spawn_subagent({
task: `SIGNAL DETECTION — scan this message:
"${text}"
1. IDEAS FIRST: Is the user expressing an original thought?
If yes -> create/update brain/originals/ with EXACT phrasing
2. ENTITIES: Extract person names, company names, media titles
For each -> check brain, create/enrich if notable
3. FACTS: New info about existing entities -> update timeline
4. CITATIONS: Every fact needs [Source: ...] attribution
5. Sync changes to brain repo`,
model: "sonnet-class", // fast + cheap
timeout: 120s
})
```
**Why Sonnet-class for detection:** Entity detection is pattern matching, not
deep reasoning. Sonnet is 5-10x cheaper than Opus and fast enough for async
detection. The main session continues on Opus while detection runs in parallel.
### Research Pipeline Pattern
For research-heavy tasks, use a multi-model pipeline:
```
1. PLANNING (Opus): Write research brief, identify what to look for
2. EXECUTION (DeepSeek): Sub-agent does the actual research (web, APIs, docs)
3. SYNTHESIS (Opus): Read research output, add strategic analysis
```
**Why this works:** The planning and synthesis steps need taste and judgment
(Opus). The execution step is mechanical data gathering (DeepSeek at 25-40x
lower cost). You get Opus-quality output at DeepSeek-level cost for 80% of
the work.
### When to Spawn Sub-Agents
| Situation | Spawn? | Model |
|-----------|--------|-------|
| Every inbound message | YES (mandatory) | Sonnet |
| Research request | YES | DeepSeek for execution |
| Quick lookup / fact check | YES | Fast model (Groq) |
| Complex analysis | NO -- handle in main session | Opus |
| Writing / editing | NO -- handle in main session | Opus |
### Cost Optimization
The main session runs on your best model. Everything else runs on the
cheapest model that can do the job. In practice, 60-70% of sub-agent
work is entity detection (Sonnet) and research execution (DeepSeek),
which are 10-40x cheaper than the main session model.
## Tricky Spots
1. **Sonnet, not Opus, for detection.** The most common mistake is running
entity detection on Opus. Detection is pattern matching, not deep reasoning.
Sonnet is 5-10x cheaper and fast enough. Reserve Opus for the main session
where reasoning quality matters.
2. **Don't block the main thread.** Sub-agents must run asynchronously. If the
signal detector runs synchronously, the user waits 30-120 seconds for every
message while entity detection completes. Spawn and forget. The user sees
a response immediately.
3. **Cost optimization is multiplicative.** Entity detection runs on every
single message. If you use Opus at $15/MTok for detection across 50
messages/day, that's $3-5/day just for detection. Sonnet at $3/MTok brings
that to $0.60-1.00/day. Over a month, the wrong model choice costs $100+
more than necessary.
## How to Verify
1. **Spawn a signal detector and check the model.** Send a message and verify
the sub-agent was spawned on Sonnet-class, not Opus. Check the model field
in the sub-agent config or logs.
2. **Check cost per day.** After running for a day with sub-agent routing,
compare total API costs against the previous day without routing. You
should see a 50-80% reduction in total cost.
3. **Verify async execution.** Send a message and measure response time. The
response should arrive in under 5 seconds. If it takes 30+ seconds, the
signal detector is running synchronously and blocking the main thread.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

View File

@@ -0,0 +1,182 @@
# Upgrades and Auto-Update Notifications
## Goal
Users get notified of new GBrain features conversationally, and the agent walks them through upgrading with post-upgrade migrations that make the new version actually work.
## What the User Gets
Without this: GBrain ships updates but nobody knows. The user stays on an old
version with stale skills and missing features. Or worse, someone runs
`gbrain upgrade` but skips the post-upgrade steps, leaving new code with old
agent behavior.
With this: the agent checks for updates daily, sells the upgrade with punchy
benefit-focused bullets, waits for explicit permission, then runs the full
upgrade flow including re-reading skills, running migrations, and syncing
schema. The user gets new capabilities automatically.
## Implementation
### The Check (cron-initiated)
```
check_for_update():
result = run("gbrain check-update --json")
if not result.update_available:
exit_silently() // do NOT message the user
// Sell the upgrade — lead with what they can DO, not what changed
message = compose_upgrade_message(
current: result.current_version,
latest: result.latest_version,
changelog: result.changelog
)
send_to_user(message, respect_quiet_hours=true)
```
### The Upgrade Message
Sell the upgrade. The user should feel "hell yeah, I want that." Lead with
what they can DO now that they couldn't before, not what files changed.
```
> **GBrain v0.5.0 is available** (you're on v0.4.0)
>
> What's new:
> - Your brain never falls behind. Live sync keeps the vector DB current
> automatically, so edits show up in search within minutes
> - New verification runbook catches silent failures before they bite you
> - New installs set up live sync automatically. No more manual setup step
>
> Want me to upgrade? I'll update everything and refresh my playbook.
>
> (Reply **yes** to upgrade, **not now** to skip, **weekly** to check
> less often, or **stop** to turn off update checks)
```
### Handling Responses
| User says | Action |
|-----------|--------|
| yes / y / sure / ok / do it / upgrade | Run the full upgrade flow (below) |
| not now / later / skip / snooze | Acknowledge, check again next cycle |
| weekly | Store preference, switch cron to weekly |
| daily | Store preference, switch cron back to daily |
| stop / unsubscribe / no more | Disable the cron. Tell user how to resume |
**Never auto-upgrade.** Always wait for explicit confirmation.
### The Full Upgrade Flow (after user says yes)
```
full_upgrade():
// Step 1: Update the binary/package
run("gbrain upgrade")
// Step 2: Re-read all updated skills
for skill in find("skills/*/SKILL.md"):
read_and_internalize(skill) // updated skills = better agent behavior
// Step 3: Re-read production reference docs
read("docs/GBRAIN_SKILLPACK.md")
read("docs/GBRAIN_RECOMMENDED_SCHEMA.md")
// Step 4: Check for version-specific migration directives
for version in range(old_version, new_version):
migration = find(f"skills/migrations/v{version}.md")
if migration exists:
read_and_execute(migration) // in order, don't skip
// Step 5: Schema sync — suggest new, respect declined
state = read("~/.gbrain/update-state.json")
for recommendation in new_schema_recommendations:
if recommendation not in state.declined:
suggest_to_user(recommendation)
update(state, new_choices)
// Step 6: Report what changed
summarize_to_user(actions_taken)
```
### Migration Files
Migration files live at `skills/migrations/vX.Y.Z.md`. They contain agent
instructions (not scripts) for post-upgrade actions that make the new version
work for existing users. Example: v0.5.0 migration sets up live sync and
runs the verification runbook.
The agent reads migration files in version order and executes them step by
step. Without migrations, the agent has new code but the user's environment
hasn't changed.
### Cron Registration
```
Name: gbrain-update-check
Default schedule: 0 9 * * * (daily 9 AM)
Weekly schedule: 0 9 * * 1 (Monday 9 AM)
Prompt: "Run gbrain check-update --json. If update_available is true,
summarize the changelog and message me asking if I'd like to upgrade.
If false, stay silent."
```
### Frequency Preferences
Default: daily. Store in agent memory as `gbrain_update_frequency: daily|weekly|off`.
Also persist in `~/.gbrain/update-state.json` so it survives agent context resets.
### Standalone Skillpack Users
If you loaded this SKILLPACK directly (copied or read from GitHub) without
installing gbrain, you can still stay current. Both GBRAIN_SKILLPACK.md and
GBRAIN_RECOMMENDED_SCHEMA.md have version markers:
```bash
curl -s https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_SKILLPACK.md | head -1
# Returns: <!-- skillpack-version: X.Y.Z -->
```
If the remote version is newer, fetch the full file and replace your local
copy. Set up a weekly cron to check automatically.
## Tricky Spots
1. **Never auto-install.** The upgrade must always wait for the user's explicit
"yes." Even if the cron detects an update at 9 AM and the changelog looks
great, the agent messages the user and waits. Auto-installing can break
workflows, introduce breaking changes, or interrupt work in progress.
2. **Migration files are agent instructions, not scripts.** They tell the agent
what to do step by step in plain language. They are NOT bash scripts to
execute blindly. The agent reads them, understands the context, and adapts
to the user's specific environment (e.g., skip a step if the user already
has live sync configured).
3. **check-update should run on a daily cron.** Don't rely on the user
remembering to check for updates. The cron runs `gbrain check-update --json`
daily at 9 AM (respecting quiet hours). If there's nothing new, it stays
completely silent. The user only hears about updates when there IS something
worth upgrading to.
## How to Verify
1. **Run check-update and verify detection.** Execute
`gbrain check-update --json`. Verify it returns the current version and
correctly reports whether an update is available. If `update_available`
is false, verify the version matches the latest release on GitHub.
2. **Verify migration files are readable.** List `skills/migrations/` and
check that each file follows the naming convention `vX.Y.Z.md`. Open one
and verify it contains step-by-step agent instructions, not raw scripts.
The agent should be able to read and execute each step.
3. **Test the full upgrade flow end-to-end.** If an update is available, say
"yes" and watch the agent execute the full flow: upgrade, re-read skills,
run migrations, sync schema, report. Verify each step completes and the
agent reports what changed.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md).*

104
docs/integrations/README.md Normal file
View File

@@ -0,0 +1,104 @@
# Getting Data Into Your Brain
GBrain is the retrieval layer. But retrieval is only as good as what you put in.
This directory covers how to get data flowing into your brain automatically.
## How Data Flows In
```
Signal arrives (phone call, email, tweet, calendar event)
Collector captures it (deterministic code, reliable)
Agent analyzes it (LLM, judgment, entity detection)
Brain pages created/updated (compiled truth + timeline)
GBrain indexes it (chunking, embedding, search-ready)
Next query is smarter (the compounding effect)
```
## Available Integrations
### Self-Installing Recipes
These are integration recipes your agent can set up for you. Run
`gbrain integrations` to see what's available and their status.
| Recipe | Category | Requires | What It Does | Setup Time |
|--------|----------|----------|-------------|------------|
| [ngrok-tunnel](../../recipes/ngrok-tunnel.md) | Infra | — | Fixed public URL for MCP + voice ($8/mo) | 10 min |
| [credential-gateway](../../recipes/credential-gateway.md) | Infra | — | Gmail + Calendar access (ClawVisor or Google OAuth) | 15 min |
| [voice-to-brain](../../recipes/twilio-voice-brain.md) | Sense | ngrok-tunnel | Phone calls create brain pages via Twilio + OpenAI Realtime | 30 min |
| [email-to-brain](../../recipes/email-to-brain.md) | Sense | credential-gateway | Gmail messages flow into entity pages via deterministic collector | 20 min |
| [x-to-brain](../../recipes/x-to-brain.md) | Sense | — | Twitter timeline, mentions, keyword monitoring with deletion detection | 15 min |
| [calendar-to-brain](../../recipes/calendar-to-brain.md) | Sense | credential-gateway | Google Calendar events become searchable daily brain pages | 20 min |
| [meeting-sync](../../recipes/meeting-sync.md) | Sense | — | Circleback meeting transcripts auto-import with attendee propagation | 15 min |
### Manual Integration Guides
These require manual setup (no self-installing recipe yet):
| Guide | What It Does |
|-------|-------------|
| [Credential Gateway](credential-gateway.md) | Set up ClawVisor or Hermes for Gmail, Calendar, Contacts access |
| [Meeting & Call Webhooks](meeting-webhooks.md) | Circleback meeting transcripts + Quo/OpenPhone SMS/calls |
## How to Read a Recipe
Integration recipes are markdown files with YAML frontmatter. Your agent reads
the recipe and walks you through setup.
```yaml
---
id: voice-to-brain # unique identifier
name: Voice-to-Brain # human-readable name
version: 0.7.0 # recipe version
description: Phone calls... # what it does
category: sense # sense (data input) or reflex (automated response)
requires: [] # other recipes that must be set up first
secrets: # API keys and credentials needed
- name: TWILIO_ACCOUNT_SID
description: Twilio account SID
where: https://console.twilio.com # exact URL to get this key
health_checks: # commands to verify the integration is working
- "curl -sf https://api.twilio.com/..."
setup_time: 30 min # estimated time to complete setup
---
[Setup instructions the agent follows step by step...]
```
**The recipe IS the installer.** Your agent (OpenClaw, Hermes, Claude Code) reads
the markdown body and executes the setup steps. It asks you for API keys, validates
each one, configures the integration, and runs a smoke test.
## The Deterministic Collector Pattern
When an LLM keeps failing at a mechanical task despite repeated prompt fixes,
stop fighting the LLM. Move the mechanical work to code.
**Code for data. LLMs for judgment.**
- Email collection: code pulls emails with baked-in links (100% reliable).
LLM reads the digest, classifies, enriches brain entries (judgment).
- Tweet collection: code pulls timeline, detects deletions, tracks engagement
(deterministic). LLM extracts entities, writes brain updates (judgment).
- Calendar sync: code pulls events and attendees (deterministic). LLM enriches
attendee brain pages (judgment).
This pattern prevents the "LLM forgot the links" failure mode. Mechanical work
must be 100% reliable. Judgment work is where LLMs shine.
See [Deterministic Collectors](../guides/deterministic-collectors.md) for the
full pattern.
## Architecture
For details on the shared infrastructure that all integrations build on
(import pipeline, chunking, embedding, search), see the
[Infrastructure Layer](../architecture/infra-layer.md).
For the philosophy behind thin harness + fat skills, see
[Thin Harness, Fat Skills](../ethos/THIN_HARNESS_FAT_SKILLS.md).

View File

@@ -0,0 +1,52 @@
# Credential Gateway (ClawVisor / Hermes)
Three integrations that make the agent real. Without these, the brain is a static
database. With them, it's alive.
### 14a. Credential Gateway (ClawVisor / Hermes Gateway)
The EA workflow needs Gmail, Calendar, Contacts, and messaging access. The agent
should never hold API keys directly. Use a credential gateway that enforces policies
and injects credentials at request time.
**OpenClaw: ClawVisor.** [ClawVisor](https://clawvisor.com) is a credential vaulting
and authorization gateway with task-scoped authorization.
**Services:** Gmail (list, read, send, draft), Google Calendar (CRUD), Google Drive
(list, search, read), Google Contacts (list, search), Apple iMessage (list, read,
search, send), GitHub, Slack.
**Task-scoped authorization:** Every request must include a `task_id` from an approved
standing task. Tasks declare: purpose (verbose, 2-3 sentences), authorized actions with
expected use patterns, auto-execute flag, lifetime (standing vs ephemeral).
**Why this matters for GBrain:** The EA workflow needs Gmail (sender lookup before
triage), Calendar (meeting prep, attendee pages), Contacts (enrichment trigger), and
iMessage (direct instructions). ClawVisor gives the agent access without giving it
raw credentials.
**Setup:**
1. Create agent in ClawVisor dashboard, copy agent token
2. Set `CLAWVISOR_URL` and `CLAWVISOR_AGENT_TOKEN` in env
3. Activate services (Google, iMessage, etc.) in the dashboard
4. Create standing tasks with expansive scopes (narrow purposes cause false blocks)
5. Store standing task IDs in agent memory for reuse
**Critical scoping rule:** Be expansive in task purposes. "Full executive assistant
email management including inbox triage, searching by any criteria, reading emails,
tracking threads" works. "Email triage" gets rejected. The intent verification model
uses the purpose to judge whether each request is consistent -- if your purpose is
narrow, legitimate requests fail verification.
**Hermes Agent: Built-in gateway.** Hermes has multi-platform messaging (Telegram,
Discord, Slack, WhatsApp, Signal, Email) and tool access built into its gateway. Use
`config.yaml` to configure API credentials. The gateway daemon manages connections
and routes webhooks to agent sessions. For Google services, configure OAuth credentials
in the gateway config. Hermes's scheduled automations can run the same EA workflows
(email triage, calendar prep, contact enrichment) through the gateway's tool system.
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Getting Data In](README.md)*

View File

@@ -0,0 +1,63 @@
# Meeting & Call Webhooks
### 14b. Circleback -- Meeting Ingestion via Webhooks
[Circleback](https://circleback.ai) records meetings, generates transcripts with
speaker diarization, and fires webhooks on completion.
**Webhook setup:**
1. In Circleback dashboard -> Automations -> add webhook
2. URL: `{your_agent_gateway}/hooks/circleback-meetings`
3. Circleback provides a signing secret for HMAC-SHA256 signature verification
4. Store the signing secret in your webhook transform for verification
**Webhook payload:** Meeting JSON with id, name, attendees, notes, action items, full
transcript, calendar event context.
**Signature verification:** Header `X-Circleback-Signature` contains `sha256=<hex>`.
Verify with `HMAC-SHA256(body, signing_secret)`. Reject unverified webhooks.
**OAuth for API access:** Circleback uses dynamic client registration (OAuth 2.0).
Access tokens expire in ~24h, auto-refresh via refresh token. Store credentials in
agent memory.
**Flow:** Webhook fires -> transform validates signature + normalizes -> agent wakes ->
pulls full transcript via API -> creates brain meeting page -> propagates to entity
pages -> commits to brain repo -> `gbrain sync`.
### 14c. Quo (OpenPhone) -- SMS and Call Integration
[Quo](https://openphone.com) (formerly OpenPhone) provides business phone numbers with
SMS, calls, voicemail, and AI transcripts.
**Webhook setup:**
1. In Quo dashboard -> Integrations -> Webhooks
2. Register webhooks for: `message.received`, `call.completed`, `call.summary.completed`, `call.transcript.completed`
3. Point all to: `{your_agent_gateway}/hooks/quo-events`
4. Store registered webhook IDs in agent memory
**How inbound texts work:**
- Webhook fires with sender phone, message text, conversation context
- Agent looks up sender in brain by phone number
- Surfaces to user's messaging platform with sender identity + brain context
- Drafts reply for approval (never auto-replies without explicit permission)
**How inbound calls work:**
- `call.completed` fires -> if duration > 30s, fetch transcript + AI summary via API
- Ingest to brain (meeting-style page at `meetings/`)
- Update relevant people and company pages
**API auth:** Bare API key in `Authorization` header (no Bearer prefix).
**Key endpoints:** `POST /v1/messages` (send SMS), `GET /v1/messages` (list),
`GET /v1/call-transcripts/{id}`, `GET /v1/conversations`.
---
---
*Part of the [GBrain Skillpack](../GBRAIN_SKILLPACK.md). See also: [Getting Data In](README.md)*

View File

@@ -1,6 +1,6 @@
{ {
"name": "gbrain", "name": "gbrain",
"version": "0.5.0", "version": "0.7.0",
"description": "Postgres-native personal knowledge brain with hybrid RAG search", "description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module", "type": "module",
"main": "src/core/index.ts", "main": "src/core/index.ts",

View File

@@ -0,0 +1,396 @@
---
id: calendar-to-brain
name: Calendar-to-Brain
version: 0.7.0
description: Google Calendar events become searchable brain pages. Daily files with attendees, locations, and meeting prep context.
category: sense
requires: [credential-gateway]
secrets:
- name: CLAWVISOR_URL
description: ClawVisor gateway URL (Option A — recommended, handles OAuth for you)
where: https://clawvisor.com — create an agent, activate Google Calendar service
- name: CLAWVISOR_AGENT_TOKEN
description: ClawVisor agent token (Option A)
where: https://clawvisor.com — agent settings, copy the agent token
- name: GOOGLE_CLIENT_ID
description: Google OAuth2 client ID (Option B — direct API access, you manage tokens)
where: https://console.cloud.google.com/apis/credentials — create OAuth 2.0 Client ID
- name: GOOGLE_CLIENT_SECRET
description: Google OAuth2 client secret (Option B)
where: https://console.cloud.google.com/apis/credentials — same page as client ID
health_checks:
- "[ -n \"$CLAWVISOR_URL\" ] && curl -sf $CLAWVISOR_URL/health > /dev/null && echo 'ClawVisor: OK' || [ -n \"$GOOGLE_CLIENT_ID\" ] && echo 'Google OAuth: configured' || echo 'No calendar auth configured'"
setup_time: 20 min
cost_estimate: "$0 (both options are free)"
---
# Calendar-to-Brain: Your Schedule Becomes Searchable Memory
Every calendar event becomes a searchable brain page. Your agent knows who you're
meeting tomorrow, what you discussed last time, and what context matters. Meeting
prep happens automatically because the brain already has the history.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Follow these steps precisely.
**Why this matters:** Calendar data is the richest source of relationship history.
13 years of calendar data tells you who you've met with, how often, where, and
with whom. When someone emails you, the brain already knows your meeting history.
When you have a meeting tomorrow, the agent pulls attendee dossiers automatically.
**The output is daily markdown files:** One file per day at
`brain/daily/calendar/{YYYY}/{YYYY-MM-DD}.md` with all events, attendees, and
locations. These files are the foundation for meeting prep, relationship tracking,
and pattern detection.
**Do not skip steps. Verify after each step.**
## Architecture
```
Google Calendar (multiple accounts)
↓ (ClawVisor credential gateway, paginated)
Calendar Sync Script (deterministic Node.js)
↓ Outputs:
├── brain/daily/calendar/{YYYY}/{YYYY-MM-DD}.md (daily event files)
├── brain/daily/calendar/.raw/events-{range}.json (raw API responses)
└── brain/daily/calendar/INDEX.md (date ranges + monthly summary)
Agent reads daily files
↓ Judgment calls:
├── Attendee enrichment (create/update brain pages for people)
├── Meeting prep (pull context before tomorrow's meetings)
└── Pattern detection (meeting frequency, relationship temperature)
```
## Opinionated Defaults
**Multiple calendar accounts:**
- Work calendar (company domain)
- Personal calendar (gmail.com)
- Previous company calendars (if still accessible)
**Daily file format:**
```markdown
# 2026-04-10 (Thursday)
- 09:00-09:30 **Team standup** (Work) — with Alice, Bob, Carol
- 10:00-11:00 **Board meeting** (Work) 📍 Office — with Diana, Eduardo, Fiona
- 12:00-13:00 **Lunch with Pedro** (Personal) 📍 Chez Panisse — with Pedro Franceschi
- 14:00-14:30 **1:1 with Jordan** (Work) — with Jordan Lee
```
All-day events listed first. Timed events sorted by start time.
Cancelled events are skipped. Attendee names extracted (no email addresses in output).
Calendar label in parentheses. Location with 📍 emoji.
**Historical backfill:** Sync years of calendar data, not just recent. Common ranges:
- Work: 2020-present
- Personal: 2014-present
This builds the full relationship graph from day one.
## Prerequisites
1. **GBrain installed and configured** (`gbrain doctor` passes)
2. **Node.js 18+** (for the sync script)
3. **Google Calendar access** via ONE of:
- **Option A: ClawVisor** (recommended, handles OAuth for you, no token management)
- **Option B: Google OAuth2 directly** (you manage tokens, no extra service needed)
## Setup Flow
### Step 1: Choose and Configure Calendar Access
Ask the user: "How do you want to connect to Google Calendar?
**Option A: ClawVisor (recommended)**
ClawVisor handles OAuth, token refresh, and encryption. You never touch Google
credentials directly. If you already use ClawVisor for email, this uses the same setup.
**Option B: Google OAuth2 directly**
Connect to Google Calendar API directly. No extra service needed, but you manage
OAuth tokens yourself. Good if you don't want another dependency."
#### Option A: ClawVisor Setup
Tell the user:
"I need your ClawVisor URL and agent token.
1. Go to https://clawvisor.com
2. Create an agent (or use existing)
3. Activate the **Google Calendar** service
4. Create a standing task with purpose: 'Full calendar access for historical
backfill and ongoing sync. List events, read event details, search across
all calendars.'
IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes block requests.
5. Copy the gateway URL and agent token"
Validate:
```bash
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
```
**STOP until ClawVisor validates.**
#### Option B: Google OAuth2 Setup
Tell the user:
"I need Google OAuth2 credentials. Here's exactly how to set them up:
1. Go to https://console.cloud.google.com/apis/credentials
(create a Google Cloud project if you don't have one)
2. Click **'+ CREATE CREDENTIALS'** at the top, select **'OAuth client ID'**
3. If prompted, configure the OAuth consent screen first:
- User type: **External** (or Internal if you have Google Workspace)
- App name: anything (e.g., 'GBrain Calendar')
- Scopes: add **'Google Calendar API .../auth/calendar.readonly'**
- Test users: add your own email
4. Back on Credentials, create the OAuth client ID:
- Application type: **Desktop app**
- Name: anything (e.g., 'GBrain')
5. Click **'Create'**. You'll see the Client ID and Client Secret.
6. Copy both and paste them to me.
Also enable the Calendar API:
7. Go to https://console.cloud.google.com/apis/library/calendar-json.googleapis.com
8. Click **'Enable'**"
Validate the credentials are set:
```bash
[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \
&& echo "PASS: Google OAuth credentials set" \
|| echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET"
```
Then run the OAuth flow to get an access token:
```bash
# The sync script should handle the OAuth flow:
# 1. Open browser to Google auth URL with calendar.readonly scope
# 2. User grants access
# 3. Script receives auth code, exchanges for access + refresh token
# 4. Stores tokens in ~/.gbrain/google-tokens.json
# 5. Auto-refreshes on expiry
```
**STOP until OAuth flow completes and tokens are stored.**
### Step 2: Identify Calendar Accounts
Ask the user: "Which Google Calendar accounts should I sync? Common setup:
- Work email (e.g., you@company.com)
- Personal email (e.g., you@gmail.com)
- Any previous company emails with calendar history"
For each account, note:
- Email address
- Start year (how far back to sync)
- Label (Work, Personal, etc.)
### Step 3: Set Up the Calendar Sync Script
Create the sync directory:
```bash
mkdir -p calendar-sync
cd calendar-sync
npm init -y
```
The sync script needs these capabilities:
1. **Paginated event retrieval** — Google Calendar API returns max 50 events per
request. The script must paginate through large date ranges. Use monthly chunks
for sparse periods, weekly for dense ones.
2. **Daily markdown generation** — group events by date, format as markdown with
times, attendees, locations, calendar labels
3. **Merge with existing files** — if a daily file already has manual notes, preserve
them when updating calendar data
4. **Index generation** — create INDEX.md with date ranges, event counts, monthly summary
5. **Raw JSON preservation** — save raw API responses to `.raw/` for provenance
### Step 4: Run Historical Backfill
This is the big initial sync. It may take 10-30 minutes depending on how many
years of calendar data you have.
```bash
node calendar-sync.mjs --start 2020-01-01 --end $(date +%Y-%m-%d)
```
Tell the user: "Syncing calendar history from [start year]. This creates one
markdown file per day. For 4 years of data, expect ~1,400 daily files."
Verify:
```bash
ls brain/daily/calendar/2026/ | head -10
```
Should show daily files like `2026-04-01.md`, `2026-04-02.md`, etc.
### Step 5: Import Calendar Data to GBrain
```bash
gbrain import brain/daily/calendar/ --no-embed
gbrain embed --stale
```
Verify:
```bash
gbrain search "meeting" --limit 3
```
Should return calendar pages with event details.
### Step 6: Attendee Enrichment
This is YOUR job (the agent). For each person who appears in calendar events:
1. **Check brain**: `gbrain search "attendee name"` — do they have a page?
2. **Create page if missing**: notable attendees (appears 3+ times) get a brain page
3. **Update existing pages**: add meeting history to timeline:
`- YYYY-MM-DD | Meeting: {event title} [Source: Google Calendar]`
4. **Relationship tracking**: note meeting frequency in compiled truth:
"Met 12 times in last 6 months. Regular 1:1 cadence."
### Step 7: Set Up Weekly Sync
The calendar should sync weekly to stay current:
```bash
# Cron: every Sunday at 10 AM
0 10 * * 0 cd /path/to/calendar-sync && node calendar-sync.mjs --start $(date -v-7d +%Y-%m-%d) --end $(date +%Y-%m-%d)
```
After sync, import new data:
```bash
gbrain sync --no-pull --no-embed && gbrain embed --stale
```
### Step 8: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/calendar-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"accounts":"ACCOUNT_COUNT","start_year":"YYYY"}}' >> ~/.gbrain/integrations/calendar-to-brain/heartbeat.jsonl
```
Tell the user: "Calendar-to-brain is set up. You have [N] days of calendar history
indexed. I can now prep you for meetings by pulling attendee context from the brain.
Weekly sync keeps it current."
## Implementation Guide
These are production-tested patterns from syncing 13 years of calendar data.
### Smart Chunking (Monthly vs Weekly)
```
generate_chunks(start, end, dense_after='2023-01-01'):
chunks = []
current = start
while current < end:
if current < dense_after:
next = current + 1_MONTH // sparse period: monthly
else:
next = current + 7_DAYS // dense period: weekly
chunks.append({from: current, to: min(next, end)})
current = next
return chunks
```
**Why:** Monthly chunks for sparse years (2014-2023) = ~96 API calls for 8 years.
Weekly for everything would be ~600+ calls. Per-calendar `startYear` avoids
pulling empty months (e.g., don't query 2014-2020 for a calendar created in 2020).
### Attendee Filtering
```
filter_attendees(attendees):
return attendees.filter(a =>
!a.email?.includes('@resource.calendar.google.com') AND // conference rooms
!a.email?.includes('@group.calendar.google.com') AND // mailing lists
!a.name?.startsWith('YC-SF-') // internal distros
)
```
Without this, your attendee list is polluted with "Conference Room A" and
"engineering-all@company.com". You want actual people.
### Merge with Existing Files (Preserve Manual Notes)
```
write_daily_file(date, events, dir):
path = f'{dir}/{date}.md'
calendar_md = format_events(events)
if file_exists(path):
existing = read(path)
if '## Calendar' in existing:
// Replace ONLY the calendar section, keep everything else
before = existing.split('## Calendar')[0]
after_match = regex_search(existing, /## [A-Z](?!alendar)/) // next section
after = after_match ? existing[match_index:] : ''
write(path, f'{before}## Calendar\n\n{calendar_md}\n{after}')
else:
write(path, f'## Calendar\n\n{calendar_md}\n---\n\n{existing}')
else:
write(path, calendar_md)
```
**Critical:** Only touch `## Calendar`. Everything else is preserved. If you
manually added `## Notes` to a daily file, it survives re-sync.
### Date/Time Parsing Edge Cases
```
parse_event_date(event):
// All-day: event.start = "2024-01-15" (no T)
// Timed: event.start = "2024-01-15T10:00:00-08:00" (with T)
if 'T' in event.start:
return event.start[0:10] // extract date from datetime
return event.start // already a date
format_time(iso_str):
if not iso_str or 'T' not in iso_str: return 'all-day'
// Extract hours:minutes, convert to 12-hour
// Edge: 00:00 = 12:00 AM, 12:00 = 12:00 PM, 13:00 = 1:00 PM
```
### What the Agent Should Test After Setup
1. **Monthly vs weekly:** Run from 2014 with dense_after=2023. Verify pre-2023
makes ~12 API calls per year, post-2023 makes ~4 per month.
2. **Attendee filtering:** Create a meeting with a conference room and a mailing
list. Sync. Verify neither appears in the daily file.
3. **Merge preservation:** Add `## Notes` to a daily file manually. Sync calendar.
Verify notes are preserved.
4. **All-day events:** Create an all-day event and a timed event on the same day.
Verify all-day appears first, timed events sorted by start time.
5. **Cancelled events:** Cancel a meeting. Sync. Verify it doesn't appear.
6. **Per-calendar startYear:** Sync a calendar created in 2022 with startYear=2022.
Verify no API calls for years before 2022.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| ClawVisor (free tier) | $0 |
| Google Calendar API | $0 (within free quota) |
| **Total** | **$0** |
## Troubleshooting
**No events returned:**
- Check the calendar account email is correct
- Check ClawVisor has Google Calendar service activated
- Check the standing task purpose is expansive enough
- Some calendars may be empty for the requested date range
**Attendee names missing:**
- Google Calendar sometimes returns email addresses instead of display names
- The sync script should extract the display name from the attendee object
- If no display name, use the email prefix (before @)
**Duplicate events:**
- The sync script should be idempotent (same date range = same output)
- If running multiple times, existing daily files are overwritten (not appended)

View File

@@ -0,0 +1,180 @@
---
id: credential-gateway
name: Credential Gateway
version: 0.7.0
description: Secure access to Gmail, Google Calendar, and other Google services. ClawVisor (recommended) or direct Google OAuth.
category: infra
requires: []
secrets:
- name: CLAWVISOR_URL
description: ClawVisor gateway URL (Option A — recommended)
where: https://clawvisor.com — create an agent, copy the gateway URL
- name: CLAWVISOR_AGENT_TOKEN
description: ClawVisor agent token (Option A)
where: https://clawvisor.com — agent settings, copy the agent token
- name: GOOGLE_CLIENT_ID
description: Google OAuth2 client ID (Option B — direct API)
where: https://console.cloud.google.com/apis/credentials — create OAuth 2.0 Client ID
- name: GOOGLE_CLIENT_SECRET
description: Google OAuth2 client secret (Option B)
where: https://console.cloud.google.com/apis/credentials — same page as client ID
health_checks:
- "[ -n \"$CLAWVISOR_URL\" ] && curl -sf $CLAWVISOR_URL/health > /dev/null && echo 'ClawVisor: OK' || [ -n \"$GOOGLE_CLIENT_ID\" ] && echo 'Google OAuth: configured' || echo 'No credential gateway configured'"
setup_time: 15 min
cost_estimate: "$0 (both options are free)"
---
# Credential Gateway: Secure Access to Google Services
Gmail, Google Calendar, Google Contacts, and other services require OAuth
credentials. This recipe sets up secure access that email-to-brain and
calendar-to-brain depend on.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Other recipes depend on this one. If the user wants
email-to-brain or calendar-to-brain, set up credential-gateway FIRST.
**Two options, both free:**
- **Option A: ClawVisor** — handles OAuth, token refresh, and encryption for you.
No token management. If you use multiple Google services, set up ClawVisor once
and all recipes use it.
- **Option B: Google OAuth directly** — no extra service, but you manage tokens
yourself. Good if you don't want another dependency.
**Do not skip steps. Verify after each step.**
## Setup Flow
### Step 1: Choose Your Gateway
Ask the user: "How do you want to connect to Google services (Gmail, Calendar)?
**Option A: ClawVisor (recommended)**
ClawVisor handles OAuth, token refresh, and encryption. Set it up once and
email-to-brain, calendar-to-brain, and any future Google service recipes
all use the same credentials. No token management on your end.
**Option B: Google OAuth2 directly**
Connect to Google APIs directly. No extra service. But you manage OAuth
tokens yourself (they expire, need refresh)."
#### Option A: ClawVisor Setup
Tell the user:
"1. Go to https://clawvisor.com and create an account
2. Create an agent (or use existing one)
3. Activate the services you need:
- **Gmail** (for email-to-brain)
- **Google Calendar** (for calendar-to-brain)
- **Google Contacts** (for enrichment)
4. Create a standing task with a broad purpose. CRITICAL: be EXPANSIVE.
Good purpose: 'Full executive assistant access to Gmail, Calendar, and
Contacts including inbox triage, event listing, contact lookup, and
historical data access for all connected Google accounts.'
Bad purpose: 'email triage' — too narrow, blocks legitimate requests.
5. Copy the **Gateway URL** and **Agent Token** and paste them to me"
Validate:
```bash
curl -sf "$CLAWVISOR_URL/health" \
&& echo "PASS: ClawVisor reachable" \
|| echo "FAIL: ClawVisor not reachable — check the URL"
```
**STOP until ClawVisor validates.**
#### Option B: Google OAuth2 Setup
Tell the user:
"I need Google OAuth2 credentials. Here's exactly how:
1. Go to https://console.cloud.google.com/apis/credentials
(create a Google Cloud project if you don't have one — it's free)
2. Click **'+ CREATE CREDENTIALS'** at the top > **'OAuth client ID'**
3. If prompted to configure the consent screen:
- User type: **External** (or Internal for Google Workspace)
- App name: 'GBrain' (anything works)
- Scopes: add the ones you need:
- Gmail: `https://www.googleapis.com/auth/gmail.readonly`
- Calendar: `https://www.googleapis.com/auth/calendar.readonly`
- Contacts: `https://www.googleapis.com/auth/contacts.readonly`
- Test users: add your own email address
4. Create the OAuth client ID:
- Application type: **Desktop app**
- Name: 'GBrain'
5. Click **'Create'** — copy the **Client ID** and **Client Secret**
6. Enable the APIs you need:
- Gmail: https://console.cloud.google.com/apis/library/gmail.googleapis.com
- Calendar: https://console.cloud.google.com/apis/library/calendar-json.googleapis.com
Click **'Enable'** on each one.
Paste the Client ID and Client Secret to me."
Validate:
```bash
[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \
&& echo "PASS: Google OAuth credentials set" \
|| echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET"
```
Then run the OAuth flow:
```
// The first time a recipe uses these credentials, it will:
// 1. Open a browser to the Google consent URL
// 2. User grants access
// 3. Script receives auth code, exchanges for access + refresh token
// 4. Stores tokens in ~/.gbrain/google-tokens.json
// 5. Auto-refreshes when tokens expire (refresh token is long-lived)
```
**STOP until OAuth credentials validate.**
### Step 2: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/credential-gateway
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"type":"CLAWVISOR_OR_GOOGLE"}}' >> ~/.gbrain/integrations/credential-gateway/heartbeat.jsonl
```
Tell the user: "Credential gateway is set up. Email-to-brain and calendar-to-brain
can now access your Google services."
## Tricky Spots
1. **ClawVisor task purpose must be EXPANSIVE.** "Email triage" is too narrow and
blocks legitimate requests. Use a broad purpose that covers everything you
might want to do with email. The intent verification model checks each
request against the purpose. Narrow = blocked.
2. **Google OAuth tokens expire.** Access tokens last ~1 hour. The refresh token
is long-lived but can be revoked. Store both in `~/.gbrain/google-tokens.json`
with 0600 permissions. The script should auto-refresh on 401.
3. **Google consent screen in "Testing" mode** limits to 100 users and tokens
expire weekly. For personal use this is fine. For production, publish the app.
4. **Multiple Google accounts.** If you have work + personal Gmail, you need to
authorize each one separately in the OAuth flow. ClawVisor handles this
automatically.
## How to Verify
1. **ClawVisor:** `curl $CLAWVISOR_URL/health` returns OK.
2. **Google OAuth:** Tokens exist at `~/.gbrain/google-tokens.json`.
3. **Gmail access:** Run the email collector — it should pull recent messages.
4. **Calendar access:** Run the calendar sync — it should pull today's events.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| ClawVisor | $0 (free tier) |
| Google OAuth | $0 (free, no billing needed for personal use) |
---
*Part of the [GBrain Skillpack](../docs/GBRAIN_SKILLPACK.md). See also: [Email-to-Brain](email-to-brain.md), [Calendar-to-Brain](calendar-to-brain.md)*

333
recipes/email-to-brain.md Normal file
View File

@@ -0,0 +1,333 @@
---
id: email-to-brain
name: Email-to-Brain
version: 0.7.0
description: Gmail messages flow into brain pages. Deterministic collector pulls emails, agent analyzes and enriches entities.
category: sense
requires: [credential-gateway]
secrets:
- name: CLAWVISOR_URL
description: ClawVisor gateway URL (Option A — recommended, handles OAuth for you)
where: https://clawvisor.com — create an agent, activate Gmail service
- name: CLAWVISOR_AGENT_TOKEN
description: ClawVisor agent token (Option A)
where: https://clawvisor.com — agent settings, copy the agent token
- name: GOOGLE_CLIENT_ID
description: Google OAuth2 client ID (Option B — direct Gmail API access)
where: https://console.cloud.google.com/apis/credentials — create OAuth 2.0 Client ID
- name: GOOGLE_CLIENT_SECRET
description: Google OAuth2 client secret (Option B)
where: https://console.cloud.google.com/apis/credentials — same page as client ID
health_checks:
- "[ -n \"$CLAWVISOR_URL\" ] && curl -sf $CLAWVISOR_URL/health > /dev/null && echo 'ClawVisor: OK' || [ -n \"$GOOGLE_CLIENT_ID\" ] && echo 'Google OAuth: configured' || echo 'No email auth configured'"
setup_time: 20 min
cost_estimate: "$0 (both options are free)"
---
# Email-to-Brain: Gmail Messages That Update Your Brain
Emails arrive. Brain pages get smarter. The agent reads your inbox, detects
entities, updates person and company pages, extracts action items, and files
everything with source attribution.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Follow these steps precisely.
**The core pattern: code for data, LLMs for judgment.**
Email collection is split into two layers:
1. DETERMINISTIC: code pulls emails, generates Gmail links, detects noise/signatures.
This never fails. Links are always correct. Timestamps are always accurate.
2. LATENT: you (the agent) read the collected emails and make judgment calls.
Who is important? What entities are mentioned? What action items exist?
**Do not try to pull emails yourself.** Use the collector script. It handles
pagination, deduplication, Gmail link generation, and noise filtering. If you
try to do this via raw API calls, you WILL forget links, miss emails, or break
pagination. The collector exists because LLMs kept failing at this.
**Why sequential execution matters:**
- Step 1 validates the credential gateway. Without it, nothing connects to Gmail.
- Step 2 sets up the collector. Without it, you have no emails to analyze.
- Step 3 does the first collection. Without data, Step 4 can't enrich.
- Step 4 is YOUR job: read the digest, update brain pages.
## Architecture
```
Gmail Account(s)
↓ (ClawVisor E2E encrypted gateway)
Email Collector (deterministic Node.js script)
↓ Outputs:
├── messages/{YYYY-MM-DD}.json (structured email data)
├── digests/{YYYY-MM-DD}.md (markdown digest for agent)
└── state.json (pagination state, known IDs)
Agent reads digest
↓ Judgment calls:
├── Entity detection (people, companies mentioned)
├── Brain page updates (timeline entries, compiled truth)
├── Action item extraction
└── Priority classification (urgent / normal / noise)
```
## Opinionated Defaults
**Noise filtering (deterministic, in collector):**
- Skip: noreply@, notifications@, calendar-notification@
- Flag: DocuSign, Dropbox Sign, HelloSign, PandaDoc (signatures needing action)
- Keep: everything else
**Email accounts:** Configure multiple accounts. Common setup:
- Work email (company domain)
- Personal email (gmail.com)
**Digest format:** Daily markdown with sections:
- Signatures pending (DocuSign etc. needing action)
- Messages to triage (real emails from real people)
- Noise (filtered, available if needed)
Every email gets a baked-in Gmail link: `[Open in Gmail](https://mail.google.com/mail/u/?authuser=ACCOUNT#inbox/MESSAGE_ID)` — these are generated by code, never by the LLM, so they are always correct.
## Prerequisites
1. **GBrain installed and configured** (`gbrain doctor` passes)
2. **Node.js 18+** (for the collector script)
3. **Gmail access** via one of:
- ClawVisor (recommended: E2E encrypted credential gateway)
- Google OAuth credentials (direct API access)
- Hermes Gateway (built-in Gmail connector)
## Setup Flow
### Step 1: Validate Credential Gateway
Ask the user: "How do you access Gmail programmatically? Options:
1. ClawVisor (recommended, handles OAuth and encryption)
2. Google OAuth credentials (you manage tokens yourself)
3. Hermes Gateway (if you're using Hermes Agent)"
#### Option A: ClawVisor (recommended)
Tell the user:
"I need your ClawVisor URL and agent token.
1. Go to https://clawvisor.com
2. Create an agent (or use existing)
3. Activate the Gmail service
4. Create a standing task with purpose: 'Full executive assistant email management
including inbox triage, searching by any criteria, reading emails, tracking threads'
IMPORTANT: Be EXPANSIVE in the task purpose. Narrow purposes like 'email triage'
will cause legitimate requests to fail verification.
5. Copy the gateway URL and agent token"
Validate:
```bash
curl -sf "$CLAWVISOR_URL/health" && echo "PASS: ClawVisor reachable" || echo "FAIL"
```
**STOP until ClawVisor validates.**
#### Option B: Google OAuth2 directly
Tell the user:
"I need Google OAuth2 credentials for Gmail access. Here's how:
1. Go to https://console.cloud.google.com/apis/credentials
(create a Google Cloud project if you don't have one)
2. Click **'+ CREATE CREDENTIALS'** > **'OAuth client ID'**
3. If prompted, configure the OAuth consent screen:
- User type: **External** (or Internal for Google Workspace)
- App name: 'GBrain Email' (anything works)
- Scopes: add **'Gmail API .../auth/gmail.readonly'**
- Test users: add your own email address
4. Create the OAuth client ID:
- Application type: **Desktop app**
- Name: 'GBrain'
5. Copy the **Client ID** and **Client Secret**
6. Also enable the Gmail API:
Go to https://console.cloud.google.com/apis/library/gmail.googleapis.com
Click **'Enable'**"
Validate:
```bash
[ -n "$GOOGLE_CLIENT_ID" ] && [ -n "$GOOGLE_CLIENT_SECRET" ] \
&& echo "PASS: Google OAuth credentials set" \
|| echo "FAIL: Missing GOOGLE_CLIENT_ID or GOOGLE_CLIENT_SECRET"
```
Then run the OAuth flow to get tokens:
```bash
# The collector script handles the OAuth flow:
# 1. Opens browser to Google consent URL with gmail.readonly scope
# 2. User grants access
# 3. Script receives auth code, exchanges for access + refresh token
# 4. Stores tokens in ~/.gbrain/google-tokens.json
# 5. Auto-refreshes on expiry
```
**STOP until OAuth flow completes and tokens are stored.**
### Step 2: Set Up the Email Collector
Create the collector directory and script:
```bash
mkdir -p email-collector/data/{messages,digests}
cd email-collector
npm init -y
```
The collector script needs these capabilities:
1. **collect** — pull emails from Gmail via credential gateway, deduplicate by message ID, store as JSON with Gmail links baked in
2. **digest** — generate a markdown digest from collected emails, grouped by: signatures pending, messages to triage, noise
3. **state tracking** — remember last collection timestamp and known message IDs to avoid re-processing
Key design rules for the collector:
- Gmail links are generated by CODE, not by the LLM. Format: `[Open in Gmail](https://mail.google.com/mail/u/?authuser=ACCOUNT#inbox/MESSAGE_ID)`
- Noise filtering is deterministic: noreply, notifications, calendar invites
- Signature detection uses known patterns: DocuSign envelope, Dropbox Sign, HelloSign, PandaDoc
- All state persisted to `data/state.json` (last collect timestamp, known message IDs)
- Output is structured JSON (machine-readable) AND markdown digest (agent-readable)
### Step 3: Run First Collection
```bash
node email-collector.mjs collect
node email-collector.mjs digest
```
Verify: `ls data/digests/` should show today's digest file.
Read the digest. Confirm it contains real emails with working Gmail links.
### Step 4: Enrich Brain Pages
This is YOUR job (the agent). Read the digest. For each email:
1. **Detect entities**: who sent it? Who is mentioned? What companies?
2. **Check the brain**: `gbrain search "sender name"` — do we have a page?
3. **Update brain pages**: if sender has a brain page, append a timeline entry:
`- YYYY-MM-DD | Email from {sender}: {subject} [Source: Gmail, {date}]`
4. **Create new pages**: if sender is notable and has no page, create one
5. **Extract action items**: if the email requires a response or action, log it
6. **Sync**: run `gbrain sync --no-pull --no-embed` to index changes
### Step 5: Set Up Cron
The collector should run every 30 minutes:
```bash
*/30 * * * * cd /path/to/email-collector && node email-collector.mjs collect && node email-collector.mjs digest
```
The agent should read the digest on a schedule (e.g., 3x/day: 9 AM, 12 PM, 3 PM)
and run the enrichment flow from Step 4.
### Step 6: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/email-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok"}' >> ~/.gbrain/integrations/email-to-brain/heartbeat.jsonl
```
## Implementation Guide
These are production-tested patterns. Follow them exactly.
### Noise Filtering (Deterministic)
```
NOISE_SENDERS = ['noreply', 'no-reply', 'notifications@', 'calendar-notification',
'mailer-daemon', 'postmaster', 'donotreply']
is_noise(email):
from = email.from.toLowerCase()
return NOISE_SENDERS.some(p => from.includes(p)) // substring match
```
Simple substring matching, not regex. `notifications@slack.com` matches because
`notifications@` is in the pattern list. Order doesn't matter.
### Signature Detection
```
SIGNATURE_PATTERNS = [
/docusign/i, /dropbox sign/i, /hellosign/i, /pandadoc/i,
/please sign/i, /signature needed/i, /ready for your signature/i,
/everyone has signed/i, /you just signed/i
]
is_signature(email):
subject = email.subject || ''
from = email.from || ''
return SIGNATURE_PATTERNS.some(p => p.test(subject) || p.test(from))
```
Test BOTH subject AND from. Signature requests come from services that have
"docusign" in the sender address, not just the subject.
### Gmail Link Generation (CRITICAL)
```
gmail_link(messageId, authuser):
return `https://mail.google.com/mail/u/?authuser=${authuser}#inbox/${messageId}`
```
The `authuser` parameter is CRITICAL. Without it, the link opens in the default
Gmail account, not the right one. Each email record stores its account separately.
Generate these in CODE, never by the LLM. Links must be 100% reliable.
### Deduplication
```
collect():
state = load_state()
since = state.lastCollect ? `newer_than:${hours_since}h` : 'newer_than:1d'
for account in accounts:
inbox = gmail.list(query=since, max=50)
for msg in inbox:
if msg.id in state.knownMessageIds: continue // already seen
record = build_record(msg)
state.knownMessageIds[msg.id] = record
// ALSO pull sent mail to detect replies
sent = gmail.list(query=`from:${account.email} ${since}`, max=30)
for msg in sent:
state.knownMessageIds[msg.id] = {is_sent: true}
```
**Why sent mail matters:** Without it, the digest shows "awaiting response" on
threads you already replied to. Sent mail acts as a negative filter.
### What the Agent Should Test After Setup
1. **Noise filtering:** Send a test email from `noreply@test.com`. Run collect.
Verify it appears in noise section, not triage section.
2. **Gmail links:** Click a link from the digest. Verify it opens the correct
account (not the default one).
3. **Deduplication:** Run collect twice in 1 minute. Verify no duplicate messages.
4. **Sent mail:** Reply to an email manually. Run collect. Verify the thread is
marked as replied-to in the digest.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| ClawVisor (free tier) | $0 |
| Gmail API | $0 (within free quota) |
| **Total** | **$0** |
## Troubleshooting
**No emails collected:**
- Check ClawVisor health: `curl $CLAWVISOR_URL/health`
- Check standing task is active and has Gmail service enabled
- Check task purpose is expansive enough (narrow purposes block requests)
**Gmail links don't work:**
- Verify the `authuser` parameter matches the account email
- Gmail links require being logged into the correct Google account
**Digest is empty but collection ran:**
- Check `data/messages/` for JSON files
- All emails might be filtered as noise — check noise filtering rules

380
recipes/meeting-sync.md Normal file
View File

@@ -0,0 +1,380 @@
---
id: meeting-sync
name: Meeting Sync
version: 0.7.0
description: Meeting transcripts from Circleback auto-import into brain pages with attendee detection and entity propagation.
category: sense
requires: []
secrets:
- name: CIRCLEBACK_TOKEN
description: Circleback API token for meeting data access
where: https://app.circleback.ai — Settings > API > generate token
health_checks:
- "curl -sf -H \"Authorization: Bearer $CIRCLEBACK_TOKEN\" \"https://app.circleback.ai/api/mcp\" -X POST -H \"Content-Type: application/json\" -d '{\"jsonrpc\":\"2.0\",\"method\":\"tools/list\",\"id\":1}' > /dev/null && echo 'Circleback: OK' || echo 'Circleback: FAIL'"
setup_time: 15 min
cost_estimate: "$0-17/mo (Circleback free for 10 meetings/mo, Pro $17/mo unlimited)"
---
# Meeting Sync: Transcripts That Become Brain Pages
Every meeting is automatically recorded, transcribed, and imported into your brain
with attendee detection, entity propagation, and action item extraction. You never
take notes again. The brain remembers what was said, who said it, and what needs
to happen next.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Follow these steps precisely.
**Why this is high-value:** Meeting transcripts are the richest signal source.
A 30-minute meeting mentions 5-10 people, 3-5 companies, and generates 2-3 action
items. Each one should propagate to the relevant brain pages. Without this recipe,
meetings are black holes. With it, every meeting compounds the brain.
**The flow:**
1. Circleback records and transcribes the meeting (automatic, no user action)
2. The sync script pulls completed meetings from Circleback API
3. Each meeting becomes a brain page at `brain/meetings/{YYYY-MM-DD}-{slug}.md`
4. YOU (the agent) propagate entities to people/company pages
**Do not skip steps. Verify after each step.**
## Architecture
```
Video Call (Zoom, Google Meet, Teams)
↓ Circleback bot joins automatically
Circleback (recording + transcription + AI summary)
↓ API (JSONRPC 2.0 over HTTP, SSE responses)
Meeting Sync Script (deterministic Node.js)
↓ Outputs:
└── brain/meetings/{YYYY-MM-DD}-{slug}.md
- Frontmatter: source_id, date, duration, attendees, location
- Transcript with speaker labels and timestamps
- Tags inferred from title
Agent reads meeting page
↓ Judgment calls:
├── Entity detection (people, companies, topics)
├── Propagate to attendee brain pages (timeline entries)
├── Action item extraction
└── Cross-reference with calendar data
```
## Opinionated Defaults
**Meeting page format:**
```markdown
---
type: meeting
source_id: cb_abc123
source_type: circleback
title: Weekly Team Sync
date: 2026-04-10
duration: 32 min
attendees: [Alice Chen, Bob Park, Carol Wu]
location: Google Meet
tags: [team, weekly, sync]
---
## Key Points
- Discussed Q2 roadmap priorities
- Alice is blocked on the API migration
- Bob's prototype is ready for review
## Action Items
- [ ] Alice: unblock API migration by Friday
- [ ] Bob: share prototype link in Slack
- [ ] Carol: schedule design review for next week
---
## Transcript
**Alice Chen** (00:00): Let's start with the roadmap update...
**Bob Park** (02:15): The prototype is basically done...
**Carol Wu** (05:30): I have some design feedback on the new flow...
```
**Attendee filtering:**
- Skip calendar resources (e.g., "YC-SF Conference Room")
- Skip group addresses (e.g., "team@company.com")
- Extract display names, not email addresses
**Idempotent by source_id:** If a meeting with the same `source_id` already exists
in the brain, skip it. No duplicates.
## Prerequisites
1. **GBrain installed and configured** (`gbrain doctor` passes)
2. **Node.js 18+** (for the sync script)
3. **Circleback account** (https://circleback.ai) with meetings recorded
## Setup Flow
### Step 1: Get Circleback API Token
Tell the user:
"I need your Circleback API token. Here's where to find it:
1. Go to https://app.circleback.ai
2. Click your profile icon (top right) > Settings
3. Go to the API section
4. Generate a new API token (or copy existing)
5. Paste it to me
Note: Circleback's free tier records up to 10 meetings/month. Pro ($17/mo)
is unlimited. You need at least one recorded meeting for the sync to work."
Validate immediately:
```bash
curl -sf -H "Authorization: Bearer $CIRCLEBACK_TOKEN" \
"https://app.circleback.ai/api/mcp" \
-X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"tools/list","id":1}' \
| grep -q '"result"' \
&& echo "PASS: Circleback API connected" \
|| echo "FAIL: Circleback token invalid"
```
**If validation fails:** "That didn't work. Common issues: (1) make sure you copied
the full token, (2) tokens are long hex strings, (3) check that your Circleback
account is active."
**STOP until Circleback validates.**
### Step 2: Set Up the Meeting Sync Script
```bash
mkdir -p meeting-sync
cd meeting-sync
npm init -y
```
The sync script needs these capabilities:
1. **List meetings** — call Circleback API `list_meetings` with date range
(SSE response format, parse streaming events)
2. **Extract meeting data** — title, attendees, transcript, duration, date
3. **Slugify title** — "Weekly Team Sync" → `weekly-team-sync`
4. **Check for existing** — skip if `brain/meetings/{date}-{slug}.md` exists
5. **Format as markdown** — frontmatter + key points + action items + transcript
6. **Filter attendees** — remove calendar resources, groups, extract display names
7. **Infer tags** — from title keywords (e.g., "board" → board, "1:1" → 1-on-1)
### Step 3: Run First Sync
```bash
node meeting-sync.mjs --days 7
```
This syncs the last 7 days of meetings. For a full backfill:
```bash
node meeting-sync.mjs --start 2026-01-01 --end $(date +%Y-%m-%d)
```
Verify:
```bash
ls brain/meetings/ | head -10
```
Should show files like `2026-04-10-weekly-team-sync.md`.
Tell the user: "Found and synced N meetings. Here are the most recent: [list 3]."
### Step 4: Import to GBrain
```bash
gbrain import brain/meetings/ --no-embed
gbrain embed --stale
```
Verify:
```bash
gbrain search "meeting" --limit 3
```
### Step 5: Propagate to Entity Pages
This is YOUR job (the agent). For each meeting:
1. **Read the meeting page** — understand who attended and what was discussed
2. **For each attendee**, check brain: `gbrain search "attendee name"`
- If page exists: append timeline entry:
`- YYYY-MM-DD | Meeting: {title}. Discussed: {key points relevant to this person} [Source: Circleback]`
- If no page and person is notable: create a brain page
3. **For each company mentioned**: update company page timeline
4. **Action items**: if the meeting has action items, ensure they're tracked
5. **Cross-reference with calendar**: link meeting page to the calendar event
6. **Sync**: `gbrain sync --no-pull --no-embed`
### Step 6: Set Up Cron
Sync 3x daily on weekdays:
```bash
# 10 AM, 4 PM, 9 PM PT on weekdays
0 10,16,21 * * 1-5 cd /path/to/meeting-sync && node meeting-sync.mjs >> /tmp/meeting-sync.log 2>&1
```
Default (no flags): syncs yesterday and today.
### Step 7: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/meeting-sync
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok"}' >> ~/.gbrain/integrations/meeting-sync/heartbeat.jsonl
```
Tell the user: "Meeting sync is set up. Every meeting recorded by Circleback
automatically becomes a searchable brain page. Attendee pages get updated with
meeting history. Action items are extracted. Sync runs 3x daily on weekdays."
## Implementation Guide
These are production-tested patterns from syncing 280+ meeting transcripts.
### SSE Response Parsing
Circleback returns JSONRPC 2.0 over SSE (Server-Sent Events):
```
call_circleback(tool_name, args):
body = {jsonrpc: '2.0', id: next_id(), method: 'tools/call',
params: {name: tool_name, arguments: args}}
res = POST CIRCLEBACK_ENDPOINT, body,
headers: {Authorization: Bearer TOKEN, Accept: 'application/json, text/event-stream'}
text = res.text()
for line in text.split('\n'):
if line.startsWith('data: '):
json = JSON.parse(line[6:]) // strip "data: "
if json.result?.content?.[0]?.text:
return JSON.parse(json.result.content[0].text) // double-parse
if json.error:
throw json.error
```
**Non-obvious:** The response is JSON inside SSE inside JSONRPC. You have to:
1. Strip `data: ` prefix
2. Parse the SSE line as JSON
3. Drill into `result.content[0].text`
4. Parse THAT as JSON again (it's a string containing JSON)
### Idempotency (Double-Check)
```
meeting_exists(source_id):
// Method 1: grep all meeting files for source_id
result = shell(f'grep -rl "source_id: {source_id}" {MEETINGS_DIR}/')
if result: return true
// Method 2: check filename (backup)
slug = slugify(meeting.name)
if file_exists(f'{MEETINGS_DIR}/{date}-{slug}.md'): return true
return false
```
**Why double-check:** grep catches source_id matches even if the filename changed.
File existence catches cases where grep fails (e.g., permission issues).
### Auto-Tagging from Meeting Name
```
auto_tag(meeting_name):
name = meeting_name.toLowerCase()
tags = []
if 'office hours' in name or ' oh ' in name: tags.push('oh')
if 'standup' in name or 'sync' in name: tags.push('sync')
if '1:1' in name or '1on1' in name: tags.push('1on1')
if 'board' in name: tags.push('board')
if 'policy' in name or 'civic' in name: tags.push('civic')
if not tags: tags.push('meeting')
return tags
```
### Meeting Page Structure
```
---
title: "Weekly Team Sync"
type: meeting
date: 2026-04-10
duration: 32 min
source: circleback
source_id: cb_abc123
attendees:
- {name: Alice Chen, email: alice@company.com}
- {name: Bob Park, email: bob@company.com}
tags: [sync]
---
# Weekly Team Sync
## Summary
[Circleback AI summary]
## Attendees
- Alice Chen
- Bob Park
## Action Items
- [ ] Alice: unblock API migration by Friday
---
## Transcript
**Alice Chen** (00:00): Let's start with the roadmap...
**Bob Park** (02:15): The prototype is basically done...
```
### Git Commit After Sync
```
if new_meetings_created > 0:
shell('git add -A', cwd=BRAIN_DIR)
msg = f'sync: {count} meeting(s) from Circleback ({start} to {end})'
shell(f'git commit -m "{msg}"', cwd=BRAIN_DIR)
shell('git push', cwd=BRAIN_DIR)
```
The sync script commits and pushes automatically. This triggers GBrain's
live sync to index the new pages.
### What the Agent Should Test After Setup
1. **SSE parsing:** Verify `SearchMeetings` returns parseable data (the double-JSON
parsing is the most common failure point).
2. **Idempotency:** Sync a meeting, add a note to the file manually, sync again.
Verify the meeting is skipped (not re-created or overwritten).
3. **Attendee filtering:** Sync a meeting that includes a conference room in attendees.
Verify the room doesn't appear in the attendee list.
4. **Auto-tagging:** Sync a meeting named "1:1 with Sarah". Verify tag is `1on1`.
5. **Transcript formatting:** Verify speaker names and timestamps are formatted
correctly (speaker bold, timestamp in parentheses).
6. **Git commit:** Sync 2+ meetings. Verify the git commit message includes the count.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| Circleback Free tier | $0 (10 meetings/mo) |
| Circleback Pro | $17/mo (unlimited) |
| **Recommended** | **$17/mo (Pro)** |
## Troubleshooting
**No meetings found:**
- Check that Circleback has recorded meetings (open the Circleback dashboard)
- The Circleback bot must join the meeting for recording to work
- Check the date range: `--days 30` to widen the search
**Transcript is empty:**
- Some meetings may not have transcripts (e.g., no audio, bot was removed)
- Check the Circleback dashboard for the specific meeting's status
**Duplicate meetings:**
- The sync script checks for existing files by source_id
- If duplicates appear, the idempotency check may be failing
- Delete duplicates manually and re-run sync

245
recipes/ngrok-tunnel.md Normal file
View File

@@ -0,0 +1,245 @@
---
id: ngrok-tunnel
name: Public Tunnel
version: 0.7.0
description: Fixed public URL for your brain (MCP server, voice agent, any service). One ngrok account, never changes.
category: infra
requires: []
secrets:
- name: NGROK_AUTHTOKEN
description: ngrok auth token (Hobby tier recommended for fixed domain)
where: https://dashboard.ngrok.com/get-started/your-authtoken — sign up, then copy your authtoken
health_checks:
- "pgrep -f 'ngrok.*http' > /dev/null && echo 'ngrok: running' || echo 'ngrok: NOT running'"
- "curl -sf http://localhost:4040/api/tunnels > /dev/null && echo 'ngrok API: OK' || echo 'ngrok API: FAIL'"
setup_time: 10 min
cost_estimate: "$8/mo for Hobby tier (fixed domain). Free tier works but URLs change on restart."
---
# Public Tunnel: Fixed URL for Your Brain
Your GBrain MCP server and voice agent need public URLs so Claude Desktop,
Perplexity, and Twilio can reach them. ngrok gives you a fixed domain that
never changes.
## IMPORTANT: Instructions for the Agent
**You are the installer.** This is foundational infrastructure. Other recipes
(voice-to-brain, remote MCP) depend on this. Set it up first.
**Why this matters:**
- Voice-to-brain needs a public URL for Twilio webhooks
- Remote MCP needs a public URL for Claude Desktop and Perplexity
- Free ngrok URLs change on every restart, breaking all integrations
- Hobby tier ($8/mo) gives a fixed domain. Set it once, never touch it again.
**Do not skip steps. Verify after each step.**
## Architecture
```
Local services (your machine)
├── GBrain MCP server (port 3000) gbrain serve
└── Voice agent (port 8765) node server.mjs
ngrok tunnel (fixed domain)
└── https://your-brain.ngrok.app
├── /mcp → Claude Desktop, Claude Code, Perplexity
└── /voice → Twilio webhooks
```
## Setup Flow
### Step 1: Create ngrok Account + Get Hobby Tier
Tell the user:
"I need you to create an ngrok account. I strongly recommend Hobby tier ($8/mo)
for a fixed domain that never changes. Without it, every restart breaks your
Twilio webhooks and Claude Desktop connection.
1. Go to https://dashboard.ngrok.com/signup (sign up)
2. Go to https://dashboard.ngrok.com/billing and upgrade to **Hobby** ($8/mo)
3. Go to https://dashboard.ngrok.com/get-started/your-authtoken
4. Copy your **Authtoken** and paste it to me"
Validate:
```bash
ngrok config add-authtoken $NGROK_AUTHTOKEN \
&& echo "PASS: ngrok configured" \
|| echo "FAIL: ngrok auth token rejected"
```
If ngrok is not installed:
- **Mac:** `brew install ngrok`
- **Linux:** `curl -sL https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz | tar xz -C /usr/local/bin`
**STOP until ngrok validates.**
### Step 2: Claim a Fixed Domain
Tell the user:
"1. Go to https://dashboard.ngrok.com/domains
2. Click **'+ New Domain'**
3. Choose a name (e.g., `your-brain.ngrok.app`)
4. Click **'Create'**
5. Tell me the domain name you chose"
If user stayed on free tier (no fixed domain), note that URLs will change on
restart and the watchdog will need to update Twilio. Recommend upgrading later.
### Step 3: Start the Tunnel
```bash
# With fixed domain (Hobby):
ngrok http 8765 --url your-brain.ngrok.app
# Without fixed domain (free):
ngrok http 8765
```
Verify:
```bash
curl -sf http://localhost:4040/api/tunnels \
&& echo "PASS: ngrok tunnel active" \
|| echo "FAIL: ngrok not running"
```
### Step 4: Set Up Watchdog
The tunnel must auto-restart if it dies. Create a watchdog:
```bash
#!/bin/bash
# ngrok-watchdog.sh — run via cron every 2 minutes
# Check if ngrok is running
if ! pgrep -f "ngrok.*http" > /dev/null 2>&1; then
echo "[watchdog] ngrok not running — starting..."
# Install if missing
if ! command -v ngrok > /dev/null 2>&1; then
echo "[watchdog] ngrok not installed"
exit 1
fi
# Start with fixed domain (if configured) or free
if [ -n "$NGROK_DOMAIN" ]; then
nohup ngrok http 8765 --url "$NGROK_DOMAIN" > /dev/null 2>&1 &
else
nohup ngrok http 8765 > /dev/null 2>&1 &
fi
sleep 5
# If no fixed domain, update Twilio webhook with new URL
if [ -z "$NGROK_DOMAIN" ] && [ -n "$TWILIO_ACCOUNT_SID" ]; then
NGROK_URL=$(curl -s http://localhost:4040/api/tunnels 2>/dev/null \
| grep -o '"public_url":"https://[^"]*' | grep -o 'https://.*')
if [ -n "$NGROK_URL" ] && [ -n "$TWILIO_NUMBER_SID" ]; then
curl -s -X POST -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
"https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/IncomingPhoneNumbers/$TWILIO_NUMBER_SID.json" \
-d "VoiceUrl=${NGROK_URL}/voice" > /dev/null
echo "[watchdog] Twilio updated: $NGROK_URL"
fi
fi
echo "[watchdog] ngrok started"
else
echo "[watchdog] ngrok running"
fi
```
Add to crontab:
```bash
*/2 * * * * NGROK_DOMAIN=your-brain.ngrok.app /path/to/ngrok-watchdog.sh >> /tmp/ngrok-watchdog.log 2>&1
```
### Step 5: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/ngrok-tunnel
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"domain":"NGROK_DOMAIN","tier":"hobby"}}' >> ~/.gbrain/integrations/ngrok-tunnel/heartbeat.jsonl
```
## Connecting AI Clients (after tunnel is running)
**Claude Code:**
```bash
claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp \
-H "Authorization: Bearer YOUR_GBRAIN_TOKEN"
```
**Claude Desktop:**
Go to Settings > Integrations > Add. Enter:
`https://your-brain.ngrok.app/mcp`
IMPORTANT: Claude Desktop does NOT support remote MCP via JSON config.
You MUST use Settings > Integrations in the GUI. This is the #1 setup failure.
**Perplexity Computer:**
Settings > Connectors > Add Remote MCP.
URL: `https://your-brain.ngrok.app/mcp`
## Implementation Guide
### The Watchdog Pattern (from production)
```
watchdog():
// Check: is ngrok running?
if not process_running("ngrok.*http"):
start_ngrok()
sleep(5)
// If no fixed domain, must update Twilio
if no_fixed_domain AND twilio_configured:
new_url = get_ngrok_url() // from localhost:4040/api/tunnels
update_twilio_webhook(new_url + "/voice")
// Check: is the service behind ngrok running?
if not curl_succeeds("http://localhost:PORT/health"):
restart_service()
```
### ngrok Inspect Dashboard
`http://localhost:4040` shows all requests flowing through the tunnel. Use this
to debug MCP connection issues (see request/response headers, latency, errors).
## Tricky Spots
1. **Claude Desktop requires GUI setup.** Adding remote MCP servers via
`claude_desktop_config.json` does NOT work. It silently fails with no error.
You MUST use Settings > Integrations.
2. **Free tier URLs are ephemeral.** They change on every ngrok restart. The
watchdog handles Twilio, but Claude Desktop and Perplexity must be manually
reconfigured. This is why Hobby ($8/mo) is worth it.
3. **One domain, multiple services.** Hobby gives 1 free domain. Route by path
(`/mcp`, `/voice`) on one domain, or pay $8/mo more for a second domain.
4. **The watchdog must run on startup.** If the machine reboots, ngrok won't
auto-start unless you have a watchdog cron or systemd service.
## How to Verify
1. Start tunnel. Visit `https://your-brain.ngrok.app` in a browser.
You should see a response (health check or default page).
2. From Claude Desktop, run `gbrain search "test"`. Results should come back.
3. Kill ngrok. Wait 2 minutes. Check the watchdog restarted it.
4. From a different device (phone), access the same URL. Verify it works.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| ngrok Free | $0 (ephemeral URLs, change on restart) |
| ngrok Hobby | $8/mo (1 fixed domain, enough for MCP + voice) |
| ngrok Pro | $20/mo (2+ domains, IP restrictions) |
| **Recommended** | **$8/mo (Hobby)** |
---
*Part of the [GBrain Skillpack](../docs/GBRAIN_SKILLPACK.md). See also: [Voice-to-Brain](twilio-voice-brain.md), [Remote MCP Deployment](../docs/mcp/DEPLOY.md)*

View File

@@ -0,0 +1,466 @@
---
id: twilio-voice-brain
name: Voice-to-Brain
version: 0.7.0
description: Phone calls create brain pages via Twilio + OpenAI Realtime + GBrain MCP. Callers talk, brain pages appear.
category: sense
requires: [ngrok-tunnel]
secrets:
- name: TWILIO_ACCOUNT_SID
description: Twilio account SID (starts with AC)
where: https://www.twilio.com/console — visible on the main dashboard after login
- name: TWILIO_AUTH_TOKEN
description: Twilio auth token (click "Show" next to the SID on the dashboard)
where: https://www.twilio.com/console — click "Show" under Auth Token on the main dashboard
- name: OPENAI_API_KEY
description: OpenAI API key (needs Realtime API access enabled on your account)
where: https://platform.openai.com/api-keys — click "+ Create new secret key", copy immediately (you can't see it again)
health_checks:
- "curl -sf -u \"$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN\" \"https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json\" > /dev/null && echo 'Twilio: OK' || echo 'Twilio: FAIL'"
- "curl -sf -H \"Authorization: Bearer $OPENAI_API_KEY\" https://api.openai.com/v1/models > /dev/null && echo 'OpenAI: OK' || echo 'OpenAI: FAIL'"
setup_time: 30 min
cost_estimate: "$15-25/mo (Twilio number $1-2 + voice $0.01/min, OpenAI Realtime $0.06/min input + $0.24/min output)"
---
# Voice-to-Brain: Phone Calls That Create Brain Pages
Call a phone number. Talk. A structured brain page appears with entity detection,
cross-references, and a summary posted to your messaging app.
## IMPORTANT: Instructions for the Agent
**You are the installer.** This recipe is written for YOU (the AI agent) to execute
on behalf of the user. Follow these instructions precisely.
**Why sequential execution matters:** Each step depends on the previous one:
- Step 1 validates prerequisites. If GBrain isn't configured, nothing else works.
- Step 2 collects credentials. If a credential is wrong, Steps 5-7 will silently fail.
- Step 3 creates the ngrok tunnel. Step 5 needs the ngrok URL for the Twilio webhook.
- Step 5 configures Twilio. Step 7 (smoke test) needs Twilio configured to reach your server.
**Do not skip steps. Do not reorder steps. Do not batch multiple steps.**
**Stop points (MUST pause and verify before continuing):**
- After Step 1: all prerequisites pass? If not, fix before proceeding.
- After each credential in Step 2: validation passes? If not, help the user fix it.
- After Step 6: health check passes? If not, debug before smoke test.
- After Step 7: brain page created? If not, troubleshoot before declaring success.
**When something fails:** Tell the user EXACTLY what failed, what it means, and what
to try. Never say "something went wrong." Say "Twilio returned a 401, which means the
auth token is incorrect. Let's re-enter it."
## Architecture
```
Caller (phone)
↓ Twilio (WebSocket, g711_ulaw audio — no transcoding)
Voice Server (Node.js, your machine or cloud)
↓↑ OpenAI Realtime API (STT + LLM + TTS in one pipeline)
↓ Function calls during conversation
GBrain MCP (semantic search, page reads, page writes)
↓ Post-call
Brain page created (meetings/YYYY-MM-DD-call-{caller}.md)
Summary posted to messaging app (Telegram/Slack/Discord)
```
## Opinionated Defaults
These are production-tested defaults from a real deployment. Customize after setup.
**Caller routing (prompt-based, enforced server-side):**
- Owner: OTP challenge via secure channel, then full access (read + write + gateway)
- Trusted contacts: callback verification, scoped write access
- Known contacts (brain score >= 4): warm greeting by name, offer to transfer
- Unknown callers: screen, ask name + reason, take message
**Security:**
- Twilio signature validation on `/voice` endpoint (X-Twilio-Signature header)
- Unauthenticated callers never see write tools
- Caller ID is NOT trusted for auth (OTP or callback required)
---
## Setup Flow
### Step 1: Check Prerequisites
**STOP if any check fails. Fix before proceeding.**
Run these checks and report results to the user:
```bash
# 1. Verify GBrain is configured
gbrain doctor --json
```
If this fails: "GBrain isn't set up yet. Let's run `gbrain init --supabase` first."
```bash
# 2. Verify Node.js 18+
node --version
```
If missing or < 18: "Node.js 18+ is required. Install it: https://nodejs.org/en/download"
```bash
# 3. Check if ngrok is installed
which ngrok
```
If missing:
- **Mac:** "Run `brew install ngrok` in your terminal."
- **Linux:** "Run `snap install ngrok` or download from https://ngrok.com/download"
Tell the user: "All prerequisites checked. [N/3 passed]. [List any that failed and how to fix.]"
### Step 2: Collect and Validate Credentials
Ask for each credential ONE AT A TIME. Validate IMMEDIATELY. Do not proceed to
the next credential until the current one validates.
**Credential 1: Twilio Account SID + Auth Token**
Tell the user:
"I need your Twilio Account SID and Auth Token. Here's exactly where to find them:
1. Go to https://www.twilio.com/console (sign up free if you don't have an account)
2. After logging in, you'll see your **Account SID** right on the main dashboard
(it starts with 'AC' followed by 32 characters)
3. Below it you'll see **Auth Token** — click **'Show'** to reveal it
4. Copy both values and paste them to me"
After the user provides them, validate immediately:
```bash
curl -s -u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN" \
"https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID.json" \
| grep -q '"status"' \
&& echo "PASS: Twilio credentials valid" \
|| echo "FAIL: Twilio credentials invalid — double-check the SID starts with AC and the auth token is correct"
```
**If validation fails:** "That didn't work. Common issues: (1) the SID should start
with 'AC', (2) make sure you clicked 'Show' to reveal the auth token and copied the
full value, (3) if you just created the account, wait 30 seconds and try again."
**STOP HERE until Twilio validates.**
**Credential 2: OpenAI API Key**
Tell the user:
"I need your OpenAI API key. Here's exactly where to get one:
1. Go to https://platform.openai.com/api-keys
2. Click **'+ Create new secret key'** (top right)
3. Name it something like 'gbrain-voice'
4. Click **'Create secret key'**
5. **Copy the key immediately** — you won't be able to see it again after closing the dialog
6. Paste it to me
Note: your OpenAI account needs Realtime API access. Most accounts have it by default."
After the user provides it, validate immediately:
```bash
curl -sf -H "Authorization: Bearer $OPENAI_API_KEY" \
https://api.openai.com/v1/models > /dev/null \
&& echo "PASS: OpenAI key valid" \
|| echo "FAIL: OpenAI key invalid — make sure you copied the full key (starts with sk-)"
```
**If validation fails:** "That didn't work. Common issues: (1) the key starts with
'sk-', (2) make sure you copied the entire key (it's long), (3) if you just created
it, it's active immediately — no delay needed."
**STOP HERE until OpenAI validates.**
**Credential 3: ngrok Account (Hobby tier recommended)**
Tell the user:
"I need your ngrok auth token. **I strongly recommend the Hobby tier ($8/mo)**
because it gives you a fixed domain that never changes. With the free tier,
your URL changes every time ngrok restarts, breaking Twilio and Claude Desktop.
1. Go to https://dashboard.ngrok.com/signup (sign up)
2. **Recommended:** Go to https://dashboard.ngrok.com/billing and upgrade to
**Hobby** ($8/mo). This gives you a fixed domain.
3. If you upgraded: go to https://dashboard.ngrok.com/domains and click
**'+ New Domain'**. Choose a name (e.g., `your-brain-voice.ngrok.app`).
4. Go to https://dashboard.ngrok.com/get-started/your-authtoken
5. Copy your **Authtoken** and paste it to me
6. Also tell me your fixed domain name (if you created one)"
```bash
ngrok config add-authtoken $NGROK_TOKEN \
&& echo "PASS: ngrok configured" \
|| echo "FAIL: ngrok auth token rejected"
```
If user has a fixed domain, use `--url` flag (Step 3 below).
If user stayed on free tier, URLs will change on restart (the watchdog handles this).
**Credential 4: Messaging Platform (for call summaries)**
Ask the user: "Where should I send call summaries? Options: Telegram, Slack, or Discord."
Based on their choice:
- **Telegram:** "Create a bot via @BotFather on Telegram, copy the bot token, and
tell me which chat/group to send summaries to."
Validate: `curl -sf "https://api.telegram.org/bot$TOKEN/getMe" | grep -q '"ok":true'`
- **Slack:** "Create an Incoming Webhook at https://api.slack.com/apps → your app →
Incoming Webhooks → Add New. Copy the webhook URL."
Validate: `curl -sf -X POST -d '{"text":"GBrain voice test"}' $WEBHOOK_URL`
- **Discord:** "Go to your server → channel settings → Integrations → Webhooks →
New Webhook. Copy the webhook URL."
Validate: `curl -sf -X POST -H "Content-Type: application/json" -d '{"content":"GBrain voice test"}' $WEBHOOK_URL`
Tell the user: "All credentials validated. Moving to server setup."
### Step 3: Start ngrok Tunnel
```bash
# With fixed domain (Hobby tier — recommended):
ngrok http 8765 --url your-brain-voice.ngrok.app
# Without fixed domain (free tier — URL changes on restart):
ngrok http 8765
```
If using a fixed domain, the URL is always `https://your-brain-voice.ngrok.app`.
If using free tier, copy the URL from the ngrok output (changes every restart).
Note: ngrok runs in the foreground. Run it in a background process or new terminal tab.
The same ngrok account can also serve your GBrain MCP server (see
[ngrok Setup](docs/mcp/NGROK_SETUP.md) for the full multi-service pattern).
### Step 4: Create Voice Server
Create the voice server directory and install dependencies:
```bash
mkdir -p voice-agent && cd voice-agent
npm init -y
npm install ws express
```
The voice server needs these components in `server.mjs`:
1. **HTTP server** on port 8765 with:
- `POST /voice` — returns TwiML that opens a WebSocket media stream to `/ws`
- `GET /health` — returns `{ ok: true }`
- Twilio signature validation (`X-Twilio-Signature` header) on `/voice`
2. **WebSocket handler** at `/ws` that:
- Accepts Twilio media stream (g711_ulaw audio)
- Opens a second WebSocket to `wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview`
- Bridges audio bidirectionally (no transcoding — both sides use g711_ulaw)
- Handles `response.function_call_arguments.done` events from OpenAI (tool execution)
- Sends tool results back via `conversation.item.create` with type `function_call_output`
3. **System prompt builder** that takes caller phone number and returns:
- Appropriate greeting based on caller routing rules
- Available tools (read-only for unauthenticated, full for authenticated)
- Instructions: "You are a voice assistant. Search the brain before answering
questions. Take messages from unknown callers. Never hang up first."
4. **Tool executor** that:
- Spawns GBrain MCP client (`gbrain serve` as stdio child process)
- Routes function calls: `search_brain``gbrain query`, `lookup_person``gbrain search` + `gbrain get`
- Gates write tools behind authentication
5. **Post-call handler** that:
- Saves transcript to `brain/meetings/YYYY-MM-DD-call-{caller}.md`
- Posts summary to the user's messaging platform
- Runs `gbrain sync --no-pull --no-embed` to index the new page
6. **WebRTC endpoint** (optional, for browser-based calling):
- `POST /session` — accepts SDP offer, forwards to OpenAI Realtime `/v1/realtime/calls` as multipart form-data, returns SDP answer
- `GET /call` — serves a web client HTML page with:
- WebRTC connection to OpenAI Realtime API
- RNNoise WASM noise suppression (AudioWorklet)
- Push-to-talk AND auto-VAD mode switching
- Pipeline: Microphone → RNNoise denoise → MediaStream → WebRTC → OpenAI
- `POST /tool` — receives tool calls from the WebRTC data channel, executes them, returns results
- This lets users call the voice agent from a browser tab instead of a phone
**WebRTC session creation pseudocode:**
```
POST /session:
sdp = request.body // caller's SDP offer
form = new FormData()
form.append('sdp', sdp)
form.append('session', JSON.stringify({
type: 'realtime',
model: 'gpt-4o-realtime-preview',
audio: {output: {voice: VOICE}},
instructions: buildPrompt(null)
}))
response = POST 'https://api.openai.com/v1/realtime/calls'
Authorization: Bearer OPENAI_API_KEY
body: form
return response.text() // SDP answer
```
**Important WebRTC gotchas:**
- `voice` goes under `audio.output.voice`, not top-level
- Do NOT send `turn_detection` in session config (not accepted by `/v1/realtime/calls`)
- Do NOT send `session.update` on connect (server already configured it)
- Trigger greeting via data channel after WebRTC connects
**Reference implementation:** The architecture above and the OpenAI Realtime API
docs (https://platform.openai.com/docs/guides/realtime) provide the building blocks.
### Step 5: Configure Twilio Phone Number
Tell the user:
"Now I need to set up your Twilio phone number. Here's what to do:
1. Go to https://www.twilio.com/console/phone-numbers/search
2. Search for a number (pick your area code or any available number)
3. Click **'Buy'** next to the number you want (costs $1-2/month)
4. After purchase, go to https://www.twilio.com/console/phone-numbers/incoming
5. Click on your new number
6. Scroll to **'Voice Configuration'**
7. Under **'A call comes in'**, select **'Webhook'**
8. Enter: `https://YOUR-NGROK-URL.ngrok-free.app/voice`
9. Method: **HTTP POST**
10. Click **'Save configuration'**
11. Tell me the phone number you purchased"
Or if the user prefers CLI:
```bash
# Buy a number (US local)
twilio phone-numbers:buy:local --area-code 415
# Configure webhook
twilio phone-numbers:update PHONE_SID \
--voice-url https://YOUR-NGROK-URL.ngrok-free.app/voice \
--voice-method POST
```
### Step 6: Start Voice Server and Verify
```bash
cd voice-agent && node server.mjs
```
**STOP and verify:**
```bash
curl -sf http://localhost:8765/health && echo "Voice server: running" || echo "Voice server: NOT running"
```
If not running: check the server logs for errors. Common issues:
- Port 8765 already in use: `lsof -i :8765` to find what's using it
- Missing environment variables: make sure OPENAI_API_KEY is set
- Module not found: run `npm install` again
### Step 7: Smoke Test (Outbound Call)
**This is the magical moment.** The agent calls the USER to prove the system works.
Tell the user: "Your phone is about to ring. Pick up and talk for about 30 seconds.
Say something like 'Hey, I'm testing my new voice-to-brain system. Remind me to
check the quarterly numbers tomorrow.' When you're done, hang up."
```bash
curl -X POST "https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json" \
--data-urlencode "To=USER_PHONE_NUMBER" \
--data-urlencode "From=TWILIO_PHONE_NUMBER" \
--data-urlencode "Url=https://YOUR-NGROK-URL.ngrok-free.app/voice" \
-u "$TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN"
```
**After the call ends, verify ALL of these:**
1. Messaging notification arrived with call summary
2. Brain page exists:
```bash
gbrain search "call" --limit 1
```
3. The brain page has: transcript, entity mentions, action items
**If the smoke test fails:**
- No ring: check Twilio console for error logs at https://www.twilio.com/console/debugger
- Ring but no voice: check ngrok tunnel is up, check OpenAI key is valid
- Voice works but no brain page: check post-call handler logs, run `gbrain sync` manually
- Brain page but no messaging: check messaging bot token is valid
**STOP HERE until the smoke test passes. Do not declare success until the user
confirms they received the messaging notification AND the brain page exists.**
### Step 8: Set Up Inbound Calling
Tell the user: "The smoke test passed — voice-to-brain is live! Your number is
[TWILIO_NUMBER]. Now let's set up inbound calling."
1. Twilio webhook is already configured from Step 5
2. Ask: "Do you want calls to your existing phone to forward to this number
after a few rings? That way you answer if you can, and the voice agent
picks up if you don't."
3. Configure caller routing rules in the system prompt
4. Add the user's phone number as the "owner" number for full access
### Step 9: Watchdog (Auto-restart)
```bash
# Cron watchdog (every 2 minutes) — add to crontab
*/2 * * * * curl -sf http://localhost:8765/health > /dev/null || (cd /path/to/voice-agent && node server.mjs >> /tmp/voice-agent.log 2>&1 &)
```
If using ngrok, also set up URL monitoring (free ngrok URLs change on restart):
```bash
# Check if ngrok URL changed, update Twilio if so
NGROK_URL=$(curl -s http://localhost:4040/api/tunnels 2>/dev/null | grep -o '"public_url":"https://[^"]*' | grep -o 'https://.*')
if [ -n "$NGROK_URL" ]; then
twilio phone-numbers:update PHONE_SID --voice-url "$NGROK_URL/voice"
fi
```
### Step 10: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/twilio-voice-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"phone":"TWILIO_NUMBER","deployment":"local+ngrok"}}' >> ~/.gbrain/integrations/twilio-voice-brain/heartbeat.jsonl
```
Tell the user: "Voice-to-brain is fully set up. Your number is [NUMBER]. Here's
what happens now: anyone who calls gets screened by the voice agent. Known contacts
get a warm greeting. Unknown callers leave a message. Every call creates a brain
page with the full transcript, and you get a summary on [their messaging platform].
The watchdog restarts the server if it crashes."
## Cost Estimate
| Component | Monthly Cost | Source |
|-----------|-------------|--------|
| Twilio phone number | $1-2/mo | [Twilio pricing](https://www.twilio.com/en-us/voice/pricing) |
| Twilio voice minutes (100 min) | $1-2/mo | $0.0085-0.015/min depending on direction |
| OpenAI Realtime input (100 min) | $6/mo | [$0.06/min](https://openai.com/api/pricing/) |
| OpenAI Realtime output (50 min) | $12/mo | [$0.24/min](https://openai.com/api/pricing/) |
| ngrok (free tier) | $0 | Static domain: $8/mo |
| **Total estimate** | **$20-22/mo** | For ~100 min of calls |
## Troubleshooting
**Calls don't connect:**
- Check ngrok: `curl http://localhost:4040/api/tunnels` — if empty, ngrok isn't running
- Check voice server: `curl http://localhost:8765/health` — should return `{"ok":true}`
- Check Twilio debugger: https://www.twilio.com/console/debugger — shows webhook errors
- Check webhook URL: go to https://www.twilio.com/console/phone-numbers/incoming, click your number, verify the webhook URL matches your ngrok URL
**Voice agent doesn't respond:**
- Check OpenAI key: the validation command from Step 2 should still pass
- Check server logs for WebSocket errors (look for "connection refused" or "401")
- Verify Realtime API access: not all OpenAI accounts have it. Check https://platform.openai.com/docs/guides/realtime
**Brain pages not created after call:**
- Run `gbrain doctor` — if it fails, the database connection is broken
- Check if the post-call handler ran (look in server logs for "transcript saved")
- Run `gbrain sync` manually to force indexing
- Check file permissions on the brain repo directory
**ngrok URL keeps changing:**
- Free ngrok URLs change every time ngrok restarts
- The watchdog (Step 9) handles this automatically
- For a permanent URL: upgrade to ngrok paid ($8/mo) for a static domain, or deploy to Fly.io/Railway instead

353
recipes/x-to-brain.md Normal file
View File

@@ -0,0 +1,353 @@
---
id: x-to-brain
name: X-to-Brain
version: 0.7.0
description: Twitter timeline, mentions, and keyword monitoring flow into brain pages. Tracks deletions and engagement velocity.
category: sense
requires: []
secrets:
- name: X_BEARER_TOKEN
description: X API v2 Bearer token (Basic tier minimum, $200/mo for full archive search)
where: https://developer.x.com/en/portal/dashboard — create a project + app, copy the Bearer Token from "Keys and tokens"
health_checks:
- "curl -sf -H \"Authorization: Bearer $X_BEARER_TOKEN\" \"https://api.x.com/2/users/me\" > /dev/null && echo 'X API: OK' || echo 'X API: FAIL'"
setup_time: 15 min
cost_estimate: "$0-200/mo (Free tier: 1 app, read-only. Basic: $200/mo for search + higher limits)"
---
# X-to-Brain: Twitter Monitoring That Updates Your Brain
Your timeline, mentions, and keyword searches flow into brain pages. The collector
tracks deletions, engagement velocity, and narrative patterns. You wake up knowing
what happened on X while you slept.
## IMPORTANT: Instructions for the Agent
**You are the installer.** Follow these steps precisely.
**The core pattern: code for data, LLMs for judgment.**
The X collector is deterministic code. It pulls tweets, detects deletions, tracks
engagement. It NEVER interprets content. YOU (the agent) read the collected data
and make judgment calls: who is important, what entities are mentioned, what
narratives are forming.
**Why sequential execution matters:**
- Step 1 validates the API key. Without it, nothing connects to X.
- Step 2 sets up the collector. Without it, you have no data.
- Step 3 runs the first collection. Without data, you can't enrich.
- Step 4 is YOUR job: read the collected tweets, update brain pages.
**Do not skip steps. Do not reorder. Verify after each step.**
## Architecture
```
X API v2 (Bearer token auth)
↓ Three collection streams:
├── Own timeline: GET /users/{id}/tweets
├── Mentions: GET /users/{id}/mentions
└── Keyword searches: GET /tweets/search/recent
X Collector (deterministic Node.js script)
↓ Outputs:
├── data/tweets/{own,mentions,searches}/{id}.json
├── data/deletions/{id}.json (detected via diff)
├── data/engagement/{id}.json (velocity snapshots)
└── data/state.json (pagination, rate limits)
Agent reads collected data
↓ Judgment calls:
├── Entity detection (people, companies mentioned)
├── Brain page updates (timeline entries)
├── Narrative pattern detection
└── Engagement spike alerts
```
## Opinionated Defaults
**Three collection streams:**
1. **Own timeline** — your tweets, for your own archive and engagement tracking
2. **Mentions** — who is talking about you, for relationship tracking
3. **Keyword searches** — topics you care about, for signal detection
**Deletion detection:**
- Compare tweet IDs from previous run vs current
- If an ID is missing AND the tweet is < 7 days old, call GET /tweets/{id}
- 404 = confirmed deleted. Save the original tweet + deletion timestamp.
- Alert on deletions from accounts you track.
**Engagement velocity:**
- Snapshot likes/retweets/replies for tracked tweets
- Alert if likes doubled AND previous count >= 50
- Alert if likes gained > 100 absolute since last check
- Only write snapshot if metrics actually changed (idempotent)
**Rate limit awareness:**
- Basic tier: 1500 req/15min for timeline, 450 for mentions, 60 for search
- Collector tracks rate limits in state.json
- Back off automatically when approaching limits
## Prerequisites
1. **GBrain installed and configured** (`gbrain doctor` passes)
2. **Node.js 18+** (for the collector script)
3. **X Developer account** with API access
## Setup Flow
### Step 1: Get X API Credentials
Tell the user:
"I need your X API Bearer token. Here's exactly where to get it:
1. Go to https://developer.x.com/en/portal/dashboard
2. If you don't have a developer account, click 'Sign up' (free tier available)
3. Create a new Project (name it anything, e.g., 'GBrain')
4. Inside the project, create a new App
5. Go to the app's 'Keys and tokens' tab
6. Under 'Bearer Token', click 'Generate' (or 'Regenerate')
7. Copy the Bearer Token and paste it to me
Note: Free tier gives read-only access with low limits. Basic tier ($200/mo)
gives search/recent endpoint and higher limits. Pro tier gets full archive search."
Validate immediately:
```bash
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
"https://api.x.com/2/users/me" \
&& echo "PASS: X API connected" \
|| echo "FAIL: X API token invalid"
```
**If validation fails:** "That didn't work. Common issues: (1) make sure you copied
the Bearer Token, not the API Key or API Secret, (2) Bearer Tokens are long strings
starting with 'AAA...', (3) if you just created the app, the token is valid immediately."
**STOP until X API validates.**
### Step 2: Get Your X User ID
```bash
# Look up the user's X user ID from their handle
curl -sf -H "Authorization: Bearer $X_BEARER_TOKEN" \
"https://api.x.com/2/users/by/username/USERNAME" | grep -o '"id":"[^"]*"'
```
Ask the user for their X handle (e.g., @yourhandle). Look up their user ID.
Save it — the collector needs the numeric ID, not the handle.
### Step 3: Configure the Collector
Create the collector directory:
```bash
mkdir -p x-collector/data/{tweets/{own,mentions,searches},deletions,engagement}
cd x-collector
```
The collector script needs these capabilities:
1. **collect** — pull tweets from three streams:
- Own timeline: `GET /2/users/{id}/tweets` with max_results=100
- Mentions: `GET /2/users/{id}/mentions` with max_results=100
- Keyword searches: configurable search terms via `GET /2/tweets/search/recent`
2. **Deletion detection** — compare previous run's tweet IDs vs current. For missing IDs, verify with individual tweet lookup. 404 = deleted.
3. **Engagement tracking** — snapshot metrics for tracked tweets. Only write if metrics changed.
4. **State management** — save pagination tokens, last run timestamp, rate limit state to `data/state.json`
5. **Atomic writes** — write to .tmp file, then rename (prevents corrupt data on crash)
Configure keyword searches based on what the user cares about:
```json
{
"searches": [
"\"your name\" -from:yourhandle",
"\"your company\" OR \"your product\"",
"topic you track"
]
}
```
### Step 4: Run First Collection
```bash
node x-collector.mjs collect
```
Verify: `ls data/tweets/own/` should contain tweet JSON files.
Show the user a sample: "Found N tweets from your timeline, M mentions, K search results."
### Step 5: Enrich Brain Pages
This is YOUR job (the agent). Read the collected tweets:
1. **Detect entities**: who tweeted? Who is mentioned? What companies/topics?
2. **Check the brain**: `gbrain search "person name"` — do we have a page?
3. **Update brain pages**: for each notable person or company mentioned:
`- YYYY-MM-DD | Tweeted about {topic} [Source: X, @handle, {date}]`
4. **Track narratives**: if someone tweets about the same topic 3+ times in a week, note the pattern in their compiled truth
5. **Flag deletions**: if a tracked account deleted a tweet, note it:
`- YYYY-MM-DD | Deleted tweet: "{content}" [Source: X deletion, detected {date}]`
6. **Sync**: `gbrain sync --no-pull --no-embed`
### Step 6: Set Up Cron
The collector should run every 30 minutes:
```bash
*/30 * * * * cd /path/to/x-collector && node x-collector.mjs collect >> /tmp/x-collector.log 2>&1
```
The agent should review collected data 2-3x daily and run enrichment.
### Step 7: Log Setup Completion
```bash
mkdir -p ~/.gbrain/integrations/x-to-brain
echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","event":"setup_complete","source_version":"0.7.0","status":"ok","details":{"user_id":"X_USER_ID"}}' >> ~/.gbrain/integrations/x-to-brain/heartbeat.jsonl
```
## Implementation Guide
These are production-tested patterns from a deployment tracking 19+ accounts.
### Deletion Detection Algorithm
```
detect_deletions(prevIds, currentIds):
for id in prevIds:
if id in currentIds: continue // still exists
stored = load_tweet(id)
if not stored: continue // never stored
// HEURISTIC 1: Only check tweets < 7 days old
age = now - stored.created_at
if age > 7_DAYS: continue // aged out of API window
// HEURISTIC 2: Skip if last seen > 48h ago
staleness = now - stored.last_updated
if staleness > 48_HOURS: continue // fell out of window, not deleted
// HEURISTIC 3: Already logged?
if deletion_file_exists(id): continue
// VERIFY via direct API call
res = GET /tweets/{id}
if res.status == 404 OR (res.ok AND no data):
save_deletion(id, original_tweet, detected_at)
alert(f"DELETION: {author} deleted: {preview}")
```
**Why the heuristics matter:** Without #2 (48h staleness check), you get false
positives on old tweets that just aged out of the API search window. Without #1
(7-day cap), you'd investigate thousands of old tweets on every run.
### Engagement Velocity Tracking
```
track_engagement(id, metrics):
snapshots = load_snapshots(id)
last = snapshots[-1] if snapshots else null
if last AND metrics_equal(last, metrics): return // no change
snapshots.append({timestamp: now, metrics})
if len(snapshots) > 100: snapshots = snapshots[-100:] // cap growth
// Alert conditions (OR logic):
if last:
old_likes = last.like_count
new_likes = metrics.like_count
// Condition 1: 2x on established tweets (>= 50 likes)
if old_likes >= 50 AND new_likes >= old_likes * 2:
alert(f"VELOCITY: {id} likes {old_likes} -> {new_likes}")
// Condition 2: Absolute jump > 100
elif (new_likes - old_likes) > 100:
alert(f"VELOCITY: {id} likes {old_likes} -> {new_likes}")
```
**Threshold design:** `50` minimum prevents noise from small tweets going 2→4.
The `100` absolute jump catches big spikes on tweets with any baseline.
### Atomic File Writes
```
atomic_write(path, obj):
tmp = path + '.tmp'
writeFileSync(tmp, JSON.stringify(obj, null, 2))
renameSync(tmp, path) // atomic on most filesystems
```
If the process dies mid-write, the `.tmp` file is left behind but the original
is untouched. Critical when you have thousands of per-tweet JSON files.
### Rate Limit Handling
```
rate_limits = {} // per endpoint
after_each_request(endpoint, headers):
rate_limits[endpoint] = {
remaining: headers['x-rate-limit-remaining'],
reset: headers['x-rate-limit-reset']
}
is_rate_limited(endpoint, min_remaining=2):
r = rate_limits[endpoint]
return r AND r.remaining <= min_remaining
```
Reserve 2 requests per endpoint so other streams still work. If mentions
hits the limit, own timeline and searches can still run.
### Stdout Contract
The collector prints structured lines the cron agent can parse:
```
RUN_START:{timestamp}
OWN_TWEETS:{total} ({new} new)
MENTIONS:{total} ({new} new)
DELETION_DETECTED:{id}:{author}:{preview}
VELOCITY_ALERT:{id}:likes:{old}->{new}:{minutes}min
RUN_COMPLETE:{timestamp}:tweets_stored={N}:deletions={N}:velocity_alerts={N}
```
### What the Agent Should Test After Setup
1. **Deletion detection:** Post a tweet, collect, delete it, collect again.
Verify deletion is detected on second run.
2. **Rate limit:** Run collect with very low remaining quota. Verify it stops
gracefully and reports which streams were skipped.
3. **Engagement:** Find a tweet with 45 likes. Mock it jumping to 90 (no alert,
< 50 threshold). Then 50→100 (alert: 2x). Then 30→150 (alert: >100 jump).
4. **Deduplication:** Collect, then like one of your own tweets, collect again.
Verify `_collected_at` is preserved (not overwritten).
5. **Atomic writes:** Kill the process mid-collection. Verify no corrupted JSON.
## Cost Estimate
| Component | Monthly Cost |
|-----------|-------------|
| X API Free tier | $0 (read-only, low limits) |
| X API Basic tier | $200/mo (search + higher limits) |
| X API Pro tier | $5,000/mo (full archive) |
| **Recommended** | **$0 (free) or $200 (basic)** |
Free tier works for personal monitoring. Basic tier needed for keyword search.
## Troubleshooting
**API returns 403:**
- Check your app has the right access level (Read or Read+Write)
- Free tier apps can only use basic endpoints
- Some endpoints require Basic or Pro tier
**Rate limited (429):**
- The collector respects rate limits automatically
- If hitting limits frequently, increase the cron interval to 60 minutes
- Check `data/state.json` for rate limit tracking
**No tweets collected:**
- Verify the user ID is correct (numeric, not handle)
- Check the Bearer Token is valid (Step 1 validation)
- Some accounts may have protected tweets (requires OAuth 2.0 user context)

View File

@@ -1,6 +1,6 @@
{ {
"name": "gbrain", "name": "gbrain",
"version": "0.4.0", "version": "0.7.0",
"description": "Personal knowledge brain with hybrid RAG search", "description": "Personal knowledge brain with hybrid RAG search",
"skills": [ "skills": [
{ {
@@ -46,5 +46,6 @@
"setup": { "setup": {
"skill": "setup", "skill": "setup",
"description": "Auto-provision Supabase and configure GBrain (< 2 min)" "description": "Auto-provision Supabase and configure GBrain (< 2 min)"
} },
"recipes_dir": "recipes/"
} }

View File

@@ -0,0 +1,51 @@
# v0.7.0 Migration: Integration Recipes
This migration introduces the GBrain integration recipe system. Existing users
should be informed about the new capabilities.
## What's New
GBrain now distributes **integration recipes** — markdown files that teach your
agent how to set up new data pipelines into your brain. Your agent reads the recipe
and walks you through setup, asking for API keys and configuring everything.
### New: Voice-to-Brain Integration
Phone calls create brain pages. Powered by Twilio + OpenAI Realtime + GBrain MCP.
Call a number, talk, and a structured memory appears in your brain with entity
detection, cross-references, and a summary posted to your messaging app.
### New: `gbrain integrations` Command
Discover, check status, and monitor your integrations:
```bash
gbrain integrations # dashboard: show all senses + reflexes
gbrain integrations list --json # list available recipes (agent-parseable)
gbrain integrations show <id> # show recipe details + setup instructions
gbrain integrations status <id> # check secrets + health
gbrain integrations doctor # run health checks for all integrations
gbrain integrations stats # signal statistics
```
### New: SKILLPACK Breakout
The SKILLPACK document has been broken into 17 individual guides at `docs/guides/`.
Each guide is individually linkable and searchable. The SKILLPACK index at
`docs/GBRAIN_SKILLPACK.md` still works (backward compatible).
## Agent Instructions
When presenting this update to the user, say something like:
"GBrain v0.7.0 adds integration recipes — your brain can now get new senses
automatically. The first one is Voice-to-Brain: phone calls create brain pages.
Want me to show you what's available? Run `gbrain integrations` to see the
dashboard."
If the user is interested in voice-to-brain:
1. Run `gbrain integrations show twilio-voice-brain` to see what's needed
2. Walk them through the setup (the recipe has step-by-step instructions)
3. The smoke test calls THEM (outbound) to prove it works
Do NOT auto-install integrations. Always ask first.

View File

@@ -19,7 +19,7 @@ for (const op of operations) {
} }
// CLI-only commands that bypass the operation layer // CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'check-update', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor']); const CLI_ONLY = new Set(['init', 'upgrade', 'check-update', 'integrations', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor']);
async function main() { async function main() {
const args = process.argv.slice(2); const args = process.argv.slice(2);
@@ -238,6 +238,11 @@ async function handleCliOnly(command: string, args: string[]) {
await runCheckUpdate(args); await runCheckUpdate(args);
return; return;
} }
if (command === 'integrations') {
const { runIntegrations } = await import('./commands/integrations.ts');
await runIntegrations(args);
return;
}
// All remaining CLI-only commands need a DB connection // All remaining CLI-only commands need a DB connection
const engine = await connectEngine(); const engine = await connectEngine();
@@ -332,6 +337,7 @@ SETUP
upgrade Self-update upgrade Self-update
check-update [--json] Check for new versions check-update [--json] Check for new versions
doctor [--json] Health check (pgvector, RLS, schema, embeddings) doctor [--json] Health check (pgvector, RLS, schema, embeddings)
integrations [subcommand] Manage integration recipes (senses + reflexes)
PAGES PAGES
get <slug> Read a page get <slug> Read a page

View File

@@ -0,0 +1,686 @@
/**
* gbrain integrations — standalone CLI command for recipe discovery and health.
*
* NOT an operation (no database connection needed).
* Reads embedded recipe files and heartbeat JSONL from ~/.gbrain/integrations/.
*
* ARCHITECTURE:
* recipes/*.md (embedded at build time)
* │
* ├── list → parse frontmatter, check env vars, show status
* ├── show → display recipe details + body
* ├── status → check secrets + heartbeat
* ├── doctor → run health_checks
* ├── stats → aggregate heartbeat JSONL
* ├── test → validate recipe file
* └── (bare) → dashboard view
*
* ~/.gbrain/integrations/<id>/heartbeat.jsonl
* └── append-only, pruned to 30 days on read
*/
import matter from 'gray-matter';
import { readFileSync, existsSync, writeFileSync, mkdirSync, readdirSync } from 'fs';
import { join, basename } from 'path';
import { homedir } from 'os';
import { execSync } from 'child_process';
// --- Types ---
interface RecipeSecret {
name: string;
description: string;
where: string;
}
interface RecipeFrontmatter {
id: string;
name: string;
version: string;
description: string;
category: 'infra' | 'sense' | 'reflex';
requires: string[];
secrets: RecipeSecret[];
health_checks: string[];
setup_time: string;
cost_estimate?: string;
}
interface ParsedRecipe {
frontmatter: RecipeFrontmatter;
body: string;
filename: string;
}
interface HeartbeatEntry {
ts: string;
event: string;
source_version?: string;
status: string;
details?: Record<string, unknown>;
error?: string;
}
// --- Recipe Parsing ---
/**
* Parse a recipe markdown file. Uses gray-matter directly (NOT parseMarkdown,
* which splits on --- as timeline separator and would corrupt recipe bodies
* that use horizontal rules).
*/
export function parseRecipe(content: string, filename: string): ParsedRecipe | null {
try {
const { data, content: body } = matter(content);
if (!data.id) return null;
return {
frontmatter: {
id: data.id,
name: data.name || data.id,
version: data.version || '0.0.0',
description: data.description || '',
category: data.category || 'sense',
requires: data.requires || [],
secrets: data.secrets || [],
health_checks: data.health_checks || [],
setup_time: data.setup_time || 'unknown',
cost_estimate: data.cost_estimate,
},
body: body.trim(),
filename,
};
} catch {
return null;
}
}
// --- Embedded Recipes ---
// Recipes are loaded from the recipes/ directory at runtime.
// For compiled binaries, these should be embedded at build time.
// For source installs (bun run), they're read from disk.
function getRecipesDir(): string {
// Explicit override (for compiled binaries or custom installs)
if (process.env.GBRAIN_RECIPES_DIR && existsSync(process.env.GBRAIN_RECIPES_DIR)) {
return process.env.GBRAIN_RECIPES_DIR;
}
// Try relative to this file (source install via bun)
const sourceDir = join(import.meta.dir, '../../recipes');
if (existsSync(sourceDir)) return sourceDir;
// Try relative to CWD (development)
const cwdDir = join(process.cwd(), 'recipes');
if (existsSync(cwdDir)) return cwdDir;
// Try global install path (bun add -g)
const globalDir = join(homedir(), '.bun', 'install', 'global', 'node_modules', 'gbrain', 'recipes');
if (existsSync(globalDir)) return globalDir;
return '';
}
function loadAllRecipes(): ParsedRecipe[] {
const dir = getRecipesDir();
if (!dir || !existsSync(dir)) return [];
const files = readdirSync(dir).filter(f => f.endsWith('.md'));
const recipes: ParsedRecipe[] = [];
for (const file of files) {
try {
const content = readFileSync(join(dir, file), 'utf-8');
const recipe = parseRecipe(content, file);
if (recipe) {
recipes.push(recipe);
} else {
console.error(`Warning: skipping ${file} (invalid or missing 'id' in frontmatter)`);
}
} catch {
console.error(`Warning: skipping ${file} (unreadable)`);
}
}
return recipes;
}
function findRecipe(id: string): ParsedRecipe | null {
const recipes = loadAllRecipes();
const exact = recipes.find(r => r.frontmatter.id === id);
if (exact) return exact;
// Fuzzy: check if id is a substring match
const partial = recipes.filter(r =>
r.frontmatter.id.includes(id) || r.frontmatter.name.toLowerCase().includes(id.toLowerCase())
);
if (partial.length === 1) return partial[0];
if (partial.length > 1) {
console.error(`Recipe '${id}' not found. Did you mean one of these?`);
for (const r of partial) {
console.error(` ${r.frontmatter.id}${r.frontmatter.description}`);
}
return null;
}
console.error(`Recipe '${id}' not found.`);
const all = recipes.map(r => r.frontmatter.id);
if (all.length > 0) {
console.error(`Available recipes: ${all.join(', ')}`);
}
return null;
}
// --- Heartbeat ---
function heartbeatDir(id: string): string {
return join(homedir(), '.gbrain', 'integrations', id);
}
function heartbeatPath(id: string): string {
return join(heartbeatDir(id), 'heartbeat.jsonl');
}
function readHeartbeat(id: string): HeartbeatEntry[] {
const path = heartbeatPath(id);
if (!existsSync(path)) return [];
try {
const lines = readFileSync(path, 'utf-8').split('\n').filter(l => l.trim());
const entries: HeartbeatEntry[] = [];
const thirtyDaysAgo = Date.now() - 30 * 24 * 60 * 60 * 1000;
for (const line of lines) {
try {
const entry = JSON.parse(line) as HeartbeatEntry;
if (new Date(entry.ts).getTime() >= thirtyDaysAgo) {
entries.push(entry);
}
} catch {
// Skip malformed lines
}
}
// Prune old entries on read
if (entries.length < lines.length) {
try {
mkdirSync(heartbeatDir(id), { recursive: true });
writeFileSync(path, entries.map(e => JSON.stringify(e)).join('\n') + '\n');
} catch {
// Non-fatal: pruning failed
}
}
return entries;
} catch {
return [];
}
}
// --- Secret Checking ---
function checkSecrets(secrets: RecipeSecret[]): { set: string[]; missing: RecipeSecret[] } {
const set: string[] = [];
const missing: RecipeSecret[] = [];
for (const s of secrets) {
if (process.env[s.name]) {
set.push(s.name);
} else {
missing.push(s);
}
}
return { set, missing };
}
type IntegrationStatus = 'available' | 'configured' | 'active';
function getStatus(recipe: ParsedRecipe): IntegrationStatus {
const { set, missing } = checkSecrets(recipe.frontmatter.secrets);
// All required secrets must be set to be "configured"
if (missing.length > 0) return 'available';
const heartbeat = readHeartbeat(recipe.frontmatter.id);
const recentEvents = heartbeat.filter(e =>
Date.now() - new Date(e.ts).getTime() < 24 * 60 * 60 * 1000
);
if (recentEvents.length > 0) return 'active';
return 'configured';
}
// --- Dependency Resolution ---
function checkDependencies(recipe: ParsedRecipe, allRecipes: ParsedRecipe[]): string[] {
const warnings: string[] = [];
const visited = new Set<string>();
function check(id: string, chain: string[]): void {
if (visited.has(id)) return;
if (chain.includes(id)) {
warnings.push(`Circular dependency: ${chain.join(' -> ')} -> ${id}`);
return;
}
visited.add(id);
const r = allRecipes.find(r => r.frontmatter.id === id);
if (!r && id !== recipe.frontmatter.id) {
warnings.push(`${recipe.frontmatter.id} requires '${id}' (not found)`);
return;
}
if (r) {
for (const dep of r.frontmatter.requires) {
check(dep, [...chain, id]);
}
}
}
for (const dep of recipe.frontmatter.requires) {
check(dep, [recipe.frontmatter.id]);
}
return warnings;
}
// --- Subcommands ---
function cmdList(args: string[]): void {
const jsonMode = args.includes('--json');
const recipes = loadAllRecipes();
if (recipes.length === 0) {
if (jsonMode) {
console.log(JSON.stringify({ senses: [], reflexes: [] }));
} else {
console.log('No integrations available.');
}
return;
}
const infra = recipes.filter(r => r.frontmatter.category === 'infra');
const senses = recipes.filter(r => r.frontmatter.category === 'sense');
const reflexes = recipes.filter(r => r.frontmatter.category === 'reflex');
if (jsonMode) {
const toJson = (r: ParsedRecipe) => ({
id: r.frontmatter.id,
name: r.frontmatter.name,
version: r.frontmatter.version,
description: r.frontmatter.description,
category: r.frontmatter.category,
status: getStatus(r),
setup_time: r.frontmatter.setup_time,
requires: r.frontmatter.requires,
});
console.log(JSON.stringify({
infra: infra.map(toJson),
senses: senses.map(toJson),
reflexes: reflexes.map(toJson),
}, null, 2));
return;
}
const printSection = (title: string, items: ParsedRecipe[]) => {
if (items.length === 0) return;
console.log(`\n ${title}`);
console.log(' ' + '-'.repeat(62));
for (const r of items) {
const status = getStatus(r);
const statusStr = status === 'active' ? 'ACTIVE' : status === 'configured' ? 'CONFIGURED' : 'AVAILABLE';
const id = r.frontmatter.id.padEnd(22);
const desc = r.frontmatter.description.slice(0, 28).padEnd(28);
const deps = r.frontmatter.requires.length > 0 ? ` (needs ${r.frontmatter.requires.join(', ')})` : '';
console.log(` ${id}${desc} ${statusStr}${deps}`);
}
};
// Dashboard view
printSection('INFRASTRUCTURE (set up first)', infra);
printSection('SENSES (data inputs)', senses);
printSection('REFLEXES (automated responses)', reflexes);
// Stats summary
const allHeartbeats = recipes.flatMap(r => readHeartbeat(r.frontmatter.id));
const weekAgo = Date.now() - 7 * 24 * 60 * 60 * 1000;
const weekEvents = allHeartbeats.filter(e => new Date(e.ts).getTime() >= weekAgo);
if (weekEvents.length > 0) {
console.log(`\n This week: ${weekEvents.length} events logged.`);
}
console.log("\n Run 'gbrain integrations show <id>' for setup details.");
console.log('');
}
function cmdShow(args: string[]): void {
const id = args.find(a => !a.startsWith('-'));
if (!id) {
console.error('Usage: gbrain integrations show <recipe-id>');
return;
}
const recipe = findRecipe(id);
if (!recipe) return;
const f = recipe.frontmatter;
console.log(`\n${f.name} (${f.id} v${f.version})`);
console.log(`${f.description}\n`);
console.log(`Category: ${f.category}`);
console.log(`Setup time: ${f.setup_time}`);
if (f.cost_estimate) console.log(`Cost: ${f.cost_estimate}`);
if (f.requires.length > 0) console.log(`Requires: ${f.requires.join(', ')}`);
console.log('\nSecrets needed:');
for (const s of f.secrets) {
const isSet = process.env[s.name] ? ' [set]' : ' [missing]';
console.log(` ${s.name}${isSet}`);
console.log(` ${s.description}`);
console.log(` Get it: ${s.where}`);
}
if (f.health_checks.length > 0) {
console.log(`\nHealth checks: ${f.health_checks.length} configured`);
}
console.log('\n--- Recipe Body ---\n');
console.log(recipe.body);
}
function cmdStatus(args: string[]): void {
const jsonMode = args.includes('--json');
const id = args.find(a => !a.startsWith('-'));
if (!id) {
console.error('Usage: gbrain integrations status <recipe-id>');
return;
}
const recipe = findRecipe(id);
if (!recipe) return;
const { set, missing } = checkSecrets(recipe.frontmatter.secrets);
const heartbeat = readHeartbeat(recipe.frontmatter.id);
const status = getStatus(recipe);
if (jsonMode) {
console.log(JSON.stringify({
id: recipe.frontmatter.id,
status,
secrets: { set, missing: missing.map(m => ({ name: m.name, where: m.where })) },
heartbeat: {
total_events: heartbeat.length,
last_event: heartbeat.length > 0 ? heartbeat[heartbeat.length - 1] : null,
},
}, null, 2));
return;
}
console.log(`\n${recipe.frontmatter.name}: ${status.toUpperCase()}`);
if (set.length > 0) {
console.log('\nSecrets configured:');
for (const s of set) console.log(` ${s} [set]`);
}
if (missing.length > 0) {
console.log('\nMissing secrets:');
for (const m of missing) {
console.log(` ${m.name} [missing]`);
console.log(` Get it: ${m.where}`);
}
}
if (heartbeat.length > 0) {
const last = heartbeat[heartbeat.length - 1];
const lastDate = new Date(last.ts);
const ageMs = Date.now() - lastDate.getTime();
const ageHours = Math.floor(ageMs / (60 * 60 * 1000));
console.log(`\nLast event: ${last.event} (${ageHours}h ago)`);
if (ageMs > 24 * 60 * 60 * 1000) {
console.log(` WARNING: no events in ${Math.floor(ageMs / (24 * 60 * 60 * 1000))} days`);
console.log(' Check: is ngrok running? Is the voice server alive?');
console.log(' Run: gbrain integrations doctor');
}
} else {
console.log('\nNo heartbeat data yet.');
}
console.log('');
}
function cmdDoctor(args: string[]): void {
const jsonMode = args.includes('--json');
const recipes = loadAllRecipes();
const configured = recipes.filter(r => getStatus(r) !== 'available');
if (configured.length === 0) {
if (jsonMode) {
console.log(JSON.stringify({ checks: [], overall: 'no_integrations' }));
} else {
console.log('No configured integrations to check.');
}
return;
}
interface CheckResult {
integration: string;
check: string;
status: 'ok' | 'fail' | 'timeout';
output: string;
}
const results: CheckResult[] = [];
for (const recipe of configured) {
for (const check of recipe.frontmatter.health_checks) {
try {
const output = execSync(check, {
timeout: 10000,
encoding: 'utf-8',
env: process.env,
}).trim();
results.push({
integration: recipe.frontmatter.id,
check,
status: output.includes('FAIL') ? 'fail' : 'ok',
output,
});
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
results.push({
integration: recipe.frontmatter.id,
check,
status: msg.includes('TIMEDOUT') ? 'timeout' : 'fail',
output: msg,
});
}
}
}
if (jsonMode) {
const fails = results.filter(r => r.status !== 'ok');
console.log(JSON.stringify({
checks: results,
overall: fails.length === 0 ? 'ok' : 'issues_found',
}, null, 2));
return;
}
for (const recipe of configured) {
const checks = results.filter(r => r.integration === recipe.frontmatter.id);
const allOk = checks.every(c => c.status === 'ok');
console.log(` ${recipe.frontmatter.id}: ${allOk ? 'OK' : 'ISSUES'}`);
for (const c of checks) {
const icon = c.status === 'ok' ? ' ✓' : c.status === 'timeout' ? ' ⏱' : ' ✗';
console.log(`${icon} ${c.output}`);
}
}
const totalFails = results.filter(r => r.status !== 'ok').length;
console.log(`\n OVERALL: ${totalFails === 0 ? 'All checks passed' : `${totalFails} issue(s) found`}`);
}
function cmdStats(args: string[]): void {
const jsonMode = args.includes('--json');
const recipes = loadAllRecipes();
const allEntries: (HeartbeatEntry & { integration: string })[] = [];
for (const r of recipes) {
const entries = readHeartbeat(r.frontmatter.id);
for (const e of entries) {
allEntries.push({ ...e, integration: r.frontmatter.id });
}
}
if (allEntries.length === 0) {
if (jsonMode) {
console.log(JSON.stringify({ total_events: 0, message: 'No stats yet' }));
} else {
console.log('No stats yet. Set up an integration and start using it.');
}
return;
}
const weekAgo = Date.now() - 7 * 24 * 60 * 60 * 1000;
const weekEntries = allEntries.filter(e => new Date(e.ts).getTime() >= weekAgo);
// Count by integration
const bySense: Record<string, number> = {};
for (const e of weekEntries) {
bySense[e.integration] = (bySense[e.integration] || 0) + 1;
}
if (jsonMode) {
console.log(JSON.stringify({
total_events: allEntries.length,
week_events: weekEntries.length,
by_integration: bySense,
}, null, 2));
return;
}
console.log(`\n This week: ${weekEntries.length} events`);
const sorted = Object.entries(bySense).sort((a, b) => b[1] - a[1]);
for (const [name, count] of sorted) {
const pct = Math.round((count / weekEntries.length) * 100);
console.log(` ${name}: ${count} (${pct}%)`);
}
console.log(`\n All time: ${allEntries.length} events`);
console.log('');
}
function cmdTest(args: string[]): void {
const filePath = args.find(a => !a.startsWith('-'));
if (!filePath) {
console.error('Usage: gbrain integrations test <recipe-file.md>');
return;
}
if (!existsSync(filePath)) {
console.error(`File not found: ${filePath}`);
process.exit(1);
}
const content = readFileSync(filePath, 'utf-8');
const recipe = parseRecipe(content, basename(filePath));
if (!recipe) {
console.error('FAIL: Could not parse recipe. Missing or invalid YAML frontmatter.');
console.error('Required field: id');
process.exit(1);
}
const errors: string[] = [];
const warnings: string[] = [];
// Validate required fields
const f = recipe.frontmatter;
if (!f.id) errors.push('Missing: id');
if (!f.name) warnings.push('Missing: name (will default to id)');
if (!f.description) warnings.push('Missing: description');
if (!f.version) warnings.push('Missing: version');
if (!['sense', 'reflex'].includes(f.category)) {
errors.push(`Invalid category: '${f.category}' (must be 'sense' or 'reflex')`);
}
// Check secrets format
for (const s of f.secrets) {
if (!s.name) errors.push('Secret missing name');
if (!s.where) warnings.push(`Secret '${s.name}' missing 'where' URL`);
}
// Check dependencies
if (f.requires.length > 0) {
const allRecipes = loadAllRecipes();
const depWarnings = checkDependencies(recipe, allRecipes);
warnings.push(...depWarnings);
}
// Check body isn't empty
if (!recipe.body || recipe.body.length < 50) {
warnings.push('Recipe body is very short (< 50 chars). Is the setup guide complete?');
}
// Report
if (errors.length > 0) {
console.log('FAIL:');
for (const e of errors) console.log(`${e}`);
}
if (warnings.length > 0) {
console.log('WARNINGS:');
for (const w of warnings) console.log(`${w}`);
}
if (errors.length === 0 && warnings.length === 0) {
console.log(`PASS: ${f.id} v${f.version}${f.description}`);
}
if (errors.length > 0) process.exit(1);
}
function printHelp(): void {
console.log(`gbrain integrations — manage integration recipes
USAGE
gbrain integrations Show integration dashboard
gbrain integrations list [--json] List available integrations
gbrain integrations show <id> Show recipe details
gbrain integrations status <id> Check secrets + health
gbrain integrations doctor [--json] Run health checks
gbrain integrations stats [--json] Show signal statistics
gbrain integrations test <file> Validate a recipe file
`);
}
// --- Main Entry ---
export async function runIntegrations(args: string[]): Promise<void> {
const sub = args[0];
if (!sub || sub === '--help' || sub === '-h') {
if (!sub) {
// Bare command: show dashboard
cmdList([]);
} else {
printHelp();
}
return;
}
const subArgs = args.slice(1);
switch (sub) {
case 'list':
cmdList(subArgs);
break;
case 'show':
cmdShow(subArgs);
break;
case 'status':
cmdStatus(subArgs);
break;
case 'doctor':
cmdDoctor(subArgs);
break;
case 'stats':
cmdStats(subArgs);
break;
case 'test':
cmdTest(subArgs);
break;
default:
console.error(`Unknown subcommand: ${sub}`);
printHelp();
process.exit(1);
}
}

214
test/integrations.test.ts Normal file
View File

@@ -0,0 +1,214 @@
import { describe, test, expect, beforeAll } from 'bun:test';
import { parseRecipe } from '../src/commands/integrations.ts';
// --- parseRecipe tests ---
describe('parseRecipe', () => {
test('parses valid recipe with full frontmatter', () => {
const content = `---
id: test-recipe
name: Test Recipe
version: 1.0.0
description: A test recipe
category: sense
requires: []
secrets:
- name: API_KEY
description: Test key
where: https://example.com
health_checks:
- "echo ok"
setup_time: 5 min
---
# Setup Guide
Step 1: do the thing.
---
Step 2: do the other thing.
`;
const recipe = parseRecipe(content, 'test.md');
expect(recipe).not.toBeNull();
expect(recipe!.frontmatter.id).toBe('test-recipe');
expect(recipe!.frontmatter.name).toBe('Test Recipe');
expect(recipe!.frontmatter.version).toBe('1.0.0');
expect(recipe!.frontmatter.category).toBe('sense');
expect(recipe!.frontmatter.secrets).toHaveLength(1);
expect(recipe!.frontmatter.secrets[0].name).toBe('API_KEY');
expect(recipe!.frontmatter.secrets[0].where).toBe('https://example.com');
expect(recipe!.frontmatter.health_checks).toHaveLength(1);
// Body should contain the horizontal rule (---) without being split
expect(recipe!.body).toContain('Step 1');
expect(recipe!.body).toContain('Step 2');
expect(recipe!.body).toContain('---');
});
test('body with --- horizontal rules is NOT split as timeline', () => {
const content = `---
id: hr-test
name: HR Test
---
Section one content.
---
Section two content.
---
Section three content.
`;
const recipe = parseRecipe(content, 'hr-test.md');
expect(recipe).not.toBeNull();
// All three sections should be in the body (gray-matter doesn't split on ---)
expect(recipe!.body).toContain('Section one');
expect(recipe!.body).toContain('Section two');
expect(recipe!.body).toContain('Section three');
});
test('returns null for missing id', () => {
const content = `---
name: No ID Recipe
---
Content here.
`;
const recipe = parseRecipe(content, 'no-id.md');
expect(recipe).toBeNull();
});
test('returns null for malformed YAML', () => {
const content = `---
id: broken
this is not: valid: yaml: [
---
Content.
`;
const recipe = parseRecipe(content, 'broken.md');
expect(recipe).toBeNull();
});
test('returns null for no frontmatter', () => {
const content = `# Just a markdown file
No frontmatter here.
`;
const recipe = parseRecipe(content, 'plain.md');
expect(recipe).toBeNull();
});
test('defaults missing optional fields', () => {
const content = `---
id: minimal
---
Minimal recipe.
`;
const recipe = parseRecipe(content, 'minimal.md');
expect(recipe).not.toBeNull();
expect(recipe!.frontmatter.name).toBe('minimal');
expect(recipe!.frontmatter.version).toBe('0.0.0');
expect(recipe!.frontmatter.category).toBe('sense');
expect(recipe!.frontmatter.requires).toEqual([]);
expect(recipe!.frontmatter.secrets).toEqual([]);
expect(recipe!.frontmatter.health_checks).toEqual([]);
});
test('parses reflex category', () => {
const content = `---
id: meeting-prep
category: reflex
---
Prep for meetings.
`;
const recipe = parseRecipe(content, 'reflex.md');
expect(recipe).not.toBeNull();
expect(recipe!.frontmatter.category).toBe('reflex');
});
test('parses multiple secrets', () => {
const content = `---
id: multi-secret
secrets:
- name: KEY_A
description: First key
where: https://a.com
- name: KEY_B
description: Second key
where: https://b.com
- name: KEY_C
description: Third key
where: https://c.com
---
Content.
`;
const recipe = parseRecipe(content, 'multi.md');
expect(recipe).not.toBeNull();
expect(recipe!.frontmatter.secrets).toHaveLength(3);
expect(recipe!.frontmatter.secrets[2].name).toBe('KEY_C');
});
});
// --- CLI structure tests ---
describe('CLI integration', () => {
let cliSource: string;
beforeAll(() => {
const { readFileSync } = require('fs');
cliSource = readFileSync(new URL('../src/cli.ts', import.meta.url), 'utf-8');
});
test('CLI_ONLY set contains integrations', () => {
expect(cliSource).toContain("'integrations'");
});
test('handleCliOnly routes integrations before connectEngine', () => {
// integrations case must appear before "All remaining CLI-only commands need a DB"
const integrationsIdx = cliSource.indexOf("command === 'integrations'");
const dbComment = cliSource.indexOf('All remaining CLI-only commands need a DB');
expect(integrationsIdx).toBeGreaterThan(0);
expect(dbComment).toBeGreaterThan(0);
expect(integrationsIdx).toBeLessThan(dbComment);
});
test('help text mentions integrations', () => {
expect(cliSource).toContain('integrations');
});
});
// --- Recipe file validation ---
describe('twilio-voice-brain recipe', () => {
test('recipe file parses correctly', () => {
const { readFileSync } = require('fs');
const content = readFileSync(
new URL('../recipes/twilio-voice-brain.md', import.meta.url),
'utf-8'
);
const recipe = parseRecipe(content, 'twilio-voice-brain.md');
expect(recipe).not.toBeNull();
expect(recipe!.frontmatter.id).toBe('twilio-voice-brain');
expect(recipe!.frontmatter.category).toBe('sense');
expect(recipe!.frontmatter.secrets.length).toBeGreaterThan(0);
expect(recipe!.frontmatter.health_checks.length).toBeGreaterThan(0);
// Body should not be corrupted (contains --- horizontal rules)
expect(recipe!.body.length).toBeGreaterThan(100);
});
test('recipe has required secrets with where URLs', () => {
const { readFileSync } = require('fs');
const content = readFileSync(
new URL('../recipes/twilio-voice-brain.md', import.meta.url),
'utf-8'
);
const recipe = parseRecipe(content, 'twilio-voice-brain.md');
expect(recipe).not.toBeNull();
for (const secret of recipe!.frontmatter.secrets) {
expect(secret.name).toBeTruthy();
expect(secret.where).toBeTruthy();
expect(secret.where).toContain('https://');
}
});
});