triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
77 Commits 2 Branches 0 Tags
c22ca847721e00d0a10c000412723a2fbcee195d
Commit Graph

3 Commits

Author SHA1 Message Date
Garry Tan
013b348c28 v0.12.3: Reliability wave — sync deadlock, search timeout scoping, wikilinks, orphans (#216) 
* fix(sync): remove nested transaction that deadlocks > 10 file syncs

sync.ts wraps the add/modify loop in engine.transaction(), and each
importFromContent inside opens another one. PGLite's
_runExclusiveTransaction is a non-reentrant mutex — the second call
queues on the mutex the first is holding, and the process hangs forever
in ep_poll. Reproduced with a 15-file commit: unpatched hangs, patched
runs in 3.4s. Fix drops the outer wrap; per-file atomicity is correct
anyway (one file's failure should not roll back the others).

(cherry picked from commit 4a1ac00105226695d16fb343b44e55a52f44b95b)

* test(sync): regression guard for #132 top-level engine.transaction wrap

Reads src/commands/sync.ts verbatim and asserts no uncommented
engine.transaction() call appears above the add/modify loop. Protects
against silent reintroduction of the nested-mutex deadlock that hung
> 10-file syncs forever in ep_poll.

* feat(utils): tryParseEmbedding() skip+warn sibling for availability path

parseEmbedding() throws on structural corruption — right call for ingest/
migrate paths where silent skips would be data loss. Wrong call for
search/rescore paths where one corrupt row in 10K would kill every
query that touches it.

tryParseEmbedding() wraps parseEmbedding in try/catch: returns null on
any shape that would throw, warns once per session so the bad row is
visible in logs. Use it anywhere we'd rather degrade ranking than blow
up the whole query.

Retrofit postgres-engine.getEmbeddingsByChunkIds (the #175 slice call
site) — the 5-line rescore loop was the direct motivator. Keep the
throwing parseEmbedding() for everything else (pglite-engine rowToChunk,
migrate-engine round-trips, ingest).

* postgres-engine: scope search statement_timeout to the transaction

searchKeyword and searchVector run on a pooled postgres.js client
(max: 10 by default). The original code bounded each search with

  await sql`SET statement_timeout = '8s'`
  try { await sql`<query>` }
  finally { await sql`SET statement_timeout = '0'` }

but every tagged template is an independent round-trip that picks an
arbitrary connection from the pool. The SET, the query, and the reset
could all land on DIFFERENT connections. In practice the GUC sticks
to whichever connection ran the SET and then gets returned to the
pool — the next unrelated caller on that connection inherits the 8s
timeout (clipping legitimate long queries) or the reset-to-0 (disabling
the guard for whoever expected it). A crash in the middle leaves the
state set permanently.

Wrap each search in sql.begin(async sql => …). postgres.js reserves
a single connection for the transaction body, so the SET LOCAL, the
query, and the implicit COMMIT all run on the same connection. SET
LOCAL scopes the GUC to the transaction — COMMIT or ROLLBACK restores
the previous value automatically, regardless of the code path out.
Error paths can no longer leak the GUC.

No API change. Timeout value and semantics are identical (8s cap on
search queries, no effect on embed --all / bulk import which runs
outside these methods). Only one transaction per search — BEGIN +
COMMIT round-trips are negligible next to a ranked FTS or pgvector
query.

Also closes the earlier audit finding R4-F002 which reported the same
pattern on searchKeyword. This PR covers both searchKeyword and
searchVector so the pool-leak class is fully closed.

Tests (test/postgres-engine.test.ts, new file):
- No bare SET statement_timeout remains after stripping comments.
- searchKeyword and searchVector each wrap their query in sql.begin.
- Both use SET LOCAL.
- Neither explicitly clears the timeout with SET statement_timeout=0.

Source-level guardrails keep the fast unit suite DB-free. Live
Postgres coverage of the search path is in test/e2e/search-quality.test.ts,
which continues to exercise these methods end-to-end against
pgvector when DATABASE_URL is set.

(cherry picked from commit 6146c3b470dce7380da024a238eab9e6b2174296)

* feat(orphans): add gbrain orphans command for finding under-connected pages

Surfaces pages with zero inbound wikilinks. Essential for content
enrichment cycles in KBs with 1000+ pages. By default filters out
auto-generated pages, raw sources, and pseudo-pages where no inbound
links is expected; --include-pseudo to disable.

Supports text (grouped by domain), --json, --count outputs.
Also exposed as find_orphans MCP operation.

Tests cover basic detection, filtering, all output modes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit f50954f8e03f85803c6133c85c530bd45e9aceaa)

* feat(extract): support Obsidian wikilinks + wiki-style domain slugs in canonical extractor

extractEntityRefs now recognizes both syntaxes equally:
  [Name](people/slug)      -- upstream original
  [[people/slug|Name]]     -- Obsidian wikilink (new)

Extends DIR_PATTERN to include domain-organized wiki slugs used by
Karpathy-style knowledge bases:
  - entities  (legacy prefix some brains keep during migration)
  - projects  (gbrain canonical, was missing from regex)
  - tech, finance, personal, openclaw (domain-organized wiki roots)

Before this change, a 2,100-page brain with wikilinks throughout extracted
zero auto-links on put_page because the regex only matched markdown-style
[name](path). After: 1,377 new typed edges on a single extract --source db
pass over the same corpus.

Matches the behavior of the extract.ts filesystem walker (which already
handled wikilinks as of the wiki-markdown-compat fix wave), so the db and
fs sources now produce the same link graph from the same content.

Both patterns share the DIR_PATTERN constant so adding a new entity dir
only requires updating one string.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1cfb15679a684e94bec5a48c537a0a40a85f57ab)

* feat(doctor): jsonb_integrity + markdown_body_completeness detection

Add two v0.12.1-era reliability checks to `gbrain doctor`:

- `jsonb_integrity` scans the 4 known write sites from the v0.12.0
  double-encode bug (pages.frontmatter, raw_data.data,
  ingest_log.pages_updated, files.metadata) and reports rows where
  jsonb_typeof(col) = 'string'. The fix hint points at
  `gbrain repair-jsonb` (the standalone repair command shipped in
  v0.12.1).

- `markdown_body_completeness` flags pages whose compiled_truth is
  <30% of the raw source content length when raw has multiple H2/H3
  boundaries. Heuristic only; suggests `gbrain sync --force` or
  `gbrain import --force <slug>`.

Also adds test/e2e/jsonb-roundtrip.test.ts — the regression coverage
that should have caught the original double-encode bug. Hits all four
write sites against real Postgres and asserts jsonb_typeof='object'
plus `->>'key'` returns the expected scalar.

Detection only: doctor diagnoses, `gbrain repair-jsonb` treats.
No overlap with the standalone repair path.

* chore: bump to v0.12.3 + changelog (reliability wave)

Master shipped v0.12.1 (extract N+1 + migration timeout) and v0.12.2
(JSONB double-encode + splitBody + wiki types + parseEmbedding) while
this wave was mid-flight. Ships the remaining pieces as v0.12.3:

- sync deadlock (#132, @sunnnybala)
- statement_timeout scoping (#158, @garagon)
- Obsidian wikilinks + domain patterns (#187 slice, @knee5)
- gbrain orphans command (#187 slice, @knee5)
- tryParseEmbedding() availability helper
- doctor detection for jsonb_integrity + markdown_body_completeness

No schema, no migration, no data touch.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: update project documentation for v0.12.3

CLAUDE.md:
- Add src/commands/orphans.ts entry
- Expand src/commands/doctor.ts with v0.12.3 jsonb_integrity +
  markdown_body_completeness check descriptions
- Update src/core/link-extraction.ts to mention Obsidian wikilinks +
  extended DIR_PATTERN (entities/projects/tech/finance/personal/openclaw)
- Update src/core/utils.ts to mention tryParseEmbedding sibling
- Update src/core/postgres-engine.ts to note statement_timeout scoping +
  tryParseEmbedding usage in getEmbeddingsByChunkIds
- Add Key commands added in v0.12.3 section (orphans, doctor checks)
- Add test/orphans.test.ts, test/postgres-engine.test.ts, updated
  descriptions for test/sync.test.ts, test/doctor.test.ts,
  test/utils.test.ts
- Add test/e2e/jsonb-roundtrip.test.ts with note on intentional overlap
- Bump operation count from ~36 to ~41 (find_orphans shipped in v0.12.3)

README.md:
- Add gbrain orphans to ADMIN commands block

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: sunnnybala <dhruvagarwal5018@gmail.com>
Co-authored-by: Gustavo Aragon <gustavoraularagon@gmail.com>
Co-authored-by: Clevin Canales <clevin@Clevins-MacBook-Pro.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Clevin Canales <clev.canales@gmail.com>
 2026-04-19 18:23:02 +08:00
Garry Tan
7bbfc3e36a security: fix wave 3 — 9 vulns (file_upload, SSRF, recipe trust, prompt injection) (#174) 
* feat(engine): add cap parameter to clampSearchLimit (H6)

clampSearchLimit(limit, defaultLimit, cap = MAX_SEARCH_LIMIT) — third arg
is a caller-specified cap so operation handlers can enforce limits below
MAX_SEARCH_LIMIT. Backward compatible: existing two-arg callers still cap
at MAX_SEARCH_LIMIT.

This fixes a Codex-caught semantics bug: the prior signature took (limit,
defaultLimit) where the second arg was misread as a cap. clampSearchLimit(x, 20)
was actually allowing values up to 100, not 20.

* feat(integrations): SSRF defense + recipe trust boundary (B1, B2, Fix 2, Fix 4, B3, B4)

- B1: split loadAllRecipes into trusted (package-bundled) and untrusted
  (cwd/recipes, $GBRAIN_RECIPES_DIR) tiers. Only package-bundled recipes
  get embedded=true. Closes the fake trust boundary that let any cwd-local
  recipe bypass health-check gates.
- B2: hard-block string health_checks for non-embedded recipes (was previously
  only blocked when isUnsafeHealthCheck regex matched, which the cwd recipe
  exploit bypassed). Embedded recipes still get the regex defense.
- Fix 2: gate command DSL health_checks on isEmbedded. Non-embedded
  recipes cannot spawnSync.
- Fix 4 + B3 + B4: gate http DSL health_checks on isEmbedded; for embedded
  recipes, validate URLs via new isInternalUrl() before fetch:
  - Scheme allowlist (http/https only): blocks file:, data:, blob:, ftp:, javascript:
  - IPv4 range check covering hex/octal/decimal/single-integer bypass forms
  - IPv6 loopback ::1 + IPv4-mapped ::ffff: (canonicalized hex hextets handled)
  - Metadata hostnames (AWS, GCP, instance-data) blocked
  - fetch with redirect: 'manual' + per-hop re-validation up to 3 hops

Original PRs #105-109 by @garagon. Wave 3 collector branch reimplemented
the fixes after Codex outside-voice review found that PRs #106/#108 alone
did not actually gate cwd-local recipes (B1) and that PR #108 missed
redirect-following SSRF (B3) and non-http schemes (B4).

* feat(file_upload): path/slug/filename validation + remote-caller confinement (Fix 1, B5, H5, M4, Fix 5)

- Fix 1 + B5 + H1: validateUploadPath uses realpathSync + path.relative
  to defeat symlink-parent traversal. lstatSync alone (the original PR #105
  approach) only catches final-component symlinks; a symlinked parent dir
  still followed to /etc/passwd. Now the entire path chain is resolved.
- H5: validatePageSlug uses an allowlist regex (alphanumeric + hyphens,
  slash-separated segments). Closes URL-encoded traversal (%2e%2e%2f),
  Unicode lookalikes, backslashes, control chars implicitly.
- M4: validateFilename allowlist regex. Rejects control chars, backslash,
  RTL override (\u202E), leading dot/dash. Filename flows into storage_path
  so this matters for every storage backend.
- Fix 5: clamp list_pages and get_ingest_log limits at the operation layer
  via new clampSearchLimit cap parameter (list_pages caps at 100,
  get_ingest_log at 50). Internal bulk commands bypass the operation
  layer and remain uncapped.
- New OperationContext.remote flag distinguishes trusted local CLI from
  untrusted MCP callers. file_upload uses strict cwd confinement when
  remote=true (default), loose mode when remote=false (CLI). MCP stdio
  server sets remote=true; cli.ts and handleToolCall (gbrain call) set
  remote=false.

Original PR #105 by @garagon. Issue #139 reported by @Hybirdss.

* feat(search): query sanitization + structural prompt boundary (Fix 3, M1, M2, M3)

- M1: restructure callHaikuForExpansion to use a system message that declares
  the user query as untrusted data, plus an XML-tagged <user_query> boundary
  in the user message. Layered defense with the existing tool_choice constraint
  (3 layers vs 1).
- Fix 3 (regex sanitizer, defense-in-depth): sanitizeQueryForPrompt strips
  triple-backtick code fences, XML/HTML tags, leading injection prefixes,
  and caps at 500 chars. Original query is still used for downstream search;
  only the LLM-facing copy is sanitized.
- M2: sanitizeExpansionOutput validates the model's alternative_queries array
  before it flows into search. Strips control chars, caps length, dedupes
  case-insensitively, drops empty/non-string items, caps to 2 items.
- M3: console.warn on stripped content NEVER logs the query text — privacy-safe
  debug signal only.

Original PR #107 by @garagon. M1/M2/M3 are wave 3 hardening per Codex review.

* chore: bump version and changelog (v0.10.2)

Security wave 3: 9 vulnerabilities closed across file_upload, recipe trust
boundary, SSRF defense, prompt injection, and limit clamping. See CHANGELOG
for full details.

Contributors:
- @garagon (PRs #105-109)
- @Hybirdss (Issue #139)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync documentation with v0.10.2 security wave 3

- CLAUDE.md: document OperationContext.remote, new security helpers
  (validateUploadPath, validatePageSlug, validateFilename, isInternalUrl,
  parseOctet, hostnameToOctets, isPrivateIpv4, getRecipeDirs,
  sanitizeQueryForPrompt, sanitizeExpansionOutput), updated clampSearchLimit
  signature, recipe trust boundary, new test files
- docs/integrations/README.md: replace string-form health_check example
  with typed DSL (string checks now hard-block for non-embedded recipes);
  add recipe trust boundary subsection
- docs/mcp/DEPLOY.md: document file_upload remote-caller cwd confinement,
  symlink rejection, slug/filename allowlists

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-16 23:03:15 -07:00
Garry Tan
ce15062694 feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39) 
* docs: break SKILLPACK into 17 individual guides

The 1,281-line SKILLPACK monolith is now 17 individually linkable guides
in docs/guides/, organized by category: core patterns, data pipelines,
operations, search, and administration.

GBRAIN_SKILLPACK.md becomes a structured index with categorized tables
linking to each guide. The URL stays stable for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add integration guides, architecture docs, and ethos

New documentation directories:
- docs/integrations/ — "Getting Data In" landing page, credential gateway,
  meeting webhooks. Includes recipe format documentation.
- docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search)
- docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide
- docs/designs/ — "Homebrew for Personal AI" 10-star vision document

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add gbrain integrations command + voice-to-brain recipe

New CLI command: gbrain integrations (list/show/status/doctor/stats/test)
- Standalone command, no database connection needed
- Uses gray-matter directly for recipe parsing (not parseMarkdown)
- --json flag on every subcommand for agent-parseable output
- Bare command shows senses/reflexes dashboard
- Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl

First recipe: recipes/twilio-voice-brain.md
- Phone calls create brain pages via Twilio + OpenAI Realtime
- Opinionated defaults: caller screening, brain-first lookup, quiet hours
- Outbound call smoke test (GBrain calls the user to prove it works)
- Validate-as-you-go credential testing
- Twilio signature validation for webhook security

Migration file for v0.7.0 with agent-readable changelog.
13 unit tests covering parseRecipe, CLI routing, and recipe validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Getting Data In to README, update CLAUDE.md and manifest

README: voice calls in intro bullet list, new "Getting Data In" section
with integration table (voice, email, X, calendar) and recipe philosophy.

CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/,
docs/integrations/, docs/architecture/, docs/ethos/).

manifest.json: bump to v0.7.0, add recipes_dir field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: v0.7.0 CHANGELOG, TODOS, VERSION bump

CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain,
gbrain integrations command, SKILLPACK breakout, and new documentation.

TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL,
community recipe submission, always-on deployment recipes).

VERSION + package.json: bump to 0.7.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite voice recipe with agent instructions and verified links

Major improvements to recipes/twilio-voice-brain.md:

- Agent preamble: explains WHY sequential execution matters (each step
  depends on the previous), defines 4 stop points where the agent MUST
  pause and verify, tells agent to never say "something went wrong"
  but instead explain the exact error and fix

- User actions are now specific: exact URLs for every credential
  (Twilio console, OpenAI API keys page, ngrok dashboard), what
  buttons to click, what fields to copy, common failure modes

- All URLs verified via web search against current 2026 documentation:
  Twilio SID/token at twilio.com/console, OpenAI keys at
  platform.openai.com/api-keys, ngrok token at
  dashboard.ngrok.com/get-started/your-authtoken

- Cost estimate corrected: OpenAI Realtime is $0.06/min input +
  $0.24/min output (was understated), total ~$20-22/mo for 100 min

- Validate-as-you-go: each credential tested immediately with exact
  curl commands, failure messages explain what went wrong and how to fix

- Smoke test flow: tells user exactly what to say, verifies ALL
  three outputs (messaging notification + brain page + search result)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add "Homebrew for Personal AI" essay (markdown is code)

New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution
corollary to "Thin Harness, Fat Skills." Argues that markdown skill files
are simultaneously documentation, specification, package, and source code.
The agent is the package manager. The git repo is the app store.

Referenced from SKILLPACK index and CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite agent instructions as command language, promote skills

The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide.
Every step is an imperative command with exact verification criteria and
explicit stop-on-failure behavior. No FYI, no suggestions, just rails.

Key changes:
- 11-step setup with STOP points after each step
- Exact user instructions for Supabase connection string (what to click,
  what NOT to give the agent, what the string looks like)
- "Verify: run X. You must see Y. If not: Z" after every step
- Skills table now links to both skill files AND guide docs
- Integration recipes table simplified (no "coming soon" placeholders)
- Docs section reorganized: for agents / for humans / reference

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 4 codex findings + add email-to-brain recipe

Codex review found 4 issues, all fixed:

1. getStatus() returned "configured" if ANY secret was set (e.g. just
   OPENAI_API_KEY). Now requires ALL required secrets before marking
   configured. Prevents false "configured" status and spurious doctor runs.

2. Twilio health check hit unauthenticated endpoint (always 401). Now
   uses authenticated curl with SID:token, matching the setup validation.

3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after
   SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md.

4. Compiled binary can't find recipes/ via import.meta.dir. Added
   GBRAIN_RECIPES_DIR env var override + global bun install path fallback.

Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern
with ClawVisor credential gateway, validate-as-you-go, agent instructions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add email, X, calendar, and meeting sync recipes

Four new integration recipes extracted from production wintermute patterns:

- recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector
  pattern (code pulls emails with baked-in links, agent does judgment),
  noise filtering, signature detection, digest generation

- recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search,
  deletion detection (diffs previous run, verifies 404), engagement
  velocity tracking, rate limit awareness

- recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical
  backfill (years of data), daily markdown files with attendees + locations,
  attendee enrichment for brain pages

- recipes/meeting-sync.md: Circleback API, transcript import with speaker
  labels, attendee detection + filtering, entity propagation to people/
  company pages, action item extraction, idempotent by source_id

All recipes follow the same format: agent preamble with sequential execution
rules, validate-as-you-go credentials, exact URLs for API key setup,
stop-on-failure verification, and heartbeat logging.

Updated README, SKILLPACK index, and integrations landing page with all 5 recipes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes

Both recipes now offer two auth options:
- Option A: ClawVisor (recommended, handles OAuth + token refresh)
- Option B: Google OAuth2 directly (no extra service, you manage tokens)

Option B includes step-by-step instructions for Google Cloud Console:
exact URLs, which buttons to click, which scopes to add, how to enable
the API, and the OAuth flow for token exchange.

This removes ClawVisor as a hard dependency for getting started.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add implementation guides with pseudocode and test suggestions

Every recipe now includes an "Implementation Guide" section with:

- Production-tested pseudocode the agent can follow to build each collector
- Edge cases and failure modes discovered in real deployment
- Non-obvious implementation details (why the 48h staleness heuristic,
  why Gmail links need authuser, why SSE responses need double-parsing)
- Test suggestions: what the agent should verify after setup

email-to-brain: noise filtering algorithm, signature detection patterns,
  Gmail link generation (authuser is critical), sent-mail dedup

x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness,
  API verification), engagement velocity thresholds (50 min for 2x, 100
  absolute jump), atomic writes, stdout contract, rate limit handling

calendar-to-brain: smart chunking (monthly for sparse years, weekly for
  dense), attendee filtering (rooms, groups, distros), merge-with-existing
  (only replace ## Calendar section), date/time parsing edge cases

meeting-sync: SSE double-JSON parsing, idempotency double-check (grep +
  filename), auto-tagging from meeting names, git commit after sync

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: 6 new guides from production patterns (wintermute extraction)

New guides extracted and generalized from production deployment:

- repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge).
  Strict boundary rules, decision tree, hard rule: never write knowledge
  to the agent repo.

- sub-agent-routing.md: Model routing table by task type. Signal detector
  pattern (spawn Sonnet on every message). Research pipeline pattern
  (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization.

- skill-development.md: 5-step cycle (concept, prototype, evaluate, codify,
  cron). MECE discipline (no overlapping skills). Quality bar checklist.
  "If you ask twice, it should already be a skill."

- idea-capture.md: Originality distribution rating (0-100 across 4
  populations). Depth test ("could someone unfamiliar understand WHY?").
  Deep cross-linking mandate. Notability filtering.

- quiet-hours.md: Hold notifications 11pm-8am local time. Held messages
  directory pattern. Timezone-aware delivery. Morning briefing pickup.

- diligence-ingestion.md: 9-step pipeline for data room materials. Detection
  patterns (PDF filenames, spreadsheet tabs, user language). Index.md
  template with bull/bear case. Company page enrichment.

All PII scrubbed. Patterns generalized for any user.
SKILLPACK index updated with 6 new entries. CLAUDE.md references added.
All 37 SKILLPACK links verified.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: upgrade all guides to operational playbooks with pseudocode

Every guide now follows the playbook structure:
- Goal: one sentence, what this achieves
- What the User Gets: without this / with this
- Implementation: pseudocode with actual gbrain commands
- Tricky Spots: production-tested gotchas
- How to Verify: test steps the agent runs after setup

Guides upgraded (15 files):
- brain-agent-loop: on_message() loop with read/write/sync pseudocode
- brain-first-lookup: 4-step lookup cascade with exact commands
- brain-vs-memory: routing algorithm for 3 knowledge layers
- compiled-truth: page structure + rewrite vs append rules
- content-media: 3 ingest patterns (YouTube, social, PDFs)
- cron-schedule: full schedule table + dream cycle pseudocode
- enrichment-pipeline: 7-step protocol with tier classification
- entity-detection: spawn pattern + detection prompt + notability filter
- executive-assistant: 3 workflow algorithms (triage, prep, post-inbox)
- meeting-ingestion: 6-step transcript-to-brain flow
- operational-disciplines: 5 executable discipline blocks
- originals-folder: detection + exact-phrasing capture + cross-linking
- search-modes: decision tree for keyword vs hybrid vs direct
- source-attribution: citation format + hierarchy + conflict resolution
- Plus Goal/What User Gets headers on 6 newer guides

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add WebRTC to voice recipe + ngrok Hobby setup guide

Voice recipe updates:
- Added WebRTC endpoint (POST /session, GET /call, POST /tool) for
  browser-based calling with RNNoise noise suppression
- WebRTC pseudocode with the 4 non-obvious gotchas from production
  (voice under audio.output.voice, no turn_detection, no session.update
  on connect, trigger greeting via data channel)
- Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier
- Fixed domain means URLs never change, Twilio never breaks

New guide: docs/mcp/NGROK_SETUP.md
- How to set up ngrok Hobby for both MCP and voice agent
- Fixed domain setup, watchdog pattern, AI client configuration
- Claude Desktop requires Settings > Integrations (not JSON config)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add dependency graph + ngrok-tunnel + credential-gateway recipes

Recipes now have real dependencies via the `requires` field:
- voice-to-brain requires ngrok-tunnel (needs public URL for Twilio)
- email-to-brain requires credential-gateway (needs Gmail access)
- calendar-to-brain requires credential-gateway (needs Calendar access)
- x-to-brain and meeting-sync are standalone (direct API keys)

Two new infrastructure recipes:
- ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby
  ($8/mo) for a domain that never changes. Includes watchdog pattern.
- credential-gateway: secure Google service access via ClawVisor
  (recommended) or direct OAuth2. One setup, all Google recipes use it.

Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure,
not MCP-specific.

README and integrations landing page show dependency chains.
When agent installs voice-to-brain, it sets up ngrok-tunnel first.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add infra category, fix dashboard alignment, show dependencies

DX audit found two bugs in gbrain integrations dashboard:

1. Column alignment broken — IDs > 18 chars ran into descriptions
   with no space. Fixed: pad to 22 chars.

2. ngrok-tunnel and credential-gateway showed as SENSES but they're
   infrastructure. Added 'infra' category. Dashboard now shows three
   sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES.

3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)"

Also added 'requires' field to JSON output for agent consumption.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add frontier model requirement disclaimer to README

GBrain's markdown-is-code approach requires models capable of
interpreting intent and implementing from architecture descriptions.
Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models
will struggle with the recipe format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add PGLite → Supabase upgrade path to README

Clarify the database progression: start with PGLite (Postgres as WASM,
zero infrastructure, pgvector built in, nothing to install). Graduate
to Supabase or self-hosted Postgres when you need connection pooling,
concurrency, and remote MCP access from Claude Desktop, Cowork,
ChatGPT, Perplexity Computer, or any MCP-compatible agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: revert PGLite mention (coming in next branch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify)

Every guide now has exactly these sections in this order:
- ## Goal (one sentence)
- ## What the User Gets (without this / with this)
- ## Implementation (pseudocode with gbrain commands)
- ## Tricky Spots (3-5 numbered gotchas)
- ## How to Verify (3-5 numbered test steps)

11 guides restructured from non-standard headings:
- deterministic-collectors, live-sync, upgrades-auto-update (full rewrites)
- entity-detection, diligence-ingestion, idea-capture, quiet-hours,
  repo-architecture, skill-development, sub-agent-routing (restructured)

23/23 guides now pass consistency audit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: restructure README around the #1 blocker (getting data in)

The README was leading with Postgres and database architecture. Most
users are stuck at step zero: "I have an agent but it doesn't know
anything about my life."

New structure:
1. The Problem — your agent doesn't know your life
2. Getting Data In — integration recipes, front and center
3. The Compounding Thesis — why this matters
4. How this happened — credibility, origin story
5. When you need Postgres — scale, not starting point

Postgres is de-emphasized from a full section to two paragraphs:
"You don't need Postgres to start" and "When you need Postgres"
(1,000+ files, remote MCP access, multiple AI clients).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move Install to top of README, remove duplicate section

Install now appears right after Getting Data In (line 38), not buried
at line 295. The user sees: Problem → Getting Data In → Install.

Removed the duplicate Install section (262 lines) that was lower in
the README. The agent instructions block, CLI quickstart, and all
content is now in the single Install section near the top.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: move agent install block to first thing in README

"Start here: paste this into your agent" is now the first section,
right after the one-line pitch. No scrolling, no context, no preamble.
User opens the README, sees the paste block, copies it into OpenClaw
or Hermes, and the agent takes over.

Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: compress install block from 11 steps to 5

The agent install block was 102 lines and 11 steps. Now it's 40 lines
and 5 steps. Same coverage, half the text.

Changes:
- Merged "prove keyword search" + "embed" + "prove hybrid search"
  into one SEARCH step (the user doesn't care about the intermediate)
- Merged skillpack, sync, auto-update, integrations, verification
  into one GO LIVE step with sub-items (post-install polish, not install)
- Shortened database instructions (one line instead of 5 sub-steps)
- Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now
  just "Do not skip steps. Verify each step.")

The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: gitignore all .env files, not just specific ones

CSO audit found .gitignore covered .env.testing and .env.production
but not bare .env. A user creating .env with database credentials
could accidentally commit it.

Fix: .env and .env.* are now gitignored. .env.*.example files are
explicitly un-ignored so templates remain tracked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* security: scrub PII from essay and recipe examples

- 510-MY-GARRY phone mnemonic → "Your Phone Number"
- "Garry → Authenticated Mode" → "Owner → Authenticated Mode"
- "Telegram" → "secure channel" in auth example
- @garrytan → @yourhandle in X recipe example

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 23:39:06 -10:00