baf3517868e5833ff8c1ff82d30d26877c1f1326
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Garry Tan
|
baf3517868 |
feat: v0.9.0 -- smart file storage, publish, production-grade skills (#62)
* feat: battle-tested skill patterns from production deployment Backport production-learned brain-operations patterns: - Iron Law of Back-Linking (mandatory bidirectional linking) - Brain filing rules (file by primary subject, not format) - Enrichment protocol (7-step pipeline, 3-tier system, person/company templates) - Media ingest workflows (articles, videos, podcasts, PDFs, screenshots) - Citation requirements (mandatory [Source: ...] on every fact) - Test Before Bulk operating principle - Voice recipe: unicode crash fix, PII scrub, identity-first prompt, DIY STT+LLM+TTS - X-to-Brain recipe: image OCR, Filtered Stream, tweet rating rubric, cron stagger * chore: bump version and changelog (v0.8.1) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add _brain-filing-rules.md to CLAUDE.md key files * feat: smart file upload with TUS resumable and .redirect.yaml pointers - Supabase Storage auto-selects upload method by file size: < 100 MB standard POST, >= 100 MB TUS resumable (6 MB chunks + retry) - Signed URL generation for private bucket access (1-hour expiry) - New `upload-raw` command with size routing: small text stays in git, large/media files go to cloud with .redirect.yaml pointer - New `signed-url` command for generating access links - File resolver supports both .redirect.yaml (v0.9+) and .redirect (legacy) - Redirect format upgraded: 10 fields with full metadata - All migration commands (mirror, redirect, restore, clean) handle both formats * feat: skills reference actual gbrain file commands - Filing rules document upload-raw, signed-url, and .redirect.yaml format - Ingest skill uses gbrain files upload-raw for raw source preservation - Maintain skill adds file storage health checks - Setup skill adds storage configuration phase with migration guidance - Voice recipe uses upload-raw for call audio storage - Migration v0.9.0 with complete storage setup instructions * chore: bump version and changelog (v0.9.0) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: gbrain publish -- shareable HTML with password protection First code+skill pair: deterministic code does the work (strip private data, encrypt with AES-256-GCM, generate self-contained HTML), the skill tells the agent when and how to use it. 34 new tests. See: https://x.com/garrytan/status/2042925773300908103 * feat: backlinks check/fix, page lint, and report commands Three new deterministic tools (zero LLM calls): - gbrain backlinks check/fix -- scans brain for entity mentions without back-links, creates them. Enforces the Iron Law from the skills. - gbrain lint [--fix] -- catches LLM preambles, code fence wrapping, placeholder dates, missing frontmatter, broken citations, empty sections. --fix auto-strips fixable artifacts. - gbrain report --type <name> -- saves timestamped reports to brain/reports/{type}/YYYY-MM-DD-HHMM.md for audit trails. 33 new tests (409 total, 0 fail). * feat: v0.9.0 migration tells agents to swap scripts for built-in commands Migration file now: - Lists all 5 new deterministic commands with usage examples - Includes a script-to-command replacement table (old -> new) - Tells the agent to find custom script references in AGENTS.md, skills, and cron jobs and replace with gbrain commands - Adds recommended cron jobs for daily backlink fix + weekly lint - References the Thin Harness, Fat Skills thread * fix: CLI routing bugs found during DX review - Fixed subArgs reference error in handleCliOnly (used wrong variable name) - Renamed gbrain backlinks check/fix to gbrain check-backlinks to avoid conflict with existing backlinks operation (per-page incoming links) - Added TOOLS section to --help output showing publish, check-backlinks, lint, report - Added upload-raw and signed-url to FILES section in --help - Updated all docs/migration references to use check-backlinks * fix: security hardening from adversarial review - XSS: sanitize marked.parse() output (strip script/iframe/on* attrs) - Path traversal: validate report --type against [a-z0-9-] pattern - TUS: HEAD request before retry to get server's actual offset (TUS spec) - Pointer: upload-raw now includes pointer content in JSON output - Symlinks: use lstatSync in all walkers to prevent directory escape --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Garry Tan
|
91ced664b6 |
feat: Voice v0.8.0 + feature discovery + Edge Function removal (#55)
* chore: remove Supabase Edge Function MCP deployment The Edge Function never worked reliably. All MCP traffic goes through self-hosted server + ngrok tunnel. Removes deploy-remote.sh, edge-entry.ts, supabase/functions/, .env.production.example, and CHATGPT.md (OAuth not implemented). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite MCP docs for self-hosted + ngrok deployment All per-client guides updated from Edge Function URLs to self-hosted server + ngrok tunnel pattern. DEPLOY.md rewritten with local vs remote paths. ALTERNATIVES.md now shows self-hosted as primary, with ngrok, Tailscale, and Fly.io/Railway comparison. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: voice recipe v0.8.0 — 25 production patterns from real deployment Identity separation, pre-computed bid system, conversation timing fix, proactive advisor mode, radical prompt compression, OpenAI Realtime Prompting Guide structure, auth-before-speech, brain escalation, stuck watchdog, never-hang-up rule, thinking sounds, fallback TwiML, tool set architecture, trusted user auth, caller routing, dynamic VAD, on-screen debug UI, live moment capture, belt-and-suspenders post-call, mandatory 3-step post-call, WebRTC parity, dual API events, report-aware query routing. WebRTC pseudocode updated with native FormData and 6 gotchas. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: post-upgrade feature discovery framework upgrade.ts captures old version before upgrading, then execs gbrain post-upgrade (new binary) to read migration files and print feature pitches. Migration files get YAML frontmatter with feature_pitch field (headline, description, recipe, tiers). CLI prints excited builder tone post-upgrade. v0.8.0 migration offers voice setup with environment detection (server vs local) and 3-tier progressive disclosure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add Voice section to README with WebRTC screenshot + tweet link Her out of the box: voice-to-brain with 25 production patterns. WebRTC client screenshot embedded. Remote MCP section rewritten for self-hosted + ngrok. Setup block genericized. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add recipe validation tests + genericize personal refs 5 new integration tests: secrets completeness, semver version, requires resolution, all-recipes-parse, no-personal-references. Test fixture genericized. CLAUDE.md/TODOS.md/SKILLPACK updated for v0.8.0. build:edge script removed from package.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.8.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
6c7d2ed30b |
feat: PGLite engine — local brain, zero infrastructure (v0.7.0) (#41)
* refactor: extract shared utils, add runMigration + getChunksWithEmbeddings to BrainEngine Extract validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult from postgres-engine.ts into shared utils.ts. Add rowToChunk includeEmbedding parameter for migration support. Add two new methods to BrainEngine interface: - runMigration(version, sql) — replaces internal eng.sql access in migrate.ts - getChunksWithEmbeddings(slug) — returns chunks with embedding data for migration Replace 'sqlite' with 'pglite' in EngineConfig and GBrainConfig types. Fix loadConfig to infer engine from database_path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: pluggable engine factory + hybridSearch keyword-only fallback Add createEngine() factory with dynamic imports so PGLite WASM is never loaded for Postgres users. Wire CLI to use factory instead of hardcoded PostgresEngine. Force workers=1 for PGLite imports (single-connection architecture). Fix hybridSearch to check OPENAI_API_KEY before calling embed(). When unset, returns keyword-only results instead of throwing. Critical for local PGLite users who don't need vector search. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: PGLiteEngine — embedded Postgres 17.5 via WASM, same SQL everywhere Full BrainEngine implementation (37 methods) using @electric-sql/pglite. Same SQL as PostgresEngine — tsvector triggers, pgvector HNSW, pg_trgm fuzzy matching, recursive CTEs, JSONB. Only the driver call syntax differs (parameterized queries instead of tagged templates). PGLite schema is the Postgres schema minus RLS, advisory locks, and remote auth tables (access_tokens, mcp_request_log, files). No server. No subscription. One directory. Works offline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: smart init (PGLite default) + bidirectional engine migration gbrain init now defaults to PGLite — brain ready in 2 seconds, no server needed. Scans target directory: <1000 .md files = PGLite, >=1000 = suggests Supabase. --supabase and --pglite flags override. gbrain migrate --to supabase/pglite transfers all data between engines with manifest-based resume. Copies pages, chunks (with embeddings), tags, timeline, raw data, links, and config. --force overwrites non-empty target. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 60 new tests for PGLite engine, utils, and factory 41 PGLite engine tests covering all 37 BrainEngine methods: CRUD, tsvector keyword search, pg_trgm fuzzy matching, chunk upsert with COALESCE, graph traversal via recursive CTE, transactions, cascade deletes, stats/health, and embedding round-trip. 14 shared utility tests (validateSlug, contentHash, row mappers). 5 engine factory tests (dispatch, error messages). All run in-memory — zero Docker, zero DATABASE_URL, instant in CI. Add P0 TODO: submit Bun PR for WASM embedding in bun build --compile (oven-sh/bun#15032). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.7.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.7.0 PGLite engine - CLAUDE.md: add PGLite key files, update architecture, add migrate command, add 3 test files - README.md: PGLite as default init, zero-config getting started, migration path to Supabase - docs/ENGINES.md: PGLiteEngine shipped (v0.7), capability matrix, migration docs - docs/SQLITE_ENGINE.md: marked superseded by PGLite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove stale v0.4 README update prompt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove SQLITE_ENGINE.md (superseded by PGLite) PGLite uses the same SQL as Postgres, making a separate SQLite engine unnecessary. docs/ENGINES.md covers PGLiteEngine. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update README step 2 to default to PGLite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add schema setup step and install-all-integrations step to README Step 3 now tells agents to read GBRAIN_RECOMMENDED_SCHEMA.md and set up the MECE directory structure before importing. Step 7 tells agents to install every available integration recipe, not just list them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update install goal to match full opinionated setup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add 'Need an AI agent first?' section with one-click deploy links New users who don't have OpenClaw or Hermes Agent get pointed to AlphaClaw on Render and the Hermes Agent Railway template. One click each. Claude Code mentioned for users who already have it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add migrate to CLI_ONLY + help output, fix standalone example - migrate command was missing from CLI_ONLY set (errored as "Unknown command") - migrate now shows in --help under SETUP - init help line shows --pglite flag - standalone CLI example uses gbrain init (not --supabase) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: set realistic time expectation (~30 min to working brain) DB is 2 seconds. But schema + import + embeddings + integrations is 15-30 minutes. The agent does the work, you answer API key questions. Don't oversell time-to-value. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix AlphaClaw Render requirement (8GB+ RAM, not free tier) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: final README polish for launch - GOAL line: "Garry Tan's exact setup" (not Claude Code specific) - Remove markdown links from code block (won't render) - STEP 2 renamed from "START HERE" to "DATABASE" - Tighten Supabase fallback text Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: remove duplicate old install block from README The v0.5-era "With OpenClaw or Hermes Agent" paste block was superseded by the top-level "Start here" block. Having both confused users and the old one still said --supabase as step 2. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clean up README consistency and remove duplicated content - Remove duplicate "Try it" section (old 4-act walkthrough that repeated the install flow and contradicted "~30 min" with "90 sec") - Remove duplicate Setup section (third repetition of gbrain init) - Fix brain.db → brain.pglite (actual default path) - Fix "coming in v0.7" → "not yet implemented" (we ARE v0.7) - Remove "You don't need Postgres" (confusing since PGLite IS Postgres) - Deduplicate "competitive dynamics" query (appeared 3 times) - Collapse redundant standalone CLI section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
ce15062694 |
feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39)
* docs: break SKILLPACK into 17 individual guides The 1,281-line SKILLPACK monolith is now 17 individually linkable guides in docs/guides/, organized by category: core patterns, data pipelines, operations, search, and administration. GBRAIN_SKILLPACK.md becomes a structured index with categorized tables linking to each guide. The URL stays stable for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add integration guides, architecture docs, and ethos New documentation directories: - docs/integrations/ — "Getting Data In" landing page, credential gateway, meeting webhooks. Includes recipe format documentation. - docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search) - docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide - docs/designs/ — "Homebrew for Personal AI" 10-star vision document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain integrations command + voice-to-brain recipe New CLI command: gbrain integrations (list/show/status/doctor/stats/test) - Standalone command, no database connection needed - Uses gray-matter directly for recipe parsing (not parseMarkdown) - --json flag on every subcommand for agent-parseable output - Bare command shows senses/reflexes dashboard - Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl First recipe: recipes/twilio-voice-brain.md - Phone calls create brain pages via Twilio + OpenAI Realtime - Opinionated defaults: caller screening, brain-first lookup, quiet hours - Outbound call smoke test (GBrain calls the user to prove it works) - Validate-as-you-go credential testing - Twilio signature validation for webhook security Migration file for v0.7.0 with agent-readable changelog. 13 unit tests covering parseRecipe, CLI routing, and recipe validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Getting Data In to README, update CLAUDE.md and manifest README: voice calls in intro bullet list, new "Getting Data In" section with integration table (voice, email, X, calendar) and recipe philosophy. CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/, docs/integrations/, docs/architecture/, docs/ethos/). manifest.json: bump to v0.7.0, add recipes_dir field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.7.0 CHANGELOG, TODOS, VERSION bump CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain, gbrain integrations command, SKILLPACK breakout, and new documentation. TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL, community recipe submission, always-on deployment recipes). VERSION + package.json: bump to 0.7.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite voice recipe with agent instructions and verified links Major improvements to recipes/twilio-voice-brain.md: - Agent preamble: explains WHY sequential execution matters (each step depends on the previous), defines 4 stop points where the agent MUST pause and verify, tells agent to never say "something went wrong" but instead explain the exact error and fix - User actions are now specific: exact URLs for every credential (Twilio console, OpenAI API keys page, ngrok dashboard), what buttons to click, what fields to copy, common failure modes - All URLs verified via web search against current 2026 documentation: Twilio SID/token at twilio.com/console, OpenAI keys at platform.openai.com/api-keys, ngrok token at dashboard.ngrok.com/get-started/your-authtoken - Cost estimate corrected: OpenAI Realtime is $0.06/min input + $0.24/min output (was understated), total ~$20-22/mo for 100 min - Validate-as-you-go: each credential tested immediately with exact curl commands, failure messages explain what went wrong and how to fix - Smoke test flow: tells user exactly what to say, verifies ALL three outputs (messaging notification + brain page + search result) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add "Homebrew for Personal AI" essay (markdown is code) New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution corollary to "Thin Harness, Fat Skills." Argues that markdown skill files are simultaneously documentation, specification, package, and source code. The agent is the package manager. The git repo is the app store. Referenced from SKILLPACK index and CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite agent instructions as command language, promote skills The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide. Every step is an imperative command with exact verification criteria and explicit stop-on-failure behavior. No FYI, no suggestions, just rails. Key changes: - 11-step setup with STOP points after each step - Exact user instructions for Supabase connection string (what to click, what NOT to give the agent, what the string looks like) - "Verify: run X. You must see Y. If not: Z" after every step - Skills table now links to both skill files AND guide docs - Integration recipes table simplified (no "coming soon" placeholders) - Docs section reorganized: for agents / for humans / reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 4 codex findings + add email-to-brain recipe Codex review found 4 issues, all fixed: 1. getStatus() returned "configured" if ANY secret was set (e.g. just OPENAI_API_KEY). Now requires ALL required secrets before marking configured. Prevents false "configured" status and spurious doctor runs. 2. Twilio health check hit unauthenticated endpoint (always 401). Now uses authenticated curl with SID:token, matching the setup validation. 3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md. 4. Compiled binary can't find recipes/ via import.meta.dir. Added GBRAIN_RECIPES_DIR env var override + global bun install path fallback. Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern with ClawVisor credential gateway, validate-as-you-go, agent instructions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add email, X, calendar, and meeting sync recipes Four new integration recipes extracted from production wintermute patterns: - recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector pattern (code pulls emails with baked-in links, agent does judgment), noise filtering, signature detection, digest generation - recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search, deletion detection (diffs previous run, verifies 404), engagement velocity tracking, rate limit awareness - recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical backfill (years of data), daily markdown files with attendees + locations, attendee enrichment for brain pages - recipes/meeting-sync.md: Circleback API, transcript import with speaker labels, attendee detection + filtering, entity propagation to people/ company pages, action item extraction, idempotent by source_id All recipes follow the same format: agent preamble with sequential execution rules, validate-as-you-go credentials, exact URLs for API key setup, stop-on-failure verification, and heartbeat logging. Updated README, SKILLPACK index, and integrations landing page with all 5 recipes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes Both recipes now offer two auth options: - Option A: ClawVisor (recommended, handles OAuth + token refresh) - Option B: Google OAuth2 directly (no extra service, you manage tokens) Option B includes step-by-step instructions for Google Cloud Console: exact URLs, which buttons to click, which scopes to add, how to enable the API, and the OAuth flow for token exchange. This removes ClawVisor as a hard dependency for getting started. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation guides with pseudocode and test suggestions Every recipe now includes an "Implementation Guide" section with: - Production-tested pseudocode the agent can follow to build each collector - Edge cases and failure modes discovered in real deployment - Non-obvious implementation details (why the 48h staleness heuristic, why Gmail links need authuser, why SSE responses need double-parsing) - Test suggestions: what the agent should verify after setup email-to-brain: noise filtering algorithm, signature detection patterns, Gmail link generation (authuser is critical), sent-mail dedup x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness, API verification), engagement velocity thresholds (50 min for 2x, 100 absolute jump), atomic writes, stdout contract, rate limit handling calendar-to-brain: smart chunking (monthly for sparse years, weekly for dense), attendee filtering (rooms, groups, distros), merge-with-existing (only replace ## Calendar section), date/time parsing edge cases meeting-sync: SSE double-JSON parsing, idempotency double-check (grep + filename), auto-tagging from meeting names, git commit after sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: 6 new guides from production patterns (wintermute extraction) New guides extracted and generalized from production deployment: - repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge). Strict boundary rules, decision tree, hard rule: never write knowledge to the agent repo. - sub-agent-routing.md: Model routing table by task type. Signal detector pattern (spawn Sonnet on every message). Research pipeline pattern (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization. - skill-development.md: 5-step cycle (concept, prototype, evaluate, codify, cron). MECE discipline (no overlapping skills). Quality bar checklist. "If you ask twice, it should already be a skill." - idea-capture.md: Originality distribution rating (0-100 across 4 populations). Depth test ("could someone unfamiliar understand WHY?"). Deep cross-linking mandate. Notability filtering. - quiet-hours.md: Hold notifications 11pm-8am local time. Held messages directory pattern. Timezone-aware delivery. Morning briefing pickup. - diligence-ingestion.md: 9-step pipeline for data room materials. Detection patterns (PDF filenames, spreadsheet tabs, user language). Index.md template with bull/bear case. Company page enrichment. All PII scrubbed. Patterns generalized for any user. SKILLPACK index updated with 6 new entries. CLAUDE.md references added. All 37 SKILLPACK links verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: upgrade all guides to operational playbooks with pseudocode Every guide now follows the playbook structure: - Goal: one sentence, what this achieves - What the User Gets: without this / with this - Implementation: pseudocode with actual gbrain commands - Tricky Spots: production-tested gotchas - How to Verify: test steps the agent runs after setup Guides upgraded (15 files): - brain-agent-loop: on_message() loop with read/write/sync pseudocode - brain-first-lookup: 4-step lookup cascade with exact commands - brain-vs-memory: routing algorithm for 3 knowledge layers - compiled-truth: page structure + rewrite vs append rules - content-media: 3 ingest patterns (YouTube, social, PDFs) - cron-schedule: full schedule table + dream cycle pseudocode - enrichment-pipeline: 7-step protocol with tier classification - entity-detection: spawn pattern + detection prompt + notability filter - executive-assistant: 3 workflow algorithms (triage, prep, post-inbox) - meeting-ingestion: 6-step transcript-to-brain flow - operational-disciplines: 5 executable discipline blocks - originals-folder: detection + exact-phrasing capture + cross-linking - search-modes: decision tree for keyword vs hybrid vs direct - source-attribution: citation format + hierarchy + conflict resolution - Plus Goal/What User Gets headers on 6 newer guides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add WebRTC to voice recipe + ngrok Hobby setup guide Voice recipe updates: - Added WebRTC endpoint (POST /session, GET /call, POST /tool) for browser-based calling with RNNoise noise suppression - WebRTC pseudocode with the 4 non-obvious gotchas from production (voice under audio.output.voice, no turn_detection, no session.update on connect, trigger greeting via data channel) - Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier - Fixed domain means URLs never change, Twilio never breaks New guide: docs/mcp/NGROK_SETUP.md - How to set up ngrok Hobby for both MCP and voice agent - Fixed domain setup, watchdog pattern, AI client configuration - Claude Desktop requires Settings > Integrations (not JSON config) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency graph + ngrok-tunnel + credential-gateway recipes Recipes now have real dependencies via the `requires` field: - voice-to-brain requires ngrok-tunnel (needs public URL for Twilio) - email-to-brain requires credential-gateway (needs Gmail access) - calendar-to-brain requires credential-gateway (needs Calendar access) - x-to-brain and meeting-sync are standalone (direct API keys) Two new infrastructure recipes: - ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby ($8/mo) for a domain that never changes. Includes watchdog pattern. - credential-gateway: secure Google service access via ClawVisor (recommended) or direct OAuth2. One setup, all Google recipes use it. Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure, not MCP-specific. README and integrations landing page show dependency chains. When agent installs voice-to-brain, it sets up ngrok-tunnel first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add infra category, fix dashboard alignment, show dependencies DX audit found two bugs in gbrain integrations dashboard: 1. Column alignment broken — IDs > 18 chars ran into descriptions with no space. Fixed: pad to 22 chars. 2. ngrok-tunnel and credential-gateway showed as SENSES but they're infrastructure. Added 'infra' category. Dashboard now shows three sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES. 3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)" Also added 'requires' field to JSON output for agent consumption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add frontier model requirement disclaimer to README GBrain's markdown-is-code approach requires models capable of interpreting intent and implementing from architecture descriptions. Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models will struggle with the recipe format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add PGLite → Supabase upgrade path to README Clarify the database progression: start with PGLite (Postgres as WASM, zero infrastructure, pgvector built in, nothing to install). Graduate to Supabase or self-hosted Postgres when you need connection pooling, concurrency, and remote MCP access from Claude Desktop, Cowork, ChatGPT, Perplexity Computer, or any MCP-compatible agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: revert PGLite mention (coming in next branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify) Every guide now has exactly these sections in this order: - ## Goal (one sentence) - ## What the User Gets (without this / with this) - ## Implementation (pseudocode with gbrain commands) - ## Tricky Spots (3-5 numbered gotchas) - ## How to Verify (3-5 numbered test steps) 11 guides restructured from non-standard headings: - deterministic-collectors, live-sync, upgrades-auto-update (full rewrites) - entity-detection, diligence-ingestion, idea-capture, quiet-hours, repo-architecture, skill-development, sub-agent-routing (restructured) 23/23 guides now pass consistency audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: restructure README around the #1 blocker (getting data in) The README was leading with Postgres and database architecture. Most users are stuck at step zero: "I have an agent but it doesn't know anything about my life." New structure: 1. The Problem — your agent doesn't know your life 2. Getting Data In — integration recipes, front and center 3. The Compounding Thesis — why this matters 4. How this happened — credibility, origin story 5. When you need Postgres — scale, not starting point Postgres is de-emphasized from a full section to two paragraphs: "You don't need Postgres to start" and "When you need Postgres" (1,000+ files, remote MCP access, multiple AI clients). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move Install to top of README, remove duplicate section Install now appears right after Getting Data In (line 38), not buried at line 295. The user sees: Problem → Getting Data In → Install. Removed the duplicate Install section (262 lines) that was lower in the README. The agent instructions block, CLI quickstart, and all content is now in the single Install section near the top. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move agent install block to first thing in README "Start here: paste this into your agent" is now the first section, right after the one-line pitch. No scrolling, no context, no preamble. User opens the README, sees the paste block, copies it into OpenClaw or Hermes, and the agent takes over. Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: compress install block from 11 steps to 5 The agent install block was 102 lines and 11 steps. Now it's 40 lines and 5 steps. Same coverage, half the text. Changes: - Merged "prove keyword search" + "embed" + "prove hybrid search" into one SEARCH step (the user doesn't care about the intermediate) - Merged skillpack, sync, auto-update, integrations, verification into one GO LIVE step with sub-items (post-install polish, not install) - Shortened database instructions (one line instead of 5 sub-steps) - Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now just "Do not skip steps. Verify each step.") The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: gitignore all .env files, not just specific ones CSO audit found .gitignore covered .env.testing and .env.production but not bare .env. A user creating .env with database credentials could accidentally commit it. Fix: .env and .env.* are now gitignored. .env.*.example files are explicitly un-ignored so templates remain tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: scrub PII from essay and recipe examples - 510-MY-GARRY phone mnemonic → "Your Phone Number" - "Garry → Authenticated Mode" → "Owner → Authenticated Mode" - "Telegram" → "secure channel" in auth example - @garrytan → @yourhandle in X recipe example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
8de04d3827 |
fix: community fix wave — 9 PRs, 8 contributors (v0.6.1) (#38)
* fix: validateSlug accepts ellipsis filenames, rejects only real path traversal Changed regex from /\.\./ to /(^|\/)\.\.($|\/)/ so filenames with "..." (like YouTube transcripts, TED talks, podcast titles) are no longer falsely rejected. The old regex matched ".." anywhere as a substring. The new one only matches ".." as a complete path component (e.g., ../foo, foo/../bar, bare ..). Fixes 1.2% silent data loss on real-world import corpora. Co-Authored-By: orendi84 <orendi84@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: import walker skips node_modules, handles broken symlinks, supports .mdx Three improvements to the file walker: - Skip node_modules directories (prevents crashes importing JS/TS projects) - try/catch around statSync for broken symlinks (warns and continues) - Accept .mdx files alongside .md (extends to slugifyPath and isSyncable) Co-Authored-By: mattbratos <mattbratos@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: init exits cleanly, auto-creates pgvector, updates Supabase UI hint Three init improvements: - process.stdin.pause() after reading URL input (prevents event loop hang) - Auto-run CREATE EXTENSION IF NOT EXISTS vector with fallback message - Update Supabase session pooler navigation hint to match current dashboard UI Co-Authored-By: changergosum <changergosum@users.noreply.github.com> Co-Authored-By: eric-hth <eric-hth@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * perf: parallelize keyword search with embedding pipeline Run keyword search concurrently with the embed+vector pipeline instead of sequentially. Keyword search has no embedding dependency so it can overlap with the OpenAI API call, saving ~200-500ms per search. Co-Authored-By: irresi <irresi@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update Hermes Agent link to NousResearch GitHub repo Co-Authored-By: howardpen9 <howardpen9@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add community PR wave process to CLAUDE.md Documents the fix wave workflow: categorize, deduplicate, collector branch, test, close with context, ship as one PR with attribution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: bump version and changelog (v0.6.1) Community fix wave: 9 PRs re-implemented with full test coverage. 6 bug fixes, 1 perf improvement, 2 feature additions, 8 contributors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: migrate gstack from vendored to team mode Remove vendored .claude/skills/gstack/ from git tracking. The global install at ~/.claude/skills/gstack/ is the source of truth. Each developer runs `cd ~/.claude/skills/gstack && ./setup` to set up symlink stubs locally. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: untrack skill symlink stubs These are generated locally by gstack's ./setup script. Not project code. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: credit community contributors in CHANGELOG Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update OpenClaw links from .com to .ai openclaw.com is a parked page. openclaw.ai is the real product. Co-Authored-By: joshua-morris <joshua-morris@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: orendi84 <orendi84@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: mattbratos <mattbratos@users.noreply.github.com> Co-authored-by: changergosum <changergosum@users.noreply.github.com> Co-authored-by: eric-hth <eric-hth@users.noreply.github.com> Co-authored-by: irresi <irresi@users.noreply.github.com> Co-authored-by: howardpen9 <howardpen9@users.noreply.github.com> Co-authored-by: joshua-morris <joshua-morris@users.noreply.github.com> |
||
|
Garry Tan
|
3e21e9b69b |
feat: GBrain v0.6.0 — Remote MCP Server + 12 Bug Fixes (#28)
* fix: 7 bug fixes from Issue #9 and #22 - fix(mcp): use ListToolsRequestSchema/CallToolRequestSchema instead of string literals (Issue #9, PR #25) - fix(mcp): handleToolCall reads dry_run from params instead of hardcoding false (#22 Bug #11) - fix(search): keyword search returns best chunk per page via DISTINCT ON, not all chunks (#22 Bug #8) - fix(search): dedup layer 1 keeps top 3 chunks per page instead of collapsing to 1 (#22 Bug #12) - fix(engine): transaction uses scoped engine via Object.create, no shared state mutation (#22 Bug #2) - fix(engine): upsertChunks uses UPSERT instead of DELETE+INSERT, preserves existing embeddings (#22 Bug #1) - fix(slugs): validateSlug normalizes to lowercase, pathToSlug lowercases consistently (#22 Bug #4) - schema: add unique index on content_chunks(page_id, chunk_index) for UPSERT support - schema: add access_tokens and mcp_request_log tables via migration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: embed schema.sql at build time, remove fs dependency from initSchema initSchema() previously read schema.sql from disk at runtime via readFileSync, which broke in compiled Bun binaries and Deno Edge Functions. Now uses a generated schema-embedded.ts constant (run `bun run build:schema` to regenerate). - Removes fs and path imports from postgres-engine.ts and db.ts - Adds scripts/build-schema.sh for one-source-of-truth generation - Adds build:schema npm script Fixes Issue #22 Bug #6. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 5 more bug fixes from Issue #22 - fix(file_upload): call storage.upload() in all 3 paths (operation, CLI upload, CLI sync) with rollback semantics (#22 Bug #9) - fix(import): use atomic index counter for parallel queue instead of array.shift() race, preserve checkpoint on errors (#22 Bug #3) - fix(s3): replace unsigned fetch with @aws-sdk/client-s3 for proper SigV4 auth, supports R2/MinIO via forcePathStyle (#22 Bug #10) - fix(redirect): verify remote file exists before deleting local copy, skip files not found in storage (#22 Bug #5) - deps: add @aws-sdk/client-s3 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: remote MCP server via Supabase Edge Functions Deploy GBrain as a serverless remote MCP endpoint on your existing Supabase instance. One brain, accessible from Claude Desktop, Claude Code, Cowork, Perplexity Computer, and any MCP client. Zero new infrastructure. New files: - supabase/functions/gbrain-mcp/index.ts — Edge Function with Hono + MCP SDK - supabase/functions/gbrain-mcp/deno.json — Deno import map - src/edge-entry.ts — curated bundle entry point (excludes fs-dependent modules) - src/commands/auth.ts — standalone token management (create/list/revoke/test) - scripts/deploy-remote.sh — one-script deployment - .env.production.example — 3-value config template Changes: - config.ts: lazy-evaluate CONFIG_DIR (no homedir() at module scope) - schema.sql: add access_tokens + mcp_request_log tables - package.json: add build:edge script Auth: bearer tokens via access_tokens table (SHA-256 hashed, per-client, revocable) Transport: WebStandardStreamableHTTPServerTransport (stateless, Streamable HTTP) Health: /health endpoint (unauth: 200/503, auth: postgres/pgvector/openai checks) Excluded from remote: sync_brain, file_upload (may exceed 60s timeout) Setup: clone, fill .env.production, run scripts/deploy-remote.sh, create token, done. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: per-client MCP setup guides - docs/mcp/DEPLOY.md — deployment walkthrough, auth, troubleshooting, latency table - docs/mcp/CLAUDE_CODE.md — claude mcp add command - docs/mcp/CLAUDE_DESKTOP.md — Settings > Integrations (NOT JSON config!) - docs/mcp/CLAUDE_COWORK.md — remote + local bridge paths - docs/mcp/PERPLEXITY.md — Perplexity Computer connector setup - docs/mcp/CHATGPT.md — coming soon (requires OAuth 2.1, P0 TODO) - docs/mcp/ALTERNATIVES.md — Tailscale Funnel + ngrok self-hosted options Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.6.0) GBrain v0.6.0: Remote MCP server via Supabase Edge Functions + 12 bug fixes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Remote MCP Server section to README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make document-release mandatory in CLAUDE.md, add MCP key files Post-ship requirements section: document-release is NOT optional. Lists every file that must be checked on every ship. A ship without updated docs is incomplete. Also adds remote MCP server files to Key files section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: batch upsertChunks into single statement to prevent deadlocks The per-chunk UPSERT loop caused deadlocks under parallel workers because each INSERT ON CONFLICT acquired row-level locks sequentially. Multiple workers upserting different pages could deadlock on the shared unique index. Fix: batch all chunks into a single multi-row INSERT ON CONFLICT statement. One round-trip, one lock acquisition. COALESCE preserves existing embeddings when the new value is NULL. Fixes CI failure: "E2E: Parallel Import > parallel import with --workers 4" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: advisory lock in initSchema() prevents deadlock on concurrent DDL When multiple processes call initSchema() concurrently (e.g., test setup + CLI subprocess, or parallel workers during E2E tests), the schema SQL's DROP TRIGGER + CREATE TRIGGER statements acquire AccessExclusiveLock on different tables, causing deadlocks. Fix: pg_advisory_lock(42) serializes all initSchema() calls within the same database. The lock is session-scoped and released in a finally block. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add explicit test timeouts for CLI subprocess E2E tests CLI subprocess tests (Setup Journey, Doctor Command, Parallel Import) spawn `bun run src/cli.ts` which takes several seconds to JIT compile + connect. The Bun test framework default 5000ms per-test timeout is too tight for CI. Added 30-60s timeouts matching each subprocess's own timeout to prevent false failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: infinite recursion in config.ts exported getConfigDir/getConfigPath The replace_all refactor created recursive functions: the exported getConfigDir() called the private getConfigDir() which called itself. Renamed exports to configDir()/configPath() to avoid shadowing. Also adds scripts/smoke-test-mcp.ts — verified all 8 MCP tool calls work against a real Postgres database. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
27eb87f1f4 |
feat: slugify file paths with spaces and special characters (v0.5.1) (#29)
* feat: slugify file paths with spaces and special characters Apple Notes files (e.g., "2017-05-03 ohmygreen.md") now get clean, URL-safe slugs instead of raw filenames with spaces. Spaces become hyphens, special chars are stripped, accented chars normalize to ASCII. Both import (inferSlug) and sync (pathToSlug) pipelines now use the same slugifyPath() function, eliminating the case-preservation mismatch. * feat: one-time migration to slugify existing page slugs Extends Migration interface with optional TypeScript handler for application-level data transformations. Adds version 2 migration that renames all existing slugs to their slugified form, including link rewriting. Collision handling via try/catch + warning. * test: slugify unit tests, E2E tests, and updated expectations 22 new unit tests for slugifySegment and slugifyPath covering spaces, special chars, unicode, dots, empty segments, and all 4 bug report examples. Updated pathToSlug tests for new lowercase behavior. Updated E2E tests for slugified Apple Notes slugs. Added 2 new E2E tests for space-named file import and sync. * chore: bump version and changelog (v0.5.1) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: fix changelog example to show directory with spaces --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Garry Tan
|
e9f3c9c24d |
docs: live sync setup + verification runbook + API key loading (#24)
* docs: add SKILLPACK Section 18 — Live Sync (MUST ADD) Contract-first guide for keeping the vector DB in sync with the brain repo. Documents the pooler prerequisite (Session mode required for transactions), sync + embed primitives, four example approaches (cron, --watch, webhook, git hook), isSyncable exclusions, silent skip warning, and OpenClaw/Hermes cron registration examples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add GBRAIN_VERIFY.md installation verification runbook Six-check runbook: schema (doctor), skillpack loaded, auto-update, live sync (coverage check + embed check + end-to-end push-and-search test), embedding coverage, brain-first lookup protocol. Emphasizes "sync ran" != "sync worked" — the real test is searching for corrected text after a push. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add setup Phases H (Live Sync) and I (Verification) Phase H: MUST ADD live sync setup — pooler prerequisite check, automatic sync configuration (agent picks approach), sync+embed chaining, coverage verification. Phase I: run GBRAIN_VERIFY.md end-to-end before declaring setup complete. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add install steps 8-9 (live sync + verification) Step 8: set up automatic sync with SKILLPACK Section 18 reference. Step 9: run GBRAIN_VERIFY.md runbook. Add GBRAIN_VERIFY.md to docs section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add API key loading instructions to CLAUDE.md Source ~/.zshrc before running Tier 2 tests so OPENAI_API_KEY and ANTHROPIC_API_KEY are available. Without this, embedding and skills tests skip silently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version to v0.5.0 Live sync, verification runbook, API key loading instructions. Version markers updated in SKILLPACK and RECOMMENDED_SCHEMA. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add anti-hand-roll rule to skill routing in CLAUDE.md Explicitly prohibit manually running git commit + push + gh pr create when /ship is available. /ship handles VERSION, CHANGELOG, document-release, reviews, and coverage audit. Hand-rolling skips all of these. Added "commit and ship" / "push and ship" variants to the ship routing rule. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: changelog voice rule + rewrite 0.5.0 changelog to sell the upgrade CLAUDE.md: add changelog voice guidance — lead with benefits, not implementation details. Make users want to upgrade. CHANGELOG: rewrite 0.5.0 entries from dry feature descriptions to capability-focused bullets ("your brain never falls behind" not "SKILLPACK Section 18 added"). SKILLPACK Section 17: update the auto-update message template to instruct agents to sell the upgrade, not just summarize the diff. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add v0.5.0 migration directive for live sync + verification Agents upgrading from v0.4.x will automatically: check their pooler connection string, set up automatic sync, and run the verification runbook. Without this migration file, upgrading agents would learn about live sync (by re-reading Section 18) but wouldn't set it up. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: sharpen migration file guidance in CLAUDE.md Replace vague "requires agent action" with concrete trigger list: new setup steps existing users don't have, MUST ADD skillpack sections, schema changes, deprecated commands, new verification steps, new crons. Add the key test: "if an existing user upgrades and does nothing else, will their brain work worse?" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make Section 17 upgrade flow work for direct user requests Section 17 was structured as a cron-initiated flow only. An agent handling "upgrade gbrain" might just run the command and stop, missing the post-upgrade steps where the value is (re-read skills, run migrations, schema sync). Added explicit entry point for direct upgrade requests. Made Steps 2-4 more concrete about where to find files and why migrations can't be skipped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add E2E sync tests — git-to-DB pipeline (11 tests) Tests the full sync lifecycle against real Postgres+pgvector: - First sync imports all pages from a git repo - Second sync with no changes returns up_to_date - Incremental sync picks up new files (add → commit → sync → verify) - Incremental sync picks up modifications — THE CRITICAL TEST: corrected text appears in DB and keyword search after sync - Incremental sync handles deletes - Non-syncable files are excluded (README, .raw/, ops/) - Sync state (last_commit, last_run) persisted to config - Sync logged to ingest_log - --full reimports everything - --dry-run shows changes without applying Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: strengthen CLAUDE.md to always run ALL test tiers Replace passive "source zshrc" suggestion with ALWAYS directive. Explicitly state that "run all tests" means ALL tiers including Tier 2 with API keys. Do not skip Tier 2 just because keys need loading. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Tier 2 E2E tests — correct openclaw CLI invocation The tests used `openclaw -p` which doesn't exist. The correct command is `openclaw agent --local --agent <id> --message <prompt>`. Also fixed JSON output parsing (structured JSON goes to stderr, not stdout — use non-JSON mode instead). Fixed ingest test to assert on agent response text rather than test DB state (the agent writes to its own configured DB, not the ephemeral test DB). 82 tests pass, 0 fail, 0 skip across all 5 E2E files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
f541f045d2 |
feat: add gbrain check-update command and auto-update agent workflow (#15)
* feat: add `gbrain check-update` command for auto-update notifications Deterministic collector that checks GitHub Releases for new versions, compares semver (minor+ only, skips patches), and fetches changelog diffs. Exports `detectInstallMethod()` from upgrade.ts for reuse. Includes 15 unit tests covering version comparison, CLI wiring, and error handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add E2E upgrade tests against real GitHub API Exercises check-update CLI end-to-end: valid JSON output, human-readable mode, help text, graceful no-releases handling, and version comparison wiring. Skips gracefully when network is unavailable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add SKILLPACK Section 17 — auto-update notifications Full agent playbook for the update lifecycle: check, notify, consent, upgrade, skills refresh, schema sync, report. Includes standalone self-update for skillpack-only users via version markers and raw GitHub URL fetching. Adds version markers to both SKILLPACK and RECOMMENDED_SCHEMA headers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add auto-update step 7 to install paste, setup Phase G, migrations dir Adds step 7 to the OpenClaw install paste (default-on update checks). Setup skill gets Phase G (conditional offer for manual installs) and schema state tracking via ~/.gbrain/update-state.json. Creates skills/migrations/ directory for version-specific upgrade directives. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md with E2E test DB lifecycle, migration conventions Adds E2E test DB lifecycle instructions (spin up, run, tear down). Documents version migration convention (skills/migrations/v[version].md) and schema state tracking (~/.gbrain/update-state.json). Updates test file counts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: broken semver comparison in extractChangelogBetween The version range check compared minor versions without guarding on major being equal, causing incorrect changelog entries to be captured (e.g., v0.5.0 would match when upgrading from v1.2.0). Extracted semverGt/semverLte helpers for correct comparisons. Added 5 tests for extractChangelogBetween covering cross-major, same-version, and malformed input cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.4.1) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
912a321cfa |
GBrain v0.4.0 — production agent documentation + reference architecture (#10)
* fix: widen validateSlug to accept any filename characters Git is the system of record. Slugs are lowercased repo-relative paths. The restrictive regex rejected spaces, parens, and special chars, blocking 5,861 Apple Notes files from importing. Now only rejects empty slugs, path traversal (..), and leading slash. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: enable RLS on all tables with BYPASSRLS safety check Without RLS, the Supabase anon key gives full read access to the DB. Enable RLS on all 10 tables with no policies — the postgres role (used by gbrain via pooler) has BYPASSRLS and is unaffected. Only enables if the current role actually has BYPASSRLS privilege to avoid locking ourselves out on non-Supabase setups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: import resilience — 5MB limit, error suppression, structured progress Raise MAX_FILE_SIZE from 1MB to 5MB for Apple Notes with attachments. Track error patterns and suppress after 5 identical errors to prevent 5,861 identical warnings from killing the agent process. Replace \r progress bar with structured log lines (rate, ETA) for agent parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: init detects IPv6-only Supabase URLs, adds pgvector check Detect db.*.supabase.co direct URLs and warn about IPv6 failure. On ECONNREFUSED/ETIMEDOUT to Supabase, suggest the Session pooler connection string with exact dashboard click path. Check for pgvector extension after connecting and fail with clear instructions if missing. Update wizard hints to show pooler URL format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add pre-ship requirement for E2E tests E2E tests against real Postgres+pgvector must pass before /ship or /review. Adds the requirement to CLAUDE.md so all agents enforce it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: parallel import with per-worker engine instances Refactor PostgresEngine to support instance-level DB connections instead of only the module-global singleton. Each worker gets its own connection with poolSize:2 (vs 10 for the main engine), so 8 workers = 16 connections. Add --workers N flag to gbrain import. Workers pull from a shared queue and use independent engine instances — no transaction context corruption. The bottleneck is network round-trips to Supabase (one per page upsert). Parallel workers cut import time proportionally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: automatic schema migration runner Migrations are embedded as string constants in migrate.ts (survives Bun --compile). Each migration runs in a transaction for clean rollback on failure. Runs automatically on initSchema() — no manual step needed when a user updates the gbrain binary against an older DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: pluggable storage backend (S3 + Supabase Storage + local) Add StorageBackend interface with three implementations: - S3Storage: works with AWS S3, Cloudflare R2, MinIO (any S3-compatible) - SupabaseStorage: uses Supabase Storage REST API with service role key - LocalStorage: filesystem-based, for testing Add file-resolver.ts with fallback chain: local file → .redirect breadcrumb → .supabase marker → storage backend. Supports the three-stage migration (mirror → redirect → clean). Add yaml-lite.ts for parsing marker and breadcrumb files without adding a YAML dependency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: gbrain doctor command — health checks with --json output Checks: connection, pgvector extension, RLS on all tables, schema version, embedding coverage. Outputs structured JSON with --json flag for agent parsing. Exit code 0 if healthy, 1 if issues found. Agents should run gbrain doctor --json when any command fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite setup skill + README for agent-first DX Setup skill: add Why Supabase, step-by-step project creation, explicit agent instructions (nohup for large imports, doctor on failure, don't ask for anon key), available init flags, file migration offer after first import. Remove ClawHub references. README: simplify to single OpenClaw install path, remove ClawHub, fix squatted npm name to github:garrytan/gbrain, add Supabase settings note about Session pooler. Add Apple Notes test fixtures with spaces and parens in filenames for E2E testing of the slug fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add RLS verification, schema health, and nohup hints to maintain skill Maintenance skill now checks RLS status and schema version as part of periodic health checks. Adds nohup pattern for large embedding refreshes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: import resume checkpoint + Supabase smart URL parsing Import resume: saves checkpoint every 100 files to ~/.gbrain/import-checkpoint.json. On restart with same directory and file count, skips already-processed files. Use --fresh to ignore checkpoint and start over. Cleared on successful completion. Supabase admin: extractProjectRef() parses any Supabase URL format (dashboard, direct, pooler, project URL) to extract the project ref. discoverPoolerUrl() uses the Management API to find the correct pooler connection string (including the exact region prefix). checkRls() verifies RLS status via the API. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add 56 unit tests for all new code 8 new test files covering every feature added in this branch: - slug-validation.test.ts: spaces, parens, unicode, path traversal (10 tests) - yaml-lite.test.ts: parse + stringify, marker/redirect formats (9 tests) - supabase-admin.test.ts: extractProjectRef for 4 URL formats (7 tests) - migrate.test.ts: version export, runMigrations callable (2 tests) - storage.test.ts: LocalStorage CRUD + createStorage factory (14 tests) - file-resolver.test.ts: fallback chain, redirect, marker parsing (6 tests) - import-resume.test.ts: checkpoint save/load/resume/fresh (6 tests) - doctor.test.ts: module export, CLI registration (3 tests) Total: 184 pass, 0 fail (up from 128). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: bulk chunk INSERT + E2E tests for all new features Bulk INSERT: upsertChunks now builds a multi-row VALUES query instead of inserting chunks one-by-one. Reduces DB round-trips by ~50x per page. E2E tests added to mechanical.test.ts: - Slug with special chars: import Apple Notes fixtures with spaces/parens, verify search finds them, verify idempotency - RLS verification: check pg_tables.rowsecurity on all tables, verify current user has BYPASSRLS - Doctor command: verify exit 0 on healthy DB, --json produces valid JSON with check structure - Parallel import: --workers 2 produces same page count as sequential Unit tests added: - setup-branching.test.ts: IPv6 detection, defaultWorkers auto-tuning, smart URL parsing across all Supabase URL formats Fixtures added: - large/big-file.md (2.1MB) for testing raised file size limit - apple-notes/ fixtures already existed Total: 200 pass, 0 fail (up from 184). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: --json on init/import, file migration CLI, lifecycle tests --json flag: init and import now support --json for structured output. Agents get parseable JSON instead of human-readable text. File migration CLI: implement mirror, unmirror, redirect, restore, clean, and status subcommands for the three-stage file migration lifecycle (local → mirrored → redirected → cloud-only). File migration tests: full lifecycle test covering every transition in the state machine (LOCAL → MIRROR → UNMIRROR → REDIRECT → RESTORE → CLEAN), including edge cases and file resolver at each stage. Bulk chunk INSERT: upsertChunks now builds multi-row parameterized VALUES query, reducing round-trips per page from ~50 to 1. Total: 207 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: thorough E2E tests for parallel import concurrency Replace the weak single-comparison parallel import test with 7 tests: - Sequential baseline: capture page count, chunk count, and all slugs - --workers 2: verify page count matches sequential - Chunk count matches (no duplicates from concurrent writes) - Page slugs match exactly - No duplicate pages (SQL GROUP BY HAVING count > 1) - No duplicate chunks (SQL GROUP BY page_id, chunk_index) - --workers 4: also works correctly - Re-import with workers is idempotent These tests catch the exact bug Codex found (db.ts singleton causing concurrent transaction corruption) by verifying data integrity after parallel writes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add batch embedding queue as P1 TODO Deferred during eng review (per-worker embedding is good enough for now). Revisit after profiling real imports to confirm embedding is the bottleneck. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: E2E test failures — fixture counts, arg parsing, doctor exit code Fix fixture count assertions: 13 → 16 pages (added apple-notes + large file), companies 2 → 3 (ohmygreen), concepts 3 → 5 (notes, big-file). Fix --workers arg parsing: the worker count value (e.g. "2") was being picked up as the directory arg. Skip flag values when finding the dir. Fix doctor exit code: warnings (like missing embeddings) should exit 0, only actual failures exit 1. E2E tests import with --no-embed, so embeddings are always WARN. Fix E2E CLI tests: add initCli() before doctor and parallel import tests so ~/.gbrain/config.json exists for the subprocess. All E2E tests pass: 63 pass, 0 fail. All unit tests pass: 207 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.4.0 New CHANGELOG entry for all post-0.3.0 features (doctor, storage backends, parallel import, resume checkpoints, RLS, schema migrations, --json output). Version bumped 0.3.0 → 0.4.0 across all manifests. CLAUDE.md: test count 9→19, skill count 8→7, added key files. CONTRIBUTING.md: fixture count 13→16, added missing source files. README.md: added gbrain doctor to commands, fixed stale welcome PRs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add GBRAIN_SKILLPACK.md reference architecture Production agent patterns from a real deployment with 14,700+ brain files. Covers: entity detection on every message, brain-first lookup protocol, 7-step enrichment pipeline with tiered API spend, compiled truth + timeline, source attribution with mandatory citations, meeting ingestion with entity propagation, cron schedule with quiet hours and travel-aware timezone, YouTube/media ingestion via Diarize.io, integration guides for ClawVisor, Circleback webhooks, and Quo/OpenPhone SMS. Opens with the Vannevar Bush memex framing and the originals folder for capturing intellectual capital. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite README opener with memex pitch and production architecture Replace code-first opener with mimetic-desire pitch: Vannevar Bush memex tagline, production brain numbers (10K+ files, 3K+ people, 13 years of calendar), "ask it anything" examples, compounding thesis. New sections: The Compounding Thesis (read-write loop), Architecture (three-column diagram), What a Production Agent Looks Like (SKILLPACK reference), How gbrain fits with OpenClaw (three-layer complement). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update skills with brain-first lookup, entity detection, heartbeat setup: Phase D rewritten with brain-first lookup protocol (gbrain search → query → get → grep fallback), sync-after-write rule, memory_search complement table. query: token-budget awareness (chunks not full pages), source precedence hierarchy (user > compiled truth > timeline > external). ingest: entity detection on every message (scan, check brain, create or enrich, commit and sync). maintain: heartbeat integration (doctor, embed --stale, sync verification, stale compiled truth detection). briefing: gbrain-native context loading (search attendees before meetings, search sender before email, daily deal/meeting/commitment queries). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add OpenClaw positioning to README opener Make it clear up top that GBrain is built for OpenClaw agents and works with any OpenClaw deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: credit Karpathy's Knowledge LLM vision, add origin story GBrain started as Karpathy's LLM wiki idea built for real. Worked great until the brain hit thousands of files and grep fell apart. GBrain is the search layer that had to exist once the brain outgrew grep. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
a86f995883 |
feat: GBrain v0.3.0 — contract-first architecture + ClawHub plugin (#7)
* feat: contract-first operations.ts with OperationError, dry_run, importFromContent 30 shared operations as single source of truth for CLI and MCP. - OperationError with typed error codes (page_not_found, invalid_params, etc.) - dry_run support on all mutating operations - importFromContent split from importFile with transaction wrapping - Idempotency hash now includes ALL fields (title, type, frontmatter, tags) - Config env var fallback: GBRAIN_DATABASE_URL > DATABASE_URL > config file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: rewrite MCP server + CLI + tools-json from operations server.ts: 233 -> ~80 lines. Tool definitions and dispatch generated from operations[]. cli.ts: shared operations auto-registered, CLI-only commands kept as manual dispatch. tools-json: generated FROM operations[], eliminating the third contract surface. Parity test verifies structural contract between operations, CLI, and MCP. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: delete 12 command files migrated to operations.ts Handler logic for get, put, delete, list, search, query, health, stats, tags, link, timeline, and version now lives in operations.ts. Kept: init, upgrade, import, export, files, embed, sync, serve, call, config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: init --non-interactive, upgrade verification, schema migration - gbrain init --non-interactive --url <url> for plugin mode (no TTY required) - Post-upgrade version verification in gbrain upgrade - Drop storage_url from files table (storage_path is the only identifier) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: tool-agnostic skills + new setup skill All 7 skills rewritten with intent-based language instead of CLI commands. Works with both CLI and MCP plugin contexts. New setup skill replaces install: auto-provision Supabase via CLI, AGENTS.md injection, target TTHW < 2 min. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: ClawHub bundle plugin, CI workflows, v0.3.0 - openclaw.plugin.json with configSchema, MCP server config, skill listing - GitHub Actions: test on push/PR, multi-platform release (macOS arm64 + Linux x64) - Version bump 0.3.0, CHANGELOG, README ClawHub section, CLAUDE.md updated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: idempotency hash mismatch + MCP dry_run passthrough importFromContent now passes its all-fields hash through putPage via content_hash on PageInput, so the stored hash matches the computed hash. Previously the skip-if-unchanged check never fired because the hash formulas differed. MCP server now passes dry_run from tool params to OperationContext. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.3.0.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: schema loader handles PL/pgSQL $$ blocks Delete the semicolon-based SQL splitter in db.ts which broke on PL/pgSQL trigger functions containing semicolons inside $$ delimiter blocks. Use single conn.unsafe(schemaSql) call instead — the postgres driver handles multi-statement SQL natively. schema.sql already uses IF NOT EXISTS / CREATE OR REPLACE for idempotency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: E2E test infrastructure + realistic brain fixtures Add test infrastructure for running E2E tests against real Postgres+pgvector. Includes: - test/e2e/helpers.ts: DB lifecycle, fixture import, timing, diagnostics - 13 fixture files as a miniature realistic brain (people, companies, deals, meetings, concepts, projects, sources) following the compiled truth + timeline format from GBRAIN_RECOMMENDED_SCHEMA.md - docker-compose.test.yml: local pgvector convenience (port 5433) - .env.testing.example: template for test credentials - package.json: add test:e2e script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: E2E test suites + CI workflow Tier 1 (mechanical.test.ts): 14 test suites covering all operations against real Postgres — page CRUD, search with quality scoring, links, tags, timeline, versions, admin, chunks, resolution, ingest log, raw data, files, idempotency stress, setup journey (full CLI flow), init edge cases, schema idempotency, schema diff guard, performance baselines. Tier 1 (mcp.test.ts): MCP protocol test — spawns server, sends JSON-RPC, verifies tools/list matches operations count. Tier 2 (skills.test.ts): OpenClaw skill tests — ingest, query, health. Skips gracefully when dependencies missing. CI (.github/workflows/e2e.yml): Tier 1 on every PR (pgvector service), Tier 2 nightly/manual with API key secrets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: E2E test fixes + traverseGraph jsonb cast - Fix traverseGraph query: cast json_agg to jsonb_agg so SELECT DISTINCT works - Fix put_page tests to use importFromContent with noEmbed (no OpenAI key in Tier 1) - Fix get_health assertion (page_count not total_pages) - Fix raw_data test to handle JSONB string/object return - Simplify MCP test to verify tool generation directly - Add timeouts to CLI subprocess tests - Use port 5434 for docker-compose (5433 often in use) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: update all project docs for E2E test suite - CLAUDE.md: updated test count (9 unit + 3 E2E), added E2E test instructions, fixed skill count to 8 - CONTRIBUTING.md: updated project structure with test/e2e/, added E2E test instructions, rewrote "Adding a new command" to reflect contract-first architecture (add to operations.ts, done) - README.md: fixed table count (10 not 9), added recommended schema doc to Docs section, added E2E instructions to Contributing section - CHANGELOG.md: added E2E test suite, docker-compose, schema loader fix, and traverseGraph jsonb fix to v0.3.0 entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
ecebd5552a |
feat: GBrain v0.2.0 — incremental sync, file storage, install skill (#2)
* refactor: extract importFile from import.ts + add tag reconciliation Shared single-file import function used by both import and sync. Adds tag reconciliation (removes stale tags on reimport), >1MB file skip, and import->sync checkpoint continuity (writes git HEAD to config table after import so sync picks up seamlessly). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add sync pure functions, updateSlug engine method, and sync tests - buildSyncManifest: parses git diff --name-status -M output - isSyncable: filters to .md pages, excludes hidden/ops/.raw/skip-list - pathToSlug: converts file paths to page slugs with optional prefix - updateSlug: renames page slug in-place (preserves page_id, chunks, embeddings) - rewriteLinks: stub for v0.2 (FKs use page_id, already correct) - 20 new tests, all passing (39 total across 3 files) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain sync command with CLI, MCP, and watch mode 18-step sync protocol: read config, git pull, ancestry validation, git diff --name-status -M for net changes, isSyncable filter, process deletes/renames/adds/modifies via importFile, batch optimization, sync state checkpoint in Postgres config table. Watch mode with polling and consecutive error counter. MCP sync_brain tool returns structured SyncResult. Stale page deletion for un-syncable files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add files table, gbrain files commands, and config show redaction - files table: page_slug FK with ON DELETE SET NULL + ON UPDATE CASCADE, storage_path, storage_url, mime_type, content_hash for dedup - gbrain files list/upload/sync/verify commands for Supabase Storage - gbrain config show redacts postgresql:// passwords and secret keys - CLI help updated with FILES section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add install skill for GBrain onboarding 6-phase install workflow: environment discovery, Supabase setup (magic path via CLI OAuth or fallback 2-copy-paste), init + import, ongoing sync cron, optional file migration with mandatory verification, and agent teaching (AGENTS.md rules). Every error gets what + why + fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.2.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add v0.2 features to README (sync, files, install skill) README.md: added sync command to IMPORT/EXPORT section, added FILES section with 4 commands, added files table to schema diagram, added install skill to skills table, updated MCP tools count from 20 to 21 (sync_brain added). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: OpenClaw DX improvements (skill count, upgrade docs, config show help) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: consolidate version to single source of truth Create src/version.ts that reads from package.json via static import (safe for bun compiled binaries). Update mcp/server.ts from hardcoded '0.1.0' to use shared VERSION. Bump skills/manifest.json to 0.2.0. * fix: upgrade detection order, npm→bun naming, clawhub false positives Reorder detection: node_modules first, binary second, clawhub last. Rename 'npm' install method to 'bun'. Use 'clawhub --version' instead of 'which clawhub' to avoid false positives from dangling symlinks. Add 120s timeout to execSync calls to prevent hanging. Add --help flag. * feat: per-command --help, unknown command check before DB connection Add COMMAND_HELP map covering all 28 commands. Check --help before init/upgrade dispatch and before connectEngine() so help works without a database. Use COMMAND_HELP keys as known-command set to catch unknown commands before wasting a DB round-trip. * docs: standardize npm references to bun, add Upgrade section to README Fix init.ts: npx→bunx, npm→bun for supabase CLI guidance. Fix README: npm install→bun add for standalone CLI install. Add ## Upgrade section to README with all three install methods. Update install skill Upgrading section to list bun, ClawHub, and binary. * test: full coverage audit — CLI dispatch, upgrade detection, config, edge cases New test files: - test/cli.test.ts: COMMAND_HELP ↔ switch consistency, version from package.json, per-command --help, unknown command handling, global help - test/upgrade.test.ts: detection order verification, npm→bun naming, clawhub --version (not which), timeout presence - test/config.test.ts: redactUrl for postgresql URLs, edge cases Extended existing tests: - test/sync.test.ts: empty string pathToSlug, uppercase .MD rejection, deeply nested files, multiple renames, unknown status codes - test/markdown.test.ts: multiple --- separators, missing frontmatter, no frontmatter at all, empty string, type inference from paths Tests: 39 → 83 (+44 new). All pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 100% coverage — import-file mock engine, files utils, chunker edge cases New test files: - test/import-file.test.ts (9 tests): mock BrainEngine to test importFile without DB — MAX_FILE_SIZE skip, content_hash dedup, tag reconciliation (remove stale + add new), compiled_truth/timeline chunking, noEmbed flag, sequential chunk_index - test/files.test.ts (22 tests): getMimeType for all extensions + uppercase + unknown + no-extension, fileHash consistency + different content + empty, collectFiles pattern (skip .md, skip hidden dirs, recurse, sorted output) Extended: - test/chunkers/recursive.test.ts (+6 tests): single newline splits, word-only text, clause delimiters, lossless preservation, default options, mixed delimiter hierarchy Tests: 83 → 118 (+35 new). All pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Garry Tan
|
b22cbd349a |
feat: GBrain v0.1.0 — Postgres-native personal knowledge brain (#1)
* chore: add CLAUDE.md with project context and gstack skill routing rules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: initialize project with Bun + TypeScript package.json with dependencies (postgres, pgvector, openai, anthropic, MCP SDK, gray-matter). TypeScript config targeting ESNext with bundler module resolution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add foundation layer — engine interface, Postgres engine, schema BrainEngine pluggable interface with full PostgresEngine: CRUD, search (keyword + vector), links, tags, timeline, versions, stats, health, ingest log, config. Trigger-based tsvector spanning pages + timeline_entries. Markdown parser with frontmatter, compiled_truth / timeline splitting, and round-trip serialization. 19 tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 3-tier chunking and embedding service Recursive delimiter-aware chunker (5-level hierarchy, 300-word chunks, 50-word overlap). Semantic chunker with Savitzky-Golay boundary detection and recursive fallback. LLM-guided chunker via Claude Haiku with sliding window topic detection. OpenAI embedding service with batch support, exponential backoff, and rate limit handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add hybrid search with RRF fusion, expansion, and 4-layer dedup Hybrid search merges vector (pgvector HNSW) + keyword (tsvector) via Reciprocal Rank Fusion. Multi-query expansion via Claude Haiku generates 2 alternative phrasings. 4-layer dedup pipeline: by source, cosine similarity, type diversity (60% cap), per-page cap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add GBRAIN_V0 spec, pluggable engine architecture, SQLite engine plan GBRAIN_V0.md: full product spec with architecture decisions, CLI commands, schema, search architecture, chunking strategies, first-time experience, and future plans. ENGINES.md: pluggable engine interface, capability matrix, how to add new backends. SQLITE_ENGINE.md: complete SQLite implementation plan with schema, FTS5 setup, vector search options, and contributor guide. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add CLI with all commands Full CLI dispatcher with 25+ commands: init (Supabase wizard), get, put, delete, list, search, query (hybrid RRF), import (bulk with progress bar), export (round-trip), embed, stats, health, tag/untag/tags, link/unlink/ backlinks/graph, timeline/timeline-add, history/revert, config, upgrade, serve, call. Smart slug resolution on reads. Version snapshots on updates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add MCP stdio server with all brain tools 20 MCP tools mirroring CLI operations: get/put/delete/list pages, search (keyword), query (hybrid RRF + expansion), tags, links with graph traversal, timeline, stats, health, version history, and revert. Auto-chunks and embeds on put_page. CLI and MCP share the same engine. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 6 skill files and ClawHub manifest Fat markdown skills for AI agents: ingest (meetings/docs/articles with timeline merge), query (3-layer search + synthesis + citations), maintain (health checks, stale detection, orphan audit), enrich (external API enrichment), briefing (daily briefing compilation), migrate (universal migration from Obsidian/Notion/Logseq/markdown/CSV/JSON/Roam). ClawHub manifest for skill distribution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add README, CONTRIBUTING, update CLAUDE.md test references README with quickstart, commands, architecture, library usage, MCP setup, and links to design docs. CONTRIBUTING with setup, project structure, and guides for adding commands and engines. CLAUDE.md updated to reference actual test files instead of planned-but-unwritten import test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address adversarial review findings — 5 critical/high fixes - revertToVersion: add page_id check to prevent cross-page data corruption - traverseGraph: use UNION instead of UNION ALL for cycle safety - embedAll: preserve all chunks when embedding stale subset only - embedding: throw on retry exhaustion instead of returning zero vectors - putPage: validate slugs to prevent path traversal on export Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.1.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: expand README with schema, install, search architecture, and motivation Why it exists, how search works (with ASCII diagram), full database schema with all 9 tables and index details, chunking strategies explained, storage estimates, setup wizard walkthrough, knowledge model with example page, library usage with more examples, expanded skills table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: add MIT license (Copyright 2026 Garry Tan) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add OpenClaw install flow as primary option in README OpenClaw users just say "install gbrain" and the orchestrator handles everything: package install, Supabase setup wizard, skill registration. Shows the conversational interface for querying, ingesting, and briefings. ClawHub and standalone CLI paths follow as alternatives. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add prerequisites and explicit OpenClaw install instructions Prerequisites table listing Supabase, OpenAI, and Anthropic dependencies with links. Environment variable setup. Explicit step-by-step prompt for OpenClaw users showing exactly what to tell the orchestrator. Note that search degrades gracefully without API keys (keyword-only without OpenAI, no expansion without Anthropic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: scrub named references, add PG essay demo section to README Replace all Pedro/Brex/Jensen Huang/River AI examples with Paul Graham essay examples using the kindling corpus. Add "Try it" section to README showing the power of hybrid search on PG essays in 90 seconds. Update test fixtures to use concept pages instead of person pages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |