6c7d2ed30b
* refactor: extract shared utils, add runMigration + getChunksWithEmbeddings to BrainEngine Extract validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult from postgres-engine.ts into shared utils.ts. Add rowToChunk includeEmbedding parameter for migration support. Add two new methods to BrainEngine interface: - runMigration(version, sql) — replaces internal eng.sql access in migrate.ts - getChunksWithEmbeddings(slug) — returns chunks with embedding data for migration Replace 'sqlite' with 'pglite' in EngineConfig and GBrainConfig types. Fix loadConfig to infer engine from database_path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: pluggable engine factory + hybridSearch keyword-only fallback Add createEngine() factory with dynamic imports so PGLite WASM is never loaded for Postgres users. Wire CLI to use factory instead of hardcoded PostgresEngine. Force workers=1 for PGLite imports (single-connection architecture). Fix hybridSearch to check OPENAI_API_KEY before calling embed(). When unset, returns keyword-only results instead of throwing. Critical for local PGLite users who don't need vector search. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: PGLiteEngine — embedded Postgres 17.5 via WASM, same SQL everywhere Full BrainEngine implementation (37 methods) using @electric-sql/pglite. Same SQL as PostgresEngine — tsvector triggers, pgvector HNSW, pg_trgm fuzzy matching, recursive CTEs, JSONB. Only the driver call syntax differs (parameterized queries instead of tagged templates). PGLite schema is the Postgres schema minus RLS, advisory locks, and remote auth tables (access_tokens, mcp_request_log, files). No server. No subscription. One directory. Works offline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: smart init (PGLite default) + bidirectional engine migration gbrain init now defaults to PGLite — brain ready in 2 seconds, no server needed. Scans target directory: <1000 .md files = PGLite, >=1000 = suggests Supabase. --supabase and --pglite flags override. gbrain migrate --to supabase/pglite transfers all data between engines with manifest-based resume. Copies pages, chunks (with embeddings), tags, timeline, raw data, links, and config. --force overwrites non-empty target. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 60 new tests for PGLite engine, utils, and factory 41 PGLite engine tests covering all 37 BrainEngine methods: CRUD, tsvector keyword search, pg_trgm fuzzy matching, chunk upsert with COALESCE, graph traversal via recursive CTE, transactions, cascade deletes, stats/health, and embedding round-trip. 14 shared utility tests (validateSlug, contentHash, row mappers). 5 engine factory tests (dispatch, error messages). All run in-memory — zero Docker, zero DATABASE_URL, instant in CI. Add P0 TODO: submit Bun PR for WASM embedding in bun build --compile (oven-sh/bun#15032). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.7.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.7.0 PGLite engine - CLAUDE.md: add PGLite key files, update architecture, add migrate command, add 3 test files - README.md: PGLite as default init, zero-config getting started, migration path to Supabase - docs/ENGINES.md: PGLiteEngine shipped (v0.7), capability matrix, migration docs - docs/SQLITE_ENGINE.md: marked superseded by PGLite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove stale v0.4 README update prompt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove SQLITE_ENGINE.md (superseded by PGLite) PGLite uses the same SQL as Postgres, making a separate SQLite engine unnecessary. docs/ENGINES.md covers PGLiteEngine. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update README step 2 to default to PGLite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add schema setup step and install-all-integrations step to README Step 3 now tells agents to read GBRAIN_RECOMMENDED_SCHEMA.md and set up the MECE directory structure before importing. Step 7 tells agents to install every available integration recipe, not just list them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update install goal to match full opinionated setup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add 'Need an AI agent first?' section with one-click deploy links New users who don't have OpenClaw or Hermes Agent get pointed to AlphaClaw on Render and the Hermes Agent Railway template. One click each. Claude Code mentioned for users who already have it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add migrate to CLI_ONLY + help output, fix standalone example - migrate command was missing from CLI_ONLY set (errored as "Unknown command") - migrate now shows in --help under SETUP - init help line shows --pglite flag - standalone CLI example uses gbrain init (not --supabase) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: set realistic time expectation (~30 min to working brain) DB is 2 seconds. But schema + import + embeddings + integrations is 15-30 minutes. The agent does the work, you answer API key questions. Don't oversell time-to-value. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix AlphaClaw Render requirement (8GB+ RAM, not free tier) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: final README polish for launch - GOAL line: "Garry Tan's exact setup" (not Claude Code specific) - Remove markdown links from code block (won't render) - STEP 2 renamed from "START HERE" to "DATABASE" - Tighten Supabase fallback text Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: remove duplicate old install block from README The v0.5-era "With OpenClaw or Hermes Agent" paste block was superseded by the top-level "Start here" block. Having both confused users and the old one still said --supabase as step 2. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clean up README consistency and remove duplicated content - Remove duplicate "Try it" section (old 4-act walkthrough that repeated the install flow and contradicted "~30 min" with "90 sec") - Remove duplicate Setup section (third repetition of gbrain init) - Fix brain.db → brain.pglite (actual default path) - Fix "coming in v0.7" → "not yet implemented" (we ARE v0.7) - Remove "You don't need Postgres" (confusing since PGLite IS Postgres) - Deduplicate "competitive dynamics" query (appeared 3 times) - Collapse redundant standalone CLI section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6.2 KiB
TODOS
P1
Batch embedding queue across files
What: Shared embedding queue that collects chunks from all parallel import workers and flushes to OpenAI in batches of 100, instead of each worker batching independently.
Why: With 4 workers importing files that average 5 chunks each, you get 4 concurrent OpenAI API calls with small batches (5-10 chunks). A shared queue would batch 100 chunks across workers into one API call, cutting embedding cost and latency roughly in half.
Pros: Fewer API calls (500 chunks = 5 calls instead of ~100), lower cost, faster embedding.
Cons: Adds coordination complexity: backpressure when queue is full, error attribution back to source file, worker pausing. Medium implementation effort.
Context: Deferred during eng review because per-worker embedding is simpler and the parallel workers themselves are the bigger speed win (network round-trips). Revisit after profiling real import workloads to confirm embedding is actually the bottleneck. If most imports use --no-embed, this matters less.
Implementation sketch: src/core/embedding-queue.ts with a Promise-based semaphore. Workers await queue.submit(chunks) which resolves when the queue has room. Queue flushes to OpenAI in batches of 100 with max 2-3 concurrent API calls. Track source file per chunk for error propagation.
Depends on: Part 5 (parallel import with per-worker engines) -- already shipped.
P0
Fix bun build --compile WASM embedding for PGLite
What: Submit PR to oven-sh/bun fixing WASM file embedding in bun build --compile (issue oven-sh/bun#15032).
Why: PGLite's WASM files (~3MB) can't be embedded in the compiled binary. Users who install via bun install -g gbrain are fine (WASM resolves from node_modules), but the compiled binary can't use PGLite. Jarred Sumner (Bun founder, YC W22) would likely be receptive.
Pros: Single-binary distribution includes PGLite. No sidecar files needed.
Cons: Requires understanding Bun's bundler internals. May be a large PR.
Context: Issue has been open since Nov 2024. The root cause is that bun build --compile generates virtual filesystem paths (/$bunfs/root/...) that PGLite can't resolve. Multiple users have reported this. A fix would benefit any WASM-dependent package, not just PGLite.
Depends on: PGLite engine shipping (to have a real use case for the PR).
ChatGPT MCP support (OAuth 2.1)
What: Add OAuth 2.1 with Dynamic Client Registration to the Edge Function so ChatGPT can connect.
Why: ChatGPT requires OAuth 2.1 for MCP connectors. Bearer token auth is NOT supported. This is the only major AI client that can't use GBrain remotely.
Pros: Completes the "every AI client" promise. ChatGPT has the largest user base.
Cons: OAuth 2.1 is a significant implementation: authorization endpoint, token endpoint, PKCE flow, dynamic client registration. Estimated CC: ~3-4 hours.
Context: Discovered during DX review (2026-04-10). All other clients (Claude Desktop/Code/Cowork, Perplexity) work with bearer tokens. See docs/mcp/CHATGPT.md for current status.
Depends on: v0.6.0 remote MCP server (shipped).
P1 (new from v0.7.0)
Constrained health_check DSL for third-party recipes
What: Replace shell command health_checks with a typed DSL: {type: "env_exists", name: "KEY"}, {type: "url_responds", url: "..."}, {type: "heartbeat_fresh", max_age: "24h"}.
Why: Shell commands in recipe frontmatter = arbitrary code execution from markdown. Currently trusted because recipes are first-party only. This DSL is the mandatory gate before opening community recipe submissions.
Pros: Eliminates RCE risk from third-party recipes. Health checks become machine-parseable.
Cons: Less flexible than shell commands for novel checks. Need to define enough check types to cover common cases.
Context: From CEO review + Codex outside voice (2026-04-11). User approved shell commands for first-party but explicitly requested constrained DSL before third-party recipes.
Depends on: v0.7.0 recipe format (shipped).
P2
Community recipe submission (gbrain integrations submit)
What: Package a user's custom integration recipe as a PR to the GBrain repo. Validates frontmatter, checks constrained DSL health_checks, creates PR with template.
Why: Turns GBrain from "Garry's integrations" into a community ecosystem. The recipe format IS the contribution format.
Pros: Community-driven integration library. Users build Slack-to-brain, RSS-to-brain, Discord-to-brain.
Cons: Support burden. Need constrained DSL (P1) before accepting third-party recipes. Need review process for recipe quality.
Context: From CEO review (2026-04-11). User explicitly deferred due to bandwidth constraints. Target v0.9.0.
Depends on: Constrained health_check DSL (P1).
Always-on deployment recipes (Fly.io, Railway)
What: Alternative deployment recipes for voice-to-brain and future integrations that run on cloud servers instead of local + ngrok.
Why: ngrok free URLs are ephemeral (change on restart). Always-on deployment eliminates the watchdog complexity and gives a stable webhook URL.
Pros: Stable URLs, no ngrok dependency, production-grade uptime.
Cons: Costs $5-10/mo per integration. Requires cloud account.
Context: From DX review (2026-04-11). v0.7.0 ships local+ngrok as v1 deployment path.
Depends on: v0.7.0 recipe format (shipped).
Fly.io HTTP server as alternative deployment
What: Add gbrain serve --http and a Dockerfile/fly.toml for users who prefer a traditional server over Edge Functions.
Why: Avoids the Deno bundling seam. Bun runs natively. No 60s timeout. No cold start. Codex flagged the bundle strategy as "permanent maintenance tax."
Pros: Simpler code path, no edge-entry.ts needed, no Deno compat concerns. Supports sync_brain and file_upload remotely.
Cons: Users need a Fly.io account. Not zero-infra.
Context: From CEO review (2026-04-10). Edge Functions are the primary path. Fly.io is for power users who want full operation support remotely.
Depends on: v0.6.0 remote MCP server (shipped).
Completed
Implement AWS Signature V4 for S3 storage backend
Completed: v0.6.0 (2026-04-10) — replaced with @aws-sdk/client-s3 for proper SigV4 signing.