triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: GBrain v0.6.0 — Remote MCP Server + 12 Bug Fixes (#28)

Browse Source 
* fix: 7 bug fixes from Issue #9 and #22

- fix(mcp): use ListToolsRequestSchema/CallToolRequestSchema instead of string literals (Issue #9, PR #25)
- fix(mcp): handleToolCall reads dry_run from params instead of hardcoding false (#22 Bug #11)
- fix(search): keyword search returns best chunk per page via DISTINCT ON, not all chunks (#22 Bug #8)
- fix(search): dedup layer 1 keeps top 3 chunks per page instead of collapsing to 1 (#22 Bug #12)
- fix(engine): transaction uses scoped engine via Object.create, no shared state mutation (#22 Bug #2)
- fix(engine): upsertChunks uses UPSERT instead of DELETE+INSERT, preserves existing embeddings (#22 Bug #1)
- fix(slugs): validateSlug normalizes to lowercase, pathToSlug lowercases consistently (#22 Bug #4)
- schema: add unique index on content_chunks(page_id, chunk_index) for UPSERT support
- schema: add access_tokens and mcp_request_log tables via migration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: embed schema.sql at build time, remove fs dependency from initSchema

initSchema() previously read schema.sql from disk at runtime via readFileSync,
which broke in compiled Bun binaries and Deno Edge Functions. Now uses a
generated schema-embedded.ts constant (run `bun run build:schema` to regenerate).

- Removes fs and path imports from postgres-engine.ts and db.ts
- Adds scripts/build-schema.sh for one-source-of-truth generation
- Adds build:schema npm script

Fixes Issue #22 Bug #6.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 5 more bug fixes from Issue #22

- fix(file_upload): call storage.upload() in all 3 paths (operation, CLI upload, CLI sync) with rollback semantics (#22 Bug #9)
- fix(import): use atomic index counter for parallel queue instead of array.shift() race, preserve checkpoint on errors (#22 Bug #3)
- fix(s3): replace unsigned fetch with @aws-sdk/client-s3 for proper SigV4 auth, supports R2/MinIO via forcePathStyle (#22 Bug #10)
- fix(redirect): verify remote file exists before deleting local copy, skip files not found in storage (#22 Bug #5)
- deps: add @aws-sdk/client-s3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: remote MCP server via Supabase Edge Functions

Deploy GBrain as a serverless remote MCP endpoint on your existing Supabase
instance. One brain, accessible from Claude Desktop, Claude Code, Cowork,
Perplexity Computer, and any MCP client. Zero new infrastructure.

New files:
- supabase/functions/gbrain-mcp/index.ts — Edge Function with Hono + MCP SDK
- supabase/functions/gbrain-mcp/deno.json — Deno import map
- src/edge-entry.ts — curated bundle entry point (excludes fs-dependent modules)
- src/commands/auth.ts — standalone token management (create/list/revoke/test)
- scripts/deploy-remote.sh — one-script deployment
- .env.production.example — 3-value config template

Changes:
- config.ts: lazy-evaluate CONFIG_DIR (no homedir() at module scope)
- schema.sql: add access_tokens + mcp_request_log tables
- package.json: add build:edge script

Auth: bearer tokens via access_tokens table (SHA-256 hashed, per-client, revocable)
Transport: WebStandardStreamableHTTPServerTransport (stateless, Streamable HTTP)
Health: /health endpoint (unauth: 200/503, auth: postgres/pgvector/openai checks)
Excluded from remote: sync_brain, file_upload (may exceed 60s timeout)

Setup: clone, fill .env.production, run scripts/deploy-remote.sh, create token, done.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: per-client MCP setup guides

- docs/mcp/DEPLOY.md — deployment walkthrough, auth, troubleshooting, latency table
- docs/mcp/CLAUDE_CODE.md — claude mcp add command
- docs/mcp/CLAUDE_DESKTOP.md — Settings > Integrations (NOT JSON config!)
- docs/mcp/CLAUDE_COWORK.md — remote + local bridge paths
- docs/mcp/PERPLEXITY.md — Perplexity Computer connector setup
- docs/mcp/CHATGPT.md — coming soon (requires OAuth 2.1, P0 TODO)
- docs/mcp/ALTERNATIVES.md — Tailscale Funnel + ngrok self-hosted options

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.6.0)

GBrain v0.6.0: Remote MCP server via Supabase Edge Functions + 12 bug fixes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Remote MCP Server section to README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make document-release mandatory in CLAUDE.md, add MCP key files

Post-ship requirements section: document-release is NOT optional. Lists every
file that must be checked on every ship. A ship without updated docs is incomplete.

Also adds remote MCP server files to Key files section.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: batch upsertChunks into single statement to prevent deadlocks

The per-chunk UPSERT loop caused deadlocks under parallel workers because
each INSERT ON CONFLICT acquired row-level locks sequentially. Multiple
workers upserting different pages could deadlock on the shared unique index.

Fix: batch all chunks into a single multi-row INSERT ON CONFLICT statement.
One round-trip, one lock acquisition. COALESCE preserves existing embeddings
when the new value is NULL.

Fixes CI failure: "E2E: Parallel Import > parallel import with --workers 4"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: advisory lock in initSchema() prevents deadlock on concurrent DDL

When multiple processes call initSchema() concurrently (e.g., test setup +
CLI subprocess, or parallel workers during E2E tests), the schema SQL's
DROP TRIGGER + CREATE TRIGGER statements acquire AccessExclusiveLock on
different tables, causing deadlocks.

Fix: pg_advisory_lock(42) serializes all initSchema() calls within the
same database. The lock is session-scoped and released in a finally block.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add explicit test timeouts for CLI subprocess E2E tests

CLI subprocess tests (Setup Journey, Doctor Command, Parallel Import)
spawn `bun run src/cli.ts` which takes several seconds to JIT compile +
connect. The Bun test framework default 5000ms per-test timeout is too
tight for CI. Added 30-60s timeouts matching each subprocess's own
timeout to prevent false failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: infinite recursion in config.ts exported getConfigDir/getConfigPath

The replace_all refactor created recursive functions: the exported
getConfigDir() called the private getConfigDir() which called itself.
Renamed exports to configDir()/configPath() to avoid shadowing.

Also adds scripts/smoke-test-mcp.ts — verified all 8 MCP tool calls
work against a real Postgres database.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-10 15:23:00 -10:00
committed by GitHub
parent 27eb87f1f4
commit 3e21e9b69b
39 changed files with 2502 additions and 166 deletions
Download Patch File Download Diff File Expand all files Collapse all files

241
src/commands/auth.ts Normal file
View File

@@ -0,0 +1,241 @@
#!/usr/bin/env bun
/**
* GBrain token management — standalone script, no gbrain CLI dependency.
*
* Usage:
* DATABASE_URL=... bun run src/commands/auth.ts create "claude-desktop"
* DATABASE_URL=... bun run src/commands/auth.ts list
* DATABASE_URL=... bun run src/commands/auth.ts revoke "claude-desktop"
* DATABASE_URL=... bun run src/commands/auth.ts test <url> --token <token>
*/
import postgres from 'postgres';
import { createHash, randomBytes } from 'crypto';
const DATABASE_URL = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL;
if (!DATABASE_URL && process.argv[2] !== 'test') {
console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.');
process.exit(1);
}
function hashToken(token: string): string {
return createHash('sha256').update(token).digest('hex');
}
function generateToken(): string {
return 'gbrain_' + randomBytes(32).toString('hex');
}
async function create(name: string) {
if (!name) { console.error('Usage: auth create <name>'); process.exit(1); }
const sql = postgres(DATABASE_URL!);
const token = generateToken();
const hash = hashToken(token);
try {
await sql`
INSERT INTO access_tokens (name, token_hash)
VALUES (${name}, ${hash})
`;
console.log(`Token created for "${name}":\n`);
console.log(` ${token}\n`);
console.log('Save this token — it will not be shown again.');
console.log(`Revoke with: bun run src/commands/auth.ts revoke "${name}"`);
} catch (e: any) {
if (e.code === '23505') {
console.error(`A token named "${name}" already exists. Revoke it first or use a different name.`);
} else {
console.error('Error:', e.message);
}
process.exit(1);
} finally {
await sql.end();
}
}
async function list() {
const sql = postgres(DATABASE_URL!);
try {
const rows = await sql`
SELECT name, created_at, last_used_at, revoked_at
FROM access_tokens
ORDER BY created_at DESC
`;
if (rows.length === 0) {
console.log('No tokens found. Create one: bun run src/commands/auth.ts create "my-client"');
return;
}
console.log('Name Created Last Used Status');
console.log('─'.repeat(80));
for (const r of rows) {
const name = (r.name as string).padEnd(20);
const created = new Date(r.created_at as string).toISOString().slice(0, 19);
const lastUsed = r.last_used_at ? new Date(r.last_used_at as string).toISOString().slice(0, 19) : 'never'.padEnd(19);
const status = r.revoked_at ? 'REVOKED' : 'active';
console.log(`${name} ${created} ${lastUsed} ${status}`);
}
} finally {
await sql.end();
}
}
async function revoke(name: string) {
if (!name) { console.error('Usage: auth revoke <name>'); process.exit(1); }
const sql = postgres(DATABASE_URL!);
try {
const result = await sql`
UPDATE access_tokens SET revoked_at = now()
WHERE name = ${name} AND revoked_at IS NULL
`;
if (result.count === 0) {
console.error(`No active token found with name "${name}".`);
process.exit(1);
}
console.log(`Token "${name}" revoked.`);
} finally {
await sql.end();
}
}
async function test(url: string, token: string) {
if (!url || !token) {
console.error('Usage: auth test <url> --token <token>');
process.exit(1);
}
const startTime = Date.now();
console.log(`Testing MCP server at ${url}...\n`);
// Step 1: Initialize
try {
const initRes = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${token}`,
'Accept': 'application/json, text/event-stream',
},
body: JSON.stringify({
jsonrpc: '2.0',
method: 'initialize',
params: {
protocolVersion: '2025-03-26',
capabilities: {},
clientInfo: { name: 'gbrain-smoke-test', version: '1.0' },
},
id: 1,
}),
});
if (!initRes.ok) {
console.error(` Initialize failed: ${initRes.status} ${initRes.statusText}`);
const body = await initRes.text();
if (body) console.error(` ${body}`);
process.exit(1);
}
console.log(' ✓ Initialize handshake');
} catch (e: any) {
console.error(` ✗ Connection failed: ${e.message}`);
process.exit(1);
}
// Step 2: List tools
try {
const listRes = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${token}`,
'Accept': 'application/json, text/event-stream',
},
body: JSON.stringify({
jsonrpc: '2.0',
method: 'tools/list',
params: {},
id: 2,
}),
});
if (!listRes.ok) {
console.error(` ✗ tools/list failed: ${listRes.status}`);
process.exit(1);
}
const text = await listRes.text();
// Parse SSE or JSON response
let toolCount = 0;
if (text.includes('event:')) {
// SSE format: extract data lines
const dataLines = text.split('\n').filter(l => l.startsWith('data:'));
for (const line of dataLines) {
try {
const data = JSON.parse(line.slice(5));
if (data.result?.tools) toolCount = data.result.tools.length;
} catch { /* skip non-JSON lines */ }
}
} else {
try {
const data = JSON.parse(text);
toolCount = data.result?.tools?.length || 0;
} catch { /* parse error */ }
}
console.log(` ✓ tools/list: ${toolCount} tools available`);
} catch (e: any) {
console.error(` ✗ tools/list failed: ${e.message}`);
process.exit(1);
}
// Step 3: Call get_stats (real tool call)
try {
const statsRes = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${token}`,
'Accept': 'application/json, text/event-stream',
},
body: JSON.stringify({
jsonrpc: '2.0',
method: 'tools/call',
params: { name: 'get_stats', arguments: {} },
id: 3,
}),
});
if (!statsRes.ok) {
console.error(` ✗ get_stats failed: ${statsRes.status}`);
process.exit(1);
}
console.log(' ✓ get_stats: brain is responding');
} catch (e: any) {
console.error(` ✗ get_stats failed: ${e.message}`);
process.exit(1);
}
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
console.log(`\n🧠 Your brain is live! (${elapsed}s)`);
}
// CLI dispatch
const [cmd, ...args] = process.argv.slice(2);
switch (cmd) {
case 'create': await create(args[0]); break;
case 'list': await list(); break;
case 'revoke': await revoke(args[0]); break;
case 'test': {
const tokenIdx = args.indexOf('--token');
const url = args.find(a => !a.startsWith('--') && a !== args[tokenIdx + 1]);
const token = tokenIdx >= 0 ? args[tokenIdx + 1] : '';
await test(url || '', token || '');
break;
}
default:
console.log(`GBrain Token Management
Usage:
bun run src/commands/auth.ts create <name> Create a new access token
bun run src/commands/auth.ts list List all tokens
bun run src/commands/auth.ts revoke <name> Revoke a token
bun run src/commands/auth.ts test <url> --token <token> Smoke test a remote MCP server
`);
}

34
src/commands/files.ts
View File

@@ -131,6 +131,16 @@ async function uploadFile(args: string[]) {
return;
}
// Upload to storage backend if configured
const { loadConfig } = await import('../core/config.ts');
const config = loadConfig();
if (config?.storage) {
const { createStorage } = await import('../core/storage.ts');
const storage = await createStorage(config.storage as any);
const content = readFileSync(filePath);
await storage.upload(storagePath, content, mimeType || undefined);
}
await sql`
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${mimeType}, ${stat.size}, ${hash}, ${'{}'}::jsonb)
@@ -308,10 +318,31 @@ async function redirectFiles(args: string[]) {
return;
}
// Verify remote files exist before deleting locals
const { loadConfig } = await import('../core/config.ts');
const config = loadConfig();
let storage: any = null;
if (config?.storage) {
const { createStorage } = await import('../core/storage.ts');
storage = await createStorage(config.storage as any);
}
let redirected = 0;
let skippedMissing = 0;
for (const filePath of files) {
const relPath = relative(dir, filePath);
const hash = fileHash(filePath);
// Verify remote exists before deleting local
if (storage) {
const remoteExists = await storage.exists(relPath);
if (!remoteExists) {
console.error(` Skipping ${relPath}: not found in remote storage (would lose data)`);
skippedMissing++;
continue;
}
}
const breadcrumb = stringify({
moved_to: 'storage',
bucket: marker.bucket || 'brain-files',
@@ -325,6 +356,9 @@ async function redirectFiles(args: string[]) {
}
console.log(`Redirected ${redirected} files. Originals removed, breadcrumbs created.`);
if (skippedMissing > 0) {
console.log(`Skipped ${skippedMissing} files (not found in remote storage — run 'gbrain files mirror' first).`);
}
console.log('To undo: gbrain files restore <dir>');
}

22
src/commands/import.ts
View File

@@ -109,13 +109,15 @@ export async function runImport(engine: BrainEngine, args: string[]) {
processed++;
if (processed % 100 === 0 || processed === files.length) {
logProgress();
// Save checkpoint every 100 files
// Save checkpoint every 100 files — track completed file set, not just a counter
if (processed % 100 === 0) {
try {
const cpDir = join(homedir(), '.gbrain');
if (!existsSync(cpDir)) { const { mkdirSync } = await import('fs'); mkdirSync(cpDir, { recursive: true }); }
writeFileSync(checkpointPath, JSON.stringify({
dir, totalFiles: allFiles.length, processedIndex: resumeIndex + processed,
dir, totalFiles: allFiles.length,
processedIndex: resumeIndex + processed,
completedFiles: importedSlugs.length + skipped,
timestamp: new Date().toISOString(),
}));
} catch { /* non-fatal */ }
@@ -134,11 +136,13 @@ export async function runImport(engine: BrainEngine, args: string[]) {
})
);
const queue = [...files];
// Thread-safe queue: use an atomic index counter instead of array.shift()
let queueIndex = 0;
await Promise.all(workerEngines.map(async (eng) => {
while (queue.length > 0) {
const file = queue.shift()!;
await processFile(eng, file);
while (true) {
const idx = queueIndex++;
if (idx >= files.length) break;
await processFile(eng, files[idx]);
}
}));
@@ -157,9 +161,11 @@ export async function runImport(engine: BrainEngine, args: string[]) {
}
}
// Clear checkpoint on successful completion
if (existsSync(checkpointPath)) {
// Clear checkpoint only on successful completion (no errors)
if (errors === 0 && existsSync(checkpointPath)) {
try { unlinkSync(checkpointPath); } catch { /* non-fatal */ }
} else if (errors > 0 && existsSync(checkpointPath)) {
console.log(` Checkpoint preserved (${errors} errors). Run again to retry failed files.`);
}
const totalTime = ((Date.now() - startTime) / 1000).toFixed(1);

21
src/core/config.ts
View File

@@ -3,8 +3,9 @@ import { join } from 'path';
import { homedir } from 'os';
import type { EngineConfig } from './types.ts';
const CONFIG_DIR = join(homedir(), '.gbrain');
const CONFIG_PATH = join(CONFIG_DIR, 'config.json');
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in Deno Edge Functions)
function getConfigDir() { return join(homedir(), '.gbrain'); }
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
export interface GBrainConfig {
engine: 'postgres' | 'sqlite';
@@ -21,7 +22,7 @@ export interface GBrainConfig {
export function loadConfig(): GBrainConfig | null {
let fileConfig: GBrainConfig | null = null;
try {
const raw = readFileSync(CONFIG_PATH, 'utf-8');
const raw = readFileSync(getConfigPath(), 'utf-8');
fileConfig = JSON.parse(raw) as GBrainConfig;
} catch { /* no config file */ }
@@ -40,10 +41,10 @@ export function loadConfig(): GBrainConfig | null {
}
export function saveConfig(config: GBrainConfig): void {
mkdirSync(CONFIG_DIR, { recursive: true });
writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2) + '\n', { mode: 0o600 });
mkdirSync(getConfigDir(), { recursive: true });
writeFileSync(getConfigPath(), JSON.stringify(config, null, 2) + '\n', { mode: 0o600 });
try {
chmodSync(CONFIG_PATH, 0o600);
chmodSync(getConfigPath(), 0o600);
} catch {
// chmod may fail on some platforms
}
@@ -57,10 +58,10 @@ export function toEngineConfig(config: GBrainConfig): EngineConfig {
};
}
export function getConfigDir(): string {
return CONFIG_DIR;
export function configDir(): string {
return join(homedir(), '.gbrain');
}
export function getConfigPath(): string {
return CONFIG_PATH;
export function configPath(): string {
return join(configDir(), 'config.json');
}

30
src/core/db.ts
View File

@@ -1,7 +1,6 @@
import postgres from 'postgres';
import { readFileSync } from 'fs';
import { join, dirname } from 'path';
import { GBrainError, type EngineConfig } from './types.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
let sql: ReturnType<typeof postgres> | null = null;
@@ -61,27 +60,18 @@ export async function disconnect(): Promise<void> {
export async function initSchema(): Promise<void> {
const conn = getConnection();
// Read schema SQL and execute as a single statement.
// The postgres driver handles multi-statement SQL natively, including
// PL/pgSQL functions with $$ delimiter blocks that contain semicolons.
// The schema uses IF NOT EXISTS / CREATE OR REPLACE for idempotency.
const schemaPath = join(dirname(new URL(import.meta.url).pathname), '..', 'schema.sql');
const schemaSql = readFileSync(schemaPath, 'utf-8');
await conn.unsafe(schemaSql);
// Advisory lock prevents concurrent initSchema() calls from deadlocking
await conn`SELECT pg_advisory_lock(42)`;
try {
await conn.unsafe(SCHEMA_SQL);
} finally {
await conn`SELECT pg_advisory_unlock(42)`;
}
}
export async function withTransaction<T>(fn: () => Promise<T>): Promise<T> {
export async function withTransaction<T>(fn: (tx: ReturnType<typeof postgres>) => Promise<T>): Promise<T> {
const conn = getConnection();
return conn.begin(async (tx) => {
// Temporarily swap global connection to transaction
const prev = sql;
sql = tx as unknown as ReturnType<typeof postgres>;
try {
return await fn();
} finally {
sql = prev;
}
return fn(tx as unknown as ReturnType<typeof postgres>);
});
}

34
src/core/migrate.ts
View File

@@ -48,6 +48,40 @@ const MIGRATIONS: Migration[] = [
if (renamed > 0) console.log(` Renamed ${renamed} slugs`);
},
},
{
version: 3,
name: 'unique_chunk_index',
sql: `
-- Deduplicate any existing duplicate (page_id, chunk_index) rows before adding constraint
DELETE FROM content_chunks a USING content_chunks b
WHERE a.page_id = b.page_id AND a.chunk_index = b.chunk_index AND a.id > b.id;
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
`,
},
{
version: 4,
name: 'access_tokens_and_mcp_log',
sql: `
CREATE TABLE IF NOT EXISTS access_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
token_hash TEXT NOT NULL UNIQUE,
scopes TEXT[],
created_at TIMESTAMPTZ DEFAULT now(),
last_used_at TIMESTAMPTZ,
revoked_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_access_tokens_hash ON access_tokens (token_hash) WHERE revoked_at IS NULL;
CREATE TABLE IF NOT EXISTS mcp_request_log (
id SERIAL PRIMARY KEY,
token_name TEXT,
operation TEXT NOT NULL,
latency_ms INTEGER,
status TEXT NOT NULL DEFAULT 'success',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0

39
src/core/operations.ts
View File

@@ -581,14 +581,37 @@ const file_upload: Operation = {
return { status: 'already_exists', storage_path: storagePath };
}
await sql`
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${mimeType}, ${stat.size}, ${hash}, ${'{}'}::jsonb)
ON CONFLICT (storage_path) DO UPDATE SET
content_hash = EXCLUDED.content_hash,
size_bytes = EXCLUDED.size_bytes,
mime_type = EXCLUDED.mime_type
`;
// Upload to storage backend if configured
if (ctx.config.storage) {
const { createStorage } = await import('./storage.ts');
const storage = await createStorage(ctx.config.storage as any);
try {
await storage.upload(storagePath, content, mimeType || undefined);
} catch (uploadErr) {
throw new OperationError('storage_error', `Upload failed: ${uploadErr instanceof Error ? uploadErr.message : String(uploadErr)}`);
}
}
try {
await sql`
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${mimeType}, ${stat.size}, ${hash}, ${'{}'}::jsonb)
ON CONFLICT (storage_path) DO UPDATE SET
content_hash = EXCLUDED.content_hash,
size_bytes = EXCLUDED.size_bytes,
mime_type = EXCLUDED.mime_type
`;
} catch (dbErr) {
// Rollback: clean up storage if DB write failed
if (ctx.config.storage) {
try {
const { createStorage } = await import('./storage.ts');
const storage = await createStorage(ctx.config.storage as any);
await storage.delete(storagePath);
} catch { /* best effort cleanup */ }
}
throw dbErr;
}
return { status: 'uploaded', storage_path: storagePath, size_bytes: stat.size };
},

87
src/core/postgres-engine.ts
View File

@@ -1,9 +1,8 @@
import postgres from 'postgres';
import { createHash } from 'crypto';
import { readFileSync } from 'fs';
import { join, dirname } from 'path';
import type { BrainEngine } from './engine.ts';
import { runMigrations } from './migrate.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput,
@@ -58,31 +57,31 @@ export class PostgresEngine implements BrainEngine {
async initSchema(): Promise<void> {
const conn = this.sql;
const schemaPath = join(dirname(new URL(import.meta.url).pathname), '..', 'schema.sql');
const schemaSql = readFileSync(schemaPath, 'utf-8');
await conn.unsafe(schemaSql);
// Advisory lock prevents concurrent initSchema() calls from deadlocking
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock)
await conn`SELECT pg_advisory_lock(42)`;
try {
await conn.unsafe(SCHEMA_SQL);
// Run any pending migrations automatically
const { applied } = await runMigrations(this);
if (applied > 0) {
console.log(` ${applied} migration(s) applied`);
// Run any pending migrations automatically
const { applied } = await runMigrations(this);
if (applied > 0) {
console.log(` ${applied} migration(s) applied`);
}
} finally {
await conn`SELECT pg_advisory_unlock(42)`;
}
}
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
if (this._sql) {
// Instance connection: use .begin() directly, no global swap
return this._sql.begin(async (tx) => {
const prev = this._sql;
this._sql = tx as unknown as ReturnType<typeof postgres>;
try {
return await fn(this);
} finally {
this._sql = prev;
}
});
}
return db.withTransaction(() => fn(this));
const conn = this._sql || db.getConnection();
return conn.begin(async (tx) => {
// Create a scoped engine with tx as its connection, no shared state mutation
const txEngine = Object.create(this) as PostgresEngine;
Object.defineProperty(txEngine, 'sql', { get: () => tx });
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
return fn(txEngine);
});
}
// Pages CRUD
@@ -97,7 +96,7 @@ export class PostgresEngine implements BrainEngine {
}
async putPage(slug: string, page: PageInput): Promise<Page> {
validateSlug(slug);
slug = validateSlug(slug);
const sql = this.sql;
const hash = page.content_hash || contentHash(page.compiled_truth, page.timeline || '');
const frontmatter = page.frontmatter || {};
@@ -182,7 +181,7 @@ export class PostgresEngine implements BrainEngine {
const limit = opts?.limit || 20;
const rows = await sql`
SELECT
SELECT DISTINCT ON (p.slug)
p.slug, p.id as page_id, p.title, p.type,
cc.chunk_text, cc.chunk_source,
ts_rank(p.search_vector, websearch_to_tsquery('english', ${query})) AS score,
@@ -192,9 +191,11 @@ export class PostgresEngine implements BrainEngine {
FROM pages p
JOIN content_chunks cc ON cc.page_id = p.id
WHERE p.search_vector @@ websearch_to_tsquery('english', ${query})
ORDER BY score DESC
LIMIT ${limit}
ORDER BY p.slug, score DESC
`;
// Re-sort by score (DISTINCT ON requires ORDER BY slug first) and apply limit
rows.sort((a: any, b: any) => b.score - a.score);
rows.splice(limit);
return rows.map(rowToSearchResult);
}
@@ -231,14 +232,17 @@ export class PostgresEngine implements BrainEngine {
if (pages.length === 0) throw new Error(`Page not found: ${slug}`);
const pageId = pages[0].id;
// Delete existing chunks for this page
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId}`;
// Remove chunks that no longer exist (chunk_index beyond new count)
const newIndices = chunks.map(c => c.chunk_index);
if (newIndices.length > 0) {
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId} AND chunk_index != ALL(${newIndices})`;
} else {
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId}`;
return;
}
// Bulk insert chunks — build multi-row VALUES to reduce round-trips
if (chunks.length === 0) return;
// postgres.js tagged templates don't handle vector casting in bulk,
// so we build a parameterized raw SQL query
// Batch upsert: build a single multi-row INSERT ON CONFLICT statement
// This avoids per-row round-trips and reduces lock contention under parallel workers
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at)';
const rows: string[] = [];
const params: unknown[] = [];
@@ -258,8 +262,16 @@ export class PostgresEngine implements BrainEngine {
}
}
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}`,
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
chunk_text = EXCLUDED.chunk_text,
chunk_source = EXCLUDED.chunk_source,
embedding = COALESCE(EXCLUDED.embedding, content_chunks.embedding),
model = COALESCE(EXCLUDED.model, content_chunks.model),
token_count = EXCLUDED.token_count,
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)`,
params,
);
}
@@ -584,7 +596,7 @@ export class PostgresEngine implements BrainEngine {
// Sync
async updateSlug(oldSlug: string, newSlug: string): Promise<void> {
validateSlug(newSlug);
newSlug = validateSlug(newSlug);
const sql = this.sql;
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug}`;
}
@@ -613,12 +625,13 @@ export class PostgresEngine implements BrainEngine {
}
// Helpers
function validateSlug(slug: string): void {
function validateSlug(slug: string): string {
// Git is the system of record — slugs are lowercased repo-relative paths.
// Only reject empty, path traversal (..), and leading slash.
if (!slug || /\.\./.test(slug) || /^\//.test(slug)) {
throw new Error(`Invalid slug: "${slug}". Slugs cannot be empty, start with /, or contain path traversal.`);
}
// Normalize to lowercase — all entry points (pathToSlug, inferSlug, frontmatter, direct writes) go through here
return slug.toLowerCase();
}
function contentHash(compiledTruth: string, timeline: string): string {

279
src/core/schema-embedded.ts Normal file
View File

@@ -0,0 +1,279 @@
// AUTO-GENERATED — do not edit. Run: bun run build:schema
// Source: src/schema.sql
export const SCHEMA_SQL = `
-- GBrain Postgres + pgvector schema
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- ============================================================
-- pages: the core content table
-- ============================================================
CREATE TABLE IF NOT EXISTS pages (
id SERIAL PRIMARY KEY,
slug TEXT NOT NULL UNIQUE,
type TEXT NOT NULL,
title TEXT NOT NULL,
compiled_truth TEXT NOT NULL DEFAULT '',
timeline TEXT NOT NULL DEFAULT '',
frontmatter JSONB NOT NULL DEFAULT '{}',
content_hash TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
-- ============================================================
-- content_chunks: chunked content with embeddings
-- ============================================================
CREATE TABLE IF NOT EXISTS content_chunks (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
chunk_index INTEGER NOT NULL,
chunk_text TEXT NOT NULL,
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
embedding vector(1536),
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
token_count INTEGER,
embedded_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
-- ============================================================
-- links: cross-references between pages
-- ============================================================
CREATE TABLE IF NOT EXISTS links (
id SERIAL PRIMARY KEY,
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
link_type TEXT NOT NULL DEFAULT '',
context TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(from_page_id, to_page_id)
);
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
-- ============================================================
-- tags
-- ============================================================
CREATE TABLE IF NOT EXISTS tags (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
tag TEXT NOT NULL,
UNIQUE(page_id, tag)
);
CREATE INDEX IF NOT EXISTS idx_tags_tag ON tags(tag);
CREATE INDEX IF NOT EXISTS idx_tags_page_id ON tags(page_id);
-- ============================================================
-- raw_data: sidecar data (replaces .raw/ JSON files)
-- ============================================================
CREATE TABLE IF NOT EXISTS raw_data (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
source TEXT NOT NULL,
data JSONB NOT NULL,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(page_id, source)
);
CREATE INDEX IF NOT EXISTS idx_raw_data_page ON raw_data(page_id);
-- ============================================================
-- timeline_entries: structured timeline
-- ============================================================
CREATE TABLE IF NOT EXISTS timeline_entries (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
date DATE NOT NULL,
source TEXT NOT NULL DEFAULT '',
summary TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
-- ============================================================
-- page_versions: snapshot history for compiled_truth
-- ============================================================
CREATE TABLE IF NOT EXISTS page_versions (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
compiled_truth TEXT NOT NULL,
frontmatter JSONB NOT NULL DEFAULT '{}',
snapshot_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_versions_page ON page_versions(page_id);
-- ============================================================
-- ingest_log
-- ============================================================
CREATE TABLE IF NOT EXISTS ingest_log (
id SERIAL PRIMARY KEY,
source_type TEXT NOT NULL,
source_ref TEXT NOT NULL,
pages_updated JSONB NOT NULL DEFAULT '[]',
summary TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ============================================================
-- config: brain-level settings
-- ============================================================
CREATE TABLE IF NOT EXISTS config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
INSERT INTO config (key, value) VALUES
('version', '1'),
('embedding_model', 'text-embedding-3-large'),
('embedding_dimensions', '1536'),
('chunk_strategy', 'semantic')
ON CONFLICT (key) DO NOTHING;
-- ============================================================
-- access_tokens: bearer tokens for remote MCP access
-- ============================================================
CREATE TABLE IF NOT EXISTS access_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
token_hash TEXT NOT NULL UNIQUE,
scopes TEXT[],
created_at TIMESTAMPTZ DEFAULT now(),
last_used_at TIMESTAMPTZ,
revoked_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_access_tokens_hash ON access_tokens (token_hash) WHERE revoked_at IS NULL;
-- ============================================================
-- mcp_request_log: usage logging for remote MCP requests
-- ============================================================
CREATE TABLE IF NOT EXISTS mcp_request_log (
id SERIAL PRIMARY KEY,
token_name TEXT,
operation TEXT NOT NULL,
latency_ms INTEGER,
status TEXT NOT NULL DEFAULT 'success',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
CREATE TABLE IF NOT EXISTS files (
id SERIAL PRIMARY KEY,
page_slug TEXT REFERENCES pages(slug) ON DELETE SET NULL ON UPDATE CASCADE,
filename TEXT NOT NULL,
storage_path TEXT NOT NULL,
mime_type TEXT,
size_bytes BIGINT,
content_hash TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(storage_path)
);
-- Migration: drop storage_url if it exists (renamed to storage_path only)
ALTER TABLE files DROP COLUMN IF EXISTS storage_url;
CREATE INDEX IF NOT EXISTS idx_files_page ON files(page_slug);
CREATE INDEX IF NOT EXISTS idx_files_hash ON files(content_hash);
-- ============================================================
-- Trigger-based search_vector (spans pages + timeline_entries)
-- ============================================================
ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
-- Function to rebuild search_vector for a page
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger AS \$\$
DECLARE
timeline_text TEXT;
BEGIN
-- Gather timeline_entries text for this page
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
-- Build weighted tsvector
NEW.search_vector :=
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
\$\$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_pages_search_vector ON pages;
CREATE TRIGGER trg_pages_search_vector
BEFORE INSERT OR UPDATE ON pages
FOR EACH ROW
EXECUTE FUNCTION update_page_search_vector();
-- When timeline_entries change, update the parent page's search_vector
CREATE OR REPLACE FUNCTION update_page_search_vector_from_timeline() RETURNS trigger AS \$\$
DECLARE
page_row pages%ROWTYPE;
BEGIN
-- Touch the page to re-fire its trigger
UPDATE pages SET updated_at = now()
WHERE id = coalesce(NEW.page_id, OLD.page_id);
RETURN NEW;
END;
\$\$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_timeline_search_vector ON timeline_entries;
CREATE TRIGGER trg_timeline_search_vector
AFTER INSERT OR UPDATE OR DELETE ON timeline_entries
FOR EACH ROW
EXECUTE FUNCTION update_page_search_vector_from_timeline();
-- ============================================================
-- Row Level Security: block anon access, postgres role bypasses
-- ============================================================
-- The postgres role (used by gbrain via pooler) has BYPASSRLS.
-- Enabling RLS with no policies means the anon key can't read anything.
-- Only enable if the current role actually has BYPASSRLS privilege,
-- otherwise we'd lock ourselves out.
DO \$\$
DECLARE
has_bypass BOOLEAN;
BEGIN
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
IF has_bypass THEN
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
ALTER TABLE content_chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE links ENABLE ROW LEVEL SECURITY;
ALTER TABLE tags ENABLE ROW LEVEL SECURITY;
ALTER TABLE raw_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE timeline_entries ENABLE ROW LEVEL SECURITY;
ALTER TABLE page_versions ENABLE ROW LEVEL SECURITY;
ALTER TABLE ingest_log ENABLE ROW LEVEL SECURITY;
ALTER TABLE config ENABLE ROW LEVEL SECURITY;
ALTER TABLE files ENABLE ROW LEVEL SECURITY;
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
ELSE
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
END IF;
END \$\$;
`;

20
src/core/search/dedup.ts
View File

@@ -45,19 +45,25 @@ export function dedupResults(
}
/**
* Layer 1: Keep only the highest-scoring chunk per page.
* Layer 1: Keep top 3 chunks per page (not just 1).
* Later layers (text similarity, cap per page) handle further reduction.
*/
function dedupBySource(results: SearchResult[]): SearchResult[] {
const byPage = new Map<string, SearchResult>();
const byPage = new Map<string, SearchResult[]>();
for (const r of results) {
const existing = byPage.get(r.slug);
if (!existing || r.score > existing.score) {
byPage.set(r.slug, r);
}
const existing = byPage.get(r.slug) || [];
existing.push(r);
byPage.set(r.slug, existing);
}
return Array.from(byPage.values()).sort((a, b) => b.score - a.score);
const kept: SearchResult[] = [];
for (const chunks of byPage.values()) {
chunks.sort((a, b) => b.score - a.score);
kept.push(...chunks.slice(0, 3));
}
return kept.sort((a, b) => b.score - a.score);
}
/**

122
src/core/storage/s3.ts
View File

@@ -1,79 +1,109 @@
import {
S3Client,
PutObjectCommand,
GetObjectCommand,
DeleteObjectCommand,
HeadObjectCommand,
ListObjectsV2Command,
} from '@aws-sdk/client-s3';
import type { StorageBackend, StorageConfig } from '../storage.ts';
/**
* S3-compatible storage — works with AWS S3, Cloudflare R2, MinIO, etc.
*
* Uses fetch() directly against the S3 REST API with AWS Signature V4.
* No SDK dependency needed — keeps the binary small.
* Uses @aws-sdk/client-s3 for proper authentication and request signing.
*/
export class S3Storage implements StorageBackend {
private client: S3Client;
private bucket: string;
private region: string;
private endpoint: string;
private accessKeyId: string;
private secretAccessKey: string;
constructor(config: StorageConfig) {
this.bucket = config.bucket;
this.region = config.region || 'us-east-1';
this.endpoint = config.endpoint || `https://s3.${this.region}.amazonaws.com`;
this.accessKeyId = config.accessKeyId || '';
this.secretAccessKey = config.secretAccessKey || '';
if (!this.accessKeyId || !this.secretAccessKey) {
const region = config.region || 'us-east-1';
if (!config.accessKeyId || !config.secretAccessKey) {
throw new Error('S3 storage requires accessKeyId and secretAccessKey in config');
}
}
private url(path: string): string {
return `${this.endpoint}/${this.bucket}/${path}`;
}
private async signedFetch(method: string, path: string, body?: Buffer, mime?: string): Promise<Response> {
// Simplified S3 request — for production, use proper AWS Sig V4
// For now, works with public buckets and pre-signed URLs
const url = this.url(path);
const headers: Record<string, string> = {};
if (mime) headers['Content-Type'] = mime;
return fetch(url, { method, body, headers });
this.client = new S3Client({
region,
...(config.endpoint ? {
endpoint: config.endpoint,
forcePathStyle: true, // Required for R2, MinIO, and custom endpoints
} : {}),
credentials: {
accessKeyId: config.accessKeyId,
secretAccessKey: config.secretAccessKey,
},
});
}
async upload(path: string, data: Buffer, mime?: string): Promise<void> {
const res = await this.signedFetch('PUT', path, data, mime || 'application/octet-stream');
if (!res.ok) throw new Error(`S3 upload failed: ${res.status} ${res.statusText}`);
await this.client.send(new PutObjectCommand({
Bucket: this.bucket,
Key: path,
Body: data,
ContentType: mime || 'application/octet-stream',
}));
}
async download(path: string): Promise<Buffer> {
const res = await this.signedFetch('GET', path);
if (!res.ok) throw new Error(`S3 download failed: ${res.status} ${res.statusText}`);
return Buffer.from(await res.arrayBuffer());
const res = await this.client.send(new GetObjectCommand({
Bucket: this.bucket,
Key: path,
}));
if (!res.Body) throw new Error(`S3 download returned empty body: ${path}`);
return Buffer.from(await res.Body.transformToByteArray());
}
async delete(path: string): Promise<void> {
const res = await this.signedFetch('DELETE', path);
if (!res.ok && res.status !== 404) throw new Error(`S3 delete failed: ${res.status}`);
await this.client.send(new DeleteObjectCommand({
Bucket: this.bucket,
Key: path,
}));
}
async exists(path: string): Promise<boolean> {
const res = await this.signedFetch('HEAD', path);
return res.ok;
try {
await this.client.send(new HeadObjectCommand({
Bucket: this.bucket,
Key: path,
}));
return true;
} catch (e: any) {
if (e.name === 'NotFound' || e.$metadata?.httpStatusCode === 404) return false;
throw e;
}
}
async list(prefix: string): Promise<string[]> {
const url = `${this.endpoint}/${this.bucket}?list-type=2&prefix=${encodeURIComponent(prefix)}`;
const res = await fetch(url);
if (!res.ok) throw new Error(`S3 list failed: ${res.status}`);
const xml = await res.text();
const keys: string[] = [];
const regex = /<Key>([^<]+)<\/Key>/g;
let match;
while ((match = regex.exec(xml)) !== null) {
keys.push(match[1]);
}
return keys;
const res = await this.client.send(new ListObjectsV2Command({
Bucket: this.bucket,
Prefix: prefix,
}));
return (res.Contents || []).map(obj => obj.Key!).filter(Boolean);
}
async getUrl(path: string): Promise<string> {
return this.url(path);
// For custom endpoints (R2, MinIO), use the endpoint URL
const endpoint = (this.client.config as any).endpoint;
if (endpoint) {
const base = typeof endpoint === 'function' ? (await endpoint()).url.toString() : endpoint;
return `${base}/${this.bucket}/${path}`;
}
const region = await this.client.config.region();
return `https://${this.bucket}.s3.${region}.amazonaws.com/${path}`;
}
async getContentHash(path: string): Promise<string | null> {
try {
const res = await this.client.send(new HeadObjectCommand({
Bucket: this.bucket,
Key: path,
}));
// ETag is typically the MD5 hash (quoted), but for multipart uploads it's different
return res.ETag?.replace(/"/g, '') || null;
} catch {
return null;
}
}
}

2
src/core/sync.ts
View File

@@ -131,5 +131,5 @@ export function slugifyPath(filePath: string): string {
export function pathToSlug(filePath: string, repoPrefix?: string): string {
let slug = slugifyPath(filePath);
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
return slug;
return slug.toLowerCase();
}

16
src/edge-entry.ts Normal file
View File

@@ -0,0 +1,16 @@
/**
* Edge Function bundle entry point.
*
* Curated exports for Supabase Edge Functions (Deno runtime).
* Excludes modules that depend on Node.js filesystem APIs:
* - db.ts (reads schema.sql from disk — now uses schema-embedded.ts)
* - config.ts (reads ~/.gbrain/config.json via homedir())
* - import-file.ts (uses readFileSync/statSync)
* - sync.ts (git-based, local filesystem)
*/
export { operations, operationsByName, OperationError } from './core/operations.ts';
export type { Operation, OperationContext, ParamDef } from './core/operations.ts';
export { PostgresEngine } from './core/postgres-engine.ts';
export type { BrainEngine } from './core/engine.ts';
export * from './core/types.ts';
export { VERSION } from './version.ts';

7
src/mcp/server.ts
View File

@@ -1,5 +1,6 @@
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js';
import type { BrainEngine } from '../core/engine.ts';
import { operations, OperationError } from '../core/operations.ts';
import type { OperationContext } from '../core/operations.ts';
@@ -13,7 +14,7 @@ export async function startMcpServer(engine: BrainEngine) {
);
// Generate tool definitions from operations
server.setRequestHandler('tools/list' as any, async () => ({
server.setRequestHandler(ListToolsRequestSchema, async () => ({
tools: operations.map(op => ({
name: op.name,
description: op.description,
@@ -35,7 +36,7 @@ export async function startMcpServer(engine: BrainEngine) {
}));
// Dispatch tool calls to operation handlers
server.setRequestHandler('tools/call' as any, async (request: any) => {
server.setRequestHandler(CallToolRequestSchema, async (request: any) => {
const { name, arguments: params } = request.params;
const op = operations.find(o => o.name === name);
if (!op) {
@@ -82,7 +83,7 @@ export async function handleToolCall(
engine,
config: loadConfig() || { engine: 'postgres' },
logger: { info: console.log, warn: console.warn, error: console.error },
dryRun: false,
dryRun: !!(params?.dry_run),
};
return op.handler(ctx, params);

28
src/schema.sql
View File

@@ -39,6 +39,7 @@ CREATE TABLE IF NOT EXISTS content_chunks (
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
@@ -141,6 +142,33 @@ INSERT INTO config (key, value) VALUES
('chunk_strategy', 'semantic')
ON CONFLICT (key) DO NOTHING;
-- ============================================================
-- access_tokens: bearer tokens for remote MCP access
-- ============================================================
CREATE TABLE IF NOT EXISTS access_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
token_hash TEXT NOT NULL UNIQUE,
scopes TEXT[],
created_at TIMESTAMPTZ DEFAULT now(),
last_used_at TIMESTAMPTZ,
revoked_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_access_tokens_hash ON access_tokens (token_hash) WHERE revoked_at IS NULL;
-- ============================================================
-- mcp_request_log: usage logging for remote MCP requests
-- ============================================================
CREATE TABLE IF NOT EXISTS mcp_request_log (
id SERIAL PRIMARY KEY,
token_name TEXT,
operation TEXT NOT NULL,
latency_ms INTEGER,
status TEXT NOT NULL DEFAULT 'success',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================