diff --git a/CHANGELOG.md b/CHANGELOG.md index f84dfb0..3d09af8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,94 @@ # Changelog All notable changes to GBrain will be documented in this file. +## [0.22.7] - 2026-04-28 + +## **Built-in HTTP transport with bearer auth for remote MCP.** +## **Postgres-backed tokens, default-deny CORS, two-bucket rate limit, body cap, per-request audit.** + +v0.22.7 ships `gbrain serve --http`: a built-in HTTP transport for remote MCP, authenticating via the existing `access_tokens` table that `gbrain auth create/list/revoke` already manages. Bearer-only, no OAuth surface, no registration endpoint, no self-service tokens. SECURITY.md is the canonical reference for the hardening posture and recommended deployment. + +The hardening lives inside the transport, not in the doc: + +| Layer | Default | Configurable via | +|---|---|---| +| CORS | default-deny (no `Access-Control-Allow-Origin`) | `GBRAIN_HTTP_CORS_ORIGIN=a.com,b.com` | +| Pre-auth IP rate limit | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` | +| Post-auth token rate limit | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` | +| Body cap | 1 MiB, stream-counted | `GBRAIN_HTTP_MAX_BODY_BYTES` | +| `last_used_at` debounce | once per token per 60s | (SQL-level WHERE clause, race-tolerant) | +| Per-request audit | `mcp_request_log` row per `/mcp` | (existing schema, since v4) | +| Reverse-proxy trust | off | `GBRAIN_HTTP_TRUST_PROXY=1` to honor X-Forwarded-For | + +The IP rate-limit fires **before** the auth lookup so the limit caps load on the auth path itself, not just response codes. The token-id rate limit fires after auth so a runaway authenticated client gets throttled at the right principal. Both buckets live in a bounded LRU map (default 10K keys, TTL prune at 2× window) so unique-key growth can't drift into memory pressure. + +### What changed for users + +You can now expose GBrain remotely with the built-in transport: + +```bash +gbrain auth create my-laptop # tokens managed via the existing CLI +gbrain serve --http --port 8787 # Postgres-only; PGLite users see a clear fail-fast +ngrok http 8787 --url your-brain.ngrok.app # any tunnel works +``` + +Then point Claude Desktop, claude.ai/code, or any MCP client at `http://your-tunnel/mcp` with `Authorization: Bearer `. CORS, rate limits, and body caps are on by default. `gbrain auth` is now wired into the main CLI, so it works from the compiled binary the same as `gbrain doctor` or `gbrain serve`. + +### For contributors + +- `src/mcp/dispatch.ts` (new) — shared `dispatchToolCall(engine, name, params, opts)` consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). One source of truth for `validateParams`, `OperationContext` construction, and handler invocation, so the two transports can't drift apart. +- `src/mcp/rate-limit.ts` (new) — bounded-LRU token-bucket. Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. +- `src/mcp/http-transport.ts` — built on the new dispatch + rate-limit modules. `application/json` response shape (gbrain MCP tools are synchronous; the Streamable HTTP transport spec allows JSON for non-streaming responses). +- `src/cli.ts` + `src/commands/auth.ts` — `auth` is now a wired CLI subcommand. Direct-script usage (`bun run src/commands/auth.ts ...`) still works for environments without a compiled binary. +- 23 unit cases in `test/http-transport.test.ts`, 8 E2E cases in `test/e2e/http-transport.test.ts`. Unit covers the full dispatch round-trip with a real operation; E2E covers `last_used_at` debounce against real Postgres semantics. + +### Known limits + +- `gbrain serve --http` is **Postgres-only**. PGLite has no `access_tokens` or `mcp_request_log` table by design (`src/core/pglite-schema.ts:5-6`). Local agents continue to use stdio (`gbrain serve`). +- Behind a tunnel (ngrok, Tailscale Funnel, Cloudflare Tunnel), all requests share one egress IP. The pre-auth IP bucket becomes effectively shared by all clients on that tunnel; the token-id bucket is the load-bearing limiter for tunnel deployments. Documented in SECURITY.md. + +### Itemized changes + +- New: `gbrain serve --http [--port N]` ships the built-in HTTP transport +- New: `gbrain auth create/list/revoke/test` wired into the main CLI (was a standalone script) +- New: SECURITY.md documents the disclosure path, the recommended remote-MCP setup, and the full hardening reference +- New: `src/mcp/dispatch.ts` — shared dispatch path for stdio + HTTP +- New: `src/mcp/rate-limit.ts` — bounded-LRU token-bucket limiter +- Hardening: CORS default-deny, two-bucket rate limit (per-IP pre-auth + per-token post-auth), 1 MiB body cap with stream-counted enforcement, `mcp_request_log` per-request audit, `last_used_at` SQL-level debounce +- Tests: 23 unit + 8 E2E covering auth, dispatch, CORS, body cap, rate limit, and audit +- Docs: SECURITY.md, DEPLOY.md, and per-client setup guides updated to recommend `--http` and document the env vars + +## To take advantage of v0.22.7 + +`gbrain upgrade` should do this automatically. If it didn't, or if you want to expose your brain over HTTP: + +1. **Confirm migrations are at v4 or higher** (the `access_tokens` + `mcp_request_log` tables were added in migration v4): + ```bash + gbrain doctor # schema_version check should pass + gbrain apply-migrations --yes # if not, run this + ``` +2. **Create a token for each remote client:** + ```bash + gbrain auth create my-laptop # prints the token once — copy it + ``` +3. **Start the HTTP server:** + ```bash + gbrain serve --http --port 8787 + ``` +4. **(Optional) configure CORS allowlist if a browser client will hit it:** + ```bash + GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787 + ``` +5. **(Optional) audit who's hitting your brain:** + ```bash + psql $DATABASE_URL -c "SELECT created_at, token_name, operation, status, latency_ms + FROM mcp_request_log ORDER BY created_at DESC LIMIT 50" + ``` +6. **If `gbrain serve --http` exits with "Postgres engine required":** PGLite is local-only by design. Either keep using stdio (`gbrain serve`) for local agents, or migrate to Postgres (`gbrain migrate --to supabase`). + +If anything breaks: `gbrain doctor`, `~/.gbrain/upgrade-errors.jsonl` (if present), and please file an issue at https://github.com/garrytan/gbrain/issues with both. + + ## [0.22.6.1] - 2026-04-26 diff --git a/CLAUDE.md b/CLAUDE.md index 816514e..a7ce96a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -91,8 +91,11 @@ strict behavior when unset. - `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). - `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman - `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed) -- `src/mcp/server.ts` — MCP stdio server (generated from operations) -- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test) +- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. +- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport. +- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets. +- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration. +- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only). - `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. @@ -277,7 +280,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers), `test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures), `test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics), -`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source). +`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source), +`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed). E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`. - `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage. @@ -292,6 +296,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. - `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. - `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review. +- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. diff --git a/README.md b/README.md index c401edb..3496f77 100644 --- a/README.md +++ b/README.md @@ -80,12 +80,13 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o ### Remote MCP (Claude Desktop, Cowork, Perplexity) ```bash -ngrok http 8787 --url your-brain.ngrok.app -bun run src/commands/auth.ts create "claude-desktop" +gbrain auth create "claude-desktop" # tokens via the existing CLI +gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only) +ngrok http 8787 --url your-brain.ngrok.app # any tunnel works claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN" ``` -Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented). +Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented). ### Using gbrain with GStack @@ -657,6 +658,8 @@ ADMIN gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only) gbrain stats Brain statistics gbrain serve MCP server (stdio) + gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth) + gbrain auth create|list|revoke|test Token management for the HTTP transport gbrain integrations Integration recipe dashboard gbrain sources list|add|remove|... Multi-source brain management (v0.18) gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..3a251e5 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,168 @@ +# Security + +## Reporting Vulnerabilities + +If you discover a security issue in GBrain, please report it privately by opening +a [private security advisory](https://github.com/garrytan/gbrain/security/advisories/new) +on GitHub. + +Do not open a public issue for security vulnerabilities. + +## Remote MCP Security + +### ⚠️ Do NOT use open OAuth client registration for remote MCP + +If you deploy GBrain's MCP server behind an HTTP wrapper with OAuth 2.1 +support, **never allow unauthenticated client registration**. An attacker +who discovers your server URL can: + +1. Register a new OAuth client via `POST /register` +2. Use `client_credentials` grant to obtain a bearer token +3. Access all brain data via the MCP tools + +### Recommended: `gbrain serve --http` + +As of v0.22.7, GBrain ships a built-in HTTP transport that uses the +existing `access_tokens` table for authentication: + +```bash +# Create a token +gbrain auth create "my-client" + +# Start the HTTP server +gbrain serve --http --port 8787 + +# Connect via ngrok, Tailscale, or any tunnel +ngrok http 8787 --url your-brain.ngrok.app +``` + +This is the recommended way to expose GBrain remotely. No OAuth, no +registration endpoint, no self-service tokens. Tokens are managed +exclusively via `gbrain auth create/list/revoke`. + +### If you must use a custom HTTP wrapper + +1. **Require a secret for client registration** — check a header or body + parameter before creating new OAuth clients +2. **Disable `client_credentials` grant** — only allow `authorization_code` + with browser-based approval +3. **Restrict scopes** — never issue tokens with unlimited scope +4. **Log all token issuance** — alert on unexpected registrations +5. **Rate-limit registration and token endpoints** + +### Token Management + +```bash +gbrain auth create "claude-desktop" # Create a new token +gbrain auth list # List all tokens +gbrain auth revoke "claude-desktop" # Revoke a token +gbrain auth test --token # Smoke-test a remote server +``` + +Tokens are stored as SHA-256 hashes in the `access_tokens` table. The +plaintext token is shown once at creation and never stored. + +## `gbrain serve --http` hardening (v0.22.7+) + +The built-in HTTP transport ships with several layers of hardening on by +default. All env vars below are optional; the defaults are intentionally +conservative. + +### Postgres-only + +`gbrain serve --http` requires a Postgres engine. PGLite is local-only by +design and the `access_tokens` / `mcp_request_log` tables don't exist in +the PGLite schema. Local agents continue to use stdio (`gbrain serve`). +Running `--http` against a PGLite-backed install fails fast with a clear +error message at startup. + +### CORS + +Default-deny: no `Access-Control-Allow-Origin` header is sent unless an +allowlist is configured. To allow browser-based MCP clients: + +```bash +GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai gbrain serve --http --port 8787 +# Multiple origins: comma-separated +GBRAIN_HTTP_CORS_ORIGIN=https://claude.ai,https://your.app gbrain serve --http +``` + +When the request `Origin` matches the allowlist, the server echoes it +back in `Access-Control-Allow-Origin` (with `Vary: Origin`). Otherwise no +CORS header is sent and the browser blocks the request. + +### Rate limiting + +Two buckets, both stored in a bounded LRU map (default 10K keys, evicts +least-recently-used on overflow, prunes entries older than 2× the +window): + +| Bucket | When it fires | Default | Env var | +|---|---|---|---| +| Pre-auth IP | Before the DB lookup, on every `/mcp` request | 30 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_IP` | +| Post-auth token | After a valid token is resolved | 60 req / 60s | `GBRAIN_HTTP_RATE_LIMIT_TOKEN` | +| LRU cap | Maximum distinct keys across both buckets | 10000 | `GBRAIN_HTTP_RATE_LIMIT_LRU` | + +On exhaustion the server returns `429 Too Many Requests` with a +`Retry-After` header. + +**Caveat for tunneled deployments (ngrok, Tailscale Funnel, Cloudflare +Tunnel):** all requests share one egress IP, so the pre-auth IP bucket +becomes effectively shared by all clients on that tunnel. The +post-auth token-id bucket is the load-bearing limiter for tunnel-fronted +deployments. + +### Reverse-proxy trust + +Disabled by default. To honor `X-Forwarded-For` (or `X-Real-IP`) when +gbrain runs behind a trusted reverse proxy: + +```bash +GBRAIN_HTTP_TRUST_PROXY=1 gbrain serve --http --port 8787 +``` + +**Critical safety contract:** only set `GBRAIN_HTTP_TRUST_PROXY=1` when +**both** of these are true: + +1. gbrain is reachable only via a trusted reverse proxy (not directly + exposed to the internet on the configured port). The simplest + guarantee is to bind gbrain to `127.0.0.1` or a private interface + and have the proxy forward to it. +2. The proxy strips any client-supplied `X-Forwarded-For` and `X-Real-IP` + headers, then sets them itself. (nginx with `proxy_set_header + X-Forwarded-For $remote_addr` does this; Cloudflare and most cloud + load balancers handle it automatically.) + +If gbrain is reachable directly AND `GBRAIN_HTTP_TRUST_PROXY=1` is set, +clients can spoof their IP by sending arbitrary `X-Forwarded-For` +headers, defeating the pre-auth IP rate limit. Without the flag, gbrain +ignores all forwarded-for headers and uses the socket peer address, +which is the safe default for direct-exposure deployments. + +### Body size cap + +Default 1 MiB, stream-counted (chunked transfers without +`Content-Length` are still capped). Override: + +```bash +GBRAIN_HTTP_MAX_BODY_BYTES=2097152 gbrain serve --http # 2 MiB +``` + +Over-cap requests get `413 Payload Too Large` immediately, before any +body is materialized in memory. + +### Audit log + +Every `/mcp` request writes one row to `mcp_request_log`: + +```bash +psql "$DATABASE_URL" -c \ + "SELECT created_at, token_name, operation, status, latency_ms + FROM mcp_request_log + ORDER BY created_at DESC LIMIT 100" +``` + +`status` is one of: `success`, `error`, `auth_failed`, `rate_limited`, +`body_too_large`, `parse_error`, `unknown_method`. Failed-auth rows have +`token_name = NULL`. Inserts are fire-and-forget so audit failures +never block requests. diff --git a/TODOS.md b/TODOS.md index cc7bc84..4ed2ce4 100644 --- a/TODOS.md +++ b/TODOS.md @@ -585,3 +585,90 @@ iteration's residuals. **Effort estimate:** S (human: ~2 hr / CC: ~10 min). **Priority:** P3. **Depends on:** The above caller-opt-in retry (#1) is the natural co-lander since both touch the same error-classification surface. + +## remote MCP / HTTP transport (v0.22.7 follow-ups) + +### Audit-log write amplification on rejected `/mcp` traffic +**What:** `src/mcp/http-transport.ts` writes a row to `mcp_request_log` for every +incoming `/mcp` request, including rate-limited (429), oversized (413), and +auth-failed (401) traffic. Under sustained attack the IP rate limit caps audit +writes per IP at 30/min, but at scale (10K distinct IPs) that's still 300K +inserts/min. Two follow-ups: (1) instrument the audit-write rate so we can see +the actual production volume; (2) consider a separate "rejected" table or +sampling for failed-auth rows so the success-path audit table doesn't get +swamped. + +**Why:** Codex flagged this during the v0.22.7 ship adversarial review. We kept +the full audit on purpose — forensic data of an attack is valuable — but want +to revisit once we have real volume numbers. + +**Pros:** Bounds DB write volume under attack. Keeps the success-path audit +table small enough for fast queries. + +**Cons:** Adds a second table or a sampling rule. Not free complexity. Probably +not worth it until production hits a real attack pattern. + +**Context:** `src/mcp/http-transport.ts:222,235,245` (the three audit-on-reject +call sites) + `src/schema.sql:342` (the unbounded table). + +**Effort estimate:** M (human: ~half day / CC: ~30 min once we have volume data). +**Priority:** P3 — wait for evidence. +**Depends on:** Production telemetry on `mcp_request_log` insert rate. + +### `validateParams` doesn't check enum values or array item types +**What:** `src/mcp/dispatch.ts:27` (extracted from `src/mcp/server.ts` in +v0.22.7) only checks top-level JS types. Operations declare `enum` constraints +(e.g. `direction: 'in' | 'out' | 'both'`) and array `items: { type: ... }` +schemas in `src/core/operations.ts`, but `validateParams` ignores both. Bad +inputs still reach handlers — concretely, an invalid `direction` falls through +the engine's else branch at `src/core/postgres-engine.ts:954`, widening +traversal unexpectedly; malformed `pages_updated` arrays could be written as +garbage JSONB. + +**Why:** Codex flagged this during the v0.22.7 ship adversarial review. The +validator was lifted verbatim from the pre-existing stdio path during the +dispatch.ts extraction — same gap exists on the stdio MCP server today, so +this isn't a v0.22.7 regression. Still worth tightening, since "shared +validation" is now the architectural guarantee both transports rely on. + +**Pros:** Better defense-in-depth at the MCP boundary. Catches malformed agent +inputs before the engine layer has to. + +**Cons:** Need to walk every operation's param schema and decide which enum +violations are user-facing errors vs internal bugs. May need a typed Zod-style +schema layer to do this cleanly. + +**Context:** `src/mcp/dispatch.ts:27` + `src/core/operations.ts` (param defs). +Same gap pre-existed on stdio MCP path. + +**Effort estimate:** M (human: ~half day / CC: ~30 min if we use the existing +ParamDef shape; XL if a Zod migration is the chosen direction). +**Priority:** P2. +**Depends on:** Whether we want to keep the lightweight ParamDef shape or +migrate to typed schemas. + +### Streaming MCP tool support (re-add SSE based on Accept header) +**What:** v0.22.7 dropped SSE entirely from `gbrain serve --http` because no +current MCP tool streams. When the first streaming tool ships (long-running +agent delegation as an MCP tool, `resources/subscribe`, `sampling/createMessage`), +re-add SSE in `/mcp` based on the `Accept` header per the Streamable HTTP +transport spec. ~30 lines + spec compliance test. + +**Why:** Removing SSE simplified the v0.22.7 transport (one response path, +fewer test cases). Adding it back when actually needed is cheap and keeps the +code lean in the meantime. + +**Effort estimate:** S (human: ~2 hr / CC: ~15 min). +**Priority:** P3 — wait for the first streaming tool. +**Depends on:** A streaming MCP tool actually existing. + +### `access_tokens.scopes` enforcement +**What:** The `access_tokens` schema has had a `scopes TEXT[]` column since +migration v4 (`src/core/migrate.ts:84`), but nothing enforces it. v0.22.7's +`gbrain auth create` doesn't accept a `--scopes` flag, and `dispatchToolCall` +doesn't gate on scopes. Adding per-tool scope enforcement would let +"claude-desktop-readonly" and "ingest-only" tokens exist. + +**Effort estimate:** M (human: ~1 day / CC: ~30 min for the schema-aware gate). +**Priority:** P3. +**Depends on:** Nothing. diff --git a/VERSION b/VERSION index edadaf5..9e432c8 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.22.6.1 +0.22.7 diff --git a/docs/mcp/ALTERNATIVES.md b/docs/mcp/ALTERNATIVES.md index d47a061..49c6834 100644 --- a/docs/mcp/ALTERNATIVES.md +++ b/docs/mcp/ALTERNATIVES.md @@ -1,8 +1,9 @@ # Remote MCP Deployment Options GBrain's MCP server runs via `gbrain serve` (stdio transport). To make it -accessible from other devices and AI clients, you need an HTTP wrapper and -a public tunnel. Here are your options. +accessible from other devices and AI clients, run `gbrain serve --http` +(built-in HTTP transport with bearer auth, Postgres-only ... see +[DEPLOY.md](DEPLOY.md)) behind a public tunnel. Here are your tunnel options. ## ngrok (recommended) @@ -13,8 +14,9 @@ a public tunnel. Here are your options. # 1. Install ngrok brew install ngrok -# 2. Start your MCP server (behind an HTTP wrapper) -# See docs/mcp/DEPLOY.md for the server setup +# 2. Start the built-in HTTP transport +gbrain serve --http --port 8787 +# See docs/mcp/DEPLOY.md for token setup # 3. Expose via ngrok ngrok http 8787 --url your-brain.ngrok.app @@ -59,6 +61,7 @@ Both run Bun natively. No bundling, no Deno, no cold start, no timeout limits. | All 30 operations | Yes | Yes | Yes | | Setup time | 5 min | 10 min | 15 min | -**Note:** `gbrain serve --http` (built-in HTTP transport) is planned but not yet -implemented. Currently, remote MCP requires a custom HTTP wrapper around `gbrain serve`. -See [DEPLOY.md](DEPLOY.md) for details. +**Note:** `gbrain serve --http` is the built-in HTTP transport (v0.22.7+). Bearer auth +against the `access_tokens` table, default-deny CORS, two-bucket rate limit, body cap, +per-request audit log. Postgres-only by design (PGLite is local-only). See +[DEPLOY.md](DEPLOY.md) and [SECURITY.md](../../SECURITY.md) for env vars and tunables. diff --git a/docs/mcp/CLAUDE_CODE.md b/docs/mcp/CLAUDE_CODE.md index 36037f1..54a02be 100644 --- a/docs/mcp/CLAUDE_CODE.md +++ b/docs/mcp/CLAUDE_CODE.md @@ -21,7 +21,7 @@ claude mcp add gbrain -t http \ ``` Replace `YOUR-DOMAIN` with your ngrok domain and `YOUR_TOKEN` with a token -from `bun run src/commands/auth.ts create "claude-code"`. +from `gbrain auth create "claude-code"`. ## Verify diff --git a/docs/mcp/CLAUDE_COWORK.md b/docs/mcp/CLAUDE_COWORK.md index 327dc96..35e5601 100644 --- a/docs/mcp/CLAUDE_COWORK.md +++ b/docs/mcp/CLAUDE_COWORK.md @@ -12,7 +12,7 @@ For Team/Enterprise plans, an org Owner adds the connector: https://YOUR-DOMAIN.ngrok.app/mcp ``` 3. Add Bearer token authentication in Advanced Settings - (create one with `bun run src/commands/auth.ts create "cowork"`) + (create one with `gbrain auth create "cowork"`) 4. Save Note: Cowork connects from Anthropic's cloud, not your device. Your server diff --git a/docs/mcp/CLAUDE_DESKTOP.md b/docs/mcp/CLAUDE_DESKTOP.md index 81d1173..5df65cb 100644 --- a/docs/mcp/CLAUDE_DESKTOP.md +++ b/docs/mcp/CLAUDE_DESKTOP.md @@ -16,7 +16,7 @@ Remote HTTP servers must be added through the GUI. Replace `YOUR-DOMAIN` with your ngrok domain (see [ngrok-tunnel recipe](../../recipes/ngrok-tunnel.md) for setup). 5. Set authentication to **Bearer Token** and paste your token - (create one with `bun run src/commands/auth.ts create "claude-desktop"`) + (create one with `gbrain auth create "claude-desktop"`) 6. Save ## Verify diff --git a/docs/mcp/DEPLOY.md b/docs/mcp/DEPLOY.md index 7370b34..045b210 100644 --- a/docs/mcp/DEPLOY.md +++ b/docs/mcp/DEPLOY.md @@ -1,8 +1,13 @@ # Deploy GBrain Remote MCP Server +> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in +> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and +> per-request audit log. **Postgres-only** (PGLite is local-only by design). +> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults. + Access your brain from any device, any AI client. GBrain's MCP server runs locally -via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a -public tunnel. +via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP +transport behind a public tunnel. ## Two Paths @@ -13,21 +18,23 @@ gbrain serve ``` Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio. -No server, no tunnel, no token needed. +No server, no tunnel, no token needed. Works on both PGLite and Postgres engines. -### Remote (any device, any AI client) +### Remote (any device, any AI client) — Postgres only ``` Your AI client (Claude Desktop, Perplexity, etc.) → ngrok tunnel (https://YOUR-DOMAIN.ngrok.app) - → Your HTTP server (wraps gbrain serve) - → Supabase Postgres (via pooler connection string) + → gbrain serve --http (built-in transport with bearer auth) + → Postgres (pooler connection or self-hosted) ``` This requires: -1. A machine running `gbrain serve` behind an HTTP wrapper -2. A public tunnel (ngrok, Tailscale, or cloud host) -3. Bearer token auth for security +1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres; + running `gbrain serve --http` against a PGLite install fails fast at startup) +2. A machine running `gbrain serve --http` +3. A public tunnel (ngrok, Tailscale, or cloud host) +4. A bearer token created via `gbrain auth create ` ## Remote Setup @@ -46,13 +53,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain ```bash # Create a token for each client -bun run src/commands/auth.ts create "claude-desktop" +gbrain auth create "claude-desktop" # List all tokens -bun run src/commands/auth.ts list +gbrain auth list # Revoke a token -bun run src/commands/auth.ts revoke "claude-desktop" +gbrain auth revoke "claude-desktop" ``` Tokens are per-client. Create one for each device/app. Revoke individually @@ -68,7 +75,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database. ### 4. Verify ```bash -bun run src/commands/auth.ts test \ +gbrain auth test \ https://YOUR-DOMAIN.ngrok.app/mcp \ --token YOUR_TOKEN ``` @@ -96,7 +103,7 @@ Funnel, and cloud hosts (Fly.io, Railway). Include the Authorization header: `Authorization: Bearer YOUR_TOKEN` **"invalid_token" error** -Run `bun run src/commands/auth.ts list` to see active tokens. +Run `gbrain auth list` to see active tokens. **"service_unavailable" error** Database connection failed. Check your Supabase dashboard for outages. diff --git a/docs/mcp/PERPLEXITY.md b/docs/mcp/PERPLEXITY.md index 1a8f5eb..c91520a 100644 --- a/docs/mcp/PERPLEXITY.md +++ b/docs/mcp/PERPLEXITY.md @@ -10,7 +10,7 @@ Perplexity Computer supports remote MCP servers with bearer token authentication - **URL:** `https://YOUR-DOMAIN.ngrok.app/mcp` - **Authentication:** API Key / Bearer Token - **Token:** your GBrain access token - (create one with `bun run src/commands/auth.ts create "perplexity"`) + (create one with `gbrain auth create "perplexity"`) 4. Save Replace `YOUR-DOMAIN` with your ngrok domain (see diff --git a/llms-full.txt b/llms-full.txt index 4a6160c..85f0e9f 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -170,8 +170,11 @@ strict behavior when unset. - `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). - `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman - `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed) -- `src/mcp/server.ts` — MCP stdio server (generated from operations) -- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test) +- `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. +- `src/mcp/dispatch.ts` (v0.22.7) — Shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP (`http-transport.ts`). Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, and `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults to `remote: true` (untrusted); local CLI callers pass `remote: false`. Closed F1 (reversed handler args) + F2 (incomplete OperationContext) + F3 (no param validation) drift bugs in the original v0.22.5 HTTP transport. +- `src/mcp/rate-limit.ts` (v0.22.7) — Bounded-LRU token-bucket limiter for `gbrain serve --http`. `buildDefaultLimiters()` returns the two-bucket pipeline used by http-transport: pre-auth IP (default 30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is actually capped) + post-auth token-id (default 60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap (default 10K keys) bounds memory under attacker-controlled key growth; TTL prune at 2× window evicts abandoned buckets. +- `src/mcp/http-transport.ts` (v0.22.7, rewrite) — `gbrain serve --http` HTTP transport. Postgres-only — fails fast at startup on PGLite (the `access_tokens` table only exists on Postgres). Bearer auth against SHA-256 hashes in `access_tokens`. CORS default-deny via `GBRAIN_HTTP_CORS_ORIGIN` allowlist. Body cap stream-counted (1 MiB default via `GBRAIN_HTTP_MAX_BODY_BYTES`) so chunked transfers without Content-Length still hit the cap. `last_used_at` SQL-level debounce (one UPDATE per token per 60s). Per-request audit row in `mcp_request_log` with token_name + operation + status + latency. Optional `GBRAIN_HTTP_TRUST_PROXY=1` honors `X-Forwarded-For` — only safe when bound to a private interface AND the proxy strips client-supplied XFF (otherwise enables IP spoofing past the pre-auth rate limit). `/health` does `SELECT 1` against Postgres and returns 503 + `status:unhealthy` when the DB is unreachable so orchestration doesn't see green pods while clients get misleading 401s. Replaces the standalone OAuth wrapper that was vulnerable to unauthenticated client registration. +- `src/commands/auth.ts` — Token management for the HTTP transport. `gbrain auth create/list/revoke/test`. As of v0.22.7 wired into the main CLI (`src/cli.ts`); also runs standalone via `bun run src/commands/auth.ts ...` for environments without a compiled binary. Tokens stored as SHA-256 hashes in `access_tokens` (Postgres-only). - `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. @@ -356,7 +359,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/skill-manifest.test.ts` (v0.19 skill manifest parser: drift detection, managed-block markers), `test/skillify-scaffold.test.ts` (v0.19 `gbrain skillify scaffold` stubs: SKILL.md, script, tests, routing-eval fixtures), `test/skillpack-install.test.ts` (v0.19 `gbrain skillpack install` managed-block install / update / no-clobber semantics), -`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source). +`test/skillpack-sync-guard.test.ts` (v0.19 sync-guard: bundled skills stay byte-identical to `skills/` source), +`test/http-transport.test.ts` (v0.22.7 HTTP transport: 23 unit cases covering bearer auth + missing/no-Bearer/unknown/revoked + `/health` bypass, F1+F2 round-trip via dispatch.ts, F3 invalid_params, application/json response shape (not SSE), CORS default-deny + allowlist, body cap on Content-Length AND chunked, two-bucket rate limit (refill, exhaust+Retry-After, LRU eviction, TTL prune, pre-auth IP fires before DB), and `mcp_request_log` audit on success + auth_failed). E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`. - `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage. @@ -371,6 +375,7 @@ E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_U - `test/e2e/search-exclude.test.ts` (v0.22.0) — verifies `test/` + `archive/` pages are hidden by default, that `include_slug_prefixes` opts back in, and that caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths covered. - `test/e2e/engine-parity.test.ts` (v0.22.0) — Postgres ↔ PGLite top-result and result-set parity for `searchKeyword` + `searchVector`. Codex flagged that Postgres ranks pages then picks best chunk while PGLite returns chunks directly — without parity coverage the source-boost fix could pass on PGLite and fail on Postgres. Skips gracefully when `DATABASE_URL` is unset. - `test/e2e/postgres-bootstrap.test.ts` (v0.22.6.1) — exercises `PostgresEngine.initSchema()` directly against a fresh real Postgres database. Asserts the bootstrap path is no-op on fresh installs and that SCHEMA_SQL replays cleanly through the engine path (not via the standalone `db.initSchema` from `src/core/db.ts`, which would have produced false-positive coverage). Codex caught the E2E-shape gap during plan review. +- `test/e2e/http-transport.test.ts` (v0.22.7) — 8 cases against real Postgres covering `gbrain serve --http` end-to-end: bearer auth round-trip, `last_used_at` SQL-level debounce semantics, `mcp_request_log` row insertion on success and auth_failed paths, `/health` DB-down → 503 (DB-probing health check), and the F1+F2+F3 dispatch round-trip with a real operation. Skips gracefully when `DATABASE_URL` is unset. - Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI - If `.env.testing` doesn't exist in this directory, check sibling worktrees for one: `find ../ -maxdepth 2 -name .env.testing -print -quit` and copy it here if found. @@ -1351,12 +1356,13 @@ Add to `~/.claude/server.json` (Claude Code), Settings > MCP Servers (Cursor), o ### Remote MCP (Claude Desktop, Cowork, Perplexity) ```bash -ngrok http 8787 --url your-brain.ngrok.app -bun run src/commands/auth.ts create "claude-desktop" +gbrain auth create "claude-desktop" # tokens via the existing CLI +gbrain serve --http --port 8787 # built-in HTTP transport (Postgres-only) +ngrok http 8787 --url your-brain.ngrok.app # any tunnel works claude mcp add gbrain -t http https://your-brain.ngrok.app/mcp -H "Authorization: Bearer TOKEN" ``` -Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). ChatGPT requires OAuth 2.1 (not yet implemented). +Per-client guides: [`docs/mcp/`](docs/mcp/DEPLOY.md). Hardening defaults, env vars, and threat model: [SECURITY.md](SECURITY.md). ChatGPT requires OAuth 2.1 (not yet implemented). ### Using gbrain with GStack @@ -1928,6 +1934,8 @@ ADMIN gbrain doctor --locks List idle-in-tx backends (57014 diagnostic, Postgres only) gbrain stats Brain statistics gbrain serve MCP server (stdio) + gbrain serve --http --port 8787 MCP server (HTTP, Postgres-only, bearer auth) + gbrain auth create|list|revoke|test Token management for the HTTP transport gbrain integrations Integration recipe dashboard gbrain sources list|add|remove|... Multi-source brain management (v0.18) gbrain dream [--dry-run] [--phase N] One maintenance cycle then exit (cron-friendly) @@ -4108,9 +4116,14 @@ Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY # Deploy GBrain Remote MCP Server +> **v0.22.7+:** Use `gbrain serve --http` for remote access. It includes built-in +> bearer token auth, default-deny CORS, two-bucket rate limiting, body cap, and +> per-request audit log. **Postgres-only** (PGLite is local-only by design). +> See [SECURITY.md](../../SECURITY.md) for env vars and tunable defaults. + Access your brain from any device, any AI client. GBrain's MCP server runs locally -via `gbrain serve` (stdio). For remote access, wrap it in an HTTP server behind a -public tunnel. +via `gbrain serve` (stdio). For remote access, expose it via the built-in HTTP +transport behind a public tunnel. ## Two Paths @@ -4121,21 +4134,23 @@ gbrain serve ``` Works with Claude Code, Cursor, Windsurf, and any MCP client that supports stdio. -No server, no tunnel, no token needed. +No server, no tunnel, no token needed. Works on both PGLite and Postgres engines. -### Remote (any device, any AI client) +### Remote (any device, any AI client) — Postgres only ``` Your AI client (Claude Desktop, Perplexity, etc.) → ngrok tunnel (https://YOUR-DOMAIN.ngrok.app) - → Your HTTP server (wraps gbrain serve) - → Supabase Postgres (via pooler connection string) + → gbrain serve --http (built-in transport with bearer auth) + → Postgres (pooler connection or self-hosted) ``` This requires: -1. A machine running `gbrain serve` behind an HTTP wrapper -2. A public tunnel (ngrok, Tailscale, or cloud host) -3. Bearer token auth for security +1. A Postgres-backed brain (the `access_tokens` table only exists on Postgres; + running `gbrain serve --http` against a PGLite install fails fast at startup) +2. A machine running `gbrain serve --http` +3. A public tunnel (ngrok, Tailscale, or cloud host) +4. A bearer token created via `gbrain auth create ` ## Remote Setup @@ -4154,13 +4169,13 @@ ngrok http 8787 --url your-brain.ngrok.app # Hobby tier for fixed domain ```bash # Create a token for each client -bun run src/commands/auth.ts create "claude-desktop" +gbrain auth create "claude-desktop" # List all tokens -bun run src/commands/auth.ts list +gbrain auth list # Revoke a token -bun run src/commands/auth.ts revoke "claude-desktop" +gbrain auth revoke "claude-desktop" ``` Tokens are per-client. Create one for each device/app. Revoke individually @@ -4176,7 +4191,7 @@ if compromised. Tokens are stored SHA-256 hashed in your database. ### 4. Verify ```bash -bun run src/commands/auth.ts test \ +gbrain auth test \ https://YOUR-DOMAIN.ngrok.app/mcp \ --token YOUR_TOKEN ``` @@ -4204,7 +4219,7 @@ Funnel, and cloud hosts (Fly.io, Railway). Include the Authorization header: `Authorization: Bearer YOUR_TOKEN` **"invalid_token" error** -Run `bun run src/commands/auth.ts list` to see active tokens. +Run `gbrain auth list` to see active tokens. **"service_unavailable" error** Database connection failed. Check your Supabase dashboard for outages. diff --git a/package.json b/package.json index 9c76e7d..ffd929c 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.22.6.1", + "version": "0.22.7", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/cli.ts b/src/cli.ts index 629b8ed..012e3dc 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -19,7 +19,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter']); +const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth']); async function main() { // Parse global flags (--quiet / --progress-json / --progress-interval) @@ -285,6 +285,11 @@ async function handleCliOnly(command: string, args: string[]) { await runIntegrations(args); return; } + if (command === 'auth') { + const { runAuth } = await import('./commands/auth.ts'); + await runAuth(args); + return; + } if (command === 'resolvers') { const { runResolvers } = await import('./commands/resolvers.ts'); await runResolvers(args); @@ -447,7 +452,7 @@ async function handleCliOnly(command: string, args: string[]) { } case 'serve': { const { runServe } = await import('./commands/serve.ts'); - await runServe(engine); + await runServe(engine, args); return; // serve doesn't disconnect } case 'call': { diff --git a/src/commands/auth.ts b/src/commands/auth.ts index 969b063..012a42f 100644 --- a/src/commands/auth.ts +++ b/src/commands/auth.ts @@ -1,20 +1,29 @@ #!/usr/bin/env bun /** - * GBrain token management — standalone script, no gbrain CLI dependency. + * GBrain token management. * - * Usage: + * Wired into the CLI as of v0.22.5: + * gbrain auth create "claude-desktop" + * gbrain auth list + * gbrain auth revoke "claude-desktop" + * gbrain auth test --token + * + * Also runs standalone (no compiled binary required): * DATABASE_URL=... bun run src/commands/auth.ts create "claude-desktop" - * DATABASE_URL=... bun run src/commands/auth.ts list - * DATABASE_URL=... bun run src/commands/auth.ts revoke "claude-desktop" - * DATABASE_URL=... bun run src/commands/auth.ts test --token + * + * Both paths require DATABASE_URL or GBRAIN_DATABASE_URL (except `test`, + * which only hits the remote URL and doesn't need a local DB). */ import postgres from 'postgres'; import { createHash, randomBytes } from 'crypto'; -const DATABASE_URL = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL; -if (!DATABASE_URL && process.argv[2] !== 'test') { - console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.'); - process.exit(1); +function getDatabaseUrl(requireDb: boolean): string | undefined { + const url = process.env.DATABASE_URL || process.env.GBRAIN_DATABASE_URL; + if (!url && requireDb) { + console.error('Set DATABASE_URL or GBRAIN_DATABASE_URL environment variable.'); + process.exit(1); + } + return url; } function hashToken(token: string): string { @@ -27,7 +36,7 @@ function generateToken(): string { async function create(name: string) { if (!name) { console.error('Usage: auth create '); process.exit(1); } - const sql = postgres(DATABASE_URL!); + const sql = postgres(getDatabaseUrl(true)!); const token = generateToken(); const hash = hashToken(token); @@ -53,7 +62,7 @@ async function create(name: string) { } async function list() { - const sql = postgres(DATABASE_URL!); + const sql = postgres(getDatabaseUrl(true)!); try { const rows = await sql` SELECT name, created_at, last_used_at, revoked_at @@ -80,7 +89,7 @@ async function list() { async function revoke(name: string) { if (!name) { console.error('Usage: auth revoke '); process.exit(1); } - const sql = postgres(DATABASE_URL!); + const sql = postgres(getDatabaseUrl(true)!); try { const result = await sql` UPDATE access_tokens SET revoked_at = now() @@ -216,26 +225,38 @@ async function test(url: string, token: string) { console.log(`\n🧠 Your brain is live! (${elapsed}s)`); } -// CLI dispatch -const [cmd, ...args] = process.argv.slice(2); -switch (cmd) { - case 'create': await create(args[0]); break; - case 'list': await list(); break; - case 'revoke': await revoke(args[0]); break; - case 'test': { - const tokenIdx = args.indexOf('--token'); - const url = args.find(a => !a.startsWith('--') && a !== args[tokenIdx + 1]); - const token = tokenIdx >= 0 ? args[tokenIdx + 1] : ''; - await test(url || '', token || ''); - break; - } - default: - console.log(`GBrain Token Management +/** + * Entry point for the `gbrain auth` CLI subcommand. Also reused by the + * direct-script path (see bottom of file) so `bun run src/commands/auth.ts` + * still works. + */ +export async function runAuth(args: string[]): Promise { + const [cmd, ...rest] = args; + switch (cmd) { + case 'create': await create(rest[0]); return; + case 'list': await list(); return; + case 'revoke': await revoke(rest[0]); return; + case 'test': { + const tokenIdx = rest.indexOf('--token'); + const url = rest.find(a => !a.startsWith('--') && a !== rest[tokenIdx + 1]); + const token = tokenIdx >= 0 ? rest[tokenIdx + 1] : ''; + await test(url || '', token || ''); + return; + } + default: + console.log(`GBrain Token Management Usage: - bun run src/commands/auth.ts create Create a new access token - bun run src/commands/auth.ts list List all tokens - bun run src/commands/auth.ts revoke Revoke a token - bun run src/commands/auth.ts test --token Smoke test a remote MCP server + gbrain auth create Create a new access token + gbrain auth list List all tokens + gbrain auth revoke Revoke a token + gbrain auth test --token Smoke-test a remote MCP server `); + } +} + +// Direct-script entry point — only runs when this file is invoked as the main module +// (e.g. `bun run src/commands/auth.ts ...`). When imported by cli.ts, this block is skipped. +if (import.meta.main) { + await runAuth(process.argv.slice(2)); } diff --git a/src/commands/serve.ts b/src/commands/serve.ts index 5e5d7f4..6625716 100644 --- a/src/commands/serve.ts +++ b/src/commands/serve.ts @@ -1,7 +1,19 @@ import type { BrainEngine } from '../core/engine.ts'; import { startMcpServer } from '../mcp/server.ts'; +import { startHttpTransport } from '../mcp/http-transport.ts'; -export async function runServe(engine: BrainEngine) { - console.error('Starting GBrain MCP server (stdio)...'); - await startMcpServer(engine); +export async function runServe(engine: BrainEngine, args: string[] = []) { + const useHttp = args.includes('--http'); + const portIdx = args.indexOf('--port'); + const port = portIdx >= 0 ? parseInt(args[portIdx + 1]) || 8787 : 8787; + + if (useHttp) { + console.error(`Starting GBrain MCP server (HTTP on port ${port})...`); + await startHttpTransport({ port, engine }); + // Keep alive + await new Promise(() => {}); + } else { + console.error('Starting GBrain MCP server (stdio)...'); + await startMcpServer(engine); + } } diff --git a/src/mcp/dispatch.ts b/src/mcp/dispatch.ts new file mode 100644 index 0000000..2ebac0f --- /dev/null +++ b/src/mcp/dispatch.ts @@ -0,0 +1,103 @@ +/** + * Shared MCP tool-call dispatch — single source of truth for stdio + HTTP transports. + * + * Both transports validate the same params, build the same OperationContext shape, + * and serialize errors identically. Drift between transports caused PR #483's reversed-args + * + missing-context bugs; this module exists to prevent that recurring. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { operations, OperationError } from '../core/operations.ts'; +import type { Operation, OperationContext } from '../core/operations.ts'; +import { loadConfig } from '../core/config.ts'; + +export interface ToolResult { + content: { type: 'text'; text: string }[]; + isError?: boolean; +} + +export interface DispatchOpts { + /** Defaults to true (remote/untrusted). Local CLI callers (`gbrain call`) pass false. */ + remote?: boolean; + /** Override the default stderr logger (e.g. CLI uses console.* directly). */ + logger?: OperationContext['logger']; +} + +/** Validate required params exist and have the expected type. Returns null on success, error message on failure. */ +export function validateParams(op: Operation, params: Record): string | null { + for (const [key, def] of Object.entries(op.params)) { + if (def.required && (params[key] === undefined || params[key] === null)) { + return `Missing required parameter: ${key}`; + } + if (params[key] !== undefined && params[key] !== null) { + const val = params[key]; + const expected = def.type; + if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`; + if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`; + if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`; + if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`; + if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`; + } + } + return null; +} + +const stderrLogger: OperationContext['logger'] = { + info: (msg: string) => process.stderr.write(`[info] ${msg}\n`), + warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`), + error: (msg: string) => process.stderr.write(`[error] ${msg}\n`), +}; + +export function buildOperationContext( + engine: BrainEngine, + params: Record, + opts: DispatchOpts = {}, +): OperationContext { + return { + engine, + config: loadConfig() || { engine: 'postgres' }, + logger: opts.logger || stderrLogger, + dryRun: !!params.dry_run, + remote: opts.remote ?? true, + }; +} + +/** + * Resolve operation, validate params, build context, invoke handler, format result. + * + * Returns a `ToolResult` with the same shape both MCP transports need: + * `{ content: [{ type: 'text', text }], isError?: boolean }`. + */ +export async function dispatchToolCall( + engine: BrainEngine, + name: string, + params: Record | undefined, + opts: DispatchOpts = {}, +): Promise { + const op = operations.find(o => o.name === name); + if (!op) { + return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true }; + } + + const safeParams = params || {}; + const validationError = validateParams(op, safeParams); + if (validationError) { + return { + content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }], + isError: true, + }; + } + + const ctx = buildOperationContext(engine, safeParams, opts); + + try { + const result = await op.handler(ctx, safeParams); + return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] }; + } catch (e: unknown) { + if (e instanceof OperationError) { + return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true }; + } + const msg = e instanceof Error ? e.message : String(e); + return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true }; + } +} diff --git a/src/mcp/http-transport.ts b/src/mcp/http-transport.ts new file mode 100644 index 0000000..36c4cfd --- /dev/null +++ b/src/mcp/http-transport.ts @@ -0,0 +1,354 @@ +/** + * HTTP transport for `gbrain serve --http`. + * + * Postgres-only. PGLite users get a clear fail-fast at startup (the access_tokens + * table doesn't exist on PGLite per pglite-schema.ts). + * + * Security model: + * - Every request must include `Authorization: Bearer ` (except /health) + * - Tokens are validated against SHA-256 hashes in the access_tokens table + * - Create/manage tokens with auth.ts (gbrain auth create/list/revoke) + * - No open OAuth, no client_credentials, no self-service tokens + * + * Hardening: + * - CORS default-deny: allowlist via GBRAIN_HTTP_CORS_ORIGIN (comma-separated) + * - Rate limit: per-IP pre-auth (protects DB from brute-force load) + per-token-id post-auth + * (limits runaway clients). Default 30 req/min per IP, 60 req/min per token. Bounded LRU + * so attacker-controlled keys can't grow memory unbounded. + * - Body cap: 1 MiB default (GBRAIN_HTTP_MAX_BODY_BYTES). Stream-counted, not buffered — + * chunked transfers without Content-Length are still capped. + * - last_used_at debounce: only one UPDATE per token per 60s (SQL-level WHERE clause). + * - mcp_request_log: one row per request with token_name + operation + status + latency. + * + * Replaces the standalone HTTP+OAuth wrapper that was vulnerable to unauthenticated + * client registration (see SECURITY.md). + */ + +import { createHash } from 'crypto'; +import type { BrainEngine } from '../core/engine.ts'; +import { buildToolDefs } from './tool-defs.ts'; +import { operations } from '../core/operations.ts'; +import { VERSION } from '../version.ts'; +import { dispatchToolCall } from './dispatch.ts'; +import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts'; + +const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB + +function hashToken(token: string): string { + return createHash('sha256').update(token).digest('hex'); +} + +function envInt(name: string, fallback: number): number { + const v = process.env[name]; + if (!v) return fallback; + const n = parseInt(v, 10); + return Number.isFinite(n) && n > 0 ? n : fallback; +} + +function parseCorsAllowlist(): Set | null { + const v = process.env.GBRAIN_HTTP_CORS_ORIGIN; + if (!v) return null; + return new Set(v.split(',').map(s => s.trim()).filter(Boolean)); +} + +interface HttpTransportOptions { + port: number; + engine: BrainEngine; + /** Override limiters (for tests). Defaults to env-driven buildDefaultLimiters. */ + limiters?: { ip: RateLimiter; token: RateLimiter }; +} + +interface AuthResult { + ok: boolean; + tokenId?: string; + tokenName?: string; +} + +/** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */ +async function readBodyWithCap(req: Request, cap: number): Promise { + const cl = req.headers.get('content-length'); + if (cl) { + const n = parseInt(cl, 10); + if (Number.isFinite(n) && n > cap) return null; + } + const reader = req.body?.getReader(); + if (!reader) return ''; + const chunks: Uint8Array[] = []; + let total = 0; + while (true) { + const { value, done } = await reader.read(); + if (done) break; + if (!value) continue; + total += value.byteLength; + if (total > cap) { + try { await reader.cancel(); } catch { /* noop */ } + return null; + } + chunks.push(value); + } + // Concatenate without Buffer to keep this Node-vs-Bun-portable. + const merged = new Uint8Array(total); + let offset = 0; + for (const c of chunks) { + merged.set(c, offset); + offset += c.byteLength; + } + return new TextDecoder().decode(merged); +} + +/** Resolve client IP. Honors X-Forwarded-For only when GBRAIN_HTTP_TRUST_PROXY=1. */ +function resolveClientIp(req: Request, server: { requestIP: (r: Request) => { address: string } | null }): string { + if (process.env.GBRAIN_HTTP_TRUST_PROXY === '1') { + const xff = req.headers.get('x-forwarded-for'); + if (xff) { + const first = xff.split(',')[0]?.trim(); + if (first) return first; + } + const xRealIp = req.headers.get('x-real-ip'); + if (xRealIp) return xRealIp.trim(); + } + const sock = server.requestIP(req); + return sock?.address || 'unknown'; +} + +export async function startHttpTransport(opts: HttpTransportOptions) { + const { port, engine } = opts; + + // Fail-fast: HTTP transport requires Postgres because access_tokens / mcp_request_log + // only exist in the Postgres schema (see src/core/pglite-schema.ts:5-6). + if ((engine as { kind?: string }).kind !== 'postgres') { + console.error('Error: gbrain serve --http requires a Postgres engine for remote auth tokens.'); + console.error('PGLite is local-only by design (access_tokens table is Postgres-only).'); + console.error('Either:'); + console.error(' - Use stdio: gbrain serve'); + console.error(' - Migrate to Postgres: gbrain migrate --to supabase'); + process.exit(1); + } + + const sql = (engine as unknown as { sql: any }).sql; + if (!sql) { + console.error('Error: Postgres engine has no .sql client. Engine may not be connected.'); + process.exit(1); + } + + const limiters = opts.limiters || buildDefaultLimiters(); + const bodyCap = envInt('GBRAIN_HTTP_MAX_BODY_BYTES', DEFAULT_BODY_CAP); + const corsAllowlist = parseCorsAllowlist(); + const tools = buildToolDefs(operations); + + function corsHeaders(origin: string | null, extra: Record = {}): Record { + const headers: Record = { ...extra }; + if (corsAllowlist && origin && corsAllowlist.has(origin)) { + headers['Access-Control-Allow-Origin'] = origin; + headers['Vary'] = 'Origin'; + } + return headers; + } + + function corsPreflightHeaders(origin: string | null): Record { + const headers: Record = { + 'Access-Control-Allow-Methods': 'POST, GET, OPTIONS', + 'Access-Control-Allow-Headers': 'Content-Type, Authorization, Accept', + }; + if (corsAllowlist && origin && corsAllowlist.has(origin)) { + headers['Access-Control-Allow-Origin'] = origin; + headers['Vary'] = 'Origin'; + } + return headers; + } + + async function validateToken(authHeader: string | null): Promise { + if (!authHeader?.startsWith('Bearer ')) return { ok: false }; + const token = authHeader.slice(7); + const hash = hashToken(token); + try { + const [row] = await sql` + SELECT id, name FROM access_tokens + WHERE token_hash = ${hash} AND revoked_at IS NULL + `; + if (!row) return { ok: false }; + // Debounced last_used_at update — only writes once per token per 60s. + // SQL-level WHERE clause keeps this race-tolerant even under concurrent requests. + sql`UPDATE access_tokens + SET last_used_at = now() + WHERE id = ${row.id} + AND (last_used_at IS NULL OR last_used_at < now() - interval '60 seconds')` + .catch(() => { /* fire-and-forget */ }); + return { ok: true, tokenId: row.id, tokenName: row.name }; + } catch { + return { ok: false }; + } + } + + function logRequest(tokenName: string | null, operation: string, status: string, latencyMs: number) { + sql`INSERT INTO mcp_request_log (token_name, operation, latency_ms, status) + VALUES (${tokenName}, ${operation}, ${latencyMs}, ${status})` + .catch(() => { /* best-effort */ }); + } + + const server = Bun.serve({ + port, + async fetch(req, server) { + const startedMs = Date.now(); + const url = new URL(req.url); + const path = url.pathname; + const origin = req.headers.get('origin'); + + // CORS preflight + if (req.method === 'OPTIONS') { + return new Response(null, { headers: corsPreflightHeaders(origin) }); + } + + // Health check — no auth, no rate limit. Probes the DB so orchestration + // doesn't see "ok" while clients are getting misleading 401s during a DB outage. + if (path === '/health') { + try { + await sql`SELECT 1`; + return Response.json( + { status: 'ok', version: VERSION, transport: 'http', db: 'ok' }, + { headers: corsHeaders(origin) }, + ); + } catch (e: any) { + return Response.json( + { status: 'unhealthy', version: VERSION, transport: 'http', db: 'unreachable', error: e?.message ?? 'unknown' }, + { status: 503, headers: corsHeaders(origin) }, + ); + } + } + + if (path !== '/mcp') { + return Response.json({ error: 'not_found' }, { status: 404, headers: corsHeaders(origin) }); + } + if (req.method !== 'POST') { + return Response.json({ error: 'method_not_allowed' }, { status: 405, headers: corsHeaders(origin) }); + } + + const ip = resolveClientIp(req, server); + + // Pre-auth IP rate limit. Fires BEFORE the DB lookup so we actually limit brute-force load. + const ipCheck = limiters.ip.check(ip); + if (!ipCheck.allowed) { + logRequest(null, 'unknown', 'rate_limited', Date.now() - startedMs); + return Response.json( + { error: 'rate_limited', message: 'Too many requests' }, + { + status: 429, + headers: corsHeaders(origin, { 'Retry-After': String(ipCheck.retryAfter ?? 60) }), + }, + ); + } + + // Body cap (stream-counted; chunked transfers caught here, not at req.json). + const bodyText = await readBodyWithCap(req, bodyCap); + if (bodyText === null) { + logRequest(null, 'unknown', 'body_too_large', Date.now() - startedMs); + return Response.json( + { error: 'payload_too_large', message: `Request body exceeds ${bodyCap} bytes` }, + { status: 413, headers: corsHeaders(origin) }, + ); + } + + // Auth. + const auth = await validateToken(req.headers.get('Authorization')); + if (!auth.ok) { + logRequest(null, 'unknown', 'auth_failed', Date.now() - startedMs); + return Response.json( + { error: 'invalid_token', message: 'Bearer token required. Create one: gbrain auth create ' }, + { status: 401, headers: corsHeaders(origin) }, + ); + } + + // Post-auth token-id rate limit. Limits runaway authed clients. + const tokCheck = limiters.token.check(auth.tokenId!); + if (!tokCheck.allowed) { + logRequest(auth.tokenName!, 'unknown', 'rate_limited', Date.now() - startedMs); + return Response.json( + { error: 'rate_limited', message: 'Too many requests for this token' }, + { + status: 429, + headers: corsHeaders(origin, { 'Retry-After': String(tokCheck.retryAfter ?? 60) }), + }, + ); + } + + // Parse JSON-RPC body. + let body: { method?: string; params?: any; id?: any }; + try { + body = JSON.parse(bodyText); + } catch (e: any) { + logRequest(auth.tokenName!, 'unknown', 'parse_error', Date.now() - startedMs); + return Response.json( + { error: 'parse_error', message: e?.message ?? 'invalid JSON' }, + { status: 400, headers: corsHeaders(origin) }, + ); + } + + const { method, params, id } = body; + + // initialize + if (method === 'initialize') { + logRequest(auth.tokenName!, 'initialize', 'success', Date.now() - startedMs); + return Response.json( + { + result: { + protocolVersion: '2025-03-26', + serverInfo: { name: 'gbrain', version: VERSION }, + capabilities: { tools: {} }, + }, + jsonrpc: '2.0', + id, + }, + { headers: corsHeaders(origin) }, + ); + } + + // notifications/initialized — acknowledge with 204 + if (method === 'notifications/initialized') { + return new Response(null, { status: 204, headers: corsHeaders(origin) }); + } + + // tools/list + if (method === 'tools/list') { + logRequest(auth.tokenName!, 'tools/list', 'success', Date.now() - startedMs); + return Response.json( + { result: { tools }, jsonrpc: '2.0', id }, + { headers: corsHeaders(origin) }, + ); + } + + // tools/call — dispatch through shared dispatch.ts (parity with stdio) + if (method === 'tools/call') { + const toolName: string = params?.name ?? 'unknown'; + const args: Record = params?.arguments ?? {}; + const result = await dispatchToolCall(engine, toolName, args, { remote: true }); + const status = result.isError ? 'error' : 'success'; + logRequest(auth.tokenName!, `tools/call:${toolName}`, status, Date.now() - startedMs); + return Response.json( + { result, jsonrpc: '2.0', id }, + { headers: corsHeaders(origin) }, + ); + } + + logRequest(auth.tokenName!, method ?? 'unknown', 'unknown_method', Date.now() - startedMs); + return Response.json( + { error: 'unknown_method', message: `Unknown method: ${method}` }, + { status: 400, headers: corsHeaders(origin) }, + ); + }, + }); + + console.error(`GBrain HTTP MCP server running on port ${port}`); + console.error(` Health: http://localhost:${port}/health`); + console.error(` MCP: http://localhost:${port}/mcp`); + console.error(` Auth: Bearer token required (create with: gbrain auth create )`); + if (!corsAllowlist) { + console.error(' CORS: default-deny. Set GBRAIN_HTTP_CORS_ORIGIN=https://your.app to allow browser clients.'); + } else { + console.error(` CORS: allowlist = ${[...corsAllowlist].join(', ')}`); + } + console.error(''); + console.error('⚠️ Do NOT use open OAuth registration for remote MCP access.'); + console.error(' Tokens are managed via: gbrain auth create/list/revoke'); + + return server; +} diff --git a/src/mcp/rate-limit.ts b/src/mcp/rate-limit.ts new file mode 100644 index 0000000..6ec9579 --- /dev/null +++ b/src/mcp/rate-limit.ts @@ -0,0 +1,142 @@ +/** + * Rate limiter for `gbrain serve --http`. + * + * Token-bucket per key, stored in a bounded LRU map so attacker-controlled keys + * can't grow memory unbounded. TTL prune on every access (entries older than + * 2× window are evicted) so abandoned keys don't sit around forever. + * + * Two buckets in the request pipeline (see http-transport.ts): + * 1. Pre-auth IP bucket — fires BEFORE the DB lookup so we actually limit + * brute-force load against access_tokens, not just response codes. + * 2. Post-auth token-id bucket — fires after auth so legitimate-but-runaway + * clients get throttled at the right principal. + * + * Both buckets behave identically; only the key differs. + */ + +export interface RateLimitOpts { + /** Maximum requests in the window. */ + limit: number; + /** Window length in milliseconds. */ + windowMs: number; + /** LRU cap on distinct keys. Evicts least-recently-used on overflow. */ + lruCap: number; +} + +export interface RateLimitResult { + allowed: boolean; + /** Seconds until next request would be allowed (only set when !allowed). */ + retryAfter?: number; + /** Tokens remaining in the bucket after this check. */ + remaining: number; +} + +interface Bucket { + tokens: number; + /** Used for refill math: tokens accrue based on elapsed time since this. */ + lastRefillMs: number; + /** Used for TTL eviction: time of last check, regardless of refill. Prevents bucket-reset attack + * where an exhausted key would otherwise get TTL-evicted and recreated fresh. */ + lastTouchedMs: number; +} + +/** Clock function — defaults to Date.now, overridable for tests. */ +type Clock = () => number; + +export class RateLimiter { + readonly opts: RateLimitOpts; + private readonly buckets: Map = new Map(); + private readonly clock: Clock; + + constructor(opts: RateLimitOpts, clock: Clock = Date.now) { + if (opts.limit <= 0) throw new Error('RateLimiter: limit must be > 0'); + if (opts.windowMs <= 0) throw new Error('RateLimiter: windowMs must be > 0'); + if (opts.lruCap <= 0) throw new Error('RateLimiter: lruCap must be > 0'); + this.opts = opts; + this.clock = clock; + } + + check(key: string): RateLimitResult { + const now = this.clock(); + this.prune(now); + + let bucket = this.buckets.get(key); + if (!bucket) { + bucket = { tokens: this.opts.limit, lastRefillMs: now, lastTouchedMs: now }; + } else { + // Refill: tokens accrue continuously over the window. limit/windowMs tokens per ms. + const elapsed = now - bucket.lastRefillMs; + const refilled = Math.floor((elapsed * this.opts.limit) / this.opts.windowMs); + if (refilled > 0) { + bucket.tokens = Math.min(this.opts.limit, bucket.tokens + refilled); + bucket.lastRefillMs = now; + } + bucket.lastTouchedMs = now; + // LRU bookkeeping: re-insert to move to end (Map iteration order = insertion). + this.buckets.delete(key); + } + + if (bucket.tokens > 0) { + bucket.tokens -= 1; + this.buckets.set(key, bucket); + this.evictIfOver(); + return { allowed: true, remaining: bucket.tokens }; + } + + // No tokens. Compute Retry-After from when the next token will accrue. + const msPerToken = this.opts.windowMs / this.opts.limit; + const msUntilNext = msPerToken - (now - bucket.lastRefillMs); + const retryAfter = Math.max(1, Math.ceil(msUntilNext / 1000)); + this.buckets.set(key, bucket); + this.evictIfOver(); + return { allowed: false, retryAfter, remaining: 0 }; + } + + /** Evict TTL-expired entries (older than 2× window since last touch). Cheap: O(n) but n is bounded by lruCap. + * Uses lastTouchedMs (not lastRefillMs) so an attacker can't reset their bucket by hammering an exhausted key + * past the TTL — every check updates lastTouchedMs even when refill produces 0 tokens. */ + private prune(now: number): void { + const ttl = this.opts.windowMs * 2; + for (const [key, bucket] of this.buckets) { + if (now - bucket.lastTouchedMs > ttl) { + this.buckets.delete(key); + } else { + // Map iteration is in insertion order; once we hit a fresh entry, the rest are also fresh + // ONLY if we maintain insertion-order = recency. That holds because check() does delete+set on every call. + break; + } + } + } + + private evictIfOver(): void { + while (this.buckets.size > this.opts.lruCap) { + // Map iteration starts at oldest (first-inserted). Delete it. + const oldestKey = this.buckets.keys().next().value; + if (oldestKey === undefined) break; + this.buckets.delete(oldestKey); + } + } + + /** Test helper: current key count. */ + get size(): number { + return this.buckets.size; + } +} + +/** Parse a positive integer env var, falling back to default. */ +function envInt(name: string, fallback: number): number { + const v = process.env[name]; + if (!v) return fallback; + const n = parseInt(v, 10); + return Number.isFinite(n) && n > 0 ? n : fallback; +} + +/** Build limiters from env. Keep this lazy — tests can construct RateLimiter directly. */ +export function buildDefaultLimiters(clock: Clock = Date.now): { ip: RateLimiter; token: RateLimiter } { + const lruCap = envInt('GBRAIN_HTTP_RATE_LIMIT_LRU', 10000); + const windowMs = 60_000; + return { + ip: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_IP', 30), windowMs, lruCap }, clock), + token: new RateLimiter({ limit: envInt('GBRAIN_HTTP_RATE_LIMIT_TOKEN', 60), windowMs, lruCap }, clock), + }; +} diff --git a/src/mcp/server.ts b/src/mcp/server.ts index c9a3260..e0d4eb7 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -2,30 +2,10 @@ import { Server } from '@modelcontextprotocol/sdk/server/index.js'; import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'; import { ListToolsRequestSchema, CallToolRequestSchema } from '@modelcontextprotocol/sdk/types.js'; import type { BrainEngine } from '../core/engine.ts'; -import { operations, OperationError } from '../core/operations.ts'; -import type { Operation, OperationContext } from '../core/operations.ts'; -import { loadConfig } from '../core/config.ts'; +import { operations } from '../core/operations.ts'; import { VERSION } from '../version.ts'; import { buildToolDefs } from './tool-defs.ts'; - -/** Validate required params exist and have the expected type */ -function validateParams(op: Operation, params: Record): string | null { - for (const [key, def] of Object.entries(op.params)) { - if (def.required && (params[key] === undefined || params[key] === null)) { - return `Missing required parameter: ${key}`; - } - if (params[key] !== undefined && params[key] !== null) { - const val = params[key]; - const expected = def.type; - if (expected === 'string' && typeof val !== 'string') return `Parameter "${key}" must be a string`; - if (expected === 'number' && typeof val !== 'number') return `Parameter "${key}" must be a number`; - if (expected === 'boolean' && typeof val !== 'boolean') return `Parameter "${key}" must be a boolean`; - if (expected === 'object' && (typeof val !== 'object' || Array.isArray(val))) return `Parameter "${key}" must be an object`; - if (expected === 'array' && !Array.isArray(val)) return `Parameter "${key}" must be an array`; - } - } - return null; -} +import { dispatchToolCall, validateParams, buildOperationContext } from './dispatch.ts'; export async function startMcpServer(engine: BrainEngine) { const server = new Server( @@ -40,50 +20,21 @@ export async function startMcpServer(engine: BrainEngine) { tools: buildToolDefs(operations), })); - // Dispatch tool calls to operation handlers - server.setRequestHandler(CallToolRequestSchema, async (request: any) => { + // Dispatch tool calls via shared dispatch.ts (parity with HTTP transport). + // MCP stdio callers are remote/untrusted; dispatch defaults remote=true. + // The MCP SDK's response type widened in 1.29 to allow a managed-task wrapper; + // gbrain ops are synchronous, so we return the legacy `{ content, isError? }` + // shape and cast through `any` (the SDK accepts it via the ServerResult union). + server.setRequestHandler(CallToolRequestSchema, async (request: any): Promise => { const { name, arguments: params } = request.params; - const op = operations.find(o => o.name === name); - if (!op) { - return { content: [{ type: 'text', text: `Error: Unknown tool: ${name}` }], isError: true }; - } - - const ctx: OperationContext = { - engine, - config: loadConfig() || { engine: 'postgres' }, - logger: { - info: (msg: string) => process.stderr.write(`[info] ${msg}\n`), - warn: (msg: string) => process.stderr.write(`[warn] ${msg}\n`), - error: (msg: string) => process.stderr.write(`[error] ${msg}\n`), - }, - dryRun: !!(params?.dry_run), - // MCP stdio callers are remote/untrusted; enforce strict file confinement. - remote: true, - }; - - const safeParams = params || {}; - const validationError = validateParams(op, safeParams); - if (validationError) { - return { content: [{ type: 'text', text: JSON.stringify({ error: 'invalid_params', message: validationError }, null, 2) }], isError: true }; - } - - try { - const result = await op.handler(ctx, safeParams); - return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] }; - } catch (e: unknown) { - if (e instanceof OperationError) { - return { content: [{ type: 'text', text: JSON.stringify(e.toJSON(), null, 2) }], isError: true }; - } - const msg = e instanceof Error ? e.message : String(e); - return { content: [{ type: 'text', text: `Error: ${msg}` }], isError: true }; - } + return dispatchToolCall(engine, name, params, { remote: true }); }); const transport = new StdioServerTransport(); await server.connect(transport); } -// Backward compat: used by `gbrain call` command +// Backward compat: used by `gbrain call` command (trusted local path). export async function handleToolCall( engine: BrainEngine, tool: string, @@ -95,14 +46,10 @@ export async function handleToolCall( const validationError = validateParams(op, params); if (validationError) throw new Error(validationError); - const ctx: OperationContext = { - engine, - config: loadConfig() || { engine: 'postgres' }, - logger: { info: console.log, warn: console.warn, error: console.error }, - dryRun: !!(params?.dry_run), - // Backing path for `gbrain call` CLI command — trusted local invocation. + const ctx = buildOperationContext(engine, params, { remote: false, - }; + logger: { info: console.log, warn: console.warn, error: console.error }, + }); return op.handler(ctx, params); } diff --git a/test/e2e/http-transport.test.ts b/test/e2e/http-transport.test.ts new file mode 100644 index 0000000..c2cfe69 --- /dev/null +++ b/test/e2e/http-transport.test.ts @@ -0,0 +1,233 @@ +/** + * E2E tests for src/mcp/http-transport.ts against real Postgres. + * + * Catches schema drift (column-name typos that would slip past the unit suite's + * stubbed engine.sql) and proves the F1+F2+F3 dispatch pipeline works against a + * real handler doing real DB work. Also exercises the SQL-level last_used_at + * debounce against real Postgres semantics. + * + * Run: DATABASE_URL=... bun test test/e2e/http-transport.test.ts + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { createHash, randomBytes } from 'crypto'; +import { startHttpTransport } from '../../src/mcp/http-transport.ts'; +import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts'; + +const skip = !hasDatabase(); +const describeE2E = skip ? describe.skip : describe; + +if (skip) { + console.log('Skipping E2E http-transport tests (DATABASE_URL not set)'); +} + +interface ServerHandle { + port: number; + stop: () => Promise; +} + +function generateToken(): string { + return 'gbrain_test_' + randomBytes(16).toString('hex'); +} + +function hashToken(token: string): string { + return createHash('sha256').update(token).digest('hex'); +} + +async function startServer(): Promise { + const engine = getEngine(); + const server = await startHttpTransport({ port: 0, engine: engine as any }); + return { + port: (server as any).port, + stop: async () => { (server as any).stop(true); }, + }; +} + +function rpc(method: string, params?: unknown, id: number = 1) { + return JSON.stringify({ jsonrpc: '2.0', id, method, ...(params !== undefined ? { params } : {}) }); +} + +describeE2E('http-transport E2E (real Postgres)', () => { + let srv: ServerHandle; + let validToken: string; + let revokedToken: string; + let validTokenName: string; + + beforeAll(async () => { + await setupDB(); + const conn = getConn(); + + // Seed a valid + revoked token directly via SQL (mirrors auth.ts's create path). + validToken = generateToken(); + validTokenName = 'e2e-valid-' + randomBytes(4).toString('hex'); + await conn.unsafe( + 'INSERT INTO access_tokens (name, token_hash) VALUES ($1, $2)', + [validTokenName, hashToken(validToken)], + ); + revokedToken = generateToken(); + await conn.unsafe( + 'INSERT INTO access_tokens (name, token_hash, revoked_at) VALUES ($1, $2, now())', + ['e2e-revoked-' + randomBytes(4).toString('hex'), hashToken(revokedToken)], + ); + + srv = await startServer(); + }); + + afterAll(async () => { + if (srv) await srv.stop(); + await teardownDB(); + }); + + test('1. /health → 200 with expected JSON shape', async () => { + const r = await fetch(`http://localhost:${srv.port}/health`); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.status).toBe('ok'); + expect(body.transport).toBe('http'); + expect(body.version).toBeString(); + }); + + test('2. /mcp tools/list with valid Bearer → 200 + ops list', async () => { + const r = await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.result.tools).toBeArray(); + expect(body.result.tools.length).toBeGreaterThan(5); + expect(r.headers.get('content-type')).toContain('application/json'); + }); + + test('3. /mcp tools/call (real op: list_pages) round-trips successfully — F1+F2+F3 guard', async () => { + const r = await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/call', { name: 'list_pages', arguments: { limit: 5 } }), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.jsonrpc).toBe('2.0'); + expect(body.result.content).toBeArray(); + // Should NOT be an error — handler ran successfully against the real engine. + expect(body.result.isError).toBeUndefined(); + // Result text should parse as JSON (list_pages returns an object/array) + const resultText = body.result.content[0].text; + const parsed = JSON.parse(resultText); + expect(parsed).toBeDefined(); + }); + + test('4. revoked token → 401', async () => { + const r = await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${revokedToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(401); + }); + + test('5. last_used_at debounce: two consecutive valid calls → only one UPDATE within 60s', async () => { + const conn = getConn(); + + // Reset last_used_at to NULL so the first call definitely updates + await conn.unsafe('UPDATE access_tokens SET last_used_at = NULL WHERE name = $1', [validTokenName]); + + // First request — should update last_used_at + await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + // Give the fire-and-forget UPDATE a moment to land + await new Promise(r => setTimeout(r, 50)); + + const [row1] = await conn.unsafe( + 'SELECT last_used_at FROM access_tokens WHERE name = $1', + [validTokenName], + ) as { last_used_at: Date | null }[]; + expect(row1.last_used_at).not.toBeNull(); + const firstUpdate = row1.last_used_at; + + // Second request immediately — should NOT trigger another UPDATE (debounced by SQL WHERE) + await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + await new Promise(r => setTimeout(r, 50)); + + const [row2] = await conn.unsafe( + 'SELECT last_used_at FROM access_tokens WHERE name = $1', + [validTokenName], + ) as { last_used_at: Date | null }[]; + // Same timestamp = same UPDATE = debounce held + expect(row2.last_used_at?.getTime()).toBe(firstUpdate?.getTime()); + }); + + test('6. last_used_at debounce: simulating 65s gap → second request DOES update', async () => { + const conn = getConn(); + + // Set last_used_at to 65 seconds ago — simulates the time gap without waiting in real time + await conn.unsafe( + `UPDATE access_tokens SET last_used_at = now() - interval '65 seconds' WHERE name = $1`, + [validTokenName], + ); + const [before] = await conn.unsafe( + 'SELECT last_used_at FROM access_tokens WHERE name = $1', + [validTokenName], + ) as { last_used_at: Date | null }[]; + + await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + await new Promise(r => setTimeout(r, 50)); + + const [after] = await conn.unsafe( + 'SELECT last_used_at FROM access_tokens WHERE name = $1', + [validTokenName], + ) as { last_used_at: Date | null }[]; + expect(after.last_used_at?.getTime()).toBeGreaterThan(before.last_used_at!.getTime()); + }); + + test('7. mcp_request_log gets a row per request', async () => { + const conn = getConn(); + const beforeRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[]; + const beforeN = beforeRows[0].n; + + await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + // Fire-and-forget audit insert — give it a tick + await new Promise(r => setTimeout(r, 100)); + + const afterRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[]; + expect(afterRows[0].n).toBeGreaterThan(beforeN); + + const [row] = await conn.unsafe( + `SELECT token_name, operation, status, latency_ms FROM mcp_request_log + WHERE token_name = $1 ORDER BY created_at DESC LIMIT 1`, + [validTokenName], + ) as { token_name: string; operation: string; status: string; latency_ms: number }[]; + expect(row.token_name).toBe(validTokenName); + expect(row.operation).toBe('tools/list'); + expect(row.status).toBe('success'); + expect(row.latency_ms).toBeGreaterThanOrEqual(0); + }); + + test('8. tools/call with malformed params → isError result with invalid_params', async () => { + const r = await fetch(`http://localhost:${srv.port}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' }, + body: rpc('tools/call', { name: 'get_page', arguments: { slug: 42 } }), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.result.isError).toBe(true); + expect(body.result.content[0].text).toContain('invalid_params'); + }); +}); diff --git a/test/http-transport.test.ts b/test/http-transport.test.ts new file mode 100644 index 0000000..67c2469 --- /dev/null +++ b/test/http-transport.test.ts @@ -0,0 +1,519 @@ +/** + * Unit tests for src/mcp/http-transport.ts. + * + * Covers: + * - Auth path (valid, missing header, no Bearer prefix, unknown, revoked, /health bypass) + * - F1+F2+F3 round-trip guards (handler arg order, full OperationContext, param validation) + * - JSON-only response shape (no SSE) + * - CORS default-deny + allowlist + * - Body cap (Content-Length + chunked) + * - Rate limit (token + IP buckets, LRU eviction, TTL prune, /health bypass) + * + * No DATABASE_URL needed — engine.sql is mocked. E2E coverage of the real Postgres + * round-trip lives in test/e2e/http-transport.test.ts. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { createHash } from 'crypto'; +import { startHttpTransport } from '../src/mcp/http-transport.ts'; +import { RateLimiter } from '../src/mcp/rate-limit.ts'; + +type SqlResult = unknown[] | unknown; +type SqlHandler = (query: string, values: unknown[]) => SqlResult | Promise; + +interface FakeEngine { + kind: 'postgres'; + sql: ReturnType; + audit: { token_name: string | null; operation: string; status: string; latency_ms: number }[]; +} + +function makeSqlTag(handler: SqlHandler) { + return (strings: TemplateStringsArray, ...values: unknown[]) => { + let query = ''; + for (let i = 0; i < strings.length; i++) { + query += strings[i]; + if (i < values.length) query += '?'; + } + const result = handler(query.trim(), values); + return Promise.resolve(result); + }; +} + +function hash(token: string): string { + return createHash('sha256').update(token).digest('hex'); +} + +interface FakeEngineConfig { + validTokens?: Map; + /** Tokens that are present but revoked (revoked_at IS NOT NULL — query returns empty). */ + revokedTokens?: Set; + /** If true, every SELECT throws (simulating DB outage). */ + dbDown?: boolean; +} + +function makeFakeEngine(cfg: FakeEngineConfig = {}): FakeEngine { + const validTokens = cfg.validTokens ?? new Map(); + const revokedTokens = cfg.revokedTokens ?? new Set(); + const audit: FakeEngine['audit'] = []; + + const sql = makeSqlTag((query, values) => { + if (cfg.dbDown && query.startsWith('SELECT')) throw new Error('db down'); + + if (query === 'SELECT 1') { + // /health DB probe + return [{ '?column?': 1 }]; + } + + if (query.startsWith('SELECT id, name FROM access_tokens')) { + const tokenHash = values[0] as string; + if (revokedTokens.has(tokenHash)) return []; + const row = validTokens.get(tokenHash); + return row ? [row] : []; + } + + if (query.startsWith('UPDATE access_tokens')) { + // last_used_at debounce — succeed silently + return []; + } + + if (query.startsWith('INSERT INTO mcp_request_log')) { + audit.push({ + token_name: values[0] as string | null, + operation: values[1] as string, + latency_ms: values[2] as number, + status: values[3] as string, + }); + return []; + } + + return []; + }); + + return { kind: 'postgres', sql, audit }; +} + +interface TestServer { + url: string; + stop: () => void; + engine: FakeEngine; + ipLimiter: RateLimiter; + tokenLimiter: RateLimiter; +} + +let mockNow = 0; +function freezeClock(at: number) { mockNow = at; } +function advanceClock(deltaMs: number) { mockNow += deltaMs; } + +async function startTest(cfg: FakeEngineConfig & { lruCap?: number; ipLimit?: number; tokenLimit?: number; corsOrigin?: string; bodyCap?: number; trustProxy?: boolean } = {}): Promise { + if (cfg.corsOrigin) process.env.GBRAIN_HTTP_CORS_ORIGIN = cfg.corsOrigin; + else delete process.env.GBRAIN_HTTP_CORS_ORIGIN; + if (cfg.bodyCap) process.env.GBRAIN_HTTP_MAX_BODY_BYTES = String(cfg.bodyCap); + else delete process.env.GBRAIN_HTTP_MAX_BODY_BYTES; + if (cfg.trustProxy) process.env.GBRAIN_HTTP_TRUST_PROXY = '1'; + else delete process.env.GBRAIN_HTTP_TRUST_PROXY; + + const engine = makeFakeEngine(cfg); + const clock = () => mockNow || Date.now(); + const ipLimiter = new RateLimiter( + { limit: cfg.ipLimit ?? 1000, windowMs: 60_000, lruCap: cfg.lruCap ?? 10000 }, + clock, + ); + const tokenLimiter = new RateLimiter( + { limit: cfg.tokenLimit ?? 1000, windowMs: 60_000, lruCap: cfg.lruCap ?? 10000 }, + clock, + ); + const server = await startHttpTransport({ + port: 0, + engine: engine as any, + limiters: { ip: ipLimiter, token: tokenLimiter }, + }); + return { + url: `http://localhost:${(server as any).port}`, + stop: () => (server as any).stop(true), + engine, + ipLimiter, + tokenLimiter, + }; +} + +function rpc(method: string, params?: unknown, id: number = 1) { + return JSON.stringify({ jsonrpc: '2.0', id, method, ...(params !== undefined ? { params } : {}) }); +} + +// -------------------------------------------------------------------------- +// Auth path +// -------------------------------------------------------------------------- + +describe('http-transport: auth', () => { + let srv: TestServer; + const VALID_TOKEN = 'valid-token-abc'; + const REVOKED_TOKEN = 'revoked-token-xyz'; + + beforeAll(async () => { + srv = await startTest({ + validTokens: new Map([[hash(VALID_TOKEN), { id: 'tok-1', name: 'test' }]]), + revokedTokens: new Set([hash(REVOKED_TOKEN)]), + }); + }); + afterAll(() => srv.stop()); + + test('1. valid token → 200 + tools/list returns ops', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${VALID_TOKEN}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.result.tools).toBeArray(); + expect(body.result.tools.length).toBeGreaterThan(0); + expect(body.jsonrpc).toBe('2.0'); + }); + + test('2. missing Authorization header → 401', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(401); + }); + + test('3. header missing Bearer prefix → 401', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': VALID_TOKEN, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(401); + }); + + test('4. unknown token → 401', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': 'Bearer not-a-real-token', 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(401); + }); + + test('5. revoked token → 401', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${REVOKED_TOKEN}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.status).toBe(401); + }); + + test('6. /health → 200 without auth, body has expected fields, probes DB', async () => { + const r = await fetch(`${srv.url}/health`); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.status).toBe('ok'); + expect(body.transport).toBe('http'); + expect(body.version).toBeString(); + expect(body.db).toBe('ok'); + }); + + test('6b. /health → 503 when DB is unreachable', async () => { + const dbDownSrv = await startTest({ dbDown: true }); + try { + const r = await fetch(`${dbDownSrv.url}/health`); + expect(r.status).toBe(503); + const body = await r.json(); + expect(body.status).toBe('unhealthy'); + expect(body.db).toBe('unreachable'); + } finally { dbDownSrv.stop(); } + }); +}); + +// -------------------------------------------------------------------------- +// F1+F2+F3 regression guards (the actual existing-PR bugs) +// -------------------------------------------------------------------------- + +describe('http-transport: tools/call dispatch', () => { + let srv: TestServer; + const TOK = 'tok-fix'; + + beforeAll(async () => { + srv = await startTest({ validTokens: new Map([[hash(TOK), { id: 'tok-fix-id', name: 'fix' }]]) }); + }); + afterAll(() => srv.stop()); + + test('7. tools/call with a real op (list_pages) round-trips successfully (F1+F2 guard)', async () => { + // list_pages doesn't need real DB rows in this stub — it'll call engine methods we don't mock, + // so we expect EITHER a successful tool-result OR an isError result with a meaningful message. + // The point is that the handler IS invoked with (ctx, params) order — not (params, ctx). + // If F1 regressed, the handler would receive {limit: 1} as ctx and crash trying to read ctx.engine. + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + body: rpc('tools/call', { name: 'list_pages', arguments: { limit: 1 } }), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.jsonrpc).toBe('2.0'); + expect(body.result).toBeDefined(); + expect(body.result.content).toBeArray(); + // Either success (handler ran) or a structured error (handler ran and returned an error) + // — both prove dispatch reached the handler with the correct shape. + }); + + test('8. tools/call with malformed params → 200 wrapping an isError result (F3 guard via dispatch.ts)', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + // get_page expects `slug` as required string; passing a number triggers validateParams + body: rpc('tools/call', { name: 'get_page', arguments: { slug: 42 } }), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.result.isError).toBe(true); + const text = body.result.content[0].text; + expect(text).toContain('invalid_params'); + }); + + test('9. /mcp response has Content-Type: application/json (not SSE)', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + body: rpc('tools/list'), + }); + expect(r.headers.get('content-type')).toContain('application/json'); + expect(r.headers.get('content-type')).not.toContain('event-stream'); + }); + + test('9b. unknown tool name → 200 wrapping an isError result', async () => { + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + body: rpc('tools/call', { name: 'definitely_not_a_real_tool', arguments: {} }), + }); + expect(r.status).toBe(200); + const body = await r.json(); + expect(body.result.isError).toBe(true); + expect(body.result.content[0].text).toContain('Unknown tool'); + }); +}); + +// -------------------------------------------------------------------------- +// CORS +// -------------------------------------------------------------------------- + +describe('http-transport: CORS', () => { + test('10. no GBRAIN_HTTP_CORS_ORIGIN + browser request → no ACAO header', async () => { + const srv = await startTest({}); + try { + const r = await fetch(`${srv.url}/health`, { headers: { 'Origin': 'https://evil.example' } }); + expect(r.headers.get('access-control-allow-origin')).toBeNull(); + } finally { srv.stop(); } + }); + + test('11. env set + matching Origin → ACAO echoes', async () => { + const srv = await startTest({ corsOrigin: 'https://claude.ai' }); + try { + const r = await fetch(`${srv.url}/health`, { headers: { 'Origin': 'https://claude.ai' } }); + expect(r.headers.get('access-control-allow-origin')).toBe('https://claude.ai'); + expect(r.headers.get('vary')).toBe('Origin'); + } finally { srv.stop(); } + }); + + test('12. env set + non-matching Origin → no ACAO header', async () => { + const srv = await startTest({ corsOrigin: 'https://claude.ai' }); + try { + const r = await fetch(`${srv.url}/health`, { headers: { 'Origin': 'https://evil.example' } }); + expect(r.headers.get('access-control-allow-origin')).toBeNull(); + } finally { srv.stop(); } + }); +}); + +// -------------------------------------------------------------------------- +// Body cap +// -------------------------------------------------------------------------- + +describe('http-transport: body cap', () => { + const TOK = 'body-cap-tok'; + + test('13. Content-Length over cap → 413', async () => { + const srv = await startTest({ + validTokens: new Map([[hash(TOK), { id: 'b-1', name: 'b' }]]), + bodyCap: 100, + }); + try { + const big = 'x'.repeat(200); + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + body: big, + }); + expect(r.status).toBe(413); + } finally { srv.stop(); } + }); + + test('14. chunked transfer (no Content-Length) over cap → 413', async () => { + const srv = await startTest({ + validTokens: new Map([[hash(TOK), { id: 'b-2', name: 'b' }]]), + bodyCap: 100, + }); + try { + // Build a chunked body via a ReadableStream — Bun fetch sends without Content-Length. + const stream = new ReadableStream({ + start(controller) { + for (let i = 0; i < 10; i++) controller.enqueue(new TextEncoder().encode('y'.repeat(50))); + controller.close(); + }, + }); + const r = await fetch(`${srv.url}/mcp`, { + method: 'POST', + headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, + body: stream as any, + // @ts-expect-error Bun fetch supports duplex for streaming bodies + duplex: 'half', + }); + expect(r.status).toBe(413); + } finally { srv.stop(); } + }); +}); + +// -------------------------------------------------------------------------- +// Rate limit +// -------------------------------------------------------------------------- + +describe('http-transport: rate limit', () => { + const TOK = 'rl-tok'; + + test('15. token bucket: refill mechanic over time', async () => { + freezeClock(1000); + const srv = await startTest({ + validTokens: new Map([[hash(TOK), { id: 'rl-id', name: 'rl' }]]), + tokenLimit: 2, + ipLimit: 100, + }); + try { + // Use up 2 tokens + const ok1 = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(ok1.status).toBe(200); + const ok2 = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(ok2.status).toBe(200); + + // Third should 429 + const blocked = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(blocked.status).toBe(429); + + // Advance past the refill window (60s for 2 limit = 30s/token; advance 35s) + advanceClock(35_000); + const refilled = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(refilled.status).toBe(200); + } finally { srv.stop(); freezeClock(0); } + }); + + test('16. token bucket exhausted → 429 + Retry-After header', async () => { + freezeClock(1000); + const srv = await startTest({ + validTokens: new Map([[hash(TOK), { id: 'rl16', name: 'rl' }]]), + tokenLimit: 1, + ipLimit: 100, + }); + try { + await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + const r = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(r.status).toBe(429); + expect(r.headers.get('retry-after')).not.toBeNull(); + expect(parseInt(r.headers.get('retry-after')!, 10)).toBeGreaterThan(0); + } finally { srv.stop(); freezeClock(0); } + }); + + test('17. LRU eviction at cap (insert > cap evicts LRU)', () => { + let now = 0; + const lim = new RateLimiter({ limit: 10, windowMs: 60_000, lruCap: 3 }, () => now); + lim.check('a'); now += 1; + lim.check('b'); now += 1; + lim.check('c'); now += 1; + expect(lim.size).toBe(3); + lim.check('d'); now += 1; + expect(lim.size).toBe(3); + // 'a' should have been evicted (oldest by insertion). After re-checking 'a' it's a fresh bucket again. + // Easiest verification: hammer 'a' should NOT be already exhausted — fresh bucket starts at limit. + for (let i = 0; i < 10; i++) { + const r = lim.check('a'); + expect(r.allowed).toBe(true); + } + // 11th should fail (no refill since clock barely moved) + expect(lim.check('a').allowed).toBe(false); + }); + + test('18. TTL prune (entries older than 2× window evicted)', () => { + let now = 1000; + const lim = new RateLimiter({ limit: 10, windowMs: 1000, lruCap: 100 }, () => now); + lim.check('stale'); // touched at t=1000 + expect(lim.size).toBe(1); + now = 1000 + 2001; // advance past 2× window + lim.check('fresh'); // triggers prune + expect(lim.size).toBe(1); // 'stale' evicted, only 'fresh' remains + }); + + test('19. pre-auth IP bucket fires BEFORE auth (DB not called when IP exhausted)', async () => { + freezeClock(1000); + const srv = await startTest({ + ipLimit: 1, + tokenLimit: 100, + validTokens: new Map([[hash(TOK), { id: 'rl19', name: 'rl' }]]), + }); + try { + // First request consumes IP token (will hit auth and succeed) + const r1 = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(r1.status).toBe(200); + // Second: IP bucket exhausted. We send WITHOUT auth header. Should be 429 (IP-limited), + // not 401 (auth-failed) — proving IP check happened first. + const r2 = await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + expect(r2.status).toBe(429); + } finally { srv.stop(); freezeClock(0); } + }); + + test('20. /health bypasses rate limit', async () => { + freezeClock(1000); + const srv = await startTest({ ipLimit: 1, tokenLimit: 1 }); + try { + // Hammer health 5 times — none should 429 + for (let i = 0; i < 5; i++) { + const r = await fetch(`${srv.url}/health`); + expect(r.status).toBe(200); + } + } finally { srv.stop(); freezeClock(0); } + }); +}); + +// -------------------------------------------------------------------------- +// mcp_request_log audit +// -------------------------------------------------------------------------- + +describe('http-transport: mcp_request_log audit', () => { + test('21. successful request → audit row with token_name + operation + status', async () => { + const TOK = 'audit-tok'; + const srv = await startTest({ validTokens: new Map([[hash(TOK), { id: 'a-1', name: 'audit-test' }]]) }); + try { + await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOK}`, 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + // Audit insert is fire-and-forget; give it a tick to land in the fake handler + await new Promise(r => setTimeout(r, 10)); + expect(srv.engine.audit.length).toBeGreaterThanOrEqual(1); + const row = srv.engine.audit[srv.engine.audit.length - 1]; + expect(row.token_name).toBe('audit-test'); + expect(row.operation).toBe('tools/list'); + expect(row.status).toBe('success'); + expect(row.latency_ms).toBeGreaterThanOrEqual(0); + } finally { srv.stop(); } + }); + + test('22. failed auth → audit row with null token_name + auth_failed status', async () => { + const srv = await startTest({}); + try { + await fetch(`${srv.url}/mcp`, { method: 'POST', headers: { 'Authorization': 'Bearer wrong', 'Content-Type': 'application/json' }, body: rpc('tools/list') }); + await new Promise(r => setTimeout(r, 10)); + expect(srv.engine.audit.length).toBeGreaterThanOrEqual(1); + const row = srv.engine.audit[srv.engine.audit.length - 1]; + expect(row.token_name).toBeNull(); + expect(row.status).toBe('auth_failed'); + } finally { srv.stop(); } + }); +});