v0.36.3.0 feat: dynamic embedding column selection for search (#1164)

* feat: migration v68 — eval_candidates.embedding_column

Schema migration ALTERs eval_candidates to add a nullable
embedding_column TEXT column. Per-row capture metadata so
`gbrain eval replay` reproduces the same column the
capture ran against (D16 / CDX-10). NULL-tolerant: pre-v0.36
rows fall back to current default.

Renumbered v67→v68 because master claimed v67 for
facts_typed_claim_columns during this branch's lifetime.

PGLite parity via sqlFor.pglite — same ALTER IF NOT EXISTS.

* feat: dynamic embedding column — core (resolver, types, gateway, engines)

The read-path foundation for routing search through any
populated embedding column, not just OpenAI 1536.

src/core/search/embedding-column.ts (new) is the canonical
seam. Single source of truth for column → provider/dim/type
lookup. Validates registry keys via regex
(/^[a-z_][a-z0-9_]*$/), uses Object.create(null) +
Object.hasOwn so 'constructor' and other inherited names
can't masquerade as registered columns. Identifier-quoting
on SQL interpolation as defense in depth.

src/core/types.ts widens SearchOpts.embeddingColumn to
accept ResolvedColumn descriptors at the engine boundary;
adds EmbeddingColumnConfig + ResolvedColumn exports.

src/core/config.ts merges embedding_columns +
search_embedding_column from the DB plane via
loadConfigWithEngine, mirroring the existing
embedding_multimodal_model pattern. Handles the no-file
case so env-only Postgres installs see DB-plane overrides
(codex /ship #3).

src/core/ai/gateway.ts: embedQuery(text, opts) +
embed(texts, opts) accept embeddingModel + dimensions
overrides. isAvailable(touchpoint, modelOverride?) so
hybrid asks 'is the active column's provider reachable?'
not 'is the global default reachable?' (CDX-4 / D10).

Engines: searchVector accepts ResolvedColumn descriptors via
normalizeEngineColumn; engine code is config-free and
unit-testable. getEmbeddingsByChunkIds(ids, column?) so
cosineReScore hydrates from the active column instead of
always 'embedding' (CDX-3 / D9). Identifier-quoting belt at
the SQL boundary.

src/core/eval-capture.ts threads embedding_column from
hybridSearch meta into the persisted capture row.

* feat: dynamic embedding column — integration (hybrid, ops, doctor)

Wires the resolver into hybridSearch, the query op, doctor,
and the config command.

src/core/search/hybrid.ts: resolves the column once at the
boundary, threads the descriptor into engine calls, routes
embedQuery through the resolved column's provider/dims, and
calls isCacheSafe (not isDefaultColumn) for cache skip so
user overrides of the 'embedding' builtin can't leak across
vector spaces (CDX-4). cosineReScore now hydrates from the
active column.

src/core/search/mode.ts: KNOBS_HASH_VERSION 2→3, append-only
new fields col= and prov= alongside floor_ratio. Cache rows
from different columns or providers now sit in different
keyspaces — cross-column contamination impossible.

src/core/operations.ts: query op accepts embedding_column
param for per-call A/B benchmarking. search op (keyword-only)
deliberately does NOT (CDX-9 / D15) — would be silent UX.

src/commands/doctor.ts: new embedding_column_registry
check. Batch format_type probe (D13) catches dim drift
that information_schema.columns.udt_name can't.
Batch pg_indexes probe (D5) warns on missing HNSW. Coverage
% on active column, gates at <90% (D14), short-circuits on
empty brains (codex /ship #5).

src/commands/config.ts: validates embedding_columns JSON
shape at set time, runs the coverage gate when setting
search_embedding_column, uses Object.hasOwn for the
registry lookup.

src/commands/eval-replay.ts: replay re-runs queries against
the captured embedding_column so post-flip-config replays
don't surface as false-positive regressions.

* test: dynamic embedding column — unit + e2e coverage

50 unit cases for the resolver (resolution chain, registry
merge, validation, prototype pollution, descriptor
passthrough, isCacheSafe, normalizeEngineColumn).

8 gateway override cases — embeddingModel + dimensions
flow into providerOptions, isAvailable(touchpoint, override)
routes to the right recipe, unknown models throw clean.

4 cosineReScore + 6 ops + 5 knobs-hash + 7 mode + 9 PGLite
E2E + 7 Postgres E2E + 5 eval-replay column metadata.

Postgres E2E (gated on DATABASE_URL) covers halfvec(2560)
end-to-end on real pgvector, EXPLAIN-visible HNSW index
on the alternate column, format_type-based dim drift catch,
and the <90% coverage gate.

Pins every codex /ship fix: prototype-pollution rejection
('constructor' as column name), descriptor passthrough
validation (rejects SQL-shaped strings in dimensions),
isCacheSafe semantics (space-based, not name-based).

Total: 141 new + extended cases, all green.

* chore: bump version and changelog (v0.36.3.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync to v0.36.3.0

Add CLAUDE.md key-files entry for src/core/search/embedding-column.ts.
Annotate hybrid.ts, gateway.ts, doctor.ts, and migrate.ts entries with
v0.36.3.0 wave changes (ResolvedColumn threading, embedQuery model
override, embedding_column_registry check, migration v68). Document
knobs_hash v=2 → v=3 bump under the Search Mode section.

Regenerate llms-full.txt from the updated CLAUDE.md so the auto-checked
bundle matches source (build-llms.test.ts CI guard).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): two CI failures from v0.36.3.0

1. test/loadConfig-merge.test.ts: update the 'returns null when base
   config is null' contract test. Pre-v0.36 the function returned null
   for null base; the codex /ship #3 fix changed that to synthesize a
   minimal `{ engine: 'postgres' }` so env-only installs see DB-plane
   overrides. Test now pins the new contract + adds a round-trip case
   asserting the merge actually surfaces `embedding_columns` /
   `search_embedding_column` set via gbrain config set on a null base.

2. test/schema-bootstrap-coverage.test.ts was failing because
   eval_candidates.embedding_column (added by migration v68) wasn't
   covered by applyForwardReferenceBootstrap. Fix: add the column to
   PGLITE_SCHEMA_SQL's eval_candidates CREATE TABLE definition (and
   src/schema.sql for parity) so fresh installs get it natively. The
   coverage test's third tier (schemaCreateTableCols) now finds it.
   Regenerated schema-embedded.ts via bun run build:schema.

Schema-blob path is cleaner than COLUMN_EXEMPTIONS — fresh installs
skip the migration entirely; upgrade installs still run v68.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/core/postgres-engine.ts

@@ -18,6 +18,13 @@ import { runMigrations } from './migrate.ts';
 import { SCHEMA_SQL } from './schema-embedded.ts';
 import { verifySchema } from './schema-verify.ts';
 import { applyChunkEmbeddingIndexPolicy, dropZombieIndexes } from './vector-index.ts';
+import {
+  normalizeEngineColumn,
+  buildVectorCastFragment,
+  quoteIdentifier,
+  COLUMN_NAME_REGEX,
+  EmbeddingColumnNotRegisteredError,
+} from './search/embedding-column.ts';
 import type {
   Page, PageInput, PageFilters, PageType,
   Chunk, ChunkInput, StaleChunkRow,
@@ -1292,9 +1299,20 @@ export class PostgresEngine implements BrainEngine {
     // wasting candidate slots on hidden rows.
     const visibilityClause = buildVisibilityClause('p', 's');
 
-    // v0.27.1: column routing. See pglite-engine.ts searchVector for rationale.
-    const col = opts?.embeddingColumn === 'embedding_image' ? 'embedding_image' : 'embedding';
-    const modalityFilter = col === 'embedding_image' ? `AND cc.modality = 'image'` : `AND cc.modality = 'text'`;
+    // v0.36 (D11): column routing via resolved descriptor. Engine doesn't
+    // read config — caller (hybrid/op) resolved it and passed it in.
+    // normalizeEngineColumn accepts the legacy union (string literals,
+    // ResolvedColumn, undefined) and produces a canonical descriptor.
+    const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
+    const { col, castSql } = buildVectorCastFragment(resolvedCol);
+    // Modality filter: image rows live in modality='image'; text/code in
+    // 'text'. The image-column literal is the only one with image rows.
+    // For all other columns (default + user-declared), restrict to text
+    // mode to avoid cross-modality dim leaks. The check is on
+    // resolved.name (already validated, never raw input).
+    const modalityFilter = resolvedCol.name === 'embedding_image'
+      ? `AND cc.modality = 'image'`
+      : `AND cc.modality = 'text'`;
 
     const rawQuery = `
       WITH hnsw_candidates AS (
@@ -1302,7 +1320,7 @@ export class PostgresEngine implements BrainEngine {
           p.slug, p.id as page_id, p.title, p.type, p.source_id,
           p.effective_date, p.effective_date_source,
           cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
-          1 - (cc.${col} <=> $1::vector) AS raw_score
+          1 - (cc.${col} <=> ${castSql}) AS raw_score
         FROM content_chunks cc
         JOIN pages p ON p.id = cc.page_id
         JOIN sources s ON s.id = p.source_id
@@ -1318,7 +1336,7 @@ export class PostgresEngine implements BrainEngine {
           ${sourceClause}
           ${hardExcludeClause}
           ${visibilityClause}
-        ORDER BY cc.${col} <=> $1::vector
+        ORDER BY cc.${col} <=> ${castSql}
         LIMIT ${innerLimitParam}
       )
       SELECT
@@ -1340,13 +1358,27 @@ export class PostgresEngine implements BrainEngine {
     return rows.map(rowToSearchResult);
   }
 
-  async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
+  async getEmbeddingsByChunkIds(
+    ids: number[],
+    column: string = 'embedding',
+  ): Promise<Map<number, Float32Array>> {
     if (ids.length === 0) return new Map();
+    // v0.36 (D9): column parameter used by hybrid.cosineReScore so
+    // rescoring rehydrates from the active column's embedding space,
+    // not always 'embedding'. Engine has no resolver access; the
+    // caller must pass a known column name. Identifier-quoted (D12
+    // defense layer 2) plus a strict regex check (D12 defense layer 1)
+    // so even a misconfigured caller can't smuggle a SQL fragment.
+    if (!COLUMN_NAME_REGEX.test(column)) {
+      throw new EmbeddingColumnNotRegisteredError(column, []);
+    }
+    const quotedCol = quoteIdentifier(column);
     const sql = this.sql;
-    const rows = await sql`
-      SELECT id, embedding FROM content_chunks
-      WHERE id = ANY(${ids}::int[]) AND embedding IS NOT NULL
+    const rawQuery = `
+      SELECT id, ${quotedCol} AS embedding FROM content_chunks
+      WHERE id = ANY($1::int[]) AND ${quotedCol} IS NOT NULL
     `;
+    const rows = await sql.unsafe(rawQuery, [ids] as Parameters<typeof sql.unsafe>[1]);
     const result = new Map<number, Float32Array>();
     for (const row of rows) {
       const embedding = tryParseEmbedding(row.embedding);
@@ -3777,11 +3809,11 @@ export class PostgresEngine implements BrainEngine {
       INSERT INTO eval_candidates (
         tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
         expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
-        latency_ms, remote, job_id, subagent_id
+        latency_ms, remote, job_id, subagent_id, embedding_column
       ) VALUES (
         ${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids},
         ${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied},
-        ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}
+        ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}, ${input.embedding_column ?? null}
       )
       RETURNING id
     `;