triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.42.28.0 fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861) (#1927)

Browse Source 
* fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861)

addLinksBatch/addTimelineEntriesBatch/addTakesBatch passed free text through
unnest(${arr}::text[]); postgres.js serialized it to a Postgres text[] literal
that array_in rejected ("malformed array literal") on calendar/Zoom context,
aborting the whole `extract links --stale` sweep. Bind the batch as one JSONB
doc via jsonb_to_recordset(($1::jsonb)->'rows') through the audited
executeRawJsonb contract instead. Shared row builders (src/core/batch-rows.ts)
keep both engines byte-identical; NUL is stripped only from free-text body
fields (context/summary/detail/claim), while identity/security fields
(slugs/source_ids/holder/kind/dates) still reject NUL. addTakesBatch is now
batchRetry-wrapped ('addTakesBatch' audit site) and its BrainEngine signature
takes BatchOpts. Scalar addLink context is NUL-stripped too.

Regression tests on both engines: PGLite always-on poison/NUL/parity suite +
DATABASE_URL-gated Postgres lane (the engine that actually crashed).

* test: make "no Anthropic key" tests hermetic via withoutAnthropicKey

hasAnthropicKey() reads both ANTHROPIC_API_KEY and ~/.gbrain config; tests that
only deleted the env var fired a real LLM call on configured machines (warning
flipped NO_ANTHROPIC_API_KEY -> LLM_OUTPUT_NOT_JSON). New test/helpers/no-anthropic-key.ts
neutralizes both sources (env + GBRAIN_HOME temp dir) for the duration of the call.
Refactors the five no-key tests in think-pipeline + takes-mcp-allowlist to use it,
including two that previously passed only by luck of the live LLM output.

* chore: docs + version bump (v0.42.28.0)

KEY_FILES.md/RETRIEVAL.md describe the jsonb_to_recordset batch path; TODOS.md
files the #1861 follow-ups (element-isolation, remaining ::text[] sites, shared
SQL-string hoist, batch-insert edge-case tests). CHANGELOG + VERSION + package.json
to 0.42.28.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync TESTING.md batch-insert references for v0.42.28.0

The #1861 fix migrated links/timeline/takes batch inserts from
unnest(::text[]) to jsonb_to_recordset. Update the stale "postgres-js
unnest() binding" note and add the two new poison-regression test files
(test/links-timeline-jsonb-poison.test.ts PGLite half,
test/e2e/jsonb-batch-poison-postgres.test.ts Postgres lane) to the inventory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sql-query): reject top-level array jsonb params in executeRawJsonb (#1861 P2a)

The "no top-level array" rule was only a comment. A bare JS array bound to a
$N::jsonb position can serialize as a Postgres array literal (not jsonb) through
postgres.js, silently re-entering the "malformed array literal" class #1861 just
escaped. executeRawJsonb now throws a clear error steering callers to the
{ rows: [...] } object wrapper. Verified breaks zero call sites (all pass objects
or null). Codex adversarial P2a; batch-size enforcement (P2b) filed as a TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-07 08:05:34 -07:00
committed by GitHub
parent 805814451e
commit f7f8512b14
20 changed files with 872 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files

185
src/core/postgres-engine.ts
View File

@@ -21,6 +21,8 @@ import type {
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { deriveResolutionTuple, finalizeScorecard } from './takes-resolution.ts';
import { normalizeWeightForStorage } from './takes-fence.ts';
import { executeRawJsonb } from './sql-query.ts';
import { stripNul, buildLinkRows, buildTimelineRows, buildTakeRows } from './batch-rows.ts';
import { runMigrations } from './migrate.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import { verifySchema } from './schema-verify.ts';
@@ -2509,7 +2511,7 @@ export class PostgresEngine implements BrainEngine {
await sql`
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
FROM (VALUES (${from}, ${to}, ${linkType || ''}, ${context || ''}, ${src}, ${originSlug ?? null}, ${originField ?? null}, ${fromSrc}, ${toSrc}, ${originSrc}))
FROM (VALUES (${from}, ${to}, ${linkType || ''}, ${stripNul(context || '')}, ${src}, ${originSlug ?? null}, ${originField ?? null}, ${fromSrc}, ${toSrc}, ${originSrc}))
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
@@ -2526,42 +2528,33 @@ export class PostgresEngine implements BrainEngine {
}
private async _addLinksBatchOnce(links: LinkBatchInput[]): Promise<number> {
const sql = this.sql;
// unnest() pattern: 7 array-typed bound parameters regardless of batch size.
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
// identifier-escape gotcha when used inside a (VALUES) subquery.
//
// v0.13: added link_source, origin_slug, origin_field. Defaults:
// link_source → 'markdown' (back-compat with pre-v0.13 callers)
// origin_slug → NULL (resolves to origin_page_id IS NULL via LEFT JOIN)
// origin_field → NULL
const fromSlugs = links.map(l => l.from_slug);
const toSlugs = links.map(l => l.to_slug);
const linkTypes = links.map(l => l.link_type || '');
const contexts = links.map(l => l.context || '');
const linkSources = links.map(l => l.link_source || 'markdown');
const originSlugs = links.map(l => l.origin_slug || null);
const originFields = links.map(l => l.origin_field || null);
const fromSourceIds = links.map(l => l.from_source_id || 'default');
const toSourceIds = links.map(l => l.to_source_id || 'default');
const originSourceIds = links.map(l => l.origin_source_id || 'default');
// v0.41.18.0 (A10): link_kind column (v98). NULL = legacy/plain.
const linkKinds = links.map(l => l.link_kind ?? null);
const result = await sql`
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
FROM unnest(
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
${originFields}::text[], ${fromSourceIds}::text[], ${toSourceIds}::text[],
${originSourceIds}::text[], ${linkKinds}::text[]
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id, link_kind)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
RETURNING 1
`;
// #1861: pass the batch as one JSONB document via jsonb_to_recordset instead
// of N parallel unnest(${arr}::text[]). The old text[] array-literal path
// crashed Postgres ("malformed array literal") on free-text context strings
// (calendar/Zoom lines with commas, quotes, braces, em-dashes); JSONB encodes
// arbitrary text safely and dodges the 65535-param cap. Binding goes through
// executeRawJsonb (the audited cross-engine JSONB contract) with an OBJECT
// wrapper { rows } — a bare top-level array through postgres.js would re-enter
// the same array serializer this fix exists to avoid. Row construction +
// NUL-stripping + exact defaulting live in buildLinkRows (shared with PGLite).
const rows = buildLinkRows(links);
const result = await executeRawJsonb(
this,
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, link_kind, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, v.link_kind, o.id, v.origin_field
FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(
from_slug text, to_slug text, link_type text, context text, link_source text,
origin_slug text, origin_field text, from_source_id text, to_source_id text,
origin_source_id text, link_kind text
)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
RETURNING 1`,
[],
[{ rows }],
);
return result.length;
}
@@ -3202,22 +3195,24 @@ export class PostgresEngine implements BrainEngine {
}
private async _addTimelineEntriesBatchOnce(entries: TimelineBatchInput[]): Promise<number> {
const sql = this.sql;
const slugs = entries.map(e => e.slug);
const dates = entries.map(e => e.date);
const sources = entries.map(e => e.source || '');
const summaries = entries.map(e => e.summary);
const details = entries.map(e => e.detail || '');
const sourceIds = entries.map(e => e.source_id || 'default');
const result = await sql`
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT p.id, v.date::date, v.source, v.summary, v.detail
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[], ${sourceIds}::text[])
AS v(slug, date, source, summary, detail, source_id)
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
ON CONFLICT (page_id, date, summary, source) DO NOTHING
RETURNING 1
`;
// #1861: JSONB jsonb_to_recordset instead of unnest(${arr}::text[]). Meeting
// summary/detail/source are free text with the same array-literal crash
// hazard as link context. See _addLinksBatchOnce for the full rationale.
// `date` stays text in the recordset and is cast v.date::date in the SELECT,
// exactly as the old unnest shape did.
const rows = buildTimelineRows(entries);
const result = await executeRawJsonb(
this,
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT p.id, v.date::date, v.source, v.summary, v.detail
FROM jsonb_to_recordset(($1::jsonb)->'rows')
AS v(slug text, date text, source text, summary text, detail text, source_id text)
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
ON CONFLICT (page_id, date, summary, source) DO NOTHING
RETURNING 1`,
[],
[{ rows }],
);
return result.length;
}
@@ -3883,53 +3878,51 @@ export class PostgresEngine implements BrainEngine {
// v0.28: Takes (typed/weighted/attributed claims) + synthesis_evidence
// ============================================================
async addTakesBatch(rowsIn: TakeBatchInput[]): Promise<number> {
async addTakesBatch(rowsIn: TakeBatchInput[], opts?: BatchOpts): Promise<number> {
if (rowsIn.length === 0) return 0;
const sql = this.sql;
let weightClamped = 0;
const pageIds = rowsIn.map(r => r.page_id);
const rowNums = rowsIn.map(r => r.row_num);
const claims = rowsIn.map(r => r.claim);
const kinds = rowsIn.map(r => r.kind);
const holders = rowsIn.map(r => r.holder);
const weights = rowsIn.map(r => {
const { weight, clamped } = normalizeWeightForStorage(r.weight);
if (clamped) weightClamped++;
return weight;
});
const sinces = rowsIn.map(r => r.since_date ?? null);
const untils = rowsIn.map(r => r.until_date ?? null);
const sources = rowsIn.map(r => r.source ?? null);
const supersededBys = rowsIn.map(r => r.superseded_by ?? null);
// postgres-js needs boolean arrays passed as text[] then SQL-cast to boolean[],
// otherwise the driver mis-detects element type. Same pattern as how the
// existing batch methods handle bools.
const actives = rowsIn.map(r => (r.active ?? true) ? 'true' : 'false');
// v0.42.26: takes is a batch primitive too — wrap in batchRetry so a
// Supavisor circuit-breaker blip doesn't silently drop takes the way it
// could before (links/timeline already had this; takes was the gap).
return this.batchRetry(opts?.auditSite ?? 'addTakesBatch', opts?.signal, () => this._addTakesBatchOnce(rowsIn), rowsIn.length);
}
private async _addTakesBatchOnce(rowsIn: TakeBatchInput[]): Promise<number> {
// #1861: JSONB jsonb_to_recordset instead of unnest(${arr}::text[]). `claim`
// is free LLM-extracted prose with the same array-literal crash hazard as
// link context. JSONB additionally lets us declare NATIVE recordset column
// types and emit JSON-native numbers/booleans, which retires the old
// postgres-js ${actives}::text[]::boolean[] element-type workaround entirely.
// Weight clamp/round + NUL-stripping live in buildTakeRows (shared w/ PGLite).
// NOTE: ON CONFLICT here is DO UPDATE (not DO NOTHING) — an intra-batch
// duplicate (page_id, row_num) errors, identical to the pre-#1861 unnest path.
const { rows, weightClamped } = buildTakeRows(rowsIn);
if (weightClamped > 0) {
process.stderr.write(`[takes] TAKES_WEIGHT_CLAMPED: ${weightClamped} row(s) had weight outside [0,1]; clamped\n`);
}
const result = await sql`
INSERT INTO takes (page_id, row_num, claim, kind, holder, weight, since_date, until_date, source, superseded_by, active)
SELECT v.page_id::int, v.row_num::int, v.claim, v.kind, v.holder, v.weight::real,
v.since_date::text, v.until_date::text, v.source, v.superseded_by::int, v.active::boolean
FROM unnest(
${pageIds}::int[], ${rowNums}::int[], ${claims}::text[], ${kinds}::text[],
${holders}::text[], ${weights}::real[], ${sinces}::text[], ${untils}::text[],
${sources}::text[], ${supersededBys}::int[], ${actives}::text[]::boolean[]
) AS v(page_id, row_num, claim, kind, holder, weight, since_date, until_date, source, superseded_by, active)
ON CONFLICT (page_id, row_num) DO UPDATE SET
claim = EXCLUDED.claim,
kind = EXCLUDED.kind,
holder = EXCLUDED.holder,
weight = EXCLUDED.weight,
since_date = EXCLUDED.since_date,
until_date = EXCLUDED.until_date,
source = EXCLUDED.source,
superseded_by = EXCLUDED.superseded_by,
active = EXCLUDED.active,
updated_at = now()
RETURNING 1
`;
const result = await executeRawJsonb(
this,
`INSERT INTO takes (page_id, row_num, claim, kind, holder, weight, since_date, until_date, source, superseded_by, active)
SELECT v.page_id, v.row_num, v.claim, v.kind, v.holder, v.weight,
v.since_date, v.until_date, v.source, v.superseded_by, v.active
FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(
page_id int, row_num int, claim text, kind text, holder text, weight real,
since_date text, until_date text, source text, superseded_by int, active boolean
)
ON CONFLICT (page_id, row_num) DO UPDATE SET
claim = EXCLUDED.claim,
kind = EXCLUDED.kind,
holder = EXCLUDED.holder,
weight = EXCLUDED.weight,
since_date = EXCLUDED.since_date,
until_date = EXCLUDED.until_date,
source = EXCLUDED.source,
superseded_by = EXCLUDED.superseded_by,
active = EXCLUDED.active,
updated_at = now()
RETURNING 1`,
[],
[{ rows }],
);
return result.length;
}