fix: JSONB double-encode + splitBody wiki + parseEmbedding (v0.12.1) (#196)
* fix: splitBody and inferType for wiki-style markdown content - splitBody now requires explicit timeline sentinel (<!-- timeline -->, --- timeline ---, or --- directly before ## Timeline / ## History). A bare --- in body text is a markdown horizontal rule, not a separator. This fixes the 83% content truncation @knee5 reported on a 1,991-article wiki where 4,856 of 6,680 wikilinks were lost. - serializeMarkdown emits <!-- timeline --> sentinel for round-trip stability. - inferType extended with /writing/, /wiki/analysis/, /wiki/guides/, /wiki/hardware/, /wiki/architecture/, /wiki/concepts/. Path order is most-specific-first so projects/blog/writing/essay.md → writing, not project. - PageType union extended: writing, analysis, guide, hardware, architecture. Updates test/import-file.test.ts to use the new sentinel. Co-Authored-By: @knee5 (PR #187) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: JSONB double-encode bug on Postgres + parseEmbedding NaN scores Two related Postgres-string-typed-data bugs that PGLite hid: 1. JSONB double-encode (postgres-engine.ts:107,668,846 + files.ts:254): ${JSON.stringify(value)}::jsonb in postgres.js v3 stringified again on the wire, storing JSONB columns as quoted string literals. Every frontmatter->>'key' returned NULL on Postgres-backed brains; GIN indexes were inert. Switched to sql.json(value), which is the postgres.js-native JSONB encoder (Parameter with OID 3802). Affected columns: pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata. page_versions.frontmatter is downstream via INSERT...SELECT and propagates the fix. 2. pgvector embeddings returning as strings (utils.ts): getEmbeddingsByChunkIds returned "[0.1,0.2,...]" instead of Float32Array on Supabase, producing [NaN] cosine scores. Adds parseEmbedding() helper handling Float32Array, numeric arrays, and pgvector string format. Throws loud on malformed vectors (per Codex's no-silent-NaN requirement); returns null for non-vector strings (treated as "no embedding here"). rowToChunk delegates to parseEmbedding. E2E regression test at test/e2e/postgres-jsonb.test.ts asserts jsonb_typeof = 'object' AND col->>'k' returns expected scalar across all 5 affected columns — the test that should have caught the original bug. Runs in CI via the existing pgvector service. Co-Authored-By: @knee5 (PR #187 — JSONB triple-fix) Co-Authored-By: @leonardsellem (PR #175 — parseEmbedding) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: extract wikilink syntax with ancestor-search slug resolution extractMarkdownLinks now handles [[page]] and [[page|Display Text]] alongside standard [text](page.md). For wiki KBs where authors omit leading ../ (thinking in wiki-root-relative terms), resolveSlug walks ancestor directories until it finds a matching slug. Without this, wikilinks under tech/wiki/analysis/ targeting [[../../finance/wiki/concepts/foo]] silently dangled when the correct relative depth was 3 × ../ instead of 2. Co-Authored-By: @knee5 (PR #187) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: gbrain repair-jsonb + v0.12.1 migration + CI grep guard - New gbrain repair-jsonb command. Detects rows where jsonb_typeof(col) = 'string' and rewrites them via (col #>> '{}')::jsonb across 5 affected columns: pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter. Idempotent — re-running is a no-op. PGLite engines short-circuit cleanly (the bug never affected the parameterized encode path PGLite uses). --dry-run shows what would be repaired; --json for scripting. - New v0_12_1.ts migration orchestrator. Phases: schema → repair → verify. Modeled on v0_12_0 pattern, registered in migrations/index.ts. Runs automatically via gbrain upgrade / apply-migrations. - CI grep guard at scripts/check-jsonb-pattern.sh fails the build if anyone reintroduces the ${JSON.stringify(x)}::jsonb interpolation pattern. Wired into bun test via package.json. Best-effort static analysis (multi-line and helper-wrapped variants are caught by the E2E round-trip test instead). - Updates apply-migrations.test.ts expectations to account for the new v0.12.1 entry in the registry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.12.1) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.12.1 - CLAUDE.md: document repair-jsonb command, v0_12_1 migration, splitBody sentinel contract, inferType wiki subtypes, CI grep guard, new test files (repair-jsonb, migrations-v0_12_1, markdown) - README.md: add gbrain repair-jsonb to ADMIN command reference - INSTALL_FOR_AGENTS.md: fix verification count (6 -> 7), add v0.12.1 upgrade guidance for Postgres brains - docs/GBRAIN_VERIFY.md: add check #8 for JSONB integrity on Postgres-backed brains - docs/UPGRADING_DOWNSTREAM_AGENTS.md: add v0.12.1 section with migration steps, splitBody contract, wiki subtype inference - skills/migrate/SKILL.md: document native wikilink extraction via gbrain extract links (v0.12.1+) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -18,7 +18,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check', 'repair-jsonb']);
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
@@ -306,6 +306,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runApplyMigrations(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'repair-jsonb') {
|
||||
const { runRepairJsonbCli } = await import('./commands/repair-jsonb.ts');
|
||||
await runRepairJsonbCli(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'skillpack-check') {
|
||||
// Agent-readable health report. Shells out to doctor + apply-migrations
|
||||
// internally; does not need its own DB connection.
|
||||
|
||||
@@ -76,19 +76,72 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
|
||||
|
||||
// --- Link extraction ---
|
||||
|
||||
/** Extract markdown links to .md files (relative paths only) */
|
||||
/**
|
||||
* Extract markdown links to .md files (relative paths only).
|
||||
*
|
||||
* Handles two syntaxes:
|
||||
* 1. Standard markdown: [text](relative/path.md)
|
||||
* 2. Wikilinks: [[relative/path]] or [[relative/path|Display Text]]
|
||||
*
|
||||
* Both are resolved relative to the file that contains them. External URLs
|
||||
* (containing ://) are always skipped. For wikilinks, the .md suffix is added
|
||||
* if absent and section anchors (#heading) are stripped.
|
||||
*/
|
||||
export function extractMarkdownLinks(content: string): { name: string; relTarget: string }[] {
|
||||
const results: { name: string; relTarget: string }[] = [];
|
||||
const pattern = /\[([^\]]+)\]\(([^)]+\.md)\)/g;
|
||||
|
||||
const mdPattern = /\[([^\]]+)\]\(([^)]+\.md)\)/g;
|
||||
let match;
|
||||
while ((match = pattern.exec(content)) !== null) {
|
||||
while ((match = mdPattern.exec(content)) !== null) {
|
||||
const target = match[2];
|
||||
if (target.includes('://')) continue; // skip external URLs
|
||||
if (target.includes('://')) continue;
|
||||
results.push({ name: match[1], relTarget: target });
|
||||
}
|
||||
|
||||
const wikiPattern = /\[\[([^|\]]+?)(?:\|[^\]]*?)?\]\]/g;
|
||||
while ((match = wikiPattern.exec(content)) !== null) {
|
||||
const rawPath = match[1].trim();
|
||||
if (rawPath.includes('://')) continue;
|
||||
const hashIdx = rawPath.indexOf('#');
|
||||
const pagePath = hashIdx >= 0 ? rawPath.slice(0, hashIdx) : rawPath;
|
||||
if (!pagePath) continue;
|
||||
const relTarget = pagePath.endsWith('.md') ? pagePath : pagePath + '.md';
|
||||
const pipeIdx = match[0].indexOf('|');
|
||||
const displayName = pipeIdx >= 0 ? match[0].slice(pipeIdx + 1, -2).trim() : rawPath;
|
||||
results.push({ name: displayName, relTarget });
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a wikilink target to a canonical slug, given the directory of the
|
||||
* containing page and the set of all known slugs in the brain.
|
||||
*
|
||||
* Wiki KBs often use inconsistent relative depths. Authors omit one or more
|
||||
* leading `../` because they think in "wiki-root-relative" terms. Resolution
|
||||
* order (first match wins):
|
||||
* 1. Standard `join(fileDir, relTarget)` — exact relative path as written
|
||||
* 2. Ancestor search — strip leading path components from fileDir, retry
|
||||
*
|
||||
* Returns null when no matching slug is found (dangling link).
|
||||
*/
|
||||
export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<string>): string | null {
|
||||
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
|
||||
|
||||
const s1 = join(fileDir, targetNoExt);
|
||||
if (allSlugs.has(s1)) return s1;
|
||||
|
||||
const parts = fileDir.split('/').filter(Boolean);
|
||||
for (let strip = 1; strip <= parts.length; strip++) {
|
||||
const ancestor = parts.slice(0, parts.length - strip).join('/');
|
||||
const candidate = ancestor ? join(ancestor, targetNoExt) : targetNoExt;
|
||||
if (allSlugs.has(candidate)) return candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Infer link type from directory structure */
|
||||
function inferLinkType(fromDir: string, toDir: string, frontmatter?: Record<string, unknown>): string {
|
||||
const from = fromDir.split('/')[0];
|
||||
@@ -146,8 +199,8 @@ export function extractLinksFromFile(
|
||||
const fm = parseFrontmatterFromContent(content, relPath);
|
||||
|
||||
for (const { name, relTarget } of extractMarkdownLinks(content)) {
|
||||
const resolved = join(fileDir, relTarget).replace('.md', '');
|
||||
if (allSlugs.has(resolved)) {
|
||||
const resolved = resolveSlug(fileDir, relTarget, allSlugs);
|
||||
if (resolved !== null) {
|
||||
links.push({
|
||||
from_slug: slug, to_slug: resolved,
|
||||
link_type: inferLinkType(fileDir, dirname(resolved), fm),
|
||||
|
||||
@@ -251,7 +251,7 @@ async function uploadRaw(args: string[]) {
|
||||
await sql`
|
||||
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
|
||||
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${mimeType}, ${stat.size}, ${'sha256:' + hash},
|
||||
${JSON.stringify({ type: fileType, upload_method: method })}::jsonb)
|
||||
${sql.json({ type: fileType, upload_method: method })})
|
||||
ON CONFLICT (storage_path) DO UPDATE SET
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
|
||||
@@ -13,10 +13,12 @@
|
||||
import type { Migration } from './types.ts';
|
||||
import { v0_11_0 } from './v0_11_0.ts';
|
||||
import { v0_12_0 } from './v0_12_0.ts';
|
||||
import { v0_12_2 } from './v0_12_2.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
v0_12_0,
|
||||
v0_12_2,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
141
src/commands/migrations/v0_12_2.ts
Normal file
141
src/commands/migrations/v0_12_2.ts
Normal file
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* v0.12.2 migration orchestrator — JSONB double-encode repair.
|
||||
*
|
||||
* v0.12.0-and-earlier wrote JSONB columns via the buggy
|
||||
* `JSON.stringify(value)`-then-cast-to-jsonb interpolation pattern, which
|
||||
* postgres.js v3 stringified again on the wire. Result: every
|
||||
* `frontmatter->>'key'` query returned NULL on Postgres-backed brains and
|
||||
* GIN indexes on JSONB columns were inert. PGLite was unaffected (its
|
||||
* driver path uses parameterized binding, never interpolation).
|
||||
*
|
||||
* v0.12.2 fixes the writes (sql.json) AND repairs existing rows in place.
|
||||
* This is the migration. It's idempotent (only touches `jsonb_typeof = 'string'`
|
||||
* rows) and safe to re-run. PGLite engines no-op cleanly.
|
||||
*
|
||||
* Phases (all idempotent):
|
||||
* A. Schema — gbrain init --migrate-only (no schema changes in v0.12.2
|
||||
* but we still apply for consistency with v0.12.0).
|
||||
* B. Repair — gbrain repair-jsonb (the actual JSONB fix).
|
||||
* C. Verify — gbrain repair-jsonb --dry-run --json; assert 0 remaining.
|
||||
* D. Record — append completed.jsonl.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'schema', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase B — JSONB repair ──────────────────────────────────
|
||||
|
||||
function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain repair-jsonb', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'jsonb_repair', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'jsonb_repair', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase C — Verify ────────────────────────────────────────
|
||||
|
||||
function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
const out = execSync('gbrain repair-jsonb --dry-run --json', {
|
||||
encoding: 'utf-8', timeout: 60_000, env: process.env,
|
||||
});
|
||||
const parsed = JSON.parse(out) as { total_repaired?: number; engine?: string };
|
||||
const remaining = parsed.total_repaired ?? 0;
|
||||
if (remaining > 0) {
|
||||
return {
|
||||
name: 'verify',
|
||||
status: 'failed',
|
||||
detail: `${remaining} string-typed JSONB rows remain after repair`,
|
||||
};
|
||||
}
|
||||
return { name: 'verify', status: 'complete', detail: parsed.engine ? `engine=${parsed.engine}` : undefined };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'verify', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.12.2 — JSONB double-encode repair ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const b = phaseBRepair(opts);
|
||||
phases.push(b);
|
||||
if (b.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' || b.status === 'failed' ? 'failed' :
|
||||
c.status === 'failed' ? 'partial' :
|
||||
'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
|
||||
if (status !== 'failed') {
|
||||
try {
|
||||
appendCompletedMigration({ version: '0.12.2', status: status as 'complete' | 'partial' });
|
||||
} catch {
|
||||
// Recording is best-effort.
|
||||
}
|
||||
}
|
||||
return {
|
||||
version: '0.12.2',
|
||||
status,
|
||||
phases,
|
||||
};
|
||||
}
|
||||
|
||||
export const v0_12_2: Migration = {
|
||||
version: '0.12.2',
|
||||
featurePitch: {
|
||||
headline: 'Postgres frontmatter queries now work — JSONB double-encode bug fixed and existing rows auto-repaired',
|
||||
description:
|
||||
'gbrain v0.12.0-and-earlier silently stored JSONB columns as quoted string literals on ' +
|
||||
'Postgres/Supabase (PGLite was unaffected). Every `frontmatter->>\'key\'` returned NULL ' +
|
||||
'and GIN indexes were inert. v0.12.2 fixes the writes AND auto-repairs every existing ' +
|
||||
'string-typed row in pages.frontmatter, raw_data.data, ingest_log.pages_updated, ' +
|
||||
'files.metadata, and page_versions.frontmatter. The migration is idempotent. Pages ' +
|
||||
'truncated by the splitBody horizontal-rule bug can be recovered with `gbrain sync --full`.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBRepair,
|
||||
phaseCVerify,
|
||||
};
|
||||
151
src/commands/repair-jsonb.ts
Normal file
151
src/commands/repair-jsonb.ts
Normal file
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* `gbrain repair-jsonb` — repair JSONB columns that were stored as string
|
||||
* literals due to the v0.12.0-and-earlier double-encode bug.
|
||||
*
|
||||
* Background: postgres-engine.ts wrote frontmatter and other JSONB columns
|
||||
* via the buggy `JSON.stringify(value)`-then-cast-to-jsonb interpolation
|
||||
* pattern, which postgres.js v3 stringified AGAIN on the wire. Result: every `frontmatter->>'key'` query returned NULL
|
||||
* on Postgres-backed brains; GIN indexes were inert. PGLite was unaffected
|
||||
* (different driver path). v0.12.1 fixes the writes (sql.json) but existing
|
||||
* rows stay broken until they're rewritten — that's what this command does.
|
||||
*
|
||||
* Strategy: for each affected JSONB column, detect rows where
|
||||
* `jsonb_typeof(col) = 'string'` and rewrite them via `(col #>> '{}')::jsonb`,
|
||||
* which extracts the string payload and re-parses it as JSONB. Idempotent:
|
||||
* re-running is a no-op (no rows match the guard). PGLite is a no-op too
|
||||
* (it never wrote string-typed JSONB).
|
||||
*
|
||||
* Affected columns (audit of src/schema.sql):
|
||||
* - pages.frontmatter (postgres-engine.ts:107 putPage)
|
||||
* - raw_data.data (postgres-engine.ts:668 putRawData)
|
||||
* - ingest_log.pages_updated (postgres-engine.ts:846 logIngest)
|
||||
* - files.metadata (commands/files.ts:254 file upload)
|
||||
* - page_versions.frontmatter (downstream of pages.frontmatter via
|
||||
* INSERT...SELECT FROM pages)
|
||||
*
|
||||
* Other JSONB columns (minion_jobs.{data,result,progress,stacktrace},
|
||||
* minion_inbox.payload) were always written via parameterized form ($N::jsonb
|
||||
* with a string parameter, not interpolation) so they were never affected.
|
||||
*/
|
||||
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import type { EngineConfig } from '../core/types.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
|
||||
interface RepairTarget {
|
||||
table: string;
|
||||
column: string;
|
||||
/** Optional secondary key column for logging. */
|
||||
keyCol?: string;
|
||||
}
|
||||
|
||||
const TARGETS: RepairTarget[] = [
|
||||
{ table: 'pages', column: 'frontmatter', keyCol: 'slug' },
|
||||
{ table: 'raw_data', column: 'data', keyCol: 'source' },
|
||||
{ table: 'ingest_log', column: 'pages_updated', keyCol: 'source_ref' },
|
||||
{ table: 'files', column: 'metadata', keyCol: 'storage_path' },
|
||||
{ table: 'page_versions', column: 'frontmatter', keyCol: 'snapshot_at' },
|
||||
];
|
||||
|
||||
export interface RepairResult {
|
||||
engine: string;
|
||||
per_target: Array<{
|
||||
table: string;
|
||||
column: string;
|
||||
rows_repaired: number;
|
||||
}>;
|
||||
total_repaired: number;
|
||||
}
|
||||
|
||||
export interface RepairOpts {
|
||||
dryRun: boolean;
|
||||
/** Engine config override (for tests). Defaults to loadConfig() result. */
|
||||
engineConfig?: EngineConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the repair against the currently-configured engine.
|
||||
*
|
||||
* On PGLite this finds 0 rows (the bug never affected the parameterized
|
||||
* encode path PGLite uses) and exits cleanly. On Postgres it issues one
|
||||
* idempotent UPDATE per target column.
|
||||
*/
|
||||
export async function repairJsonb(opts: RepairOpts = { dryRun: false }): Promise<RepairResult> {
|
||||
let engineCfg = opts.engineConfig;
|
||||
if (!engineCfg) {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
engineCfg = toEngineConfig(config);
|
||||
}
|
||||
const engineKind = engineCfg.engine || 'postgres';
|
||||
|
||||
const result: RepairResult = {
|
||||
engine: engineKind,
|
||||
per_target: [],
|
||||
total_repaired: 0,
|
||||
};
|
||||
|
||||
if (engineKind === 'pglite') {
|
||||
for (const t of TARGETS) {
|
||||
result.per_target.push({ table: t.table, column: t.column, rows_repaired: 0 });
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
await db.connect(engineCfg);
|
||||
const sql = db.getConnection();
|
||||
|
||||
for (const t of TARGETS) {
|
||||
let repaired = 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM ${t.table} WHERE jsonb_typeof(${t.column}) = 'string'`,
|
||||
);
|
||||
repaired = (rows[0] as { n: number }).n;
|
||||
} else {
|
||||
const rows = await sql.unsafe(
|
||||
`UPDATE ${t.table}
|
||||
SET ${t.column} = (${t.column} #>> '{}')::jsonb
|
||||
WHERE jsonb_typeof(${t.column}) = 'string'
|
||||
RETURNING 1`,
|
||||
);
|
||||
repaired = rows.length;
|
||||
}
|
||||
|
||||
result.per_target.push({ table: t.table, column: t.column, rows_repaired: repaired });
|
||||
result.total_repaired += repaired;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function runRepairJsonbCli(args: string[]): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const jsonMode = args.includes('--json');
|
||||
|
||||
const result = await repairJsonb({ dryRun });
|
||||
|
||||
if (jsonMode) {
|
||||
console.log(JSON.stringify({ status: 'ok', dry_run: dryRun, ...result }));
|
||||
return;
|
||||
}
|
||||
|
||||
if (result.engine === 'pglite') {
|
||||
console.log('Engine: pglite — JSONB double-encode bug never affected this path. No-op.');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}Engine: postgres`);
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}JSONB repair across ${TARGETS.length} columns:`);
|
||||
for (const t of result.per_target) {
|
||||
const verb = dryRun ? 'would repair' : 'repaired';
|
||||
console.log(` ${t.table}.${t.column}: ${verb} ${t.rows_repaired} rows`);
|
||||
}
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}Total ${dryRun ? 'to repair' : 'repaired'}: ${result.total_repaired} rows`);
|
||||
if (!dryRun && result.total_repaired === 0) {
|
||||
console.log('Nothing to repair (already-valid JSONB or fresh install).');
|
||||
}
|
||||
}
|
||||
@@ -22,14 +22,16 @@ export interface ParsedMarkdown {
|
||||
* tags: [startups, growth]
|
||||
* ---
|
||||
* Compiled truth content here...
|
||||
* ---
|
||||
*
|
||||
* <!-- timeline -->
|
||||
* Timeline content here...
|
||||
*
|
||||
* The first --- pair is YAML frontmatter (handled by gray-matter).
|
||||
* After frontmatter, the body is split at the first standalone ---
|
||||
* (a line containing only --- with optional whitespace).
|
||||
* Everything before is compiled_truth, everything after is timeline.
|
||||
* If no body --- exists, all content is compiled_truth.
|
||||
* After frontmatter, the body is split at the first recognized timeline
|
||||
* sentinel: `<!-- timeline -->` (preferred), `--- timeline ---` (decorated),
|
||||
* or a plain `---` immediately preceding a `## Timeline` / `## History`
|
||||
* heading (backward-compat for existing files). A bare `---` in body text
|
||||
* is treated as a markdown horizontal rule, not a timeline separator.
|
||||
*/
|
||||
export function parseMarkdown(content: string, filePath?: string): ParsedMarkdown {
|
||||
const { data: frontmatter, content: body } = matter(content);
|
||||
@@ -62,26 +64,21 @@ export function parseMarkdown(content: string, filePath?: string): ParsedMarkdow
|
||||
}
|
||||
|
||||
/**
|
||||
* Split body content at first standalone --- separator.
|
||||
* Split body content at the first recognized timeline sentinel.
|
||||
* Returns compiled_truth (before) and timeline (after).
|
||||
*
|
||||
* Recognized sentinels (in order of precedence):
|
||||
* 1. `<!-- timeline -->` — preferred, unambiguous, what serializeMarkdown emits
|
||||
* 2. `--- timeline ---` — decorated separator
|
||||
* 3. `---` ONLY when the next non-empty line is `## Timeline` or `## History`
|
||||
* (backward-compat fallback for older gbrain-written files)
|
||||
*
|
||||
* A plain `---` line is a markdown horizontal rule, NOT a timeline separator.
|
||||
* Treating bare `---` as a separator caused 83% content truncation on wiki corpora.
|
||||
*/
|
||||
export function splitBody(body: string): { compiled_truth: string; timeline: string } {
|
||||
// Match a line that is only --- (with optional whitespace)
|
||||
// Must not be at the very start (that would be frontmatter)
|
||||
const lines = body.split('\n');
|
||||
let splitIndex = -1;
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const trimmed = lines[i].trim();
|
||||
if (trimmed === '---') {
|
||||
// Skip if this is the very first non-empty line (leftover from frontmatter parsing)
|
||||
const beforeContent = lines.slice(0, i).join('\n').trim();
|
||||
if (beforeContent.length > 0) {
|
||||
splitIndex = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
const splitIndex = findTimelineSplitIndex(lines);
|
||||
|
||||
if (splitIndex === -1) {
|
||||
return { compiled_truth: body, timeline: '' };
|
||||
@@ -92,6 +89,33 @@ export function splitBody(body: string): { compiled_truth: string; timeline: str
|
||||
return { compiled_truth, timeline };
|
||||
}
|
||||
|
||||
function findTimelineSplitIndex(lines: string[]): number {
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const trimmed = lines[i].trim();
|
||||
|
||||
if (trimmed === '<!-- timeline -->' || trimmed === '<!--timeline-->') {
|
||||
return i;
|
||||
}
|
||||
|
||||
if (trimmed === '--- timeline ---' || /^---\s+timeline\s+---$/i.test(trimmed)) {
|
||||
return i;
|
||||
}
|
||||
|
||||
if (trimmed === '---') {
|
||||
const beforeContent = lines.slice(0, i).join('\n').trim();
|
||||
if (beforeContent.length === 0) continue;
|
||||
|
||||
for (let j = i + 1; j < lines.length; j++) {
|
||||
const next = lines[j].trim();
|
||||
if (next.length === 0) continue;
|
||||
if (/^##\s+(timeline|history)\b/i.test(next)) return i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize a page back to markdown format.
|
||||
* Produces: frontmatter + compiled_truth + --- + timeline
|
||||
@@ -116,7 +140,7 @@ export function serializeMarkdown(
|
||||
|
||||
let body = compiled_truth;
|
||||
if (timeline) {
|
||||
body += '\n\n---\n\n' + timeline;
|
||||
body += '\n\n<!-- timeline -->\n\n' + timeline;
|
||||
}
|
||||
|
||||
return yamlContent + '\n\n' + body + '\n';
|
||||
@@ -125,8 +149,18 @@ export function serializeMarkdown(
|
||||
function inferType(filePath?: string): PageType {
|
||||
if (!filePath) return 'concept';
|
||||
|
||||
// Normalize: add leading / for consistent matching
|
||||
// Normalize: add leading / for consistent matching.
|
||||
// Wiki subtypes and /writing/ check FIRST — they're stronger signals than
|
||||
// ancestor directories. e.g. `projects/blog/writing/essay.md` is a piece of
|
||||
// writing, not a project page; `tech/wiki/analysis/foo.md` is analysis,
|
||||
// not a hit on the broader `tech/` ancestor.
|
||||
const lower = ('/' + filePath).toLowerCase();
|
||||
if (lower.includes('/writing/')) return 'writing';
|
||||
if (lower.includes('/wiki/analysis/')) return 'analysis';
|
||||
if (lower.includes('/wiki/guides/') || lower.includes('/wiki/guide/')) return 'guide';
|
||||
if (lower.includes('/wiki/hardware/')) return 'hardware';
|
||||
if (lower.includes('/wiki/architecture/')) return 'architecture';
|
||||
if (lower.includes('/wiki/concepts/') || lower.includes('/wiki/concept/')) return 'concept';
|
||||
if (lower.includes('/people/') || lower.includes('/person/')) return 'person';
|
||||
if (lower.includes('/companies/') || lower.includes('/company/')) return 'company';
|
||||
if (lower.includes('/deals/') || lower.includes('/deal/')) return 'deal';
|
||||
|
||||
@@ -17,7 +17,7 @@ import type {
|
||||
} from './types.ts';
|
||||
import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding } from './utils.ts';
|
||||
|
||||
export class PostgresEngine implements BrainEngine {
|
||||
private _sql: ReturnType<typeof postgres> | null = null;
|
||||
@@ -104,7 +104,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
const rows = await sql`
|
||||
INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES (${slug}, ${page.type}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${JSON.stringify(frontmatter)}::jsonb, ${hash}, now())
|
||||
VALUES (${slug}, ${page.type}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter)}, ${hash}, now())
|
||||
ON CONFLICT (slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
title = EXCLUDED.title,
|
||||
@@ -272,7 +272,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
const result = new Map<number, Float32Array>();
|
||||
for (const row of rows) {
|
||||
if (row.embedding) result.set(row.id as number, row.embedding as Float32Array);
|
||||
const parsed = parseEmbedding(row.embedding);
|
||||
if (parsed) result.set(row.id as number, parsed);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -711,7 +712,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
const result = await sql`
|
||||
INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, ${source}, ${JSON.stringify(data)}::jsonb
|
||||
SELECT id, ${source}, ${sql.json(data as Record<string, unknown>)}
|
||||
FROM pages WHERE slug = ${slug}
|
||||
ON CONFLICT (page_id, source) DO UPDATE SET
|
||||
data = EXCLUDED.data,
|
||||
@@ -889,7 +890,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
INSERT INTO ingest_log (source_type, source_ref, pages_updated, summary)
|
||||
VALUES (${entry.source_type}, ${entry.source_ref}, ${JSON.stringify(entry.pages_updated)}::jsonb, ${entry.summary})
|
||||
VALUES (${entry.source_type}, ${entry.source_ref}, ${sql.json(entry.pages_updated)}, ${entry.summary})
|
||||
`;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
// Page types
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media';
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media' | 'writing' | 'analysis' | 'guide' | 'hardware' | 'architecture';
|
||||
|
||||
export interface Page {
|
||||
id: number;
|
||||
|
||||
@@ -43,6 +43,51 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize an embedding value into a Float32Array.
|
||||
*
|
||||
* pgvector returns embeddings in different shapes depending on driver/path:
|
||||
* - postgres.js (Postgres): often a string like `"[0.1,0.2,...]"`
|
||||
* - pglite: typically a numeric array or Float32Array
|
||||
* - pgvector node binding: numeric array
|
||||
* - Some queries that JSON-aggregate embeddings: JSON-string array
|
||||
*
|
||||
* Without normalization, downstream cosine math sees a string and produces
|
||||
* NaN scores silently. This helper guarantees a Float32Array or throws
|
||||
* loudly on malformed input — never returns NaN.
|
||||
*/
|
||||
export function parseEmbedding(value: unknown): Float32Array | null {
|
||||
if (value === null || value === undefined) return null;
|
||||
if (value instanceof Float32Array) return value;
|
||||
if (Array.isArray(value)) {
|
||||
if (value.length === 0) return new Float32Array(0);
|
||||
if (typeof value[0] !== 'number') {
|
||||
throw new Error(`parseEmbedding: array contains non-numeric element (${typeof value[0]})`);
|
||||
}
|
||||
return Float32Array.from(value as number[]);
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const trimmed = value.trim();
|
||||
// Plain non-vector strings: treat as "no embedding here", return null.
|
||||
// Strings that LOOK like vector literals but contain garbage: throw,
|
||||
// because that's a real corruption signal worth surfacing loudly.
|
||||
if (!trimmed.startsWith('[') || !trimmed.endsWith(']')) return null;
|
||||
const inner = trimmed.slice(1, -1).trim();
|
||||
if (inner.length === 0) return new Float32Array(0);
|
||||
const parts = inner.split(',');
|
||||
const out = new Float32Array(parts.length);
|
||||
for (let i = 0; i < parts.length; i++) {
|
||||
const n = Number(parts[i].trim());
|
||||
if (!Number.isFinite(n)) {
|
||||
throw new Error(`parseEmbedding: non-finite value at index ${i}: ${parts[i]}`);
|
||||
}
|
||||
out[i] = n;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function rowToChunk(row: Record<string, unknown>, includeEmbedding = false): Chunk {
|
||||
return {
|
||||
id: row.id as number,
|
||||
@@ -50,7 +95,7 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
|
||||
chunk_index: row.chunk_index as number,
|
||||
chunk_text: row.chunk_text as string,
|
||||
chunk_source: row.chunk_source as 'compiled_truth' | 'timeline',
|
||||
embedding: includeEmbedding && row.embedding ? row.embedding as Float32Array : null,
|
||||
embedding: includeEmbedding ? parseEmbedding(row.embedding) : null,
|
||||
model: row.model as string,
|
||||
token_count: row.token_count as number | null,
|
||||
embedded_at: row.embedded_at ? new Date(row.embedded_at as string) : null,
|
||||
|
||||
Reference in New Issue
Block a user