v0.12.3: Reliability wave — sync deadlock, search timeout scoping, wikilinks, orphans (#216)

* fix(sync): remove nested transaction that deadlocks > 10 file syncs sync.ts wraps the add/modify loop in engine.transaction(), and each importFromContent inside opens another one. PGLite's _runExclusiveTransaction is a non-reentrant mutex — the second call queues on the mutex the first is holding, and the process hangs forever in ep_poll. Reproduced with a 15-file commit: unpatched hangs, patched runs in 3.4s. Fix drops the outer wrap; per-file atomicity is correct anyway (one file's failure should not roll back the others). (cherry picked from commit 4a1ac00105226695d16fb343b44e55a52f44b95b) * test(sync): regression guard for #132 top-level engine.transaction wrap Reads src/commands/sync.ts verbatim and asserts no uncommented engine.transaction() call appears above the add/modify loop. Protects against silent reintroduction of the nested-mutex deadlock that hung > 10-file syncs forever in ep_poll. * feat(utils): tryParseEmbedding() skip+warn sibling for availability path parseEmbedding() throws on structural corruption — right call for ingest/ migrate paths where silent skips would be data loss. Wrong call for search/rescore paths where one corrupt row in 10K would kill every query that touches it. tryParseEmbedding() wraps parseEmbedding in try/catch: returns null on any shape that would throw, warns once per session so the bad row is visible in logs. Use it anywhere we'd rather degrade ranking than blow up the whole query. Retrofit postgres-engine.getEmbeddingsByChunkIds (the #175 slice call site) — the 5-line rescore loop was the direct motivator. Keep the throwing parseEmbedding() for everything else (pglite-engine rowToChunk, migrate-engine round-trips, ingest). * postgres-engine: scope search statement_timeout to the transaction searchKeyword and searchVector run on a pooled postgres.js client (max: 10 by default). The original code bounded each search with await sql`SET statement_timeout = '8s'` try { await sql`<query>` } finally { await sql`SET statement_timeout = '0'` } but every tagged template is an independent round-trip that picks an arbitrary connection from the pool. The SET, the query, and the reset could all land on DIFFERENT connections. In practice the GUC sticks to whichever connection ran the SET and then gets returned to the pool — the next unrelated caller on that connection inherits the 8s timeout (clipping legitimate long queries) or the reset-to-0 (disabling the guard for whoever expected it). A crash in the middle leaves the state set permanently. Wrap each search in sql.begin(async sql => …). postgres.js reserves a single connection for the transaction body, so the SET LOCAL, the query, and the implicit COMMIT all run on the same connection. SET LOCAL scopes the GUC to the transaction — COMMIT or ROLLBACK restores the previous value automatically, regardless of the code path out. Error paths can no longer leak the GUC. No API change. Timeout value and semantics are identical (8s cap on search queries, no effect on embed --all / bulk import which runs outside these methods). Only one transaction per search — BEGIN + COMMIT round-trips are negligible next to a ranked FTS or pgvector query. Also closes the earlier audit finding R4-F002 which reported the same pattern on searchKeyword. This PR covers both searchKeyword and searchVector so the pool-leak class is fully closed. Tests (test/postgres-engine.test.ts, new file): - No bare SET statement_timeout remains after stripping comments. - searchKeyword and searchVector each wrap their query in sql.begin. - Both use SET LOCAL. - Neither explicitly clears the timeout with SET statement_timeout=0. Source-level guardrails keep the fast unit suite DB-free. Live Postgres coverage of the search path is in test/e2e/search-quality.test.ts, which continues to exercise these methods end-to-end against pgvector when DATABASE_URL is set. (cherry picked from commit 6146c3b470dce7380da024a238eab9e6b2174296) * feat(orphans): add gbrain orphans command for finding under-connected pages Surfaces pages with zero inbound wikilinks. Essential for content enrichment cycles in KBs with 1000+ pages. By default filters out auto-generated pages, raw sources, and pseudo-pages where no inbound links is expected; --include-pseudo to disable. Supports text (grouped by domain), --json, --count outputs. Also exposed as find_orphans MCP operation. Tests cover basic detection, filtering, all output modes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit f50954f8e03f85803c6133c85c530bd45e9aceaa) * feat(extract): support Obsidian wikilinks + wiki-style domain slugs in canonical extractor extractEntityRefs now recognizes both syntaxes equally: [Name](people/slug) -- upstream original [[people/slug|Name]] -- Obsidian wikilink (new) Extends DIR_PATTERN to include domain-organized wiki slugs used by Karpathy-style knowledge bases: - entities (legacy prefix some brains keep during migration) - projects (gbrain canonical, was missing from regex) - tech, finance, personal, openclaw (domain-organized wiki roots) Before this change, a 2,100-page brain with wikilinks throughout extracted zero auto-links on put_page because the regex only matched markdown-style [name](path). After: 1,377 new typed edges on a single extract --source db pass over the same corpus. Matches the behavior of the extract.ts filesystem walker (which already handled wikilinks as of the wiki-markdown-compat fix wave), so the db and fs sources now produce the same link graph from the same content. Both patterns share the DIR_PATTERN constant so adding a new entity dir only requires updating one string. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit 1cfb15679a684e94bec5a48c537a0a40a85f57ab) * feat(doctor): jsonb_integrity + markdown_body_completeness detection Add two v0.12.1-era reliability checks to `gbrain doctor`: - `jsonb_integrity` scans the 4 known write sites from the v0.12.0 double-encode bug (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata) and reports rows where jsonb_typeof(col) = 'string'. The fix hint points at `gbrain repair-jsonb` (the standalone repair command shipped in v0.12.1). - `markdown_body_completeness` flags pages whose compiled_truth is <30% of the raw source content length when raw has multiple H2/H3 boundaries. Heuristic only; suggests `gbrain sync --force` or `gbrain import --force <slug>`. Also adds test/e2e/jsonb-roundtrip.test.ts — the regression coverage that should have caught the original double-encode bug. Hits all four write sites against real Postgres and asserts jsonb_typeof='object' plus `->>'key'` returns the expected scalar. Detection only: doctor diagnoses, `gbrain repair-jsonb` treats. No overlap with the standalone repair path. * chore: bump to v0.12.3 + changelog (reliability wave) Master shipped v0.12.1 (extract N+1 + migration timeout) and v0.12.2 (JSONB double-encode + splitBody + wiki types + parseEmbedding) while this wave was mid-flight. Ships the remaining pieces as v0.12.3: - sync deadlock (#132, @sunnnybala) - statement_timeout scoping (#158, @garagon) - Obsidian wikilinks + domain patterns (#187 slice, @knee5) - gbrain orphans command (#187 slice, @knee5) - tryParseEmbedding() availability helper - doctor detection for jsonb_integrity + markdown_body_completeness No schema, no migration, no data touch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: update project documentation for v0.12.3 CLAUDE.md: - Add src/commands/orphans.ts entry - Expand src/commands/doctor.ts with v0.12.3 jsonb_integrity + markdown_body_completeness check descriptions - Update src/core/link-extraction.ts to mention Obsidian wikilinks + extended DIR_PATTERN (entities/projects/tech/finance/personal/openclaw) - Update src/core/utils.ts to mention tryParseEmbedding sibling - Update src/core/postgres-engine.ts to note statement_timeout scoping + tryParseEmbedding usage in getEmbeddingsByChunkIds - Add Key commands added in v0.12.3 section (orphans, doctor checks) - Add test/orphans.test.ts, test/postgres-engine.test.ts, updated descriptions for test/sync.test.ts, test/doctor.test.ts, test/utils.test.ts - Add test/e2e/jsonb-roundtrip.test.ts with note on intentional overlap - Bump operation count from ~36 to ~41 (find_orphans shipped in v0.12.3) README.md: - Add gbrain orphans to ADMIN commands block Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: sunnnybala <dhruvagarwal5018@gmail.com> Co-authored-by: Gustavo Aragon <gustavoraularagon@gmail.com> Co-authored-by: Clevin Canales <clevin@Clevins-MacBook-Pro.local> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Clevin Canales <clev.canales@gmail.com>
2026-04-19 18:23:02 +08:00
parent c0b621923b
commit 013b348c28
20 changed files with 1109 additions and 65 deletions
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -18,7 +18,7 @@ for (const op of operations) {
 }

 // CLI-only commands that bypass the operation layer
-const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check', 'repair-jsonb']);
+const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check', 'repair-jsonb', 'orphans']);

 async function main() {
  const args = process.argv.slice(2);
@@ -417,6 +417,11 @@ async function handleCliOnly(command: string, args: string[]) {
        await runGraphQuery(engine, args);
        break;
      }
+      case 'orphans': {
+        const { runOrphans } = await import('./commands/orphans.ts');
+        await runOrphans(engine, args);
+        break;
+      }
    }
  } finally {
    if (command !== 'serve') await engine.disconnect();
@@ -525,6 +530,7 @@ TOOLS
  publish <page.md> [--password]     Shareable HTML (strips private data, optional AES-256)
  check-backlinks <check|fix> [dir]  Find/fix missing back-links across brain
  lint <dir|file> [--fix]            Catch LLM artifacts, placeholder dates, bad frontmatter
+  orphans [--json] [--count]         Find pages with no inbound wikilinks
  report --type <name> --content ... Save timestamped report to brain/reports/

 JOBS (Minions)
--- a/src/commands/doctor.ts
+++ b/src/commands/doctor.ts
@@ -208,6 +208,74 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
    checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
  }

+  // 9. JSONB integrity (v0.12.1 reliability wave).
+  // v0.12.0's JSON.stringify()::jsonb pattern stored JSONB string literals
+  // instead of objects on real Postgres. PGLite masked this; Supabase did not.
+  // Scan the 4 known sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated,
+  // files.metadata) for rows whose top-level jsonb_typeof is 'string'.
+  try {
+    const sql = db.getConnection();
+    const targets: Array<{ table: string; col: string; expected: 'object' | 'array' }> = [
+      { table: 'pages',      col: 'frontmatter',    expected: 'object' },
+      { table: 'raw_data',   col: 'data',           expected: 'object' },
+      { table: 'ingest_log', col: 'pages_updated',  expected: 'array'  },
+      { table: 'files',      col: 'metadata',       expected: 'object' },
+    ];
+    let totalBad = 0;
+    const breakdown: string[] = [];
+    for (const { table, col } of targets) {
+      const rows = await sql.unsafe(
+        `SELECT count(*)::int AS n FROM ${table} WHERE jsonb_typeof(${col}) = 'string'`,
+      );
+      const n = Number((rows as any)[0]?.n ?? 0);
+      if (n > 0) { totalBad += n; breakdown.push(`${table}.${col}=${n}`); }
+    }
+    if (totalBad === 0) {
+      checks.push({ name: 'jsonb_integrity', status: 'ok', message: 'All JSONB columns store objects/arrays' });
+    } else {
+      checks.push({
+        name: 'jsonb_integrity',
+        status: 'warn',
+        message: `${totalBad} row(s) double-encoded (${breakdown.join(', ')}). Fix: gbrain repair-jsonb`,
+      });
+    }
+  } catch {
+    checks.push({ name: 'jsonb_integrity', status: 'warn', message: 'Could not check JSONB integrity' });
+  }
+
+  // 10. Markdown body completeness (v0.12.1 reliability wave).
+  // v0.12.0's splitBody ate everything after the first `---` horizontal rule,
+  // truncating wiki-style pages. Heuristic: pages whose body is <30% of the
+  // raw source content length when raw has multiple H2/H3 boundaries.
+  try {
+    const sql = db.getConnection();
+    const rows = await sql`
+      SELECT p.slug,
+             length(p.compiled_truth) AS body_len,
+             length(rd.data ->> 'content') AS raw_len
+      FROM pages p
+      JOIN raw_data rd ON rd.page_id = p.id
+      WHERE rd.data ? 'content'
+        AND length(rd.data ->> 'content') > 1000
+        AND length(p.compiled_truth) < length(rd.data ->> 'content') * 0.3
+        AND (rd.data ->> 'content') ~ '(^|\n)##+ '
+      LIMIT 100
+    `;
+    if (rows.length === 0) {
+      checks.push({ name: 'markdown_body_completeness', status: 'ok', message: 'No truncated bodies detected' });
+    } else {
+      const sample = rows.slice(0, 3).map((r: any) => r.slug).join(', ');
+      checks.push({
+        name: 'markdown_body_completeness',
+        status: 'warn',
+        message: `${rows.length} page(s) appear truncated (sample: ${sample}). Re-import with: gbrain sync --force`,
+      });
+    }
+  } catch {
+    // pages_raw.raw_data may not exist on older schemas; best-effort.
+    checks.push({ name: 'markdown_body_completeness', status: 'ok', message: 'Skipped (raw_data unavailable)' });
+  }
+
  const hasFail = outputResults(checks, jsonOutput);

  // Features teaser (non-JSON, non-failing only)
--- a/src/commands/orphans.ts
+++ b/src/commands/orphans.ts
@@ -0,0 +1,227 @@
+/**
+ * gbrain orphans — Surface pages with no inbound wikilinks.
+ *
+ * Deterministic: zero LLM calls. Queries the links table for pages with
+ * no entries where to_page_id = pages.id. By default filters out
+ * auto-generated pages and pseudo-pages where no inbound links is expected.
+ *
+ * Usage:
+ *   gbrain orphans                  # list orphans grouped by domain
+ *   gbrain orphans --json           # JSON output for agent consumption
+ *   gbrain orphans --count          # just the number
+ *   gbrain orphans --include-pseudo # include auto-generated/pseudo pages
+ */
+
+import type { BrainEngine } from '../core/engine.ts';
+import * as db from '../core/db.ts';
+
+// --- Types ---
+
+export interface OrphanPage {
+  slug: string;
+  title: string;
+  domain: string;
+}
+
+export interface OrphanResult {
+  orphans: OrphanPage[];
+  total_orphans: number;
+  total_linkable: number;
+  total_pages: number;
+  excluded: number;
+}
+
+// --- Filter constants ---
+
+/** Slug suffixes that are always auto-generated root files */
+const AUTO_SUFFIX_PATTERNS = ['/_index', '/log'];
+
+/** Page slugs that are pseudo-pages by convention */
+const PSEUDO_SLUGS = new Set(['_atlas', '_index', '_stats', '_orphans', '_scratch', 'claude']);
+
+/** Slug segment that marks raw sources */
+const RAW_SEGMENT = '/raw/';
+
+/** Slug prefixes where no inbound links is expected */
+const DENY_PREFIXES = [
+  'output/',
+  'dashboards/',
+  'scripts/',
+  'templates/',
+  'openclaw/config/',
+];
+
+/** First slug segments where no inbound links is expected */
+const FIRST_SEGMENT_EXCLUSIONS = new Set(['scratch', 'thoughts', 'catalog', 'entities']);
+
+// --- Filter logic ---
+
+/**
+ * Returns true if a slug should be excluded from orphan reporting by default.
+ * These are pages where having no inbound links is expected / not a content problem.
+ */
+export function shouldExclude(slug: string): boolean {
+  // Pseudo-pages (exact match)
+  if (PSEUDO_SLUGS.has(slug)) return true;
+
+  // Auto-generated suffix patterns
+  for (const suffix of AUTO_SUFFIX_PATTERNS) {
+    if (slug.endsWith(suffix)) return true;
+  }
+
+  // Raw source slugs
+  if (slug.includes(RAW_SEGMENT)) return true;
+
+  // Deny-prefix slugs
+  for (const prefix of DENY_PREFIXES) {
+    if (slug.startsWith(prefix)) return true;
+  }
+
+  // First-segment exclusions
+  const firstSegment = slug.split('/')[0];
+  if (FIRST_SEGMENT_EXCLUSIONS.has(firstSegment)) return true;
+
+  return false;
+}
+
+/**
+ * Derive domain from frontmatter or first slug segment.
+ */
+export function deriveDomain(frontmatterDomain: string | null | undefined, slug: string): string {
+  if (frontmatterDomain && typeof frontmatterDomain === 'string' && frontmatterDomain.trim()) {
+    return frontmatterDomain.trim();
+  }
+  return slug.split('/')[0] || 'root';
+}
+
+// --- Core query ---
+
+/**
+ * Find pages with no inbound links.
+ * Returns raw rows from the DB (all pages regardless of filter).
+ */
+export async function queryOrphanPages(): Promise<{ slug: string; title: string; domain: string | null }[]> {
+  const sql = db.getConnection();
+  const rows = await sql`
+    SELECT
+      p.slug,
+      COALESCE(p.title, p.slug) AS title,
+      p.frontmatter->>'domain' AS domain
+    FROM pages p
+    WHERE NOT EXISTS (
+      SELECT 1 FROM links l WHERE l.to_page_id = p.id
+    )
+    ORDER BY p.slug
+  `;
+  return rows as { slug: string; title: string; domain: string | null }[];
+}
+
+/**
+ * Find orphan pages, with optional pseudo-page filtering.
+ * Returns structured OrphanResult with totals.
+ */
+export async function findOrphans(includePseudo: boolean = false): Promise<OrphanResult> {
+  const allOrphans = await queryOrphanPages();
+  const totalPages = allOrphans.length; // pages with no inbound links
+
+  // Count total pages in DB for the summary line
+  const sql = db.getConnection();
+  const [{ count: totalPagesCount }] = await sql`SELECT count(*)::int AS count FROM pages`;
+  const total = Number(totalPagesCount);
+
+  const filtered = includePseudo
+    ? allOrphans
+    : allOrphans.filter(row => !shouldExclude(row.slug));
+
+  const orphans: OrphanPage[] = filtered.map(row => ({
+    slug: row.slug,
+    title: row.title,
+    domain: deriveDomain(row.domain, row.slug),
+  }));
+
+  const excluded = allOrphans.length - filtered.length;
+
+  return {
+    orphans,
+    total_orphans: orphans.length,
+    total_linkable: filtered.length + (total - allOrphans.length),
+    total_pages: total,
+    excluded,
+  };
+}
+
+// --- Output formatters ---
+
+export function formatOrphansText(result: OrphanResult): string {
+  const lines: string[] = [];
+
+  const { orphans, total_orphans, total_linkable, total_pages, excluded } = result;
+  lines.push(
+    `${total_orphans} orphans out of ${total_linkable} linkable pages (${total_pages} total; ${excluded} excluded)\n`,
+  );
+
+  if (orphans.length === 0) {
+    lines.push('No orphan pages found.');
+    return lines.join('\n');
+  }
+
+  // Group by domain, sort alphabetically within each group
+  const byDomain = new Map<string, OrphanPage[]>();
+  for (const page of orphans) {
+    const list = byDomain.get(page.domain) || [];
+    list.push(page);
+    byDomain.set(page.domain, list);
+  }
+
+  // Sort domains alphabetically
+  const sortedDomains = [...byDomain.keys()].sort();
+  for (const domain of sortedDomains) {
+    const pages = byDomain.get(domain)!.sort((a, b) => a.slug.localeCompare(b.slug));
+    lines.push(`[${domain}]`);
+    for (const page of pages) {
+      lines.push(`  ${page.slug}  ${page.title}`);
+    }
+    lines.push('');
+  }
+
+  return lines.join('\n').trimEnd();
+}
+
+// --- CLI entry point ---
+
+export async function runOrphans(_engine: BrainEngine, args: string[]) {
+  const json = args.includes('--json');
+  const count = args.includes('--count');
+  const includePseudo = args.includes('--include-pseudo');
+
+  if (args.includes('--help') || args.includes('-h')) {
+    console.log(`Usage: gbrain orphans [options]
+
+Find pages with no inbound wikilinks.
+
+Options:
+  --json            Output as JSON (for agent consumption)
+  --count           Output just the number of orphans
+  --include-pseudo  Include auto-generated and pseudo pages in results
+  --help, -h        Show this help
+
+Output (default): grouped by domain, sorted alphabetically within each group
+Summary line: N orphans out of M linkable pages (K total; K-M excluded)
+`);
+    return;
+  }
+
+  const result = await findOrphans(includePseudo);
+
+  if (count) {
+    console.log(String(result.total_orphans));
+    return;
+  }
+
+  if (json) {
+    console.log(JSON.stringify(result, null, 2));
+    return;
+  }
+
+  console.log(formatOrphansText(result));
+}
--- a/src/commands/sync.ts
+++ b/src/commands/sync.ts
@@ -203,29 +203,29 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
    pagesAffected.push(newSlug);
  }

-  // Process adds and modifies
-  const useTransaction = (filtered.added.length + filtered.modified.length) > 10;
-  const processAddsModifies = async () => {
-    for (const path of [...filtered.added, ...filtered.modified]) {
-      const filePath = join(repoPath, path);
-      if (!existsSync(filePath)) continue;
-      try {
-        const result = await importFile(engine, filePath, path, { noEmbed });
-        if (result.status === 'imported') {
-          chunksCreated += result.chunks;
-          pagesAffected.push(result.slug);
-        }
-      } catch (e: unknown) {
-        const msg = e instanceof Error ? e.message : String(e);
-        console.error(`  Warning: skipped ${path}: ${msg}`);
+  // Process adds and modifies.
+  //
+  // NOTE: do NOT wrap this loop in engine.transaction(). importFromContent
+  // already opens its own inner transaction per file, and PGLite transactions
+  // are not reentrant — they acquire the same _runExclusiveTransaction mutex,
+  // so a nested call from inside a user callback queues forever on the mutex
+  // the outer transaction is still holding. Result: incremental sync hangs in
+  // ep_poll whenever the diff crosses the old > 10 threshold that used to
+  // trigger the outer wrap. Per-file atomicity is also the right granularity:
+  // one file's failure should not roll back the others' successful imports.
+  for (const path of [...filtered.added, ...filtered.modified]) {
+    const filePath = join(repoPath, path);
+    if (!existsSync(filePath)) continue;
+    try {
+      const result = await importFile(engine, filePath, path, { noEmbed });
+      if (result.status === 'imported') {
+        chunksCreated += result.chunks;
+        pagesAffected.push(result.slug);
      }
+    } catch (e: unknown) {
+      const msg = e instanceof Error ? e.message : String(e);
+      console.error(`  Warning: skipped ${path}: ${msg}`);
    }
-  };
-
-  if (useTransaction) {
-    await engine.transaction(async () => { await processAddsModifies(); });
-  } else {
-    await processAddsModifies();
  }

  const elapsed = Date.now() - start;
--- a/src/core/link-extraction.ts
+++ b/src/core/link-extraction.ts
@@ -27,16 +27,41 @@ export interface EntityRef {
 }

 /**
- * Match `[Name](path)` markdown links pointing to `people/` or `companies/`
- * (and other entity directories). Accepts both filesystem-relative format
- * (`[Name](../people/slug.md)`) AND engine-slug format (`[Name](people/slug)`).
+ * Directory prefix whitelist. These are the top-level slug dirs the extractor
+ * recognizes as entity references. Upstream canonical + our extensions:
+ *   - Gbrain canonical: people, companies, meetings, concepts, deal, civic, project, source, media, yc, projects
+ *   - Our domain extensions: tech, finance, personal, openclaw (domain-organized wikis)
+ *   - Our entity prefix: entities (we kept some legacy entities/projects/ pages)
+ */
+const DIR_PATTERN = '(?:people|companies|meetings|concepts|deal|civic|project|projects|source|media|yc|tech|finance|personal|openclaw|entities)';
+
+/**
+ * Match `[Name](path)` markdown links pointing to entity directories.
+ * Accepts both filesystem-relative format (`[Name](../people/slug.md)`)
+ * AND engine-slug format (`[Name](people/slug)`).
 *
- * Captures: name, dir (people/companies/...), slug.
+ * Captures: name, slug (dir/name, possibly deeper).
 *
 * The regex permits an optional `../` prefix (any number) and an optional
 * `.md` suffix so the same function works for both filesystem and DB content.
 */
-const ENTITY_REF_RE = /\[([^\]]+)\]\((?:\.\.\/)*((?:people|companies|meetings|concepts|deal|civic|project|source|media|yc)\/([^)\s]+?))(?:\.md)?\)/g;
+const ENTITY_REF_RE = new RegExp(
+  `\\[([^\\]]+)\\]\\((?:\\.\\.\\/)*(${DIR_PATTERN}\\/[^)\\s]+?)(?:\\.md)?\\)`,
+  'g',
+);
+
+/**
+ * Match Obsidian-style `[[path]]` or `[[path|Display Text]]` wikilinks.
+ * Captures: slug (dir/...), displayName (optional).
+ *
+ * Same dir whitelist as ENTITY_REF_RE. Strips trailing `.md`, strips section
+ * anchors (`#heading`), skips external URLs. Wiki KBs use this format almost
+ * exclusively so missing it leaves the graph empty.
+ */
+const WIKILINK_RE = new RegExp(
+  `\\[\\[(${DIR_PATTERN}\\/[^|\\]#]+?)(?:#[^|\\]]*?)?(?:\\|([^\\]]+?))?\\]\\]`,
+  'g',
+);

 /**
 * Strip fenced code blocks (```...```) and inline code (`...`) from markdown,
@@ -84,16 +109,30 @@ function stripCodeBlocks(content: string): string {
 export function extractEntityRefs(content: string): EntityRef[] {
  const stripped = stripCodeBlocks(content);
  const refs: EntityRef[] = [];
-  let m: RegExpExecArray | null;
-  // Fresh regex per call (g-flag state is per-instance).
-  const re = new RegExp(ENTITY_REF_RE.source, ENTITY_REF_RE.flags);
-  while ((m = re.exec(stripped)) !== null) {
-    const name = m[1];
-    const fullPath = m[2];
-    const slug = fullPath; // dir/slug
+  let match: RegExpExecArray | null;
+
+  // 1. Markdown links: [Name](path)
+  const mdPattern = new RegExp(ENTITY_REF_RE.source, ENTITY_REF_RE.flags);
+  while ((match = mdPattern.exec(stripped)) !== null) {
+    const name = match[1];
+    const fullPath = match[2];
+    const slug = fullPath;
    const dir = fullPath.split('/')[0];
    refs.push({ name, slug, dir });
  }
+
+  // 2. Obsidian wikilinks: [[path]] or [[path|Display Text]]
+  const wikiPattern = new RegExp(WIKILINK_RE.source, WIKILINK_RE.flags);
+  while ((match = wikiPattern.exec(stripped)) !== null) {
+    let slug = match[1].trim();
+    if (!slug) continue;
+    if (slug.includes('://')) continue;
+    if (slug.endsWith('.md')) slug = slug.slice(0, -3);
+    const displayName = (match[2] || slug).trim();
+    const dir = slug.split('/')[0];
+    refs.push({ name: displayName, slug, dir });
+  }
+
  return refs;
 }

@@ -145,7 +184,10 @@ export function extractPageLinks(
  // Limited to the same entity directories ENTITY_REF_RE covers.
  // Code blocks are stripped first — slugs in code samples are not real refs.
  const strippedContent = stripCodeBlocks(content);
-  const bareRe = /\b((?:people|companies|meetings|concepts|deal|civic|project|source|media|yc)\/[a-z0-9][a-z0-9-]*)\b/g;
+  const bareRe = new RegExp(
+    `\\b(${DIR_PATTERN}\\/[a-z0-9][a-z0-9/-]*[a-z0-9])\\b`,
+    'g',
+  );
  let m: RegExpExecArray | null;
  while ((m = bareRe.exec(strippedContent)) !== null) {
    // Skip matches that are part of a markdown link (already handled above).
--- a/src/core/operations.ts
+++ b/src/core/operations.ts
@@ -1082,6 +1082,24 @@ const send_job_message: Operation = {
  },
 };

+// --- Orphans ---
+
+const find_orphans: Operation = {
+  name: 'find_orphans',
+  description: 'Find pages with no inbound wikilinks. Essential for content enrichment cycles.',
+  params: {
+    include_pseudo: {
+      type: 'boolean',
+      description: 'Include auto-generated and pseudo pages (default: false)',
+    },
+  },
+  handler: async (_ctx, p) => {
+    const { findOrphans } = await import('../commands/orphans.ts');
+    return findOrphans((p.include_pseudo as boolean) || false);
+  },
+  cliHints: { name: 'orphans', hidden: true },
+};
+
 // --- Exports ---

 export const operations: Operation[] = [
@@ -1110,6 +1128,8 @@ export const operations: Operation[] = [
  // Jobs (Minions)
  submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress,
  pause_job, resume_job, replay_job, send_job_message,
+  // Orphans
+  find_orphans,
 ];

 export const operationsByName = Object.fromEntries(
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -17,7 +17,7 @@ import type {
 } from './types.ts';
 import { GBrainError } from './types.ts';
 import * as db from './db.ts';
-import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding } from './utils.ts';
+import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';

 export class PostgresEngine implements BrainEngine {
  private _sql: ReturnType<typeof postgres> | null = null;
@@ -188,11 +188,17 @@ export class PostgresEngine implements BrainEngine {
    const detailLow = opts?.detail === 'low';

    // Search-only timeout: prevents DoS via expensive queries without
-    // affecting long-running operations like embed --all or bulk import
-    await sql`SET statement_timeout = '8s'`;
-    try {
+    // affecting long-running operations like embed --all or bulk import.
+    // SET LOCAL inside sql.begin() scopes the GUC to the transaction so
+    // it can never leak onto a pooled connection returned to other
+    // callers. A bare `SET statement_timeout` goes to an arbitrary
+    // connection from the pool, lives past this method, and either
+    // clips an unrelated caller's long-running query (DoS) or — via
+    // `SET statement_timeout = 0` — disables the guard for them.
+    const rows = await sql.begin(async sql => {
+      await sql`SET LOCAL statement_timeout = '8s'`;
      // CTE: rank pages by FTS score, then pick the best chunk per page in SQL
-      const rows = await sql`
+      return await sql`
        WITH ranked_pages AS (
          SELECT p.id, p.slug, p.title, p.type,
            ts_rank(p.search_vector, websearch_to_tsquery('english', ${query})) AS score
@@ -218,10 +224,8 @@ export class PostgresEngine implements BrainEngine {
        FROM best_chunks
        ORDER BY score DESC
      `;
-      return rows.map(rowToSearchResult);
-    } finally {
-      await sql`SET statement_timeout = '0'`;
-    }
+    });
+    return rows.map(rowToSearchResult);
  }

  async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> {
@@ -238,10 +242,12 @@ export class PostgresEngine implements BrainEngine {

    const vecStr = '[' + Array.from(embedding).join(',') + ']';

-    // Search-only timeout (see searchKeyword for rationale)
-    await sql`SET statement_timeout = '8s'`;
-    try {
-      const rows = await sql`
+    // Search-only timeout (see searchKeyword for rationale). SET LOCAL +
+    // sql.begin ensures the GUC stays transaction-scoped on the pooled
+    // connection.
+    const rows = await sql.begin(async sql => {
+      await sql`SET LOCAL statement_timeout = '8s'`;
+      return await sql`
        SELECT
          p.slug, p.id as page_id, p.title, p.type,
          cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
@@ -257,10 +263,8 @@ export class PostgresEngine implements BrainEngine {
        LIMIT ${limit}
        OFFSET ${offset}
      `;
-      return rows.map(rowToSearchResult);
-    } finally {
-      await sql`SET statement_timeout = '0'`;
-    }
+    });
+    return rows.map(rowToSearchResult);
  }

  async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
@@ -272,8 +276,8 @@ export class PostgresEngine implements BrainEngine {
    `;
    const result = new Map<number, Float32Array>();
    for (const row of rows) {
-      const parsed = parseEmbedding(row.embedding);
-      if (parsed) result.set(row.id as number, parsed);
+      const embedding = tryParseEmbedding(row.embedding);
+      if (embedding) result.set(row.id as number, embedding);
    }
    return result;
  }
--- a/src/core/utils.ts
+++ b/src/core/utils.ts
@@ -88,6 +88,28 @@ export function parseEmbedding(value: unknown): Float32Array | null {
  return null;
 }

+let _tryParseEmbeddingWarned = false;
+
+/**
+ * Availability-path sibling of parseEmbedding(). Returns null + warns once
+ * on any shape parseEmbedding would throw on. Use this on read/rescore paths
+ * where one corrupt row should degrade ranking, not kill the whole query.
+ * Use parseEmbedding() (throws) on ingest/migrate paths where silent skips
+ * would be data loss.
+ */
+export function tryParseEmbedding(value: unknown): Float32Array | null {
+  try {
+    return parseEmbedding(value);
+  } catch (err) {
+    if (!_tryParseEmbeddingWarned) {
+      _tryParseEmbeddingWarned = true;
+      const msg = err instanceof Error ? err.message : String(err);
+      console.warn(`tryParseEmbedding: skipping corrupt embedding row (${msg}). Further warnings suppressed this session.`);
+    }
+    return null;
+  }
+}
+
 export function rowToChunk(row: Record<string, unknown>, includeEmbedding = false): Chunk {
  return {
    id: row.id as number,