feat: v0.13 frontmatter relationship indexing — YAML becomes typed graph edges (#231)
* feat(schema): links provenance + engine plumbing (v0.13)
Adds link_source, origin_page_id, origin_field columns with
UNIQUE NULLS NOT DISTINCT constraint + CHECK constraint. New indexes
on link_source + origin_page_id.
migrate.ts v11 handles idempotent upgrade path for existing brains.
Both engines: addLink/addLinksBatch threads new columns (4→7 col
unnest). removeLink gains linkSource filter. getLinks/getBacklinks
return new columns.
New engine method findByTitleFuzzy(name, dirPrefix?, minSim?) uses
pg_trgm % operator + similarity(). Drives the v0.13 resolver's
fuzzy-match step with zero LLM/embedding cost.
* feat(graph): frontmatter edge extraction + slug resolver (v0.13)
Canonical FRONTMATTER_LINK_MAP: field → type + direction + dir-hint
for 10 frontmatter patterns (company/companies, key_people, investors,
attendees, partner, lead, founded, sources, source, related/see_also).
Direction semantics: "incoming" means resolved value is the FROM side
so subject-of-verb reads naturally (pedro → meeting, not backwards).
makeResolver(engine, {mode}) — two-mode resolver:
batch (migration): slug → dir-hint → pg_trgm. NEVER hits search.
live (put_page): + optional search fallback with expand=false
(dodges hidden Haiku per operations-query learning).
Per-run cache: same name → single DB lookup.
extractFrontmatterLinks handles arrays-of-objects (investors:
[{name: 'Sequoia', role: 'lead'}]), skips bad types silently,
tracks unresolved names for the summary report.
extractPageLinks is now async. LinkCandidate gains fromSlug,
linkSource, originSlug, originField. Returns {candidates, unresolved}.
22 new tests: field-map coverage, direction semantics, source vs
sources, resolver fallback chain (batch + live), cache hit, bad
types skipped, context enrichment, FRONTMATTER_LINK_MAP integrity.
* feat(auto-link): bidirectional reconciliation + unresolved response
put_page auto-link post-hook now handles incoming-direction frontmatter
edges. Reconciliation splits candidates into out (fromSlug === slug)
and in (fromSlug !== slug — frontmatter fields like key_people on a
company page emit person → company edges).
Safe reconciliation via origin_page_id scoping: we only touch
link_source='frontmatter' edges where origin_slug = the page being
written. Markdown + manual edges survive untouched. Edges created
by OTHER pages' frontmatter also survive.
put_page response extends auto_links with unresolved: Array<{field,
name}>. Agents writing attendees: [Pedro, Alex] where Alex doesn't
resolve see it in the response and can queue for enrichment.
Additive — existing agents unaffected.
extract.ts: delete the local 5-field extractFrontmatterLinks + local
inferLinkType. FS-source now calls canonical link-extraction.ts via
a synthetic resolver backed by the allSlugs Set. --include-frontmatter
flag (default OFF in v0.13 for back-compat; migration explicitly
enables for the one-time backfill). Top-20 unresolved names summary
when active.
* feat(migration): v0.13.0 orchestrator
3-phase orchestrator (schema → backfill → verify → record) follows
the v0_12_2.ts pattern. Phase A triggers migrate.ts v11 via
gbrain init --migrate-only. Phase B runs:
gbrain extract links --source db --include-frontmatter
to backfill frontmatter edges for every existing page. Uses the
batch-mode resolver (pg_trgm only, no LLM calls, zero API cost).
Ignores auto_link=false config — migration is canonical, the
auto_link flag controls per-write post-hook not one-time schema
work.
Idempotent + resumable via ON CONFLICT DO NOTHING + origin_page_id
scoping. Wall-clock budget: 2-5 min on 46K-page brains.
Registered in migrations/index.ts. apply-migrations test updated
to include v0.13.0 in skippedFuture for older installed versions.
* feat(release): upgrade-errors.jsonl trail + doctor surfacing
upgrade.ts catches post-upgrade subprocess failures as best-effort
today (line 65 comment: "post-upgrade is best-effort, don't fail
the upgrade"). When that chain silently fails, users end up with
half-upgraded brains and no signal.
v0.13: on post-upgrade failure, append a structured record to
~/.gbrain/upgrade-errors.jsonl with ts, phase, versions, error
message, and a paste-ready recovery hint.
doctor.ts reads the jsonl and surfaces the latest entry with a
warn-status check. User runs gbrain doctor, sees exactly what
failed, pastes the recovery command, files an issue if needed.
Applies to every future release — doctor grows with the codebase
without per-release edits. The CHANGELOG pattern ("To take advantage
of v[version]" block) mirrors this in user-facing form.
* chore: bump version and changelog (v0.13.0)
v0.13.0 — Frontmatter Relationship Indexing.
Adds the "To take advantage of v[version]" block pattern to
CHANGELOG format (CLAUDE.md documents the requirement going
forward). Pairs with the upgrade-errors.jsonl + doctor surfacing
to close the "half-upgraded brain, no signal" loop.
UPGRADING_DOWNSTREAM_AGENTS.md gets a v0.13 section: no-action-
required verdict for most skills, optional diffs for meeting-
ingestion / enrich / idea-ingest if they want to consume
auto_links.unresolved.
skills/migrations/v0.13.0.md is the user-facing upgrade skill.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.13): adversarial review P0s
Codex + Claude adversarial review caught 4 critical issues in the
v0.13 implementation. Fixing before ship.
1. findByTitleFuzzy SET LOCAL was a no-op. postgres.js auto-commits
each sql`` so SET LOCAL pg_trgm.similarity_threshold committed
before the `%` operator ran against it. Resolver used server
default (0.3, not 0.55) → way too many fuzzy matches, wrong
links on a 46K-page brain. Switched to inline
`similarity(title, $1) >= $N` which has no transaction scoping.
Added `ORDER BY sim DESC, slug ASC` for deterministic
tie-breaking (prevents reconciliation churn on re-runs).
2. v11 migration now checks Postgres ≥ 15 before applying
UNIQUE NULLS NOT DISTINCT. Old Supabase projects on PG14 would
have dropped the old unique constraint and failed to add the
new one, corrupting the uniqueness invariant. The check raises
a clear error with the actual PG version, leaving the old
constraint in place.
3. v11 migration now backfills NULL link_source → 'markdown' for
pre-v0.13 legacy rows. Without this, reconciliation's existKey
comparison treats NULL and 'markdown' as equivalent but the
unique constraint sees them as distinct (NULLS NOT DISTINCT
only collapses NULL with NULL, not NULL with 'markdown'). Result
was duplicate edges accumulating forever. Treating legacy as
markdown is the accurate best-guess — pre-v0.13 auto-link only
emitted markdown edges.
4. v0_13_0.ts orchestrator now uses process.execPath, not a bare
`gbrain` on PATH. After `gbrain upgrade` rewrites the binary,
alias shadowing / PATH caching / multiple installs could
resolve a stale `gbrain` binary. process.execPath is always
the binary that loaded this migration module.
Phase C verify clarified: reports page + link counts and points to
Phase B's own stdout as the authoritative signal for backfill
results (extract.ts already prints `Links: created N from M pages`).
* docs: scrub real names from public docs + add privacy rule to CLAUDE.md
Public artifacts (CHANGELOG, skills, docs) should never reveal real
contacts, companies, funds, or private agent-fork names from any
user's brain. When a doc copies a query like `gbrain graph diana-hu`
or names a fork like `Wintermute`, that real name gets indexed,
cross-referenced, and distributed with every release.
CLAUDE.md gains a "Privacy rule: scrub real names from public docs"
section with:
- What counts as public (CHANGELOG, README, docs/, skills/, PR bodies,
commit messages, code comments)
- Name mapping table (agent forks → your agent fork; example person →
alice-example; example fund → fund-a; etc.)
- Distinction between illustrative API examples with household brands
(Stripe, Brex) and queries that reveal real relationships
Applied the rule to v0.13 scope:
- CHANGELOG v0.13 entry: Pedro/Diana/Wintermute/Sequoia/Benchmark/a16z
all replaced with alice/charlie/fund-a/acme/agent-fork placeholders
- skills/migrations/v0.13.0.md: same
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: Wintermute references scrubbed
throughout (pre-v0.13 and v0.13 sections)
- CLAUDE.md: "Brain skills (from Wintermute)" → "(ported from an
upstream agent fork)", internal Wintermute provenance notes
genericized, "Garry finds fragile upgrade paths" → "the gbrain
maintainers find fragile upgrade paths" in the template
Pre-v0.13 historical CHANGELOG entries (v0.10-v0.12) left alone —
those are shipped releases; rewriting changes public history.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -97,6 +97,32 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
|
||||
// handles the "schema v7+ but no prefs" case.
|
||||
}
|
||||
|
||||
// 3b. Upgrade-error trail (v0.13+). `gbrain upgrade` silently swallows
|
||||
// best-effort failures in `gbrain post-upgrade`; the failure record is
|
||||
// appended to ~/.gbrain/upgrade-errors.jsonl so we can surface it here
|
||||
// with a paste-ready recovery hint. Without this, users end up with
|
||||
// half-upgraded brains and no signal.
|
||||
try {
|
||||
const home = process.env.HOME || '';
|
||||
const errPath = join(home, '.gbrain', 'upgrade-errors.jsonl');
|
||||
if (existsSync(errPath)) {
|
||||
const lines = readFileSync(errPath, 'utf-8').split('\n').filter(l => l.trim());
|
||||
if (lines.length > 0) {
|
||||
const latest = JSON.parse(lines[lines.length - 1]) as {
|
||||
ts: string; phase: string; from_version: string; to_version: string; hint: string;
|
||||
};
|
||||
const date = latest.ts.slice(0, 10);
|
||||
checks.push({
|
||||
name: 'upgrade_errors',
|
||||
status: 'warn',
|
||||
message: `Post-upgrade failure on ${date} (${latest.from_version} → ${latest.to_version}, phase: ${latest.phase}). Recovery: ${latest.hint}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Read/parse failure is itself best-effort; skip silently.
|
||||
}
|
||||
|
||||
// --- DB checks (skip if --fast or no engine) ---
|
||||
|
||||
if (fastMode || !engine) {
|
||||
|
||||
@@ -21,7 +21,11 @@ import { join, relative, dirname } from 'path';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import { extractPageLinks, parseTimelineEntries, inferLinkType } from '../core/link-extraction.ts';
|
||||
import {
|
||||
extractPageLinks, parseTimelineEntries, inferLinkType, makeResolver,
|
||||
extractFrontmatterLinks,
|
||||
type UnresolvedFrontmatterRef,
|
||||
} from '../core/link-extraction.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
@@ -142,8 +146,19 @@ export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<st
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Infer link type from directory structure */
|
||||
function inferLinkType(fromDir: string, toDir: string, frontmatter?: Record<string, unknown>): string {
|
||||
/**
|
||||
* Directory-based link-type inference for the fs-source path.
|
||||
*
|
||||
* FS-source operates without a BrainEngine. We have paths, not pages. This
|
||||
* helper looks at source + target directories and returns a type aligned
|
||||
* with the canonical `inferLinkType` in link-extraction.ts (calibrated
|
||||
* verb-based inference for db-source).
|
||||
*
|
||||
* v0.13: aligned type names with link-extraction.ts (was: 'mention' →
|
||||
* 'mentions', 'attendee' → 'attended'). Diverged historically; the v0_13_0
|
||||
* migration normalizes any legacy rows on existing brains.
|
||||
*/
|
||||
function inferTypeByDir(fromDir: string, toDir: string, frontmatter?: Record<string, unknown>): string {
|
||||
const from = fromDir.split('/')[0];
|
||||
const to = toDir.split('/')[0];
|
||||
if (from === 'people' && to === 'companies') {
|
||||
@@ -152,31 +167,8 @@ function inferLinkType(fromDir: string, toDir: string, frontmatter?: Record<stri
|
||||
}
|
||||
if (from === 'people' && to === 'deals') return 'involved_in';
|
||||
if (from === 'deals' && to === 'companies') return 'deal_for';
|
||||
if (from === 'meetings' && to === 'people') return 'attendee';
|
||||
return 'mention';
|
||||
}
|
||||
|
||||
/** Extract links from frontmatter fields */
|
||||
function extractFrontmatterLinks(slug: string, fm: Record<string, unknown>): ExtractedLink[] {
|
||||
const links: ExtractedLink[] = [];
|
||||
const fieldMap: Record<string, { dir: string; type: string }> = {
|
||||
company: { dir: 'companies', type: 'works_at' },
|
||||
companies: { dir: 'companies', type: 'works_at' },
|
||||
investors: { dir: 'companies', type: 'invested_in' },
|
||||
attendees: { dir: 'people', type: 'attendee' },
|
||||
founded: { dir: 'companies', type: 'founded' },
|
||||
};
|
||||
for (const [field, config] of Object.entries(fieldMap)) {
|
||||
const value = fm[field];
|
||||
if (!value) continue;
|
||||
const slugs = Array.isArray(value) ? value : [value];
|
||||
for (const s of slugs) {
|
||||
if (typeof s !== 'string') continue;
|
||||
const toSlug = `${config.dir}/${s.toLowerCase().replace(/\s+/g, '-')}`;
|
||||
links.push({ from_slug: slug, to_slug: toSlug, link_type: config.type, context: `frontmatter.${field}` });
|
||||
}
|
||||
}
|
||||
return links;
|
||||
if (from === 'meetings' && to === 'people') return 'attended';
|
||||
return 'mentions';
|
||||
}
|
||||
|
||||
/** Parse frontmatter using the project's gray-matter-based parser */
|
||||
@@ -189,10 +181,19 @@ function parseFrontmatterFromContent(content: string, relPath: string): Record<s
|
||||
}
|
||||
}
|
||||
|
||||
/** Full link extraction from a single markdown file */
|
||||
export function extractLinksFromFile(
|
||||
/**
|
||||
* Full link extraction from a single markdown file (FS-source path).
|
||||
*
|
||||
* Async (v0.13): uses the canonical `extractFrontmatterLinks` via a
|
||||
* synthetic resolver backed by the pre-loaded `allSlugs` Set. No DB,
|
||||
* no fuzzy match — FS-source resolves only when the dir-hint + slugify
|
||||
* of the frontmatter value hits an actual file path. That mirrors the
|
||||
* fs path's existing "exact match against disk" behavior.
|
||||
*/
|
||||
export async function extractLinksFromFile(
|
||||
content: string, relPath: string, allSlugs: Set<string>,
|
||||
): ExtractedLink[] {
|
||||
opts?: { includeFrontmatter?: boolean },
|
||||
): Promise<ExtractedLink[]> {
|
||||
const links: ExtractedLink[] = [];
|
||||
const slug = relPath.replace('.md', '');
|
||||
const fileDir = dirname(relPath);
|
||||
@@ -203,13 +204,51 @@ export function extractLinksFromFile(
|
||||
if (resolved !== null) {
|
||||
links.push({
|
||||
from_slug: slug, to_slug: resolved,
|
||||
link_type: inferLinkType(fileDir, dirname(resolved), fm),
|
||||
link_type: inferTypeByDir(fileDir, dirname(resolved), fm),
|
||||
context: `markdown link: [${name}]`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
links.push(...extractFrontmatterLinks(slug, fm));
|
||||
if (opts?.includeFrontmatter) {
|
||||
// Synthetic sync-ish resolver: only does step 1 (already a slug) and
|
||||
// step 2 (dir-hint + slugify), backed by the Set of all known slugs.
|
||||
const slugify = (s: string) => s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
|
||||
const fsResolver = {
|
||||
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
|
||||
if (!name) return null;
|
||||
const trimmed = name.trim();
|
||||
if (/^[a-z][a-z0-9-]*\/[a-z0-9][a-z0-9-]*$/.test(trimmed) && allSlugs.has(trimmed)) {
|
||||
return trimmed;
|
||||
}
|
||||
const hints = Array.isArray(dirHint) ? dirHint : (dirHint ? [dirHint] : []);
|
||||
for (const hint of hints) {
|
||||
if (!hint) continue;
|
||||
const candidate = `${hint}/${slugify(trimmed)}`;
|
||||
if (allSlugs.has(candidate)) return candidate;
|
||||
}
|
||||
return null;
|
||||
},
|
||||
};
|
||||
// Guess the page type from its directory for field-map filtering.
|
||||
const topDir = slug.split('/')[0];
|
||||
const pageType = topDir === 'people' ? 'person'
|
||||
: topDir === 'companies' ? 'company'
|
||||
: topDir === 'deals' || topDir === 'deal' ? 'deal'
|
||||
: topDir === 'meetings' ? 'meeting'
|
||||
: 'concept';
|
||||
const fm = parseFrontmatterFromContent(content, relPath);
|
||||
const fmLinks = await extractFrontmatterLinks(slug, pageType as never, fm, fsResolver);
|
||||
for (const c of fmLinks.candidates) {
|
||||
links.push({
|
||||
from_slug: c.fromSlug ?? slug,
|
||||
to_slug: c.targetSlug,
|
||||
link_type: c.linkType,
|
||||
context: c.context,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return links;
|
||||
}
|
||||
|
||||
@@ -300,6 +339,10 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
const since = (sinceIdx >= 0 && sinceIdx + 1 < args.length) ? args[sinceIdx + 1] : undefined;
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const jsonMode = args.includes('--json');
|
||||
// --include-frontmatter: v0.13 flag. Default OFF for back-compat. The
|
||||
// v0_13_0 migration orchestrator runs this once under the hood; users
|
||||
// opt in for subsequent runs.
|
||||
const includeFrontmatter = args.includes('--include-frontmatter');
|
||||
|
||||
// Validate --since upfront. Without this, an invalid date like
|
||||
// `--since yesterday` produces NaN which silently passes the filter check
|
||||
@@ -337,7 +380,7 @@ export async function runExtract(engine: BrainEngine, args: string[]) {
|
||||
// can opt in via mode + source.
|
||||
result = { links_created: 0, timeline_entries_created: 0, pages_processed: 0 };
|
||||
if (subcommand === 'links' || subcommand === 'all') {
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since);
|
||||
const r = await extractLinksFromDB(engine, dryRun, jsonMode, typeFilter, since, { includeFrontmatter });
|
||||
result.links_created = r.created;
|
||||
result.pages_processed = r.pages;
|
||||
}
|
||||
@@ -397,7 +440,7 @@ async function extractLinksFromDir(
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
try {
|
||||
const content = readFileSync(files[i].path, 'utf-8');
|
||||
const links = extractLinksFromFile(content, files[i].relPath, allSlugs);
|
||||
const links = await extractLinksFromFile(content, files[i].relPath, allSlugs);
|
||||
for (const link of links) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`;
|
||||
@@ -491,7 +534,7 @@ export async function extractLinksForSlugs(engine: BrainEngine, repoPath: string
|
||||
if (!existsSync(filePath)) continue;
|
||||
try {
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
for (const link of extractLinksFromFile(content, slug + '.md', allSlugs)) {
|
||||
for (const link of await extractLinksFromFile(content, slug + '.md', allSlugs)) {
|
||||
try { await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type); created++; } catch { /* skip */ }
|
||||
}
|
||||
} catch { /* skip */ }
|
||||
@@ -527,7 +570,18 @@ async function extractLinksFromDB(
|
||||
jsonMode: boolean,
|
||||
typeFilter: PageType | undefined,
|
||||
since: string | undefined,
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
opts?: { includeFrontmatter?: boolean },
|
||||
): Promise<{ created: number; pages: number; unresolved: UnresolvedFrontmatterRef[] }> {
|
||||
const includeFrontmatter = opts?.includeFrontmatter ?? false;
|
||||
// Batch resolver: pg_trgm + exact only, NO search fallback. Dodges the
|
||||
// N-thousand API call trap on 46K-page brains. Resolver has a per-run
|
||||
// cache so duplicate names (same person appearing on many pages) resolve
|
||||
// once, not once per mention.
|
||||
const resolver = makeResolver(engine, { mode: 'batch' });
|
||||
const unresolved: UnresolvedFrontmatterRef[] = [];
|
||||
const nullResolver = {
|
||||
resolve: async () => null as string | null,
|
||||
};
|
||||
const allSlugs = await engine.getAllSlugs();
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
@@ -564,25 +618,45 @@ async function extractLinksFromDB(
|
||||
}
|
||||
|
||||
const fullContent = page.compiled_truth + '\n' + page.timeline;
|
||||
const candidates = extractPageLinks(fullContent, page.frontmatter, page.type);
|
||||
// --include-frontmatter default OFF in v0.13 (codex tension 5, back-compat).
|
||||
// Migration orchestrator explicitly enables it for the one-time backfill;
|
||||
// user-invoked `gbrain extract links` stays outgoing-only.
|
||||
const activeResolver = includeFrontmatter ? resolver : nullResolver;
|
||||
const extracted = await extractPageLinks(
|
||||
slug, fullContent, page.frontmatter, page.type, activeResolver,
|
||||
);
|
||||
unresolved.push(...extracted.unresolved);
|
||||
|
||||
for (const c of candidates) {
|
||||
for (const c of extracted.candidates) {
|
||||
// Validate BOTH endpoints exist. Incoming frontmatter edges have
|
||||
// fromSlug !== the page being processed; we need that page to exist
|
||||
// too or the JOIN drops the row anyway.
|
||||
const fromSlug = c.fromSlug ?? slug;
|
||||
if (!allSlugs.has(c.targetSlug)) continue;
|
||||
if (!allSlugs.has(fromSlug)) continue;
|
||||
if (dryRunSeen) {
|
||||
const key = `${slug}::${c.targetSlug}::${c.linkType}`;
|
||||
const key = `${fromSlug}::${c.targetSlug}::${c.linkType}::${c.linkSource ?? 'markdown'}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_link', from: slug, to: c.targetSlug,
|
||||
type: c.linkType, context: c.context,
|
||||
action: 'add_link', from: fromSlug, to: c.targetSlug,
|
||||
type: c.linkType, context: c.context, link_source: c.linkSource,
|
||||
}) + '\n');
|
||||
} else {
|
||||
console.log(` ${slug} → ${c.targetSlug} (${c.linkType})`);
|
||||
console.log(` ${fromSlug} → ${c.targetSlug} (${c.linkType})${c.linkSource === 'frontmatter' ? ' [fm]' : ''}`);
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
batch.push({ from_slug: slug, to_slug: c.targetSlug, link_type: c.linkType, context: c.context });
|
||||
batch.push({
|
||||
from_slug: fromSlug,
|
||||
to_slug: c.targetSlug,
|
||||
link_type: c.linkType,
|
||||
context: c.context,
|
||||
link_source: c.linkSource,
|
||||
origin_slug: c.originSlug,
|
||||
origin_field: c.originField,
|
||||
});
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
@@ -596,8 +670,22 @@ async function extractLinksFromDB(
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
console.log(`Links: ${label} ${created} from ${processed} pages (db source)`);
|
||||
if (includeFrontmatter && unresolved.length > 0) {
|
||||
// Top-20 preview of unresolvable frontmatter names so the user can
|
||||
// see where the graph has holes (codex tension 6.4).
|
||||
console.log(`Unresolved frontmatter refs: ${unresolved.length} total`);
|
||||
const bucket = new Map<string, number>();
|
||||
for (const u of unresolved) {
|
||||
const key = `${u.field}:${u.name}`;
|
||||
bucket.set(key, (bucket.get(key) || 0) + 1);
|
||||
}
|
||||
const top = Array.from(bucket.entries()).sort((a, b) => b[1] - a[1]).slice(0, 20);
|
||||
for (const [key, count] of top) {
|
||||
console.log(` ${count}× ${key}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { created, pages: processed };
|
||||
return { created, pages: processed, unresolved };
|
||||
}
|
||||
|
||||
async function extractTimelineFromDB(
|
||||
|
||||
@@ -14,11 +14,13 @@ import type { Migration } from './types.ts';
|
||||
import { v0_11_0 } from './v0_11_0.ts';
|
||||
import { v0_12_0 } from './v0_12_0.ts';
|
||||
import { v0_12_2 } from './v0_12_2.ts';
|
||||
import { v0_13_0 } from './v0_13_0.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
v0_12_0,
|
||||
v0_12_2,
|
||||
v0_13_0,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
175
src/commands/migrations/v0_13_0.ts
Normal file
175
src/commands/migrations/v0_13_0.ts
Normal file
@@ -0,0 +1,175 @@
|
||||
/**
|
||||
* v0.13.0 migration orchestrator — frontmatter relationship indexing.
|
||||
*
|
||||
* v0.13 extends the knowledge graph to project typed edges from YAML
|
||||
* frontmatter (company, investors, attendees, key_people, etc.), not just
|
||||
* `[Name](path)` markdown refs. This migration:
|
||||
*
|
||||
* A. Schema — `gbrain init --migrate-only` triggers migrate.ts v11 which
|
||||
* adds link_source + origin_page_id + origin_field columns,
|
||||
* swaps the unique constraint to include them, and creates
|
||||
* new indexes.
|
||||
* B. Backfill — `gbrain extract links --source db --include-frontmatter`
|
||||
* walks every page and emits the frontmatter-derived edges.
|
||||
* Uses the batch-mode resolver (pg_trgm only, no LLM).
|
||||
* C. Verify — Query the links table and confirm link_source='frontmatter'
|
||||
* rows exist (> 0 on any brain with frontmatter content).
|
||||
* D. Record — append to ~/.gbrain/completed.jsonl.
|
||||
*
|
||||
* Idempotent. Resumable from `partial` via ON CONFLICT DO NOTHING on the
|
||||
* new unique constraint. Wall-clock budget on 46K-page brains: 2-5 min
|
||||
* (pg_trgm index-backed, no embedding or LLM calls).
|
||||
*
|
||||
* Ignores `auto_link=false` config: migration is canonical (CLAUDE.md),
|
||||
* not advisory. The auto_link toggle controls the put_page post-hook,
|
||||
* not one-time schema+backfill work.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
//
|
||||
// migrate.ts v11 adds the link_source/origin_page_id/origin_field columns
|
||||
// and swaps the unique constraint. Schema build time on 46K pages is
|
||||
// ~10s (ALTER + index builds). Bumped timeout accounts for slow Supabase
|
||||
// links (v0.12.1 pattern — migrations can time out on the 60s default).
|
||||
// Use the CURRENTLY-RUNNING binary path (not `gbrain` off $PATH). After
|
||||
// `gbrain upgrade` rewrites the binary, a bare `gbrain` could resolve to
|
||||
// an older installed copy via alias shadowing or stale PATH cache. The
|
||||
// active process.execPath is the one that loaded THIS migration module,
|
||||
// so recursing into it is always the right binary.
|
||||
const GBRAIN = process.execPath;
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync(`${GBRAIN} init --migrate-only`, { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'schema', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase B — Frontmatter edge backfill ─────────────────────
|
||||
|
||||
function phaseBBackfill(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'frontmatter_backfill', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// `--source db` iterates pages from the engine (no local checkout required).
|
||||
// `--include-frontmatter` is the v0.13 flag that enables the canonical
|
||||
// frontmatter link extractor. Default-OFF in the CLI for back-compat;
|
||||
// the migration explicitly opts in because this is the canonical backfill.
|
||||
execSync(`${GBRAIN} extract links --source db --include-frontmatter`, {
|
||||
stdio: 'inherit',
|
||||
timeout: 1_800_000, // 30 min hard cap; typical 2-5 min on 46K pages
|
||||
env: process.env,
|
||||
});
|
||||
return { name: 'frontmatter_backfill', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'frontmatter_backfill', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase C — Verify ────────────────────────────────────────
|
||||
|
||||
function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
// Query frontmatter edge count via get_stats + a secondary --json call
|
||||
// to `gbrain graph-query` as a smoke test: extract one random page and
|
||||
// confirm it has at least one edge. Non-blocking.
|
||||
//
|
||||
// We intentionally do NOT fail on 0 frontmatter edges: fresh installs,
|
||||
// docs-only brains, and brains with no entity pages legitimately
|
||||
// produce 0. Phase B's own stdout shows `Links: created N` which is
|
||||
// the authoritative signal — user sees it during upgrade.
|
||||
const out = execSync(`${GBRAIN} call get_stats`, {
|
||||
encoding: 'utf-8', timeout: 60_000, env: process.env,
|
||||
});
|
||||
const parsed = JSON.parse(out) as { link_count?: number; page_count?: number };
|
||||
const linkCount = parsed.link_count ?? 0;
|
||||
const pageCount = parsed.page_count ?? 0;
|
||||
return {
|
||||
name: 'verify',
|
||||
status: 'complete',
|
||||
detail: `pages=${pageCount}, links=${linkCount} (backfill output in Phase B logs)`,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'verify', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.13.0 — Frontmatter relationship indexing ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const b = phaseBBackfill(opts);
|
||||
phases.push(b);
|
||||
// Backfill failure → partial. Schema is already applied so re-running
|
||||
// only re-tries the backfill (idempotent via ON CONFLICT DO NOTHING).
|
||||
if (b.status === 'failed') return finalizeResult(phases, 'partial');
|
||||
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' || b.status === 'failed' ? 'failed' :
|
||||
c.status === 'failed' ? 'partial' :
|
||||
'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
|
||||
if (status !== 'failed') {
|
||||
try {
|
||||
appendCompletedMigration({ version: '0.13.0', status: status as 'complete' | 'partial' });
|
||||
} catch {
|
||||
// Recording is best-effort.
|
||||
}
|
||||
}
|
||||
return {
|
||||
version: '0.13.0',
|
||||
status,
|
||||
phases,
|
||||
};
|
||||
}
|
||||
|
||||
export const v0_13_0: Migration = {
|
||||
version: '0.13.0',
|
||||
featurePitch: {
|
||||
headline: 'Frontmatter becomes a graph — company, investors, attendees now create typed edges automatically',
|
||||
description:
|
||||
'v0.13 extends the knowledge graph to project typed edges from YAML frontmatter. ' +
|
||||
'Every `company: X`, `investors: [A, B]`, `attendees: [Pedro, Garry]`, `key_people`, ' +
|
||||
'`partner`, `lead`, and `related` field you already wrote now surfaces in ' +
|
||||
'`gbrain graph`. Direction semantics respect subject-of-verb (Pedro → meeting, ' +
|
||||
'not meeting → Pedro). The migration backfills every existing page in ~2-5 min ' +
|
||||
'on a 46K-page brain. Uses pg_trgm fuzzy-match for name resolution (zero LLM ' +
|
||||
'cost, zero API calls). Unresolvable names surface in the extract summary so you ' +
|
||||
'see exactly where the graph has holes.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBBackfill,
|
||||
phaseCVerify,
|
||||
};
|
||||
@@ -1,5 +1,5 @@
|
||||
import { execSync } from 'child_process';
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { VERSION } from '../version.ts';
|
||||
|
||||
@@ -61,8 +61,18 @@ export async function runUpgrade(args: string[]) {
|
||||
// autopilot install) on a v0.11.0→v0.11.1 jump. Codex H7.
|
||||
try {
|
||||
execSync('gbrain post-upgrade', { stdio: 'inherit', timeout: 300_000 });
|
||||
} catch {
|
||||
// post-upgrade is best-effort, don't fail the upgrade
|
||||
} catch (e) {
|
||||
// post-upgrade is best-effort, don't fail the upgrade. BUT leave a
|
||||
// trail so `gbrain doctor` can surface it and give the user a clear
|
||||
// paste-ready recovery command. Silent failure here is how users end
|
||||
// up with half-upgraded brains and no signal.
|
||||
recordUpgradeError({
|
||||
phase: 'post-upgrade',
|
||||
fromVersion: oldVersion,
|
||||
toVersion: newVersion,
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
hint: 'Run: gbrain apply-migrations --yes',
|
||||
});
|
||||
}
|
||||
// Run features scan to show what's new and what to fix
|
||||
try {
|
||||
@@ -84,6 +94,39 @@ function verifyUpgrade(): string {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append a structured record to ~/.gbrain/upgrade-errors.jsonl when a
|
||||
* best-effort phase of the upgrade fails (e.g., `gbrain post-upgrade`
|
||||
* silently bombing). Without this trail, users end up with half-upgraded
|
||||
* brains and no signal. `gbrain doctor` reads this file and surfaces the
|
||||
* paste-ready recovery hint. Failures here are themselves best-effort.
|
||||
*/
|
||||
export function recordUpgradeError(record: {
|
||||
phase: string;
|
||||
fromVersion: string;
|
||||
toVersion: string;
|
||||
error: string;
|
||||
hint: string;
|
||||
}): void {
|
||||
try {
|
||||
const dir = join(process.env.HOME || '', '.gbrain');
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const path = join(dir, 'upgrade-errors.jsonl');
|
||||
const line = JSON.stringify({
|
||||
ts: new Date().toISOString(),
|
||||
phase: record.phase,
|
||||
from_version: record.fromVersion,
|
||||
to_version: record.toVersion,
|
||||
error: record.error,
|
||||
hint: record.hint,
|
||||
}) + '\n';
|
||||
appendFileSync(path, line);
|
||||
} catch {
|
||||
// Recording errors is itself best-effort. The user will still see the
|
||||
// underlying failure in stdout/stderr from the original command.
|
||||
}
|
||||
}
|
||||
|
||||
function saveUpgradeState(oldVersion: string, newVersion: string) {
|
||||
try {
|
||||
const dir = join(process.env.HOME || '', '.gbrain');
|
||||
|
||||
@@ -17,6 +17,17 @@ export interface LinkBatchInput {
|
||||
to_slug: string;
|
||||
link_type?: string;
|
||||
context?: string;
|
||||
/**
|
||||
* Provenance (v0.13+). Pass 'frontmatter' for edges derived from YAML
|
||||
* frontmatter, 'markdown' for [Name](path) refs, 'manual' for user-created.
|
||||
* NULL means "legacy / unknown" and is only used by pre-v0.13 rows; new
|
||||
* writes should always set this. Missing on input defaults to 'markdown'.
|
||||
*/
|
||||
link_source?: string;
|
||||
/** For link_source='frontmatter': slug of the page whose frontmatter created this edge. */
|
||||
origin_slug?: string;
|
||||
/** Frontmatter field name (e.g. 'key_people', 'investors'). */
|
||||
origin_field?: string;
|
||||
}
|
||||
|
||||
/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
@@ -69,7 +80,20 @@ export interface BrainEngine {
|
||||
deleteChunks(slug: string): Promise<void>;
|
||||
|
||||
// Links
|
||||
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
|
||||
/**
|
||||
* Single-row link insert. linkSource defaults to 'markdown' for back-compat
|
||||
* with pre-v0.13 callers. Pass 'frontmatter' + originSlug + originField for
|
||||
* frontmatter-derived edges; 'manual' for user-initiated edges.
|
||||
*/
|
||||
addLink(
|
||||
from: string,
|
||||
to: string,
|
||||
context?: string,
|
||||
linkType?: string,
|
||||
linkSource?: string,
|
||||
originSlug?: string,
|
||||
originField?: string,
|
||||
): Promise<void>;
|
||||
/**
|
||||
* Bulk insert links via a single multi-row INSERT...SELECT FROM (VALUES) JOIN pages
|
||||
* statement with ON CONFLICT DO NOTHING. Returns the count of rows actually inserted
|
||||
@@ -80,11 +104,32 @@ export interface BrainEngine {
|
||||
/**
|
||||
* Remove links from `from` to `to`. If linkType is provided, only that specific
|
||||
* (from, to, type) row is removed. If omitted, ALL link types between the pair
|
||||
* are removed (matches pre-multi-type-link behavior).
|
||||
* are removed (matches pre-multi-type-link behavior). linkSource additionally
|
||||
* constrains the delete to a specific provenance ('frontmatter', 'markdown',
|
||||
* 'manual') — used by runAutoLink reconciliation to avoid deleting edges from
|
||||
* other provenances when pruning frontmatter-derived edges.
|
||||
*/
|
||||
removeLink(from: string, to: string, linkType?: string): Promise<void>;
|
||||
removeLink(from: string, to: string, linkType?: string, linkSource?: string): Promise<void>;
|
||||
getLinks(slug: string): Promise<Link[]>;
|
||||
getBacklinks(slug: string): Promise<Link[]>;
|
||||
/**
|
||||
* Fuzzy-match a display name to a page slug using pg_trgm similarity.
|
||||
* Zero embedding cost, zero LLM cost — designed for the v0.13 resolver used
|
||||
* during migration/batch backfill where 5K+ lookups must stay sub-second.
|
||||
*
|
||||
* Returns the best match whose title similarity is at or above `minSimilarity`
|
||||
* (default 0.55). If `dirPrefix` is given (e.g. 'people' or 'companies'),
|
||||
* only slugs starting with that prefix are considered. Returns null when no
|
||||
* page meets the threshold.
|
||||
*
|
||||
* Uses the `%` trigram operator (GIN-indexed) + the standard `similarity()`
|
||||
* function. Both engines support pg_trgm (PGLite 0.3+, Postgres always).
|
||||
*/
|
||||
findByTitleFuzzy(
|
||||
name: string,
|
||||
dirPrefix?: string,
|
||||
minSimilarity?: number,
|
||||
): Promise<{ slug: string; similarity: number } | null>;
|
||||
traverseGraph(slug: string, depth?: number): Promise<GraphNode[]>;
|
||||
/**
|
||||
* Edge-based graph traversal with optional type and direction filters.
|
||||
|
||||
@@ -139,12 +139,47 @@ export function extractEntityRefs(content: string): EntityRef[] {
|
||||
// ─── Link candidates (richer than EntityRef) ────────────────────
|
||||
|
||||
export interface LinkCandidate {
|
||||
/**
|
||||
* Source page slug for the edge. When omitted, callers default to
|
||||
* "the page being written" (operations.ts runAutoLink) or "the page
|
||||
* currently being processed" (extract.ts). Explicitly set when
|
||||
* frontmatter emits an incoming edge — e.g. a company page's
|
||||
* `key_people: [pedro-franceschi]` produces a candidate whose
|
||||
* fromSlug is `people/pedro-franceschi`, not the company.
|
||||
*/
|
||||
fromSlug?: string;
|
||||
/** Target page slug (no .md, no ../). */
|
||||
targetSlug: string;
|
||||
/** Inferred relationship type. */
|
||||
linkType: string;
|
||||
/** Surrounding text (up to ~80 chars) used for inference + storage. */
|
||||
context: string;
|
||||
/**
|
||||
* Provenance (v0.13+). Defaults to 'markdown' on older call sites;
|
||||
* frontmatter-derived candidates set 'frontmatter'; user-created edges
|
||||
* via explicit API pass 'manual'.
|
||||
*/
|
||||
linkSource?: string;
|
||||
/**
|
||||
* Origin-page slug. Only populated for link_source='frontmatter' so
|
||||
* reconciliation can scope cleanups to edges THIS page's frontmatter
|
||||
* created (never touching edges other pages authored).
|
||||
*/
|
||||
originSlug?: string;
|
||||
/** Frontmatter field name (e.g. 'key_people'), for debug + unresolved report. */
|
||||
originField?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of extractPageLinks. `candidates` includes markdown refs + bare
|
||||
* slug refs + frontmatter-derived edges (v0.13). `unresolved` lists
|
||||
* frontmatter names that did not resolve to any page — surfaced in the
|
||||
* put_page auto_links response and the extract summary so users know
|
||||
* where the graph has holes.
|
||||
*/
|
||||
export interface PageLinksResult {
|
||||
candidates: LinkCandidate[];
|
||||
unresolved: UnresolvedFrontmatterRef[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -153,16 +188,24 @@ export interface LinkCandidate {
|
||||
* Sources:
|
||||
* 1. Markdown entity refs in compiled_truth + timeline (extractEntityRefs).
|
||||
* 2. Bare slug references in text (people/slug, companies/slug).
|
||||
* 3. Frontmatter `source:` field (creates a 'source' link).
|
||||
* 3. Frontmatter fields → typed graph edges (v0.13: company, investors,
|
||||
* attendees, key_people, etc.). See FRONTMATTER_LINK_MAP.
|
||||
*
|
||||
* Within-page dedup: multiple mentions of the same (targetSlug, linkType)
|
||||
* collapse to one candidate. The first occurrence's context wins.
|
||||
* ASYNC (v0.13): frontmatter extraction resolves display names to slugs
|
||||
* via the supplied resolver, which may hit the DB. Pre-v0.13 callers
|
||||
* that don't care about frontmatter can pass a resolver that always
|
||||
* returns null; only markdown/bare-slug candidates are emitted.
|
||||
*
|
||||
* Within-page dedup: multiple mentions of the same (fromSlug, targetSlug,
|
||||
* linkType) tuple collapse to one candidate. First occurrence wins.
|
||||
*/
|
||||
export function extractPageLinks(
|
||||
export async function extractPageLinks(
|
||||
slug: string,
|
||||
content: string,
|
||||
frontmatter: Record<string, unknown>,
|
||||
pageType: PageType,
|
||||
): LinkCandidate[] {
|
||||
resolver: SlugResolver,
|
||||
): Promise<PageLinksResult> {
|
||||
const candidates: LinkCandidate[] = [];
|
||||
|
||||
// 1. Markdown entity refs.
|
||||
@@ -177,6 +220,7 @@ export function extractPageLinks(
|
||||
targetSlug: ref.slug,
|
||||
linkType: inferLinkType(pageType, context, content, ref.slug),
|
||||
context,
|
||||
linkSource: 'markdown',
|
||||
});
|
||||
}
|
||||
|
||||
@@ -198,30 +242,26 @@ export function extractPageLinks(
|
||||
targetSlug: m[1],
|
||||
linkType: inferLinkType(pageType, context, content, m[1]),
|
||||
context,
|
||||
linkSource: 'markdown',
|
||||
});
|
||||
}
|
||||
|
||||
// 3. Frontmatter source field.
|
||||
const source = frontmatter.source;
|
||||
if (typeof source === 'string' && source.length > 0 && /^[a-z][a-z0-9-]*\/[a-z0-9][a-z0-9-]*$/.test(source)) {
|
||||
candidates.push({
|
||||
targetSlug: source,
|
||||
linkType: 'source',
|
||||
context: `frontmatter source: ${source}`,
|
||||
});
|
||||
}
|
||||
// 3. Frontmatter-derived edges (v0.13). Includes the legacy `source:`
|
||||
// field along with the full field map.
|
||||
const fm = await extractFrontmatterLinks(slug, pageType, frontmatter, resolver);
|
||||
candidates.push(...fm.candidates);
|
||||
|
||||
// Within-page dedup: same (targetSlug, linkType) collapses to one entry.
|
||||
// First occurrence wins (preserves the most natural/earliest context).
|
||||
// Within-page dedup: same (fromSlug, targetSlug, linkType, linkSource)
|
||||
// collapses to one entry. First occurrence wins.
|
||||
const seen = new Set<string>();
|
||||
const result: LinkCandidate[] = [];
|
||||
for (const c of candidates) {
|
||||
const key = `${c.targetSlug}\u0000${c.linkType}`;
|
||||
const key = `${c.fromSlug ?? ''}\u0000${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? ''}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(c);
|
||||
}
|
||||
return result;
|
||||
return { candidates: result, unresolved: fm.unresolved };
|
||||
}
|
||||
|
||||
/** Excerpt a window of `width` chars around `idx`, collapsed to one line. */
|
||||
@@ -311,6 +351,272 @@ export function inferLinkType(pageType: PageType, context: string, globalContext
|
||||
return 'mentions';
|
||||
}
|
||||
|
||||
// ─── Frontmatter link extraction (v0.13) ────────────────────────
|
||||
//
|
||||
// YAML frontmatter on entity pages carries rich relationship data:
|
||||
//
|
||||
// company: "Stripe" # person page
|
||||
// companies: [Stripe, Plaid] # person page (alias of company)
|
||||
// key_people: [Patrick Collison, John] # company page (incoming works_at)
|
||||
// investors: [{name: Sequoia}, Benchmark] # deal page (incoming invested_in)
|
||||
// attendees: [Pedro, Garry] # meeting page (incoming attended)
|
||||
//
|
||||
// Each maps to a typed graph edge. The mapping lives here (one source of
|
||||
// truth) so the three entry points — operations.ts auto-link, extract.ts
|
||||
// fs source, extract.ts db source — emit identical edges for the same
|
||||
// frontmatter. This is the point of the v0.13 rewrite.
|
||||
//
|
||||
// DIRECTION: "incoming" means the page being written is the TO side;
|
||||
// the FROM side is the resolved frontmatter value. E.g. `key_people:
|
||||
// [Pedro]` on company/stripe emits `people/pedro -> companies/stripe
|
||||
// type=works_at`, preserving subject-of-verb semantics for graph reads.
|
||||
//
|
||||
// MULTI-DIR HINTS: investors can be companies, funds, or people. The
|
||||
// resolver tries each hint in order and takes the first match.
|
||||
|
||||
export interface FrontmatterFieldMapping {
|
||||
/** Field name(s). Multiple entries are aliases (e.g. company + companies). */
|
||||
fields: string[];
|
||||
/**
|
||||
* Only applies when page.type matches. Omitted = any page type. String
|
||||
* (not PageType) because some page types like 'meeting' exist in the
|
||||
* pages table without being in the TypeScript PageType enum.
|
||||
*/
|
||||
pageType?: string;
|
||||
/** Edge link_type. */
|
||||
type: string;
|
||||
/** 'outgoing' = page→target. 'incoming' = target→page (subject of verb = from). */
|
||||
direction: 'outgoing' | 'incoming';
|
||||
/**
|
||||
* Target directory hints for slug resolution. Single string or ordered
|
||||
* array; resolver tries each. E.g. investors → ['companies', 'funds', 'people'].
|
||||
*/
|
||||
dirHint: string | string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonical field → (type, direction, dir-hint) map. Consulted by
|
||||
* extractFrontmatterLinks for every YAML field on every written page.
|
||||
*
|
||||
* NOT normalization: kept as a flat array so duplicate field names with
|
||||
* different pageType filters coexist cleanly (vs an object-literal which
|
||||
* would last-write-wins on key collision).
|
||||
*/
|
||||
export const FRONTMATTER_LINK_MAP: FrontmatterFieldMapping[] = [
|
||||
// Person pages → companies
|
||||
{ fields: ['company', 'companies'], pageType: 'person', type: 'works_at', direction: 'outgoing', dirHint: 'companies' },
|
||||
{ fields: ['founded'], pageType: 'person', type: 'founded', direction: 'outgoing', dirHint: 'companies' },
|
||||
// Company pages (incoming relationships — subject of the verb lives elsewhere)
|
||||
{ fields: ['key_people'], pageType: 'company', type: 'works_at', direction: 'incoming', dirHint: 'people' },
|
||||
{ fields: ['partner'], pageType: 'company', type: 'yc_partner', direction: 'incoming', dirHint: 'people' },
|
||||
{ fields: ['investors'], pageType: 'company', type: 'invested_in', direction: 'incoming',
|
||||
dirHint: ['companies', 'funds', 'people'] },
|
||||
// Deal pages (all incoming — deals are the object)
|
||||
{ fields: ['investors'], pageType: 'deal', type: 'invested_in', direction: 'incoming',
|
||||
dirHint: ['companies', 'funds', 'people'] },
|
||||
{ fields: ['lead'], pageType: 'deal', type: 'led_round', direction: 'incoming',
|
||||
dirHint: ['companies', 'funds', 'people'] },
|
||||
// Meeting pages
|
||||
{ fields: ['attendees'], pageType: 'meeting', type: 'attended', direction: 'incoming', dirHint: 'people' },
|
||||
// Any page type
|
||||
{ fields: ['sources'], type: 'discussed_in', direction: 'incoming', dirHint: ['source', 'media'] },
|
||||
{ fields: ['source'], type: 'source', direction: 'outgoing', dirHint: '' /* already slug-shaped */ },
|
||||
{ fields: ['related', 'see_also'], type: 'related_to', direction: 'outgoing', dirHint: '' },
|
||||
];
|
||||
|
||||
// ─── Slug resolver ──────────────────────────────────────────────
|
||||
|
||||
export interface SlugResolver {
|
||||
/**
|
||||
* Resolve a display name to a canonical slug.
|
||||
* Returns null when no match meets confidence threshold — callers should
|
||||
* skip (not write a dead link) and the unresolved name goes into the
|
||||
* extract/put_page summary so the user can see the gap.
|
||||
*/
|
||||
resolve(name: string, dirHint?: string | string[]): Promise<string | null>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a resolver scoped to a single extract run or single put_page call.
|
||||
*
|
||||
* mode: 'batch' (migration / gbrain extract) — pg_trgm only, NO search
|
||||
* fallback. On a 46K-page brain this avoids N-thousand OpenAI embedding
|
||||
* calls + Anthropic Haiku expansion calls (see operations-query-hidden-haiku
|
||||
* learning) and keeps the backfill deterministic + under a wall-clock budget.
|
||||
*
|
||||
* mode: 'live' (put_page auto-link) — can afford the (rare, bounded) search
|
||||
* fallback for names that don't fuzzy-match. Still passes expand=false to
|
||||
* dodge Haiku.
|
||||
*
|
||||
* cache: per-resolver instance. Same name → same slug lookup every call.
|
||||
* Callers never need to dedupe names themselves.
|
||||
*/
|
||||
export function makeResolver(
|
||||
engine: BrainEngine,
|
||||
opts: { mode: 'batch' | 'live' } = { mode: 'live' },
|
||||
): SlugResolver {
|
||||
const cache = new Map<string, string | null>();
|
||||
|
||||
const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9\s-]/g, '').trim().replace(/\s+/g, '-');
|
||||
|
||||
return {
|
||||
async resolve(name: string, dirHint?: string | string[]): Promise<string | null> {
|
||||
if (!name || typeof name !== 'string') return null;
|
||||
const trimmed = name.trim();
|
||||
if (!trimmed) return null;
|
||||
|
||||
const cacheKey = `${trimmed}\u0000${Array.isArray(dirHint) ? dirHint.join(',') : (dirHint || '')}`;
|
||||
if (cache.has(cacheKey)) return cache.get(cacheKey)!;
|
||||
|
||||
const hints = Array.isArray(dirHint) ? dirHint : (dirHint ? [dirHint] : []);
|
||||
|
||||
// Step 1: already a slug? (dir/name shape, lowercase, hyphenated)
|
||||
if (/^[a-z][a-z0-9-]*\/[a-z0-9][a-z0-9-]*$/.test(trimmed)) {
|
||||
const page = await engine.getPage(trimmed);
|
||||
if (page) {
|
||||
cache.set(cacheKey, trimmed);
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
|
||||
// Step 2: dir-hint + slugify → exact getPage
|
||||
const slugified = norm(trimmed);
|
||||
for (const hint of hints) {
|
||||
if (!hint) continue;
|
||||
const candidate = `${hint}/${slugified}`;
|
||||
const page = await engine.getPage(candidate);
|
||||
if (page) {
|
||||
cache.set(cacheKey, candidate);
|
||||
return candidate;
|
||||
}
|
||||
}
|
||||
|
||||
// Step 3: pg_trgm fuzzy title match — both modes. Tries each hint in
|
||||
// order; first hint with a ≥0.55 similarity match wins. If no hints,
|
||||
// try the whole pages table.
|
||||
const searchHints = hints.length > 0 ? hints : [undefined];
|
||||
for (const hint of searchHints) {
|
||||
const match = await engine.findByTitleFuzzy(trimmed, hint, 0.55);
|
||||
if (match) {
|
||||
cache.set(cacheKey, match.slug);
|
||||
return match.slug;
|
||||
}
|
||||
}
|
||||
|
||||
// Step 4: live-mode ONLY — fall back to hybrid search. expand: false
|
||||
// is MANDATORY (see operations-query-hidden-haiku learning). Batch
|
||||
// mode skips this step entirely to keep migration deterministic.
|
||||
if (opts.mode === 'live') {
|
||||
try {
|
||||
const results = await engine.searchKeyword(trimmed, { limit: 3 });
|
||||
if (results.length > 0 && results[0].score >= 0.8) {
|
||||
// Filter by dir hint if provided.
|
||||
const top = hints.length > 0
|
||||
? results.find(r => hints.some(h => r.slug.startsWith(`${h}/`)))
|
||||
: results[0];
|
||||
if (top) {
|
||||
cache.set(cacheKey, top.slug);
|
||||
return top.slug;
|
||||
}
|
||||
}
|
||||
} catch { /* search errors are non-fatal; fall through to null */ }
|
||||
}
|
||||
|
||||
// Null = unresolvable. Caller records for the unresolved report.
|
||||
cache.set(cacheKey, null);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Frontmatter extractor ──────────────────────────────────────
|
||||
|
||||
export interface UnresolvedFrontmatterRef {
|
||||
/** The frontmatter field name. */
|
||||
field: string;
|
||||
/** The name that did not resolve. */
|
||||
name: string;
|
||||
}
|
||||
|
||||
export interface FrontmatterExtractResult {
|
||||
candidates: LinkCandidate[];
|
||||
unresolved: UnresolvedFrontmatterRef[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract typed graph edges from YAML frontmatter. Async because the
|
||||
* resolver may need to query the DB for fuzzy matches.
|
||||
*
|
||||
* Arrays of strings: each entry resolved independently.
|
||||
* Arrays of objects: uses the `name` or `slug` property (codex tension 6.3).
|
||||
* Non-string / non-object entries: silently skipped (log-only).
|
||||
*/
|
||||
export async function extractFrontmatterLinks(
|
||||
slug: string,
|
||||
pageType: PageType,
|
||||
frontmatter: Record<string, unknown>,
|
||||
resolver: SlugResolver,
|
||||
): Promise<FrontmatterExtractResult> {
|
||||
const candidates: LinkCandidate[] = [];
|
||||
const unresolved: UnresolvedFrontmatterRef[] = [];
|
||||
|
||||
for (const mapping of FRONTMATTER_LINK_MAP) {
|
||||
if (mapping.pageType && mapping.pageType !== pageType) continue;
|
||||
for (const field of mapping.fields) {
|
||||
const value = frontmatter[field];
|
||||
if (value == null) continue;
|
||||
const entries = Array.isArray(value) ? value : [value];
|
||||
|
||||
for (const entry of entries) {
|
||||
// Extract the name to resolve. Strings pass through; objects use
|
||||
// the `name` / `slug` / `title` field in that preference order.
|
||||
let name: string | null = null;
|
||||
let contextExtra = '';
|
||||
if (typeof entry === 'string') {
|
||||
name = entry;
|
||||
} else if (entry && typeof entry === 'object') {
|
||||
const obj = entry as Record<string, unknown>;
|
||||
const n = obj.name ?? obj.slug ?? obj.title;
|
||||
if (typeof n === 'string') {
|
||||
name = n;
|
||||
// Carry interesting object fields (role, title) into the context.
|
||||
const extras: string[] = [];
|
||||
if (typeof obj.role === 'string') extras.push(obj.role);
|
||||
if (typeof obj.title === 'string' && obj.title !== n) extras.push(obj.title);
|
||||
if (extras.length > 0) contextExtra = ` (${extras.join(', ')})`;
|
||||
}
|
||||
}
|
||||
if (!name) continue; // skip numbers, nulls, malformed objects
|
||||
|
||||
const resolved = await resolver.resolve(name, mapping.dirHint);
|
||||
if (!resolved) {
|
||||
unresolved.push({ field, name });
|
||||
continue;
|
||||
}
|
||||
|
||||
// Outgoing: page → resolved. Incoming: resolved → page.
|
||||
const fromSlug = mapping.direction === 'outgoing' ? slug : resolved;
|
||||
const toSlug = mapping.direction === 'outgoing' ? resolved : slug;
|
||||
// Context enrichment (review Finding 7): readable in backlink panels
|
||||
// and search snippets instead of bare `frontmatter.key_people`.
|
||||
const context = `frontmatter.${field}: ${name}${contextExtra}`;
|
||||
|
||||
candidates.push({
|
||||
fromSlug,
|
||||
targetSlug: toSlug,
|
||||
linkType: mapping.type,
|
||||
context,
|
||||
linkSource: 'frontmatter',
|
||||
originSlug: slug, // the page whose frontmatter created this edge
|
||||
originField: field,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { candidates, unresolved };
|
||||
}
|
||||
|
||||
// ─── Timeline parsing ───────────────────────────────────────────
|
||||
|
||||
export interface TimelineCandidate {
|
||||
|
||||
@@ -284,6 +284,70 @@ export const MIGRATIONS: Migration[] = [
|
||||
DROP FUNCTION IF EXISTS update_page_search_vector_from_timeline();
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 11,
|
||||
name: 'links_provenance_columns',
|
||||
// v0.13: adds provenance columns so frontmatter-derived edges can be
|
||||
// distinguished from markdown/manual edges. Reconciliation on put_page
|
||||
// scopes by (link_source='frontmatter' AND origin_page_id = written_page)
|
||||
// so edges from other pages never get mis-deleted.
|
||||
//
|
||||
// Unique constraint swaps: old (from, to, type) blocks coexistence of
|
||||
// markdown + frontmatter + manual edges with the same tuple. New tuple
|
||||
// includes link_source + origin_page_id.
|
||||
//
|
||||
// Existing rows keep link_source IS NULL (legacy marker) — they are NOT
|
||||
// backfilled to 'markdown' because existing rows may be manual/imported
|
||||
// /inferred; mislabeling them as markdown would corrupt provenance.
|
||||
//
|
||||
// Idempotent via IF NOT EXISTS / DROP IF EXISTS.
|
||||
sql: `
|
||||
-- Postgres version gate: UNIQUE NULLS NOT DISTINCT requires PG15+.
|
||||
-- PGLite ships PG17.5, current Supabase is PG15+. Old Supabase projects
|
||||
-- on PG14 hit an explicit error rather than half-applying (drop old
|
||||
-- constraint but fail to add new one → brain loses uniqueness guarantee).
|
||||
DO $$ BEGIN
|
||||
IF current_setting('server_version_num')::int < 150000 THEN
|
||||
RAISE EXCEPTION
|
||||
'v0.13 migration requires Postgres 15+. Current: %. '
|
||||
'Upgrade your Postgres (Supabase: migrate project to a newer PG major). '
|
||||
'This migration intentionally stops before touching the schema to preserve data integrity.',
|
||||
current_setting('server_version');
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
|
||||
DO $$ BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint WHERE conname = 'links_link_source_check'
|
||||
) THEN
|
||||
ALTER TABLE links ADD CONSTRAINT links_link_source_check
|
||||
CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual'));
|
||||
END IF;
|
||||
END $$;
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
|
||||
REFERENCES pages(id) ON DELETE SET NULL;
|
||||
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_field TEXT;
|
||||
-- Backfill NULL link_source → 'markdown' for existing rows. Codex review
|
||||
-- caught that without this, pre-v0.13 legacy rows coexist with new
|
||||
-- 'markdown' writes under NULLS NOT DISTINCT (NULL ≠ 'markdown'),
|
||||
-- causing duplicate edges to accumulate. Treating legacy as markdown
|
||||
-- is the accurate best-guess: pre-v0.13 auto-link only emitted markdown
|
||||
-- edges. User-created 'manual' edges are a v0.13+ concept anyway.
|
||||
UPDATE links SET link_source = 'markdown' WHERE link_source IS NULL;
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique;
|
||||
DO $$ BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint WHERE conname = 'links_from_to_type_source_origin_unique'
|
||||
) THEN
|
||||
ALTER TABLE links ADD CONSTRAINT links_from_to_type_source_origin_unique
|
||||
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id);
|
||||
END IF;
|
||||
END $$;
|
||||
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -13,7 +13,7 @@ import { importFromContent } from './import-file.ts';
|
||||
import { hybridSearch } from './search/hybrid.ts';
|
||||
import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled } from './link-extraction.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import * as db from './db.ts';
|
||||
|
||||
// --- Types ---
|
||||
@@ -248,7 +248,11 @@ const put_page: Operation = {
|
||||
// Combined with the backlink boost in hybridSearch, attacker-placed targets
|
||||
// would surface higher in search. Local CLI users (ctx.remote=false) opt
|
||||
// into this behavior; MCP/remote writes do not.
|
||||
let autoLinks: { created: number; removed: number; errors: number } | { error: string } | { skipped: 'remote' } | undefined;
|
||||
let autoLinks:
|
||||
| { created: number; removed: number; errors: number; unresolved: UnresolvedFrontmatterRef[] }
|
||||
| { error: string }
|
||||
| { skipped: 'remote' }
|
||||
| undefined;
|
||||
if (ctx.remote === true) {
|
||||
autoLinks = { skipped: 'remote' };
|
||||
} else if (result.parsedPage) {
|
||||
@@ -286,43 +290,114 @@ async function runAutoLink(
|
||||
engine: BrainEngine,
|
||||
slug: string,
|
||||
parsed: { type: PageType; compiled_truth: string; timeline: string; frontmatter: Record<string, unknown> },
|
||||
): Promise<{ created: number; removed: number; errors: number }> {
|
||||
): Promise<{ created: number; removed: number; errors: number; unresolved: UnresolvedFrontmatterRef[] }> {
|
||||
const fullContent = parsed.compiled_truth + '\n' + parsed.timeline;
|
||||
const candidates = extractPageLinks(fullContent, parsed.frontmatter, parsed.type);
|
||||
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
|
||||
const resolver = makeResolver(engine, { mode: 'live' });
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
slug, fullContent, parsed.frontmatter, parsed.type, resolver,
|
||||
);
|
||||
|
||||
// Resolve which targets exist (skip refs to non-existent pages to avoid FK
|
||||
// violation churn in addLink). One getAllSlugs call upfront, O(1) lookup.
|
||||
const allSlugs = await engine.getAllSlugs();
|
||||
const valid = candidates.filter(c => allSlugs.has(c.targetSlug));
|
||||
const valid = candidates.filter(c =>
|
||||
allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug))
|
||||
);
|
||||
|
||||
// Split candidates by direction. Outgoing (fromSlug === slug or unset) are
|
||||
// this page's own edges, reconciled against getLinks(slug). Incoming
|
||||
// (fromSlug !== slug — frontmatter with `direction: incoming`) are edges
|
||||
// where this page is the TO side; reconciled against getBacklinks(slug)
|
||||
// but SCOPED to the frontmatter edges this page authored via
|
||||
// (link_source='frontmatter' AND origin_slug = slug). We never touch
|
||||
// frontmatter edges authored by OTHER pages.
|
||||
const out = valid.filter(c => !c.fromSlug || c.fromSlug === slug);
|
||||
const inc = valid.filter(c => c.fromSlug && c.fromSlug !== slug);
|
||||
|
||||
// Run getLinks + addLink/removeLink loops inside a single transaction so that
|
||||
// concurrent put_page calls on the same slug can't race the reconciliation:
|
||||
// without this, two simultaneous writes both read stale `existingKeys` and
|
||||
// re-create links the other side just removed (lost-update). The transaction
|
||||
// serializes via row-level locks on `links` rows touched by addLink/removeLink.
|
||||
return await engine.transaction(async (tx) => {
|
||||
const existing = await tx.getLinks(slug);
|
||||
const desiredKeys = new Set(valid.map(c => `${c.targetSlug}\u0000${c.linkType}`));
|
||||
const existingKeys = new Set(existing.map(l => `${l.to_slug}\u0000${l.link_type}`));
|
||||
const result = await engine.transaction(async (tx) => {
|
||||
const existingOut = await tx.getLinks(slug);
|
||||
// Incoming: we only look at frontmatter edges WE authored (origin_slug=slug).
|
||||
// Non-frontmatter and other-page frontmatter edges survive untouched.
|
||||
const existingInRaw = await tx.getBacklinks(slug);
|
||||
const existingIn = existingInRaw.filter(
|
||||
l => l.link_source === 'frontmatter' && l.origin_slug === slug,
|
||||
);
|
||||
|
||||
// Reconcilable outgoing edges: markdown + our own frontmatter edges.
|
||||
// Manual edges (link_source='manual') are NEVER touched by reconciliation.
|
||||
const reconcilableOut = existingOut.filter(
|
||||
l => l.link_source === 'markdown' || l.link_source == null ||
|
||||
(l.link_source === 'frontmatter' && l.origin_slug === slug),
|
||||
);
|
||||
|
||||
const outKeys = new Set(out.map(c =>
|
||||
`${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? 'markdown'}`
|
||||
));
|
||||
const incKeys = new Set(inc.map(c =>
|
||||
`${c.fromSlug}\u0000${c.linkType}`
|
||||
));
|
||||
|
||||
let created = 0, removed = 0, errors = 0;
|
||||
|
||||
// Add new + update existing.
|
||||
for (const c of valid) {
|
||||
// Add outgoing edges.
|
||||
for (const c of out) {
|
||||
try {
|
||||
await tx.addLink(slug, c.targetSlug, c.context, c.linkType);
|
||||
if (!existingKeys.has(`${c.targetSlug}\u0000${c.linkType}`)) created++;
|
||||
await tx.addLink(
|
||||
slug, c.targetSlug, c.context, c.linkType,
|
||||
c.linkSource, c.originSlug, c.originField,
|
||||
);
|
||||
const existKey = `${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? 'markdown'}`;
|
||||
const exists = reconcilableOut.some(l =>
|
||||
`${l.to_slug}\u0000${l.link_type}\u0000${l.link_source ?? 'markdown'}` === existKey
|
||||
);
|
||||
if (!exists) created++;
|
||||
} catch {
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
// Remove stale (in DB but not in desired set).
|
||||
for (const l of existing) {
|
||||
const key = `${l.to_slug}\u0000${l.link_type}`;
|
||||
if (!desiredKeys.has(key)) {
|
||||
// Add incoming edges (other page → slug).
|
||||
for (const c of inc) {
|
||||
try {
|
||||
await tx.addLink(
|
||||
c.fromSlug!, c.targetSlug, c.context, c.linkType,
|
||||
'frontmatter', c.originSlug, c.originField,
|
||||
);
|
||||
const existKey = `${c.fromSlug}\u0000${c.linkType}`;
|
||||
const exists = existingIn.some(l =>
|
||||
`${l.from_slug}\u0000${l.link_type}` === existKey
|
||||
);
|
||||
if (!exists) created++;
|
||||
} catch {
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
// Remove stale outgoing (markdown or our-frontmatter, not in desired set).
|
||||
for (const l of reconcilableOut) {
|
||||
const key = `${l.to_slug}\u0000${l.link_type}\u0000${l.link_source ?? 'markdown'}`;
|
||||
if (!outKeys.has(key)) {
|
||||
try {
|
||||
await tx.removeLink(slug, l.to_slug, l.link_type);
|
||||
await tx.removeLink(slug, l.to_slug, l.link_type, l.link_source ?? undefined);
|
||||
removed++;
|
||||
} catch {
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Remove stale incoming (our frontmatter → slug, not in desired set).
|
||||
for (const l of existingIn) {
|
||||
const key = `${l.from_slug}\u0000${l.link_type}`;
|
||||
if (!incKeys.has(key)) {
|
||||
try {
|
||||
await tx.removeLink(l.from_slug, slug, l.link_type, 'frontmatter');
|
||||
removed++;
|
||||
} catch {
|
||||
errors++;
|
||||
@@ -332,6 +407,8 @@ async function runAutoLink(
|
||||
|
||||
return { created, removed, errors };
|
||||
});
|
||||
|
||||
return { ...result, unresolved };
|
||||
}
|
||||
|
||||
const delete_page: Operation = {
|
||||
|
||||
@@ -321,43 +321,67 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Links
|
||||
async addLink(from: string, to: string, context?: string, linkType?: string): Promise<void> {
|
||||
async addLink(
|
||||
from: string,
|
||||
to: string,
|
||||
context?: string,
|
||||
linkType?: string,
|
||||
linkSource?: string,
|
||||
originSlug?: string,
|
||||
originField?: string,
|
||||
): Promise<void> {
|
||||
const src = linkSource ?? 'markdown';
|
||||
await this.db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, $3, $4
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, $3, $4, $5,
|
||||
(SELECT id FROM pages WHERE slug = $6),
|
||||
$7
|
||||
FROM pages f, pages t
|
||||
WHERE f.slug = $1 AND t.slug = $2
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO UPDATE SET
|
||||
context = EXCLUDED.context`,
|
||||
[from, to, linkType || '', context || '']
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO UPDATE SET
|
||||
context = EXCLUDED.context,
|
||||
origin_field = EXCLUDED.origin_field`,
|
||||
[from, to, linkType || '', context || '', src, originSlug ?? null, originField ?? null]
|
||||
);
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// Same shape as PostgresEngine. Avoids the 65535-parameter cap entirely.
|
||||
// unnest() pattern: 7 array-typed bound parameters regardless of batch size.
|
||||
// Same shape as PostgresEngine (v0.13). Avoids the 65535-parameter cap.
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const linkSources = links.map(l => l.link_source || 'markdown');
|
||||
const originSlugs = links.map(l => l.origin_slug || null);
|
||||
const originFields = links.map(l => l.origin_field || null);
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[], $6::text[], $7::text[])
|
||||
AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
LEFT JOIN pages o ON o.slug = v.origin_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[fromSlugs, toSlugs, linkTypes, contexts]
|
||||
[fromSlugs, toSlugs, linkTypes, contexts, linkSources, originSlugs, originFields]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
if (linkType !== undefined) {
|
||||
async removeLink(from: string, to: string, linkType?: string, linkSource?: string): Promise<void> {
|
||||
if (linkType !== undefined && linkSource !== undefined) {
|
||||
await this.db.query(
|
||||
`DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = $1)
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = $2)
|
||||
AND link_type = $3
|
||||
AND link_source IS NOT DISTINCT FROM $4`,
|
||||
[from, to, linkType, linkSource]
|
||||
);
|
||||
} else if (linkType !== undefined) {
|
||||
await this.db.query(
|
||||
`DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = $1)
|
||||
@@ -365,6 +389,14 @@ export class PGLiteEngine implements BrainEngine {
|
||||
AND link_type = $3`,
|
||||
[from, to, linkType]
|
||||
);
|
||||
} else if (linkSource !== undefined) {
|
||||
await this.db.query(
|
||||
`DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = $1)
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = $2)
|
||||
AND link_source IS NOT DISTINCT FROM $3`,
|
||||
[from, to, linkSource]
|
||||
);
|
||||
} else {
|
||||
await this.db.query(
|
||||
`DELETE FROM links
|
||||
@@ -377,10 +409,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
|
||||
async getLinks(slug: string): Promise<Link[]> {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = $1`,
|
||||
[slug]
|
||||
);
|
||||
@@ -389,16 +424,44 @@ export class PGLiteEngine implements BrainEngine {
|
||||
|
||||
async getBacklinks(slug: string): Promise<Link[]> {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = $1`,
|
||||
[slug]
|
||||
);
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
|
||||
async findByTitleFuzzy(
|
||||
name: string,
|
||||
dirPrefix?: string,
|
||||
minSimilarity: number = 0.55,
|
||||
): Promise<{ slug: string; similarity: number } | null> {
|
||||
// Inline threshold comparison instead of `SET LOCAL pg_trgm.similarity_threshold`.
|
||||
// The GUC only scopes to the current transaction and pglite auto-commits each
|
||||
// .query() call, so the SET LOCAL would be a no-op. Using similarity() >= $N
|
||||
// directly gives predictable behavior. Tie-breaker: sort by slug so re-runs
|
||||
// pick the same winner.
|
||||
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT slug, similarity(title, $1) AS sim
|
||||
FROM pages
|
||||
WHERE similarity(title, $1) >= $3
|
||||
AND slug LIKE $2
|
||||
ORDER BY sim DESC, slug ASC
|
||||
LIMIT 1`,
|
||||
[name, prefixPattern, minSimilarity]
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
const row = rows[0] as { slug: string; sim: number };
|
||||
return { slug: row.slug, similarity: row.sim };
|
||||
}
|
||||
|
||||
async traverseGraph(slug: string, depth: number = 5): Promise<GraphNode[]> {
|
||||
// Cycle prevention: visited array tracks page IDs already in the path.
|
||||
// Prevents exponential blowup on cyclic subgraphs (e.g., A->B->A).
|
||||
|
||||
@@ -62,18 +62,25 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (em
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
-- ============================================================
|
||||
-- See src/schema.sql for full design notes on link_source + origin_page_id.
|
||||
CREATE TABLE IF NOT EXISTS links (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT links_from_to_type_unique UNIQUE(from_page_id, to_page_id, link_type)
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT links_from_to_type_source_origin_unique
|
||||
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
|
||||
|
||||
-- ============================================================
|
||||
-- tags
|
||||
|
||||
@@ -355,7 +355,15 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// Links
|
||||
async addLink(from: string, to: string, context?: string, linkType?: string): Promise<void> {
|
||||
async addLink(
|
||||
from: string,
|
||||
to: string,
|
||||
context?: string,
|
||||
linkType?: string,
|
||||
linkSource?: string,
|
||||
originSlug?: string,
|
||||
originField?: string,
|
||||
): Promise<void> {
|
||||
const sql = this.sql;
|
||||
// Pre-check existence so we can throw a clear error (ON CONFLICT DO UPDATE
|
||||
// returns 0 rows when source SELECT is empty, indistinguishable from missing page).
|
||||
@@ -367,48 +375,82 @@ export class PostgresEngine implements BrainEngine {
|
||||
if (exists.length === 0) {
|
||||
throw new Error(`addLink failed: page "${from}" or "${to}" not found`);
|
||||
}
|
||||
// Default link_source to 'markdown' for back-compat with pre-v0.13 callers.
|
||||
// origin_page_id resolves from originSlug via the pages join (NULL if no slug).
|
||||
const src = linkSource ?? 'markdown';
|
||||
await sql`
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, ${linkType || ''}, ${context || ''}
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, ${linkType || ''}, ${context || ''}, ${src},
|
||||
(SELECT id FROM pages WHERE slug = ${originSlug ?? null}),
|
||||
${originField ?? null}
|
||||
FROM pages f, pages t
|
||||
WHERE f.slug = ${from} AND t.slug = ${to}
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO UPDATE SET
|
||||
context = EXCLUDED.context
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO UPDATE SET
|
||||
context = EXCLUDED.context,
|
||||
origin_field = EXCLUDED.origin_field
|
||||
`;
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// unnest() pattern: 7 array-typed bound parameters regardless of batch size.
|
||||
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
|
||||
// identifier-escape gotcha when used inside a (VALUES) subquery.
|
||||
//
|
||||
// v0.13: added link_source, origin_slug, origin_field. Defaults:
|
||||
// link_source → 'markdown' (back-compat with pre-v0.13 callers)
|
||||
// origin_slug → NULL (resolves to origin_page_id IS NULL via LEFT JOIN)
|
||||
// origin_field → NULL
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const linkSources = links.map(l => l.link_source || 'markdown');
|
||||
const originSlugs = links.map(l => l.origin_slug || null);
|
||||
const originFields = links.map(l => l.origin_field || null);
|
||||
const result = await sql`
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest(${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[], ${contexts}::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
||||
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
|
||||
FROM unnest(
|
||||
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
|
||||
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
|
||||
${originFields}::text[]
|
||||
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
LEFT JOIN pages o ON o.slug = v.origin_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
async removeLink(from: string, to: string, linkType?: string, linkSource?: string): Promise<void> {
|
||||
const sql = this.sql;
|
||||
if (linkType !== undefined) {
|
||||
// Build up filters dynamically. linkType + linkSource are independent
|
||||
// optional constraints; all four combinations are valid.
|
||||
if (linkType !== undefined && linkSource !== undefined) {
|
||||
await sql`
|
||||
DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
||||
AND link_type = ${linkType}
|
||||
AND link_source IS NOT DISTINCT FROM ${linkSource}
|
||||
`;
|
||||
} else if (linkType !== undefined) {
|
||||
await sql`
|
||||
DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
||||
AND link_type = ${linkType}
|
||||
`;
|
||||
} else if (linkSource !== undefined) {
|
||||
await sql`
|
||||
DELETE FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
||||
AND link_source IS NOT DISTINCT FROM ${linkSource}
|
||||
`;
|
||||
} else {
|
||||
await sql`
|
||||
@@ -422,10 +464,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
async getLinks(slug: string): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
@@ -434,15 +479,48 @@ export class PostgresEngine implements BrainEngine {
|
||||
async getBacklinks(slug: string): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
|
||||
async findByTitleFuzzy(
|
||||
name: string,
|
||||
dirPrefix?: string,
|
||||
minSimilarity: number = 0.55,
|
||||
): Promise<{ slug: string; similarity: number } | null> {
|
||||
const sql = this.sql;
|
||||
// Use the `similarity()` function directly with an explicit threshold
|
||||
// comparison. DO NOT use `SET LOCAL pg_trgm.similarity_threshold` +
|
||||
// the `%` operator here — postgres.js auto-commits each sql`` call
|
||||
// so `SET LOCAL` is a no-op across statement boundaries. Inline
|
||||
// comparison is the only way to get predictable threshold behavior
|
||||
// without wrapping the caller in a transaction.
|
||||
//
|
||||
// Tie-breaker: sort by slug after similarity so re-runs return the
|
||||
// same winner when multiple pages score equally (prevents churn
|
||||
// in put_page auto-link reconciliation).
|
||||
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
|
||||
const rows = await sql`
|
||||
SELECT slug, similarity(title, ${name}) AS sim
|
||||
FROM pages
|
||||
WHERE similarity(title, ${name}) >= ${minSimilarity}
|
||||
AND slug LIKE ${prefixPattern}
|
||||
ORDER BY sim DESC, slug ASC
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const row = rows[0] as { slug: string; sim: number };
|
||||
return { slug: row.slug, similarity: row.sim };
|
||||
}
|
||||
|
||||
async traverseGraph(slug: string, depth: number = 5): Promise<GraphNode[]> {
|
||||
const sql = this.sql;
|
||||
// Cycle prevention: visited array tracks page IDs already in the path.
|
||||
|
||||
@@ -52,18 +52,38 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (em
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
-- ============================================================
|
||||
-- Provenance model (v0.13):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
|
||||
-- (NULL = legacy row written before v0.13; unknown source)
|
||||
-- origin_page_id — for link_source='frontmatter', the page whose YAML
|
||||
-- frontmatter created this edge; scopes reconciliation
|
||||
-- origin_field — the frontmatter field name (e.g. 'key_people')
|
||||
--
|
||||
-- The unique constraint includes link_source + origin_page_id so a manual edge
|
||||
-- and a frontmatter-derived edge with the same (from, to, type) tuple coexist.
|
||||
-- Reconciliation on put_page filters by (link_source='frontmatter' AND
|
||||
-- origin_page_id = written_page) — never touches other pages' edges.
|
||||
CREATE TABLE IF NOT EXISTS links (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT links_from_to_type_unique UNIQUE(from_page_id, to_page_id, link_type)
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- NULLS NOT DISTINCT (PG15+) so two rows with link_source IS NULL or
|
||||
-- origin_page_id IS NULL collide as expected. Without this, every row with
|
||||
-- NULL origin_page_id (markdown/manual edges) would be treated as unique.
|
||||
CONSTRAINT links_from_to_type_source_origin_unique
|
||||
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
|
||||
|
||||
-- ============================================================
|
||||
-- tags
|
||||
|
||||
@@ -82,6 +82,26 @@ export interface Link {
|
||||
to_slug: string;
|
||||
link_type: string;
|
||||
context: string;
|
||||
/**
|
||||
* Provenance (v0.13+). NULL = legacy row (pre-v0.13, unknown source).
|
||||
* 'markdown' = extracted from `[Name](path)` refs. 'frontmatter' = extracted
|
||||
* from YAML frontmatter fields (company, investors, attendees, etc.).
|
||||
* 'manual' = user-created via addLink with explicit source.
|
||||
* Reconciliation in runAutoLink filters on link_source to avoid touching
|
||||
* markdown / manual edges when rewriting a page's frontmatter.
|
||||
*/
|
||||
link_source?: string | null;
|
||||
/**
|
||||
* For link_source='frontmatter': the slug of the page whose frontmatter
|
||||
* created this edge. Lets reconciliation scope "my edges" precisely when
|
||||
* multiple pages reference the same (from, to, type) tuple.
|
||||
*/
|
||||
origin_slug?: string | null;
|
||||
/**
|
||||
* The frontmatter field name that created this edge (e.g. 'key_people',
|
||||
* 'investors'). Used for debug output and the `unresolved` response list.
|
||||
*/
|
||||
origin_field?: string | null;
|
||||
}
|
||||
|
||||
export interface GraphNode {
|
||||
|
||||
@@ -48,18 +48,38 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (em
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
-- ============================================================
|
||||
-- Provenance model (v0.13):
|
||||
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
|
||||
-- (NULL = legacy row written before v0.13; unknown source)
|
||||
-- origin_page_id — for link_source='frontmatter', the page whose YAML
|
||||
-- frontmatter created this edge; scopes reconciliation
|
||||
-- origin_field — the frontmatter field name (e.g. 'key_people')
|
||||
--
|
||||
-- The unique constraint includes link_source + origin_page_id so a manual edge
|
||||
-- and a frontmatter-derived edge with the same (from, to, type) tuple coexist.
|
||||
-- Reconciliation on put_page filters by (link_source='frontmatter' AND
|
||||
-- origin_page_id = written_page) — never touches other pages' edges.
|
||||
CREATE TABLE IF NOT EXISTS links (
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CONSTRAINT links_from_to_type_unique UNIQUE(from_page_id, to_page_id, link_type)
|
||||
id SERIAL PRIMARY KEY,
|
||||
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
||||
link_type TEXT NOT NULL DEFAULT '',
|
||||
context TEXT NOT NULL DEFAULT '',
|
||||
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
||||
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
||||
origin_field TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- NULLS NOT DISTINCT (PG15+) so two rows with link_source IS NULL or
|
||||
-- origin_page_id IS NULL collide as expected. Without this, every row with
|
||||
-- NULL origin_page_id (markdown/manual edges) would be treated as unique.
|
||||
CONSTRAINT links_from_to_type_source_origin_unique
|
||||
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
|
||||
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
|
||||
|
||||
-- ============================================================
|
||||
-- tags
|
||||
|
||||
Reference in New Issue
Block a user