v0.41.29.0 feat(conversation-parser): bold-name-no-time builtin + fix(orphans): source-scoped orphan_ratio (supersedes #1613) (#1620)
* feat(conversation-parser): add bold-name-no-time builtin (Circleback/Granola/Zoom, no timestamp) The 14th built-in pattern parses `**Speaker:** text` transcripts with NO per-line timestamp — the shape Circleback / Granola / Zoom emit. Every prior builtin required a time anchor, so this shape matched nothing: a production brain had 104 conversation pages + 3,423 eligible pages silently extracting zero facts. Messages anchor at T00:00:00Z of the frontmatter date (no fabricated wall-clock; line order preserves sequence), same convention as irc-classic. Hardening beyond the original community proposal: - regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`: the colon-inside-bold (NOT declaration order) is what prevents shadowing bold-paren-time; the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling telegram-bracket yields an honest no_match instead of speaker="[18:37] Name". - new optional PatternEntry.score_full_body: `**Label:** text` is a common prose idiom, so a notes page with bold labels clustered in its first 10 lines scored 0.3 on the head pass (NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so the full-body fallback never fired) and cleared the 0.05 floor. parse.ts now recomputes the winner's score over the full body before the floor, so such a page drops to its true low density and stays no_match. - scrubbed pre-existing real names from bold-paren-time test_positive samples (privacy rule). Fixtures use placeholder names only. Pinned by new bold-name-no-time + clustered-head no_match cases in parse.test.ts and the eval corpus. Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(orphans): scope orphan_ratio + find_orphans by source; fix total_linkable denominator `gbrain doctor --source <id>` and `gbrain orphans --source <id>` now scope the orphan scan to that source instead of reporting brain-wide. Three fixes: - findOrphanPages(opts?: { sourceId?, sourceIds? }) on both engines scopes the CANDIDATE set (scalar `= $1` or federated `= ANY($1::text[])`). Inbound links from ANY source still count, so a page in source X linked FROM source Y is reachable and NOT an orphan of X (the deliberate, less-surprising definition). - corrected the total_linkable denominator in findOrphans: it now enumerates all live pages (scoped) and subtracts every excluded-by-slug page, not just excluded orphans. The old `total - excludedOrphans` left excluded NON-orphan pages (templates/, scratch/) with inbound links in the denominator, inflating it and suppressing warnings. Changes orphan_ratio output for every brain, in the accurate direction. - the find_orphans MCP op threads sourceScopeOpts(ctx), closing a cross-source read leak where a source-bound OAuth client saw brain-wide orphans (v0.34.1 source-isolation class). doctor uses an explicit `--source` flag parse (NOT resolveSourceWithTier, which would scope bare invocations to a default), and under explicit --source reports the ratio with a low-scale caveat below 100 entity pages instead of a vacuous "ok". Thin-client doctor --source orphan_ratio deferred (TODOS.md). Pinned by test/orphans-source-scope.test.ts (PGLite: scoping, cross-source inbound, denominator, find_orphans op scope) + a Postgres↔PGLite parity case in test/e2e/engine-parity.test.ts (scalar + federated binding). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: v0.41.29.0 — bold-name-no-time + orphan source scoping VERSION + package.json → 0.41.29.0; CHANGELOG entry; CLAUDE.md conversation-parser (13→14 patterns) + orphans source-scoping notes; regenerated llms bundles; TODOS for thin-client doctor --source + check-test-real-names widening. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2857,13 +2857,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
return out;
|
||||
}
|
||||
|
||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
async findOrphanPages(opts?: {
|
||||
sourceId?: string;
|
||||
sourceIds?: string[];
|
||||
}): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
const sql = this.sql;
|
||||
// Soft-delete filter on BOTH sides:
|
||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||
// Without the link-source filter, a live page can hide from orphan results purely
|
||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||
//
|
||||
// v0.41.29.0: scope ONLY the candidate side (`p.source_id`) when opts.sourceId
|
||||
// is set. The inbound-link NOT EXISTS deliberately counts links from ANY source:
|
||||
// a page in source X linked FROM source Y is reachable, so NOT an orphan of X.
|
||||
// Do NOT add `src.source_id = p.source_id` here — that would be the stricter
|
||||
// intra-source-only definition we deliberately reject.
|
||||
const sourceFilter =
|
||||
opts?.sourceIds && opts.sourceIds.length > 0
|
||||
? sql`AND p.source_id = ANY(${opts.sourceIds}::text[])`
|
||||
: opts?.sourceId
|
||||
? sql`AND p.source_id = ${opts.sourceId}`
|
||||
: sql``;
|
||||
const rows = await sql`
|
||||
SELECT
|
||||
p.slug,
|
||||
@@ -2871,6 +2886,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
${sourceFilter}
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM links l
|
||||
|
||||
Reference in New Issue
Block a user