triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.42.8.0 feat: content-quality gate on sync — quarantine junk + flag boilerplate (#1699) (#1756)

Browse Source 
* feat: content-quality gate on sync — quarantine junk + flag boilerplate (#1699)

Three-tier disposition at the importFromContent narrow waist:
- High-confidence junk (Cloudflare/CAPTCHA interstitial patterns + operator
  literals) -> quarantine (hidden from search, zero chunks) or reject.
- Fuzzy markup-heavy (prose-vs-markup ratio, warn-tier window, code-exempt)
  -> content_flag marker, stays searchable, agent warned.
- Oversize -> existing embed_skip soft-block + content_flag:oversized warning.

Agent-warning channel: SearchResult.content_flag (stamped in hybridSearch +
the keyword-only search op) and a top-level content_flag on get_page.
New quarantine.ts markers, gbrain quarantine CLI (list/clear/scan), doctor
quarantined_pages + flagged_pages checks (engine.executeRaw, works on PGLite),
sources-audit disposition awareness, markup-heavy lint rule, config keys.

Security: gate-owned markers stripped from untrusted (remote MCP) frontmatter
so a write-scoped client can't hide pages or inject the warning channel.
Markers excluded from content_hash so flagged pages don't re-embed every sync.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.8.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document content-quality gate (quarantine + content_flag) for v0.42.8.0

Add CLAUDE.md Key Files + Commands entries for the #1699 content-quality
gate: src/core/quarantine.ts, gbrain quarantine CLI (list/clear/scan),
the agent-warning channel (SearchResult.content_flag + get_page), doctor
quarantined_pages/flagged_pages checks, the markup-heavy lint rule,
sources-audit disposition awareness, and the three new content_sanity
config keys. Regenerate llms-full.txt from CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-01 23:03:09 -07:00
committed by GitHub
parent ca68a551db
commit 0bfe0d0c7e
34 changed files with 1982 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
src/core/postgres-engine.ts
View File

@@ -2979,6 +2979,27 @@ export class PostgresEngine implements BrainEngine {
return result;
}
async getContentFlagsByPageIds(
pageIds: number[],
): Promise<Map<number, { reason: string; detail: string }>> {
const result = new Map<number, { reason: string; detail: string }>();
if (pageIds.length === 0) return result;
const sql = this.sql;
const rows = await sql`
SELECT id,
frontmatter -> 'content_flag' ->> 'reason' AS reason,
frontmatter -> 'content_flag' ->> 'detail' AS detail
FROM pages
WHERE id = ANY(${pageIds}::int[])
AND frontmatter ? 'content_flag'
`;
for (const r of rows as unknown as { id: number; reason: string | null; detail: string | null }[]) {
if (!r.reason) continue;
result.set(Number(r.id), { reason: r.reason, detail: r.detail ?? '' });
}
return result;
}
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
if (slugs.length === 0) return new Map();
const sql = this.sql;