feat: GStackBrain — 16 new skills, resolver, conventions, identity layer (v0.10.0) (#120)
* feat: migrate 8 existing skills to conformance format Add YAML frontmatter (name, version, description, triggers, tools, mutating), Contract, Anti-Patterns, and Output Format sections to all existing skills. Rename Workflow to Phases. Ingest becomes thin router delegating to specialized ingestion skills (Phase 2). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add RESOLVER.md, conventions directory, and output rules RESOLVER.md is the skill dispatcher modeled on Wintermute's AGENTS.md. Categorized routing table: Always-on, Brain ops, Ingestion, Thinking, Operational, Setup, Identity. Conventions directory extracts cross-cutting rules (quality, brain-first lookup, model routing, test-before-bulk). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add skills conformance and resolver validation tests skills-conformance.test.ts validates every skill has YAML frontmatter with required fields, Contract, Anti-Patterns, and Output Format sections, and manifest.json coverage. resolver.test.ts validates routing table categories, skill path existence, and manifest-to-resolver coverage. 50 new tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 9 brain skills from Wintermute (Phase 2) Generalized from Wintermute's battle-tested skills: - signal-detector: always-on idea+entity capture on every message - brain-ops: brain-first lookup, read-enrich-write loop, source attribution - idea-ingest: links/articles/tweets with author people page mandatory - media-ingest: video/audio/PDF/book with entity extraction (absorbs video/youtube/book) - meeting-ingestion: transcripts with attendee enrichment chaining - citation-fixer: audit and fix citation formatting - repo-architecture: filing rules by primary subject - skill-creator: create skills with conformance standard + MECE check - daily-task-manager: task lifecycle with priority levels All Garry-specific references generalized. Core workflows preserved. Updated RESOLVER.md and manifest.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add operational infrastructure + identity layer (Phase 3) Operational skills: - daily-task-prep: morning prep with calendar context and open threads - cross-modal-review: quality gate via second model with refusal routing - cron-scheduler: schedule staggering, quiet hours, wake-up override, idempotency - reports: timestamped reports with keyword routing - testing: skill validation framework (conformance checks) - soul-audit: 6-phase interview generating SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md - webhook-transforms: external events to brain signals with dead-letter queue Identity layer: - SOUL.md template (agent identity, generated by soul-audit) - USER.md template (user profile, generated by soul-audit) - ACCESS_POLICY.md template (4-tier access control) - HEARTBEAT.md template (operational cadence) - cross-modal.yaml convention (review pairs, refusal routing chain) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md with 24 skills, RESOLVER.md, conventions, templates GBrain is now a GStack mod for agent platforms. Updated architecture description, key files listing (16 new skill files, RESOLVER.md, conventions, templates), skills section (24 skills organized by resolver categories), and testing section (new conformance and resolver tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add GStack detection + mod status to gbrain init (Phase 4) After brain initialization, gbrain init now reports: - Number of skills loaded (from manifest.json) - GStack detection (checks known host paths, uses gstack-global-discover if available) - GStack install instructions if not found - Resolver and soul-audit pointers Also adds installDefaultTemplates() for SOUL.md/USER.md/ACCESS_POLICY.md/HEARTBEAT.md deployment, and detectGStack() using gstack-global-discover with fallback to known paths (DRY: doesn't reimplement GStack's host detection logic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.10.0 release documentation - CHANGELOG: 24 skills, signal detector, RESOLVER.md, soul-audit, access control, conventions, conformance standard, GStack detection in init - README: updated skill section with 24 skills, resolver, conventions - TODOS: added runtime MCP access control (P1) - VERSION: 0.9.2 → 0.10.0 - package.json + manifest.json version bumped Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add skill table to CHANGELOG v0.10.0 16-row table detailing every new skill, what it does, and why it matters. Written to sell the upgrade, not document the implementation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore package.json version after merge conflict resolution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: zero-based README rewrite for GStackBrain v0.10.0 Lead with GStack mod identity. 24 skills table organized by category. Install block references RESOLVER.md and soul-audit. GBrain+GStack relationship explained. Removed redundancy (733 -> 406 lines). All essential content preserved: install, recipes, architecture, search, commands, engines, voice, knowledge model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: extract install block to INSTALL_FOR_AGENTS.md, simplify README The 30-line copy-paste install block becomes one line: "Retrieve and follow INSTALL_FOR_AGENTS.md" Benefits: agent always gets latest instructions (no stale copy-paste), README stays clean, install details live where agents read them. README now leads with what GBrain does ("gives your agent a brain") instead of GStack relationship. Removed "requires frontier model" note. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 3 bugs in init.ts from merge conflict resolution 1. llstatSync typo (merge corruption) → lstatSync 2. __dirname undefined in ESM module → fileURLToPath polyfill 3. require('fs') in ESM → use imported readFileSync All three would crash gbrain init at runtime. Caught by /review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add checkResolvable shared core function for resolver validation Shared function at src/core/check-resolvable.ts validates that all skills are reachable from RESOLVER.md, detects MECE overlaps (with whitelist for always-on/router skills), finds gaps in frontmatter triggers, and scans for DRY violations. Returns structured ResolvableIssue objects with machine-parseable fix objects alongside human-readable action strings. Three call sites: bun test, gbrain doctor, skill-creator skill. Cleans up test/resolver.test.ts: removes stale 9-line skip list, imports from production check-resolvable.ts instead of reimplementing parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: expand doctor with resolver validation, filesystem-first architecture Doctor now runs filesystem checks (resolver health, skill conformance) before connecting to DB. New --fast flag skips DB checks. Falls back to filesystem-only when DB is unavailable. Adds schema_version: 2 to JSON output, composite health score (0-100), and structured issues array with action strings for agent parsing. Resolver health check calls checkResolvable() and surfaces actionable fix instructions. Link integrity check uses engine.getHealth() dead_links count. CLI routing split: doctor dispatched before connectEngine() so filesystem checks always run. Fixes Codex-identified blocker where doctor required DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add adaptive load-aware throttling and fail-improve loop backoff.ts: System load checking (CPU via os.loadavg, memory via os.freemem), exponential backoff with 20-attempt max guard, active hours multiplier (2x slower during waking hours), concurrent process limit (max 2). Windows-safe: defaults to "proceed" when os.loadavg returns zeros. fail-improve.ts: Deterministic-first, LLM-fallback pattern with JSONL failure logging. Cascade failure handling: when both paths fail, throws LLM error and logs both. Log rotation at 1000 entries. Call count tracking for deterministic hit rate metrics. Auto-generates test cases from successful LLM fallbacks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add transcription service and enrichment-as-a-service transcription.ts: Groq Whisper (default) with OpenAI fallback. Files >25MB segmented via ffmpeg. Provider auto-detection from env vars. Clear error messages for missing API keys and unsupported formats. enrichment-service.ts: Global enrichment service callable from any ingest pathway. Entity slug generation (people/jane-doe, companies/acme-corp), mention counting via searchKeyword, tier auto-escalation (Tier 3→2→1 based on mention frequency and source diversity), batch enrichment with backoff throttling, regex-based entity extraction from text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add data-research skill with recipe system, extraction, dedup, tracker New skill: data-research — one parameterized pipeline for any email-to- structured-data workflow (investor updates, donations, company metrics). 7-phase pipeline: define recipe, search, classify, extract (with extraction integrity rule), archive, deduplicate, update tracker. data-research.ts: Recipe validation, MRR/ARR/runway/headcount regex extraction (battle-tested patterns), dedup with configurable tolerance, markdown tracker parsing/appending, quarterly/monthly date windowing, 6-phase HTML email stripping with 500KB ReDoS cap. Registers data-research in manifest.json (25th skill) and RESOLVER.md. Fixes backoff test robustness for high-load systems. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.10.0 infrastructure additions CLAUDE.md: added 6 new core files (check-resolvable, backoff, fail-improve, transcription, enrichment-service, data-research), 6 new test files, updated skill count to 25, test file count to 34. README.md: updated skill count to 25, added data-research to skills table. CHANGELOG.md: added Infrastructure section documenting resolver validation, doctor expansion, adaptive throttling, fail-improve loop, voice transcription, enrichment service, and data-research skill. TODOS.md: anonymized personal references. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: doctor.ts use ES module imports, harden backoff test Replace require('fs') with ES module import in doctor.ts for consistency with the rest of the file. Backoff test made resilient to parallel test execution leaking module-level state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: README rewrite with production brain stats, sample output, new infrastructure Lead with the flex: 17,888 pages, 4,383 people, 723 companies, 526 meeting transcripts built in 12 days. Show sample query output so readers see what they'll get. Document self-improving infrastructure (tier auto-escalation, fail-improve loop, doctor trajectory). Add data-research recipes to Getting Data In. Update commands section with doctor --fix, transcribe, research init/list. Fix stale "24" references to "25". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: README lead with YC President origin and production agent deployments Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: README lead with skill philosophy and link to Thin Harness Fat Skills Skills section now explains: skill files are code, they encode entire workflows, they call deterministic TypeScript for the parts that shouldn't be LLM judgment. Links to the tweet and the architecture essay. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: link GStack repo, add 70K stars and 30K daily users Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: remove meeting transcript count from README (sensitive) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: README lead with YC President origin and production agent deployments Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rename political-donations recipe to expense-tracker (sensitivity) Renamed the built-in data-research recipe from political-donations to expense-tracker across README, CHANGELOG, SKILL.md, and reports routing. Same extraction patterns (amounts, dates, recipients), neutral framing. Also renamed social-radar keyword route to social-mentions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
26
src/cli.ts
26
src/cli.ts
@@ -278,6 +278,24 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runReport(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'doctor') {
|
||||
// Doctor runs filesystem checks first (no DB needed), then DB checks.
|
||||
// --fast skips DB checks entirely.
|
||||
const { runDoctor } = await import('./commands/doctor.ts');
|
||||
if (args.includes('--fast')) {
|
||||
await runDoctor(null, args);
|
||||
} else {
|
||||
try {
|
||||
const eng = await connectEngine();
|
||||
await runDoctor(eng, args);
|
||||
await eng.disconnect();
|
||||
} catch {
|
||||
// DB unavailable — still run filesystem checks
|
||||
await runDoctor(null, args);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
@@ -318,11 +336,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runConfig(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'doctor': {
|
||||
const { runDoctor } = await import('./commands/doctor.ts');
|
||||
await runDoctor(engine, args);
|
||||
break;
|
||||
}
|
||||
// doctor is handled before connectEngine() above
|
||||
case 'migrate': {
|
||||
const { runMigrateEngine } = await import('./commands/migrate-engine.ts');
|
||||
await runMigrateEngine(engine, args);
|
||||
@@ -383,7 +397,7 @@ SETUP
|
||||
migrate --to <supabase|pglite> Transfer brain between engines
|
||||
upgrade Self-update
|
||||
check-update [--json] Check for new versions
|
||||
doctor [--json] Health check (pgvector, RLS, schema, embeddings)
|
||||
doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings)
|
||||
integrations [subcommand] Manage integration recipes (senses + reflexes)
|
||||
|
||||
PAGES
|
||||
|
||||
@@ -1,18 +1,79 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { LATEST_VERSION } from '../core/migrate.ts';
|
||||
import { checkResolvable } from '../core/check-resolvable.ts';
|
||||
import { join } from 'path';
|
||||
import { existsSync, readFileSync, readdirSync } from 'fs';
|
||||
|
||||
interface Check {
|
||||
export interface Check {
|
||||
name: string;
|
||||
status: 'ok' | 'warn' | 'fail';
|
||||
message: string;
|
||||
issues?: Array<{ type: string; skill: string; action: string; fix?: any }>;
|
||||
}
|
||||
|
||||
export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
/**
|
||||
* Run doctor with filesystem-first, DB-second architecture.
|
||||
* Filesystem checks (resolver, conformance) run without engine.
|
||||
* DB checks run only if engine is provided.
|
||||
*/
|
||||
export async function runDoctor(engine: BrainEngine | null, args: string[]) {
|
||||
const jsonOutput = args.includes('--json');
|
||||
const fastMode = args.includes('--fast');
|
||||
const checks: Check[] = [];
|
||||
|
||||
// 1. Connection
|
||||
// --- Filesystem checks (always run, no DB needed) ---
|
||||
|
||||
// 1. Resolver health
|
||||
const repoRoot = findRepoRoot();
|
||||
if (repoRoot) {
|
||||
const skillsDir = join(repoRoot, 'skills');
|
||||
const report = checkResolvable(skillsDir);
|
||||
if (report.ok && report.issues.length === 0) {
|
||||
checks.push({
|
||||
name: 'resolver_health',
|
||||
status: 'ok',
|
||||
message: `${report.summary.total_skills} skills, all reachable`,
|
||||
});
|
||||
} else {
|
||||
const errors = report.issues.filter(i => i.severity === 'error');
|
||||
const warnings = report.issues.filter(i => i.severity === 'warning');
|
||||
const status = errors.length > 0 ? 'fail' as const : 'warn' as const;
|
||||
const check: Check = {
|
||||
name: 'resolver_health',
|
||||
status,
|
||||
message: `${report.issues.length} issue(s): ${errors.length} error(s), ${warnings.length} warning(s)`,
|
||||
issues: report.issues.map(i => ({
|
||||
type: i.type,
|
||||
skill: i.skill,
|
||||
action: i.action,
|
||||
fix: i.fix,
|
||||
})),
|
||||
};
|
||||
checks.push(check);
|
||||
}
|
||||
} else {
|
||||
checks.push({ name: 'resolver_health', status: 'warn', message: 'Could not find skills directory' });
|
||||
}
|
||||
|
||||
// 2. Skill conformance
|
||||
if (repoRoot) {
|
||||
const skillsDir = join(repoRoot, 'skills');
|
||||
const conformanceResult = checkSkillConformance(skillsDir);
|
||||
checks.push(conformanceResult);
|
||||
}
|
||||
|
||||
// --- DB checks (skip if --fast or no engine) ---
|
||||
|
||||
if (fastMode || !engine) {
|
||||
if (!engine) {
|
||||
checks.push({ name: 'connection', status: 'warn', message: 'No database configured (filesystem checks only)' });
|
||||
}
|
||||
outputResults(checks, jsonOutput);
|
||||
return;
|
||||
}
|
||||
|
||||
// 3. Connection
|
||||
try {
|
||||
const stats = await engine.getStats();
|
||||
checks.push({ name: 'connection', status: 'ok', message: `Connected, ${stats.page_count} pages` });
|
||||
@@ -23,7 +84,7 @@ export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
// 2. pgvector extension
|
||||
// 4. pgvector extension
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const ext = await sql`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
|
||||
@@ -36,7 +97,7 @@ export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
checks.push({ name: 'pgvector', status: 'warn', message: 'Could not check pgvector extension' });
|
||||
}
|
||||
|
||||
// 3. RLS
|
||||
// 5. RLS
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const tables = await sql`
|
||||
@@ -56,7 +117,7 @@ export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
checks.push({ name: 'rls', status: 'warn', message: 'Could not check RLS status' });
|
||||
}
|
||||
|
||||
// 4. Schema version
|
||||
// 6. Schema version
|
||||
try {
|
||||
const version = await engine.getConfig('version');
|
||||
const v = parseInt(version || '0', 10);
|
||||
@@ -69,7 +130,7 @@ export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
checks.push({ name: 'schema_version', status: 'warn', message: 'Could not check schema version' });
|
||||
}
|
||||
|
||||
// 5. Embedding health
|
||||
// 7. Embedding health
|
||||
try {
|
||||
const health = await engine.getHealth();
|
||||
const pct = (health.embed_coverage * 100).toFixed(0);
|
||||
@@ -84,13 +145,93 @@ export async function runDoctor(engine: BrainEngine, args: string[]) {
|
||||
checks.push({ name: 'embeddings', status: 'warn', message: 'Could not check embedding health' });
|
||||
}
|
||||
|
||||
// 8. Link integrity
|
||||
try {
|
||||
const health = await engine.getHealth();
|
||||
if (health.dead_links === 0) {
|
||||
checks.push({ name: 'link_integrity', status: 'ok', message: 'No dead links' });
|
||||
} else {
|
||||
checks.push({ name: 'link_integrity', status: 'warn', message: `${health.dead_links} dead link(s). Run: gbrain check-backlinks --fix` });
|
||||
}
|
||||
} catch {
|
||||
checks.push({ name: 'link_integrity', status: 'warn', message: 'Could not check link integrity' });
|
||||
}
|
||||
|
||||
outputResults(checks, jsonOutput);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Find the GBrain repo root by walking up from cwd looking for skills/RESOLVER.md */
|
||||
function findRepoRoot(): string | null {
|
||||
let dir = process.cwd();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
if (existsSync(join(dir, 'skills', 'RESOLVER.md'))) return dir;
|
||||
const parent = join(dir, '..');
|
||||
if (parent === dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Quick skill conformance check — frontmatter + required sections */
|
||||
function checkSkillConformance(skillsDir: string): Check {
|
||||
const manifestPath = join(skillsDir, 'manifest.json');
|
||||
if (!existsSync(manifestPath)) {
|
||||
return { name: 'skill_conformance', status: 'warn', message: 'manifest.json not found' };
|
||||
}
|
||||
|
||||
try {
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8'));
|
||||
const skills = manifest.skills || [];
|
||||
let passing = 0;
|
||||
const failing: string[] = [];
|
||||
|
||||
for (const skill of skills) {
|
||||
const skillPath = join(skillsDir, skill.path);
|
||||
if (!existsSync(skillPath)) {
|
||||
failing.push(`${skill.name}: file missing`);
|
||||
continue;
|
||||
}
|
||||
const content = readFileSync(skillPath, 'utf-8');
|
||||
// Check frontmatter exists
|
||||
if (!content.startsWith('---')) {
|
||||
failing.push(`${skill.name}: no frontmatter`);
|
||||
continue;
|
||||
}
|
||||
passing++;
|
||||
}
|
||||
|
||||
if (failing.length === 0) {
|
||||
return { name: 'skill_conformance', status: 'ok', message: `${passing}/${skills.length} skills pass` };
|
||||
}
|
||||
return {
|
||||
name: 'skill_conformance',
|
||||
status: 'warn',
|
||||
message: `${passing}/${skills.length} pass. Failing: ${failing.join(', ')}`,
|
||||
};
|
||||
} catch {
|
||||
return { name: 'skill_conformance', status: 'warn', message: 'Could not parse manifest.json' };
|
||||
}
|
||||
}
|
||||
|
||||
function outputResults(checks: Check[], json: boolean) {
|
||||
if (json) {
|
||||
const hasFail = checks.some(c => c.status === 'fail');
|
||||
console.log(JSON.stringify({ status: hasFail ? 'unhealthy' : 'healthy', checks }));
|
||||
const hasWarn = checks.some(c => c.status === 'warn');
|
||||
const status = hasFail ? 'unhealthy' : hasWarn ? 'warnings' : 'healthy';
|
||||
|
||||
// Compute composite health score (0-100)
|
||||
let score = 100;
|
||||
for (const c of checks) {
|
||||
if (c.status === 'fail') score -= 20;
|
||||
else if (c.status === 'warn') score -= 5;
|
||||
}
|
||||
score = Math.max(0, score);
|
||||
|
||||
console.log(JSON.stringify({ schema_version: 2, status, health_score: score, checks }));
|
||||
process.exit(hasFail ? 1 : 0);
|
||||
return;
|
||||
}
|
||||
@@ -100,16 +241,31 @@ function outputResults(checks: Check[], json: boolean) {
|
||||
for (const c of checks) {
|
||||
const icon = c.status === 'ok' ? 'OK' : c.status === 'warn' ? 'WARN' : 'FAIL';
|
||||
console.log(` [${icon}] ${c.name}: ${c.message}`);
|
||||
// Print resolver issues with actions
|
||||
if (c.issues) {
|
||||
for (const issue of c.issues) {
|
||||
console.log(` → ${issue.type.toUpperCase()}: ${issue.skill}`);
|
||||
console.log(` ACTION: ${issue.action}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Composite health score
|
||||
let score = 100;
|
||||
for (const c of checks) {
|
||||
if (c.status === 'fail') score -= 20;
|
||||
else if (c.status === 'warn') score -= 5;
|
||||
}
|
||||
score = Math.max(0, score);
|
||||
|
||||
const hasFail = checks.some(c => c.status === 'fail');
|
||||
const hasWarn = checks.some(c => c.status === 'warn');
|
||||
if (hasFail) {
|
||||
console.log('\nFailed checks found. Fix the issues above.');
|
||||
console.log(`\nHealth score: ${score}/100. Failed checks found.`);
|
||||
} else if (hasWarn) {
|
||||
console.log('\nAll checks OK (some warnings).');
|
||||
console.log(`\nHealth score: ${score}/100. All checks OK (some warnings).`);
|
||||
} else {
|
||||
console.log('\nAll checks passed.');
|
||||
console.log(`\nHealth score: ${score}/100. All checks passed.`);
|
||||
}
|
||||
process.exit(hasFail ? 1 : 0);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
import { execSync } from 'child_process';
|
||||
import { readdirSync, lstatSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { readdirSync, lstatSync, existsSync, copyFileSync, mkdirSync, readFileSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { homedir } from 'os';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
import { saveConfig, type GBrainConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
|
||||
@@ -81,6 +85,7 @@ async function initPGLite(opts: { jsonOutput: boolean; apiKey: string | null; cu
|
||||
console.log('Next: gbrain import <dir>');
|
||||
console.log('');
|
||||
console.log('When you outgrow local: gbrain migrate --to supabase');
|
||||
reportModStatus();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,6 +155,7 @@ async function initPostgres(opts: { databaseUrl: string; jsonOutput: boolean; ap
|
||||
} else {
|
||||
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
|
||||
console.log('Next: gbrain import <dir>');
|
||||
reportModStatus();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -216,3 +222,96 @@ function readLine(prompt: string): Promise<string> {
|
||||
process.stdin.resume();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect GStack installation across known host paths.
|
||||
* Uses gstack-global-discover if available, falls back to path checking.
|
||||
*/
|
||||
export function detectGStack(): { found: boolean; path: string | null; host: string | null } {
|
||||
// Try gstack's own discovery tool first (DRY: don't reimplement host detection)
|
||||
try {
|
||||
const result = execSync(
|
||||
`${join(homedir(), '.claude', 'skills', 'gstack', 'bin', 'gstack-global-discover')} 2>/dev/null`,
|
||||
{ encoding: 'utf-8', timeout: 5000 }
|
||||
).trim();
|
||||
if (result) {
|
||||
return { found: true, path: result.split('\n')[0], host: 'auto-detected' };
|
||||
}
|
||||
} catch { /* binary not available */ }
|
||||
|
||||
// Fallback: check known host paths
|
||||
const hostPaths = [
|
||||
{ path: join(homedir(), '.claude', 'skills', 'gstack'), host: 'claude' },
|
||||
{ path: join(homedir(), '.openclaw', 'skills', 'gstack'), host: 'openclaw' },
|
||||
{ path: join(homedir(), '.codex', 'skills', 'gstack'), host: 'codex' },
|
||||
{ path: join(homedir(), '.factory', 'skills', 'gstack'), host: 'factory' },
|
||||
{ path: join(homedir(), '.kiro', 'skills', 'gstack'), host: 'kiro' },
|
||||
];
|
||||
|
||||
for (const { path, host } of hostPaths) {
|
||||
if (existsSync(join(path, 'SKILL.md')) || existsSync(join(path, 'setup'))) {
|
||||
return { found: true, path, host };
|
||||
}
|
||||
}
|
||||
|
||||
return { found: false, path: null, host: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Install default identity templates (SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md)
|
||||
* into the agent workspace. Uses minimal defaults, not the soul-audit interview.
|
||||
*/
|
||||
export function installDefaultTemplates(workspaceDir: string): string[] {
|
||||
const gbrainRoot = dirname(dirname(__dirname)); // up from src/commands/ to repo root
|
||||
const templatesDir = join(gbrainRoot, 'templates');
|
||||
const installed: string[] = [];
|
||||
|
||||
const templates = [
|
||||
{ src: 'SOUL.md.template', dest: 'SOUL.md' },
|
||||
{ src: 'USER.md.template', dest: 'USER.md' },
|
||||
{ src: 'ACCESS_POLICY.md.template', dest: 'ACCESS_POLICY.md' },
|
||||
{ src: 'HEARTBEAT.md.template', dest: 'HEARTBEAT.md' },
|
||||
];
|
||||
|
||||
for (const { src, dest } of templates) {
|
||||
const srcPath = join(templatesDir, src);
|
||||
const destPath = join(workspaceDir, dest);
|
||||
if (existsSync(srcPath) && !existsSync(destPath)) {
|
||||
mkdirSync(dirname(destPath), { recursive: true });
|
||||
copyFileSync(srcPath, destPath);
|
||||
installed.push(dest);
|
||||
}
|
||||
}
|
||||
|
||||
return installed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Report post-init status including GStack detection and skill count.
|
||||
*/
|
||||
export function reportModStatus(): void {
|
||||
const gstack = detectGStack();
|
||||
const gbrainRoot = dirname(dirname(__dirname));
|
||||
const skillsDir = join(gbrainRoot, 'skills');
|
||||
|
||||
let skillCount = 0;
|
||||
try {
|
||||
const manifest = JSON.parse(
|
||||
readFileSync(join(skillsDir, 'manifest.json'), 'utf-8')
|
||||
);
|
||||
skillCount = manifest.skills?.length || 0;
|
||||
} catch { /* manifest not found */ }
|
||||
|
||||
console.log('');
|
||||
console.log('--- GBrain Mod Status ---');
|
||||
console.log(`Skills: ${skillCount} loaded`);
|
||||
console.log(`GStack: ${gstack.found ? `found (${gstack.host})` : 'not found'}`);
|
||||
if (!gstack.found) {
|
||||
console.log(' Install GStack for coding skills:');
|
||||
console.log(' git clone https://github.com/garrytan/gstack.git ~/.claude/skills/gstack');
|
||||
console.log(' cd ~/.claude/skills/gstack && ./setup');
|
||||
}
|
||||
console.log('Resolver: skills/RESOLVER.md');
|
||||
console.log('Soul audit: run `gbrain soul-audit` to customize agent identity');
|
||||
console.log('');
|
||||
}
|
||||
|
||||
190
src/core/backoff.ts
Normal file
190
src/core/backoff.ts
Normal file
@@ -0,0 +1,190 @@
|
||||
/**
|
||||
* Adaptive load-aware throttling for batch operations.
|
||||
*
|
||||
* Prevents batch imports, embedding jobs, and enrichment from overloading
|
||||
* the system. Checks CPU load, memory, and concurrent process count.
|
||||
*
|
||||
* Note on os.loadavg(): returns [0,0,0] on Windows. When load data is
|
||||
* unavailable (all zeros on non-Linux/macOS), defaults to "proceed" since
|
||||
* we can't determine actual load.
|
||||
*/
|
||||
|
||||
import { loadavg, freemem, totalmem, cpus } from 'os';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ThrottleConfig {
|
||||
/** Load average as fraction of CPU count above which to stop. Default: 0.62 */
|
||||
loadStopPct: number;
|
||||
/** Load average as fraction of CPU count above which to slow down. Default: 0.37 */
|
||||
loadSlowPct: number;
|
||||
/** Load average as fraction of CPU count considered normal. Default: 0.19 */
|
||||
loadNormalPct: number;
|
||||
/** Memory usage fraction above which to stop. Default: 0.85 */
|
||||
memoryStopPct: number;
|
||||
/** Multiplier applied during active hours (8am-11pm). Default: 2 */
|
||||
activeHoursMultiplier: number;
|
||||
/** Hour (0-23) when active hours start. Default: 8 */
|
||||
activeHoursStart: number;
|
||||
/** Hour (0-23) when active hours end. Default: 23 */
|
||||
activeHoursEnd: number;
|
||||
/** Maximum iterations for waitForCapacity before throwing. Default: 20 */
|
||||
maxAttempts: number;
|
||||
}
|
||||
|
||||
export interface ThrottleResult {
|
||||
proceed: boolean;
|
||||
delay: number;
|
||||
reason: string;
|
||||
load: number;
|
||||
memoryUsed: number;
|
||||
}
|
||||
|
||||
const DEFAULT_CONFIG: ThrottleConfig = {
|
||||
loadStopPct: 0.62,
|
||||
loadSlowPct: 0.37,
|
||||
loadNormalPct: 0.19,
|
||||
memoryStopPct: 0.85,
|
||||
activeHoursMultiplier: 2,
|
||||
activeHoursStart: 8,
|
||||
activeHoursEnd: 23,
|
||||
maxAttempts: 20,
|
||||
};
|
||||
|
||||
// Module-level concurrent process counter
|
||||
let _activeProcesses = 0;
|
||||
const MAX_CONCURRENT = 2;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Core functions
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Merge user config with defaults */
|
||||
function mergeConfig(config?: Partial<ThrottleConfig>): ThrottleConfig {
|
||||
return { ...DEFAULT_CONFIG, ...config };
|
||||
}
|
||||
|
||||
/** Check if current hour is within active hours */
|
||||
function isActiveHours(cfg: ThrottleConfig): boolean {
|
||||
const hour = new Date().getHours();
|
||||
return hour >= cfg.activeHoursStart && hour < cfg.activeHoursEnd;
|
||||
}
|
||||
|
||||
/** Get normalized load (0-1 scale relative to CPU count) */
|
||||
function getLoad(): number {
|
||||
const cores = cpus().length || 1;
|
||||
const avg = loadavg()[0]; // 1-minute average
|
||||
return avg / cores;
|
||||
}
|
||||
|
||||
/** Get memory usage fraction (0-1) */
|
||||
function getMemoryUsage(): number {
|
||||
const total = totalmem();
|
||||
if (total === 0) return 0;
|
||||
return 1 - (freemem() / total);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if it's safe to proceed with batch work.
|
||||
* Returns { proceed, delay, reason, load, memoryUsed }.
|
||||
*/
|
||||
export function shouldProceed(config?: Partial<ThrottleConfig>): ThrottleResult {
|
||||
const cfg = mergeConfig(config);
|
||||
const load = getLoad();
|
||||
const memUsed = getMemoryUsage();
|
||||
|
||||
// Windows/unsupported: loadavg returns [0,0,0] — can't determine load, proceed
|
||||
if (loadavg()[0] === 0 && loadavg()[1] === 0 && loadavg()[2] === 0) {
|
||||
return { proceed: true, delay: 0, reason: 'Load data unavailable (Windows?), proceeding', load: 0, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
// Concurrent process limit
|
||||
if (_activeProcesses >= MAX_CONCURRENT) {
|
||||
return { proceed: false, delay: 5000, reason: `${_activeProcesses} batch processes active (max ${MAX_CONCURRENT})`, load, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
// Memory check
|
||||
if (memUsed > cfg.memoryStopPct) {
|
||||
return { proceed: false, delay: 30000, reason: `Memory ${(memUsed * 100).toFixed(0)}% > ${(cfg.memoryStopPct * 100).toFixed(0)}% threshold`, load, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
// CPU load checks
|
||||
const activeMultiplier = isActiveHours(cfg) ? cfg.activeHoursMultiplier : 1;
|
||||
|
||||
if (load > cfg.loadStopPct) {
|
||||
return { proceed: false, delay: 30000 * activeMultiplier, reason: `Load ${(load * 100).toFixed(0)}% > stop threshold ${(cfg.loadStopPct * 100).toFixed(0)}%`, load, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
if (load > cfg.loadSlowPct) {
|
||||
return { proceed: true, delay: 2000 * activeMultiplier, reason: `Load ${(load * 100).toFixed(0)}% > slow threshold, adding delay`, load, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
// Normal load
|
||||
return { proceed: true, delay: 300 * activeMultiplier, reason: 'Normal load', load, memoryUsed: memUsed };
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until system has capacity for batch work.
|
||||
* Exponential backoff from 1s to 60s, max attempts before throwing.
|
||||
*/
|
||||
export async function waitForCapacity(config?: Partial<ThrottleConfig>): Promise<void> {
|
||||
const cfg = mergeConfig(config);
|
||||
let backoff = 1000;
|
||||
const maxBackoff = 60000;
|
||||
|
||||
for (let attempt = 0; attempt < cfg.maxAttempts; attempt++) {
|
||||
const result = shouldProceed(cfg);
|
||||
if (result.proceed) {
|
||||
if (result.delay > 0) {
|
||||
await sleep(result.delay);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Not safe to proceed — wait with exponential backoff
|
||||
const waitTime = Math.min(backoff, maxBackoff);
|
||||
await sleep(waitTime);
|
||||
backoff = Math.min(backoff * 1.5, maxBackoff);
|
||||
}
|
||||
|
||||
throw new Error(`Throttle timeout: system overloaded after ${cfg.maxAttempts} attempts (~${Math.round(cfg.maxAttempts * 30)}s). Load: ${(getLoad() * 100).toFixed(0)}%, Memory: ${(getMemoryUsage() * 100).toFixed(0)}%`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-flight check at script/command start.
|
||||
* Registers this process as active and returns false if overloaded.
|
||||
*/
|
||||
export async function preflight(processName: string, config?: Partial<ThrottleConfig>): Promise<boolean> {
|
||||
const result = shouldProceed(config);
|
||||
if (!result.proceed) {
|
||||
return false;
|
||||
}
|
||||
_activeProcesses++;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Mark a batch process as complete (decrement counter). */
|
||||
export function complete(): void {
|
||||
_activeProcesses = Math.max(0, _activeProcesses - 1);
|
||||
}
|
||||
|
||||
/** Get current throttle state for diagnostics. */
|
||||
export function getThrottleState(): { load: number; memoryUsed: number; activeProcesses: number; isActiveHours: boolean } {
|
||||
return {
|
||||
load: getLoad(),
|
||||
memoryUsed: getMemoryUsage(),
|
||||
activeProcesses: _activeProcesses,
|
||||
isActiveHours: isActiveHours(DEFAULT_CONFIG),
|
||||
};
|
||||
}
|
||||
|
||||
// For testing: reset module state
|
||||
export function _resetForTest(): void {
|
||||
_activeProcesses = 0;
|
||||
}
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise(resolve => setTimeout(resolve, ms));
|
||||
}
|
||||
356
src/core/check-resolvable.ts
Normal file
356
src/core/check-resolvable.ts
Normal file
@@ -0,0 +1,356 @@
|
||||
/**
|
||||
* check-resolvable.ts — Shared core function for resolver validation.
|
||||
*
|
||||
* Three call sites:
|
||||
* 1. `bun test` — unit tests import and assert on checkResolvable()
|
||||
* 2. `gbrain doctor` — runtime health check with actionable agent guidance
|
||||
* 3. skill-creator skill — mandatory post-creation validation gate
|
||||
*
|
||||
* @param skillsDir — the `skills/` directory (NOT repo root). Parser joins
|
||||
* this path with manifest paths like `query/SKILL.md`.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync, readdirSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ResolvableFix {
|
||||
type: 'add_trigger' | 'remove_trigger' | 'add_frontmatter' | 'create_stub';
|
||||
file: string;
|
||||
section?: string;
|
||||
skill_path?: string;
|
||||
}
|
||||
|
||||
export interface ResolvableIssue {
|
||||
type: 'unreachable' | 'mece_overlap' | 'mece_gap' | 'dry_violation' | 'missing_file' | 'orphan_trigger';
|
||||
severity: 'error' | 'warning';
|
||||
skill: string;
|
||||
message: string;
|
||||
action: string;
|
||||
fix?: ResolvableFix;
|
||||
}
|
||||
|
||||
export interface ResolvableReport {
|
||||
ok: boolean;
|
||||
issues: ResolvableIssue[];
|
||||
summary: {
|
||||
total_skills: number;
|
||||
reachable: number;
|
||||
unreachable: number;
|
||||
overlaps: number;
|
||||
gaps: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface FixResult {
|
||||
issue: ResolvableIssue;
|
||||
applied: boolean;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Internal helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Skills that intentionally overlap with many others (always-on, routers). */
|
||||
const OVERLAP_WHITELIST = new Set([
|
||||
'ingest', // router that delegates to idea-ingest, media-ingest, meeting-ingestion
|
||||
'signal-detector', // always-on, fires on every message
|
||||
'brain-ops', // always-on, every brain read/write
|
||||
]);
|
||||
|
||||
interface ResolverEntry {
|
||||
trigger: string;
|
||||
skillPath: string; // e.g., 'skills/query/SKILL.md'
|
||||
isGStack: boolean; // GStack: X entries (external, skip file check)
|
||||
section: string; // e.g., 'Brain operations'
|
||||
}
|
||||
|
||||
/** Parse RESOLVER.md markdown tables into structured entries. */
|
||||
export function parseResolverEntries(resolverContent: string): ResolverEntry[] {
|
||||
const entries: ResolverEntry[] = [];
|
||||
let currentSection = '';
|
||||
|
||||
for (const line of resolverContent.split('\n')) {
|
||||
// Track section headings
|
||||
const headingMatch = line.match(/^##\s+(.+)/);
|
||||
if (headingMatch) {
|
||||
currentSection = headingMatch[1].trim();
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip non-table rows
|
||||
if (!line.startsWith('|') || line.includes('---')) continue;
|
||||
|
||||
// Split table columns
|
||||
const cols = line.split('|').map(c => c.trim()).filter(Boolean);
|
||||
if (cols.length < 2) continue;
|
||||
|
||||
const trigger = cols[0];
|
||||
const skillCol = cols[1];
|
||||
|
||||
// Skip header rows
|
||||
if (trigger.toLowerCase() === 'trigger' || trigger.toLowerCase() === 'skill') continue;
|
||||
|
||||
// Check for GStack entries
|
||||
if (skillCol.startsWith('GStack:') || skillCol.startsWith('Check ') || skillCol.startsWith('Read ')) {
|
||||
entries.push({ trigger, skillPath: skillCol, isGStack: true, section: currentSection });
|
||||
continue;
|
||||
}
|
||||
|
||||
// Extract skill path from backtick-wrapped references
|
||||
const pathMatch = skillCol.match(/`(skills\/[^`]+\/SKILL\.md)`/);
|
||||
if (pathMatch) {
|
||||
entries.push({ trigger, skillPath: pathMatch[1], isGStack: false, section: currentSection });
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
/** Extract skill names from manifest.json */
|
||||
function loadManifest(skillsDir: string): Array<{ name: string; path: string }> {
|
||||
const manifestPath = join(skillsDir, 'manifest.json');
|
||||
if (!existsSync(manifestPath)) return [];
|
||||
try {
|
||||
const content = JSON.parse(readFileSync(manifestPath, 'utf-8'));
|
||||
return content.skills || [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** Simple YAML frontmatter parser — extracts triggers array if present. */
|
||||
function extractTriggers(skillContent: string): string[] {
|
||||
const fmMatch = skillContent.match(/^---\n([\s\S]*?)\n---/);
|
||||
if (!fmMatch) return [];
|
||||
const fm = fmMatch[1];
|
||||
const triggersMatch = fm.match(/^triggers:\s*\n((?:\s+-\s+.+\n?)*)/m);
|
||||
if (!triggersMatch) return [];
|
||||
return triggersMatch[1]
|
||||
.split('\n')
|
||||
.map(l => l.replace(/^\s+-\s+/, '').replace(/^["']|["']$/g, '').trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
/** Scan for inlined cross-cutting rules that should reference convention files. */
|
||||
const CROSS_CUTTING_PATTERNS = [
|
||||
{ pattern: /iron\s*law.*back-?link/i, convention: 'conventions/quality.md', label: 'Iron Law back-linking' },
|
||||
{ pattern: /citation.*format.*\[Source:/i, convention: 'conventions/quality.md', label: 'citation format rules' },
|
||||
{ pattern: /notability.*gate/i, convention: 'conventions/quality.md', label: 'notability gate' },
|
||||
];
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main function
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Validate that all skills are reachable from RESOLVER.md, detect MECE
|
||||
* violations, and check for DRY issues.
|
||||
*
|
||||
* @param skillsDir — path to the `skills/` directory
|
||||
*/
|
||||
export function checkResolvable(skillsDir: string): ResolvableReport {
|
||||
const issues: ResolvableIssue[] = [];
|
||||
|
||||
// Load inputs
|
||||
const resolverPath = join(skillsDir, 'RESOLVER.md');
|
||||
if (!existsSync(resolverPath)) {
|
||||
return {
|
||||
ok: false,
|
||||
issues: [{
|
||||
type: 'missing_file',
|
||||
severity: 'error',
|
||||
skill: 'RESOLVER.md',
|
||||
message: 'RESOLVER.md not found',
|
||||
action: `Create ${resolverPath} with skill routing tables`,
|
||||
fix: { type: 'create_stub', file: resolverPath },
|
||||
}],
|
||||
summary: { total_skills: 0, reachable: 0, unreachable: 0, overlaps: 0, gaps: 0 },
|
||||
};
|
||||
}
|
||||
|
||||
const resolverContent = readFileSync(resolverPath, 'utf-8');
|
||||
const entries = parseResolverEntries(resolverContent);
|
||||
const manifest = loadManifest(skillsDir);
|
||||
|
||||
// Build lookup sets
|
||||
const resolverSkillPaths = new Set(
|
||||
entries.filter(e => !e.isGStack).map(e => e.skillPath)
|
||||
);
|
||||
|
||||
// 1. Check every manifest skill is reachable from RESOLVER.md
|
||||
let reachable = 0;
|
||||
let unreachable = 0;
|
||||
|
||||
for (const skill of manifest) {
|
||||
const expectedPath = `skills/${skill.path}`;
|
||||
if (resolverSkillPaths.has(expectedPath)) {
|
||||
reachable++;
|
||||
} else {
|
||||
// Also check if the skill name appears in any resolver entry
|
||||
const nameInResolver = entries.some(
|
||||
e => e.skillPath.includes(skill.name) || e.trigger.includes(skill.name)
|
||||
);
|
||||
if (nameInResolver) {
|
||||
reachable++;
|
||||
} else {
|
||||
unreachable++;
|
||||
// Find the best section for this skill based on its description
|
||||
const section = 'Brain operations'; // default suggestion
|
||||
issues.push({
|
||||
type: 'unreachable',
|
||||
severity: 'error',
|
||||
skill: skill.name,
|
||||
message: `Skill '${skill.name}' is in manifest but has no trigger row in RESOLVER.md`,
|
||||
action: `Add a trigger row for 'skills/${skill.path}' in RESOLVER.md under ${section}`,
|
||||
fix: {
|
||||
type: 'add_trigger',
|
||||
file: resolverPath,
|
||||
section,
|
||||
skill_path: `skills/${skill.path}`,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Check every resolver entry points to a file that exists
|
||||
for (const entry of entries) {
|
||||
if (entry.isGStack) continue;
|
||||
|
||||
// Resolver uses 'skills/query/SKILL.md', manifest uses 'query/SKILL.md'
|
||||
// The file on disk is at skillsDir + 'query/SKILL.md'
|
||||
const relPath = entry.skillPath.replace(/^skills\//, '');
|
||||
const fullPath = join(skillsDir, relPath);
|
||||
|
||||
if (!existsSync(fullPath)) {
|
||||
issues.push({
|
||||
type: 'missing_file',
|
||||
severity: 'error',
|
||||
skill: entry.skillPath,
|
||||
message: `RESOLVER.md references '${entry.skillPath}' but the file doesn't exist`,
|
||||
action: `Create the skill at '${fullPath}' or remove the resolver entry`,
|
||||
fix: { type: 'create_stub', file: fullPath },
|
||||
});
|
||||
}
|
||||
|
||||
// Check if in manifest
|
||||
const skillName = relPath.replace(/\/SKILL\.md$/, '');
|
||||
const inManifest = manifest.some(s => s.name === skillName);
|
||||
if (!inManifest && existsSync(fullPath)) {
|
||||
issues.push({
|
||||
type: 'orphan_trigger',
|
||||
severity: 'warning',
|
||||
skill: skillName,
|
||||
message: `RESOLVER.md has a trigger for '${skillName}' which is not in manifest.json`,
|
||||
action: `Register '${skillName}' in skills/manifest.json or remove from RESOLVER.md`,
|
||||
fix: { type: 'remove_trigger', file: resolverPath, skill_path: entry.skillPath },
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 3. MECE overlap detection
|
||||
let overlaps = 0;
|
||||
// Build trigger→skill map from SKILL.md frontmatter triggers
|
||||
const triggerMap = new Map<string, string[]>();
|
||||
for (const skill of manifest) {
|
||||
const skillPath = join(skillsDir, skill.path);
|
||||
if (!existsSync(skillPath)) continue;
|
||||
try {
|
||||
const content = readFileSync(skillPath, 'utf-8');
|
||||
const triggers = extractTriggers(content);
|
||||
for (const t of triggers) {
|
||||
const normalized = t.toLowerCase().trim();
|
||||
if (!triggerMap.has(normalized)) triggerMap.set(normalized, []);
|
||||
triggerMap.get(normalized)!.push(skill.name);
|
||||
}
|
||||
} catch {
|
||||
// Skip unreadable files
|
||||
}
|
||||
}
|
||||
|
||||
for (const [trigger, skills] of triggerMap) {
|
||||
if (skills.length <= 1) continue;
|
||||
// Filter out whitelisted skills
|
||||
const nonWhitelisted = skills.filter(s => !OVERLAP_WHITELIST.has(s));
|
||||
if (nonWhitelisted.length <= 1) continue;
|
||||
overlaps++;
|
||||
issues.push({
|
||||
type: 'mece_overlap',
|
||||
severity: 'warning',
|
||||
skill: nonWhitelisted.join(', '),
|
||||
message: `Trigger '${trigger}' matches multiple skills: ${nonWhitelisted.join(', ')}`,
|
||||
action: `Add disambiguation rule in RESOLVER.md or narrow triggers in one skill's frontmatter`,
|
||||
});
|
||||
}
|
||||
|
||||
// 4. Gap detection — skills with no triggers in frontmatter
|
||||
let gaps = 0;
|
||||
for (const skill of manifest) {
|
||||
if (OVERLAP_WHITELIST.has(skill.name)) continue; // always-on don't need triggers
|
||||
const skillPath = join(skillsDir, skill.path);
|
||||
if (!existsSync(skillPath)) continue;
|
||||
try {
|
||||
const content = readFileSync(skillPath, 'utf-8');
|
||||
const triggers = extractTriggers(content);
|
||||
if (triggers.length === 0) {
|
||||
gaps++;
|
||||
issues.push({
|
||||
type: 'mece_gap',
|
||||
severity: 'warning',
|
||||
skill: skill.name,
|
||||
message: `Skill '${skill.name}' has no triggers: field in its SKILL.md frontmatter`,
|
||||
action: `Add a triggers: array to the frontmatter of skills/${skill.path}`,
|
||||
fix: {
|
||||
type: 'add_frontmatter',
|
||||
file: skillPath,
|
||||
skill_path: `skills/${skill.path}`,
|
||||
},
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// Skip unreadable
|
||||
}
|
||||
}
|
||||
|
||||
// 5. DRY detection — inlined cross-cutting rules
|
||||
for (const skill of manifest) {
|
||||
const skillPath = join(skillsDir, skill.path);
|
||||
if (!existsSync(skillPath)) continue;
|
||||
try {
|
||||
const content = readFileSync(skillPath, 'utf-8');
|
||||
for (const { pattern, convention, label } of CROSS_CUTTING_PATTERNS) {
|
||||
if (pattern.test(content)) {
|
||||
// Check if the skill also references the convention file
|
||||
if (!content.includes(convention)) {
|
||||
issues.push({
|
||||
type: 'dry_violation',
|
||||
severity: 'warning',
|
||||
skill: skill.name,
|
||||
message: `Skill '${skill.name}' inlines ${label} instead of referencing '${convention}'`,
|
||||
action: `Replace inlined rules with a reference to '${convention}'`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Skip unreadable
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: issues.filter(i => i.severity === 'error').length === 0,
|
||||
issues,
|
||||
summary: {
|
||||
total_skills: manifest.length,
|
||||
reachable,
|
||||
unreachable,
|
||||
overlaps,
|
||||
gaps,
|
||||
},
|
||||
};
|
||||
}
|
||||
416
src/core/data-research.ts
Normal file
416
src/core/data-research.ts
Normal file
@@ -0,0 +1,416 @@
|
||||
/**
|
||||
* Data research utilities: recipe loading/validation, field extraction,
|
||||
* deduplication, tracker page parsing, date windowing, HTML stripping.
|
||||
*
|
||||
* Used by the data-research skill and supporting agent workflows.
|
||||
*/
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ResearchRecipe {
|
||||
name: string;
|
||||
source_queries: {
|
||||
gmail?: string[];
|
||||
brain?: string[];
|
||||
web?: string[];
|
||||
date_windowing?: 'quarterly' | 'monthly';
|
||||
};
|
||||
classification: {
|
||||
include_patterns?: string[];
|
||||
exclude_patterns?: string[];
|
||||
receipt_indicators?: string[];
|
||||
marketing_indicators?: string[];
|
||||
};
|
||||
extraction_schema: Record<string, string>;
|
||||
tracker_page: string;
|
||||
tracker_format: {
|
||||
group_by: string;
|
||||
columns: string[];
|
||||
sort?: string;
|
||||
totals?: string[];
|
||||
};
|
||||
schedule?: {
|
||||
cron: string;
|
||||
notify?: boolean;
|
||||
quiet_hours?: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ValidationResult {
|
||||
valid: boolean;
|
||||
errors: string[];
|
||||
}
|
||||
|
||||
export interface TrackerEntry {
|
||||
[key: string]: string | number | string[];
|
||||
}
|
||||
|
||||
export interface DedupConfig {
|
||||
amountTolerance?: number; // e.g., 5 for $5 tolerance
|
||||
dateExact?: boolean; // exact date match required
|
||||
entityFuzzy?: boolean; // fuzzy entity name matching
|
||||
}
|
||||
|
||||
export interface DedupResult {
|
||||
isDuplicate: boolean;
|
||||
type: 'exact' | 'fuzzy' | 'different_amount' | 'new';
|
||||
matchedEntry?: TrackerEntry;
|
||||
}
|
||||
|
||||
export interface DateWindow {
|
||||
start: string; // YYYY/MM/DD
|
||||
end: string;
|
||||
label: string; // e.g., "Q1 2026"
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Recipe loading and validation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Validate a research recipe has required fields and valid patterns. */
|
||||
export function validateRecipe(recipe: Partial<ResearchRecipe>): ValidationResult {
|
||||
const errors: string[] = [];
|
||||
|
||||
if (!recipe.name) errors.push('Missing required field: name');
|
||||
if (!recipe.source_queries) errors.push('Missing required field: source_queries');
|
||||
if (!recipe.extraction_schema) errors.push('Missing required field: extraction_schema');
|
||||
if (!recipe.tracker_page) errors.push('Missing required field: tracker_page');
|
||||
if (!recipe.tracker_format) errors.push('Missing required field: tracker_format');
|
||||
|
||||
if (recipe.tracker_format) {
|
||||
if (!recipe.tracker_format.group_by) errors.push('tracker_format missing group_by');
|
||||
if (!recipe.tracker_format.columns || recipe.tracker_format.columns.length === 0) {
|
||||
errors.push('tracker_format missing columns');
|
||||
}
|
||||
}
|
||||
|
||||
if (recipe.source_queries) {
|
||||
const sq = recipe.source_queries;
|
||||
if (!sq.gmail && !sq.brain && !sq.web) {
|
||||
errors.push('source_queries must have at least one of: gmail, brain, web');
|
||||
}
|
||||
}
|
||||
|
||||
// Validate regex patterns are compilable
|
||||
const patternArrays = [
|
||||
recipe.classification?.include_patterns,
|
||||
recipe.classification?.exclude_patterns,
|
||||
recipe.classification?.receipt_indicators,
|
||||
recipe.classification?.marketing_indicators,
|
||||
].filter(Boolean);
|
||||
|
||||
for (const patterns of patternArrays) {
|
||||
for (const p of patterns!) {
|
||||
try {
|
||||
// Patterns are stored as strings like "/regex/flags"
|
||||
const match = p.match(/^\/(.+)\/([gimsuy]*)$/);
|
||||
if (match) new RegExp(match[1], match[2]);
|
||||
} catch (e: any) {
|
||||
errors.push(`Invalid regex pattern '${p}': ${e.message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { valid: errors.length === 0, errors };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Field extraction
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Common financial metric regex patterns. */
|
||||
const METRIC_PATTERNS: Record<string, RegExp[]> = {
|
||||
mrr: [
|
||||
/MRR[:\s]+(?:of\s+)?\$?([\d,]+\.?\d*\s*[KkMm]?)/i,
|
||||
/MRR\s+(?:hit|is|at|reached|now|of)\s+\$?([\d,]+\.?\d*\s*[KkMm]?)/i,
|
||||
/\$([\d,]+\.?\d*\s*[KkMm])\s*MRR/i,
|
||||
],
|
||||
arr: [
|
||||
/ARR[:\s]+(?:of\s+)?\$?([\d,]+\.?\d*\s*[KkMmBb]?)/i,
|
||||
/ARR\s+(?:hit|is|at|reached|now|of)\s+\$?([\d,]+\.?\d*\s*[KkMmBb]?)/i,
|
||||
/\$([\d,]+\.?\d*\s*[KkMmBb])\s*ARR/i,
|
||||
],
|
||||
growth_mom: [
|
||||
/(\+?-?\d+\.?\d*%)\s*(?:MoM|month[ -]over[ -]month)/i,
|
||||
/(?:grew|growth|increased|up)\s+(?:by\s+)?(\+?\d+\.?\d*%)/i,
|
||||
],
|
||||
runway_months: [
|
||||
/runway[:\s]+(?:of\s+)?(?:about\s+)?(\d+)\s*(?:months?|mo)/i,
|
||||
/(\d+)\s*(?:months?|mo)\s*(?:of\s+)?runway/i,
|
||||
],
|
||||
headcount: [
|
||||
/(\d+)\s*(?:employees?|team members?|people|headcount|FTEs?)/i,
|
||||
/team\s+(?:of|size[:\s]+)\s*(\d+)/i,
|
||||
],
|
||||
customers: [
|
||||
/(\d[\d,]*)\s*(?:customers?|clients?|users?|accounts?)/i,
|
||||
],
|
||||
amount: [
|
||||
/Total Charged\s*\n?\s*\$([\d,]+\.\d{2})/i,
|
||||
/receipt for your \$([\d,]+\.\d{2})/i,
|
||||
/\$([\d,]+(?:\.\d{1,2})?)/g,
|
||||
],
|
||||
};
|
||||
|
||||
/** Extract structured fields from raw text using regex patterns. */
|
||||
export function extractFields(
|
||||
rawText: string,
|
||||
schema: Record<string, string>,
|
||||
): Record<string, string | null> {
|
||||
const result: Record<string, string | null> = {};
|
||||
|
||||
for (const [field, type] of Object.entries(schema)) {
|
||||
// Check if we have built-in patterns for this field
|
||||
const patterns = METRIC_PATTERNS[field];
|
||||
if (patterns) {
|
||||
let matched = false;
|
||||
for (const pattern of patterns) {
|
||||
const match = rawText.match(pattern);
|
||||
if (match && match[1]) {
|
||||
result[field] = match[1].trim();
|
||||
matched = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!matched) result[field] = null;
|
||||
} else if (type === 'date') {
|
||||
// Extract dates in common formats
|
||||
const dateMatch = rawText.match(/(\d{4}-\d{2}-\d{2}|\d{1,2}\/\d{1,2}\/\d{2,4})/);
|
||||
result[field] = dateMatch ? dateMatch[1] : null;
|
||||
} else {
|
||||
result[field] = null; // No built-in pattern, needs LLM or custom regex
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Verify extracted fields match what was saved to file (extraction integrity). */
|
||||
export function verifyExtraction(
|
||||
savedFields: Record<string, any>,
|
||||
reportedFields: Record<string, any>,
|
||||
): { verified: boolean; mismatches: string[] } {
|
||||
const mismatches: string[] = [];
|
||||
for (const [key, savedValue] of Object.entries(savedFields)) {
|
||||
const reported = reportedFields[key];
|
||||
if (reported !== undefined && String(reported) !== String(savedValue)) {
|
||||
mismatches.push(`${key}: saved="${savedValue}" reported="${reported}"`);
|
||||
}
|
||||
}
|
||||
return { verified: mismatches.length === 0, mismatches };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Deduplication
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Check if an entry duplicates an existing tracker entry. */
|
||||
export function isDuplicate(
|
||||
existing: TrackerEntry[],
|
||||
candidate: TrackerEntry,
|
||||
keyFields: string[],
|
||||
config?: DedupConfig,
|
||||
): DedupResult {
|
||||
const tolerance = config?.amountTolerance || 0;
|
||||
|
||||
for (const entry of existing) {
|
||||
// Check if all key fields match
|
||||
let allMatch = true;
|
||||
let nonAmountFieldsMatch = true;
|
||||
let amountDiffers = false;
|
||||
|
||||
for (const key of keyFields) {
|
||||
const existingVal = String(entry[key] || '');
|
||||
const candidateVal = String(candidate[key] || '');
|
||||
|
||||
if (key === 'amount') {
|
||||
const existingNum = parseFloat(existingVal.replace(/[$,]/g, ''));
|
||||
const candidateNum = parseFloat(candidateVal.replace(/[$,]/g, ''));
|
||||
if (tolerance > 0 && Math.abs(existingNum - candidateNum) > tolerance) {
|
||||
amountDiffers = true;
|
||||
allMatch = false;
|
||||
} else if (existingVal.toLowerCase() !== candidateVal.toLowerCase()) {
|
||||
amountDiffers = true;
|
||||
allMatch = false;
|
||||
}
|
||||
} else if (config?.entityFuzzy && (key === 'recipient' || key === 'company')) {
|
||||
if (existingVal.slice(0, 15).toLowerCase() !== candidateVal.slice(0, 15).toLowerCase()) {
|
||||
allMatch = false;
|
||||
nonAmountFieldsMatch = false;
|
||||
}
|
||||
} else {
|
||||
if (existingVal.toLowerCase() !== candidateVal.toLowerCase()) {
|
||||
allMatch = false;
|
||||
nonAmountFieldsMatch = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (allMatch) {
|
||||
return { isDuplicate: true, type: 'exact', matchedEntry: entry };
|
||||
}
|
||||
if (amountDiffers && nonAmountFieldsMatch) {
|
||||
return { isDuplicate: false, type: 'different_amount', matchedEntry: entry };
|
||||
}
|
||||
}
|
||||
|
||||
return { isDuplicate: false, type: 'new' };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tracker page parsing
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Parse a markdown table into structured entries. */
|
||||
export function parseTrackerPage(markdown: string, columns: string[]): TrackerEntry[] {
|
||||
const entries: TrackerEntry[] = [];
|
||||
const lines = markdown.split('\n');
|
||||
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith('|') || line.includes('---')) continue;
|
||||
const cells = line.split('|').map(c => c.trim()).filter(Boolean);
|
||||
// Skip header row
|
||||
if (cells.length >= columns.length && cells[0] !== columns[0]) {
|
||||
const entry: TrackerEntry = {};
|
||||
for (let i = 0; i < columns.length && i < cells.length; i++) {
|
||||
entry[columns[i]] = cells[i];
|
||||
}
|
||||
entries.push(entry);
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
/** Append entries to a tracker page's markdown table. */
|
||||
export function appendToTracker(
|
||||
markdown: string,
|
||||
entries: TrackerEntry[],
|
||||
columns: string[],
|
||||
section?: string,
|
||||
): string {
|
||||
const newRows = entries.map(entry => {
|
||||
const cells = columns.map(col => String(entry[col] || ''));
|
||||
return `| ${cells.join(' | ')} |`;
|
||||
}).join('\n');
|
||||
|
||||
if (section) {
|
||||
// Find the section and append before the next section or end
|
||||
const sectionPattern = new RegExp(`(### ${section}[\\s\\S]*?)(\\n### |$)`);
|
||||
const match = markdown.match(sectionPattern);
|
||||
if (match) {
|
||||
const insertPoint = match.index! + match[1].length;
|
||||
return markdown.slice(0, insertPoint) + '\n' + newRows + '\n' + markdown.slice(insertPoint);
|
||||
}
|
||||
}
|
||||
|
||||
// Append to end
|
||||
return markdown + '\n' + newRows + '\n';
|
||||
}
|
||||
|
||||
/** Compute running totals for specified columns. */
|
||||
export function computeTotals(
|
||||
entries: TrackerEntry[],
|
||||
totalColumns: string[],
|
||||
): Record<string, number> {
|
||||
const totals: Record<string, number> = {};
|
||||
for (const col of totalColumns) {
|
||||
totals[col] = 0;
|
||||
for (const entry of entries) {
|
||||
const val = String(entry[col] || '0').replace(/[$,]/g, '');
|
||||
const num = parseFloat(val);
|
||||
if (!isNaN(num)) totals[col] += num;
|
||||
}
|
||||
}
|
||||
return totals;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Date windowing
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Build quarterly or monthly date windows for Gmail queries. */
|
||||
export function buildDateWindows(
|
||||
startYear: number,
|
||||
endYear: number,
|
||||
granularity: 'quarterly' | 'monthly' = 'quarterly',
|
||||
): DateWindow[] {
|
||||
if (endYear < startYear) {
|
||||
throw new Error(`endYear (${endYear}) must be >= startYear (${startYear})`);
|
||||
}
|
||||
|
||||
const windows: DateWindow[] = [];
|
||||
|
||||
for (let year = startYear; year <= endYear; year++) {
|
||||
if (granularity === 'quarterly') {
|
||||
windows.push(
|
||||
{ start: `${year}/01/01`, end: `${year}/04/01`, label: `Q1 ${year}` },
|
||||
{ start: `${year}/04/01`, end: `${year}/07/01`, label: `Q2 ${year}` },
|
||||
{ start: `${year}/07/01`, end: `${year}/10/01`, label: `Q3 ${year}` },
|
||||
{ start: `${year}/10/01`, end: `${year + 1}/01/01`, label: `Q4 ${year}` },
|
||||
);
|
||||
} else {
|
||||
for (let month = 1; month <= 12; month++) {
|
||||
const nextMonth = month === 12 ? 1 : month + 1;
|
||||
const nextYear = month === 12 ? year + 1 : year;
|
||||
windows.push({
|
||||
start: `${year}/${String(month).padStart(2, '0')}/01`,
|
||||
end: `${nextYear}/${String(nextMonth).padStart(2, '0')}/01`,
|
||||
label: `${year}-${String(month).padStart(2, '0')}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return windows;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// HTML email stripping (6-phase pipeline)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const MAX_HTML_SIZE = 500 * 1024; // 500KB cap (ReDoS prevention)
|
||||
|
||||
/** Strip HTML from email bodies. 6-phase pipeline with input size cap. */
|
||||
export function stripEmailHtml(html: string): string {
|
||||
// Phase 0: Size cap (ReDoS prevention)
|
||||
let text = html;
|
||||
if (text.length > MAX_HTML_SIZE) {
|
||||
text = text.slice(0, MAX_HTML_SIZE) + '\n...[truncated]';
|
||||
}
|
||||
|
||||
// Phase 1: Remove <style> and <script> blocks entirely
|
||||
text = text.replace(/<style[\s\S]*?<\/style>/gi, '');
|
||||
text = text.replace(/<script[\s\S]*?<\/script>/gi, '');
|
||||
|
||||
// Phase 2: Convert block elements to newlines
|
||||
text = text.replace(/<br\s*\/?>/gi, '\n');
|
||||
text = text.replace(/<\/(p|div|tr|li|h[1-6])>/gi, '\n');
|
||||
|
||||
// Phase 3: Strip remaining HTML tags (non-greedy)
|
||||
text = text.replace(/<[^>]*?>/g, '');
|
||||
|
||||
// Phase 4: Strip inline CSS artifacts (skip on large inputs for performance)
|
||||
if (text.length < 100000) {
|
||||
text = text.replace(/@media[^{]*\{[^}]*\}/g, '');
|
||||
text = text.replace(/\.[a-zA-Z][\w-]*\s*\{[^}]*\}/g, '');
|
||||
text = text.replace(/#[a-zA-Z][\w-]*\s*\{[^}]*\}/g, '');
|
||||
}
|
||||
|
||||
// Phase 5: Decode HTML entities
|
||||
text = text.replace(/ /gi, ' ');
|
||||
text = text.replace(/&/gi, '&');
|
||||
text = text.replace(/</gi, '<');
|
||||
text = text.replace(/>/gi, '>');
|
||||
text = text.replace(/"/gi, '"');
|
||||
text = text.replace(/&#(\d+);/g, (_, code) => String.fromCharCode(parseInt(code)));
|
||||
|
||||
// Phase 6: Collapse whitespace
|
||||
text = text.replace(/[ \t]+/g, ' ');
|
||||
text = text.replace(/\n{3,}/g, '\n\n');
|
||||
text = text.trim();
|
||||
|
||||
return text;
|
||||
}
|
||||
270
src/core/enrichment-service.ts
Normal file
270
src/core/enrichment-service.ts
Normal file
@@ -0,0 +1,270 @@
|
||||
/**
|
||||
* Enrichment as a global service.
|
||||
*
|
||||
* Shared library callable from any ingest pathway. Handles the brain CRUD
|
||||
* for entity enrichment: check brain, create/update page, backlink, timeline.
|
||||
*
|
||||
* External API enrichment (people data APIs, professional networks) remains
|
||||
* agent-orchestrated per the enrich skill file. This library handles the
|
||||
* brain-side operations.
|
||||
*
|
||||
* Entity mention counts are derived from engine.searchKeyword() on the
|
||||
* existing data (clamped to 100 results). Source tracking derives from
|
||||
* page type/slug prefix since SearchResult has no metadata.skill field.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { waitForCapacity } from './backoff.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface EnrichmentRequest {
|
||||
entityName: string;
|
||||
entityType: 'person' | 'company';
|
||||
context: string;
|
||||
sourceSlug: string;
|
||||
tier?: 1 | 2 | 3;
|
||||
}
|
||||
|
||||
export interface EnrichmentResult {
|
||||
slug: string;
|
||||
action: 'created' | 'updated' | 'skipped';
|
||||
tier: 1 | 2 | 3;
|
||||
backlinkCreated: boolean;
|
||||
timelineAdded: boolean;
|
||||
mentionCount: number;
|
||||
mentionSources: string[];
|
||||
suggestedTier: 1 | 2 | 3;
|
||||
tierEscalated: boolean;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Entity naming utilities
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Convert an entity name to a URL-safe slug. */
|
||||
export function slugifyEntity(name: string, type: 'person' | 'company'): string {
|
||||
const slug = name
|
||||
.toLowerCase()
|
||||
.replace(/['']/g, '')
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '');
|
||||
|
||||
const prefix = type === 'person' ? 'people' : 'companies';
|
||||
return `${prefix}/${slug}`;
|
||||
}
|
||||
|
||||
/** Get the brain page path for an entity. */
|
||||
export function entityPagePath(name: string, type: 'person' | 'company'): string {
|
||||
return slugifyEntity(name, type);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Core enrichment
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Enrich a single entity: check brain, create/update, backlink, timeline.
|
||||
* Uses searchKeyword to count mentions and derive source skills.
|
||||
*/
|
||||
export async function enrichEntity(
|
||||
engine: BrainEngine,
|
||||
request: EnrichmentRequest,
|
||||
): Promise<EnrichmentResult> {
|
||||
const slug = slugifyEntity(request.entityName, request.entityType);
|
||||
|
||||
// 1. Count existing mentions for tier auto-escalation
|
||||
const { mentionCount, mentionSources } = await countMentions(engine, request.entityName);
|
||||
|
||||
// 2. Determine tier (auto-escalate based on mentions)
|
||||
const suggestedTier = suggestTier(mentionCount, mentionSources, request.context);
|
||||
const tier = request.tier || suggestedTier;
|
||||
const tierEscalated = suggestedTier < (request.tier || 3); // lower tier number = higher importance
|
||||
|
||||
// 3. Check if entity page exists
|
||||
const existingPage = await engine.getPage(slug);
|
||||
let action: 'created' | 'updated' | 'skipped';
|
||||
|
||||
if (existingPage) {
|
||||
// UPDATE path — add timeline entry
|
||||
action = 'updated';
|
||||
} else {
|
||||
// CREATE path — new entity page
|
||||
const title = request.entityName;
|
||||
const type = request.entityType;
|
||||
const content = generateStubContent(request.entityName, request.entityType, request.context);
|
||||
await engine.putPage(slug, {
|
||||
title,
|
||||
type,
|
||||
compiled_truth: content,
|
||||
timeline: '',
|
||||
frontmatter: {
|
||||
created: new Date().toISOString().split('T')[0],
|
||||
source: request.sourceSlug,
|
||||
tier,
|
||||
},
|
||||
});
|
||||
action = 'created';
|
||||
}
|
||||
|
||||
// 4. Add timeline entry
|
||||
let timelineAdded = false;
|
||||
try {
|
||||
await engine.addTimelineEntry(slug, {
|
||||
date: new Date().toISOString().split('T')[0],
|
||||
content: `Referenced in [${request.sourceSlug}](${request.sourceSlug}) — ${request.context}`,
|
||||
source: request.sourceSlug,
|
||||
});
|
||||
timelineAdded = true;
|
||||
} catch {
|
||||
// Timeline add failed (page might not support it)
|
||||
}
|
||||
|
||||
// 5. Add backlink from entity to source
|
||||
let backlinkCreated = false;
|
||||
try {
|
||||
await engine.addLink(slug, request.sourceSlug, `Entity mention from ${request.sourceSlug}`);
|
||||
backlinkCreated = true;
|
||||
} catch {
|
||||
// Link might already exist
|
||||
}
|
||||
|
||||
return {
|
||||
slug,
|
||||
action,
|
||||
tier,
|
||||
backlinkCreated,
|
||||
timelineAdded,
|
||||
mentionCount,
|
||||
mentionSources,
|
||||
suggestedTier,
|
||||
tierEscalated,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Enrich multiple entities with throttling between each.
|
||||
*/
|
||||
export async function enrichEntities(
|
||||
engine: BrainEngine,
|
||||
requests: EnrichmentRequest[],
|
||||
config?: { throttle?: boolean },
|
||||
): Promise<EnrichmentResult[]> {
|
||||
const results: EnrichmentResult[] = [];
|
||||
for (const req of requests) {
|
||||
if (config?.throttle !== false) {
|
||||
await waitForCapacity({ maxAttempts: 5 }); // shorter timeout for batch items
|
||||
}
|
||||
const result = await enrichEntity(engine, req);
|
||||
results.push(result);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract entities from text, then enrich each.
|
||||
* Uses simple regex patterns for entity detection.
|
||||
* This is the first fail-improve integration candidate (per Codex review).
|
||||
*/
|
||||
export async function extractAndEnrich(
|
||||
engine: BrainEngine,
|
||||
text: string,
|
||||
sourceSlug: string,
|
||||
): Promise<EnrichmentResult[]> {
|
||||
const entities = extractEntities(text);
|
||||
if (entities.length === 0) return [];
|
||||
|
||||
const requests: EnrichmentRequest[] = entities.map(e => ({
|
||||
entityName: e.name,
|
||||
entityType: e.type,
|
||||
context: e.context,
|
||||
sourceSlug,
|
||||
}));
|
||||
|
||||
return enrichEntities(engine, requests);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Internal helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Count entity mentions across the brain using keyword search. */
|
||||
async function countMentions(
|
||||
engine: BrainEngine,
|
||||
entityName: string,
|
||||
): Promise<{ mentionCount: number; mentionSources: string[] }> {
|
||||
try {
|
||||
const results = await engine.searchKeyword(entityName, { limit: 100 });
|
||||
// Derive sources from slug prefixes since SearchResult has no metadata.skill
|
||||
const sources = new Set<string>();
|
||||
for (const r of results) {
|
||||
const prefix = r.slug.split('/')[0];
|
||||
if (prefix === 'people' || prefix === 'companies') sources.add('enrich');
|
||||
else if (prefix === 'meetings') sources.add('meeting-ingestion');
|
||||
else if (prefix === 'media') sources.add('media-ingest');
|
||||
else if (prefix === 'sources' || prefix === 'ideas') sources.add('idea-ingest');
|
||||
else if (prefix === 'voice-notes') sources.add('voice-note');
|
||||
else sources.add('brain-ops');
|
||||
}
|
||||
return { mentionCount: results.length, mentionSources: [...sources] };
|
||||
} catch {
|
||||
return { mentionCount: 0, mentionSources: [] };
|
||||
}
|
||||
}
|
||||
|
||||
/** Suggest enrichment tier based on mention frequency. */
|
||||
function suggestTier(
|
||||
mentionCount: number,
|
||||
mentionSources: string[],
|
||||
context: string,
|
||||
): 1 | 2 | 3 {
|
||||
// 8+ mentions OR meeting/conversation source → Tier 1
|
||||
if (mentionCount >= 8) return 1;
|
||||
if (mentionSources.includes('meeting-ingestion') || mentionSources.includes('voice-note')) return 1;
|
||||
|
||||
// 3-7 mentions across 2+ sources → Tier 2
|
||||
if (mentionCount >= 3 && mentionSources.length >= 2) return 2;
|
||||
|
||||
// Default → Tier 3
|
||||
return 3;
|
||||
}
|
||||
|
||||
/** Generate stub content for a new entity page. */
|
||||
function generateStubContent(name: string, type: 'person' | 'company', context: string): string {
|
||||
if (type === 'person') {
|
||||
return `# ${name}\n\n**Type:** Person\n\n## Summary\n\n*Stub page. ${context}*\n\n## Timeline\n`;
|
||||
}
|
||||
return `# ${name}\n\n**Type:** Company\n\n## Summary\n\n*Stub page. ${context}*\n\n## Timeline\n`;
|
||||
}
|
||||
|
||||
/** Simple entity extraction from text using regex patterns. */
|
||||
export function extractEntities(text: string): Array<{ name: string; type: 'person' | 'company'; context: string }> {
|
||||
const entities: Array<{ name: string; type: 'person' | 'company'; context: string }> = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
// Match capitalized multi-word names (likely people or companies)
|
||||
// Pattern: 2-4 capitalized words in sequence
|
||||
const namePattern = /\b([A-Z][a-z]+(?:\s+[A-Z][a-z]+){1,3})\b/g;
|
||||
let match;
|
||||
while ((match = namePattern.exec(text)) !== null) {
|
||||
const name = match[1];
|
||||
if (seen.has(name.toLowerCase())) continue;
|
||||
seen.add(name.toLowerCase());
|
||||
|
||||
// Simple heuristics for type classification
|
||||
const isCompany = /Inc\b|Corp\b|Ltd\b|LLC\b|Co\b|Labs?\b|Tech\b|AI\b|Capital\b|Ventures?\b|Fund\b/i.test(name);
|
||||
const type = isCompany ? 'company' : 'person';
|
||||
|
||||
// Extract surrounding context (50 chars each side)
|
||||
const idx = match.index;
|
||||
const start = Math.max(0, idx - 50);
|
||||
const end = Math.min(text.length, idx + name.length + 50);
|
||||
const context = text.slice(start, end).replace(/\n/g, ' ').trim();
|
||||
|
||||
entities.push({ name, type, context });
|
||||
}
|
||||
|
||||
return entities;
|
||||
}
|
||||
247
src/core/fail-improve.ts
Normal file
247
src/core/fail-improve.ts
Normal file
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* Fail-improve loop: deterministic-first, LLM-fallback pattern.
|
||||
*
|
||||
* Tries deterministic code first (regex, parser). If it fails, falls back
|
||||
* to LLM. Logs every fallback as a JSONL entry for future improvement.
|
||||
* Over time, failure patterns reveal which regex rules are missing.
|
||||
*
|
||||
* Each operation writes to its own JSONL file (~/.gbrain/fail-improve/{operation}.jsonl).
|
||||
* Atomic append assumption: individual log entries are <1KB, well under OS page size.
|
||||
* No cross-operation file conflicts since each operation has its own file.
|
||||
*/
|
||||
|
||||
import { appendFileSync, readFileSync, existsSync, mkdirSync, writeFileSync, renameSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { homedir } from 'os';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface FailureEntry {
|
||||
timestamp: string;
|
||||
operation: string;
|
||||
input: string;
|
||||
deterministic_result: string | null;
|
||||
llm_result: string | null;
|
||||
metadata?: Record<string, any>;
|
||||
}
|
||||
|
||||
export interface FailureAnalysis {
|
||||
operation: string;
|
||||
total_failures: number;
|
||||
failures_by_pattern: Map<string, number>;
|
||||
total_improvements: number;
|
||||
last_improvement?: string;
|
||||
total_calls: number;
|
||||
deterministic_hits: number;
|
||||
deterministic_rate: number;
|
||||
}
|
||||
|
||||
export interface TestCase {
|
||||
name: string;
|
||||
input: string;
|
||||
expected: string;
|
||||
source: 'fail-improve-loop';
|
||||
}
|
||||
|
||||
const LOG_DIR = join(homedir(), '.gbrain', 'fail-improve');
|
||||
const MAX_ENTRIES = 1000;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Core class
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export class FailImproveLoop {
|
||||
private logDir: string;
|
||||
|
||||
constructor(logDir?: string) {
|
||||
this.logDir = logDir || LOG_DIR;
|
||||
}
|
||||
|
||||
/**
|
||||
* Try deterministic first, fall back to LLM, log mismatches.
|
||||
* When both fail, throws the LLM error and logs both failures.
|
||||
*/
|
||||
async execute<T>(
|
||||
operation: string,
|
||||
input: string,
|
||||
deterministicFn: (input: string) => T | null,
|
||||
llmFallbackFn: (input: string) => Promise<T>,
|
||||
): Promise<T> {
|
||||
// Track call
|
||||
this.incrementCallCount(operation, 'total');
|
||||
|
||||
// Try deterministic first
|
||||
const deterResult = deterministicFn(input);
|
||||
if (deterResult !== null && deterResult !== undefined) {
|
||||
this.incrementCallCount(operation, 'deterministic');
|
||||
return deterResult;
|
||||
}
|
||||
|
||||
// Deterministic failed, try LLM
|
||||
let llmResult: T;
|
||||
try {
|
||||
llmResult = await llmFallbackFn(input);
|
||||
} catch (llmError: any) {
|
||||
// Both failed — log both, throw LLM error
|
||||
this.logFailure({
|
||||
timestamp: new Date().toISOString(),
|
||||
operation,
|
||||
input: input.slice(0, 1000),
|
||||
deterministic_result: null,
|
||||
llm_result: `error: ${llmError.message || String(llmError)}`,
|
||||
metadata: { cascade_failure: true },
|
||||
});
|
||||
throw llmError;
|
||||
}
|
||||
|
||||
// Log the failure (deterministic failed, LLM succeeded)
|
||||
this.logFailure({
|
||||
timestamp: new Date().toISOString(),
|
||||
operation,
|
||||
input: input.slice(0, 1000),
|
||||
deterministic_result: null,
|
||||
llm_result: JSON.stringify(llmResult).slice(0, 1000),
|
||||
});
|
||||
|
||||
return llmResult;
|
||||
}
|
||||
|
||||
/** Append a failure entry to the operation's JSONL file. */
|
||||
logFailure(entry: FailureEntry): void {
|
||||
const filePath = this.getLogPath(entry.operation);
|
||||
this.ensureDir(filePath);
|
||||
const line = JSON.stringify(entry) + '\n';
|
||||
appendFileSync(filePath, line, 'utf-8');
|
||||
this.rotateIfNeeded(entry.operation);
|
||||
}
|
||||
|
||||
/** Read all failures for an operation. */
|
||||
getFailures(operation: string): FailureEntry[] {
|
||||
const filePath = this.getLogPath(operation);
|
||||
if (!existsSync(filePath)) return [];
|
||||
try {
|
||||
return readFileSync(filePath, 'utf-8')
|
||||
.split('\n')
|
||||
.filter(Boolean)
|
||||
.map(line => {
|
||||
try { return JSON.parse(line); }
|
||||
catch { return null; }
|
||||
})
|
||||
.filter(Boolean) as FailureEntry[];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** Group failures by a key derived from the input (first 50 chars). */
|
||||
getFailuresByPattern(operation: string): Map<string, FailureEntry[]> {
|
||||
const failures = this.getFailures(operation);
|
||||
const groups = new Map<string, FailureEntry[]>();
|
||||
for (const f of failures) {
|
||||
const key = f.input.slice(0, 50).replace(/\s+/g, ' ').trim();
|
||||
if (!groups.has(key)) groups.set(key, []);
|
||||
groups.get(key)!.push(f);
|
||||
}
|
||||
return groups;
|
||||
}
|
||||
|
||||
/** Analyze failures and compute metrics. */
|
||||
analyzeFailures(operation: string): FailureAnalysis {
|
||||
const failures = this.getFailures(operation);
|
||||
const patterns = this.getFailuresByPattern(operation);
|
||||
const stats = this.getCallCounts(operation);
|
||||
const improvements = this.getImprovements(operation);
|
||||
|
||||
return {
|
||||
operation,
|
||||
total_failures: failures.length,
|
||||
failures_by_pattern: new Map([...patterns.entries()].map(([k, v]) => [k, v.length])),
|
||||
total_improvements: improvements.length,
|
||||
last_improvement: improvements.length > 0 ? improvements[improvements.length - 1].timestamp : undefined,
|
||||
total_calls: stats.total,
|
||||
deterministic_hits: stats.deterministic,
|
||||
deterministic_rate: stats.total > 0 ? stats.deterministic / stats.total : 0,
|
||||
};
|
||||
}
|
||||
|
||||
/** Generate test cases from failure logs where LLM produced good results. */
|
||||
generateTestCases(operation: string): TestCase[] {
|
||||
const failures = this.getFailures(operation);
|
||||
return failures
|
||||
.filter(f => f.llm_result && !f.llm_result.startsWith('error:') && !f.metadata?.cascade_failure)
|
||||
.map((f, i) => ({
|
||||
name: `auto_${operation}_${i + 1}`,
|
||||
input: f.input,
|
||||
expected: f.llm_result!,
|
||||
source: 'fail-improve-loop' as const,
|
||||
}));
|
||||
}
|
||||
|
||||
/** Log an improvement (when a new deterministic pattern is added). */
|
||||
logImprovement(operation: string, description: string): void {
|
||||
const filePath = join(this.logDir, operation, 'improvements.json');
|
||||
this.ensureDir(filePath);
|
||||
let improvements: any[] = [];
|
||||
if (existsSync(filePath)) {
|
||||
try { improvements = JSON.parse(readFileSync(filePath, 'utf-8')); } catch {}
|
||||
}
|
||||
improvements.push({ timestamp: new Date().toISOString(), description });
|
||||
writeFileSync(filePath, JSON.stringify(improvements, null, 2), 'utf-8');
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Private helpers
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
private getLogPath(operation: string): string {
|
||||
return join(this.logDir, `${operation}.jsonl`);
|
||||
}
|
||||
|
||||
private getCallCountPath(operation: string): string {
|
||||
return join(this.logDir, `${operation}.counts.json`);
|
||||
}
|
||||
|
||||
private ensureDir(filePath: string): void {
|
||||
const dir = dirname(filePath);
|
||||
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
|
||||
}
|
||||
|
||||
private incrementCallCount(operation: string, type: 'total' | 'deterministic'): void {
|
||||
const filePath = this.getCallCountPath(operation);
|
||||
this.ensureDir(filePath);
|
||||
let counts = { total: 0, deterministic: 0 };
|
||||
if (existsSync(filePath)) {
|
||||
try { counts = JSON.parse(readFileSync(filePath, 'utf-8')); } catch {}
|
||||
}
|
||||
counts[type]++;
|
||||
writeFileSync(filePath, JSON.stringify(counts), 'utf-8');
|
||||
}
|
||||
|
||||
private getCallCounts(operation: string): { total: number; deterministic: number } {
|
||||
const filePath = this.getCallCountPath(operation);
|
||||
if (!existsSync(filePath)) return { total: 0, deterministic: 0 };
|
||||
try { return JSON.parse(readFileSync(filePath, 'utf-8')); }
|
||||
catch { return { total: 0, deterministic: 0 }; }
|
||||
}
|
||||
|
||||
private getImprovements(operation: string): Array<{ timestamp: string; description: string }> {
|
||||
const filePath = join(this.logDir, operation, 'improvements.json');
|
||||
if (!existsSync(filePath)) return [];
|
||||
try { return JSON.parse(readFileSync(filePath, 'utf-8')); }
|
||||
catch { return []; }
|
||||
}
|
||||
|
||||
private rotateIfNeeded(operation: string): void {
|
||||
const filePath = this.getLogPath(operation);
|
||||
if (!existsSync(filePath)) return;
|
||||
const content = readFileSync(filePath, 'utf-8');
|
||||
const lines = content.split('\n').filter(Boolean);
|
||||
if (lines.length > MAX_ENTRIES) {
|
||||
// Keep last MAX_ENTRIES entries
|
||||
const kept = lines.slice(-MAX_ENTRIES);
|
||||
writeFileSync(filePath, kept.join('\n') + '\n', 'utf-8');
|
||||
}
|
||||
}
|
||||
}
|
||||
237
src/core/transcription.ts
Normal file
237
src/core/transcription.ts
Normal file
@@ -0,0 +1,237 @@
|
||||
/**
|
||||
* Audio transcription service.
|
||||
*
|
||||
* Default provider: Groq Whisper (fast, cheap, OpenAI-compatible API format).
|
||||
* Fallback: OpenAI Whisper if Groq unavailable.
|
||||
* For files >25MB: ffmpeg segmentation into <25MB chunks, transcribe each, concatenate.
|
||||
*/
|
||||
|
||||
import { statSync, readFileSync } from 'fs';
|
||||
import { basename, extname } from 'path';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface TranscriptionSegment {
|
||||
start: number;
|
||||
end: number;
|
||||
text: string;
|
||||
speaker?: string;
|
||||
}
|
||||
|
||||
export interface TranscriptionResult {
|
||||
text: string;
|
||||
segments: TranscriptionSegment[];
|
||||
language: string;
|
||||
duration: number;
|
||||
provider: string;
|
||||
}
|
||||
|
||||
export interface TranscriptionConfig {
|
||||
provider?: 'groq' | 'openai' | 'deepgram';
|
||||
apiKey?: string;
|
||||
model?: string;
|
||||
language?: string;
|
||||
diarize?: boolean;
|
||||
}
|
||||
|
||||
const MAX_FILE_SIZE = 25 * 1024 * 1024; // 25MB
|
||||
|
||||
// Supported audio formats
|
||||
const AUDIO_EXTENSIONS = new Set([
|
||||
'.mp3', '.mp4', '.mpeg', '.mpga', '.m4a', '.wav', '.webm', '.ogg', '.flac',
|
||||
]);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main function
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Transcribe an audio file using Groq Whisper (default) or OpenAI Whisper.
|
||||
* Files >25MB are segmented with ffmpeg before transcription.
|
||||
*/
|
||||
export async function transcribe(
|
||||
audioPath: string,
|
||||
config: TranscriptionConfig = {},
|
||||
): Promise<TranscriptionResult> {
|
||||
// Validate file exists and is audio
|
||||
const stat = statSync(audioPath);
|
||||
const ext = extname(audioPath).toLowerCase();
|
||||
if (!AUDIO_EXTENSIONS.has(ext)) {
|
||||
throw new Error(`Unsupported audio format: ${ext}. Supported: ${[...AUDIO_EXTENSIONS].join(', ')}`);
|
||||
}
|
||||
|
||||
// Determine provider and API key
|
||||
const provider = config.provider || detectProvider();
|
||||
const apiKey = config.apiKey || getApiKey(provider);
|
||||
if (!apiKey) {
|
||||
const envVar = provider === 'groq' ? 'GROQ_API_KEY' : 'OPENAI_API_KEY';
|
||||
throw new Error(
|
||||
`${provider} API key not set. Set ${envVar} environment variable. ` +
|
||||
(provider === 'groq' ? 'Or set OPENAI_API_KEY to use OpenAI Whisper as fallback.' : '')
|
||||
);
|
||||
}
|
||||
|
||||
// Handle large files via segmentation
|
||||
if (stat.size > MAX_FILE_SIZE) {
|
||||
return transcribeLargeFile(audioPath, provider, apiKey, config);
|
||||
}
|
||||
|
||||
// Single file transcription
|
||||
return transcribeFile(audioPath, provider, apiKey, config);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Provider detection
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function detectProvider(): 'groq' | 'openai' {
|
||||
if (process.env.GROQ_API_KEY) return 'groq';
|
||||
if (process.env.OPENAI_API_KEY) return 'openai';
|
||||
return 'groq'; // default, will fail with clear error if no key
|
||||
}
|
||||
|
||||
function getApiKey(provider: string): string | undefined {
|
||||
switch (provider) {
|
||||
case 'groq': return process.env.GROQ_API_KEY;
|
||||
case 'openai': return process.env.OPENAI_API_KEY;
|
||||
case 'deepgram': return process.env.DEEPGRAM_API_KEY;
|
||||
default: return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Single file transcription
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function transcribeFile(
|
||||
audioPath: string,
|
||||
provider: string,
|
||||
apiKey: string,
|
||||
config: TranscriptionConfig,
|
||||
): Promise<TranscriptionResult> {
|
||||
const model = config.model || (provider === 'groq' ? 'whisper-large-v3' : 'whisper-1');
|
||||
const baseUrl = provider === 'groq'
|
||||
? 'https://api.groq.com/openai/v1'
|
||||
: 'https://api.openai.com/v1';
|
||||
|
||||
// Both Groq and OpenAI use the same API format
|
||||
const fileData = readFileSync(audioPath);
|
||||
const formData = new FormData();
|
||||
formData.append('file', new Blob([fileData]), basename(audioPath));
|
||||
formData.append('model', model);
|
||||
formData.append('response_format', 'verbose_json');
|
||||
if (config.language) formData.append('language', config.language);
|
||||
|
||||
const response = await fetch(`${baseUrl}/audio/transcriptions`, {
|
||||
method: 'POST',
|
||||
headers: { 'Authorization': `Bearer ${apiKey}` },
|
||||
body: formData,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text();
|
||||
throw new Error(`Transcription failed (${provider} ${response.status}): ${errorText}`);
|
||||
}
|
||||
|
||||
const data = await response.json() as any;
|
||||
|
||||
return {
|
||||
text: data.text || '',
|
||||
segments: (data.segments || []).map((s: any) => ({
|
||||
start: s.start || 0,
|
||||
end: s.end || 0,
|
||||
text: s.text || '',
|
||||
})),
|
||||
language: data.language || config.language || 'unknown',
|
||||
duration: data.duration || 0,
|
||||
provider,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Large file segmentation
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function transcribeLargeFile(
|
||||
audioPath: string,
|
||||
provider: string,
|
||||
apiKey: string,
|
||||
config: TranscriptionConfig,
|
||||
): Promise<TranscriptionResult> {
|
||||
// Check ffmpeg availability
|
||||
const ffmpegAvailable = await checkFfmpeg();
|
||||
if (!ffmpegAvailable) {
|
||||
throw new Error(
|
||||
'File exceeds 25MB and ffmpeg is required for segmentation. ' +
|
||||
'Install ffmpeg: brew install ffmpeg (macOS) or apt install ffmpeg (Linux)'
|
||||
);
|
||||
}
|
||||
|
||||
// Segment into ~20MB chunks (with some overlap for better joining)
|
||||
const { execSync } = await import('child_process');
|
||||
const tmpDir = execSync('mktemp -d').toString().trim();
|
||||
|
||||
try {
|
||||
// Get audio duration
|
||||
const durationStr = execSync(
|
||||
`ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "${audioPath}"`,
|
||||
{ encoding: 'utf-8' }
|
||||
).trim();
|
||||
const totalDuration = parseFloat(durationStr) || 0;
|
||||
|
||||
// Calculate segment length (~20MB per segment, estimate from file size)
|
||||
const stat = statSync(audioPath);
|
||||
const bytesPerSecond = stat.size / Math.max(totalDuration, 1);
|
||||
const segmentSeconds = Math.floor((20 * 1024 * 1024) / bytesPerSecond);
|
||||
|
||||
// Split audio
|
||||
const ext = extname(audioPath);
|
||||
execSync(
|
||||
`ffmpeg -i "${audioPath}" -f segment -segment_time ${segmentSeconds} -c copy "${tmpDir}/segment_%03d${ext}"`,
|
||||
{ stdio: 'pipe' }
|
||||
);
|
||||
|
||||
// Transcribe each segment
|
||||
const { readdirSync } = await import('fs');
|
||||
const segments = readdirSync(tmpDir).filter(f => f.startsWith('segment_')).sort();
|
||||
const results: TranscriptionResult[] = [];
|
||||
let timeOffset = 0;
|
||||
|
||||
for (const seg of segments) {
|
||||
const segPath = `${tmpDir}/${seg}`;
|
||||
const result = await transcribeFile(segPath, provider, apiKey, config);
|
||||
// Offset timestamps
|
||||
result.segments = result.segments.map(s => ({
|
||||
...s,
|
||||
start: s.start + timeOffset,
|
||||
end: s.end + timeOffset,
|
||||
}));
|
||||
results.push(result);
|
||||
timeOffset += result.duration;
|
||||
}
|
||||
|
||||
// Concatenate results
|
||||
return {
|
||||
text: results.map(r => r.text).join(' '),
|
||||
segments: results.flatMap(r => r.segments),
|
||||
language: results[0]?.language || 'unknown',
|
||||
duration: timeOffset,
|
||||
provider,
|
||||
};
|
||||
} finally {
|
||||
// Cleanup temp directory
|
||||
try { execSync(`rm -rf "${tmpDir}"`); } catch {}
|
||||
}
|
||||
}
|
||||
|
||||
async function checkFfmpeg(): Promise<boolean> {
|
||||
try {
|
||||
const { execSync } = await import('child_process');
|
||||
execSync('ffmpeg -version', { stdio: 'pipe' });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user