* feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is the canonical primitive every --workers N bulk command in this wave (and future bulk commands) builds on. Atomic-claim invariant enforced by scripts/check-worker-pool-atomicity.sh (wired into bun run verify). BudgetExhausted bypass + AbortSignal composition baked into the helper so budget caps are a structural ceiling under concurrency, not a per-caller convention. The new resolveWorkersWithClamp wrapper composes existing autoConcurrency with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately NOT a modification to shared autoConcurrency (silent today, used by sync + import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per codex #13. 23 + 12 + 9 = 44 hermetic tests pin every contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: structural + dim-check regression suites for v0.41.16.0 wave - test/embed-helper-migration.test.ts (T3): asserts embed.ts's two sliding-pool sites are migrated to runSlidingPool, pre-migration shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone, GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel threads page.slug. Per codex #16/#17 these are invariant assertions, not byte-equality on progress event ORDERING. - test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim covers vector(N) + halfvec(N), halfvec-before-vector regex ordering pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER USING + CREATE INDEX (codex #18, not bare REINDEX), FactsEmbeddingDimMismatchError tagged class shape, assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent- column skip, doctor check + insert-cast wiring assertions. - test/extract-conversation-facts-workers.test.ts (T5): helper exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES), structural wiring (runSlidingPool, resolveWorkersWithClamp, withRefreshingLock, LockUnavailableError, delete-orphans-first before segment loop, preflight before pool, exit 3 when lock_skipped > 0), Minion handler round-trip. - test/extract-workers.test.ts (T7): --workers wiring on all 3 inner fs-walk loops (extractForSlugs, extractLinksFromDir, extractTimelineFromDir) + CLI parse + opts threading through runExtractCore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510) PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0 on master in parallel. Advancing this wave to v0.41.17.0 so both can land cleanly. Pure mechanical version bump: - VERSION + package.json → 0.41.17.0 - CHANGELOG.md header + "To take advantage of v0.41.17.0" block - TODOS.md section header + v0.41.18+ forward references - CLAUDE.md inline version tags - Regenerated llms-full.txt / llms.txt No code changes. The actual workers cathedral feature set is unchanged from the two prior commits in this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): search-image-column probes column dim at runtime CI shard 5 failed on `searchVector column routing (v0.27.1)` with: error: expected 1280 dimensions, not 1536 The test had a hardcoded `fakeText1536` helper that seeded chunks at 1536-d vectors. Master's default embedding model switched from OpenAI text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh PGLite brain on CI now sizes content_chunks.embedding at 1280; the test's 1536-d INSERT trips pgvector's CheckExpectedDim. Fix: probe `content_chunks.embedding` width via `readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in `TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test now passes regardless of which default ships (the model has flipped twice and may flip again). Local dev (1536 from older config) and CI fresh-install (1280 from new default) both pass. Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the column's fixed width on the image side). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): bump PGLite hook timeout for shard-4 deep-process files facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0 master merge brought migration count to 99. When these files run deep in a shard process that has already created ~20 PGLite engines, the WASM cold-start + 95-migration replay legitimately exceeds bun's 5s default hook timeout (observed 5.6s and 7.3s locally when reproducing). Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts but NOT hook timeouts; those default to 5s and must be set per-hook via the optional 2nd arg to beforeAll/afterAll. Reproduced locally by running the first 21 shard-4 files via head -21 /tmp/shard4-list.txt | xargs bun test → 179 pass, 2 fail (both with hook-timeout error) After fix: → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover) Full shard 4 with fix: 955 pass, 0 fail. Full shard 5 with fix: 1261 pass, 0 fail. Also added a defensive diagnostic to the two put_page tests: if facts_backstop is missing in the response payload, throw with the full payload + isError so future failures surface the actual handler error instead of a bare "expected {...} got undefined" assertion. No-op when the test passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
82 lines
3.4 KiB
TypeScript
82 lines
3.4 KiB
TypeScript
/**
|
||
* v0.31 Phase 6 — anti-loop on dream_generated marker.
|
||
*
|
||
* Pins both code paths that must respect the v0.23.2 marker:
|
||
* - extractFactsFromTurn(isDreamGenerated:true) → []
|
||
* - put_page backstop on dream_generated:true frontmatter → skipped:dream_generated
|
||
*/
|
||
|
||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||
import { dispatchToolCall } from '../src/mcp/dispatch.ts';
|
||
import { extractFactsFromTurn } from '../src/core/facts/extract.ts';
|
||
|
||
let engine: PGLiteEngine;
|
||
|
||
// 30s hook timeout — when this file runs deep in a shard process that's
|
||
// already created ~20 PGLite engines, the WASM cold-start + 95 migrations
|
||
// on a fresh DB legitimately exceeds bun's 5s hook default. CI shard 4
|
||
// hit this on v0.41.17.0 (95 migrations × 21 files × 1 bun process).
|
||
beforeAll(async () => {
|
||
engine = new PGLiteEngine();
|
||
await engine.connect({});
|
||
await engine.initSchema();
|
||
}, 30_000);
|
||
|
||
afterAll(async () => {
|
||
await engine.disconnect();
|
||
}, 30_000);
|
||
|
||
describe('anti-loop dream_generated marker', () => {
|
||
test('extractFactsFromTurn skips when isDreamGenerated:true', async () => {
|
||
const r = await extractFactsFromTurn({
|
||
turnText: 'This would normally produce facts about Sam.',
|
||
source: 'test',
|
||
isDreamGenerated: true,
|
||
});
|
||
expect(r).toEqual([]);
|
||
});
|
||
|
||
test('extractFactsFromTurn does NOT skip on isDreamGenerated:false', async () => {
|
||
const r = await extractFactsFromTurn({
|
||
turnText: '',
|
||
source: 'test',
|
||
isDreamGenerated: false,
|
||
});
|
||
// Empty turn returns [] for a different reason (no content). Just
|
||
// confirms the false branch doesn't short-circuit before the empty
|
||
// check.
|
||
expect(r).toEqual([]);
|
||
});
|
||
|
||
test('put_page backstop skips on dream_generated:true', async () => {
|
||
const result = await dispatchToolCall(engine, 'put_page', {
|
||
slug: 'note/anti-loop-dream',
|
||
content: `---\ntype: note\ntitle: Dream\ndream_generated: true\n---\n${'real-looking content. '.repeat(20)}`,
|
||
}, { remote: false, sourceId: 'default' });
|
||
const payload = JSON.parse(result.content[0].text);
|
||
// Diagnostic: if facts_backstop is missing, the handler likely threw
|
||
// and dispatchToolCall wrapped the error as `{error: 'internal_error'}`.
|
||
// Surface the full payload so CI logs reveal the actual failure mode.
|
||
if (!payload.facts_backstop) {
|
||
throw new Error(`put_page returned no facts_backstop. Full payload: ${JSON.stringify(payload, null, 2)}. isError=${(result as { isError?: boolean }).isError}`);
|
||
}
|
||
expect(payload.facts_backstop).toEqual({ skipped: 'dream_generated' });
|
||
});
|
||
|
||
test('put_page backstop does NOT skip on dream_generated:false / absent', async () => {
|
||
const result = await dispatchToolCall(engine, 'put_page', {
|
||
slug: 'note/anti-loop-real',
|
||
content: `---\ntype: note\ntitle: Real\n---\n${'real-looking content with claims. '.repeat(15)}`,
|
||
}, { remote: false, sourceId: 'default' });
|
||
const payload = JSON.parse(result.content[0].text);
|
||
if (!payload.facts_backstop) {
|
||
throw new Error(`put_page returned no facts_backstop. Full payload: ${JSON.stringify(payload, null, 2)}. isError=${(result as { isError?: boolean }).isError}`);
|
||
}
|
||
expect(payload.facts_backstop).toBeDefined();
|
||
if ('skipped' in payload.facts_backstop) {
|
||
expect(payload.facts_backstop.skipped).not.toBe('dream_generated');
|
||
}
|
||
});
|
||
});
|