v0.30.0 feat: calibration scorecards (Slice A1 of v0.30 wave) (#731)
* feat(schema): migration v40 — takes_resolved_quality + drift_decisions Slice A1 of the v0.30 wave. Bundles all wave schema in one migration so A2/B1/C1 carry no schema of their own (codex F6 schema-first ordering). - takes.resolved_quality TEXT with CHECK (correct/incorrect/partial). - takes_resolution_consistency CHECK enforces (quality, outcome) tuple consistency at the DB layer. partial → outcome=NULL. - One-shot backfill maps legacy resolved_outcome → resolved_quality so v0.28 brains keep working with no manual reclassification. - idx_takes_scorecard partial index on (holder, kind, resolved_quality) WHERE resolved_quality IS NOT NULL — scorecard hot path. - drift_decisions audit table (consumed by Slice C1 in v0.30.3). - PGLite branch via sqlFor.pglite mirrors the same shape; RLS DO-block is Postgres-only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(takes-fence): extend ParsedTake + parser + conditional renderer (codex F3) A codex consult on the v0.30 plan caught a real bug: the v0.28 parser had no concept of resolution columns, so every cmdUpdate after a cmdResolve silently deleted resolution data on the next render. This commit kills that data-loss path. ParsedTake gains optional resolvedAt, resolvedQuality, resolvedOutcome, resolvedEvidence, resolvedValue, resolvedUnit, resolvedBy. parseTakesFence detects v0.30-shape headers and reads resolution cells when present; v0.28 7-column fences round-trip byte-identical. renderTakesFence emits the resolution columns ONLY when at least one row on the page has resolvedQuality set — pages with no resolved rows keep the narrow shape exactly as before. 11 new test cases including the round-trip preservation regression gate. Without those tests, the silent-delete bug returns the moment the parser shape drifts. Tests cover: parsing v0.30 + v0.28 shapes, conditional rendering, partial quality round-trip, upsertTakeRow + supersedeRow preservation when a page already has resolved rows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(engine): getScorecard + getCalibrationCurve + 3-state TakeResolution Adds the calibration aggregate methods on BrainEngine. Both engines implement them with SQL-level allow-list filtering inside the GROUP BY (D4 fail-closed): hidden-holder rows contribute zero to aggregates. TakeResolution gains optional `quality` (correct|incorrect|partial). When both quality and outcome are supplied AND inconsistent, the engine throws TAKE_RESOLUTION_INVALID rather than silently overwriting. resolveTake writes both columns: quality directly, outcome derived (correct→true, incorrect→false, partial→NULL). Schema CHECK is the defense-in-depth backstop. Brier scope (D5 + D11): the SQL aggregation excludes partial rows from the Brier denominator — partial isn't a binary outcome to compare a probability against. partial_rate is reported alongside as a separate counter so hedging behavior stays visible. The 20% threshold lives in src/core/takes-resolution.ts and the CLI surfaces it in v0.30.0's cmdScorecard. New module src/core/takes-resolution.ts holds shared pure helpers (deriveResolutionTuple, finalizeScorecard) consumed by both engines so the math stays identical across backends. takeRowToTake (utils.ts) reads resolved_quality through to the Take row shape. 23 new test cases: 16 for the helpers (Brier hand-calc against a 4-bet reference at 0.205, n=0 no-divide, contradictory-input rejection, partial-exclusion contract, threshold constant); 7 against PGLite for the engine path (3-state quality writes, contradictory throws, scorecard hand-calc, n=0, SQL-level allow-list privacy filter). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): gbrain takes resolve --quality, takes scorecard, takes calibration cmdResolve widened: --quality correct|incorrect|partial is the new primary input. --outcome true|false stays as a back-compat alias auto-mapping to quality, with a stderr deprecation warning on use. Mutually exclusive with --quality. --evidence is a semantic alias for --source on the resolve subcommand. cmdResolve mirrors resolution metadata into the takes-fence on disk via the page-lock-aware path. Round-trip preservation through parseTakesFence + renderTakesFence keeps resolution data intact across unrelated edits to other rows on the same page. Removes the v0.28 deferred-rendering warning. cmdScorecard prints `correct | incorrect | partial`, accuracy, Brier (correct ∨ incorrect only; lower is better; 0.25 = always-50% baseline), and partial_rate. When partial_rate > 20% the CLI prints "[!] partial_rate is high — calibration may be optimistic" so hedging behavior stays visible even though it doesn't enter the math (D11). Small-N note when resolved < 100. JSON output via --json. cmdCalibration bins resolved correct/incorrect bets by stated weight (--bucket-size, default 0.1) and prints observed vs predicted vs delta per bucket. Diagonal alignment = perfect calibration. Both new subcommands wire allow-list as undefined for local CLI callers (trusted); MCP path will thread it from access_tokens.permissions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(mcp): register takes_scorecard + takes_calibration ops Both ops are read-scope, MCP-callable, allow-list-honoring. Handlers thread ctx.takesHoldersAllowList into the engine method's required allowList parameter, which applies WHERE holder = ANY at SQL aggregation level (D4 fail-closed). Local CLI callers leave the allow-list undefined and see all holders. Updates the OperationContext.takesHoldersAllowList contract comment to list the new aggregate ops alongside takes_list, takes_search, query. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.0 release: calibration core (Slice A1 of v0.30 wave) VERSION + package.json bump. CHANGELOG entry covers the release-summary (headline + math table + privacy note + data-loss-bug-killed note), "## To take advantage of v0.30.0" upgrade path, and itemized changes. llms-full.txt regenerated to capture the v0.28.x annotations that had been merged but not yet rolled into the docs bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): scorecard + calibration parity on real Postgres + NUMERIC fix Adds end-to-end coverage for v0.30.0 (Slice A1) against real Postgres: - test/e2e/takes-scorecard-parity.test.ts (new): seeds the same 6-bet fixture (4 binary garry + 1 partial garry + 1 binary harj) into both Postgres and PGLite, asserts getScorecard + getCalibrationCurve return byte-identical results across engines, runs the 4-bet hand-calc Brier reference (0.205) on real PG, and verifies the SQL-level allow-list filter strictly subtracts hidden-holder rows on both engines. - test/e2e/takes-postgres.test.ts: extended with 8 v0.30 cases — quality semantics (correct/partial/back-compat) writes the expected (quality, outcome) tuple on real PG; the takes_resolution_consistency CHECK constraint actually fires on a contradictory raw UPDATE; getScorecard + getCalibrationCurve coherent shape + ordered-bucket invariants; PRIVACY allow-list filter on real PG; MCP dispatch path for takes_scorecard + takes_calibration with allow-list threading. While writing the parity test, the e2e harness caught a real bug PGLite tolerated: postgres.js sends scalar `${bucketSize}` params as text by default, so `FLOOR(weight / $N)` tried to coerce '0.1' to integer and threw `invalid input syntax for type integer: "0.1"`. The NUMERIC fix also kills a separate FP-precision divergence — `FLOOR(0.7 / 0.1)` returns 6 on real PG (IEEE 754 rounds 0.7/0.1 to 6.9999...) and 7 on PGLite. Both engines now bucket via `weight::numeric / $N::numeric` which is exact decimal arithmetic and engine-agnostic. This is the v0.30.0 wave's first cross-engine parity test. Same shape will guard A2's getTrajectory + getAnnualReview when those land. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): correct v40→v43 reference and expand v0.30.0 test note Two fixes to the v0.30.0 entry after the master merge renumbered the migration: - The "#### Added" bullet still said "Schema migration v40"; bumped to v43. - The "#### Tests" section only enumerated unit tests. The PR also ships 19 E2E cases (11 in takes-scorecard-parity, 8 extending takes-postgres) that exercise the calibration math against real Postgres and the PG↔PGLite engine parity. Added the count + a note about the two real bugs the parity test caught (postgres.js string-typed scalar params and IEEE 754 bucketing divergence) that PGLite tolerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -228,3 +228,177 @@ d('v0.28 MCP allow-list — Postgres dispatch', () => {
|
||||
expect(env.remote_persisted_blocked).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0 (Slice A1): 3-state quality + scorecard + calibration on real PG.
|
||||
// Mirrors the unit-test invariants (PGLite) against postgres.js and the
|
||||
// real CHECK constraint enforcement.
|
||||
// ============================================================
|
||||
d('v0.30.0 takes resolve --quality on real Postgres', () => {
|
||||
test('quality=correct writes both columns; CHECK constraint is enforced', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 50, claim: 'Series A correct', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 50, { quality: 'correct', resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 50],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBe(true);
|
||||
expect(rows[0].resolved_quality).toBe('correct');
|
||||
});
|
||||
|
||||
test('quality=partial writes (partial, NULL) + CHECK accepts it', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 51, claim: 'partial scope bet', kind: 'bet', holder: 'garry', weight: 0.55 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 51, { quality: 'partial', resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean | null; resolved_quality: string | null }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 51],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBeNull();
|
||||
expect(rows[0].resolved_quality).toBe('partial');
|
||||
});
|
||||
|
||||
test('CHECK constraint rejects contradictory raw UPDATE', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 52, claim: 'unresolved bet', kind: 'bet', holder: 'garry', weight: 0.6 },
|
||||
]);
|
||||
// Bypass the engine method's deriveResolutionTuple guard and write a
|
||||
// contradictory tuple directly. The schema CHECK should refuse.
|
||||
await expect(
|
||||
engine.executeRaw(
|
||||
`UPDATE takes SET resolved_at = now(), resolved_outcome = true, resolved_quality = 'incorrect' WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 52],
|
||||
),
|
||||
).rejects.toThrow(/takes_resolution_consistency|check constraint/i);
|
||||
});
|
||||
|
||||
test('back-compat: resolveTake with outcome=true → quality=correct on real PG', async () => {
|
||||
const engine = getEngine();
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: acmePageId, row_num: 53, claim: 'legacy v0.28 callers', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
]);
|
||||
await engine.resolveTake(acmePageId, 53, { outcome: true, resolvedBy: 'garry' });
|
||||
const rows = await engine.executeRaw<{ resolved_outcome: boolean; resolved_quality: string }>(
|
||||
`SELECT resolved_outcome, resolved_quality FROM takes WHERE page_id = $1 AND row_num = $2`,
|
||||
[acmePageId, 53],
|
||||
);
|
||||
expect(rows[0].resolved_outcome).toBe(true);
|
||||
expect(rows[0].resolved_quality).toBe('correct');
|
||||
});
|
||||
});
|
||||
|
||||
d('v0.30.0 scorecard + calibration on real Postgres', () => {
|
||||
// Note: this suite runs after the other Postgres takes suites in this
|
||||
// file, so the takes table already has resolved data from the earlier
|
||||
// tests. We don't need a fresh seed — we just verify the aggregate
|
||||
// queries work end-to-end against postgres.js bind shapes.
|
||||
test('getScorecard returns coherent shape against real PG', async () => {
|
||||
const engine = getEngine();
|
||||
const card = await engine.getScorecard({ holder: 'garry' }, undefined);
|
||||
expect(card.total_bets).toBeGreaterThan(0);
|
||||
expect(card.resolved).toBeGreaterThanOrEqual(0);
|
||||
expect(card.correct + card.incorrect + card.partial).toBe(card.resolved);
|
||||
if (card.correct + card.incorrect > 0) {
|
||||
expect(card.brier).not.toBeNull();
|
||||
expect(card.brier!).toBeGreaterThanOrEqual(0);
|
||||
expect(card.brier!).toBeLessThanOrEqual(1);
|
||||
}
|
||||
});
|
||||
|
||||
test('getCalibrationCurve returns ordered buckets against real PG', async () => {
|
||||
const engine = getEngine();
|
||||
const buckets = await engine.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
// Buckets must be ordered by bucket_lo ascending.
|
||||
for (let i = 1; i < buckets.length; i++) {
|
||||
expect(buckets[i].bucket_lo).toBeGreaterThanOrEqual(buckets[i - 1].bucket_lo);
|
||||
}
|
||||
// Every bucket has n > 0 (empty buckets aren't returned by GROUP BY).
|
||||
for (const b of buckets) expect(b.n).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('PRIVACY: scorecard SQL allow-list excludes hidden holders on real PG', async () => {
|
||||
const engine = getEngine();
|
||||
// Seed a holder that the scorecard with allow-list ['garry'] should
|
||||
// not see. Use a fresh page so the assertion is local and not noisy.
|
||||
const harjPage = await engine.putPage('companies/scorecard-allowlist-fixture', {
|
||||
title: 'Allow-list fixture', type: 'company', compiled_truth: '## Takes\n',
|
||||
});
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: harjPage.id, row_num: 1, claim: 'g bet', kind: 'bet', holder: 'garry', weight: 0.7 },
|
||||
{ page_id: harjPage.id, row_num: 2, claim: 'h bet', kind: 'bet', holder: 'harj-taggar', weight: 0.6 },
|
||||
]);
|
||||
await engine.resolveTake(harjPage.id, 1, { quality: 'correct', resolvedBy: 'garry' });
|
||||
await engine.resolveTake(harjPage.id, 2, { quality: 'incorrect', resolvedBy: 'harj-taggar' });
|
||||
|
||||
const garryOnly = await engine.getScorecard(
|
||||
{ domainPrefix: 'companies/scorecard-allowlist-fixture' },
|
||||
['garry'],
|
||||
);
|
||||
const trustedFull = await engine.getScorecard(
|
||||
{ domainPrefix: 'companies/scorecard-allowlist-fixture' },
|
||||
undefined,
|
||||
);
|
||||
expect(garryOnly.resolved).toBe(1);
|
||||
expect(garryOnly.correct).toBe(1);
|
||||
expect(trustedFull.resolved).toBe(2);
|
||||
// Allow-list strictly subtracts the harj row.
|
||||
expect(trustedFull.resolved - garryOnly.resolved).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================
|
||||
// v0.30.0: MCP dispatch path for takes_scorecard + takes_calibration.
|
||||
// ============================================================
|
||||
d('v0.30.0 MCP dispatch — Postgres', () => {
|
||||
test('takes_scorecard via MCP returns correct counts with allow-list', async () => {
|
||||
const engine = getEngine();
|
||||
const result = await dispatchToolCall(engine, 'takes_scorecard', { holder: 'garry' }, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['garry'],
|
||||
});
|
||||
expect(result.isError).toBeFalsy();
|
||||
const card = JSON.parse(result.content[0].text);
|
||||
expect(card).toHaveProperty('correct');
|
||||
expect(card).toHaveProperty('incorrect');
|
||||
expect(card).toHaveProperty('partial');
|
||||
expect(card).toHaveProperty('brier');
|
||||
expect(card).toHaveProperty('partial_rate');
|
||||
});
|
||||
|
||||
test('takes_calibration via MCP returns bucket array with allow-list', async () => {
|
||||
const engine = getEngine();
|
||||
const result = await dispatchToolCall(engine, 'takes_calibration', { holder: 'garry', bucket_size: 0.1 }, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['garry'],
|
||||
});
|
||||
expect(result.isError).toBeFalsy();
|
||||
const buckets = JSON.parse(result.content[0].text);
|
||||
expect(Array.isArray(buckets)).toBe(true);
|
||||
if (buckets.length > 0) {
|
||||
expect(buckets[0]).toHaveProperty('bucket_lo');
|
||||
expect(buckets[0]).toHaveProperty('bucket_hi');
|
||||
expect(buckets[0]).toHaveProperty('n');
|
||||
expect(buckets[0]).toHaveProperty('observed');
|
||||
expect(buckets[0]).toHaveProperty('predicted');
|
||||
}
|
||||
});
|
||||
|
||||
test('PRIVACY: takes_scorecard with allow-list ["world"] excludes garry rows', async () => {
|
||||
const engine = getEngine();
|
||||
// 'world' has only fact-kind takes in the seed; bets are garry-only.
|
||||
// Scorecard scoped to world should report zero resolved.
|
||||
const result = await dispatchToolCall(engine, 'takes_scorecard', {}, {
|
||||
remote: true,
|
||||
takesHoldersAllowList: ['world'],
|
||||
});
|
||||
const card = JSON.parse(result.content[0].text);
|
||||
// No resolved bets exist with holder='world' in our seed.
|
||||
expect(card.resolved).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
202
test/e2e/takes-scorecard-parity.test.ts
Normal file
202
test/e2e/takes-scorecard-parity.test.ts
Normal file
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* v0.30.0 (Slice A1) E2E: scorecard + calibration parity between Postgres
|
||||
* and PGLite. Same fixture, same query, same numbers.
|
||||
*
|
||||
* Seeds 4 binary bets + 1 partial bet into both engines (mirrors the
|
||||
* `takes-resolution.test.ts` 4-bet hand-calc reference: Brier=0.205).
|
||||
* Asserts getScorecard returns byte-identical numeric output and
|
||||
* getCalibrationCurve emits the same buckets.
|
||||
*
|
||||
* Privacy gate: also asserts the SQL-level allow-list filter (D4
|
||||
* fail-closed) returns identical results across engines — hidden-holder
|
||||
* rows must contribute zero on both sides.
|
||||
*
|
||||
* Skips gracefully when DATABASE_URL is unset.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import type { TakesScorecardOpts, CalibrationCurveOpts } from '../../src/core/engine.ts';
|
||||
|
||||
const SKIP_PG = !hasDatabase();
|
||||
const d = SKIP_PG ? describe.skip : describe;
|
||||
|
||||
let pglite: PGLiteEngine;
|
||||
let pgEngine: BrainEngine;
|
||||
|
||||
interface FixtureBet {
|
||||
rowNum: number;
|
||||
claim: string;
|
||||
holder: string;
|
||||
weight: number;
|
||||
resolveAs?: 'correct' | 'incorrect' | 'partial';
|
||||
}
|
||||
|
||||
// Same fixture used by takes-resolution.test.ts hand-calc.
|
||||
// 4 binary garry bets at varied weights (Brier=0.205) + 1 partial garry +
|
||||
// 1 binary harj bet so the allow-list assertion has signal.
|
||||
const FIXTURE_BETS: FixtureBet[] = [
|
||||
{ rowNum: 1, claim: 'b1', holder: 'garry', weight: 0.9, resolveAs: 'correct' },
|
||||
{ rowNum: 2, claim: 'b2', holder: 'garry', weight: 0.6, resolveAs: 'correct' },
|
||||
{ rowNum: 3, claim: 'b3', holder: 'garry', weight: 0.7, resolveAs: 'incorrect' },
|
||||
{ rowNum: 4, claim: 'b4', holder: 'garry', weight: 0.4, resolveAs: 'incorrect' },
|
||||
{ rowNum: 5, claim: 'b5', holder: 'garry', weight: 0.5, resolveAs: 'partial' },
|
||||
{ rowNum: 6, claim: 'h1', holder: 'harj-taggar', weight: 0.8, resolveAs: 'correct' },
|
||||
];
|
||||
|
||||
const SCORECARD_PAGE = 'companies/scorecard-parity-fixture';
|
||||
|
||||
async function seedFixture(engine: BrainEngine): Promise<number> {
|
||||
const page = await engine.putPage(SCORECARD_PAGE, {
|
||||
title: 'Scorecard parity fixture',
|
||||
type: 'company' as const,
|
||||
compiled_truth: '## Takes\n',
|
||||
});
|
||||
await engine.addTakesBatch(
|
||||
FIXTURE_BETS.map(b => ({
|
||||
page_id: page.id,
|
||||
row_num: b.rowNum,
|
||||
claim: b.claim,
|
||||
kind: 'bet' as const,
|
||||
holder: b.holder,
|
||||
weight: b.weight,
|
||||
})),
|
||||
);
|
||||
for (const b of FIXTURE_BETS) {
|
||||
if (b.resolveAs === undefined) continue;
|
||||
await engine.resolveTake(page.id, b.rowNum, {
|
||||
quality: b.resolveAs,
|
||||
resolvedBy: b.holder,
|
||||
});
|
||||
}
|
||||
return page.id;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
if (SKIP_PG) return;
|
||||
// PGLite (in-memory, no DATABASE_URL needed; runs even on the skipped
|
||||
// suite path so we'd still know if its seed broke).
|
||||
pglite = new PGLiteEngine();
|
||||
await pglite.connect({});
|
||||
await pglite.initSchema();
|
||||
await seedFixture(pglite);
|
||||
|
||||
// Real Postgres
|
||||
await setupDB();
|
||||
pgEngine = getEngine();
|
||||
await seedFixture(pgEngine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (pglite) await pglite.disconnect();
|
||||
if (!SKIP_PG) await teardownDB();
|
||||
});
|
||||
|
||||
d('v0.30.0 e2e: scorecard parity (PG vs PGLite)', () => {
|
||||
const queries: Array<{ name: string; opts: TakesScorecardOpts; allowList?: string[] }> = [
|
||||
{ name: 'all garry, no allow-list', opts: { holder: 'garry', domainPrefix: SCORECARD_PAGE } },
|
||||
{ name: 'all rows, no holder filter', opts: { domainPrefix: SCORECARD_PAGE } },
|
||||
{ name: 'allow-list garry-only', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['garry'] },
|
||||
{ name: 'allow-list world-only (zero match)', opts: { domainPrefix: SCORECARD_PAGE }, allowList: ['world'] },
|
||||
];
|
||||
|
||||
for (const q of queries) {
|
||||
test(`scorecard: ${q.name}`, async () => {
|
||||
const pgCard = await pgEngine.getScorecard(q.opts, q.allowList);
|
||||
const pgliteCard = await pglite.getScorecard(q.opts, q.allowList);
|
||||
expect(pgCard.total_bets).toBe(pgliteCard.total_bets);
|
||||
expect(pgCard.resolved).toBe(pgliteCard.resolved);
|
||||
expect(pgCard.correct).toBe(pgliteCard.correct);
|
||||
expect(pgCard.incorrect).toBe(pgliteCard.incorrect);
|
||||
expect(pgCard.partial).toBe(pgliteCard.partial);
|
||||
// Float fields: byte-equality is too strict due to engine-driver
|
||||
// numeric coercion (Postgres returns string-coerced floats; PGLite
|
||||
// returns JS numbers). Use 6-decimal closeness — well below any
|
||||
// user-visible precision.
|
||||
const closeOrBothNull = (a: number | null, b: number | null) => {
|
||||
if (a === null && b === null) return;
|
||||
expect(a).not.toBeNull();
|
||||
expect(b).not.toBeNull();
|
||||
expect(a).toBeCloseTo(b!, 6);
|
||||
};
|
||||
closeOrBothNull(pgCard.accuracy, pgliteCard.accuracy);
|
||||
closeOrBothNull(pgCard.brier, pgliteCard.brier);
|
||||
closeOrBothNull(pgCard.partial_rate, pgliteCard.partial_rate);
|
||||
});
|
||||
}
|
||||
|
||||
test('Brier on the 4-bet hand-calc reference matches 0.205 on both engines', async () => {
|
||||
// Filter to garry bets only — the harj bet would skew the Brier.
|
||||
const opts: TakesScorecardOpts = { holder: 'garry', domainPrefix: SCORECARD_PAGE };
|
||||
const pgCard = await pgEngine.getScorecard(opts, undefined);
|
||||
const pgliteCard = await pglite.getScorecard(opts, undefined);
|
||||
expect(pgCard.brier).toBeCloseTo(0.205, 4);
|
||||
expect(pgliteCard.brier).toBeCloseTo(0.205, 4);
|
||||
// Also: partial_rate = 1/5 = 0.2 (4 binary + 1 partial = 5 resolved garry rows).
|
||||
expect(pgCard.partial_rate).toBeCloseTo(0.2, 6);
|
||||
expect(pgliteCard.partial_rate).toBeCloseTo(0.2, 6);
|
||||
});
|
||||
|
||||
test('PRIVACY: allow-list ["garry"] gives same totals on both engines AND excludes harj', async () => {
|
||||
const opts: TakesScorecardOpts = { domainPrefix: SCORECARD_PAGE };
|
||||
const pgGarry = await pgEngine.getScorecard(opts, ['garry']);
|
||||
const pgliteGarry = await pglite.getScorecard(opts, ['garry']);
|
||||
|
||||
// Both engines see exactly the 5 garry rows.
|
||||
expect(pgGarry.resolved).toBe(5);
|
||||
expect(pgliteGarry.resolved).toBe(5);
|
||||
|
||||
// Without allow-list, both see all 6 resolved rows (5 garry + 1 harj).
|
||||
const pgFull = await pgEngine.getScorecard(opts, undefined);
|
||||
const pgliteFull = await pglite.getScorecard(opts, undefined);
|
||||
expect(pgFull.resolved).toBe(6);
|
||||
expect(pgliteFull.resolved).toBe(6);
|
||||
|
||||
// Allow-list strictly subtracts (defense in depth: if SQL filter were
|
||||
// post-filtered or applied wrong, this assertion catches it).
|
||||
expect(pgFull.resolved - pgGarry.resolved).toBe(1);
|
||||
expect(pgliteFull.resolved - pgliteGarry.resolved).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
d('v0.30.0 e2e: calibration curve parity (PG vs PGLite)', () => {
|
||||
const queries: Array<{ name: string; opts: CalibrationCurveOpts; allowList?: string[] }> = [
|
||||
{ name: 'garry-only, default bucket', opts: { holder: 'garry' } },
|
||||
{ name: 'all-rows, default bucket', opts: {} },
|
||||
{ name: 'allow-list garry-only', opts: {}, allowList: ['garry'] },
|
||||
];
|
||||
for (const q of queries) {
|
||||
test(`calibration: ${q.name}`, async () => {
|
||||
const pgBuckets = await pgEngine.getCalibrationCurve(q.opts, q.allowList);
|
||||
const pgliteBuckets = await pglite.getCalibrationCurve(q.opts, q.allowList);
|
||||
expect(pgBuckets.length).toBe(pgliteBuckets.length);
|
||||
for (let i = 0; i < pgBuckets.length; i++) {
|
||||
expect(pgBuckets[i].bucket_lo).toBeCloseTo(pgliteBuckets[i].bucket_lo, 6);
|
||||
expect(pgBuckets[i].bucket_hi).toBeCloseTo(pgliteBuckets[i].bucket_hi, 6);
|
||||
expect(pgBuckets[i].n).toBe(pgliteBuckets[i].n);
|
||||
if (pgBuckets[i].observed !== null) {
|
||||
expect(pgBuckets[i].observed).toBeCloseTo(pgliteBuckets[i].observed!, 6);
|
||||
} else {
|
||||
expect(pgliteBuckets[i].observed).toBeNull();
|
||||
}
|
||||
if (pgBuckets[i].predicted !== null) {
|
||||
expect(pgBuckets[i].predicted).toBeCloseTo(pgliteBuckets[i].predicted!, 6);
|
||||
} else {
|
||||
expect(pgliteBuckets[i].predicted).toBeNull();
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
test('partial bet (weight 0.5) is excluded from calibration curve on both engines', async () => {
|
||||
const pgBuckets = await pgEngine.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
const pgliteBuckets = await pglite.getCalibrationCurve({ holder: 'garry' }, undefined);
|
||||
const pgTotal = pgBuckets.reduce((s, b) => s + b.n, 0);
|
||||
const pgliteTotal = pgliteBuckets.reduce((s, b) => s + b.n, 0);
|
||||
// 4 binary garry rows; partial excluded.
|
||||
expect(pgTotal).toBe(4);
|
||||
expect(pgliteTotal).toBe(4);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user