v0.42.18.0 fix: sync orphan-pileup watchdog (#1633) + links-lag µs stamp (#1768) (#1807)

* fix(extract): links_extraction_lag never clears on Postgres (#1768) Stamp the full-microsecond updated_at (via to_char ... AT TIME ZONE UTC) instead of the millisecond-truncated JS Date, so links_extracted_at equals the DB updated_at exactly and the staleness predicate clears. Stamp SQL unchanged: version-arm backdating still works, D4 preserved, CDX-1 strengthened. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(core): out-of-band hard-deadline watchdog primitive (#1633) Bun eval-Worker that SIGTERM->grace->SIGKILLs its own process from a separate OS thread, so a sync whose main event loop is starved (ReDoS spin) still dies. Signals SELF (no PID-reuse footgun). Empirically validated on Bun 1.3.13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): arm hard-deadline watchdog + graceful SIGINT cancel (#1633) cli.ts installs the watchdog before connectEngine (bounds connect hangs); resolveSyncHardDeadline + composeAbortSignals in sync.ts; SIGINT graceful cancel on single-source + --all; withRefreshingLock timer unref'd. Non-TTY default 3600s makes cron orphan-pileup structurally impossible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(claude): annotate process-watchdog + #1768/#1633 fixes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: rebump v0.42.13.0 -> v0.42.18.0 (queue collision) Sibling workspaces claimed v0.42.13-v0.42.17; advance this branch's slot. VERSION + package.json + CHANGELOG header + CLAUDE.md annotations + llms bundles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state phrasing for #1633/#1768 entries (fix check:doc-history) The doc-history guard bans the bolded **v0.X release-clause marker in reference docs (history belongs in CHANGELOG + git). Rewrote the extract.ts/sync.ts additions as current-state prose and de-versioned the process-watchdog entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 08:00:57 -07:00
parent fd2fde9d26
commit bde11bb18f
18 changed files with 655 additions and 9 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,31 @@

 All notable changes to GBrain will be documented in this file.

+## [0.42.18.0] - 2026-06-03
+
+**A scheduled `gbrain sync` can no longer spin forever and pile up dead processes, and `gbrain doctor` stops showing "100% of pages need link extraction" right after you ran the thing that's supposed to fix it.**
+
+Two unrelated bugs, both reported from real Postgres/Supabase brains, both fixed here.
+
+The first one is the scary one. A `gbrain sync --source <id>` fired from cron could get stuck in a busy loop, peg a CPU core, and ignore `Ctrl-C` and `kill` (only `kill -9` stopped it). When the cron parent exited, the stuck sync was left orphaned, and the next cron tick spawned another. One reporter woke up to 13 of them, 24+ hours old, thrashing a 16 GB Mac mini down to 121 MB free. The root cause: when a sync spins on synchronous work, it starves its own event loop, so the SIGTERM handler and `--timeout` that gbrain already had could never actually run.
+
+The fix is a watchdog that runs on a separate OS thread and kills the process from outside the starved loop. On a non-interactive run (cron), gbrain now arms a hard deadline by default (1 hour), sends SIGTERM at the deadline for a clean exit, and SIGKILL shortly after if the process is too wedged to respond. A runaway sync now dies on its own instead of piling up. Sync is resumable, so a deadline-hit run just picks up next tick.
+
+- **Default on for cron, off for you at the keyboard.** Interactive (TTY) syncs stay unbounded. Tune the cron deadline with `GBRAIN_SYNC_MAX_RUNTIME_SECONDS=N`, set a one-off with `gbrain sync --hard-deadline 600`, or opt out with `--no-hard-deadline`. `gbrain sync --source x --timeout 300` now also arms the hard backstop automatically.
+- **`Ctrl-C` is clean now too.** Hitting Ctrl-C during a long sync returns a partial result and releases the lock through the normal path, instead of a hard cut that could leave the sync lock stuck until its TTL expired.
+- **The spin itself isn't root-caused yet** (it needs a live reproduction; the leading suspect is a pathological regex in a schema pack's link rules, already partly mitigated). The watchdog makes the *symptom* impossible. A `[sync-watchdog]` heartbeat line in your logs, plus the existing `[gbrain phase]` lines, will pinpoint where the next one hangs.
+
+The second fix: on Postgres, `gbrain doctor`'s `links_extraction_lag` check was permanently stuck at 100%. You'd run `gbrain extract --stale`, it would stamp every page as extracted, and the check would still say every page needs extraction. The stamp was being truncated to millisecond precision while the database kept microseconds, so "last extracted" always looked a hair older than "last updated." Now the stamp carries full microsecond precision and the check clears the moment extraction runs. (Postgres-only; the health score stops being dragged down by a check that could never pass.)
+
+## To take advantage of v0.42.18.0
+
+`gbrain upgrade` is all that's required — both fixes are automatic. Two things worth knowing:
+
+1. **Scheduled (non-interactive) syncs now have a 1-hour hard deadline by default.** If you run a legitimately long sync from cron (e.g. a first import of a very large brain), raise it: `GBRAIN_SYNC_MAX_RUNTIME_SECONDS=14400 gbrain sync ...` (4h), pass `gbrain sync --hard-deadline <seconds>`, or opt out with `--no-hard-deadline`. Interactive runs at your keyboard are unbounded as before.
+2. **Verify the link-extraction fix (Postgres):** `gbrain extract --stale` then `gbrain doctor` — the `links_extraction_lag` check should now read near 0% and stay there on a re-run (it was stuck at 100%).
+
+### For contributors
+- New reusable primitive `src/core/process-watchdog.ts` (Bun worker-thread self-kill, `eval:true` so it survives `bun --compile`); `resolveSyncHardDeadline` + `composeAbortSignals` in `sync.ts`; watchdog armed in `cli.ts` before `connectEngine` (bounds connect-phase hangs). #1768 fix threads a full-µs `updated_at_iso` (projected via `to_char(... AT TIME ZONE 'UTC', '…US"Z"')`) into `StalePageRow`; the `markPagesExtractedBatch` SQL is unchanged so the version-arm / CDX-1 tests stay green. New tests: `test/process-watchdog.test.ts` (+ `.serial`), `test/sync-hard-deadline.test.ts`, and a deterministic µs regression in `test/extract-stale.test.ts`. Eng review + Codex outside-voice both cleared the plan (Codex empirically validated the worker-self-kill on Bun 1.3.13).
 ## [0.42.17.0] - 2026-06-03

 **A huge `gbrain sync` can no longer get stuck forever losing all its progress when it's killed partway through.** If your brain suddenly grows by tens of thousands of pages (say a background process is enriching one page per commit, all night long), the next sync has a giant backlog to import. If that sync gets killed before it finishes — a session timeout, a laptop sleep, anything — it used to throw away **everything** it had done and start over from zero. The next hour the backlog was even bigger, so it got killed again, and again. It could never catch up. This release makes sync **resumable**: a killed sync banks what it imported, and the next run picks up where it left off. It converges.
--- a/2
+++ b/2
@@ -1 +1 @@
-0.42.17.0
+0.42.18.0
--- a/docs/architecture/KEY_FILES.md
+++ b/docs/architecture/KEY_FILES.md
--- a/package.json
+++ b/package.json
@@ -143,5 +143,5 @@
    "bun": ">=1.3.10"
  },
  "license": "MIT",
-  "version": "0.42.17.0"
+  "version": "0.42.18.0"
 }
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -1439,6 +1439,39 @@ async function handleCliOnly(command: string, args: string[]) {
    return;
  }

+  // #1633: out-of-band hard-deadline watchdog for `gbrain sync`. Installed
+  // BEFORE connectEngine so a connect-phase hang (the reported zombie class) is
+  // bounded too. A Bun Worker on its own OS thread SIGKILLs the process at the
+  // deadline even when the main event loop is starved by a synchronous spin —
+  // the only thing that stops the cron orphan-pileup. Disposed in the finally.
+  let syncWatchdog: { dispose(): void } | null = null;
+  if (command === 'sync') {
+    try {
+      const { resolveSyncHardDeadline } = await import('./commands/sync.ts');
+      const res = resolveSyncHardDeadline(args, {
+        isTty: Boolean(process.stdout.isTTY),
+        env: process.env,
+      });
+      if (res) {
+        const { installProcessWatchdog } = await import('./core/process-watchdog.ts');
+        syncWatchdog = installProcessWatchdog({
+          deadlineMs: res.deadlineMs,
+          graceMs: res.graceMs,
+          label: 'sync-watchdog',
+          heartbeatMs: 60_000,
+        });
+        process.stderr.write(
+          `[sync-watchdog] hard deadline armed: ${Math.round(res.deadlineMs / 1000)}s ` +
+          `+ ${Math.round(res.graceMs / 1000)}s grace (${res.reason}); disable with --no-hard-deadline\n`,
+        );
+      }
+    } catch (e) {
+      // A bad --hard-deadline value throws here (same posture as --timeout).
+      console.error(e instanceof Error ? e.message : String(e));
+      process.exit(1);
+    }
+  }
+
  // All remaining CLI-only commands need a DB connection
  const engine = await connectEngine();
  try {
@@ -1875,6 +1908,7 @@ async function handleCliOnly(command: string, args: string[]) {
      }
    }
  } finally {
+    syncWatchdog?.dispose(); // #1633: tear down the hard-deadline watchdog on clean exit
    if (command !== 'serve') await engine.disconnect();
  }
 }
--- a/src/commands/extract.ts
+++ b/src/commands/extract.ts
@@ -1636,7 +1636,12 @@ async function extractStaleFromDB(
      // landing between this SELECT and the stamp advances updated_at past the
      // stamped value, so the page stays stale and re-extracts next run instead
      // of being marked fresh-with-stale-content.
-      processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at.toISOString() });
+      //
+      // #1768: stamp the FULL-µs `updated_at_iso` (projected via to_char), NOT
+      // `page.updated_at.toISOString()` — the JS Date is ms-truncated, so the
+      // µs-precision DB updated_at stayed strictly greater and the page never
+      // cleared on Postgres. Stamping the exact value makes them equal.
+      processedRefs.push({ slug: page.slug, source_id: page.source_id, extractedAt: page.updated_at_iso });
    }

    // Flush NON-swallowing (CDX-4): a throw here propagates out of the sweep so
--- a/src/commands/sync.ts
+++ b/src/commands/sync.ts
@@ -2348,6 +2348,85 @@ async function performFullSync(
  };
 }

+/**
+ * Grace window (seconds) between the watchdog's SIGTERM and SIGKILL. SIGTERM
+ * gives a responsive loop a clean shutdown; SIGKILL is the starvation backstop.
+ */
+export const HARD_DEADLINE_GRACE_SEC = 30;
+
+export interface HardDeadlineResolution {
+  deadlineMs: number;
+  graceMs: number;
+  /** Where the deadline came from (for the armed-log line + tests). */
+  reason: string;
+}
+
+/**
+ * Resolve the out-of-band hard-deadline for a `gbrain sync` invocation (#1633).
+ * Pure + argv/env-only so it runs BEFORE `connectEngine` (so a connect-phase hang
+ * is also bounded — `timeout-layer-vs-connectengine` learning). DB-plane config
+ * (`gbrain config set`) is unreadable pre-connect, so the operator knob is the
+ * `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env var; bare cron is covered by the non-TTY
+ * default. Returns null when no watchdog should arm (TTY interactive with no
+ * flag, or an explicit opt-out / 0).
+ *
+ * Precedence: --no-hard-deadline > --hard-deadline > --timeout(non-all) > env >
+ * non-TTY default (3600s) > none.
+ */
+export function resolveSyncHardDeadline(
+  args: string[],
+  opts: { isTty: boolean; env?: Record<string, string | undefined>; defaultNonTtySec?: number },
+): HardDeadlineResolution | null {
+  const env = opts.env ?? {};
+  const graceMs = HARD_DEADLINE_GRACE_SEC * 1000;
+  const mk = (sec: number, reason: string): HardDeadlineResolution | null =>
+    sec > 0 ? { deadlineMs: sec * 1000, graceMs, reason } : null;
+
+  if (args.includes('--no-hard-deadline')) return null;
+
+  const hardStr = args.find((a, i) => args[i - 1] === '--hard-deadline');
+  if (hardStr !== undefined) {
+    // Throws on a bad value (cli.ts surfaces it + exits 1) — same posture as --timeout.
+    const sec = parseDurationSeconds(hardStr, '--hard-deadline');
+    return mk(sec ?? 0, 'flag:--hard-deadline');
+  }
+
+  // --timeout auto-arms a hard backstop at timeout(+grace), but ONLY single-source.
+  // For --all, per-source budgets don't collapse to one wall-clock; fall through.
+  const isAll = args.includes('--all');
+  const timeoutStr = args.find((a, i) => args[i - 1] === '--timeout');
+  if (timeoutStr !== undefined && !isAll) {
+    const sec = parseDurationSeconds(timeoutStr, '--timeout');
+    if (sec && sec > 0) return mk(sec, 'flag:--timeout');
+  }
+
+  const envRaw = env.GBRAIN_SYNC_MAX_RUNTIME_SECONDS;
+  if (envRaw !== undefined && envRaw !== '') {
+    const n = Number(envRaw);
+    if (Number.isFinite(n)) return mk(n, 'env:GBRAIN_SYNC_MAX_RUNTIME_SECONDS'); // n<=0 disables
+  }
+
+  if (!opts.isTty) return mk(opts.defaultNonTtySec ?? 3600, 'default:non-tty');
+
+  return null;
+}
+
+/**
+ * Compose 1..N AbortSignals into one (CQ2). Undefined inputs are dropped; the
+ * result aborts when ANY input aborts. Returns a single signal directly (no
+ * wrapper), `undefined` when nothing is set. Used at both performSync call sites
+ * so the SIGINT graceful-cancel signal and the per-source `--timeout` signal
+ * compose without duplicating `AbortSignal.any` logic.
+ */
+export function composeAbortSignals(
+  ...signals: Array<AbortSignal | undefined>
+): AbortSignal | undefined {
+  const live = signals.filter((s): s is AbortSignal => s !== undefined);
+  if (live.length === 0) return undefined;
+  if (live.length === 1) return live[0];
+  return AbortSignal.any(live);
+}
+
 export async function runSync(engine: BrainEngine, args: string[]) {
  // v0.40 Federated Sync v2: `gbrain sync trigger` subcommand
  // Routes to runSyncTrigger which queues a 'sync' minion job with
@@ -2827,6 +2906,13 @@ See also:
    };
    const perSourceResults: PerSourceResult[] = [];

+    // #1633 (Part B): one shared SIGINT controller for the whole --all fan-out.
+    // process-cleanup.ts doesn't own SIGINT, so without this Ctrl-C hard-cuts the
+    // run and can leak per-source locks; here it aborts every in-flight source so
+    // each performSync returns `partial` + releases its lock cleanly.
+    const allInterrupt = new AbortController();
+    const onAllSigint = () => { try { allInterrupt.abort(new Error('SIGINT')); } catch { /* */ } };
+
    const runOne = async (src: typeof sources[number]): Promise<SyncResult> => {
      const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
      // D18: parallel path defers embed; auto-enqueue embed-backfill after.
@@ -2862,7 +2948,7 @@ See also:
        sourceId: src.id,
        strategy: cfg.strategy,
        concurrency,
-        signal: controller?.signal,
+        signal: composeAbortSignals(allInterrupt.signal, controller?.signal),
      };
      // v0.40.6.0 (D6): wrap performSync in withSourcePrefix so every slog /
      // serr line emitted from inside the sync code path gets prefixed with
@@ -2941,6 +3027,8 @@ See also:
      ? Math.min(activeSources.length, maxSources ?? 8)
      : 1;

+    process.on('SIGINT', onAllSigint);
+    try {
    if (parallelEligible) {
      const { pMapAllSettled } = await import('../core/parallel.ts');
      const cap = effectiveParallel;
@@ -3013,6 +3101,9 @@ See also:
        }
      }
    }
+    } finally {
+      process.off('SIGINT', onAllSigint);
+    }

    const okCount = perSourceResults.filter((r) => r.status === 'ok').length;
    const errCount = perSourceResults.filter((r) => r.status === 'error').length;
@@ -3062,10 +3153,17 @@ See also:
    ? setTimeout(() => singleSourceController!.abort(), timeoutSeconds * 1000)
    : undefined;
  singleSourceTimer?.unref?.();
+  // #1633 (Part B): graceful SIGINT cancel. process-cleanup.ts owns SIGTERM
+  // (lock release + exit) and the watchdog owns the hard deadline; SIGINT is
+  // deliberately left to callers, so Ctrl-C during a long sync aborts the
+  // in-flight import cleanly (performSync returns `partial`, bookmark unadvanced,
+  // lock released by its own finally) instead of a hard cut.
+  const singleSourceInterrupt = new AbortController();
+  const onSingleSourceSigint = () => { try { singleSourceInterrupt.abort(new Error('SIGINT')); } catch { /* */ } };
  const opts: SyncOpts = {
    repoPath, dryRun, full, noPull, noEmbed, noExtract, skipFailed, retryFailed, noSchemaPack, sourceId,
    strategy: strategyArg, concurrency,
-    signal: singleSourceController?.signal,
+    signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
  };

  // Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
@@ -3086,10 +3184,12 @@ See also:
    // v0.41.13.0 (T6): try/finally clears the single-source timer so it
    // doesn't fire after performSync resolves OR throws.
    let result: SyncResult;
+    process.on('SIGINT', onSingleSourceSigint);
    try {
      result = await performSync(engine, opts);
    } finally {
      if (singleSourceTimer !== undefined) clearTimeout(singleSourceTimer);
+      process.off('SIGINT', onSingleSourceSigint);
    }
    printSyncResult(result);
    // v0.42.7 (#1696, D5): extraction-lag nudge after a completed single-source
--- a/src/core/db-lock.ts
+++ b/src/core/db-lock.ts
@@ -672,6 +672,10 @@ export async function withRefreshingLock<T>(
      }
    })();
  }, refreshIntervalMs);
+  // #1633: don't let the refresh timer keep the process alive on its own. The
+  // finally clearInterval is the primary cleanup; unref is belt-and-suspenders
+  // so a missed clear can't pin the event loop open past real work completion.
+  (interval as unknown as { unref?: () => void }).unref?.();

  try {
    return await work();
--- a/src/core/pglite-engine.ts
+++ b/src/core/pglite-engine.ts
@@ -2358,7 +2358,10 @@ export class PGLiteEngine implements BrainEngine {
    params.push(opts.batchSize);
    const limitIdx = params.length;
    const { rows } = await this.db.query(
-      `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at
+      // #1768: engine parity — project the same deterministic full-µs UTC string
+      // as postgres-engine.ts so extractStaleFromDB stamps the exact updated_at.
+      `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at,
+              to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso
         FROM pages
         WHERE ${where}${afterClause}
         ORDER BY id
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -2420,7 +2420,11 @@ export class PostgresEngine implements BrainEngine {
    params.push(opts.batchSize);
    const limitIdx = params.length;
    const rows = await this.sql.unsafe(
-      `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at
+      // #1768: project a deterministic full-µs UTC string alongside updated_at.
+      // to_char (not ::text — DateStyle-fragile) so extractStaleFromDB can stamp
+      // links_extracted_at = the exact updated_at and the staleness predicate clears.
+      `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at,
+              to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso
         FROM pages
         WHERE ${where}${afterClause}
         ORDER BY id
--- a/src/core/process-watchdog.ts
+++ b/src/core/process-watchdog.ts
@@ -0,0 +1,172 @@
+/**
+ * Out-of-band hard-deadline watchdog (#1633).
+ *
+ * THE PROBLEM. A `gbrain sync` that spins (e.g. synchronous catastrophic-regex
+ * in pack link-inference) STARVES the main event loop. When the loop never
+ * yields, the SIGTERM handler (process-cleanup.ts) can't run, a `--timeout`
+ * `setTimeout` can't fire, and the abort-flag checks between import iterations
+ * can't run either. The process becomes unkillable-by-SIGTERM and, under cron,
+ * orphans pile up for 24h+ (the reported incident). The ONLY thing that kills a
+ * loop-starved process is an OS signal delivered from OUTSIDE that loop.
+ *
+ * THE MECHANISM. A Bun `worker_threads` Worker runs on a real, independent OS
+ * thread with its own event loop. Its timer fires even while the main thread is
+ * in an unyielding synchronous loop. At the deadline it sends SIGTERM to its own
+ * process (a clean-shutdown chance if the loop happens to be responsive); at
+ * deadline+grace it sends SIGKILL (uncatchable — guaranteed death even when
+ * starved). Signaling SELF (`process.kill(process.pid, ...)`) has no PID-reuse
+ * footgun: the current process's PID is never reused while it's alive. (The
+ * rejected alternative — a detached child that signals the PARENT pid — CAN hit
+ * PID reuse and kill an innocent process.)
+ *
+ * `eval: true` keeps the worker body an inline string so it bakes into the
+ * `bun build --compile` binary with no separate-file embedding to worry about.
+ * Empirically validated on Bun 1.3.13 (a Worker timer fired + SIGKILLed the
+ * process while main was in `while(true){}`).
+ *
+ *   ┌─ main thread (may be starved) ──────────────┐   ┌─ watchdog worker (OS thread) ─┐
+ *   │ sync work / ReDoS spin / connect hang        │   │ t=deadline      -> SIGTERM     │
+ *   │   ...never yields...                          │   │ t=deadline+grace-> SIGKILL     │
+ *   │ on clean finish: handle.dispose()  ──────────┼──▶│   worker.terminate()           │
+ *   └──────────────────────────────────────────────┘   └────────────────────────────────┘
+ *
+ * Reusable beyond sync (autopilot / cycle are follow-up adopters): the API is
+ * just (deadline, grace, label).
+ */
+
+import { Worker } from 'node:worker_threads';
+
+export type WatchdogAction = 'wait' | 'sigterm' | 'sigkill';
+
+/**
+ * Pure decision function — the watchdog's whole state machine, extracted so it's
+ * unit-testable without spawning threads or real timers.
+ *   elapsed < deadline            -> 'wait'
+ *   deadline <= elapsed < +grace  -> 'sigterm' (clean-shutdown chance)
+ *   elapsed >= deadline + grace   -> 'sigkill' (guaranteed)
+ */
+export function watchdogDecision(elapsedMs: number, deadlineMs: number, graceMs: number): WatchdogAction {
+  if (elapsedMs >= deadlineMs + graceMs) return 'sigkill';
+  if (elapsedMs >= deadlineMs) return 'sigterm';
+  return 'wait';
+}
+
+export interface ProcessWatchdogOpts {
+  /** Wall-clock ms after which SIGTERM is sent. Must be > 0 or the watchdog is a no-op. */
+  deadlineMs: number;
+  /** ms after the deadline before SIGKILL. Default 30_000. */
+  graceMs?: number;
+  /** Prefix for stderr log lines, e.g. 'sync-watchdog'. Default 'watchdog'. */
+  label?: string;
+  /** Periodic "still alive, kill in Ns" heartbeat interval ms. 0 = off (default). */
+  heartbeatMs?: number;
+  /** Injectable warn sink (tests). Default writes to process.stderr. */
+  onWarn?: (msg: string) => void;
+}
+
+export interface WatchdogHandle {
+  /** Tear down the watchdog (clean completion). Idempotent. */
+  dispose(): void;
+  /** True when an out-of-band worker is actually running (false on no-op / fallback). */
+  readonly active: boolean;
+}
+
+const DEFAULT_GRACE_MS = 30_000;
+
+function defaultWarn(msg: string): void {
+  try { process.stderr.write(msg + '\n'); } catch { /* stderr may be broken */ }
+}
+
+const INERT: WatchdogHandle = { dispose() {}, get active() { return false; } };
+
+/**
+ * Worker body (runs on its own OS thread). Inline string so `eval: true` bakes
+ * it into the compiled binary. Uses only built-ins available in a Bun worker.
+ *
+ * `label` is validated by the caller to a safe charset before it reaches here,
+ * so it can't break the string literal or inject log lines.
+ */
+const WORKER_SRC = `
+const { workerData } = require('node:worker_threads');
+const { deadlineMs, graceMs, label, heartbeatMs } = workerData;
+const t0 = Date.now();
+function w(m) { try { process.stderr.write('[' + label + '] ' + m + '\\n'); } catch (e) {} }
+if (heartbeatMs > 0) {
+  const hb = setInterval(() => {
+    const elapsed = Math.round((Date.now() - t0) / 1000);
+    const killIn = Math.round((deadlineMs + graceMs - (Date.now() - t0)) / 1000);
+    w('parent alive ' + elapsed + 's elapsed, hard-kill in ~' + killIn + 's');
+  }, heartbeatMs);
+  if (typeof hb.unref === 'function') hb.unref();
+}
+setTimeout(() => {
+  w('deadline reached (' + Math.round(deadlineMs/1000) + 's) — sending SIGTERM for graceful shutdown');
+  try { process.kill(process.pid, 'SIGTERM'); } catch (e) {}
+}, deadlineMs);
+setTimeout(() => {
+  w('grace expired — sending SIGKILL (event loop was starved; this is the orphan-pileup backstop)');
+  try { process.kill(process.pid, 'SIGKILL'); } catch (e) {}
+}, deadlineMs + graceMs);
+`;
+
+/**
+ * Install the out-of-band hard-deadline watchdog. Returns a handle whose
+ * `dispose()` MUST be called on clean completion (a `finally`) so the worker is
+ * torn down. If the deadline is non-positive, returns an inert no-op handle.
+ *
+ * Fallback: if the Worker can't be constructed (unexpected on Bun), degrades to
+ * an in-process timer with a loud warning. The in-process timer canNOT fire
+ * under event-loop starvation — it only covers the responsive case — so the
+ * warning tells the operator the hard guarantee is degraded.
+ */
+export function installProcessWatchdog(opts: ProcessWatchdogOpts): WatchdogHandle {
+  const warn = opts.onWarn ?? defaultWarn;
+  const deadlineMs = Math.floor(opts.deadlineMs);
+  const graceMs = Math.max(0, Math.floor(opts.graceMs ?? DEFAULT_GRACE_MS));
+  // Sanitize label to a safe charset (defends the inline worker string + log lines).
+  const label = (opts.label ?? 'watchdog').replace(/[^A-Za-z0-9_.:-]/g, '').slice(0, 40) || 'watchdog';
+  const heartbeatMs = Math.max(0, Math.floor(opts.heartbeatMs ?? 0));
+
+  if (!Number.isFinite(deadlineMs) || deadlineMs <= 0) return INERT;
+
+  try {
+    const worker = new Worker(WORKER_SRC, {
+      eval: true,
+      workerData: { deadlineMs, graceMs, label, heartbeatMs },
+    });
+    // Don't let the watchdog keep the process alive past clean completion.
+    (worker as unknown as { unref?: () => void }).unref?.();
+    // A worker-side error must never crash the host; log and move on.
+    worker.on('error', (err) => warn(`[${label}] watchdog worker error: ${err instanceof Error ? err.message : String(err)}`));
+    let disposed = false;
+    return {
+      dispose() {
+        if (disposed) return;
+        disposed = true;
+        void worker.terminate();
+      },
+      get active() { return !disposed; },
+    };
+  } catch (err) {
+    // Fallback: in-process timer. Starvation-vulnerable — say so loudly.
+    warn(
+      `[${label}] could not start out-of-band watchdog (${err instanceof Error ? err.message : String(err)}); ` +
+      `falling back to an in-process timer that will NOT fire if the event loop is starved.`,
+    );
+    let killed = false;
+    const term = setTimeout(() => { try { process.kill(process.pid, 'SIGTERM'); } catch { /* */ } }, deadlineMs);
+    const kill = setTimeout(() => { killed = true; try { process.kill(process.pid, 'SIGKILL'); } catch { /* */ } }, deadlineMs + graceMs);
+    (term as unknown as { unref?: () => void }).unref?.();
+    (kill as unknown as { unref?: () => void }).unref?.();
+    let disposed = false;
+    return {
+      dispose() {
+        if (disposed || killed) return;
+        disposed = true;
+        clearTimeout(term);
+        clearTimeout(kill);
+      },
+      get active() { return !disposed; },
+    };
+  }
+}
--- a/src/core/types.ts
+++ b/src/core/types.ts
@@ -608,6 +608,15 @@ export interface StalePageRow {
  timeline: string;
  frontmatter: Record<string, unknown>;
  updated_at: Date;
+  /**
+   * Full-precision (microsecond) UTC ISO string of `updated_at`, projected
+   * straight from the DB via `to_char(... AT TIME ZONE 'UTC', '...US"Z"')`.
+   * `updated_at` (a JS `Date`) is millisecond-truncated, so stamping it back as
+   * `links_extracted_at` left the row permanently stale on Postgres (#1768).
+   * `extractStaleFromDB` stamps THIS instead, so `links_extracted_at` equals the
+   * DB `updated_at` exactly and the staleness predicate clears.
+   */
+  updated_at_iso: string;
 }

 export interface ChunkInput {
--- a/src/core/utils.ts
+++ b/src/core/utils.ts
@@ -139,6 +139,13 @@ export function rowToStalePage(row: Record<string, unknown>): StalePageRow {
    timeline: (row.timeline as string | null) ?? '',
    frontmatter: (fm == null ? {} : (typeof fm === 'string' ? JSON.parse(fm) : fm)) as Record<string, unknown>,
    updated_at: new Date(row.updated_at as string),
+    // #1768: full-µs UTC string projected by the SELECT (`updated_at_iso`).
+    // Fallback derives an ISO string from the Date — NEVER String(Date), which
+    // yields "Mon Jun 02 2026 …" that `::timestamptz` misparses. Pre-#1768
+    // callers that don't project the column still get a valid (ms) ISO value.
+    updated_at_iso: row.updated_at_iso != null
+      ? String(row.updated_at_iso)
+      : new Date(row.updated_at as string).toISOString(),
  };
 }

--- a/test/extract-stale.test.ts
+++ b/test/extract-stale.test.ts
@@ -178,6 +178,37 @@ describe('gbrain extract --stale', () => {
    expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
  });

+  test('CRITICAL (#1768): microsecond updated_at clears after --stale (no permanent lag)', async () => {
+    // Repro of #1768: extractStaleFromDB used to stamp page.updated_at.toISOString()
+    // (a JS Date, millisecond-truncated). The DB updated_at keeps microseconds, so
+    // `updated_at > links_extracted_at` stayed true forever and links_extraction_lag
+    // was stuck at 100% on Postgres. We inject a microsecond updated_at explicitly so
+    // the precision gap is deterministic regardless of the engine's now() granularity.
+    await engine.putPage('people/alice', personPage('Alice'));
+    await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) advises [Acme](companies/acme).'));
+    // Microsecond-precision updated_at, recent (after LINK_EXTRACTOR_VERSION_TS) so the
+    // version arm doesn't fire — the edited arm is what must clear.
+    await engine.executeRaw(`UPDATE pages SET updated_at = '2026-06-02 08:18:58.999166+00'`);
+    expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(2);
+
+    await runExtract(engine, ['--stale']);
+    // Fixed: links_extracted_at stamped at the EXACT microsecond updated_at, so the
+    // edited arm clears. Pre-fix this stayed at 2 (the bug).
+    expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
+
+    // And it stays cleared on a re-run (the "no matter how many times I re-run" symptom).
+    await runExtract(engine, ['--stale']);
+    expect(await engine.countStalePagesForExtraction({ versionTs: LINK_EXTRACTOR_VERSION_TS })).toBe(0);
+
+    // The stamp itself carries microsecond precision (not millisecond-truncated).
+    const stamp = await stampOf('companies/acme');
+    expect(stamp).not.toBeNull();
+    const usRows = await engine.executeRaw<{ eq: boolean }>(
+      `SELECT links_extracted_at >= updated_at AS eq FROM pages WHERE slug = 'companies/acme'`,
+    );
+    expect(usRows[0]?.eq).toBe(true);
+  });
+
  test('CDX-4 (D2): a link-flush throw aborts the sweep and leaves pages UNSTAMPED', async () => {
    await engine.putPage('people/alice', personPage('Alice'));
    await engine.putPage('companies/acme', companyPage('Acme', '[Alice](people/alice) founded [Acme](companies/acme).'));
--- a/test/fixtures/watchdog-harness.ts
+++ b/test/fixtures/watchdog-harness.ts
@@ -0,0 +1,33 @@
+/**
+ * Fixture for test/process-watchdog.serial.test.ts. Spawned via `bun`.
+ *
+ * Usage: bun watchdog-harness.ts <mode> <deadlineMs> <graceMs>
+ *   starve-with    — install the watchdog, then starve the event loop forever.
+ *                    The watchdog must SIGKILL this process by deadline+grace.
+ *   starve-without — no watchdog, just starve. Proves the busy loop truly hangs
+ *                    (the test kills it). Isolates the watchdog as cause of death.
+ *   clean-dispose  — install with a long deadline, dispose immediately, exit 0.
+ *                    The watchdog must NOT kill a cleanly-disposed process.
+ *
+ * Safety net: the busy loop self-exits after 8s so a failed test kill can't hang CI.
+ */
+import { installProcessWatchdog } from '../../src/core/process-watchdog.ts';
+
+const mode = process.argv[2] ?? 'starve-with';
+const deadlineMs = Number(process.argv[3] ?? 300);
+const graceMs = Number(process.argv[4] ?? 150);
+
+if (mode === 'starve-with' || mode === 'clean-dispose') {
+  const handle = installProcessWatchdog({ deadlineMs, graceMs, label: 'test-wd' });
+  if (mode === 'clean-dispose') {
+    handle.dispose();
+    process.stdout.write('DISPOSED\n');
+    process.exit(0);
+  }
+}
+
+// Starve the main event loop with a synchronous busy loop (simulates ReDoS).
+const start = Date.now();
+while (Date.now() - start < 8000) { /* spin — no await, no yield */ }
+process.stdout.write('SURVIVED\n'); // must NOT print under starve-with
+process.exit(0);
--- a/test/process-watchdog.serial.test.ts
+++ b/test/process-watchdog.serial.test.ts
@@ -0,0 +1,67 @@
+/**
+ * Bun-pinned integration for the out-of-band watchdog (#1633, plan A4).
+ *
+ * Bun's worker_threads Worker is flagged "experimental", and the whole #1633
+ * fix rests on a worker timer firing + SIGKILLing the process while the MAIN
+ * thread is starved by a synchronous loop. These tests spawn a real harness
+ * process that starves its own loop and assert the watchdog kills it anyway.
+ *
+ * Serial because they use real subprocesses + wall-clock timing.
+ */
+import { describe, test, expect } from 'bun:test';
+import { join } from 'node:path';
+
+const HARNESS = join(import.meta.dir, 'fixtures', 'watchdog-harness.ts');
+
+async function runHarness(
+  mode: string,
+  deadlineMs: number,
+  graceMs: number,
+  hardCapMs: number,
+): Promise<{ exitCode: number | null; signalled: boolean; elapsedMs: number; stdout: string; killedByTest: boolean }> {
+  const proc = Bun.spawn(['bun', HARNESS, mode, String(deadlineMs), String(graceMs)], {
+    stdout: 'pipe',
+    stderr: 'pipe',
+  });
+  const start = Date.now();
+  let killedByTest = false;
+  const cap = setTimeout(() => { killedByTest = true; proc.kill('SIGKILL'); }, hardCapMs);
+  await proc.exited;
+  clearTimeout(cap);
+  const elapsedMs = Date.now() - start;
+  const stdout = await new Response(proc.stdout).text();
+  // Bun surfaces signal death via exitCode === null + signalCode, or a negative
+  // exitCode on some platforms. Treat "not a clean 0" as signalled for our purpose.
+  const signalled = proc.exitCode !== 0;
+  return { exitCode: proc.exitCode, signalled, elapsedMs, stdout, killedByTest };
+}
+
+describe('process-watchdog integration (Bun-pinned)', () => {
+  test('starved process IS killed by the watchdog around deadline+grace', async () => {
+    // deadline 300 + grace 200 = ~500ms expected death. Hard cap 4s: if the
+    // watchdog failed, the test's own SIGKILL fires and the assertion catches it.
+    const r = await runHarness('starve-with', 300, 200, 4000);
+    expect(r.stdout).not.toContain('SURVIVED'); // the bug symptom
+    expect(r.killedByTest).toBe(false);          // watchdog, not the test, killed it
+    expect(r.signalled).toBe(true);
+    // Died well before the harness's 8s self-exit safety net, near deadline+grace.
+    expect(r.elapsedMs).toBeLessThan(3000);
+  }, 15000);
+
+  test('control: a starved process WITHOUT the watchdog does not self-exit', async () => {
+    // Proves the busy loop genuinely starves (so the death above is the watchdog).
+    // No watchdog installed; the test's hard cap (1.2s) is what kills it.
+    const r = await runHarness('starve-without', 300, 200, 1200);
+    expect(r.killedByTest).toBe(true);   // only the test's SIGKILL stopped it
+    expect(r.stdout).not.toContain('SURVIVED');
+  }, 15000);
+
+  test('clean dispose: a disposed watchdog never kills the process', async () => {
+    // Long deadline, disposed immediately, process exits 0 fast and prints DISPOSED.
+    const r = await runHarness('clean-dispose', 60000, 60000, 5000);
+    expect(r.exitCode).toBe(0);
+    expect(r.killedByTest).toBe(false);
+    expect(r.stdout).toContain('DISPOSED');
+    expect(r.elapsedMs).toBeLessThan(4000);
+  }, 15000);
+});
--- a/test/process-watchdog.test.ts
+++ b/test/process-watchdog.test.ts
@@ -0,0 +1,61 @@
+/**
+ * Pure-function coverage for the watchdog state machine (#1633). No threads, no
+ * real timers — the spawn-based integration lives in
+ * test/process-watchdog.serial.test.ts (Bun-pinned, real processes).
+ */
+import { describe, test, expect } from 'bun:test';
+import { watchdogDecision, installProcessWatchdog } from '../src/core/process-watchdog.ts';
+
+describe('watchdogDecision', () => {
+  const deadline = 1000;
+  const grace = 300;
+
+  test('waits before the deadline', () => {
+    expect(watchdogDecision(0, deadline, grace)).toBe('wait');
+    expect(watchdogDecision(999, deadline, grace)).toBe('wait');
+  });
+
+  test('SIGTERM at the deadline boundary (inclusive)', () => {
+    expect(watchdogDecision(1000, deadline, grace)).toBe('sigterm');
+    expect(watchdogDecision(1299, deadline, grace)).toBe('sigterm');
+  });
+
+  test('SIGKILL at deadline+grace boundary (inclusive)', () => {
+    expect(watchdogDecision(1300, deadline, grace)).toBe('sigkill');
+    expect(watchdogDecision(5000, deadline, grace)).toBe('sigkill');
+  });
+
+  test('zero grace goes straight to SIGKILL at the deadline', () => {
+    expect(watchdogDecision(999, deadline, 0)).toBe('wait');
+    expect(watchdogDecision(1000, deadline, 0)).toBe('sigkill');
+  });
+});
+
+describe('installProcessWatchdog (handle contract)', () => {
+  test('non-positive deadline returns an inert no-op handle', () => {
+    const warns: string[] = [];
+    const h0 = installProcessWatchdog({ deadlineMs: 0, onWarn: (m) => warns.push(m) });
+    expect(h0.active).toBe(false);
+    h0.dispose(); // idempotent, no throw
+    const hNeg = installProcessWatchdog({ deadlineMs: -5, onWarn: (m) => warns.push(m) });
+    expect(hNeg.active).toBe(false);
+  });
+
+  test('active handle disposes idempotently without killing the test process', () => {
+    // Long deadline so it never fires during the test; dispose tears it down.
+    const h = installProcessWatchdog({ deadlineMs: 60_000, graceMs: 60_000, label: 'unit-wd' });
+    expect(h.active).toBe(true);
+    h.dispose();
+    expect(h.active).toBe(false);
+    h.dispose(); // second dispose is a no-op
+    expect(h.active).toBe(false);
+  });
+
+  test('label is sanitized to a safe charset', () => {
+    // A nasty label must not throw at construction (it is stripped before the
+    // inline worker string). We dispose immediately so nothing fires.
+    const h = installProcessWatchdog({ deadlineMs: 60_000, label: "evil'; \n process.exit(1) //" });
+    expect(h.active).toBe(true);
+    h.dispose();
+  });
+});
--- a/test/sync-hard-deadline.test.ts
+++ b/test/sync-hard-deadline.test.ts
@@ -0,0 +1,90 @@
+/**
+ * #1633 wiring: resolveSyncHardDeadline precedence + composeAbortSignals.
+ * Pure (no engine, no env mutation — env is injected per-call).
+ */
+import { describe, test, expect } from 'bun:test';
+import {
+  resolveSyncHardDeadline,
+  composeAbortSignals,
+  HARD_DEADLINE_GRACE_SEC,
+} from '../src/commands/sync.ts';
+
+const GRACE_MS = HARD_DEADLINE_GRACE_SEC * 1000;
+
+describe('resolveSyncHardDeadline', () => {
+  test('--no-hard-deadline disables everything (even with --timeout)', () => {
+    const r = resolveSyncHardDeadline(['--source', 'x', '--timeout', '60', '--no-hard-deadline'], { isTty: false });
+    expect(r).toBeNull();
+  });
+
+  test('--hard-deadline wins and sets the deadline (with grace)', () => {
+    const r = resolveSyncHardDeadline(['--hard-deadline', '120'], { isTty: true });
+    expect(r).toEqual({ deadlineMs: 120_000, graceMs: GRACE_MS, reason: 'flag:--hard-deadline' });
+  });
+
+  test('--hard-deadline accepts s/m/h suffix', () => {
+    expect(resolveSyncHardDeadline(['--hard-deadline', '2m'], { isTty: true })?.deadlineMs).toBe(120_000);
+  });
+
+  test('--hard-deadline with a bad value throws (same posture as --timeout)', () => {
+    expect(() => resolveSyncHardDeadline(['--hard-deadline', 'nope'], { isTty: true })).toThrow();
+    expect(() => resolveSyncHardDeadline(['--hard-deadline', '0'], { isTty: true })).toThrow();
+  });
+
+  test('--timeout (single-source) auto-arms the hard backstop', () => {
+    const r = resolveSyncHardDeadline(['--source', 'briefings', '--timeout', '480'], { isTty: false });
+    expect(r).toEqual({ deadlineMs: 480_000, graceMs: GRACE_MS, reason: 'flag:--timeout' });
+  });
+
+  test('--timeout + --all does NOT auto-arm (per-source budgets); falls through', () => {
+    // Non-TTY → falls to the default; TTY → null.
+    const nonTty = resolveSyncHardDeadline(['--all', '--timeout', '60'], { isTty: false });
+    expect(nonTty?.reason).toBe('default:non-tty');
+    const tty = resolveSyncHardDeadline(['--all', '--timeout', '60'], { isTty: true });
+    expect(tty).toBeNull();
+  });
+
+  test('env GBRAIN_SYNC_MAX_RUNTIME_SECONDS sets the deadline', () => {
+    const r = resolveSyncHardDeadline([], { isTty: true, env: { GBRAIN_SYNC_MAX_RUNTIME_SECONDS: '900' } });
+    expect(r).toEqual({ deadlineMs: 900_000, graceMs: GRACE_MS, reason: 'env:GBRAIN_SYNC_MAX_RUNTIME_SECONDS' });
+  });
+
+  test('env 0 disables (overrides the non-TTY default)', () => {
+    const r = resolveSyncHardDeadline([], { isTty: false, env: { GBRAIN_SYNC_MAX_RUNTIME_SECONDS: '0' } });
+    expect(r).toBeNull();
+  });
+
+  test('non-TTY default is 3600s', () => {
+    const r = resolveSyncHardDeadline([], { isTty: false });
+    expect(r).toEqual({ deadlineMs: 3_600_000, graceMs: GRACE_MS, reason: 'default:non-tty' });
+  });
+
+  test('TTY interactive with no flag/env arms nothing', () => {
+    expect(resolveSyncHardDeadline([], { isTty: true })).toBeNull();
+  });
+
+  test('defaultNonTtySec override is honored', () => {
+    const r = resolveSyncHardDeadline([], { isTty: false, defaultNonTtySec: 60 });
+    expect(r?.deadlineMs).toBe(60_000);
+  });
+});
+
+describe('composeAbortSignals', () => {
+  test('all-undefined returns undefined', () => {
+    expect(composeAbortSignals(undefined, undefined)).toBeUndefined();
+  });
+
+  test('single signal is returned directly (no wrapper)', () => {
+    const c = new AbortController();
+    expect(composeAbortSignals(c.signal, undefined)).toBe(c.signal);
+  });
+
+  test('composite aborts when ANY input aborts', () => {
+    const a = new AbortController();
+    const b = new AbortController();
+    const sig = composeAbortSignals(a.signal, b.signal)!;
+    expect(sig.aborted).toBe(false);
+    b.abort(new Error('boom'));
+    expect(sig.aborted).toBe(true);
+  });
+});