triton6564685/gbrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.19.0 check-resolvable: add OpenClaw skills-dir fallback + docs/tests (#326)

Browse Source 
* Add OpenClaw skills fallback for check-resolvable

* feat: v0.17.0 foundation — errors/warnings split + AGENTS.md support + auto-manifest

First two workstreams of the v0.17.0 "skillify end-to-end" release. Landed
together because the D-CX-3 exit-code refactor is a prerequisite for W1's
warning-surfaced filing audit in Workstream 3.

## D-CX-3: split ResolvableReport into errors[] + warnings[] + --strict

Prior: `env.ok = report.issues.length === 0` treated warnings and errors
identically for exit status. Any warning forced exit 1, which meant the
planned filing-audit (W3) would break CI for every OpenClaw deployment
emitting advisory warnings.

New contract:
- `ResolvableReport.errors[]` and `warnings[]` as separate arrays.
- `issues[]` stays as deprecated backcompat union (remove in v0.18).
- Default: exit 0 unless any errors. Warnings are advisory.
- `--strict` flag promotes warnings to fail CI (explicit opt-in).

Files: src/core/check-resolvable.ts, src/commands/check-resolvable.ts
(added --strict flag + help text + header doc), src/commands/doctor.ts
(use new fields), test/check-resolvable-cli.test.ts (rewrite REGRESSION-GATE
to document the new contract, add 3 D-CX-3 cases).

## W1: AGENTS.md support + auto-manifest + priority fix

The reference OpenClaw deployment uses AGENTS.md (not RESOLVER.md) at the
workspace root, and ships without a manifest.json. check-resolvable
silently false-passed against it pre-W1: 0 manifest entries meant 0
reachability iterations meant 0 errors reported.

Post-W1 behavior against ~/git/<redacted>/workspace (smoke-tested live):
- Detects 102 skills via SKILL.md walk (no manifest.json needed)
- Flags 15 unreachable errors (exactly the essay's '~15% dark' finding)
- Flags 108 warnings (overlaps, gaps) — advisory, not blocking
- Auto-detects via \$OPENCLAW_WORKSPACE without --skills-dir

Changes:

- NEW src/core/resolver-filenames.ts: one source of truth for the
  filename policy. \`RESOLVER_FILENAMES = ['RESOLVER.md', 'AGENTS.md']\`.
  Callers import from here, never hardcode either name.

- NEW src/core/skill-manifest.ts: \`loadOrDeriveManifest()\` — reads
  manifest.json when present+valid, otherwise walks \`skillsDir/*/SKILL.md\`
  to derive a synthetic manifest. Both check-resolvable.ts AND dry-fix.ts
  now call this, replacing the two duplicated loaders that silently
  returned [] on missing file (F-ENG-1, D-CX-12).

- src/core/repo-root.ts (rewrite): auto-detect priority changed to put
  \$OPENCLAW_WORKSPACE ahead of findRepoRoot() walk when explicitly set
  (D-CX-4). Adds workspace-root AGENTS.md detection — OpenClaw layout
  places routing at workspace/AGENTS.md with skills/ below. New
  SkillsDirSource variants \`openclaw_workspace_env_root\` and
  \`openclaw_workspace_home_root\` for --verbose log clarity.

- src/core/check-resolvable.ts: accepts RESOLVER.md or AGENTS.md at the
  skills dir or one level up (workspace root). Uses loadOrDeriveManifest
  for reachability. Updated error messages reference both filenames.

- src/core/dry-fix.ts: unified manifest loader — auto-fix now works in
  AGENTS.md-only workspaces where it previously no-op'd silently.

- src/commands/check-resolvable.ts: new AUTO_DETECT_HINT import for
  clearer missing-skills-dir errors; updated sourceLabel map for the
  two new workspace-root variants.

Tests:
- test/skill-manifest.test.ts: 14 cases covering explicit-manifest,
  derived-manifest, malformed JSON, wrong shape, empty explicit array
  (honored as 'zero skills' declaration), dirname fallback when no
  name: frontmatter, underscore/dotfile dir skipping.
- test/repo-root.test.ts: new tests for the priority swap, AGENTS.md
  skills-dir variant, AGENTS.md workspace-root variant, both-files
  present (RESOLVER.md wins).
- test/check-resolvable-cli.test.ts: updated regression-gate to the
  new contract; added three D-CX-3 cases.

All 105 tests passing across the foundation surface.

Plan + reviews: ~/.claude/plans/p1-lets-just-vast-blanket.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 W2 — Check 5 trigger routing eval (structural)

Check 5 of the 10-step skillify checklist (the essay's "resolver
trigger eval") now runs structurally by default and has a dedicated
CLI verb for CI. Ships Layer A; Layer B (LLM tie-break) is reserved
for v0.18.

## New module: src/core/routing-eval.ts

The harness. Pure functions:

- `normalizeText(s)`: lowercase, strip non-alnum to spaces, collapse
  whitespace. Unicode-friendly, quote-agnostic, punctuation-tolerant.
- `extractTriggerPhrases(cellText)`: split quoted alternatives like
  `"search for", "find me"` into separate normalized phrases; fall
  back to the whole cell when unquoted (OpenClaw-style descriptions).
- `indexResolverTriggers(resolverContent)`: build a skill-slug →
  normalized-trigger-phrases map from the resolver table.
- `structuralRouteMatch(intent, index)`: substring-match the
  normalized intent against every trigger phrase; return the set of
  matched skills + whether the match was ambiguous (more than one
  specific skill, excluding always-on family).
- `lintRoutingFixtures`: rejects fixtures whose intent is
  verbatim-equal to a trigger (D-CX-6: fixtures must paraphrase the
  framing, not copy the trigger text) and unknown expected_skill
  references.
- `loadRoutingFixtures(skillsDir)`: walks `skills/<name>/routing-eval.jsonl`,
  handles JSONL line-comments (`//` / `#`), collects malformed lines
  separately without crashing.
- `runRoutingEval(resolver, fixtures)`: pure scoring. Supports
  negative cases (`expected_skill: null` — nothing should match) and
  an `ambiguous_with` allow-list for skills that co-fire with
  always-on handlers (signal-detector, brain-ops, ingest).

Outcomes per fixture: `pass`, `missed`, `ambiguous`, `false_positive`.
Metrics: `top1Accuracy`, `passed`, `missed`, `ambiguous`,
`falsePositives`.

## Integration: check-resolvable runs Layer A by default

`checkResolvable()` now loads `routing-eval.jsonl` fixtures from every
skill, runs the structural eval, and appends non-pass outcomes as
warning-severity issues. New issue types:

- `routing_miss`        — expected skill did not match
- `routing_ambiguous`   — expected matched AND unexpected skills
- `routing_false_positive` — negative case unexpectedly matched
- `routing_fixture_lint` — linter or malformed-JSONL finding

All four are warnings — routing issues don't break exit in default
mode, but `--strict` promotes them (D-CX-3 contract). Advisories
without breaking CI.

## New CLI verb: `gbrain routing-eval`

Standalone Check 5 runner. `--json` envelope, `--llm` flag reserved,
`--skills-dir` override. Exit codes: 0 clean, 1 any failure/lint, 2
setup error. Suitable for CI gating separately from check-resolvable.

Removed from DEFERRED in CLI: `{check: 5, name: trigger_routing_eval}`.
Check 6 (brain_filing) still deferred; lands in W3.

## Seed fixtures

- skills/query/routing-eval.jsonl
- skills/citation-fixer/routing-eval.jsonl (includes a negative case)

These are intentionally modest. Additional fixtures per skill are the
natural next step; routing-eval itself passes cleanly under
check-resolvable default mode even when fixtures surface real gaps
(they're warnings, not errors). Running `gbrain routing-eval` reveals
the gaps immediately.

## Tests (34 new cases + updated integrations)

- test/routing-eval.test.ts: full harness coverage including
  normalization, trigger extraction (quoted and unquoted), indexer,
  structural match with ambiguity + always-on exemption, fixture
  linter (verbatim-equality rule, unknown-skill rule, shape rule,
  negative-case skip), JSONL loader (comments, malformed lines,
  missing dirs, underscore/dot skipping), and every runRoutingEval
  outcome (pass, miss, ambiguous, negative-pass, false-positive, empty).

- test/check-resolvable-cli.test.ts: updated DEFERRED unit test +
  `--json` envelope test + `--verbose` test to reflect Check 5
  shipping.

140/140 passing across the W1 + W2 surface.

## Live smoke

`gbrain routing-eval --json` against the current gbrain repo: 6
fixtures, 1 passing, 5 missed. The misses correctly surface
resolver-trigger narrowness (intents users naturally phrase differently
than trigger text). Fixtures will iterate in follow-up PRs; the
machinery ships now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 W3 — Check 6 brain filing audit

Check 6 ships. Every skill that writes brain pages is now audited
against a machine-readable filing-rules doc at
`skills/_brain-filing-rules.json`.

## New: skills/_brain-filing-rules.json

Canonical filing rules, JSON (D-CX-8: the pre-existing yaml-lite
parser handles flat maps only, so YAML would have needed a new
dependency for one file). The companion `_brain-filing-rules.md`
stays as the human explainer. 14 rule entries + explicit
`sources_dir` carve-out for bulk/raw data.

## New module: src/core/filing-audit.ts

- `loadFilingRules(skillsDir)`: returns parsed doc or null (missing
  file → no-op; malformed JSON throws loud).
- `allowedDirectories(rules)`: normalized set of every rules[]
  directory + sources_dir.
- `runFilingAudit(skillsDir)`: walks skills/*/SKILL.md, parses
  frontmatter, audits any skill with `writes_pages: true`.

Two checks per qualifying skill:
  1. `writes_to:` list is non-empty.
  2. Every entry in `writes_to:` appears in allowedDirectories.

Both failures emit warning-severity issues. No errors — advisories
only, per D-CX-3.

## Distinction: writes_pages vs mutating (D-CX-7)

v0.17 introduces a new boolean frontmatter field `writes_pages:`.
`mutating: true` already means "has any side effect" (cron
schedulers, report writers, config mutators). Filing audit targets
ONLY skills with `writes_pages: true`, correctly excluding side-
effect-but-not-page-writing skills. The codex outside voice caught
this: conflating the two fields would drag ~100 skills into
filing-audit noise in the reference OpenClaw deployment.

## Integration: check-resolvable runs Check 6 by default

`checkResolvable()` calls `runFilingAudit(skillsDir)` and appends
issues as warnings. On missing/malformed rules doc, surfaces a
single advisory rather than bailing.

`DEFERRED` array in the CLI is now empty — v0.17 ships both Check 5
(W2) and Check 6 (W3). The export stays in place (stable --json
field) for future deferred checks.

## Seeded frontmatter on 7 canonical writers

Added `writes_pages: true` + `writes_to:` to:
- brain-ops (people, companies, deals, concepts, meetings)
- enrich (people, companies)
- ingest (people, companies, concepts, meetings, sources)
- idea-ingest (people, concepts, sources)
- media-ingest (concepts, people, companies, sources)
- meeting-ingestion (meetings, people, companies)
- signal-detector (people, companies, concepts)

Live smoke: `gbrain check-resolvable --json` on gbrain repo shows
`ok: true`, zero filing errors, zero filing warnings on seeded
skills. Every other mutating:true skill (citation-fixer,
cron-scheduler, data-research, maintain, migrate, minion-orchestrator,
reports, setup, skill-creator, soul-audit, webhook-transforms)
correctly skipped as side-effectful-but-not-page-writing.

## Tests (17 new cases + 3 updated CLI integrations)

test/filing-audit.test.ts covers:
  - rules loader: missing (null), valid, malformed (throw),
    non-array rules (throw)
  - directory normalization (trailing slash, leading slash)
  - clean case
  - missing writes_to on writes_pages:true
  - unknown directory
  - D-CX-7: mutating:true alone does not trigger audit
  - writes_pages:false skips
  - no frontmatter skips
  - inline `writes_to: [a, b]` syntax
  - block `writes_to:\n  - a` syntax
  - sources/ allowed
  - underscore/dot dir skipping
  - total counts (totalScanned vs writesPagesSkills)
  - missing dir graceful
  - action string quality guard

Plus: CLI integration tests updated for empty DEFERRED array (Checks
5 and 6 both shipped).

158/158 passing across the v0.17 foundation + W1 + W2 + W3 surface.

## v0.18 preview (D-CX-13)

v0.17 filing-audit is declaration-level only. A future
`gbrain filing-audit --pages` walks the brain itself, infers primary
subject from page content via LLM judgment, and flags actual
misfilings vs. declarations. Declaration audit is the leading
indicator; pages audit is the ground truth.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 W4 — gbrain skillify {scaffold,check} subcommand namespace

The essay's "skillify it!" verb becomes a CLI primitive pair. Two
subcommands, both promoted/factored so there's one source of truth:

## `gbrain skillify scaffold <name>` (mechanical)

Pure file generation. Zero LLM, zero judgment. Writes 5 stub files
atomically:

  1. skills/<name>/SKILL.md              frontmatter + body template
  2. skills/<name>/scripts/<name>.mjs    deterministic-code stub
  3. skills/<name>/routing-eval.jsonl    routing fixture seed
  4. test/<name>.test.ts                 vitest skeleton
  5. Appended trigger row to the detected resolver file (RESOLVER.md
     or AGENTS.md — whatever W1's auto-detect found)

Flags: --description (required), --triggers, --writes-to,
--writes-pages, --mutating, --force, --dry-run, --json, --skills-dir.

Kebab-case name validation (`^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$`).
Works against gbrain-native RESOLVER.md layout AND OpenClaw-native
AGENTS.md-at-workspace-root layout (W1 interop).

## `gbrain skillify check [path]` (audit)

Promoted from scripts/skillify-check.ts per codex D-CX-2. The legacy
script stays as a 12-line shim that delegates to the new module so
existing callers (docs, cron, tests) keep working.

Wrapped in a subcommand namespace: `gbrain skillify {scaffold, check}`
is one coherent verb for the whole post-task loop. The essay's
"skillify it!" triggers the markdown skill, which orchestrates the
CLI primitives.

## Idempotency contract (D-CX-7)

`skillify scaffold --force` regenerates stub FILES but never re-appends
a resolver row that already references `skills/<name>/SKILL.md`.
Unit test pins this: two applies produce one resolver row, not two.

## D-CX-9 SKILLIFY_STUB sentinel

Every scaffolded script + SKILL.md body carries a SKILLIFY_STUB
sentinel. `check-resolvable` walks every skill's script dir looking
for the marker and emits a `skillify_stub_unreplaced` warning when
found. Default mode: advisory. `--strict` mode: error, blocks CI.

This is the gate that catches "we scaffolded and forgot to implement"
— the exact failure codex flagged as "scaffold verification is
theater" in the outside-voice review.

## Files

- NEW src/core/skillify/templates.ts (template strings)
- NEW src/core/skillify/generator.ts (planScaffold / applyScaffold +
  SkillifyScaffoldError with typed error codes)
- NEW src/commands/skillify.ts (top-level dispatcher + scaffold handler)
- NEW src/commands/skillify-check.ts (promoted check logic)
- scripts/skillify-check.ts: rewritten to 12-line shim
- skills/skillify/SKILL.md: Phase 2 now references the scaffold
  primitive; legacy manual path kept for extending existing skills
- src/cli.ts: `skillify` added to CLI_ONLY + dispatcher
- src/core/check-resolvable.ts: SKILLIFY_STUB sentinel scan + new
  issue type `skillify_stub_unreplaced`

## Tests (14 new scaffold cases)

test/skillify-scaffold.test.ts covers:
  - SKILL_NAME_PATTERN validation (kebab-case, no spaces, no
    leading digit, no underscores/uppercase)
  - planScaffold against fresh + existing-file + --force paths
  - SKILLIFY_STUB sentinel presence in SKILL.md AND script stub
    (both gate paths)
  - D-CX-7 idempotency: resolverAppend null when row pre-exists,
    second apply doesn't duplicate the row
  - TBD-trigger placeholder when --triggers empty
  - writes_pages / writes_to / mutating flow through to frontmatter
  - applyScaffold writes files + appends resolver
  - Full AGENTS.md-layout workspace interop (W1)

Existing test/skillify-check.test.ts still passes against the legacy
shim — zero regression for downstream consumers.

178/178 passing across v0.17 foundation + W1..W4.

## Live smoke

\`gbrain skillify scaffold webhook-verify --description "verify incoming
webhook signatures" --triggers "verify webhook,check tunnel"
--skills-dir /tmp/smoke --dry-run\` produces the expected 4-file plan
plus a 115-byte resolver append. \`--help\` works on both the top-level
and scaffold levels.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 W5 — gbrain skillpack install (deps closure + lockfile + diff/dry-run)

The essay's "drop it into YOUR OpenClaw" promise lands as a CLI
verb. One command installs a curated bundle of gbrain skills + the
shared convention files they depend on into a target OpenClaw
workspace. Data-loss protected, concurrency-safe, atomic on the
AGENTS.md managed block.

## openclaw.plugin.json refresh

- Bumped version from stale 0.4.1 → 0.17.0 (codex flagged this drift
  F-ENG-4 / D-CX-4).
- Expanded curated skill list from 7 → 25. Uses skills/manifest.json
  top-level (v0.10.0 sourced) minus setup/migrate/publish
  (install-time / code+skill pairs) minus private skills.
- Added \`shared_deps: [...]\` listing convention files every skill
  references: conventions/, _brain-filing-rules.{md,json},
  _output-rules.md. Installer always pulls these (D-CX-10
  dependency closure).
- Added \`excluded_from_install: [...]\` for setup/migrate/publish —
  surfaces the intentional exclusion as data rather than a comment.

## New module: src/core/skillpack/bundle.ts

- \`findGbrainRoot(start)\` — walks up looking for openclaw.plugin.json
  + src/cli.ts. The pair identifies a gbrain checkout.
- \`loadBundleManifest(root)\` — strict validation + typed BundleError
  codes (manifest_not_found, manifest_malformed, skill_not_found).
- \`enumerateBundle({gbrainRoot, skillSlug?, manifest})\` — flat list
  of source → target-relative paths. When skillSlug is set, scopes
  to that one skill BUT always pulls shared_deps. \`--all\` walks every
  skill in the manifest.
- \`bundledSkillSlugs(manifest)\` — sorted slugs for \`skillpack list\`.

## New module: src/core/skillpack/installer.ts

- \`planInstall(opts)\` — builds InstallPlan with per-file
  existing/identical diff state. Pure; no writes.
- \`applyInstall(plan, opts)\` — writes files + managed block with
  the contracts below.
- \`diffSkill(root, slug, skillsDir)\` — read-only per-file status
  for \`skillpack diff <name>\`.

**Per-file diff protection (D-CX-3 / F4):**
  wrote_new            fresh file
  wrote_overwrite      local diff + --overwrite-local passed
  skipped_identical    bytes match the bundle (silent re-install)
  skipped_locally_modified  target differs + no --overwrite-local
  → PROTECTED DEFAULT

**Concurrency + atomic AGENTS.md (D-CX-11):**
  - \`.gbrain-skillpack.lock\` at workspace root. Acquired on the
    first write, released in finally.
  - Lock stale threshold configurable (default 10min). --force-unlock
    overrides.
  - Managed-block writes via tmp-file-plus-rename (atomic on POSIX).

**Managed-block format:**
  <!-- gbrain:skillpack:begin -->
  <!-- Installed by gbrain <version> — do not hand-edit between markers. -->
  | Trigger | Skill |
  |---------|-------|
  | "alpha" | \`skills/alpha/SKILL.md\` |
  | ...
  <!-- gbrain:skillpack:end -->

  extractManagedSlugs() roundtrips: single-skill installs accumulate
  into the same block rather than overwriting each other.

## New CLI: gbrain skillpack {list, install, diff, check}

Namespaced alongside W4's \`gbrain skillify\`. Subcommands:
  list             bundle inventory (human + --json)
  install <name>   single skill + deps closure
  install --all    entire curated bundle
  diff <name>      per-file diff vs target; read-only
  check            delegates to the pre-existing skillpack-check
                   (same CLI just namespaced)

Flags on install: --overwrite-local, --force-unlock, --dry-run,
--json, --skills-dir, --workspace.

Exit codes: 0 clean, 1 files skipped (protected local edits),
2 setup error / lock held.

## Live smoke

\`gbrain skillpack list\`: 25 skills. \`skillpack install query --dry-run\`
against a fresh temp workspace: 12 files planned (SKILL.md,
routing-eval.jsonl, 7 convention files, 3 rule files, managed block
to AGENTS.md). All shared_deps flagged [shared].

## Tests (36 new cases)

test/skillpack-install.test.ts:
  - findGbrainRoot walks up, returns null when absent
  - loadBundleManifest validates + rejects malformed
  - enumerateBundle pulls shared_deps on single-skill scope (D-CX-10)
  - buildManagedBlock + updateManagedBlock: append when absent,
    in-place replace when present, extractManagedSlugs roundtrip
  - planInstall + applyInstall: fresh install, dry-run, idempotency
    (skipped_identical), local-edit protection, --overwrite-local,
    lock-held concurrency (D-CX-11), --force-unlock, atomic
    managed-block write, multi-skill accumulation in managed block,
    AGENTS.md-at-workspace-root interop (W1 cross-check)
  - diffSkill: missing, identical, differs

test/skillpack-sync-guard.test.ts (F-ENG-4):
  - both manifests exist
  - every skill in plugin.json exists on disk
  - every shared_dep exists on disk
  - plugin.json skills ⊂ skills/manifest.json
  - excluded skills aren't in the install list
  - plugin version ≥ 0.17 (kills the 0.4.1 stale drift)

204/204 passing across the v0.17 foundation + W1..W5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 guards — privacy scrub + OpenClaw-reference E2E + v0.16.4 regression

Three ship-blocker work items from the eng review + codex outside
voice round out v0.17:

## scripts/check-privacy.sh (CLAUDE.md:550 enforcement)

Greps for the banned OpenClaw fork name (case-insensitive) across
tracked files. Two modes:
  scripts/check-privacy.sh           scan working tree
  scripts/check-privacy.sh --staged  scan git-staged files (pre-commit)

Exit 1 on any finding outside the allow-list. Allow-list covers files
where the name is legitimately present: this script itself (defines
the rule), CLAUDE.md (the canonical rule text), llms-full.txt
(auto-generated from CLAUDE.md), the historical upgrade guide, and
test/integrations.test.ts (whose personal-info regex ENFORCES the
rule against recipes/).

Scrubbed existing leaks:
  - CHANGELOG.md:366 reference in a closes-# line → "from the
    OpenClaw reference deployment"
  - test/doctor-minions-check.test.ts:171 comment → "an OpenClaw
    host's cron script"
  - test/plugin-loader.test.ts fixture plugin name → "openclaw-ref"

## test/e2e/openclaw-reference-compat.test.ts (ship-blocker gate)

The test that proves v0.17 delivers on the headline claim. New
fixture at test/fixtures/openclaw-reference-minimal/ mimics the
reference OpenClaw deployment layout: AGENTS.md at workspace root,
skills/ below, no manifest.json. Four fixture skills
(signal-detector, query, brain-ops, context-now).

Every v0.17 surface gets exercised end-to-end:
  - autoDetectSkillsDir with $OPENCLAW_WORKSPACE (D-CX-4 priority)
  - loadOrDeriveManifest walks SKILL.md (F-ENG-1 auto-manifest)
  - checkResolvable accepts AGENTS.md at workspace root, all 4
    skills reachable via resolver rows, zero errors
  - Filing audit clean (brain-ops declares writes_pages+writes_to)
  - CLI subprocess via `--skills-dir` → exit 0
  - CLI subprocess via $OPENCLAW_WORKSPACE (no flag) → exit 0,
    correct skillsDir detection
  - skillpack install against the layout writes managed block into
    AGENTS.md at workspace root

This is THE ship-blocker test. If the W1 + W5 stack ever regresses
against an AGENTS.md-layout workspace, this fails first.

## test/regression-v0_16_4.test.ts (F-ENG-8)

Guards v0.17 against adding "surprise" warnings. Builds a clean
fixture matching v0.16.4 canonical shape (manifest.json, RESOLVER.md,
2 skills, no routing-eval fixtures, no writes_pages). Runs v0.17
checkResolvable and asserts:
  - zero errors, zero routing_*/filing_*/skillify_stub_* warnings
  - JSON envelope keys unchanged (errors, warnings, issues, ok,
    summary) — deprecated `issues[]` still equals errors ∪ warnings
  - summary shape unchanged

If someone adds a new check that fires unexpectedly on a v0.16.4-era
fixture, this test catches it immediately.

## Fixture

test/fixtures/openclaw-reference-minimal/
├── AGENTS.md                       (4 rows, 3 sections)
└── skills/
    ├── brain-ops/SKILL.md          (writes_pages+writes_to)
    ├── context-now/SKILL.md
    ├── query/SKILL.md
    └── signal-detector/SKILL.md

Intentionally small (4 skills, 1 AGENTS.md, ~30 lines total) so the
fixture is maintainable. The OPENCLAW-reference deployment has 107
skills — this fixture is the minimum shape that exercises the full
v0.17 code path.

## Tests

215/215 passing across the full v0.17 surface:
  - foundation + W1 + W2 + W3 + W4 + W5 (204)
  - regression-v0_16_4 (3)
  - openclaw-reference-compat (7)
  - privacy guard (separate bash; exits 0 clean)

Plus: privacy pre-commit hook is a drop-in wrapper (documented in
the script header). Wiring into .github/workflows is a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* release: v0.17.0 — skillify goes end-to-end

Every skill. Every check. Every install. One command each.

Five workstreams land in one release:
  - W1: AGENTS.md + auto-manifest + env-priority
  - W2: Check 5 routing eval
  - W3: Check 6 brain filing
  - W4: gbrain skillify {scaffold,check}
  - W5: gbrain skillpack {list,install,diff}

Plus D-CX-3 foundation (errors/warnings split + --strict), plus
codex outside-voice fixes (D-CX-1..12 applied), plus privacy pre-
commit guard, plus OpenClaw-reference E2E fixture, plus v0.16.4
regression guard.

Live against the reference OpenClaw deployment: 102 skills detected
via auto-manifest, 15 unreachable errors + 108 warnings surfaced —
exactly the essay's "~15% dark" finding. The magic word from the
essay finally works the way the essay describes.

Tests: 2156 unit (178 new) + 152 E2E Tier 1 + 3 Tier 2 + 8 new
openclaw-reference fixture cases. 0 failures across all tiers.
Plan + reviews: ~/.claude/plans/p1-lets-just-vast-blanket.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): add missing 'strict' field to 5 Flags literals in check-resolvable-cli.test.ts

CI failed `tsc --noEmit` after the D-CX-3 errors/warnings split added
`strict: boolean` as a required field on the `Flags` interface. Five
test sites in test/check-resolvable-cli.test.ts still construct
Flags object literals (for direct `resolveSkillsDir()` calls) and
hadn't been updated.

Added `strict: false` to all five literals:
  - line 129  --skills-dir absolute path
  - line 135  --skills-dir relative path
  - line 148  no --skills-dir
  - line 160  no --skills-dir + no env
  - line 178  --skills-dir + OPENCLAW_WORKSPACE (REGRESSION-GATE)

Unit tests: 207/207 pass across the v0.19 surface. tsc --noEmit
exits 0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: adopt gstack's branch-scoped CHANGELOG rule + rewrite v0.19.0 entry

CLAUDE.md gains a new top section before "CHANGELOG voice" that codifies
what gstack's CLAUDE.md already says: CHANGELOG is user-facing product
release notes, not a log of internal decisions. Every entry describes
what THIS branch adds vs master. Plan-file IDs, decision tags (D-CX-#,
F-ENG-#), review rounds, test counts as marketing, and contributor-
facing metrics don't belong in it.

The v0.19.0 entry is rewritten to the new bar:

Removed:
- Version-collision note about v0.17.0/v0.18.0 shipping on master
- All D-CX-## and W# tags (meaningless outside the plan file)
- "codex caught" / CEO + Eng review round-up narrative
- Plan file path reference
- "215 new cases across 13 test files" marketing metrics
- W1..W5 bucketing in itemized changes

Kept / sharpened:
- User-facing headline (what your agent can now do)
- Numbers that mean something to users (unreachable-skills count,
  scaffold timing, pre/post AGENTS.md support)
- Upgrade instructions
- Added/Changed/Fixed/For-contributors itemized sections (standard
  keep-a-changelog shape)

Version sequence (`grep "^## \["`) is contiguous v0.19.0 → v0.16.4.
Privacy guard clean. Tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update README/CLAUDE/TODOS for v0.19.0 skills + skillify loop

Skill count was stale (README said 26, actual is 28: skillify + skillpack-check
were missing from the tables and count). Corrected throughout. Marked TODOS item
"Checks 5 + 6 deferred in PR #325" as completed in v0.19 — they shipped as real
implementations, not just filed issues.

README:
- Skill count 26 → 28 (headline, install flow, table section, architecture diagram)
- Added `skillify` + `skillpack-check` rows to the operational skills table
- Rewrote the "Skillify" section to lead with the four v0.19 CLI verbs
  (`gbrain skillify scaffold/check`, `gbrain skillpack list/install/diff`,
  `gbrain routing-eval`, `gbrain check-resolvable --strict`) instead of
  describing the pre-v0.19 state. Added the "works on your OpenClaw" pitch
  around AGENTS.md + auto-manifest. Added the "drop 25 curated skills into
  your OpenClaw" section for skillpack install.
- Added v0.19 skills block + v0.18 multi-source + v0.17 dream to the Commands
  reference at the bottom.
- Standalone instruction sets count: 25 → 28 (with a parenthetical noting
  the curated 25-skill bundle that `skillpack install` ships).

CLAUDE.md:
- Skill count 26 → 28 in the Skills section.
- New "Skillify loop (v0.19)" sub-bullet listing skillify + skillpack-check.
- Noted that `AGENTS.md` is also accepted as a resolver filename.

TODOS.md:
- Created "## Completed" section at the top.
- Moved the "Checks 5 + 6" item there with completion note linking to the
  actual implementation files (routing-eval.ts + filing-audit.ts).

Privacy scan clean. Version sequence contiguous v0.19.0 → v0.16.4.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): regenerate llms-full.txt + llms.txt after README/CLAUDE edits

CI failed on `build-llms generator > committed llms.txt + llms-full.txt
match current generator output`. The drift was expected: the prior
commit edited README.md and CLAUDE.md (skill count + skillify section),
both of which are inlined into llms-full.txt by `scripts/build-llms.ts`.

Fix: `bun run build:llms` + commit the regenerated output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-23 19:34:27 -07:00
committed by GitHub
parent 08b3698e90
commit 78ba0b5b53
56 changed files with 6595 additions and 548 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
test/check-resolvable-cli.test.ts
View File

@@ -126,15 +126,16 @@ describe('check-resolvable — unit: parseFlags', () => {
describe('check-resolvable — unit: resolveSkillsDir', () => {
it('resolves absolute --skills-dir unchanged', () => {
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, skillsDir: '/tmp/absolute-path' });
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, strict: false, skillsDir: '/tmp/absolute-path' });
expect(r.dir).toBe('/tmp/absolute-path');
expect(r.error).toBeNull();
});
it('resolves relative --skills-dir against cwd', () => {
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, skillsDir: 'skills' });
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, strict: false, skillsDir: 'skills' });
expect(r.dir).toMatch(/\/skills$/);
expect(r.error).toBeNull();
expect(r.source).toBe('explicit');
});
it('REGRESSION-GATE: returns no_skills_dir error when no --skills-dir and findRepoRoot fails', () => {
@@ -144,7 +145,7 @@ describe('check-resolvable — unit: resolveSkillsDir', () => {
const original = process.cwd();
try {
process.chdir(empty);
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, skillsDir: null });
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, strict: false, skillsDir: null });
expect(r.error).toBe('no_skills_dir');
expect(r.dir).toBeNull();
expect(typeof r.message).toBe('string');
@@ -156,19 +157,51 @@ describe('check-resolvable — unit: resolveSkillsDir', () => {
it('finds skills via findRepoRoot when cwd is inside a repo (no --skills-dir)', () => {
// Running from this test file — we're inside the real gbrain repo.
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, skillsDir: null });
const r = resolveSkillsDir({ help: false, json: false, fix: false, dryRun: false, verbose: false, strict: false, skillsDir: null });
expect(r.error).toBeNull();
expect(r.dir).toMatch(/\/skills$/);
expect(r.source).toBe('repo_root');
});
it('REGRESSION-GATE: --skills-dir override takes precedence over OpenClaw env auto-detection', () => {
const explicit = mkdtempSync(join(tmpdir(), 'explicit-skills-'));
mkdirSync(explicit, { recursive: true });
writeFileSync(join(explicit, 'RESOLVER.md'), '# RESOLVER\n');
const workspace = mkdtempSync(join(tmpdir(), 'openclaw-ws-'));
mkdirSync(join(workspace, 'skills'), { recursive: true });
writeFileSync(join(workspace, 'skills', 'RESOLVER.md'), '# RESOLVER\n');
const prev = process.env.OPENCLAW_WORKSPACE;
process.env.OPENCLAW_WORKSPACE = workspace;
try {
const r = resolveSkillsDir({
help: false,
json: false,
fix: false,
dryRun: false,
verbose: false,
strict: false,
skillsDir: explicit,
});
expect(r.error).toBeNull();
expect(r.dir).toBe(explicit);
expect(r.source).toBe('explicit');
} finally {
if (prev === undefined) delete process.env.OPENCLAW_WORKSPACE;
else process.env.OPENCLAW_WORKSPACE = prev;
rmSync(explicit, { recursive: true, force: true });
rmSync(workspace, { recursive: true, force: true });
}
});
});
describe('check-resolvable — unit: DEFERRED', () => {
it('exports two deferred check entries for Checks 5 and 6', () => {
expect(DEFERRED.length).toBe(2);
expect(DEFERRED[0].check).toBe(5);
expect(DEFERRED[0].name).toBe('trigger_routing_eval');
expect(DEFERRED[1].check).toBe(6);
expect(DEFERRED[1].name).toBe('brain_filing');
it('v0.17 ships Checks 5 and 6 — DEFERRED is empty', () => {
// Pre-v0.17: both Check 5 (routing eval) and Check 6 (brain filing)
// were deferred. v0.17 W2 shipped Check 5; v0.17 W3 shipped Check 6.
// The DEFERRED export stays (stable --json field) for future checks.
expect(DEFERRED.length).toBe(0);
});
});
@@ -191,6 +224,9 @@ describe('gbrain check-resolvable CLI — integration', () => {
expect(r.stdout).toContain('gbrain check-resolvable');
expect(r.stdout).toContain('--json');
expect(r.stdout).toContain('--fix');
expect(r.stdout).toContain('--strict');
// Check 5 shipped in v0.17 (mentioned in the body); Check 6 is
// still deferred and must appear under "Deferred to separate issues".
expect(r.stdout).toContain('Check 5');
expect(r.stdout).toContain('Check 6');
});
@@ -202,9 +238,10 @@ describe('gbrain check-resolvable CLI — integration', () => {
const keys = Object.keys(r.json).sort();
expect(keys).toEqual(['autoFix', 'deferred', 'error', 'message', 'ok', 'report', 'skillsDir']);
expect(r.json.ok).toBe(true);
expect(r.json.deferred.length).toBe(2);
expect(r.json.deferred[0].check).toBe(5);
expect(r.json.deferred[1].check).toBe(6);
// v0.17 ships Checks 5 and 6; DEFERRED is empty. The key remains
// stable for future checks.
expect(Array.isArray(r.json.deferred)).toBe(true);
expect(r.json.deferred.length).toBe(0);
});
it('--json success: autoFix is null when --fix was not passed', () => {
@@ -220,22 +257,54 @@ describe('gbrain check-resolvable CLI — integration', () => {
expect(r.stdout).toContain('resolver_health: OK');
});
it('REGRESSION-GATE: exits 1 when fixture has a warning-level orphan_trigger only', () => {
it('D-CX-3: warnings-only fixture exits 0 in default mode', () => {
// "alpha" is in resolver but not manifest → orphan_trigger (warning)
// Per D-CX-3 (codex review outside voice): warnings alone do not flip
// the exit code. This is the new contract; callers who want strict
// behavior pass --strict. Prior contract (exit 1 on any warning) broke
// CI for workspaces emitting warning-level advisories like filing-audit.
const skillsDir = makeFixture(
[{ name: 'alpha', triggers: ['alpha'], inManifest: false }],
created,
);
const r = run(['--json', '--skills-dir', skillsDir]);
expect(r.json).not.toBeNull();
const warnings = r.json.report.issues.filter((i: any) => i.severity === 'warning');
const errors = r.json.report.issues.filter((i: any) => i.severity === 'error');
expect(warnings.length).toBeGreaterThan(0);
expect(errors.length).toBe(0);
// Doctor's ok=true-on-warnings-only would exit 0. check-resolvable MUST exit 1.
expect(r.json.report.warnings.length).toBeGreaterThan(0);
expect(r.json.report.errors.length).toBe(0);
// warnings.length > 0 but errors.length === 0 → exit 0 (advisory)
expect(r.status).toBe(0);
expect(r.json.ok).toBe(true);
});
it('D-CX-3: --strict promotes warnings to exit 1', () => {
const skillsDir = makeFixture(
[{ name: 'alpha', triggers: ['alpha'], inManifest: false }],
created,
);
const r = run(['--json', '--strict', '--skills-dir', skillsDir]);
expect(r.json).not.toBeNull();
expect(r.json.report.warnings.length).toBeGreaterThan(0);
expect(r.json.report.errors.length).toBe(0);
// --strict flips ok to false and exit to 1 when warnings exist
expect(r.json.ok).toBe(false);
expect(r.status).toBe(1);
});
it('D-CX-3: report has separate errors[] and warnings[] arrays alongside issues[]', () => {
const skillsDir = makeFixture(
[{ name: 'alpha', triggers: ['alpha'], inManifest: false }],
created,
);
const r = run(['--json', '--skills-dir', skillsDir]);
expect(r.json).not.toBeNull();
const rep = r.json.report;
expect(Array.isArray(rep.errors)).toBe(true);
expect(Array.isArray(rep.warnings)).toBe(true);
// Deprecated flat `issues` still present for one-release backcompat
expect(Array.isArray(rep.issues)).toBe(true);
expect(rep.issues.length).toBe(rep.errors.length + rep.warnings.length);
});
it('exits 1 when fixture has an error-level unreachable skill', () => {
// "alpha" is in manifest but not resolver → unreachable (error)
const skillsDir = makeFixture(
@@ -261,9 +330,12 @@ describe('gbrain check-resolvable CLI — integration', () => {
it('--verbose prints the deferred checks note in human mode', () => {
const skillsDir = makeFixture([{ name: 'alpha', triggers: ['alpha'] }], created);
const r = run(['--verbose', '--skills-dir', skillsDir]);
// v0.17 ships Checks 5 and 6 → DEFERRED is empty. Verbose still
// prints the "Deferred:" header for stable UX; downstream content
// is empty until a future release adds a new deferred check.
expect(r.stdout).toContain('Deferred:');
expect(r.stdout).toContain('trigger_routing_eval');
expect(r.stdout).toContain('brain_filing');
expect(r.stdout).not.toContain('trigger_routing_eval');
expect(r.stdout).not.toContain('brain_filing');
});
it('clean fixture human output says all skills reachable', () => {
@@ -279,4 +351,11 @@ describe('gbrain check-resolvable CLI — integration', () => {
expect(r.stdout).toContain('2 skills');
expect(r.status).toBe(0);
});
it('logs the auto-detected skills directory path in human mode', () => {
const r = run([]);
expect(r.status === 0 || r.status === 1).toBe(true);
expect(r.stdout).toContain('Auto-detected skills directory');
expect(r.stdout).toContain('/skills');
});
});

3
test/doctor-minions-check.test.ts
View File

@@ -168,7 +168,8 @@ describe('gbrain doctor — half-migrated Minions detection', () => {
test('human output: prints MINIONS HALF-INSTALLED loud banner', () => {
// Same fixture as the first test, but check the human-readable output
// includes the exact banner phrase Wintermute's cron script can grep for.
// includes the exact banner phrase an OpenClaw host's cron script
// can grep for.
const migrationsDir = join(tmp, '.gbrain', 'migrations');
mkdirSync(migrationsDir, { recursive: true });
writeFileSync(

150
test/e2e/openclaw-reference-compat.test.ts Normal file
View File

@@ -0,0 +1,150 @@
/**
* test/e2e/openclaw-reference-compat.test.ts — W1 ship-blocker gate.
*
* This is THE test that proves v0.17 delivers on its headline claim:
* `gbrain check-resolvable` against an OpenClaw-reference workspace
* layout (AGENTS.md at workspace root, skills/ below, no manifest.json)
* runs cleanly and surfaces sensible issues.
*
* Fixture: `test/fixtures/openclaw-reference-minimal/` ships 4 skills
* plus an AGENTS.md with a resolver table. Every test here exercises
* the full W1 + W2 + W3 + W4 + W5 stack against that fixture.
*/
import { describe, expect, it, afterEach } from 'bun:test';
import { existsSync, mkdtempSync, rmSync, writeFileSync, mkdirSync, readFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { spawnSync } from 'child_process';
import { checkResolvable } from '../../src/core/check-resolvable.ts';
import { autoDetectSkillsDir } from '../../src/core/repo-root.ts';
import { loadOrDeriveManifest } from '../../src/core/skill-manifest.ts';
import {
applyInstall,
planInstall,
} from '../../src/core/skillpack/installer.ts';
import { findGbrainRoot } from '../../src/core/skillpack/bundle.ts';
const FIXTURE = join(import.meta.dir, '..', 'fixtures', 'openclaw-reference-minimal');
const SKILLS_DIR = join(FIXTURE, 'skills');
const REPO = join(import.meta.dir, '..', '..');
const CLI = join(REPO, 'src', 'cli.ts');
const created: string[] = [];
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
describe('OpenClaw reference workspace compat (W1 + W2 + W3)', () => {
it('fixture exists at the expected path', () => {
expect(existsSync(FIXTURE)).toBe(true);
expect(existsSync(join(FIXTURE, 'AGENTS.md'))).toBe(true);
expect(existsSync(SKILLS_DIR)).toBe(true);
expect(existsSync(join(FIXTURE, 'skills', 'manifest.json'))).toBe(false);
});
it('auto-detects skills dir via $OPENCLAW_WORKSPACE (D-CX-4 priority)', () => {
// Priority: explicit env wins over repo-root walk. Without the
// env var, we'd get gbrain's own repo. With it set, we should
// get the fixture's skills dir.
const detected = autoDetectSkillsDir(process.cwd(), { OPENCLAW_WORKSPACE: FIXTURE });
expect(detected.dir).toBe(SKILLS_DIR);
// AGENTS.md is at the workspace root, not inside skills/, so the
// source should be the workspace-root variant.
expect(detected.source).toBe('openclaw_workspace_env_root');
});
it('auto-derives manifest from SKILL.md walk (F-ENG-1)', () => {
const result = loadOrDeriveManifest(SKILLS_DIR);
expect(result.derived).toBe(true);
expect(result.skills.map(s => s.name).sort()).toEqual([
'brain-ops',
'context-now',
'query',
'signal-detector',
]);
});
it('checkResolvable accepts AGENTS.md at workspace root and runs all checks', () => {
const report = checkResolvable(SKILLS_DIR);
// Top-level ok is errors-only (D-CX-3) — no unreachable/missing-file errors.
expect(report.ok).toBe(true);
expect(report.errors.length).toBe(0);
// All 4 skills should be reachable via AGENTS.md rows.
expect(report.summary.total_skills).toBe(4);
expect(report.summary.reachable).toBe(4);
expect(report.summary.unreachable).toBe(0);
});
it('brain-ops declares writes_pages+writes_to — filing audit clean', () => {
const report = checkResolvable(SKILLS_DIR);
const filing = report.warnings.filter(w =>
w.type === 'filing_missing_writes_to' || w.type === 'filing_unknown_directory',
);
expect(filing).toEqual([]);
});
it('CLI subprocess: gbrain check-resolvable --json --skills-dir FIXTURE clean', () => {
const r = spawnSync(
'bun',
[CLI, 'check-resolvable', '--json', '--skills-dir', SKILLS_DIR],
{ encoding: 'utf-8', cwd: REPO, maxBuffer: 10 * 1024 * 1024 },
);
expect(r.status).toBe(0);
const env = JSON.parse(r.stdout);
expect(env.ok).toBe(true);
expect(env.report.errors).toEqual([]);
expect(env.report.summary.total_skills).toBe(4);
});
it('CLI subprocess: $OPENCLAW_WORKSPACE auto-detect without --skills-dir', () => {
const r = spawnSync('bun', [CLI, 'check-resolvable', '--json'], {
encoding: 'utf-8',
cwd: REPO,
env: { ...process.env, OPENCLAW_WORKSPACE: FIXTURE },
maxBuffer: 10 * 1024 * 1024,
});
expect(r.status).toBe(0);
const env = JSON.parse(r.stdout);
expect(env.skillsDir).toBe(SKILLS_DIR);
expect(env.ok).toBe(true);
});
it('skillpack install against OpenClaw-reference layout writes managed block to AGENTS.md', () => {
// Copy fixture into a tmp workspace so we can install without
// polluting the fixture itself.
const target = mkdtempSync(join(tmpdir(), 'openclaw-ref-install-'));
created.push(target);
mkdirSync(join(target, 'skills'), { recursive: true });
// Just the AGENTS.md shell — no skills yet, install writes them.
writeFileSync(
join(target, 'AGENTS.md'),
'# AGENTS\n\n| Trigger | Skill |\n|---------|-------|\n',
);
const gbrainRoot = findGbrainRoot();
expect(gbrainRoot).not.toBeNull();
const opts = {
gbrainRoot: gbrainRoot!,
targetWorkspace: target,
targetSkillsDir: join(target, 'skills'),
skillSlug: 'brain-ops',
};
const plan = planInstall(opts);
const result = applyInstall(plan, opts);
expect(result.summary.wroteNew).toBeGreaterThan(0);
expect(result.managedBlock.applied).toBe(true);
expect(result.managedBlock.resolverFile).toBe(join(target, 'AGENTS.md'));
const agents = readFileSync(join(target, 'AGENTS.md'), 'utf-8');
expect(agents).toContain('gbrain:skillpack:begin');
expect(agents).toContain('`skills/brain-ops/SKILL.md`');
// Pre-existing resolver table preserved.
expect(agents).toContain('| Trigger | Skill |');
});
});

251
test/filing-audit.test.ts Normal file
View File

@@ -0,0 +1,251 @@
/**
* Tests for src/core/filing-audit.ts — Check 6 (W3, v0.17).
*/
import { describe, expect, it, afterEach } from 'bun:test';
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
loadFilingRules,
allowedDirectories,
runFilingAudit,
} from '../src/core/filing-audit.ts';
const created: string[] = [];
function scratch(): string {
const dir = mkdtempSync(join(tmpdir(), 'filing-audit-'));
created.push(dir);
return dir;
}
function writeRules(skillsDir: string, body?: object): void {
const doc = body ?? {
version: '1.0.0',
rules: [
{ kind: 'person', directory: 'people/' },
{ kind: 'company', directory: 'companies/' },
],
sources_dir: { directory: 'sources/', purpose: 'raw data only' },
};
writeFileSync(join(skillsDir, '_brain-filing-rules.json'), JSON.stringify(doc, null, 2));
}
function writeSkill(
skillsDir: string,
name: string,
opts: {
writes_pages?: boolean;
writes_to?: string[];
writes_to_inline?: boolean;
mutating?: boolean;
no_frontmatter?: boolean;
} = {},
): void {
const dir = join(skillsDir, name);
mkdirSync(dir, { recursive: true });
if (opts.no_frontmatter) {
writeFileSync(join(dir, 'SKILL.md'), `# ${name}\nNo frontmatter.\n`);
return;
}
const lines = ['---', `name: ${name}`];
if (opts.mutating !== undefined) lines.push(`mutating: ${opts.mutating}`);
if (opts.writes_pages !== undefined) lines.push(`writes_pages: ${opts.writes_pages}`);
if (opts.writes_to) {
if (opts.writes_to_inline) {
lines.push(`writes_to: [${opts.writes_to.map(s => `"${s}"`).join(', ')}]`);
} else {
lines.push('writes_to:');
for (const d of opts.writes_to) lines.push(` - ${d}`);
}
}
lines.push('---');
lines.push(`# ${name}\n`);
writeFileSync(join(dir, 'SKILL.md'), lines.join('\n'));
}
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
describe('loadFilingRules', () => {
it('returns null when rules file is missing', () => {
const dir = scratch();
expect(loadFilingRules(dir)).toBeNull();
});
it('parses a valid rules document', () => {
const dir = scratch();
writeRules(dir);
const rules = loadFilingRules(dir);
expect(rules).not.toBeNull();
expect(rules!.rules.length).toBe(2);
expect(rules!.rules[0].kind).toBe('person');
});
it('throws on malformed JSON', () => {
const dir = scratch();
writeFileSync(join(dir, '_brain-filing-rules.json'), '{ not valid');
expect(() => loadFilingRules(dir)).toThrow();
});
it('throws when rules is not an array', () => {
const dir = scratch();
writeFileSync(
join(dir, '_brain-filing-rules.json'),
JSON.stringify({ version: '1', rules: 'oops' }),
);
expect(() => loadFilingRules(dir)).toThrow();
});
});
describe('allowedDirectories', () => {
it('normalizes directory strings (trailing slash)', () => {
const allowed = allowedDirectories({
version: '1',
rules: [
{ kind: 'a', directory: 'people/' },
{ kind: 'b', directory: 'companies' }, // no slash
],
sources_dir: { directory: '/sources', purpose: 'raw' },
});
expect(allowed.has('people/')).toBe(true);
expect(allowed.has('companies/')).toBe(true);
expect(allowed.has('sources/')).toBe(true);
});
});
describe('runFilingAudit', () => {
it('returns empty report when rules file is missing (no-op)', () => {
const dir = scratch();
// No _brain-filing-rules.json.
writeSkill(dir, 'enrich', { writes_pages: true, writes_to: ['people/'] });
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
expect(r.totalScanned).toBe(0);
});
it('clean: declares writes_pages + valid writes_to', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'enrich', {
writes_pages: true,
writes_to: ['people/', 'companies/'],
});
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
expect(r.writesPagesSkills).toBe(1);
});
it('flags missing writes_to when writes_pages is true', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'enrich', { writes_pages: true });
const r = runFilingAudit(dir);
expect(r.issues.length).toBe(1);
expect(r.issues[0].type).toBe('filing_missing_writes_to');
expect(r.issues[0].severity).toBe('warning');
});
it('flags unknown directory in writes_to', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'enrich', {
writes_pages: true,
writes_to: ['people/', 'junk/'],
});
const r = runFilingAudit(dir);
expect(r.issues.length).toBe(1);
expect(r.issues[0].type).toBe('filing_unknown_directory');
expect(r.issues[0].directory).toBe('junk/');
});
it('D-CX-7: mutating:true alone does NOT trigger filing audit', () => {
// Cron/scheduler/report skills use mutating:true but don't write
// brain pages. Filing-audit must skip them entirely.
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'cron-scheduler', { mutating: true });
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
expect(r.writesPagesSkills).toBe(0);
});
it('skips skills with writes_pages: false', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'query', { writes_pages: false });
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
});
it('skips skills with no frontmatter', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'bare', { no_frontmatter: true });
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
});
it('parses inline writes_to: [a, b] syntax', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'enrich', {
writes_pages: true,
writes_to: ['people/', 'companies/'],
writes_to_inline: true,
});
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
});
it('allows sources/ (raw data dir)', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'importer', {
writes_pages: true,
writes_to: ['sources/'],
});
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
});
it('skips underscore and dot dirs', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, '_conventions', {
writes_pages: true,
writes_to: ['not-real/'],
});
const r = runFilingAudit(dir);
expect(r.issues).toEqual([]);
});
it('totalScanned counts every skill with SKILL.md', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'a');
writeSkill(dir, 'b', { writes_pages: true, writes_to: ['people/'] });
writeSkill(dir, 'c', { writes_pages: false });
const r = runFilingAudit(dir);
expect(r.totalScanned).toBe(3);
expect(r.writesPagesSkills).toBe(1);
});
it('handles missing skillsDir cleanly', () => {
const r = runFilingAudit('/tmp/never-exists-filing-audit-ABC');
expect(r.issues).toEqual([]);
expect(r.totalScanned).toBe(0);
});
it('action string names the exact file to edit (test coverage guard)', () => {
const dir = scratch();
writeRules(dir);
writeSkill(dir, 'enrich', { writes_pages: true, writes_to: ['bad/'] });
const r = runFilingAudit(dir);
expect(r.issues[0].action).toContain('SKILL.md');
expect(r.issues[0].action.length).toBeGreaterThan(10);
});
});

25
test/fixtures/openclaw-reference-minimal/AGENTS.md vendored Normal file
View File

@@ -0,0 +1,25 @@
# AGENTS.md
Minimal fixture mimicking the OpenClaw reference deployment layout
(for W1 compat testing). AGENTS.md lives at workspace root; skills
live under `skills/`. No manifest.json (the auto-derive path in
`src/core/skill-manifest.ts` handles this).
## Gate 0 — access control
| Trigger | Skill |
|---------|-------|
| Every inbound message | `skills/signal-detector/SKILL.md` |
## Brain operations
| Trigger | Skill |
|---------|-------|
| "what do we know about", "search for", "lookup" | `skills/query/SKILL.md` |
| any brain read/write/lookup/citation | `skills/brain-ops/SKILL.md` |
## Calendar
| Trigger | Skill |
|---------|-------|
| "am I late", "how long until", "what time is" | `skills/context-now/SKILL.md` |

14
test/fixtures/openclaw-reference-minimal/skills/brain-ops/SKILL.md vendored Normal file
View File

@@ -0,0 +1,14 @@
---
name: brain-ops
description: Core read/write cycle for the OpenClaw reference fixture.
triggers:
- any brain read/write/lookup/citation
writes_pages: true
writes_to:
- people/
- companies/
---
# brain-ops
Fixture skill for `test/e2e/openclaw-reference-compat.test.ts`.

12
test/fixtures/openclaw-reference-minimal/skills/context-now/SKILL.md vendored Normal file
View File

@@ -0,0 +1,12 @@
---
name: context-now
description: "ALWAYS-ON time-sensitivity discipline for the OpenClaw reference fixture."
triggers:
- "am I late"
- "how long until"
- "what time is"
---
# context-now
Fixture skill for `test/e2e/openclaw-reference-compat.test.ts`.

12
test/fixtures/openclaw-reference-minimal/skills/query/SKILL.md vendored Normal file
View File

@@ -0,0 +1,12 @@
---
name: query
description: Look up brain pages in the OpenClaw reference fixture.
triggers:
- "what do we know about"
- "search for"
- "lookup"
---
# query
Fixture skill for `test/e2e/openclaw-reference-compat.test.ts`.

10
test/fixtures/openclaw-reference-minimal/skills/signal-detector/SKILL.md vendored Normal file
View File

@@ -0,0 +1,10 @@
---
name: signal-detector
description: Always-on ambient signal capture for the OpenClaw reference fixture.
triggers:
- every inbound message
---
# signal-detector
Fixture skill for `test/e2e/openclaw-reference-compat.test.ts`.

4
test/plugin-loader.test.ts
View File

@@ -89,7 +89,7 @@ describe('path policy', () => {
describe('loadSinglePlugin', () => {
test('loads a minimal manifest + one subagent def', () => {
const dir = writePlugin('wintermute', {
const dir = writePlugin('openclaw-ref', {
subagents: {
'meeting-ingestion.md': `---\nname: meeting-ingestion\nmodel: sonnet\n---\n\nYou are a meeting ingester.\n`,
},
@@ -97,7 +97,7 @@ describe('loadSinglePlugin', () => {
const res = loadSinglePlugin(dir);
expect('error' in res).toBe(false);
if ('error' in res) return;
expect(res.manifest.name).toBe('wintermute');
expect(res.manifest.name).toBe('openclaw-ref');
expect(res.subagents.length).toBe(1);
expect(res.subagents[0]!.name).toBe('meeting-ingestion');
expect(res.subagents[0]!.body.trim()).toBe('You are a meeting ingester.');

125
test/regression-v0_16_4.test.ts Normal file
View File

@@ -0,0 +1,125 @@
/**
* test/regression-v0_16_4.test.ts — F-ENG-8.
*
* Guards against v0.17 regressions: a clean fixture that passed cleanly
* on v0.16.4 check-resolvable must still pass cleanly on v0.17 — same
* errors[] and warnings[] shape, no new surprise findings.
*
* The fixture matches the canonical RESOLVER.md + manifest.json +
* skills/ shape that v0.16.4 test suites used.
*/
import { describe, expect, it, afterEach } from 'bun:test';
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { checkResolvable } from '../src/core/check-resolvable.ts';
const created: string[] = [];
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
function makeCleanFixture(): string {
const root = mkdtempSync(join(tmpdir(), 'v0_16_4-regression-'));
created.push(root);
const skillsDir = join(root, 'skills');
mkdirSync(skillsDir, { recursive: true });
// Two skills, both in manifest, both reachable via RESOLVER.md.
const manifest = {
name: 'test',
version: '0.16.4-fixture',
skills: [
{ name: 'query', path: 'query/SKILL.md' },
{ name: 'brain-ops', path: 'brain-ops/SKILL.md' },
],
};
writeFileSync(join(skillsDir, 'manifest.json'), JSON.stringify(manifest, null, 2));
mkdirSync(join(skillsDir, 'query'), { recursive: true });
writeFileSync(
join(skillsDir, 'query', 'SKILL.md'),
'---\nname: query\ndescription: lookup brain pages.\ntriggers:\n - "look up"\n---\n\n# query\n',
);
mkdirSync(join(skillsDir, 'brain-ops'), { recursive: true });
writeFileSync(
join(skillsDir, 'brain-ops', 'SKILL.md'),
'---\nname: brain-ops\ndescription: core read/write cycle.\ntriggers:\n - any brain read/write\n---\n\n# brain-ops\n',
);
writeFileSync(
join(skillsDir, 'RESOLVER.md'),
[
'# RESOLVER',
'',
'## Brain operations',
'',
'| Trigger | Skill |',
'|---------|-------|',
'| "look up" | `skills/query/SKILL.md` |',
'| any brain read/write | `skills/brain-ops/SKILL.md` |',
'',
].join('\n'),
);
return skillsDir;
}
describe('v0.16.4 regression guard (F-ENG-8)', () => {
it('clean fixture passes all checks on v0.17 (no errors, no surprise warnings)', () => {
const skillsDir = makeCleanFixture();
const report = checkResolvable(skillsDir);
expect(report.ok).toBe(true);
expect(report.errors).toEqual([]);
// v0.17 adds routing_*/filing_*/skillify_stub_* warning types.
// A clean v0.16.4-style fixture has NO routing-eval fixtures,
// NO writes_pages declarations, NO SKILLIFY_STUB markers — so
// none of the new warning types should fire.
const newTypeWarnings = report.warnings.filter(w =>
w.type.startsWith('routing_') ||
w.type.startsWith('filing_') ||
w.type === 'skillify_stub_unreplaced',
);
expect(newTypeWarnings).toEqual([]);
// Summary shape unchanged.
expect(report.summary.total_skills).toBe(2);
expect(report.summary.reachable).toBe(2);
expect(report.summary.unreachable).toBe(0);
});
it('JSON envelope shape is stable (keys unchanged from v0.16.4)', () => {
const skillsDir = makeCleanFixture();
const report = checkResolvable(skillsDir);
// Top-level keys the --json envelope promises.
expect(Object.keys(report).sort()).toEqual([
'errors',
'issues',
'ok',
'summary',
'warnings',
]);
// Summary keys — new `routing_*` totals are NOT added to summary
// (kept in the issues list only).
expect(Object.keys(report.summary).sort()).toEqual([
'gaps',
'overlaps',
'reachable',
'total_skills',
'unreachable',
]);
});
it('deprecated issues[] union equals errors[] + warnings[]', () => {
const skillsDir = makeCleanFixture();
const report = checkResolvable(skillsDir);
expect(report.issues.length).toBe(report.errors.length + report.warnings.length);
});
});

111
test/repo-root.test.ts
View File

@@ -2,7 +2,7 @@ import { describe, it, expect, afterEach } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { findRepoRoot } from '../src/core/repo-root.ts';
import { autoDetectSkillsDir, findRepoRoot } from '../src/core/repo-root.ts';
describe('findRepoRoot', () => {
const created: string[] = [];
@@ -22,6 +22,13 @@ describe('findRepoRoot', () => {
function seedRepo(dir: string): void {
mkdirSync(join(dir, 'skills'), { recursive: true });
writeFileSync(join(dir, 'skills', 'RESOLVER.md'), '# RESOLVER\n');
mkdirSync(join(dir, 'src'), { recursive: true });
writeFileSync(join(dir, 'src', 'cli.ts'), '// gbrain marker\n');
}
function seedSkillsDir(dir: string): void {
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, 'RESOLVER.md'), '# RESOLVER\n');
}
it('finds skills/RESOLVER.md in the passed startDir on first iteration', () => {
@@ -50,4 +57,106 @@ describe('findRepoRoot', () => {
// and CI runners. What matters is parity: no-arg === cwd-arg.
expect(findRepoRoot()).toBe(findRepoRoot(process.cwd()));
});
it('auto-detect: falls back to $OPENCLAW_WORKSPACE/skills when repo root is absent', () => {
const cwd = scratch();
const workspace = scratch();
seedSkillsDir(join(workspace, 'skills'));
const found = autoDetectSkillsDir(cwd, { OPENCLAW_WORKSPACE: workspace });
expect(found.dir).toBe(join(workspace, 'skills'));
expect(found.source).toBe('openclaw_workspace_env');
});
it('auto-detect: falls back to ~/.openclaw/workspace/skills when env var is not set', () => {
const cwd = scratch();
const home = scratch();
seedSkillsDir(join(home, '.openclaw', 'workspace', 'skills'));
const found = autoDetectSkillsDir(cwd, { HOME: home });
expect(found.dir).toBe(join(home, '.openclaw', 'workspace', 'skills'));
expect(found.source).toBe('openclaw_workspace_home');
});
it('auto-detect: falls back to ./skills as final candidate', () => {
const cwd = scratch();
seedSkillsDir(join(cwd, 'skills'));
const found = autoDetectSkillsDir(cwd, {});
expect(found.dir).toBe(join(cwd, 'skills'));
expect(found.source).toBe('cwd_skills');
});
it('D-CX-4: $OPENCLAW_WORKSPACE wins over repo-root walk when explicitly set', () => {
// Prior priority (shadow bug): walking up from cwd found gbrain's
// repo root first and silently ignored the env var. Post-D-CX-4:
// explicit env wins. Unset env → repo-root walk still wins
// (tested below).
const root = scratch();
seedRepo(root);
const workspace = scratch();
seedSkillsDir(join(workspace, 'skills'));
const nested = join(root, 'a', 'b');
mkdirSync(nested, { recursive: true });
const found = autoDetectSkillsDir(nested, { OPENCLAW_WORKSPACE: workspace });
expect(found.dir).toBe(join(workspace, 'skills'));
expect(found.source).toBe('openclaw_workspace_env');
});
it('D-CX-4: repo-root walk still wins when OPENCLAW_WORKSPACE is NOT set', () => {
const root = scratch();
seedRepo(root);
const nested = join(root, 'a', 'b');
mkdirSync(nested, { recursive: true });
const found = autoDetectSkillsDir(nested, {});
expect(found.dir).toBe(join(root, 'skills'));
expect(found.source).toBe('repo_root');
});
it('W1: AGENTS.md at skills dir is accepted (OpenClaw skills-subdir variant)', () => {
const workspace = scratch();
const skillsDir = join(workspace, 'skills');
mkdirSync(skillsDir, { recursive: true });
// seed AGENTS.md (no RESOLVER.md)
require('fs').writeFileSync(
join(skillsDir, 'AGENTS.md'),
'# AGENTS\n\n| Trigger | Skill |\n|---|---|\n',
);
const found = autoDetectSkillsDir(scratch(), { OPENCLAW_WORKSPACE: workspace });
expect(found.dir).toBe(skillsDir);
expect(found.source).toBe('openclaw_workspace_env');
});
it('W1: AGENTS.md at workspace root (OpenClaw-native layout)', () => {
// The reference OpenClaw deployment places AGENTS.md at
// workspace/AGENTS.md, with skills in workspace/skills/. Auto-detect
// must find the skills dir and flag this as the workspace-root variant.
const workspace = scratch();
const skillsDir = join(workspace, 'skills');
mkdirSync(skillsDir, { recursive: true });
require('fs').writeFileSync(
join(workspace, 'AGENTS.md'),
'# AGENTS\n\n| Trigger | Skill |\n|---|---|\n',
);
const found = autoDetectSkillsDir(scratch(), { OPENCLAW_WORKSPACE: workspace });
expect(found.dir).toBe(skillsDir);
expect(found.source).toBe('openclaw_workspace_env_root');
});
it('W1: both RESOLVER.md and AGENTS.md present — RESOLVER.md wins', () => {
// Policy: when both exist at the same location, gbrain-native wins.
const workspace = scratch();
const skillsDir = join(workspace, 'skills');
mkdirSync(skillsDir, { recursive: true });
require('fs').writeFileSync(
join(skillsDir, 'RESOLVER.md'),
'# RESOLVER\n\n| Trigger | Skill |\n|---|---|\n',
);
require('fs').writeFileSync(
join(skillsDir, 'AGENTS.md'),
'# AGENTS\n\n| Trigger | Skill |\n|---|---|\n',
);
const found = autoDetectSkillsDir(scratch(), { OPENCLAW_WORKSPACE: workspace });
expect(found.dir).toBe(skillsDir);
// Source is still `env` — the distinction is which file was found,
// and RESOLVER.md takes precedence inside resolveWorkspaceSkillsDir.
expect(found.source).toBe('openclaw_workspace_env');
});
});

372
test/routing-eval.test.ts Normal file
View File

@@ -0,0 +1,372 @@
/**
* Tests for src/core/routing-eval.ts — Check 5 harness.
*
* Covers: normalization, trigger extraction, structural match, negative
* cases, ambiguity detection + allow-list, fixture linter (D-CX-6),
* fixture loader, and the end-to-end runner.
*/
import { describe, expect, it, afterEach } from 'bun:test';
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
extractTriggerPhrases,
indexResolverTriggers,
lintRoutingFixtures,
loadRoutingFixtures,
normalizeText,
runRoutingEval,
structuralRouteMatch,
type RoutingFixture,
} from '../src/core/routing-eval.ts';
const created: string[] = [];
function scratch(): string {
const dir = mkdtempSync(join(tmpdir(), 'routing-eval-'));
created.push(dir);
return dir;
}
function makeResolver(
rows: { trigger: string; skill: string; section?: string }[],
): string {
const bySection = new Map<string, typeof rows>();
for (const row of rows) {
const sec = row.section ?? 'Brain operations';
const list = bySection.get(sec) ?? [];
list.push(row);
bySection.set(sec, list);
}
const parts: string[] = ['# RESOLVER', ''];
for (const [sec, list] of bySection) {
parts.push(`## ${sec}`, '', '| Trigger | Skill |', '|---------|-------|');
for (const r of list) {
parts.push(`| ${r.trigger} | \`skills/${r.skill}/SKILL.md\` |`);
}
parts.push('');
}
return parts.join('\n');
}
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
describe('normalizeText', () => {
it('lowercases and strips punctuation', () => {
expect(normalizeText("What's up?")).toBe('what s up');
});
it('collapses whitespace', () => {
expect(normalizeText(' foo bar\n\tbaz ')).toBe('foo bar baz');
});
it('handles unicode punctuation', () => {
expect(normalizeText('search — fast!')).toBe('search fast');
});
it('empty input → empty', () => {
expect(normalizeText('')).toBe('');
expect(normalizeText(' !! ')).toBe('');
});
});
describe('extractTriggerPhrases', () => {
it('splits comma-separated quoted phrases', () => {
const phrases = extractTriggerPhrases('"search for", "look up", "find me"');
expect(phrases).toEqual(['search for', 'look up', 'find me']);
});
it('returns single unquoted cell as one phrase', () => {
const phrases = extractTriggerPhrases('Creating/enriching a person page');
expect(phrases).toEqual(['creating enriching a person page']);
});
it('filters phrases shorter than 3 chars', () => {
// "x" too short; "hi" normalized to "hi" (2 chars, filtered)
const phrases = extractTriggerPhrases('"x", "hi", "wave"');
expect(phrases).toEqual(['wave']);
});
it('mixes quoted and non-quoted → quoted wins (split mode)', () => {
// When ANY quoted phrase is present, we treat the cell as a list.
const phrases = extractTriggerPhrases('things like "alpha" and "beta"');
expect(phrases).toEqual(['alpha', 'beta']);
});
});
describe('indexResolverTriggers', () => {
it('indexes skills and normalizes their trigger phrases', () => {
const resolver = makeResolver([
{ trigger: '"search for", "find me"', skill: 'query' },
{ trigger: 'Fix broken citations', skill: 'citation-fixer' },
]);
const idx = indexResolverTriggers(resolver);
expect(idx.skillPhrases.get('query')).toEqual(['search for', 'find me']);
expect(idx.skillPhrases.get('citation-fixer')).toEqual(['fix broken citations']);
});
it('accumulates phrases when one skill has multiple resolver rows', () => {
const resolver = makeResolver([
{ trigger: '"search", "lookup"', skill: 'query' },
{ trigger: '"graph query"', skill: 'query' },
]);
const idx = indexResolverTriggers(resolver);
expect(idx.skillPhrases.get('query')).toEqual(['search', 'lookup', 'graph query']);
});
it('skips GStack entries (external references)', () => {
const resolver = [
'# RESOLVER',
'',
'## External',
'| Trigger | Skill |',
'|---------|-------|',
'| "plan review" | GStack: plan-ceo-review |',
'| "do thing" | `skills/foo/SKILL.md` |',
'',
].join('\n');
const idx = indexResolverTriggers(resolver);
expect(Array.from(idx.skillPhrases.keys())).toEqual(['foo']);
});
});
describe('structuralRouteMatch', () => {
const index = indexResolverTriggers(
makeResolver([
{ trigger: '"search", "lookup", "find"', skill: 'query' },
{ trigger: '"fix citations", "broken sources"', skill: 'citation-fixer' },
{ trigger: '"every inbound message"', skill: 'signal-detector', section: 'Always-on' },
]),
);
it('matches a clean unambiguous intent', () => {
const r = structuralRouteMatch('please lookup that person', index);
expect(r.matched).toEqual(['query']);
expect(r.ambiguous).toBe(false);
});
it('returns empty match for out-of-scope intent', () => {
const r = structuralRouteMatch('deploy the app to prod', index);
expect(r.matched).toEqual([]);
expect(r.ambiguous).toBe(false);
});
it('flags ambiguity when two specific skills match', () => {
const r = structuralRouteMatch('find broken sources in notes', index);
expect(r.matched).toContain('query'); // "find"
expect(r.matched).toContain('citation-fixer'); // "broken sources"
expect(r.ambiguous).toBe(true);
});
it('does NOT flag ambiguity when always-on skill co-fires', () => {
// always-on skills are exempted from the ambiguity check.
const r = structuralRouteMatch('every inbound message lookup', index);
expect(r.matched).toContain('query');
expect(r.matched).toContain('signal-detector');
expect(r.ambiguous).toBe(false);
});
});
describe('lintRoutingFixtures (D-CX-6)', () => {
const resolver = makeResolver([
{ trigger: '"find me", "search for"', skill: 'query' },
{ trigger: 'Fix broken citations', skill: 'citation-fixer' },
]);
const index = indexResolverTriggers(resolver);
it('rejects fixture whose intent is verbatim-equal to a trigger', () => {
// Intent equals the trigger exactly (after normalization): pure
// tautology. These are the copy-paste fixtures D-CX-6 targets.
const fixtures: RoutingFixture[] = [
{ intent: 'find me', expected_skill: 'query' },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues.length).toBe(1);
expect(issues[0].reason).toBe('intent_copies_trigger');
});
it('accepts a natural-sentence fixture embedding trigger words', () => {
// Intent embeds the trigger in a natural sentence — this is exactly
// what Layer A's substring match is supposed to detect, so the
// linter must NOT flag it.
const fixtures: RoutingFixture[] = [
{ intent: 'please find me that paper', expected_skill: 'query' },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues).toEqual([]);
});
it('accepts a fully-paraphrased fixture (no trigger words at all)', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'pull up what we know about Paul', expected_skill: 'query' },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues).toEqual([]);
});
it('flags unknown expected_skill (typo / dead reference)', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'something', expected_skill: 'not-a-skill' },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues.length).toBe(1);
expect(issues[0].reason).toBe('unknown_expected_skill');
});
it('skips verbatim check for negative cases (expected_skill=null)', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'find broken citations', expected_skill: null },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues).toEqual([]);
});
it('flags invalid shape (missing intent, wrong type)', () => {
const fixtures: RoutingFixture[] = [
{ intent: '', expected_skill: 'query' },
{ intent: 'ok', expected_skill: 42 as unknown as string },
];
const issues = lintRoutingFixtures(fixtures, index);
expect(issues.length).toBe(2);
expect(issues[0].reason).toBe('invalid_shape');
expect(issues[1].reason).toBe('invalid_shape');
});
});
describe('loadRoutingFixtures', () => {
function seedFixture(skillsDir: string, name: string, lines: string[]): void {
const dir = join(skillsDir, name);
mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, 'SKILL.md'), `---\nname: ${name}\n---\n`);
writeFileSync(join(dir, 'routing-eval.jsonl'), lines.join('\n'));
}
it('walks skills/*/routing-eval.jsonl and parses JSON-per-line', () => {
const dir = scratch();
seedFixture(dir, 'query', [
'{"intent":"lookup paul","expected_skill":"query"}',
'{"intent":"find that doc","expected_skill":"query"}',
]);
seedFixture(dir, 'other', [
'{"intent":"clean citations","expected_skill":"other"}',
]);
const r = loadRoutingFixtures(dir);
expect(r.fixtures.length).toBe(3);
expect(r.malformed).toEqual([]);
expect(r.fixtures.every(f => f.source !== undefined)).toBe(true);
});
it('skips comments and blank lines', () => {
const dir = scratch();
seedFixture(dir, 'query', [
'// comment',
'',
'# also comment',
'{"intent":"lookup","expected_skill":"query"}',
]);
const r = loadRoutingFixtures(dir);
expect(r.fixtures.length).toBe(1);
});
it('collects malformed lines separately without crashing', () => {
const dir = scratch();
seedFixture(dir, 'query', [
'{"intent":"ok","expected_skill":"query"}',
'{ bad json',
'{"intent":"also ok","expected_skill":"query"}',
]);
const r = loadRoutingFixtures(dir);
expect(r.fixtures.length).toBe(2);
expect(r.malformed.length).toBe(1);
expect(r.malformed[0].line).toBe(2);
});
it('handles missing skills dir cleanly', () => {
const r = loadRoutingFixtures('/tmp/never-exists-routing-eval-XYZ');
expect(r.fixtures).toEqual([]);
expect(r.malformed).toEqual([]);
});
it('skips underscore and dot dirs', () => {
const dir = scratch();
seedFixture(dir, '_conventions', [
'{"intent":"x","expected_skill":"y"}',
]);
seedFixture(dir, '.hidden', [
'{"intent":"x","expected_skill":"y"}',
]);
const r = loadRoutingFixtures(dir);
expect(r.fixtures).toEqual([]);
});
});
describe('runRoutingEval', () => {
const resolver = makeResolver([
{ trigger: '"find me", "search for", "look up"', skill: 'query' },
{ trigger: '"broken citations", "fix citations"', skill: 'citation-fixer' },
{ trigger: '"every inbound message"', skill: 'signal-detector', section: 'Always-on' },
]);
it('scores a passing fixture', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'please look up that person', expected_skill: 'query' },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.passed).toBe(1);
expect(r.top1Accuracy).toBe(1);
expect(r.details[0].outcome).toBe('pass');
});
it('detects a missed fixture (no match)', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'deploy to prod', expected_skill: 'query' },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.missed).toBe(1);
expect(r.details[0].outcome).toBe('missed');
});
it('detects ambiguity when unexpected skills also match', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'search for broken citations', expected_skill: 'query' },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.ambiguous).toBe(1);
expect(r.details[0].outcome).toBe('ambiguous');
expect(r.details[0].note).toContain('citation-fixer');
});
it('honors ambiguous_with allow-list', () => {
const fixtures: RoutingFixture[] = [
{
intent: 'search for broken citations',
expected_skill: 'query',
ambiguous_with: ['citation-fixer'],
},
];
const r = runRoutingEval(resolver, fixtures);
expect(r.passed).toBe(1);
expect(r.ambiguous).toBe(0);
});
it('passes when only always-on skills co-fire', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'every inbound message look up', expected_skill: 'query' },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.passed).toBe(1);
});
it('passes a negative case when nothing matches', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'deploy the app tonight', expected_skill: null },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.passed).toBe(1);
});
it('fails a negative case when something matches (false positive)', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'please look up this fact', expected_skill: null },
];
const r = runRoutingEval(resolver, fixtures);
expect(r.falsePositives).toBe(1);
expect(r.details[0].outcome).toBe('false_positive');
});
it('empty fixture set → top1Accuracy = 1 (vacuous pass)', () => {
const r = runRoutingEval(resolver, []);
expect(r.top1Accuracy).toBe(1);
expect(r.totalCases).toBe(0);
});
it('mixed case report totals add up', () => {
const fixtures: RoutingFixture[] = [
{ intent: 'pull up paul graham', expected_skill: 'query' }, // pass
{ intent: 'deploy prod', expected_skill: 'query' }, // miss
{ intent: 'fix busted sources', expected_skill: null }, // false pos
];
const r = runRoutingEval(resolver, fixtures);
expect(r.totalCases).toBe(3);
expect(r.passed + r.missed + r.ambiguous + r.falsePositives).toBe(3);
});
});

177
test/skill-manifest.test.ts Normal file
View File

@@ -0,0 +1,177 @@
/**
* Tests for src/core/skill-manifest.ts — unified manifest loader.
*
* Covers the derive-from-walk path (F-ENG-1 / D-CX-1..4). When
* manifest.json is absent, walking skillsDir MUST produce a sensible
* synthetic manifest so reachability checks don't silently pass on
* OpenClaw deployments that don't ship manifest.json.
*/
import { describe, expect, it, beforeEach, afterEach } from 'bun:test';
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { loadOrDeriveManifest } from '../src/core/skill-manifest.ts';
const created: string[] = [];
function scratch(): string {
const dir = mkdtempSync(join(tmpdir(), 'skill-manifest-'));
created.push(dir);
return dir;
}
function writeSkill(skillsDir: string, name: string, frontmatterName?: string): void {
const skillDir = join(skillsDir, name);
mkdirSync(skillDir, { recursive: true });
const fm = frontmatterName !== undefined
? `---\nname: ${frontmatterName}\ndescription: test\n---\n`
: ''; // no frontmatter
writeFileSync(join(skillDir, 'SKILL.md'), `${fm}\n# ${name}\n`);
}
function writeManifest(skillsDir: string, json: unknown): void {
writeFileSync(join(skillsDir, 'manifest.json'), JSON.stringify(json, null, 2));
}
describe('loadOrDeriveManifest', () => {
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
it('uses manifest.json verbatim when present and valid', () => {
const dir = scratch();
writeSkill(dir, 'query');
writeSkill(dir, 'ingest');
writeManifest(dir, {
skills: [
{ name: 'query', path: 'query/SKILL.md' },
{ name: 'ingest', path: 'ingest/SKILL.md' },
],
});
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(false);
expect(r.skills.length).toBe(2);
expect(r.skills.map(s => s.name).sort()).toEqual(['ingest', 'query']);
});
it('derives from SKILL.md walk when manifest.json is missing (F-ENG-1)', () => {
const dir = scratch();
writeSkill(dir, 'query', 'query');
writeSkill(dir, 'ingest', 'ingest');
// No manifest.json — this is the OpenClaw-deployment scenario.
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills.length).toBe(2);
expect(r.skills.map(s => s.name).sort()).toEqual(['ingest', 'query']);
expect(r.skills.every(s => s.path.endsWith('/SKILL.md'))).toBe(true);
});
it('falls back to dirname when SKILL.md has no name: frontmatter', () => {
const dir = scratch();
writeSkill(dir, 'prose-only-skill'); // no frontmatter
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills.length).toBe(1);
expect(r.skills[0].name).toBe('prose-only-skill');
});
it('uses frontmatter name when it differs from dirname', () => {
const dir = scratch();
writeSkill(dir, 'weird-dir-name', 'canonical-skill-name');
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills[0].name).toBe('canonical-skill-name');
expect(r.skills[0].path).toBe('weird-dir-name/SKILL.md');
});
it('skips underscore-prefixed dirs (conventions, rule files)', () => {
const dir = scratch();
writeSkill(dir, 'query');
writeSkill(dir, '_conventions');
writeSkill(dir, '_brain-rules');
const r = loadOrDeriveManifest(dir);
expect(r.skills.map(s => s.name)).toEqual(['query']);
});
it('skips dot-prefixed dirs', () => {
const dir = scratch();
writeSkill(dir, 'query');
writeSkill(dir, '.git');
const r = loadOrDeriveManifest(dir);
expect(r.skills.map(s => s.name)).toEqual(['query']);
});
it('derives when manifest.json is malformed (invalid JSON)', () => {
const dir = scratch();
writeSkill(dir, 'query', 'query');
writeFileSync(join(dir, 'manifest.json'), '{ not valid json');
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills.map(s => s.name)).toEqual(['query']);
});
it('derives when manifest.json has a wrong shape (skills as object)', () => {
const dir = scratch();
writeSkill(dir, 'query', 'query');
writeManifest(dir, { skills: { bad: 'shape' } });
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills.map(s => s.name)).toEqual(['query']);
});
it('honors explicit empty skills array as a valid declaration', () => {
// An empty array is "no skills" declared intentionally, not
// malformed. Distinct from missing manifest.json (→ derive).
const dir = scratch();
writeSkill(dir, 'query', 'query'); // present on disk...
writeManifest(dir, { skills: [] }); // ...but manifest says zero.
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(false);
expect(r.skills.length).toBe(0);
});
it('derives when manifest.json entry lacks required keys', () => {
const dir = scratch();
writeSkill(dir, 'query', 'query');
// First entry missing 'path' → invalid shape → fall through to derive.
writeManifest(dir, { skills: [{ name: 'query' }] });
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
});
it('handles empty skillsDir cleanly', () => {
const dir = scratch();
const r = loadOrDeriveManifest(dir);
expect(r.derived).toBe(true);
expect(r.skills.length).toBe(0);
});
it('handles non-existent skillsDir cleanly', () => {
const r = loadOrDeriveManifest('/tmp/does-not-exist-skill-manifest-test');
expect(r.derived).toBe(true);
expect(r.skills.length).toBe(0);
});
it('sorts derived skills alphabetically by name', () => {
const dir = scratch();
writeSkill(dir, 'zebra', 'zebra');
writeSkill(dir, 'apple', 'apple');
writeSkill(dir, 'mango', 'mango');
const r = loadOrDeriveManifest(dir);
expect(r.skills.map(s => s.name)).toEqual(['apple', 'mango', 'zebra']);
});
it('treats dirs without SKILL.md as not-a-skill', () => {
const dir = scratch();
writeSkill(dir, 'query', 'query');
mkdirSync(join(dir, 'no-skill-here'), { recursive: true });
writeFileSync(join(dir, 'no-skill-here', 'README.md'), '# not a skill');
const r = loadOrDeriveManifest(dir);
expect(r.skills.map(s => s.name)).toEqual(['query']);
});
});

343
test/skillify-scaffold.test.ts Normal file
View File

@@ -0,0 +1,343 @@
/**
* Tests for src/core/skillify/generator.ts (W4).
* Mechanical scaffold plan + apply, idempotency (D-CX-7), stub sentinel.
*/
import { describe, expect, it, afterEach } from 'bun:test';
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
applyScaffold,
planScaffold,
SkillifyScaffoldError,
SKILL_NAME_PATTERN,
} from '../src/core/skillify/generator.ts';
import { SKILLIFY_STUB_MARKER } from '../src/core/skillify/templates.ts';
const created: string[] = [];
function scratchRepo(): { root: string; skillsDir: string } {
const root = mkdtempSync(join(tmpdir(), 'skillify-repo-'));
created.push(root);
const skillsDir = join(root, 'skills');
mkdirSync(skillsDir, { recursive: true });
mkdirSync(join(root, 'test'), { recursive: true });
writeFileSync(
join(skillsDir, 'RESOLVER.md'),
'# RESOLVER\n\n## Brain operations\n\n| Trigger | Skill |\n|---------|-------|\n| "existing thing" | `skills/existing/SKILL.md` |\n',
);
return { root, skillsDir };
}
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
describe('SKILL_NAME_PATTERN', () => {
it('accepts valid kebab-case', () => {
expect(SKILL_NAME_PATTERN.test('context-now')).toBe(true);
expect(SKILL_NAME_PATTERN.test('a')).toBe(true);
expect(SKILL_NAME_PATTERN.test('calendar-recall-v2')).toBe(true);
});
it('rejects uppercase, spaces, underscores, leading digits', () => {
expect(SKILL_NAME_PATTERN.test('ContextNow')).toBe(false);
expect(SKILL_NAME_PATTERN.test('context now')).toBe(false);
expect(SKILL_NAME_PATTERN.test('context_now')).toBe(false);
expect(SKILL_NAME_PATTERN.test('2-skill')).toBe(false);
});
});
describe('planScaffold', () => {
it('throws SkillifyScaffoldError on invalid name', () => {
const { root, skillsDir } = scratchRepo();
expect(() =>
planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'Bad Name',
description: 'x',
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
},
}),
).toThrow(SkillifyScaffoldError);
});
it('plans 4 files + resolver append for a new skill', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'hello-world',
description: 'say hello',
triggers: ['say hello', 'greet me'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
expect(plan.files.length).toBe(4);
const paths = plan.files.map(f => f.path);
expect(paths).toContain(join(skillsDir, 'hello-world', 'SKILL.md'));
expect(paths).toContain(
join(skillsDir, 'hello-world', 'scripts', 'hello-world.mjs'),
);
expect(paths).toContain(join(skillsDir, 'hello-world', 'routing-eval.jsonl'));
expect(paths).toContain(join(root, 'test', 'hello-world.test.ts'));
expect(plan.files.every(f => f.kind === 'new')).toBe(true);
expect(plan.resolverFile).toBe(join(skillsDir, 'RESOLVER.md'));
expect(plan.resolverAppend).not.toBeNull();
expect(plan.resolverAppend!).toContain('`skills/hello-world/SKILL.md`');
});
it('SKILL.md includes the SKILLIFY_STUB sentinel', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'foo',
description: 'foo skill',
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
},
});
const skillMd = plan.files.find(f => f.path.endsWith('SKILL.md'))!;
expect(skillMd.content).toContain(SKILLIFY_STUB_MARKER);
});
it('script stub includes the SKILLIFY_STUB sentinel (D-CX-9 gate hook)', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'foo',
description: 'foo skill',
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
},
});
const script = plan.files.find(f => f.path.endsWith('foo.mjs'))!;
expect(script.content).toContain(SKILLIFY_STUB_MARKER);
});
it('refuses to scaffold over an existing file without --force', () => {
const { root, skillsDir } = scratchRepo();
mkdirSync(join(skillsDir, 'existing'), { recursive: true });
writeFileSync(
join(skillsDir, 'existing', 'SKILL.md'),
'---\nname: existing\n---\n',
);
expect(() =>
planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'existing',
description: 'x',
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
},
}),
).toThrow(SkillifyScaffoldError);
});
it('--force marks existing files as overwrite', () => {
const { root, skillsDir } = scratchRepo();
mkdirSync(join(skillsDir, 'existing'), { recursive: true });
writeFileSync(
join(skillsDir, 'existing', 'SKILL.md'),
'---\nname: existing\n---\n',
);
const plan = planScaffold({
skillsDir,
repoRoot: root,
force: true,
vars: {
name: 'existing',
description: 'x',
triggers: ['foo'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
const skillMd = plan.files.find(f => f.path.endsWith('SKILL.md'))!;
expect(skillMd.kind).toBe('overwrite');
});
it('D-CX-7: resolverAppend is null when row already present (idempotent)', () => {
const { root, skillsDir } = scratchRepo();
// Prime the resolver with an existing row for the skill we're about
// to scaffold. The plan must NOT queue a duplicate append, even
// when --force regenerates files.
const resolverPath = join(skillsDir, 'RESOLVER.md');
const before = readFileSync(resolverPath, 'utf-8');
writeFileSync(
resolverPath,
before +
'\n## Uncategorized\n\n| Trigger | Skill |\n|---------|-------|\n' +
'| "do thing" | `skills/demo/SKILL.md` |\n',
);
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'demo',
description: 'demo',
triggers: ['do thing'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
expect(plan.resolverAppend).toBeNull();
});
it('handles --triggers omitted by seeding TBD placeholder', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'empty-triggers',
description: 'test',
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
},
});
const skillMd = plan.files.find(f => f.path.endsWith('SKILL.md'))!;
expect(skillMd.content).toContain('TBD-trigger');
});
it('writes_pages + writes_to flow through to frontmatter', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'writer',
description: 'writer',
triggers: ['write me'],
writesTo: ['people/', 'companies/'],
writesPages: true,
mutating: true,
},
});
const skillMd = plan.files.find(f => f.path.endsWith('SKILL.md'))!;
expect(skillMd.content).toContain('writes_pages: true');
expect(skillMd.content).toContain('- people/');
expect(skillMd.content).toContain('- companies/');
expect(skillMd.content).toContain('mutating: true');
});
});
describe('applyScaffold', () => {
it('writes all planned files and appends the resolver row', () => {
const { root, skillsDir } = scratchRepo();
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'hello',
description: 'hi',
triggers: ['say hi'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
applyScaffold(plan);
for (const f of plan.files) {
expect(existsSync(f.path)).toBe(true);
}
const resolver = readFileSync(join(skillsDir, 'RESOLVER.md'), 'utf-8');
expect(resolver).toContain('`skills/hello/SKILL.md`');
});
it('second apply with same name + --force overwrites files but does NOT duplicate resolver row (D-CX-7)', () => {
const { root, skillsDir } = scratchRepo();
const firstPlan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'idem',
description: 'first',
triggers: ['t'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
applyScaffold(firstPlan);
const secondPlan = planScaffold({
skillsDir,
repoRoot: root,
force: true,
vars: {
name: 'idem',
description: 'second',
triggers: ['t'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
expect(secondPlan.resolverAppend).toBeNull();
applyScaffold(secondPlan);
const resolver = readFileSync(join(skillsDir, 'RESOLVER.md'), 'utf-8');
const count = (resolver.match(/`skills\/idem\/SKILL\.md`/g) ?? []).length;
expect(count).toBe(1);
});
it('applies against an AGENTS.md-layout workspace (W1 interop)', () => {
const root = mkdtempSync(join(tmpdir(), 'skillify-openclaw-'));
created.push(root);
const skillsDir = join(root, 'workspace', 'skills');
mkdirSync(skillsDir, { recursive: true });
mkdirSync(join(root, 'test'), { recursive: true });
// AGENTS.md at workspace root, NOT inside skills/.
writeFileSync(
join(root, 'workspace', 'AGENTS.md'),
'# AGENTS\n\n## Ops\n\n| Trigger | Skill |\n|---------|-------|\n',
);
const plan = planScaffold({
skillsDir,
repoRoot: root,
vars: {
name: 'openclaw-demo',
description: 'demo',
triggers: ['do it'],
writesTo: [],
writesPages: false,
mutating: false,
},
});
expect(plan.resolverFile).toBe(join(root, 'workspace', 'AGENTS.md'));
expect(plan.resolverAppend).not.toBeNull();
applyScaffold(plan);
const agents = readFileSync(join(root, 'workspace', 'AGENTS.md'), 'utf-8');
expect(agents).toContain('`skills/openclaw-demo/SKILL.md`');
});
});

447
test/skillpack-install.test.ts Normal file
View File

@@ -0,0 +1,447 @@
/**
* Tests for src/core/skillpack/bundle.ts + installer.ts (W5).
* Bundle enumeration, dependency closure, per-file diff, managed
* block, lockfile concurrency, atomic writes.
*/
import { describe, expect, it, afterEach } from 'bun:test';
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from 'fs';
import { dirname, join } from 'path';
import { tmpdir } from 'os';
import {
bundledSkillSlugs,
findGbrainRoot,
loadBundleManifest,
enumerateBundle,
BundleError,
} from '../src/core/skillpack/bundle.ts';
import {
applyInstall,
buildManagedBlock,
diffSkill,
extractManagedSlugs,
planInstall,
updateManagedBlock,
InstallError,
} from '../src/core/skillpack/installer.ts';
const created: string[] = [];
function scratchGbrain(): { gbrainRoot: string; skillsDir: string } {
const root = mkdtempSync(join(tmpdir(), 'skillpack-gbrain-'));
created.push(root);
mkdirSync(join(root, 'src'), { recursive: true });
writeFileSync(join(root, 'src', 'cli.ts'), '// stub');
const skillsDir = join(root, 'skills');
mkdirSync(skillsDir, { recursive: true });
// Two bundled skills + shared deps.
mkdirSync(join(skillsDir, 'alpha'), { recursive: true });
writeFileSync(
join(skillsDir, 'alpha', 'SKILL.md'),
'---\nname: alpha\n---\n# alpha\n',
);
mkdirSync(join(skillsDir, 'alpha', 'scripts'), { recursive: true });
writeFileSync(
join(skillsDir, 'alpha', 'scripts', 'alpha.mjs'),
'export function run() { return "alpha"; }\n',
);
mkdirSync(join(skillsDir, 'beta'), { recursive: true });
writeFileSync(
join(skillsDir, 'beta', 'SKILL.md'),
'---\nname: beta\n---\n# beta\n',
);
// Shared deps.
mkdirSync(join(skillsDir, 'conventions'), { recursive: true });
writeFileSync(
join(skillsDir, 'conventions', 'quality.md'),
'# quality conventions\n',
);
writeFileSync(join(skillsDir, '_output-rules.md'), '# output rules\n');
// RESOLVER.md (so find-resolver finds it).
writeFileSync(
join(skillsDir, 'RESOLVER.md'),
'# RESOLVER\n\n| Trigger | Skill |\n|---------|-------|\n| "alpha" | `skills/alpha/SKILL.md` |\n| "beta" | `skills/beta/SKILL.md` |\n',
);
// Plugin manifest.
writeFileSync(
join(root, 'openclaw.plugin.json'),
JSON.stringify(
{
name: 'gbrain-test',
version: '0.17.0-test',
skills: ['skills/alpha', 'skills/beta'],
shared_deps: ['skills/conventions', 'skills/_output-rules.md'],
},
null,
2,
),
);
return { gbrainRoot: root, skillsDir };
}
function scratchTarget(): { workspace: string; skillsDir: string } {
const workspace = mkdtempSync(join(tmpdir(), 'skillpack-target-'));
created.push(workspace);
const skillsDir = join(workspace, 'skills');
mkdirSync(skillsDir, { recursive: true });
// Seed a RESOLVER.md so managed block has a home.
writeFileSync(
join(skillsDir, 'RESOLVER.md'),
'# Target RESOLVER\n\n| Trigger | Skill |\n|---------|-------|\n',
);
return { workspace, skillsDir };
}
afterEach(() => {
while (created.length) {
const d = created.pop();
if (d && existsSync(d)) rmSync(d, { recursive: true, force: true });
}
});
describe('findGbrainRoot', () => {
it('walks up to find openclaw.plugin.json + src/cli.ts', () => {
const { gbrainRoot } = scratchGbrain();
const nested = join(gbrainRoot, 'a', 'b', 'c');
mkdirSync(nested, { recursive: true });
expect(findGbrainRoot(nested)).toBe(gbrainRoot);
});
it('returns null when no gbrain root above', () => {
expect(findGbrainRoot('/tmp/definitely-not-a-gbrain-repo-XYZ')).toBeNull();
});
});
describe('loadBundleManifest', () => {
it('loads + validates a valid manifest', () => {
const { gbrainRoot } = scratchGbrain();
const m = loadBundleManifest(gbrainRoot);
expect(m.skills).toEqual(['skills/alpha', 'skills/beta']);
expect(m.shared_deps.length).toBe(2);
});
it('throws BundleError on missing manifest', () => {
const root = mkdtempSync(join(tmpdir(), 'skillpack-empty-'));
created.push(root);
expect(() => loadBundleManifest(root)).toThrow(BundleError);
});
it('throws BundleError on malformed JSON', () => {
const { gbrainRoot } = scratchGbrain();
writeFileSync(join(gbrainRoot, 'openclaw.plugin.json'), '{ nope');
expect(() => loadBundleManifest(gbrainRoot)).toThrow(BundleError);
});
});
describe('enumerateBundle (D-CX-10 dependency closure)', () => {
it('includes skill files AND shared_deps for a single-skill install', () => {
const { gbrainRoot } = scratchGbrain();
const m = loadBundleManifest(gbrainRoot);
const entries = enumerateBundle({ gbrainRoot, skillSlug: 'alpha', manifest: m });
const targets = entries.map(e => e.relTarget).sort();
expect(targets).toContain('alpha/SKILL.md');
expect(targets).toContain('alpha/scripts/alpha.mjs');
// Shared deps pulled in despite single-skill scope.
expect(targets).toContain('conventions/quality.md');
expect(targets).toContain('_output-rules.md');
// beta NOT included.
expect(targets.find(t => t.startsWith('beta/'))).toBeUndefined();
});
it('throws BundleError for unknown skill slug', () => {
const { gbrainRoot } = scratchGbrain();
const m = loadBundleManifest(gbrainRoot);
expect(() =>
enumerateBundle({ gbrainRoot, skillSlug: 'nope', manifest: m }),
).toThrow(BundleError);
});
it('enumerates everything when skillSlug is undefined (--all)', () => {
const { gbrainRoot } = scratchGbrain();
const m = loadBundleManifest(gbrainRoot);
const entries = enumerateBundle({ gbrainRoot, manifest: m });
const targets = entries.map(e => e.relTarget).sort();
expect(targets.some(t => t.startsWith('alpha/'))).toBe(true);
expect(targets.some(t => t.startsWith('beta/'))).toBe(true);
});
});
describe('buildManagedBlock + updateManagedBlock', () => {
it('builds a block with all installed slugs as rows', () => {
const m = loadBundleManifest(scratchGbrain().gbrainRoot);
const block = buildManagedBlock(m, ['alpha', 'beta']);
expect(block).toContain('gbrain:skillpack:begin');
expect(block).toContain('gbrain:skillpack:end');
expect(block).toContain('`skills/alpha/SKILL.md`');
expect(block).toContain('`skills/beta/SKILL.md`');
});
it('appends block when none exists', () => {
const block = buildManagedBlock(loadBundleManifest(scratchGbrain().gbrainRoot), ['alpha']);
const updated = updateManagedBlock('# AGENTS\n\nSome prose.\n', block);
expect(updated).toContain('gbrain:skillpack:begin');
expect(updated).toContain('Some prose.');
});
it('replaces existing block in place, keeping surrounding content', () => {
const m = loadBundleManifest(scratchGbrain().gbrainRoot);
const original =
'# AGENTS\n\nBefore\n\n' +
buildManagedBlock(m, ['alpha']) +
'\n\nAfter\n';
const replaced = updateManagedBlock(original, buildManagedBlock(m, ['alpha', 'beta']));
expect(replaced).toContain('Before');
expect(replaced).toContain('After');
expect(replaced).toContain('`skills/beta/SKILL.md`');
});
it('extractManagedSlugs roundtrips with buildManagedBlock', () => {
const m = loadBundleManifest(scratchGbrain().gbrainRoot);
const block = buildManagedBlock(m, ['alpha', 'beta']);
expect(extractManagedSlugs(block).sort()).toEqual(['alpha', 'beta']);
});
it('extractManagedSlugs returns [] when no block present', () => {
expect(extractManagedSlugs('# hello\n\nno block here\n')).toEqual([]);
});
});
describe('planInstall + applyInstall', () => {
it('dry-run: plans file writes but does not touch target', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const plan = planInstall({
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
});
const result = applyInstall(plan, {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
dryRun: true,
});
expect(result.dryRun).toBe(true);
expect(result.summary.wroteNew).toBeGreaterThan(0);
expect(existsSync(join(skillsDir, 'alpha', 'SKILL.md'))).toBe(false);
});
it('installs a fresh skill and its shared deps', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const plan = planInstall({
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
});
const result = applyInstall(plan, {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
});
expect(result.summary.wroteNew).toBeGreaterThan(0);
expect(existsSync(join(skillsDir, 'alpha', 'SKILL.md'))).toBe(true);
expect(existsSync(join(skillsDir, 'alpha', 'scripts', 'alpha.mjs'))).toBe(true);
expect(existsSync(join(skillsDir, 'conventions', 'quality.md'))).toBe(true);
expect(result.managedBlock.applied).toBe(true);
});
it('re-install is idempotent (skipped_identical on unchanged files)', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(opts), opts);
const second = applyInstall(planInstall(opts), opts);
expect(second.summary.wroteNew).toBe(0);
expect(second.summary.skippedIdentical).toBeGreaterThan(0);
});
it('skips locally-modified files without --overwrite-local', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(opts), opts);
// Locally edit the installed SKILL.md
const localFile = join(skillsDir, 'alpha', 'SKILL.md');
writeFileSync(localFile, readFileSync(localFile, 'utf-8') + '\n<!-- local edit -->\n');
const result = applyInstall(planInstall(opts), opts);
expect(result.summary.skippedLocallyModified).toBeGreaterThan(0);
// Local edit preserved.
expect(readFileSync(localFile, 'utf-8')).toContain('local edit');
});
it('--overwrite-local replaces locally-modified files', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(opts), opts);
const localFile = join(skillsDir, 'alpha', 'SKILL.md');
writeFileSync(localFile, 'local garbage');
const overwriteOpts = { ...opts, overwriteLocal: true };
const result = applyInstall(planInstall(overwriteOpts), overwriteOpts);
expect(result.summary.wroteOverwrite).toBeGreaterThan(0);
expect(readFileSync(localFile, 'utf-8')).not.toContain('local garbage');
});
it('D-CX-11: concurrent install attempt fails with lock_held', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
// Simulate a peer holding the lock.
writeFileSync(join(workspace, '.gbrain-skillpack.lock'), '99999');
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
const plan = planInstall(opts);
expect(() => applyInstall(plan, opts)).toThrow(InstallError);
});
it('D-CX-11: --force-unlock overrides a stale lock', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
// Stale lock is handled by setting lockStaleMs small and sleeping
// — simulate by writing the lock and passing lockStaleMs=0 so
// any age looks stale.
writeFileSync(join(workspace, '.gbrain-skillpack.lock'), '99999');
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
forceUnlock: true,
lockStaleMs: 0,
};
const plan = planInstall(opts);
const result = applyInstall(plan, opts);
expect(result.summary.wroteNew).toBeGreaterThan(0);
});
it('managed block is written atomically (tmp then rename)', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: null,
};
applyInstall(planInstall(opts), opts);
const resolver = readFileSync(join(skillsDir, 'RESOLVER.md'), 'utf-8');
expect(resolver).toContain('gbrain:skillpack:begin');
expect(resolver).toContain('gbrain:skillpack:end');
expect(resolver).toContain('`skills/alpha/SKILL.md`');
expect(resolver).toContain('`skills/beta/SKILL.md`');
});
it('managed block accumulates across separate single-skill installs', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const alphaOpts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(alphaOpts), alphaOpts);
const betaOpts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'beta',
};
applyInstall(planInstall(betaOpts), betaOpts);
const resolver = readFileSync(join(skillsDir, 'RESOLVER.md'), 'utf-8');
expect(resolver).toContain('`skills/alpha/SKILL.md`');
expect(resolver).toContain('`skills/beta/SKILL.md`');
});
it('works against AGENTS.md-at-workspace-root layout', () => {
const { gbrainRoot } = scratchGbrain();
const workspace = mkdtempSync(join(tmpdir(), 'skillpack-root-agents-'));
created.push(workspace);
const skillsDir = join(workspace, 'skills');
mkdirSync(skillsDir, { recursive: true });
// No RESOLVER.md in skills — AGENTS.md at workspace root instead.
writeFileSync(
join(workspace, 'AGENTS.md'),
'# AGENTS\n\n| Trigger | Skill |\n|---------|-------|\n',
);
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
const result = applyInstall(planInstall(opts), opts);
expect(result.managedBlock.applied).toBe(true);
const agents = readFileSync(join(workspace, 'AGENTS.md'), 'utf-8');
expect(agents).toContain('`skills/alpha/SKILL.md`');
});
});
describe('diffSkill', () => {
it('reports missing files', () => {
const { gbrainRoot } = scratchGbrain();
const { skillsDir } = scratchTarget();
const diffs = diffSkill(gbrainRoot, 'alpha', skillsDir);
expect(diffs.every(d => !d.existing)).toBe(true);
});
it('reports identical after install', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(opts), opts);
const diffs = diffSkill(gbrainRoot, 'alpha', skillsDir);
expect(diffs.every(d => d.existing && d.identical)).toBe(true);
});
it('reports differs after local edit', () => {
const { gbrainRoot } = scratchGbrain();
const { workspace, skillsDir } = scratchTarget();
const opts = {
gbrainRoot,
targetWorkspace: workspace,
targetSkillsDir: skillsDir,
skillSlug: 'alpha',
};
applyInstall(planInstall(opts), opts);
writeFileSync(join(skillsDir, 'alpha', 'SKILL.md'), 'edited locally');
const diffs = diffSkill(gbrainRoot, 'alpha', skillsDir);
expect(diffs.some(d => !d.identical && d.existing)).toBe(true);
});
});

82
test/skillpack-sync-guard.test.ts Normal file
View File

@@ -0,0 +1,82 @@
/**
* test/skillpack-sync-guard.test.ts — F-ENG-4 / D-CX-4.
*
* Guards against drift between:
* - openclaw.plugin.json#skills (what skillpack install ships)
* - skills/manifest.json#skills[].path (what the overall skill manifest knows)
*
* If someone adds a skill directory but forgets the plugin manifest,
* or vice versa, this test fails. The sync guard exists because the
* codex outside-voice flagged version drift on the plugin manifest.
*/
import { describe, expect, it } from 'bun:test';
import { existsSync, readFileSync } from 'fs';
import { join } from 'path';
const REPO = join(import.meta.dir, '..');
function readJson(path: string): any {
return JSON.parse(readFileSync(path, 'utf-8'));
}
describe('skillpack sync-guard', () => {
const pluginPath = join(REPO, 'openclaw.plugin.json');
const skillsManifestPath = join(REPO, 'skills', 'manifest.json');
it('both manifests exist at the expected paths', () => {
expect(existsSync(pluginPath)).toBe(true);
expect(existsSync(skillsManifestPath)).toBe(true);
});
it('every openclaw.plugin.json skill path exists on disk', () => {
const plugin = readJson(pluginPath);
for (const skillPath of plugin.skills) {
const skillMd = join(REPO, skillPath, 'SKILL.md');
expect(existsSync(skillMd)).toBe(true);
}
});
it('every shared_dep in openclaw.plugin.json exists on disk', () => {
const plugin = readJson(pluginPath);
for (const dep of plugin.shared_deps) {
const abs = join(REPO, dep);
expect(existsSync(abs)).toBe(true);
}
});
it('openclaw.plugin.json skills ⊂ skills/manifest.json skill paths', () => {
// Each entry in the plugin manifest's "skills" list must correspond
// to a skill that manifest.json knows about. Installing something
// the rest of gbrain doesn't register is a bug.
const plugin = readJson(pluginPath);
const skillsManifest = readJson(skillsManifestPath);
const knownSlugs = new Set(
skillsManifest.skills.map((s: { path: string }) =>
s.path.replace(/\/SKILL\.md$/, ''),
),
);
for (const skillPath of plugin.skills) {
const slug = skillPath.replace(/^skills\//, '');
expect(knownSlugs.has(slug)).toBe(true);
}
});
it('excluded skills are not listed in plugin.skills (install list is curated)', () => {
const plugin = readJson(pluginPath);
const excluded = new Set(plugin.excluded_from_install ?? []);
for (const skillPath of plugin.skills) {
expect(excluded.has(skillPath)).toBe(false);
}
});
it('plugin version tracks a real gbrain release line', () => {
// Loose check: version must be semver-ish, not the stale 0.4.1
// pre-v0.17 placeholder the codex review flagged.
const plugin = readJson(pluginPath);
const major = parseInt(plugin.version.split('.')[0], 10);
const minor = parseInt(plugin.version.split('.')[1], 10);
expect(major).toBe(0);
expect(minor).toBeGreaterThanOrEqual(17);
});
});