skill-verifier

github.com/punit-labs/skill-verifier

An eval/reliability harness that tests six real, pipeline-chained Claude Code Skills against their own invariants — and then deliberately corrupts known-good outputs to prove those checks actually have teeth.

Why this exists

On a prior project (chatgpt-chat-cleanup), an unofficial API endpoint returned HTTP 200 for an archive request that silently never persisted — the response shape said success; the backend disagreed. It was only caught by manually re-checking server state after the fact.

skill-verifier catches that failure mode one level up, against a different kind of "success response": Claude Code Skills whose JSON parses cleanly and whose markdown renders without error, but whose numbers have quietly drifted, whose dates are out of order, or whose "critical path" no longer has zero slack. It runs six real, pipeline-chained skills through their actual CLIs against real fixtures, asserts the invariants each one's output is supposed to guarantee, confirms each rejects bad input loudly instead of accepting it quietly, and then deliberately corrupts known-good outputs to prove its own checks would actually catch the corruption — because a passing invariant that can't detect an injected bug is exactly as trustworthy as a 200 OK that didn't persist.

What it tests

Six punit-labs skills, checked out as pinned git submodules under targets/ (see targets/README.md) rather than vendored — this project's whole point is not trusting an unverified copy, so it shouldn't ship one of its own:

• critical-path-mapper — CPM scheduling math (slack, critical path, weekday rolling)
• prd-to-jira-skill — PRD → Jira backlog (dependency referential integrity, Fibonacci points)
• sprint-slippage-predictor — Monte Carlo slippage forecasting (percentile ordering, load-bearing isolation)
• exec-status-report — status.json → exec report (RAG/severity enums, structural echo)
• onboarding-copilot — onboarding.json → onboarding brief (enums, structural echo)
• tpm-command-center — the pipeline capstone: every number in its dashboard must trace back verbatim to one of the four upstream files

These six chain together on one shared fictional dataset ("Nimbus Expense"): each skill's real examples/output/*.json is the next skill's real examples/*.json input, so the harness's end-to-end pipeline test chains real fixtures rather than fabricating new ones.

What it checks, per skill

• Invariants — run the skill's real script against its real example input, then assert the domain-specific guarantees its output is supposed to hold (date ordering, slack formula, enum membership, referential integrity, percentile ordering, traceability of rendered numbers back to source JSON).
• Negatives — deliberately break the input (dangling dependency id, invalid enum, negative points, dependency cycle) and confirm the script rejects it loudly (non-zero exit, expected error message) instead of producing bad output quietly.
• Mutations — two distinct modes:
  • Mode A corrupts a known-good output fixture directly (no script re-run) and re-checks it, asserting the relevant invariant fails. If nothing catches it, that's reported as a harness gap — a distinct category from an ordinary failure, since it's a statement about this project's own blind spot, not the skill's.
  • Mode B corrupts an upstream input (e.g. a number in schedule.json), re-runs tpm-command-center for real, and reuses its ordinary invariant check — since that check always recomputes expected values from whatever input was actually fed in, a failure here means the skill silently failed to propagate a changed input.

Setup

git clone --recurse-submodules https://github.com/punit-labs/skill-verifier.git
cd skill-verifier
./setup.sh   # if you cloned without --recurse-submodules

Pure Python 3 stdlib — no pip install required.

Usage

python3 run.py                              # everything, all 6 skills
python3 run.py --skill critical-path-mapper  # one skill (repeatable)
python3 run.py --only invariants             # one category (repeatable): invariants | negatives | mutations
python3 run.py --list                        # list known skill keys

Each check prints a progress line as it runs:

[12/47] PASS critical-path-mapper/invariant: EPIC-2-1:slack_days_formula
[48/47] FAIL exec-status-report/negative: bad_epic_status -- expected non-zero exit, got 0

followed by a per-skill summary, a list of any failures and harness gaps, and a process exit code (0 iff everything passed) — suitable for CI.

Running the harness's own test suite

python3 -m unittest discover tests

This project's own reliability is part of its pitch, so it has three layers of self-verification: hand-crafted pass/fail cases for every reusable invariant helper, deep-copy isolation tests for every mutation operator, and a canary test that wires a deliberately-broken "always passes" invariant module into a mutation case to prove the harness-gap signal isn't vacuous. All of this runs on a bare clone before targets/ is even initialized; the one test suite that touches real submodule code (test_integration_smoke.py) skips cleanly with an explanation if it isn't.

License

MIT — see LICENSE.