bench: measure ref-verify end to end inside Claude Code sessions

[moonweave]

Why

The reference benchmark scores the engine alone. It doesn't tell us what an agent reports when a researcher says "check my references before I submit", or what the ref-verify skill adds over the model by itself. This PR measures that. It is measurement only: no engine or SKILL.md changes.

What

benchmarks/e2e/:

• Manuscripts: eight short manuscripts covering English Markdown (APA, Vancouver, IEEE, physics style without titles), two LaTeX + BibTeX, and two Korean. Domains are materials/polymers, biomedicine, ML, and physics.
• truth.json: 40 planted errors of 10 types (fabricated with and without DOI, DOI swap, retracted, wrong year, wrong first author, title-less wrong volume/page, numeric claims with a wrong number, unit prefix, or direction). It also lists 60 correct references, 8 correct numeric claims, and 5 correct references CrossRef does not index (a Korean thesis, a KCI paper, NeurIPS, JMLR, arXiv). verify_truth.py re-checks every label live: 0 problems, with a planted counter-example caught.
• Harness: run_e2e.py runs each manuscript twice per condition, 32 runs in all (A: skill copied into <project>/.claude/skills/ref-verify/; B: same project without it).
• Scorer and decisions: score.py is stdlib only and marks anything it can't decide as review; adjudication.json holds the decisions made after reading all 32 answers.
• Engine baseline: engine_only.py runs check-bib alone on each reference list.
• Results: README.md and results/.

Isolation, verified with positive controls before any run:

claude -p "<prompt>" --output-format stream-json --verbose --include-hook-events \
  --setting-sources project --settings '{"disableAllHooks": true, "autoMemoryEnabled": false}' \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' --no-session-persistence \
  --permission-mode dontAsk --allowedTools "Bash Read Glob Grep WebFetch WebSearch Skill" \
  --disallowedTools "Write Edit NotebookEdit"
# cwd = scratch project, stdin = /dev/null, CLAUDE_REFVERIFY_EVAL_CHILD=1, REF_VERIFY_CACHE_DIR=<project>/.rv-cache

All 32 runs reported 0 hook events. The model was the subscription default (claude-opus-5-5, apiKeySource: none).

Results

	A: with skill	B: without
Planted errors caught	80/80	77/80
CLAIM_NUMBER	10/10	7/10
Each of the 9 other types	100%	100%
False alarms: correct refs / unindexed / correct claims	0/120 · 0/10 · 0/16	0/120 · 0/10 · 0/16
Engine invoked	16/16	0/16
Used WebFetch/WebSearch	2/16	13/16
Usable answer (reference + problem + fix)	16/16	16/16
Wall time, median (mean)	85 s (84.5 s)	40 s (44.7 s)

B's misses are numbers it didn't check against a source. Twice it accepted "up to 380%" for Pelrine 2000 (abstract: 215%). Once it wrote "1,200만 장 … 부분은 맞습니다" (abstract: 1.2 million). The base model knows most planted papers, so reference-level errors are caught either way on this set. The README lists the limits and the skill/engine findings.

How tested

PYTHONPATH=src python3 -m unittest discover -s tests          # OK
python3 -m py_compile src/ref_verify/*.py tests/*.py scripts/*.py benchmarks/e2e/*.py   # clean
python3 benchmarks/e2e/verify_truth.py                       # network: 0 problems

The full transcripts (32 stream-json files) are not committed; they are kept in the session's scratch directory.

🤖 Generated with Claude Code