Repository navigation
bench: e2e skill eval v2 on papers the model does not know - #40
Merged
Merged
Conversation
v1's set was built from famous papers, so it could not tell whether the skill helps when the model cannot remember the sources. Running a second set needs the scripts to take a truth file other than v1's, and a third condition with no web tools and no shell measures what the model knows on its own. The subscription default for isolated sessions moved from Opus to Sonnet after v1, so the child model can now be pinned to keep runs comparable. engine_only pairs results by cited DOI first because v2 does not list truth items in reference order. verify_truth checks the extra detail v2 records (first author, volume/page, swap target, abstract numbers) so a mislabelled item cannot pass. v1 scores, engine pairing, and live verification are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner's question after v1 was whether ref-verify still makes a difference when memory cannot help. v2 keeps v1's error types, prompts and isolation, but cites only obscure 2025-2026 papers (median 0 citations), Korean society journals and theses, and adds a condition without web or shell. With live lookups the skill and no-skill agents both caught 104/104 planted errors; the agent without the skill called CrossRef itself. With no lookups the model caught 32/52, 17 of them only as "check this", and made five false statements about correct references. The difference comes from live lookup, not from the skill. Measured on main e045a86 with claude-opus-5-5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks only. No engine,
SKILL.md, or test changes. Measured on maine045a86.Why
v1 (#37) used famous papers: 80/80 with the skill, 77/80 without. The open question was whether ref-verify makes a difference when the model can't rely on remembering the papers.
What
benchmarks/e2e/v2/: 8 new manuscripts (93 references, 30 claims, 52 planted errors per pass), with the same error types as v1.truth.jsonrecords each paper's year and citation count.verify_truth.py --truth v2/truth.jsonpasses live: 0 problems on 123 items. A truth copy with 8 planted label mistakes got all 8 reported.--truthon all four scripts.--modelto pin the child model.engine_only.pypairs results by DOI.Results (claude-opus-5-5, pinned)
curlin 15/16 runs.benchmarks/e2e/v2/README.md.Notes
curlthe way B did.--model claude-opus-5-5, so v2 differs from v1 only in the papers.🤖 Generated with Claude Code