Repository navigation
bench: rerun the v2 skill eval on Sonnet 5.5 (default model) - #41
Merged
Merged
Conversation
Run ids repeat across models, so Sonnet runs scored with the Opus adjudication would inherit its decisions. The adjudication file can now be named explicitly. The table also counts runs that looked sources up on their own, which is the behaviour that separated B from A by model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Opus the no-skill agent queried CrossRef itself and matched the skill, so whether to keep investing depended on what the default model does. On Sonnet 5.5 the no-skill agent looked up about half the cited records, mostly through publisher pages that refused it, and caught 35/52 planted errors against 51/52 with the skill. The Semantic Scholar rate-limit hedging comes from the agent's own searches that SKILL.md suggests, not from the engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmarks only. No engine,
SKILL.md, or test changes. This is the v2 set rerun on Sonnet 5.5, the current default model, with main3083597as the engine.Why
On Opus 5.5 (#40), the agent without the skill reached 104/104 because it queried CrossRef itself. Whether to keep investing depends on whether the default model does the same.
Results (1 repeat per condition on Sonnet; Opus over 2 repeats)
B's own lookups:
Semantic Scholar hedging in A:
SKILL.mdlists), not from engine output.Full tables, quotes, and limits are in
benchmarks/e2e/v2/README.md, section "Sonnet 5.5 rerun".Changes
score.py --adjudication: run ids repeat across models, so Sonnet runs need their own decisions file.score.py: new table row counting runs that looked sources up themselves.v2/adjudication-sonnet.json: every planted error decided explicitly.v2/results/2026-10-09-3083597-e2e-v2-sonnet.json: per-run results.🤖 Generated with Claude Code