Repository navigation
feat(judge): check the judge's quote, and measure it on the NEIs (20.9, 20.13) - #18
Merged
Merged
Conversation
…9, 20.13) The review prompt asked the judge to quote the passage, but nothing checked it, so an opinion from memory or a paraphrase was attached like any other (rule 1). The answer now carries a separate `quote`. A SUPPORTED or REFUTED opinion whose quote is not in the passage is dropped and noted: the match is exact after one normalisation, at word and number boundaries, with at least two words a part. The report shows the checked quote. Cached judgements are keyed by passage and by the check version, so an unchecked or differently anchored one is never served. The eval gains --judge-nei and --no-fallback. Shown each NEI's closest passage, the judge fixed 5 AVeriTeC claims and broke none (0.427 against 0.371, p = 0.062), just short of the bar set beforehand, so the product does not escalate NEIs. The quote check dropped 0 of 106 opinions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K46EEsxMvuAuM8wjvPNWLM
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
quotefield. A SUPPORTED or REFUTED opinion whose quote is not in the passage is dropped and noted. The match is exact after one normalisation: NFC, straight quotes, dashes and U+2212 to-, format characters removed, whitespace collapsed, casefold. Each ellipsis part must sit at word and number boundaries, so "2 million" is not in "1.2 million" and "3.2%" is not in "-3.2%", and must hold at least two words, counting a number as one. An NEI opinion may have no quote. The report shows the checked quote.#q1), so an unchecked opinion, or one checked against a different passage, is never served. A guarded migration adds thequotecolumn.pipeline.closest()exposes the passageaggregatealready found, anddecideis unchanged.scripts/eval_averitec.pygains--judge-neiand--no-fallback, a second hypothetical column, an exact McNemar test against the product, and counts of quote-check drops.docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md, OPEN-ITEMS §23 (decisions and result), the report, the README AVeriTeC paragraph and the CHANGELOG.Why
Rule 1 ("never assert without a passage") binds the judge too. Its prompt asked for a verbatim quote, but nothing checked it. And the 2026-10-05 eval showed the judge never sees an NEI, while NEI on a read source is the largest loss.
The change had an independent review and three re-review rounds. They tightened the matcher: word and number boundaries, a minimum length, the sign kept, no NFKC, and the range dash. The reviews also added the passage hash to the cache key, showed the checked quote in the report, and guarded the eval tests against reaching a live run. A test had once started a 16-claim live run by accident; the stray results file was deleted, and an autouse fixture now fails any such test. The documentation numbers were fact-checked against the raw results.
Checklist
ruff checkandruff format --checkare clean (andmypy --strict, plusmypy scripts/eval_averitec.py)pytestpasses (full suite)🤖 Generated with Claude Code
https://claude.ai/code/session_01K46EEsxMvuAuM8wjvPNWLM