diff --git a/CHANGELOG.md b/CHANGELOG.md index 31f0969..a60d023 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,34 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] +### Changed +- **The judge's quote is checked, and shown.** A `SUPPORTED` or `REFUTED` opinion now + has to return the exact words from the passage that decide it, in a new `quote` + field. An opinion is dropped, with a note saying why, when its quote has no words, + when a part of it is shorter than two words, or when it is not in the passage it was + shown at word boundaries ("5 million" is not in "15 million"). The match folds case, + curly quotes, dashes and the minus sign, spaces, the fraction slash and invisible + characters such as the soft hyphen, and nothing else: a superscript stays a + superscript, and a sign or a percent is never trimmed. `...` may elide the middle of + a quote as long as every part is in the passage, in order. The verified quote is + printed in the judge line, `judge (): SUPPORTED — "" — `, + and JSON gains a `quote` field. An `NEI` opinion asserts nothing and may quote + nothing. Cached opinions are now keyed by the passage they were checked against, and + those cached before this change are not served again: the next run asks. + +### Added +- **AVeriTeC eval: `--judge-nei` and `--no-fallback`** (`scripts/eval_averitec.py`). + `--judge-nei` (requires `--judge`) also sends every read source whose verdict is NEI + to the judge, with the passage the models came closest to deciding on, and the + hypothetical column gains a second number in which those opinions vote too, with its + per-label counts, its wrong-way count beside the product's, and an exact McNemar test + against the product claim by claim. The report also counts opinions by model and kind + of escalation, and the ones the quote check dropped by reason. `--no-fallback` + (requires `--judge`) turns the judge's local fallback off for the run, so every + opinion is from the configured model. Both + flags get their own results file and report name. Measurement only: the product + still never sends an NEI to the judge. + ## [0.4.11] - 2026-10-05 `proofpath update` keeps proofpath current, and it keeps the extras you installed. A diff --git a/README.md b/README.md index 0306c0a..401286f 100644 --- a/README.md +++ b/README.md @@ -442,9 +442,13 @@ were least sure about — the `low` tier, never a numeric mismatch and never a c without a quoted passage — go to the model in batches of up to 20 (about 7k tokens), each with its claim and the passage it was checked against. The model answers from the passage alone, and its label and one-sentence rationale are printed **beside** the local -verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding -kind and the report's states never change. Opinions are cached with the verdict, so a -re-run asks nothing. +verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. A `SUPPORTED` or `REFUTED` +opinion must also quote the words that decide it: at least two words, found in the +passage at word boundaries (after folding case, curly quotes, dashes and spacing; `...` +may elide a part). An opinion whose quote is not there is dropped, and the run says so +in a note; a verified quote is printed in the judge line before the rationale. The local +verdict, the finding kind and the report's states never change. Opinions are cached +with the verdict, so a re-run asks nothing. **What it cannot do.** It never sees a source document, so it cannot introduce a claim or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot @@ -511,7 +515,11 @@ The likely reason, not separately measured: the retrieval and entailment models calibrated on scientific abstracts, and a fact-check page is a different object (long, discursive, and usually quoting the claim it debunks). As shipped, the judge cannot help with those NEIs. An NEI verdict carries no passage, so it -is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Nothing was +is never escalated (rule 1); only 1 of the 165 read sources reached the judge. Shown each +NEI's closest passage in a measurement-only run, the judge answered NEI on 97 of 105, and +its 8 other opinions fixed 5 claims and broke none (0.427, exact McNemar p = 0.062). That +is just short of the bar set beforehand, so it is not in the product +([details](docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md)). Nothing was tuned after these measurements, and no blocked URL was dropped from them. What proofpath is good at is the academic path the other rows measure: finding out whether a diff --git a/docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md b/docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md new file mode 100644 index 0000000..b1a0152 --- /dev/null +++ b/docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md @@ -0,0 +1,124 @@ +# AVeriTeC dev — 2026-10-05 + +- claims: 100 (limit=100) +- measured path: gold source URLs, no search +- browser step: allowed +- fetch: cache allowed +- judge: groq openai/gpt-oss-120b (its opinions are the hypothetical column below; the verdicts are the models') +- NEI probes: on (every read NEI source was also sent to the judge, with its closest passage) +- judge fallback: off (every opinion is from the configured model) +- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json` (sha256 499793726b4a…) +- one run of the whole product: real claims, real source pages, real fetch ladder. + +## Headline + +3-way accuracy **0.371** vs majority baseline 0.708, over 89 of 100 claims. + +The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.330. + +## Per label + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | +| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 | + +## Source coverage + +| state | count | +|---|---| +| ok | 165 | +| UNVERIFIED (blocked, robots.txt) | 17 | +| not a url | 9 | +| UNVERIFIED (unreachable) | 8 | +| UNVERIFIED (blocked) | 4 | +| UNVERIFIED (reached, no text extracted) | 4 | +| UNVERIFIED (provider unavailable) | 2 | + +76 of 200 source URLs are web.archive.org snapshots (38.0 %). + +## Without the browser + +0 of 165 read sources were read by the browser step. Treating them as unread gives a 3-way accuracy of **0.371**. + +Those pages never reached the Wayback step, which a run without the browser tries next, so this is a lower bound on the pages such a run reads. It is not a bound on its accuracy: dropping a page can lose a right verdict or a wrong one. + +165 read sources came from the cache, so the step that read them is unknown; they are kept as read. + +## Judge (hypothetical) + +The product never lets the judge change a verdict (spec section 11.1). This column is what the accuracy would be if it did: a medium or high model verdict still wins; otherwise the judge's opinions on the escalated sources decide, by majority of the asserting votes (an NEI opinion abstains), a tie being NEI. An escalation the judge did not answer votes with the models' label. + +3-way accuracy **0.371**, against 0.371 for the product. 1 sources were escalated; 0 got no opinion back. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 0 | 0.000 | +| Refuted | 63 | 28 | 0.444 | +| Not Enough Evidence | 7 | 5 | 0.714 | + +| judge model | escalation | opinions | +|---|---|---| +| groq openai/gpt-oss-120b | nei | 105 | +| groq openai/gpt-oss-120b | escalated | 1 | + +| decided by | n | correct | accuracy | +|---|---|---|---| +| models (medium or high tier) | 41 | 28 | 0.683 | +| nothing escalated | 48 | 5 | 0.104 | + +### Band plus NEI probes + +`--judge-nei` also sent every read source whose verdict was NEI to the judge, as NEI at low tier, with the passage the models came closest to deciding on. Here those opinions vote too, by the same rule; a probe the judge did not answer abstains. + +3-way accuracy **0.427**, against 0.371 for the product. 105 NEI sources were probed; 0 got no opinion back. + +Against the product, claim by claim: 5 fixed, 0 broken (exact McNemar, two-sided p = 0.062). + +Refuted called SUPPORTED plus Supported called REFUTED, the wrong way: 11 (5 + 6) with the probes, against 11 (5 + 6) for the product. + +| label | n | correct | accuracy | +|---|---|---|---| +| Supported | 19 | 4 | 0.211 | +| Refuted | 63 | 29 | 0.460 | +| Not Enough Evidence | 7 | 5 | 0.714 | + +### Dropped by the quote check + +A SUPPORTED or REFUTED opinion whose quote is not in the passage it was shown is dropped (OPEN-ITEMS 20.9), and the source then counts as unanswered above. + +| escalation | reason | dropped | +|---|---|---| +| none | — | 0 | + +## Notes + +- **Same read pages as the 2026-10-05 gold run.** The 165 read sources are the same pages + as in `2026-10-05-averitec-browser-judge-fresh.md`. Every one of them carries no step, + which marks a cache hit, so they were most likely served from the fetch cache that run + filled. The per-claim predictions are identical, so the product column (0.371) is that + run's; only the judge's part is new. The unread sources differ slightly: 8 that were + "provider unavailable" there are "unreachable" here. No verdict changes because of it. +- **What the judge did with the NEIs.** 105 read sources came back NEI and were shown to + the judge with their closest passage. It answered NEI on 97, SUPPORTED on 4 and + REFUTED on 4. Counting those 8 as votes fixes 5 claims and breaks none: 0.427 against + 0.371, exact McNemar p = 0.062. The wrong-way count stays at 11 (5 + 6). `Supported` + goes from 0/19 to 4/19. +- **Below the bar set before the run.** OPEN-ITEMS §23 asked for p < 0.05 and no more + false SUPPORTED than the product (5 of 63). The second condition holds and the first + misses narrowly; with 5 discordant claims the smallest possible p is 0.0625. So this + is recorded, not shipped. A larger set (the 300 train claims planned for 20.14) could + settle it. +- **Most NEIs look like something other than a reading failure.** For 97 of the 105, + a strong LLM shown the closest passage also finds nothing that settles the claim. That + points to the deciding passage not being the closest one, or not being on the page: + retrieval and coverage rather than entailment. This is an inference, not a measurement; + the judge may also be cautious. +- **The quote check dropped nothing.** All 106 opinions (105 probes plus 1 escalation) + came from `groq openai/gpt-oss-120b` with no fallback, and every asserting opinion's + quote was found in its passage. The check costs nothing when the judge complies. +- Measured with the quote check of OPEN-ITEMS 20.9 (`#q1` cache key), `--no-fallback`, + and the default NLI profile. + diff --git a/docs/superpowers/OPEN-ITEMS.md b/docs/superpowers/OPEN-ITEMS.md index a70d4fe..5e2002e 100644 --- a/docs/superpowers/OPEN-ITEMS.md +++ b/docs/superpowers/OPEN-ITEMS.md @@ -698,3 +698,30 @@ notice outside the lock. If a concurrent re-switch announces its own notice insi microsecond window, the TUI line can show the back notice while `switched` (the footer) holds the cooling one. Fix: announce under a separate `_announce_lock`, and only when `switched` still equals the notice. + +## 23. Judge quote check (20.9) and NEI → judge (20.13) — 2026-10-05 + +The user asked for both items to be finished end to end, with the best options picked +without further questions. Design: `docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md`. + +- **20.9** (product): the review answer gains a `quote` field. A SUPPORTED or REFUTED + opinion whose quote is not in the passage, after normalisation and an + in-order check of ellipsis parts, is dropped and noted. Cached judgements from + before the change are not served as checked. +- **20.13** (measurement only): `--judge-nei` shows each NEI source's closest passage + to the judge, and `--no-fallback` keeps every opinion on Groq. It is a second + hypothetical column, compared claim by claim with the product. Shipping it needs + p < 0.05 and no more false SUPPORTED than the product (5/63); that stays a separate + decision. + +**Result (2026-10-05, `docs/eval/2026-10-05-averitec-browser-judge-nei-nofallback.md`).** + +- The judge answered 97 of 105 NEI probes NEI, 4 SUPPORTED and 4 REFUTED. +- Band plus probes: 0.427 against 0.371. 5 claims fixed, 0 broken, exact McNemar + p = 0.062. Wrong-way verdicts unchanged at 11. +- **Not shipped:** p misses the 0.05 bar. With 5 discordant claims, 0.0625 is the floor. + Re-measure on the 300 train claims of 20.14 before deciding. +- Most NEIs look like something other than a reading failure: the closest passage does + not settle the claim even for the LLM. That points (an inference, not a measurement) + to retrieval and coverage of the deciding passage as the next lever. +- **20.9 in the live run:** 0 of 106 opinions dropped by the quote check. diff --git a/docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md b/docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md new file mode 100644 index 0000000..e6d7f3b --- /dev/null +++ b/docs/superpowers/specs/2026-10-05-judge-quote-and-nei-design.md @@ -0,0 +1,132 @@ +# Judge quote check (20.9) and the NEI → judge measurement (20.13) — design (2026-10-05) + +The user asked for both items to be done end to end, with the best option chosen +without further questions. The choices below were made on that mandate and are written +here so they can be revisited. OPEN-ITEMS 20.9 and 20.13 point here. + +## 1. 20.9: the judge's quote is checked (product change) + +**Problem.** `review.md` asks for a rationale that "repeats the deciding words from the +passage verbatim", but nothing checks that it does. A judge that answers from memory, +or paraphrases, still gets its opinion attached beside the verdict. Rule 1 ("never +assert without a passage") binds the judge as much as the pipeline (spec §11). + +**Decision.** + +1. **A separate `quote` field.** The review schema and prompt add `"quote"`: the exact + words from the passage that decide the item. The rationale stays the one-sentence + reason. A structured field can be checked; a quote buried in prose cannot. +2. **Checked for asserting labels only.** A `SUPPORTED` or `REFUTED` opinion whose + quote does not occur in the passage is dropped. The drop is noted through + `Judge.skipped` as `": the quoted words are not in the passage; opinion + dropped"`, which the run already reports. An asserting opinion that came back with + no quote at all is noted separately, as `": no quoted words came back; opinion + dropped"`, so the report can count the two reasons apart. An `NEI` opinion may have + an empty quote: it asserts nothing. +3. **Matching** is exact after one normalisation, applied to both sides (revised + after review, 2026-10-05): + - Unicode NFC (not NFKC: NFKC turns `10⁶` into `106`, a false numeric match); + - every Unicode space separator (`Zs`, the no-break space among them) becomes a + space; + - curly quotes and apostrophes become straight ones; + - every `Pd` dash and the minus sign U+2212 become `-`; + - the fraction slash U+2044 becomes `/`; + - every format character (`Cf`: the soft hyphen U+00AD, the zero-width space + U+200B and the like) is removed; + - every whitespace run becomes one space; + - then casefold. A documented limit: "MW" equals "mW". Ligatures and full-width + letters are not folded, so a model that types "fi" for "fi" fails. + + The quote is then split at every ellipsis (`...` or `…`). Each part loses the + whitespace, quote marks, brackets and sentence punctuation (`.,;:!?`) at its ends, + and never `-`, `%` or `+`: "−3.2%" must not become "3.2". The checks then run in + this order, and the first failure is the reason: + - **no quote**: there is not a single word (`\w+`) left; + - **too short**: a part has fewer than two words. One word ("the", "e", a "5") + is in nearly every passage and proves nothing; + - **not in the passage**: some part does not occur in the passage at word + boundaries (the characters before and after the match are not letters or digits, + or are the passage's ends), in order and without overlapping. "5 million" is not + in "15 million", and "ploy" is not in "Unemployment". + + Each reason has its own note in `Judge.skipped` (`QUOTE_MISSING`, + `QUOTE_TOO_SHORT`, `QUOTE_NOT_FOUND`). +4. **No stale opinions, and the quote is shown.** Judgements cached before this change + were never checked, so the cache must not serve them as checked. The judgement + cache key gains `#q1` (the checks an opinion passed) and a 16-hex-digit sha256 of + the whitespace-folded passage it was checked against: + `#q1#`. The passage is in the key because the check is + about that passage: the same claim and source can be decided on another passage + after an NLI profile or `k` change, and an opinion checked against one says nothing + about the other. `JudgeOpinion` gains `quote: str = ""`: the verified quote less + its wrapping, empty for an NEI that quoted nothing or quoted wrongly (an NEI is + kept either way, but a quote the report shows is always a checked one). It is + stored in a `quote` column, added to older files by a guarded migration + (`PRAGMA table_info`, then `ALTER TABLE judgements ADD COLUMN quote TEXT NOT NULL + DEFAULT ''`). The judge line shows it as a quoted span before the rationale, + `judge (): SUPPORTED — "" — `, on every surface; JSON + gains the field and renames nothing. +5. **The batch cap holds the quote.** Every opinion now carries a quote. A typical + one is about 50 tokens; budgeted at 100 for a long quote, the existing per-batch + item cap (`BATCH_SIZE` = 20) writes 2000 tokens, under half of the review's 4096 + (`_REVIEW_TOKENS`). The other half is left to a reasoning model's thinking, which + it charges to the same budget. The arithmetic is a comment beside `_OPINION_TOKENS` + and a test. +6. The judge's other jobs, query writing and the summary, are unchanged. + +## 2. 20.13: what the judge would do with the NEIs (measurement only) + +**Problem.** On AVeriTeC dev (2026-10-05), 36 of 89 claims had a readable source and +came back NEI where the gold label was Supported or Refuted. `aggregate` drops the +passage of an NEI verdict (`pipeline.py`), so `verify.escalates` never sends an NEI to +the judge. Only 1 of 165 read sources was escalated. + +**Decision.** + +1. **The closest passage.** For each read source whose verdict is NEI, the eval takes + the passage the models came closest to deciding on: the one with the highest + SUPPORTED or REFUTED probability, which `aggregate` already finds and then + discards. Expose it through a small pure helper used by `aggregate`. **The + product's verdicts must not change**, and existing pipeline tests must stay green + unmodified. +2. **`--judge-nei`** (requires `--judge`) sends those NEI sources to the judge with + `verdict=NEI` and `tier=low`, beside the normally escalated ones. Each source + records which kind of escalation it was: `escalated` for the product's band and + `nei` for this probe. The hypothetical column then reports two numbers: + - the product's band only (as today); + - the product's band plus the NEI probes. +3. **`--no-fallback`** (requires `--judge`) sets `judge.fallback = off` for the run. The provider then waits + out a 429 as it normally does without a fallback, so every opinion is from the + configured model (`groq openai/gpt-oss-120b`), not a mix. +4. **Quote check on.** The run uses 20.9, so an opinion whose quote is not in the + passage is dropped and counted. The report says how many: per kind of escalation, + and per reason (no quote, a part under two words, quote not in the passage). +5. The setup suffix in file names gains `-judge-nei` and `-nofallback`. +6. **The run.** Gold URLs, browser allowed, cache allowed: today's pages are within the + 7-day TTL, so this reads the same pages as the 2026-10-05 gold run. `--sleep 1.0`, + 100 dev claims. +7. **Reading the result.** The NEI-probe column is compared with the product, + claim by claim (exact McNemar), and its wrong-way count is reported beside the + product's. Whether the product should escalate NEI with its closest passage stays a + separate decision. The bar is a gain that clears the noise (p < 0.05) with no more + false claims called SUPPORTED than the product (5 of 63). Unless it clears that bar, + the item is recorded and not shipped. + +## 3. Tests (no network) + +- **Quote check:** + - exact and normalised matches pass; + - a paraphrase fails; + - an ellipsis with parts in order passes, and with parts out of order fails; + - an empty quote fails for SUPPORTED and REFUTED but passes for NEI; + - the dropped opinion's note reaches `Judge.skipped`. +- **Cache:** an opinion stored under the old key is not served after the change. +- **Pipeline:** the helper returns the passage `aggregate` would have chosen, and + `aggregate`'s verdicts are unchanged (existing tests untouched). +- **Eval:** + - `--judge-nei` without `--judge` is a usage error; + - NEI sources are sent with their closest passage and recorded as `nei`; + - the two hypothetical numbers and the drop counts are scored and rendered; + - `--no-fallback` builds a client without a fallback; + - the new file-name suffixes are used; + - old results files still load. diff --git a/scripts/eval_averitec.py b/scripts/eval_averitec.py index a1ecd4e..bcd4687 100644 --- a/scripts/eval_averitec.py +++ b/scripts/eval_averitec.py @@ -2,8 +2,8 @@ Usage: uv run python scripts/eval_averitec.py [--limit 100] [--browser | --no-browser] - [--judge] [--fresh] [--sleep 1.0] [--resume] [--search] - [--out docs/eval/-averitec.md] + [--judge [--judge-nei] [--no-fallback]] [--fresh] [--sleep 1.0] [--resume] + [--search] [--out docs/eval/-averitec.md] SciFact measures retrieval and entailment over abstracts we are handed. AVeriTeC measures the whole product: a real-world claim, its real source pages, fetched over @@ -25,6 +25,14 @@ in which the judge's opinions decide the low band. The product never does the second (spec section 11.1); the report says so. +``--judge-nei`` measures what the judge would do with the NEIs (OPEN-ITEMS 20.13): +every read source whose verdict is NEI is also sent, with the passage the models came +closest to deciding on (``pipeline.closest``), and the hypothetical column gains a +second number in which those opinions vote too. The judge's quote is checked +(OPEN-ITEMS 20.9), so the report counts the opinions that check dropped. +``--no-fallback`` keeps every opinion from the configured model: a 429 is waited out +instead of handed to the local one. + The live half below ``main`` is run by hand. Importing this module touches neither the network nor a model. """ @@ -33,11 +41,12 @@ import argparse import json +import math import sys import time from collections import Counter from collections.abc import Callable, Sequence -from dataclasses import asdict, dataclass, replace +from dataclasses import asdict, dataclass, field, replace from datetime import date from pathlib import Path from typing import TYPE_CHECKING @@ -87,6 +96,21 @@ NOTHING_ESCALATED = "nothing escalated" # A row from a results file written before sources were kept: only its label is known. UNRECORDED = "not recorded (row has no per-source record)" +# The two kinds of escalation a source can have been sent to the judge under: the +# product's own band (``verify.escalates``), and the ``--judge-nei`` probe of an NEI. +ESCALATED = "escalated" +NEI_PROBE = "nei" +# Why the quote check dropped a judge's opinion, keyed by the judge's own note. Read +# off ``Judge.skipped`` by exact match on the judge's public wording, never by +# substring, so a note the judge writes for another reason is never counted here. +DROP_NO_QUOTE = "no quote" +DROP_TOO_SHORT = "a quoted part under two words" +DROP_NOT_IN_PASSAGE = "quote not in the passage" +_DROP_REASONS = { + judge_mod.QUOTE_MISSING: DROP_NO_QUOTE, + judge_mod.QUOTE_TOO_SHORT: DROP_TOO_SHORT, + judge_mod.QUOTE_NOT_FOUND: DROP_NOT_IN_PASSAGE, +} @dataclass(frozen=True) @@ -96,8 +120,14 @@ class Source: ``step`` is the ladder step that read the page, or the last one tried when it failed; ``None`` for a cache hit, which carries no step. ``label`` is the models' verdict, ``None`` when nothing was read. ``escalated`` says the verdict sat in the - band the product hands the judge (``verify.escalates``); ``judge_label`` and - ``judge_model`` are the judge's opinion of it, when one came back. + band the product hands the judge (``verify.escalates``); ``probed`` says it was an + NEI sent under ``--judge-nei`` with its closest passage. ``judge_label`` and + ``judge_model`` are the judge's opinion of it, when one came back; ``dropped`` is + why the quote check threw one away (``DROP_NO_QUOTE``, ``DROP_TOO_SHORT``, + ``DROP_NOT_IN_PASSAGE``). + + ``probed`` and ``dropped`` are new on 2026-10-05 and default to what an older + results file meant without them: no probe, nothing dropped. """ url: str @@ -109,6 +139,18 @@ class Source: escalated: bool = False judge_label: str | None = None judge_model: str | None = None + probed: bool = False + dropped: str | None = None + + @property + def escalation(self) -> str | None: + """Which kind of escalation sent this source to the judge, if any. + + Derived from the two flags rather than stored beside them, so it cannot + disagree with them, and an older file's ``escalated`` keeps its meaning.""" + if self.escalated: + return ESCALATED + return NEI_PROBE if self.probed else None @dataclass(frozen=True) @@ -152,7 +194,7 @@ def product_label(sources: Sequence[Source]) -> str | None: return Label.NEI.value if any(source.label is not None for source in sources) else None -def judged_label(sources: Sequence[Source]) -> str | None: +def judged_label(sources: Sequence[Source], *, probes: bool = False) -> str | None: """The claim's label if the judge's opinions decided the low band. The product never does this (spec section 11.1): this is the column that says @@ -161,11 +203,15 @@ def judged_label(sources: Sequence[Source]) -> str | None: votes with the judge's label, or with the models' label when the judge gave no opinion. An NEI vote abstains; the majority of the asserting votes wins, and a tie is NEI. + + ``probes`` lets the ``--judge-nei`` probes vote too, by the same rule. A probe's + own label is NEI, so a probe the judge did not answer abstains: it can only move + the claim through an opinion that passed the quote check. """ confident = [s for s in sources if _asserts(s.label) and not s.escalated] if confident: return _best(confident).label - votes = [s.judge_label or s.label for s in sources if s.escalated] + votes = [s.judge_label or s.label for s in sources if s.escalated or (probes and s.probed)] tally = Counter(vote for vote in votes if _asserts(vote)).most_common() if tally: if len(tally) > 1 and tally[0][1] == tally[1][1]: @@ -193,7 +239,15 @@ def without_browser(sources: Sequence[Source]) -> tuple[Source, ...]: """The sources as a run without the browser would have them: the pages the browser read are unread. A cache hit has no step, so it is kept (the report counts those).""" return tuple( - replace(source, label=None, escalated=False, judge_label=None, judge_model=None) + replace( + source, + label=None, + escalated=False, + judge_label=None, + judge_model=None, + probed=False, + dropped=None, + ) if source.step == BROWSER_STEP and source.label is not None else source for source in sources @@ -316,6 +370,29 @@ class Ablation: accuracy_3way: float +@dataclass(frozen=True) +class ProbedScore: + """The hypothetical column's second number: the product's band plus the NEI probes + (``--judge-nei``), and how it stands against the product claim by claim. + + ``wrong_way`` is (Refuted called SUPPORTED, Supported called REFUTED): the errors + that assert the opposite of the truth, which an accuracy gain can hide. The bar in + the spec (2026-10-05 section 2.7) is no more false claims called SUPPORTED than the + product, so the product's own pair is carried beside it. ``fixed`` and ``broken`` + are the discordant claims of the McNemar test against the product's outcome. + """ + + accuracy_3way: float + per_label: dict[str, tuple[int, int]] # gold label -> (n, correct) + probed: int # NEI sources sent with their closest passage + unanswered: int # of those, how many got no opinion back + wrong_way: tuple[int, int] + product_wrong_way: tuple[int, int] + fixed: int # claims the product got wrong and this got right + broken: int # claims the product got right and this got wrong + p_value: float # exact two-sided McNemar over ``fixed`` and ``broken`` + + @dataclass(frozen=True) class JudgedScore: """The hypothetical column: what ``judged_label`` would have scored.""" @@ -324,9 +401,14 @@ class JudgedScore: per_label: dict[str, tuple[int, int]] # gold label -> (n, correct) escalated: int # sources in the judge's band unanswered: int # of those, how many got no opinion back - opinions: Counter[str] # judge model -> opinions it gave + # (judge model, escalation kind) -> opinions it gave: the NEI probes counted + # apart from the product's band, so neither number borrows the other's answers. + opinions: Counter[tuple[str, str]] deciders: dict[str, tuple[int, int]] # ``judge_decider`` -> (3-way n, correct) unrecorded: int # rows with URLs and no per-source record: scored as the product did + # (escalation kind, drop reason) -> opinions the quote check dropped (OPEN-ITEMS 20.9) + drops: Counter[tuple[str, str]] = field(default_factory=Counter) + probed: ProbedScore | None = None # ``None`` unless the run had ``--judge-nei`` @dataclass(frozen=True) @@ -393,7 +475,66 @@ def _judged_label_of(row: Row) -> str | None: return judged_label(row.sources) if row.sources else row.predicted -def _judged(rows: Sequence[Row]) -> JudgedScore: +def _probes_label_of(row: Row) -> str | None: + return judged_label(row.sources, probes=True) if row.sources else row.predicted + + +def _is_right(label: str | None, gold: Label) -> bool: + return (Label(label) if label is not None else Label.NEI) is gold + + +def _wrong_way(rows: Sequence[Row], label_of: Callable[[Row], str | None]) -> tuple[int, int]: + """(Refuted called SUPPORTED, Supported called REFUTED) under one way of labelling.""" + refuted_supported = supported_refuted = 0 + for row in rows: + gold = averitec.to_label(row.gold) + predicted = label_of(row) + refuted_supported += int(gold is Label.REFUTED and predicted == Label.SUPPORTED.value) + supported_refuted += int(gold is Label.SUPPORTED and predicted == Label.REFUTED.value) + return refuted_supported, supported_refuted + + +def mcnemar_exact(b: int, c: int) -> float: + """The exact two-sided McNemar p-value for ``b`` and ``c`` discordant pairs. + + Exact rather than the chi-squared approximation: a hundred claims leave a handful + of discordant pairs, which is where the approximation is worst. Under the null + each discordant claim is a fair coin, so ``min(b, c)`` is a Binomial(b + c, 1/2) + tail, doubled and capped at 1. No discordant pair at all is no evidence: 1.0. + """ + n = b + c + if n == 0: + return 1.0 + tail = sum(math.comb(n, i) for i in range(min(b, c) + 1)) / (1 << n) + return min(1.0, 2.0 * tail) + + +def _probed(rows: Sequence[Row]) -> ProbedScore: + accuracy, per_label = _three_way(rows, _probes_label_of) + probes = [s for row in rows for s in row.sources if s.probed] + fixed = broken = 0 + for row in rows: + gold = averitec.to_label(row.gold) + if gold is None: + continue + product = _is_right(row.predicted, gold) + probed = _is_right(_probes_label_of(row), gold) + fixed += int(probed and not product) + broken += int(product and not probed) + return ProbedScore( + accuracy_3way=accuracy, + per_label=per_label, + probed=len(probes), + unanswered=sum(1 for s in probes if s.judge_label is None), + wrong_way=_wrong_way(rows, _probes_label_of), + product_wrong_way=_wrong_way(rows, lambda row: row.predicted), + fixed=fixed, + broken=broken, + p_value=mcnemar_exact(broken, fixed), + ) + + +def _judged(rows: Sequence[Row], *, probes: bool = False) -> JudgedScore: accuracy, per_label = _three_way(rows, _judged_label_of) escalated = [s for row in rows for s in row.sources if s.escalated] deciders: dict[str, tuple[int, int]] = {} @@ -411,15 +552,31 @@ def _judged(rows: Sequence[Row]) -> JudgedScore: per_label=per_label, escalated=len(escalated), unanswered=sum(1 for s in escalated if s.judge_label is None), - opinions=Counter(s.judge_model for s in escalated if s.judge_model is not None), + opinions=Counter( + (s.judge_model, s.escalation) + for row in rows + for s in row.sources + if s.judge_model is not None and s.escalation is not None + ), deciders=deciders, unrecorded=sum(1 for row in rows if _unrecorded(row)), + # A dropped opinion was asked for, so its source always has an escalation. + drops=Counter( + (s.escalation, s.dropped) + for row in rows + for s in row.sources + if s.dropped is not None and s.escalation is not None + ), + probed=_probed(rows) if probes else None, ) -def score(rows: Sequence[Row], *, judge: bool = False) -> AveritecResult: +def score(rows: Sequence[Row], *, judge: bool = False, probes: bool = False) -> AveritecResult: """Turn decided rows into the numbers the report prints. + ``probes`` adds the hypothetical column's second number (``--judge-nei``); it means + nothing without ``judge``. + A row with no prediction counts as NEI: having fetched nothing, NEI is the only thing the run honestly asserted. A row whose gold label is Conflicting can never be right — ``to_label`` gives it no verdict to match — so it is wrong in the @@ -458,7 +615,7 @@ def score(rows: Sequence[Row], *, judge: bool = False) -> AveritecResult: archive_urls=archive_urls, total_urls=total_urls, ablation=_ablation(rows), - judged=_judged(rows) if judge else None, + judged=_judged(rows, probes=probes) if judge else None, ) @@ -475,11 +632,15 @@ def render_report( browser: bool | None = None, fresh: bool = False, judge: str = "", + judge_nei: bool = False, + no_fallback: bool = False, ) -> str: """The markdown skeleton. ``## Notes`` is left for the controller to fill in. ``browser`` is ``None`` when the run left the browser to the config, so the report claims nothing about it; ``judge`` is the judge's name, empty when there was none. + ``judge_nei`` and ``no_fallback`` are the run's ``--judge-nei`` and + ``--no-fallback``, stated in the header so two setups' reports cannot be confused. """ # ``result.counted`` is the denominator the 3-way accuracy and its baseline are # read over. Printed beside `n`, because "0.80 over 100 claims" would otherwise @@ -500,6 +661,13 @@ def render_report( else "writes the search queries; its opinions are the hypothetical column below" ) lines.append(f"- judge: {judge} ({role})") + if judge_nei: + lines.append( + "- NEI probes: on (every read NEI source was also sent to the judge, " + "with its closest passage)" + ) + if no_fallback: + lines.append("- judge fallback: off (every opinion is from the configured model)") lines += [ f"- dataset: `{averitec.URL}` (sha256 {averitec.SHA256[:12]}…)", "- one run of the whole product: real claims, real source pages, real fetch ladder.", @@ -607,43 +775,127 @@ def _judged_lines(judged: JudgedScore, *, product: float) -> list[str]: for label in [label for label in averitec.LABELS if label in judged.per_label]: seen, right = judged.per_label[label] lines.append(_md_row(label, seen, right, f"{right / seen:.3f}" if seen else "—")) - lines.extend(["", "| judge model | opinions |", "|---|---|"]) - for model, count in judged.opinions.most_common(): - lines.append(_md_row(model, count)) + lines.extend(["", "| judge model | escalation | opinions |", "|---|---|---|"]) + for (model, kind), count in judged.opinions.most_common(): + lines.append(_md_row(model, kind, count)) if not judged.opinions: - lines.append(_md_row("none", 0)) + lines.append(_md_row("none", "—", 0)) lines.extend(["", "| decided by | n | correct | accuracy |", "|---|---|---|---|"]) for decider, (seen, right) in sorted(judged.deciders.items()): lines.append(_md_row(decider, seen, right, f"{right / seen:.3f}" if seen else "—")) + if judged.probed is not None: + lines.extend(_probed_lines(judged.probed, product=product)) + lines.extend(_drop_lines(judged.drops)) + return lines + + +def _wrong_way_text(pair: tuple[int, int]) -> str: + return f"{sum(pair)} ({pair[0]} + {pair[1]})" + + +def _probed_lines(probed: ProbedScore, *, product: float) -> list[str]: + """The second hypothetical number. Reading it (spec 2026-10-05 section 2.7): it + is a case for escalating NEIs only if the gain clears the noise (p < 0.05) and it + calls no more false claims supported than the product does.""" + lines = [ + "", + "### Band plus NEI probes", + "", + "`--judge-nei` also sent every read source whose verdict was NEI to the judge, " + "as NEI at low tier, with the passage the models came closest to deciding on. " + "Here those opinions vote too, by the same rule; a probe the judge did not " + "answer abstains.", + "", + f"3-way accuracy **{probed.accuracy_3way:.3f}**, against {product:.3f} for the " + f"product. {probed.probed} NEI sources were probed; {probed.unanswered} got no " + "opinion back.", + "", + f"Against the product, claim by claim: {probed.fixed} fixed, {probed.broken} " + f"broken (exact McNemar, two-sided p = {probed.p_value:.3f}).", + "", + "Refuted called SUPPORTED plus Supported called REFUTED, the " + f"wrong way: {_wrong_way_text(probed.wrong_way)} with the probes, against " + f"{_wrong_way_text(probed.product_wrong_way)} for the product.", + "", + "| label | n | correct | accuracy |", + "|---|---|---|---|", + ] + for label in [label for label in averitec.LABELS if label in probed.per_label]: + seen, right = probed.per_label[label] + lines.append(_md_row(label, seen, right, f"{right / seen:.3f}" if seen else "—")) + return lines + + +def _drop_lines(drops: Counter[tuple[str, str]]) -> list[str]: + lines = [ + "", + "### Dropped by the quote check", + "", + "A SUPPORTED or REFUTED opinion whose quote is not in the passage it was shown " + "is dropped (OPEN-ITEMS 20.9), and the source then counts as unanswered above.", + "", + "| escalation | reason | dropped |", + "|---|---|---|", + ] + for (kind, reason), count in sorted(drops.items()): + lines.append(_md_row(kind, reason, count)) + if not drops: + lines.append(_md_row("none", "—", 0)) return lines # --- live half: run by hand, not covered by tests ------------------------ -def _setup_suffix(*, browser: bool, judge: bool, fresh: bool) -> str: - """What sets this run apart from the legacy setup, for its file names.""" +def _setup_suffix( + *, browser: bool, judge: bool, fresh: bool, judge_nei: bool = False, no_fallback: bool = False +) -> str: + """What sets this run apart from the legacy setup, for its file names. + + ``-judge-nei`` takes the place of ``-judge``, which it requires, rather than + following it: ``-judge-judge-nei`` would say the same thing twice.""" + judged = "-judge-nei" if judge_nei else "-judge" if judge else "" return ( - ("-browser" if browser else "") + ("-judge" if judge else "") + ("-fresh" if fresh else "") + ("-browser" if browser else "") + + judged + + ("-nofallback" if no_fallback else "") + + ("-fresh" if fresh else "") ) def _results_path( - *, search: bool, browser: bool = False, judge: bool = False, fresh: bool = False + *, + search: bool, + browser: bool = False, + judge: bool = False, + fresh: bool = False, + judge_nei: bool = False, + no_fallback: bool = False, ) -> Path: """One file per run setup, so a resume does not mix rows from two setups. The legacy names stay the runs made without ``--browser``, ``--judge`` or ``--fresh``; they are shared by ``--no-browser`` and by a run that leaves the browser to the config, as they always were, so that older files can still be resumed.""" name = Path(SEARCH_RESULTS_NAME if search else RESULTS_NAME) - suffix = _setup_suffix(browser=browser, judge=judge, fresh=fresh) + suffix = _setup_suffix( + browser=browser, judge=judge, fresh=fresh, judge_nei=judge_nei, no_fallback=no_fallback + ) return cache_dir() / "datasets" / f"{name.stem}{suffix}{name.suffix}" -def _report_path(*, day: str, search: bool, browser: bool, judge: bool, fresh: bool) -> Path: +def _report_path( + *, + day: str, + search: bool, + browser: bool, + judge: bool, + fresh: bool, + judge_nei: bool = False, + no_fallback: bool = False, +) -> Path: """The default ``--out``: one report per run setup, so two runs on one day keep both.""" suffix = ("-search" if search else "") + _setup_suffix( - browser=browser, judge=judge, fresh=fresh + browser=browser, judge=judge, fresh=fresh, judge_nei=judge_nei, no_fallback=no_fallback ) return Path("docs/eval") / f"{day}-averitec{suffix}.md" @@ -695,6 +947,7 @@ def _decide_claim( anonymous: bool = False, fresh: bool = False, judge: Judge | None = None, + judge_nei: bool = False, ) -> Row: """Fetch every source of one claim and keep the strongest non-NEI verdict. @@ -702,11 +955,14 @@ def _decide_claim( as the product fetches them (round 2). ``fresh`` climbs past the cache, so every source keeps the step that read it. ``judge`` is asked about the sources whose verdict sits in the product's escalation band; its opinions are recorded beside - the verdicts and never change ``predicted`` (spec section 11.1).""" + the verdicts and never change ``predicted`` (spec section 11.1). ``judge_nei`` also + asks it about every read NEI source, shown the closest passage (OPEN-ITEMS 20.13).""" states: list[str] = [] sources: list[Source] = [] # What the judge is shown for each escalated source, by its index in ``sources``. escalated: dict[int, Verdict] = {} + # The closest passage of each NEI source probed under ``judge_nei``, the same way. + probes: dict[int, Passage] = {} for url in claim.source_urls: # Before every fetch, not between them: the ladder has no crawl delay of its # own, so this is the whole of the run's politeness and it has to hold across @@ -724,7 +980,9 @@ def _decide_claim( Passage(sentence, url, ordinal) for ordinal, sentence in enumerate(retrieval.split_sentences(fetched.text)) ] - verdict = pipeline.decide( + # ``decide``'s verdict, unchanged, and the passage an NEI verdict drops: one + # embedding and one scoring for both, so a probe never re-reads the page. + verdict, nearest = pipeline.decide_closest( claim.text, passages, engine.get_embedder(), @@ -732,8 +990,13 @@ def _decide_claim( k=engine.k, thresholds=engine.thresholds, ) + index = len(sources) if escalates(verdict): - escalated[len(sources)] = verdict + escalated[index] = verdict + elif judge is not None and judge_nei and verdict.label is Label.NEI and nearest is not None: + # No closest passage means nothing scored above zero: there is no passage + # to show, and asking without one is what rule 1 forbids. + probes[index] = nearest # A source whose verdict is NEI was still read: it stays in the row, and # ``product_label`` keeps only assertions, each with the passage it rests on # (product rule 1, enforced by ``Verdict`` itself). @@ -746,10 +1009,11 @@ def _decide_claim( score=verdict.score, tier=verdict.tier, escalated=escalates(verdict), + probed=index in probes, ) ) - if judge is not None and escalated: - sources = _ask_judge(judge, claim.text, sources, escalated) + if judge is not None and (escalated or probes): + sources = _ask_judge(judge, claim.text, sources, escalated, probes) # One state per value the dataset gave, fetchable or not, so the coverage table # accounts for every annotation rather than only the ones we could try. states.extend([NOT_A_URL] * claim.non_urls) @@ -766,32 +1030,61 @@ def _decide_claim( def _ask_judge( - judge: Judge, claim: str, sources: list[Source], escalated: dict[int, Verdict] + judge: Judge, + claim: str, + sources: list[Source], + escalated: dict[int, Verdict], + probes: dict[int, Passage] | None = None, ) -> list[Source]: """The judge's opinion on each escalated source, shown exactly what ``verify`` - shows it: the claim, the passage, the models' verdict and its tier.""" - items = [ - judge_mod.JudgeItem( - id=f"s{index}", - claim=claim, - passage=verdict.passage.text, - verdict=verdict.label, - tier=verdict.tier, + shows it: the claim, the passage, the models' verdict and its tier. + + A probe is shown its closest passage as an NEI at ``low``, which is what the + product would send were it to escalate NEIs. Both kinds go in one ``review``, and + an opinion the quote check dropped is recorded on its source by reason.""" + shown: dict[int, judge_mod.JudgeItem] = {} + for index, verdict in escalated.items(): + if verdict.passage is not None: # ``escalates`` already refused the others + shown[index] = judge_mod.JudgeItem( + id=f"s{index}", + claim=claim, + passage=verdict.passage.text, + verdict=verdict.label, + tier=verdict.tier, + ) + for index, passage in (probes or {}).items(): + shown[index] = judge_mod.JudgeItem( + id=f"s{index}", claim=claim, passage=passage.text, verdict=Label.NEI, tier="low" ) - for index, verdict in escalated.items() - if verdict.passage is not None # ``escalates`` already refused the others - ] - opinions = judge.review(items) + opinions = judge.review([shown[index] for index in sorted(shown)]) + # Read straight after the call: ``review`` resets ``skipped`` on every call, so + # these are this claim's notes and no other's. + dropped = _drops(judge.skipped) answered = list(sources) - for index in escalated: + for index in shown: opinion = opinions.get(f"s{index}") if opinion is not None: answered[index] = replace( sources[index], judge_label=opinion.label.value, judge_model=opinion.model ) + elif f"s{index}" in dropped: + answered[index] = replace(sources[index], dropped=dropped[f"s{index}"]) return answered +def _drops(skipped: Sequence[str]) -> dict[str, str]: + """Item id -> why the quote check dropped its opinion, from ``Judge.skipped``. + + The judge writes ``": "`` for a quote it refused; every other note is + about something else (no JSON, an unknown id, no rationale) and is left out.""" + found: dict[str, str] = {} + for note in skipped: + ident, separator, reason = note.partition(": ") + if separator and reason in _DROP_REASONS: + found[ident] = _DROP_REASONS[reason] + return found + + NLI_PROFILE: NliProfileName = "default" @@ -812,6 +1105,18 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="ask the configured judge about the escalated verdicts (a hypothetical column)", ) + parser.add_argument( + "--judge-nei", + action="store_true", + help="also send every read NEI source to the judge with its closest passage " + "(requires --judge; a second hypothetical number)", + ) + parser.add_argument( + "--no-fallback", + action="store_true", + help="never switch the judge to its local fallback, so every opinion is from " + "the configured model (requires --judge; sets judge.fallback = off for this run)", + ) parser.add_argument( "--fresh", action="store_true", @@ -826,10 +1131,23 @@ def main(argv: list[str] | None = None) -> int: ) parser.add_argument("--out", default="") args = parser.parse_args(argv) - - results = _results_path( - search=args.search, browser=args.browser, judge=args.judge, fresh=args.fresh - ) + if args.judge_nei and not args.judge: + # A usage error, not a silent no-op: a run asked to probe the NEIs that had no + # judge to probe them with would report a column it never measured. + parser.error("--judge-nei requires --judge") + if args.no_fallback and not args.judge: + # The same refusal: with no judge there is no fallback to turn off, and the + # "-nofallback" file name would label a run that is the plain one. + parser.error("--no-fallback requires --judge") + setup = { + "browser": args.browser, + "judge": args.judge, + "fresh": args.fresh, + "judge_nei": args.judge_nei, + "no_fallback": args.no_fallback, + } + + results = _results_path(search=args.search, **setup) if results.exists() and not args.resume: # The file is the only record of a run that costs hours of network; a fresh # run must not quietly replace it. Checked before anything is downloaded, so @@ -842,6 +1160,10 @@ def main(argv: list[str] | None = None) -> int: return 2 config = load_config() + if args.no_fallback: + # On the config the judge is built from, so ``build_client`` hands its + # ``FallbackClient`` "off" and a 429 is waited out on the configured model. + config = replace(config, judge=replace(config.judge, fallback=judge_mod.FALLBACK_OFF)) key: judge_mod.ApiKey | None = None if args.judge: # Checked before anything is downloaded, as the CLI's ``--judge`` checks it: a @@ -911,7 +1233,14 @@ def main(argv: list[str] | None = None) -> int: continue if searcher is None: stored.append( - _decide_claim(engine, claim, sleep=args.sleep, fresh=args.fresh, judge=judge) + _decide_claim( + engine, + claim, + sleep=args.sleep, + fresh=args.fresh, + judge=judge, + judge_nei=args.judge_nei, + ) ) else: # The gold URLs are the answer key; search mode reads what evidence @@ -938,6 +1267,7 @@ def main(argv: list[str] | None = None) -> int: anonymous=True, fresh=args.fresh, judge=judge, + judge_nei=args.judge_nei, ) ) scored += 1 @@ -968,26 +1298,22 @@ def main(argv: list[str] | None = None) -> int: wanted = {claim.id for claim in claims} rows = [row for row in stored if row.claim_id in wanted] report = render_report( - score(rows, judge=args.judge), + score(rows, judge=args.judge, probes=args.judge_nei), date=date.today().isoformat(), limit=args.limit, path=path, browser=browser, fresh=args.fresh, judge=judge_name, + judge_nei=args.judge_nei, + no_fallback=args.no_fallback, ) print() print(report) out = ( Path(args.out) if args.out - else _report_path( - day=date.today().isoformat(), - search=args.search, - browser=args.browser, - judge=args.judge, - fresh=args.fresh, - ) + else _report_path(day=date.today().isoformat(), search=args.search, **setup) ) out.parent.mkdir(parents=True, exist_ok=True) out.write_text(report, encoding="utf-8") diff --git a/src/proofpath/cache.py b/src/proofpath/cache.py index 9dfe390..d8d95ba 100644 --- a/src/proofpath/cache.py +++ b/src/proofpath/cache.py @@ -29,6 +29,49 @@ from proofpath.resolve import Candidate, FieldMatch, ResolveResult, Retraction, State, strip_marker SCHEMA_VERSION = "5" + +# What a cached judgement has passed, stored as a suffix on its ``judge_model`` key. +# ``q1``: the judge's quote was found in the passage (OPEN-ITEMS 20.9). An opinion +# cached before that was never checked, and serving it now would attach it as if it +# had been; under its bare model name it simply stops matching, so the next run asks +# again. The rule lives in the key suffix rather than in a schema migration: the +# unmatched rows cost a few bytes, and ``put_chunks`` still retires them with their +# source. (The ``quote`` column is a separate, guarded migration: it stores what was +# checked.) The next rule an opinion has to pass bumps this to ``q2``. +JUDGEMENT_CHECKS = "q1" +# Hex digits of the passage's hash kept in the key: 64 bits, so two passages of one +# source colliding is not a case worth a column of its own. +_PASSAGE_KEY_CHARS = 16 + + +def _judgement_key(judge_model: str, passage: str) -> str: + """The ``judge_model`` a judgement is stored under: the judge's name, the checks + its opinions passed, and the passage it was checked against. + + The passage belongs in the key because the check does: an opinion verified against + one passage says nothing about another, and the same claim and source are decided + on another passage after an NLI profile or ``k`` changes. Hashed whitespace-folded, + as the judge was shown it (``judge._one_line``). Callers pass the bare name and the + passage text; only this file builds the key.""" + folded = " ".join(passage.split()) + digest = hashlib.sha256(folded.encode("utf-8")).hexdigest()[:_PASSAGE_KEY_CHARS] + return f"{judge_model}#{JUDGEMENT_CHECKS}#{digest}" + + +def _add_judgement_quote(conn: sqlite3.Connection) -> None: + """Give an older ``judgements`` table its ``quote`` column, once. + + Guarded by ``PRAGMA table_info`` rather than by the schema version: the column + came with OPEN-ITEMS 20.9 inside schema 5, so a v5 file may or may not have it, + and ``ALTER TABLE ... ADD COLUMN`` fails if it is already there. Its rows predate + the quote check and are never served (``JUDGEMENT_CHECKS``), so ``''`` is honest. + """ + columns = {str(row[1]) for row in conn.execute("PRAGMA table_info(judgements)")} + if columns and "quote" not in columns: + with conn: + conn.execute("ALTER TABLE judgements ADD COLUMN quote TEXT NOT NULL DEFAULT ''") + + RAW_TEXT_TTL_DAYS = 7 # A resolution is a statement about a published record, which does not change; the # month is there so a reference an index had not yet ingested is looked at again. @@ -147,6 +190,7 @@ label TEXT NOT NULL, rationale TEXT NOT NULL, created_at TEXT NOT NULL, + quote TEXT NOT NULL DEFAULT '', -- the checked quote (OPEN-ITEMS 20.9) PRIMARY KEY (claim_hash, source_id, judge_model) ) """, @@ -369,6 +413,9 @@ def __init__(self, path: Path | None = None) -> None: # A new file was upgraded from nothing; an existing one lost its # embeddings, which the report may want to mention. self.migrated_from = recorded + if recorded is None or _version(recorded) <= _version(SCHEMA_VERSION): + # Never on a file from a newer proofpath, which is left exactly as it is. + _add_judgement_quote(self._conn) def close(self) -> None: self._conn.close() @@ -616,43 +663,49 @@ def put_judgement( judge_model: str, opinion: JudgeOpinion, *, + passage: str, now: datetime | None = None, ) -> None: - """Store one judge opinion. Re-asking the same judge replaces what it said.""" + """Store one judge opinion on the passage it was shown. Re-asking the same + judge about the same passage replaces what it said.""" with self._conn: self._conn.execute( "INSERT INTO judgements(claim_hash, source_id, judge_model, label, rationale, " - "created_at) VALUES (?, ?, ?, ?, ?, ?) " + "created_at, quote) VALUES (?, ?, ?, ?, ?, ?, ?) " "ON CONFLICT(claim_hash, source_id, judge_model) DO UPDATE SET " "label = excluded.label, rationale = excluded.rationale, " - "created_at = excluded.created_at", + "created_at = excluded.created_at, quote = excluded.quote", ( claim_hash_, source_id, - judge_model, + _judgement_key(judge_model, passage), opinion.label.value, opinion.rationale, _iso(now or _now()), + opinion.quote, ), ) def get_judgement( - self, claim_hash_: str, source_id: str, judge_model: str + self, claim_hash_: str, source_id: str, judge_model: str, *, passage: str ) -> JudgeOpinion | None: """What this judge said about this claim and source, or ``None`` if unasked. No TTL: an opinion is about a claim and a passage, both of which are fixed. - The passage changing is what retires it, and ``put_chunks`` does that. + The passage changing is what retires it, and ``put_chunks`` does that. A + change to what an opinion has to pass retires it too: ``_judgement_key``. """ row = self._conn.execute( - "SELECT label, rationale FROM judgements " + "SELECT label, rationale, quote FROM judgements " "WHERE claim_hash = ? AND source_id = ? AND judge_model = ?", - (claim_hash_, source_id, judge_model), + (claim_hash_, source_id, _judgement_key(judge_model, passage)), ).fetchone() if row is None: return None - label, rationale = row - return JudgeOpinion(label=Label(label), rationale=str(rationale), model=judge_model) + label, rationale, quote = row + return JudgeOpinion( + label=Label(label), rationale=str(rationale), model=judge_model, quote=str(quote) + ) # --- resolutions and retractions --------------------------------------------- diff --git a/src/proofpath/judge.py b/src/proofpath/judge.py index 0bb8cd7..ca0e975 100644 --- a/src/proofpath/judge.py +++ b/src/proofpath/judge.py @@ -22,6 +22,7 @@ import string import threading import time +import unicodedata from collections.abc import Callable, Sequence from contextlib import suppress from dataclasses import dataclass, replace @@ -1318,6 +1319,11 @@ class JudgeOpinion: label: Label rationale: str model: str + # The passage's own words the opinion rests on, as the judge wrote them less their + # wrapping, and only once ``quote_problem`` found them in the passage it was shown + # (OPEN-ITEMS 20.9). Empty for an NEI that quoted nothing or quoted wrongly. Last + # and defaulted, so every existing construction and JSON reader keeps working. + quote: str = "" @dataclass(frozen=True) @@ -1357,6 +1363,14 @@ class JudgedClaim: # sentence each for a review, a few sentences for a summary. _REASONING_EFFORT = "low" _REVIEW_TOKENS = 4096 +# What one opinion costs to write, budgeted rather than measured: ``{"id", "label", +# "rationale", "quote"}`` with a one-sentence rationale and a clause-long quote is +# ~50 tokens by ``estimate_tokens`` (a test pins a typical one), and this doubles it +# for the long quote. ``BATCH_SIZE`` opinions then take 20 x 100 = 2000 tokens, under +# half of ``_REVIEW_TOKENS`` (2048): the other half is the reasoning model's thinking, +# which it charges to the same ``max_tokens``. ``BATCH_SIZE`` is the per-batch item +# cap that holds this; raising either number has to keep the product under 2048. +_OPINION_TOKENS = 100 _SUMMARY_TOKENS = 1500 # Groq is the default because it does not train on submitted prompts. Gemini does, @@ -1380,8 +1394,11 @@ class JudgedClaim: "id": {"type": "string"}, "label": {"type": "string", "enum": [label.value for label in Label]}, "rationale": {"type": "string"}, + # The deciding words, copied from the passage: checked by + # ``quote_problem`` for an asserting label (OPEN-ITEMS 20.9). + "quote": {"type": "string"}, }, - "required": ["id", "label", "rationale"], + "required": ["id", "label", "rationale", "quote"], "additionalProperties": False, }, } @@ -1454,6 +1471,167 @@ def _one_line(text: str) -> str: return " ".join(text.split()) +# --- the quote check (OPEN-ITEMS 20.9, spec 2026-10-05 section 1) ---------------- +# +# The prompt has always asked for the deciding words verbatim; nothing checked that +# they were. A judge answering from memory, or paraphrasing, still had its opinion +# printed beside the verdict -- and rule 1 ("never assert without a passage") binds +# the judge as much as the pipeline. So the words now come back in their own field +# and are looked for in the passage the item carried. The notes are public because +# ``scripts/eval_averitec.py`` counts drops by reason off ``Judge.skipped``: a +# reworded note would quietly count none. +QUOTE_MISSING = "no quoted words came back; opinion dropped" +QUOTE_TOO_SHORT = "a quoted part is shorter than two words; opinion dropped" +QUOTE_NOT_FOUND = "the quoted words are not in the passage; opinion dropped" + +# A part has to carry at least this many words. One word -- "the", "e", a "5" -- is in +# almost every passage, so finding it proves nothing about where the opinion came from. +_MIN_QUOTE_WORDS = 2 +# A number or a contraction is one word however it is written: "43,000", "3.2" and +# "it's" each count once, so a bare figure cannot pass for a two-word quote. +_WORD = re.compile(r"\w+(?:[.,']\w+)*") +# What glues a number to its neighbours. A quote that starts at a digit must not begin +# right after a sign or after a digit and its separator ("1.2", "43,000", "-3.2"); one +# that ends at a digit must not stop before a separator and a digit, or a percent sign. +_SIGNS = "+-" +_SEPARATORS = ".," +# Typography a model does not reproduce reliably: curly quotes become straight ones, +# the fraction slash a slash. Dashes, spaces and format characters go by Unicode +# category in ``_fold_char``, so this table does not have to list every one of them. +_STRAIGHT = { + "\u2018": "'", + "\u2019": "'", + "\u201a": "'", + "\u201b": "'", + "\u201c": '"', + "\u201d": '"', + "\u201e": '"', + "\u201f": '"', + "\u2044": "/", +} +_ELLIPSIS = re.compile(r"\.\.\.|\u2026") +# What ``_trim`` may take off a quote's ends: quote marks, brackets and sentence +# punctuation, which are the model's wrapping and not the passage's words. Never ``-``, +# ``%`` or ``+``: a minus-sign "3.2%" trimmed to "3.2" would match a passage that says +# the opposite. +_WRAPPING = frozenset( + "\"'`" # straight quote marks + "\u2018\u2019\u201a\u201b\u201c\u201d\u201e\u201f" # curly ones, before folding + "\u00ab\u00bb\u2039\u203a" # guillemets + "()[]{}" # brackets + ".,;:!?" # sentence punctuation +) + + +def _fold_char(char: str) -> str: + if char in _STRAIGHT: + return _STRAIGHT[char] + if char == "\u2212": + # The minus sign is a math symbol to Unicode, but a model types it as "-". + return "-" + category = unicodedata.category(char) + if category == "Pd": + return "-" + if category == "Zs": # every space separator, the no-break space among them + return " " + if category == "Cf": # soft hyphen, zero-width space: invisible on the page + return "" + return char + + +def _fold(text: str) -> str: + """One normalisation for both sides of the match. + + NFC, then explicit folds and nothing else: NFKC would also turn "10\u2076" into + "106" and make a superscript a false numeric match. Casefolding stays, as a + documented limit: "MW" and "mW" fold to the same words. Casefolding also turns + the "\ufb01" ligature into "fi", so that one matches; full-width letters are not + folded. + """ + text = "".join(_fold_char(char) for char in unicodedata.normalize("NFC", text)) + return " ".join(text.split()).casefold() + + +def _trim(part: str) -> str: + """A quote part without the whitespace, quote marks, brackets and sentence + punctuation around it.""" + start, end = 0, len(part) + while start < end and (part[start].isspace() or part[start] in _WRAPPING): + start += 1 + while end > start and (part[end - 1].isspace() or part[end - 1] in _WRAPPING): + end -= 1 + return part[start:end] + + +def _glued_before(haystack: str, needle: str, found: int) -> bool: + if found == 0: + return False + before = haystack[found - 1] + if before.isalnum(): + return True + if not needle[0].isdigit(): + return False + # "2 million" in "1.2 million", "000" in "43,000", "3.2%" in "-3.2%". A dash right + # after a digit is a range ("2019-2020"), not a sign, so "2020 season" still stands. + after_digit = found >= 2 and haystack[found - 2].isdigit() + return (before in _SIGNS and not after_digit) or (before in _SEPARATORS and after_digit) + + +def _glued_after(haystack: str, needle: str, after: int) -> bool: + if after == len(haystack): + return False + following = haystack[after] + if following.isalnum(): + return True + if not needle[-1].isdigit(): + return False + # "1" in "1.5", "3" in "3%". + return following == "%" or ( + following in _SEPARATORS and after + 1 < len(haystack) and haystack[after + 1].isdigit() + ) + + +def _find_words(haystack: str, needle: str, start: int) -> int: + """Where ``needle`` first occurs in ``haystack`` at or after ``start`` with nothing + glued to either end, or -1. "5 million" is not in "15 million", "ploy" is not in + "unemployment", and a number's own sign, separators and percent sign belong to it: + "2 million" is not in "1.2 million", nor "3.2%" in "-3.2%".""" + found = haystack.find(needle, start) + while found >= 0: + after = found + len(needle) + if not _glued_before(haystack, needle, found) and not _glued_after(haystack, needle, after): + return found + found = haystack.find(needle, found + 1) + return -1 + + +def quote_problem(quote: str, passage: str) -> str | None: + """Why ``quote`` does not stand as the passage's own words, or ``None`` when it does. + + Exact after ``_fold`` on both sides, nothing looser: a fuzzy match would let the + paraphrase through, and the paraphrase is what this check exists to catch. An + ellipsis splits the quote; every part must be at least two words, and must occur + in the passage at word boundaries, in order and without overlapping, so a model + may elide the middle of a sentence but not stitch scraps together. The checks run + in that order: ``QUOTE_MISSING`` when there is not a word in it, + ``QUOTE_TOO_SHORT`` when a part is one word, ``QUOTE_NOT_FOUND`` otherwise. + """ + parts = [_trim(part) for part in _ELLIPSIS.split(_fold(quote))] + parts = [part for part in parts if part] + if not any(_WORD.search(part) for part in parts): + return QUOTE_MISSING + if any(len(_WORD.findall(part)) < _MIN_QUOTE_WORDS for part in parts): + return QUOTE_TOO_SHORT + haystack = _fold(passage) + position = 0 + for part in parts: + found = _find_words(haystack, part, position) + if found < 0: + return QUOTE_NOT_FOUND + position = found + len(part) + return None + + class Judge: """The escalation layer: batches of low-confidence verdicts, one opinion each. @@ -1717,7 +1895,8 @@ def _collect( ids = ", ".join(item.id for item in batch) self.skipped.append(f"the judge's answer was not JSON; {len(batch)} items ({ids})") return - known = {item.id for item in batch} + passages = {item.id: item.passage for item in batch} + known = set(passages) for entry in payload: if not isinstance(entry, dict): self.skipped.append(f"an opinion was {type(entry).__name__}, not an object") @@ -1739,10 +1918,21 @@ def _collect( # Rule 1 applies to the judge too: no quoted reason, no opinion. self.skipped.append(f"{ident} came back with no rationale") continue + # Rule 1 again, now checked rather than asked for (OPEN-ITEMS 20.9): an + # opinion that asserts must quote the passage it was shown. NEI asserts + # nothing and owes no quote, but keeps one only if it passed, so a quote + # the report shows is always a checked one. ``or ""`` so a JSON null is + # no quote, not the four letters "None". + raw = _one_line(str(entry.get("quote") or "")) + problem = quote_problem(raw, passages[ident]) + if problem is not None and label is not Label.NEI: + self.skipped.append(f"{ident}: {problem}") + continue # Named by the answer, not by ``self.name`` read now: the client may have # changed hands since this answer left it (judge-cooldown review). by = completion.by or self.name - into[ident] = JudgeOpinion(label=label, rationale=rationale, model=by) + quote = _trim(raw) if problem is None else "" + into[ident] = JudgeOpinion(label=label, rationale=rationale, model=by, quote=quote) def _render(item: JudgeItem) -> str: diff --git a/src/proofpath/pipeline.py b/src/proofpath/pipeline.py index 7e9b60e..928caff 100644 --- a/src/proofpath/pipeline.py +++ b/src/proofpath/pipeline.py @@ -99,6 +99,39 @@ def tier_note(thresholds: Thresholds) -> str: _REFUTED = LABEL_ORDER.index(Label.REFUTED) +@dataclass(frozen=True) +class Closest: + """The strongest SUPPORTED / REFUTED signal among the scored hits, decided or not. + + Never a verdict: ``aggregate`` turns it into one, and below ``decide`` that verdict + is NEI and carries no passage (product rule 1). ``passage`` is ``None`` only when + no hit scored above zero for either label, and ``label`` is then NEI. + """ + + label: Label + score: float + passage: Passage | None + + +def closest(hits: Sequence[Hit], probs: np.ndarray) -> Closest: + """The passage the models came closest to deciding on, and the label and score + they came closest with. + + Split out of ``aggregate`` so the AVeriTeC eval can show an NEI's passage to the + judge (OPEN-ITEMS 20.13) by the very rule that picks a decided verdict's passage: + a second copy of the loop could drift from this one, and the eval would then probe + a passage the product would never have chosen. The first hit wins a tie, as it + always did. + """ + best = Closest(Label.NEI, 0.0, None) + for hit, row in zip(hits, probs, strict=True): + for label, column in ((Label.SUPPORTED, _SUPPORTED), (Label.REFUTED, _REFUTED)): + value = float(row[column]) + if value > best.score: + best = Closest(label, value, hit.passage) + return best + + def aggregate( hits: Sequence[Hit], probs: np.ndarray, @@ -106,18 +139,10 @@ def aggregate( thresholds: Thresholds = DEFAULT_THRESHOLDS, ) -> Verdict: """Let the strongest SUPPORTED / REFUTED signal among the scored hits decide.""" - best_label = Label.NEI - best_score = 0.0 - best_passage: Passage | None = None - for hit, row in zip(hits, probs, strict=True): - for label, column in ((Label.SUPPORTED, _SUPPORTED), (Label.REFUTED, _REFUTED)): - value = float(row[column]) - if value > best_score: - best_label, best_score, best_passage = label, value, hit.passage - - if best_score < thresholds.decide: - return Verdict(Label.NEI, best_score, "low", None) - return Verdict(best_label, best_score, thresholds.tier(best_score), best_passage) + best = closest(hits, probs) + if best.score < thresholds.decide: + return Verdict(Label.NEI, best.score, "low", None) + return Verdict(best.label, best.score, thresholds.tier(best.score), best.passage) def decide_indexed( @@ -134,9 +159,25 @@ def decide_indexed( The index is built once per source and reused for every claim that cites it, so a document's passages are embedded once per run, not once per claim. """ + verdict, _ = _decide_indexed(claim, index, embedder, scorer, k=k, thresholds=thresholds) + return verdict + + +def _decide_indexed( + claim: str, + index: PassageIndex, + embedder: Embedder, + scorer: Scorer, + *, + k: int, + thresholds: Thresholds, +) -> tuple[Verdict, Passage | None]: + """``decide_indexed``'s verdict, and the passage the models came closest to + deciding on (``closest``): the verdict's own when it asserts, the one an NEI + dropped when it does not. The verdict is the product's, unchanged.""" hits = rank_indexed(claim, index, embedder, k=k) if not hits: - return Verdict(Label.NEI, 0.0, "low", None) + return Verdict(Label.NEI, 0.0, "low", None), None # Numbers first (spec section 10): a contradicting figure is decided by rule, # with both figures named, and never reaches the NLI model. @@ -145,11 +186,12 @@ def decide_indexed( reason = ( f"numeric mismatch: claim says {numeric.claim_text}, source says {numeric.source_text}" ) - return Verdict(Label.REFUTED, 1.0, "high", numeric.passage, reason=reason) + verdict = Verdict(Label.REFUTED, 1.0, "high", numeric.passage, reason=reason) + return verdict, numeric.passage # NLI convention: premise is the source passage, hypothesis is the claim. probs = scorer.score([(hit.passage.text, claim) for hit in hits]) - return aggregate(hits, probs, thresholds=thresholds) + return aggregate(hits, probs, thresholds=thresholds), closest(hits, probs).passage def decide( @@ -169,5 +211,27 @@ def decide( return decide_indexed(claim, index, embedder, scorer, k=k, thresholds=thresholds) +def decide_closest( + claim: str, + passages: Sequence[Passage], + embedder: Embedder, + scorer: Scorer, + *, + k: int, + thresholds: Thresholds = DEFAULT_THRESHOLDS, +) -> tuple[Verdict, Passage | None]: + """``decide``'s verdict and the closest passage, from one embedding and one scoring. + + For ``scripts/eval_averitec.py --judge-nei`` (OPEN-ITEMS 20.13), which shows an NEI + source's closest passage to the judge. The product never calls it: an NEI verdict + there carries no passage, and nothing here changes that. + """ + if not passages: + return Verdict(Label.NEI, 0.0, "low", None), None + index = PassageIndex(dim=embedder.dim) + index.add(passages, embedder.embed([p.text for p in passages])) + return _decide_indexed(claim, index, embedder, scorer, k=k, thresholds=thresholds) + + # Alias for the Phase 1 harness; removed in Phase 9 when the LLM judge lands. judge = decide diff --git a/src/proofpath/prompts/review.md b/src/proofpath/prompts/review.md index a31e146..29bd493 100644 --- a/src/proofpath/prompts/review.md +++ b/src/proofpath/prompts/review.md @@ -21,8 +21,13 @@ Labels: Rules: -- Quote the words that decide it. The rationale is one sentence and repeats the - deciding words from the passage verbatim. +- Quote the words that decide it. `quote` is the exact words from the passage + that decide the item, copied character for character: no paraphrase, no words + of your own, no fixing of its spelling. To leave out the middle of a long + quote, write `...` there; every part must still be in the passage, in order. + A `SUPPORTED` or `REFUTED` opinion whose quote is not in the passage is + discarded. For `NEI`, `quote` may be empty. +- The rationale is one sentence: why those words decide it. - Absence of evidence is not refutation. If the passage simply does not mention what the claim asserts, the label is `NEI`, never `REFUTED`. - Never invent a fact, a number, a citation or passage text. If the passage is @@ -37,4 +42,4 @@ $items Reply with JSON only — no prose, no code fence — in exactly this shape: -{"opinions": [{"id": "", "label": "SUPPORTED|REFUTED|NEI", "rationale": ""}]} +{"opinions": [{"id": "", "label": "SUPPORTED|REFUTED|NEI", "rationale": "", "quote": ""}]} diff --git a/src/proofpath/report.py b/src/proofpath/report.py index 532cf20..5e5ac3d 100644 --- a/src/proofpath/report.py +++ b/src/proofpath/report.py @@ -467,11 +467,16 @@ def _position(item: Finding) -> tuple[int, int, int]: def judge_detail(opinion: JudgeOpinion) -> str: """The one wording for the judge's opinion, shared by every surface. - ``judge (groq openai/gpt-oss-120b): SUPPORTED - ``. The model is named - because an opinion nobody can attribute is not evidence, and the label is the - judge's own: the local verdict is elsewhere on the same finding, unchanged. + ``judge (groq openai/gpt-oss-120b): SUPPORTED - "" - ``. The + model is named because an opinion nobody can attribute is not evidence, and the + label is the judge's own: the local verdict is elsewhere on the same finding, + unchanged. The quote is the passage's own words, found there by the quote check + (OPEN-ITEMS 20.9), so it is quoted the way a verdict's ``source:`` line is; an NEI + that quoted nothing shows no empty quote marks. """ head = f"{JUDGE_DETAIL_PREFIX}{opinion.model}): {opinion.label.value}" + if opinion.quote: + head = f'{head} \u2014 "{opinion.quote}"' return f"{head} \u2014 {opinion.rationale}" diff --git a/src/proofpath/verify.py b/src/proofpath/verify.py index a9b1c91..33747f9 100644 --- a/src/proofpath/verify.py +++ b/src/proofpath/verify.py @@ -1997,9 +1997,16 @@ def _judging( opinions: dict[str, JudgeOpinion] = {} fresh: dict[str, JudgeOpinion] = {} keys: dict[str, tuple[str, str]] = {} + # The passage each item is judged on: part of its cache key, because the quote an + # opinion passed was checked against that passage and no other (OPEN-ITEMS 20.9). + shown: dict[str, str] = {} ask: list[JudgeItem] = [] for index, result in escalated: ident = f"c{index}" + passage = result.verdict.passage + if passage is None: # pragma: no cover - escalates already refused these + continue + shown[ident] = passage.text # Keyed by what the models actually read (the hypothesis when the claim was # translated), matching ``_decide_claim``'s verdict cache key: a judgement # about the English text belongs to that text, not to the reader's sentence. @@ -2008,14 +2015,13 @@ def _judging( result.source_id, ) stored = ( - None if engine.cache is None else engine.cache.get_judgement(*keys[ident], judge.name) + None + if engine.cache is None + else engine.cache.get_judgement(*keys[ident], judge.name, passage=passage.text) ) if stored is not None: opinions[ident] = stored continue - passage = result.verdict.passage - if passage is None: # pragma: no cover - escalates already refused these - continue ask.append( JudgeItem( id=ident, @@ -2045,7 +2051,9 @@ def on_batch(done: int, total: int) -> None: for ident, opinion in fresh.items(): # Under the model that gave it: a judge that switched mid-review has # opinions from two models, and each belongs to its own cache row. - engine.cache.put_judgement(*keys[ident], opinion.model, opinion) + engine.cache.put_judgement( + *keys[ident], opinion.model, opinion, passage=shown[ident] + ) except sqlite3.Error as exc: # The cache is a speed-up, never a gate. A judgement that cannot be # stored -- a source row the cascade already took, a file gone read-only diff --git a/tests/test_cache.py b/tests/test_cache.py index 99e95ad..ea10cdb 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -659,29 +659,110 @@ def test_migrating_a_v1_file_adds_the_lookup_tables(tmp_path: Path) -> None: # --- judgements (schema v4) ------------------------------------------------------ OPINION = JudgeOpinion( - label=Label.SUPPORTED, rationale='the passage says "cats purr"', model="groq gpt-oss" + label=Label.SUPPORTED, + rationale='the passage says "cats purr"', + model="groq gpt-oss", + quote="cats purr when content", ) +# The passage the judge was shown: part of the judgement's key (OPEN-ITEMS 20.9 review). +PASSAGE = "Cats purr when content." def test_a_judgement_round_trips_and_is_keyed_by_the_judge_model(db: Cache) -> None: db.add_source( "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW ) - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) - assert db.get_judgement("h", "s", "groq gpt-oss") == OPINION + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION # Another judge has not been asked; absence of its opinion is not an opinion. - assert db.get_judgement("h", "s", "ollama qwen3.5:9b") is None - assert db.get_judgement("other", "s", "groq gpt-oss") is None + assert db.get_judgement("h", "s", "ollama qwen3.5:9b", passage=PASSAGE) is None + assert db.get_judgement("other", "s", "groq gpt-oss", passage=PASSAGE) is None def test_a_second_opinion_from_the_same_judge_replaces_the_first(db: Cache) -> None: db.add_source( "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW ) - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) later = JudgeOpinion(label=Label.NEI, rationale="on reflection, silent", model="groq gpt-oss") - db.put_judgement("h", "s", "groq gpt-oss", later, now=NOW) - assert db.get_judgement("h", "s", "groq gpt-oss") == later + db.put_judgement("h", "s", "groq gpt-oss", later, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == later + + +def test_an_opinion_stored_before_the_quote_check_is_not_served(db: Cache) -> None: + """OPEN-ITEMS 20.9: an opinion cached before quotes were checked was never checked, + so a run after the change must ask again rather than attach it as if it had been.""" + db.add_source( + "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW + ) + # Exactly the row ``put_judgement`` wrote before the change: keyed by the bare name. + with db._conn: + db._conn.execute( + "INSERT INTO judgements(claim_hash, source_id, judge_model, label, rationale, " + "created_at) VALUES (?, ?, ?, ?, ?, ?)", + ("h", "s", "groq gpt-oss", "SUPPORTED", "from memory", "2026-10-01T00:00:00+00:00"), + ) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) is None + # A checked opinion stored now is served, under the judge's own name. + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION + + +def test_an_opinion_on_one_passage_is_not_served_for_another(db: Cache) -> None: + """An opinion was checked against the passage it was shown. The same claim and + source can be decided on another passage (another NLI profile, another ``k``), and + that passage has not been checked against anything.""" + db.add_source( + "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW + ) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage="Dogs bark at night.") is None + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION + + +def test_the_verified_quote_round_trips(db: Cache) -> None: + db.add_source( + "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW + ) + silent = JudgeOpinion(label=Label.NEI, rationale="silent", model="groq gpt-oss") + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + db.put_judgement("h2", "s", "groq gpt-oss", silent, passage=PASSAGE, now=NOW) + stored = db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) + assert stored is not None and stored.quote == "cats purr when content" + assert db.get_judgement("h2", "s", "groq gpt-oss", passage=PASSAGE) == silent + + +def test_a_judgements_table_without_a_quote_column_gains_one(tmp_path: Path) -> None: + """A v5 file from before the quote: the column is added on open, guarded, and the + file keeps working; opening it twice does not try to add it twice.""" + path = tmp_path / "c.sqlite3" + Cache(path).close() + raw = sqlite3.connect(str(path)) + with raw: + raw.execute("DROP TABLE judgements") + raw.execute( + "CREATE TABLE judgements (claim_hash TEXT NOT NULL, source_id TEXT NOT NULL " + "REFERENCES sources(source_id) ON DELETE CASCADE, judge_model TEXT NOT NULL, " + "label TEXT NOT NULL, rationale TEXT NOT NULL, created_at TEXT NOT NULL, " + "PRIMARY KEY (claim_hash, source_id, judge_model))" + ) + raw.close() + + for _ in range(2): + with Cache(path) as db: + columns = {row[1] for row in db._conn.execute("PRAGMA table_info(judgements)")} + assert "quote" in columns + db.add_source( + "s", + scheme="academic", + title="", + url="", + text_kind="abstract", + raw_text=RAW, + now=NOW, + ) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION def test_judgements_go_with_the_verdicts_when_the_source_text_changes(db: Cache) -> None: @@ -692,11 +773,11 @@ def test_judgements_go_with_the_verdicts_when_the_source_text_changes(db: Cache) ) db.put_chunks("s", "bge@rev", PASSAGES, VECTORS, text_sha256=RAW_SHA) db.put_verdict("h", "s", "m", Verdict(Label.SUPPORTED, 0.93, "high", PASSAGES[0]), now=NOW) - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) db.put_chunks("s", "bge@rev", PASSAGES, VECTORS, text_sha256=sha256_text("different text")) assert db.get_verdict("h", "s", "m") is None - assert db.get_judgement("h", "s", "groq gpt-oss") is None + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) is None def test_re_chunking_the_same_text_keeps_the_judgements(db: Cache) -> None: @@ -704,18 +785,18 @@ def test_re_chunking_the_same_text_keeps_the_judgements(db: Cache) -> None: "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW ) db.put_chunks("s", "bge@rev", PASSAGES, VECTORS, text_sha256=RAW_SHA) - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) db.put_chunks("s", "other@rev", PASSAGES, VECTORS, text_sha256=RAW_SHA) - assert db.get_judgement("h", "s", "groq gpt-oss") == OPINION + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION def test_clearing_a_source_takes_its_judgements_with_it(db: Cache) -> None: db.add_source( "s", scheme="academic", title="", url="", text_kind="abstract", raw_text=RAW, now=NOW ) - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) db.clear() - assert db.get_judgement("h", "s", "groq gpt-oss") is None + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) is None _V3_SCHEMA = ( @@ -781,8 +862,8 @@ def test_migrating_a_v3_file_adds_judgements_and_keeps_everything_else(tmp_path: assert db.get_chunks("s", "bge@rev", text_sha256=RAW_SHA) is not None assert db.get_raw_text("s", now=NOW) == RAW assert db.get_verdict("h", "s", "m") is not None - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) - assert db.get_judgement("h", "s", "groq gpt-oss") == OPINION + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION def test_migrating_a_v1_file_runs_the_whole_chain_including_judgements(tmp_path: Path) -> None: @@ -792,8 +873,8 @@ def test_migrating_a_v1_file_runs_the_whole_chain_including_judgements(tmp_path: assert db.migrated_from == "1" db.put_resolution(RAW_REFERENCE, RESOLVED, now=NOW) assert db.get_resolution(RAW_REFERENCE, now=NOW) == RESOLVED - db.put_judgement("h", "s", "groq gpt-oss", OPINION, now=NOW) - assert db.get_judgement("h", "s", "groq gpt-oss") == OPINION + db.put_judgement("h", "s", "groq gpt-oss", OPINION, passage=PASSAGE, now=NOW) + assert db.get_judgement("h", "s", "groq gpt-oss", passage=PASSAGE) == OPINION assert db.get_verdict("h", "s", "m") is not None # nothing else was lost diff --git a/tests/test_eval_averitec.py b/tests/test_eval_averitec.py index f041e46..e4bdf10 100644 --- a/tests/test_eval_averitec.py +++ b/tests/test_eval_averitec.py @@ -35,6 +35,27 @@ def _load_script() -> ModuleType: eval_averitec = _load_script() + + +@pytest.fixture(autouse=True) +def _no_live_run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """No test in this module may reach the network or the real cache. A test that + calls ``main`` and forgets a stub now fails here instead of starting a live + AVeriTeC run, which one did once (2026-10-05). A test that needs one of these + patches it again, and its own ``setattr`` wins.""" + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path / "no-live-cache") + monkeypatch.setattr( + eval_averitec.averitec, + "ensure_downloaded", + lambda _cache: pytest.fail("a test reached averitec.ensure_downloaded"), + ) + monkeypatch.setattr( + eval_averitec.Engine, + "default", + staticmethod(lambda *a, **k: pytest.fail("a test reached Engine.default")), + ) + + Row = eval_averitec.Row score = eval_averitec.score render_report = eval_averitec.render_report @@ -709,7 +730,7 @@ def test_score_reports_the_hypothetical_judge_accuracy_and_who_decided() -> None assert judged.accuracy_3way == pytest.approx(3 / 4) assert judged.escalated == 2 assert judged.unanswered == 0 - assert judged.opinions == {GROQ: 1, LOCAL: 1} + assert judged.opinions == {(GROQ, "escalated"): 1, (LOCAL, "escalated"): 1} assert judged.deciders[eval_averitec.DECIDED_BY_MODELS] == (1, 1) assert judged.deciders[GROQ] == (1, 1) assert judged.deciders[LOCAL] == (1, 0) @@ -797,16 +818,27 @@ def fetch(self, url: str, *, anonymous: bool = False, use_cache: bool = True) -> class _Judge: - def __init__(self, answers: dict[str, str], model: str = GROQ) -> None: + def __init__( + self, answers: dict[str, str], model: str = GROQ, drops: dict[str, str] | None = None + ) -> None: self.answers = answers # passage text -> label value self.model = model + # passage text -> why the quote check dropped the judge's opinion of it + self.drops = drops or {} self.asked: list[object] = [] + self.skipped: list[str] = [] def review(self, items: list[object], **_: object) -> dict[str, object]: from proofpath.judge import JudgeOpinion from proofpath.models import Label self.asked.extend(items) + # Per call, as ``Judge.review`` resets it, in the real judge's own wording. + self.skipped = [ + f"{item.id}: {self.drops[item.passage]}" # type: ignore[attr-defined] + for item in items + if item.passage in self.drops # type: ignore[attr-defined] + ] return { item.id: JudgeOpinion(Label(self.answers[item.passage]), "why", self.model) # type: ignore[attr-defined] for item in items @@ -824,11 +856,23 @@ def _engine(fetcher: _Fetcher) -> object: ) -def _verdicts(monkeypatch: pytest.MonkeyPatch, by_url: dict[str, object]) -> None: - def decide(_claim: str, passages: list[object], *_a: object, **_k: object) -> object: - return by_url[passages[0].source_id] # type: ignore[attr-defined] +def _verdicts( + monkeypatch: pytest.MonkeyPatch, + by_url: dict[str, object], + closest: dict[str, object] | None = None, +) -> None: + """``pipeline.decide_closest`` answering per URL: the verdict, and the closest + passage (the verdict's own unless ``closest`` names another).""" + near = closest or {} - monkeypatch.setattr(eval_averitec.pipeline, "decide", decide) + def decide_closest( + _claim: str, passages: list[object], *_a: object, **_k: object + ) -> tuple[object, object]: + url = passages[0].source_id # type: ignore[attr-defined] + verdict = by_url[url] + return verdict, near.get(url, verdict.passage) # type: ignore[attr-defined] + + monkeypatch.setattr(eval_averitec.pipeline, "decide_closest", decide_closest) def _verdict(label: str, score: float, tier: str, url: str, reason: str = "") -> object: @@ -981,3 +1025,435 @@ def test_in_search_mode_the_report_says_the_judge_wrote_the_queries() -> None: assert "the verdicts are the models'" in gold assert "writes the search queries" in searched assert "writes the search queries" not in gold + + +# --- the NEI probes and the quote check (OPEN-ITEMS 20.13, spec 2026-10-05 §2) ------- + +NOT_FOUND = "the quoted words are not in the passage; opinion dropped" + + +def _nei(url: str) -> object: + """An NEI verdict as ``aggregate`` returns it: no passage (product rule 1).""" + from proofpath.models import Label, Verdict + + return Verdict(Label.NEI, 0.4, "low", None) + + +def _passage(text: str, url: str) -> object: + from proofpath.models import Passage + + return Passage(text, url, 3) + + +def test_judge_nei_without_judge_is_a_usage_error(capsys: pytest.CaptureFixture[str]) -> None: + with pytest.raises(SystemExit) as exc: + eval_averitec.main(["--judge-nei"]) + assert exc.value.code == 2 + assert "--judge-nei" in capsys.readouterr().err + + +def test_nei_sources_go_to_the_judge_with_their_closest_passage( + monkeypatch: pytest.MonkeyPatch, +) -> None: + low, nei, high = "https://a.example/low", "https://b.example/nei", "https://c.example/hi" + fetcher = _Fetcher({url: _fetched(url, step=1) for url in (low, nei, high)}) + _verdicts( + monkeypatch, + { + low: _verdict("REFUTED", 0.3, "low", low), + nei: _nei(nei), + high: _verdict("SUPPORTED", 0.9, "high", high), + }, + closest={nei: _passage("the closest sentence", nei)}, + ) + judge = _Judge({f"passage of {low}": "SUPPORTED", "the closest sentence": "SUPPORTED"}) + claim = averitec.Claim(id=4, text="A claim.", label="Supported", source_urls=(low, nei, high)) + + row = eval_averitec._decide_claim( + _engine(fetcher), + claim, + sleep=0, + judge=judge, # type: ignore[arg-type] + judge_nei=True, + ) + + asked = {item.passage: item for item in judge.asked} # type: ignore[attr-defined] + assert set(asked) == {f"passage of {low}", "the closest sentence"} + probe = asked["the closest sentence"] + assert (probe.verdict.value, probe.tier) == ("NEI", "low") + by_url = {s.url: s for s in row.sources} + assert by_url[nei].escalation == "nei" and by_url[nei].probed + assert not by_url[nei].escalated # the product's band is untouched + assert by_url[nei].judge_label == "SUPPORTED" + assert by_url[low].escalation == "escalated" + assert by_url[high].escalation is None + # Still the product's verdict: the strongest assertion, the probe changes nothing. + assert row.predicted == "SUPPORTED" == product_label(row.sources) + + +def test_without_judge_nei_an_nei_source_is_not_sent(monkeypatch: pytest.MonkeyPatch) -> None: + nei = "https://b.example/nei" + fetcher = _Fetcher({nei: _fetched(nei, step=1)}) + _verdicts(monkeypatch, {nei: _nei(nei)}, closest={nei: _passage("close", nei)}) + judge = _Judge({"close": "SUPPORTED"}) + claim = averitec.Claim(id=5, text="A claim.", label="Supported", source_urls=(nei,)) + + row = eval_averitec._decide_claim(_engine(fetcher), claim, sleep=0, judge=judge) # type: ignore[arg-type] + + assert judge.asked == [] + (source,) = row.sources + assert source.escalation is None and source.judge_label is None + + +def test_an_nei_source_with_no_closest_passage_is_not_probed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Nothing scored above zero: there is no passage, so nothing to ask (rule 1).""" + nei = "https://b.example/nei" + fetcher = _Fetcher({nei: _fetched(nei, step=1)}) + _verdicts(monkeypatch, {nei: _nei(nei)}) + judge = _Judge({}) + claim = averitec.Claim(id=6, text="A claim.", label="Refuted", source_urls=(nei,)) + + row = eval_averitec._decide_claim( + _engine(fetcher), + claim, + sleep=0, + judge=judge, # type: ignore[arg-type] + judge_nei=True, + ) + + assert judge.asked == [] + assert row.sources[0].escalation is None + + +def test_an_opinion_the_quote_check_dropped_is_recorded_on_its_source( + monkeypatch: pytest.MonkeyPatch, +) -> None: + low, nei = "https://a.example/low", "https://b.example/nei" + fetcher = _Fetcher({url: _fetched(url, step=1) for url in (low, nei)}) + _verdicts( + monkeypatch, + {low: _verdict("REFUTED", 0.3, "low", low), nei: _nei(nei)}, + closest={nei: _passage("close", nei)}, + ) + judge = _Judge( + {}, + drops={ + f"passage of {low}": "no quoted words came back; opinion dropped", + "close": NOT_FOUND, + }, + ) + claim = averitec.Claim(id=7, text="A claim.", label="Refuted", source_urls=(low, nei)) + + row = eval_averitec._decide_claim( + _engine(fetcher), + claim, + sleep=0, + judge=judge, # type: ignore[arg-type] + judge_nei=True, + ) + + by_url = {s.url: s for s in row.sources} + assert by_url[low].dropped == eval_averitec.DROP_NO_QUOTE + assert by_url[nei].dropped == eval_averitec.DROP_NOT_IN_PASSAGE + assert by_url[low].judge_label is None and by_url[nei].judge_label is None + + +def test_the_judges_note_wording_is_the_one_the_eval_reads() -> None: + """The eval counts drops off ``Judge.skipped``; a reworded note would count none.""" + from proofpath import judge as judge_mod + + assert judge_mod.QUOTE_NOT_FOUND == NOT_FOUND + assert judge_mod.QUOTE_MISSING == "no quoted words came back; opinion dropped" + + +def _probe( + *, + judge_label: str | None = None, + dropped: str | None = None, +) -> Any: + """A read NEI source sent to the judge under ``--judge-nei``.""" + return Source( + url=LIVE, + state=OK, + step=1, + label="NEI", + score=0.4, + tier="low", + probed=True, + judge_label=judge_label, + judge_model=GROQ if judge_label else None, + dropped=dropped, + ) + + +# Six claims, product accuracy 1/6, band-only 2/6, band plus probes 3/6. +PROBE_ROWS = [ + # The probe fixes it. + Row(0, SUPPORTED, "NEI", (OK,), (LIVE,), (_probe(judge_label="SUPPORTED"),)), + # The probe calls a false claim supported: the wrong way. + Row(1, REFUTED, "NEI", (OK,), (LIVE,), (_probe(judge_label="SUPPORTED"),)), + # A confident model verdict: right in every column. + Row(2, SUPPORTED, "SUPPORTED", (OK,), (LIVE,), (_read("SUPPORTED", 0.9, tier="high"),)), + # The probe's opinion was dropped by the quote check: it abstains, NEI stays. + Row( + 3, + REFUTED, + "NEI", + (OK,), + (LIVE,), + (_probe(dropped=eval_averitec.DROP_NOT_IN_PASSAGE),), + ), + # The band: the judge's NEI abstains, so the low-tier REFUTED goes. + Row( + 4, + NEI, + "REFUTED", + (OK,), + (LIVE,), + (_read("REFUTED", 0.3, tier="low", escalated=True, judge_label="NEI", judge_model=GROQ),), + ), + # The band: the opinion was dropped for want of a quote, so the model's vote stands. + Row( + 5, + SUPPORTED, + "REFUTED", + (OK,), + (LIVE,), + ( + Source( + url=LIVE, + state=OK, + step=1, + label="REFUTED", + score=0.3, + tier="low", + escalated=True, + dropped=eval_averitec.DROP_NO_QUOTE, + ), + ), + ), +] + + +def test_the_band_only_number_ignores_the_probes() -> None: + judged = score(PROBE_ROWS, judge=True).judged + assert judged is not None + assert judged.accuracy_3way == pytest.approx(2 / 6) + assert judged.escalated == 2 + assert judged.probed is None # not asked for + + +def test_the_probes_number_lets_the_nei_probes_vote_too() -> None: + result = score(PROBE_ROWS, judge=True, probes=True) + assert result.accuracy_3way == pytest.approx(1 / 6) # the product, untouched + assert result.judged is not None + probed = result.judged.probed + assert probed is not None + assert probed.accuracy_3way == pytest.approx(3 / 6) + assert probed.per_label == {SUPPORTED: (3, 2), REFUTED: (2, 0), NEI: (1, 1)} + assert probed.probed == 3 + assert probed.unanswered == 1 + # (Refuted called SUPPORTED, Supported called REFUTED) + assert probed.wrong_way == (1, 1) + assert probed.product_wrong_way == (0, 1) + # Claim by claim against the product: claims 0 and 4 fixed, none broken. + assert (probed.fixed, probed.broken) == (2, 0) + assert probed.p_value == pytest.approx(0.5) + + +def test_drops_are_counted_by_escalation_kind_and_reason() -> None: + judged = score(PROBE_ROWS, judge=True, probes=True).judged + assert judged is not None + assert judged.drops == { + ("escalated", eval_averitec.DROP_NO_QUOTE): 1, + ("nei", eval_averitec.DROP_NOT_IN_PASSAGE): 1, + } + + +@pytest.mark.parametrize( + ("b", "c", "p"), + [ + (0, 0, 1.0), + (1, 0, 1.0), + (0, 2, 0.5), + # The search-with-judge-queries run of 2026-10-05: 14 fixed, 4 broken, p = 0.031. + (14, 4, 4048 * 2 / 2**18), + (4, 14, 4048 * 2 / 2**18), + ], +) +def test_mcnemar_is_the_exact_two_sided_binomial_test(b: int, c: int, p: float) -> None: + assert eval_averitec.mcnemar_exact(b, c) == pytest.approx(p) + + +def test_the_report_renders_the_probes_number_and_the_drops() -> None: + text = render_report( + score(PROBE_ROWS, judge=True, probes=True), + date="2026-10-05", + limit=6, + judge=GROQ, + judge_nei=True, + no_fallback=True, + ) + assert "- NEI probes: on" in text + assert "- judge fallback: off" in text + assert "### Band plus NEI probes" in text + assert "**0.500**" in text + assert "2 fixed, 0 broken" in text and "p = 0.500" in text + assert "wrong way: 2 (1 + 1)" in text + assert "against 1 (0 + 1) for the product" in text + assert _md_line(REFUTED, 2, 0, "0.000") in text + assert "### Dropped by the quote check" in text + assert _md_line("escalated", eval_averitec.DROP_NO_QUOTE, 1) in text + assert _md_line("nei", eval_averitec.DROP_NOT_IN_PASSAGE, 1) in text + + +def test_a_judge_run_without_probes_has_no_probes_section_but_counts_drops() -> None: + text = render_report(score(JUDGED_ROWS, judge=True), date="2026-10-05", limit=4, judge=GROQ) + assert "### Band plus NEI probes" not in text + assert "NEI probes: on" not in text + assert "### Dropped by the quote check" in text + assert _md_line("none", "—", 0) in text + + +@pytest.mark.parametrize( + ("judge_nei", "no_fallback", "name", "report"), + [ + ( + True, + True, + "averitec_results-browser-judge-nei-nofallback.json", + "2026-10-05-averitec-browser-judge-nei-nofallback.md", + ), + ( + True, + False, + "averitec_results-browser-judge-nei.json", + "2026-10-05-averitec-browser-judge-nei.md", + ), + ( + False, + True, + "averitec_results-browser-judge-nofallback.json", + "2026-10-05-averitec-browser-judge-nofallback.md", + ), + ], +) +def test_the_new_setups_get_their_own_file_names( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + judge_nei: bool, + no_fallback: bool, + name: str, + report: str, +) -> None: + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + flags = {"browser": True, "judge": True, "judge_nei": judge_nei, "no_fallback": no_fallback} + path = eval_averitec._results_path(search=False, fresh=False, **flags) + assert path == tmp_path / "datasets" / name + out = eval_averitec._report_path(day="2026-10-05", search=False, fresh=False, **flags) + assert out == Path("docs/eval") / report + + +def test_no_fallback_builds_the_judge_with_its_fallback_off( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from proofpath.config import Config + + built: list[Any] = [] + + def build_client(config: Any, key: object, **_: object) -> object: + built.append(config) + return SimpleNamespace(provider="groq", model="openai/gpt-oss-120b", close=lambda: None) + + engine = SimpleNamespace(judge=None, escalate=False, searcher=None, close=lambda: None) + key = eval_averitec.judge_mod.ApiKey("gsk_test", source="test") + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + monkeypatch.setattr(eval_averitec, "load_config", Config) + monkeypatch.setattr(eval_averitec.judge_mod, "resolve_api_key", lambda _name: key) + monkeypatch.setattr(eval_averitec.judge_mod, "build_client", build_client) + monkeypatch.setattr(eval_averitec.averitec, "ensure_downloaded", lambda _cache: tmp_path) + monkeypatch.setattr(eval_averitec.averitec, "load", lambda _path, limit=None: []) + monkeypatch.setattr(eval_averitec.Engine, "default", staticmethod(lambda *a, **k: engine)) + out = tmp_path / "report.md" + + assert eval_averitec.main(["--judge", "--no-fallback", "--out", str(out)]) == 0 + + assert [config.fallback for config in built] == [eval_averitec.judge_mod.FALLBACK_OFF] + assert "- judge fallback: off" in out.read_text(encoding="utf-8") + + +def test_new_rows_round_trip_with_their_escalation_and_drop(tmp_path: Path) -> None: + path = tmp_path / "results.json" + eval_averitec._save_rows(path, PROBE_ROWS) + assert eval_averitec._load_rows(path) == PROBE_ROWS + + +def test_a_results_file_from_before_the_probes_still_loads(tmp_path: Path) -> None: + """Written on 2026-10-05, before ``probed`` and ``dropped`` existed: an escalated + source keeps the old meaning, the product's band.""" + path = tmp_path / "results.json" + source = ( + '{"url": "https://a.example/one", "state": "ok", "step": 1, "label": "REFUTED", ' + '"score": 0.3, "tier": "low", "escalated": true, "judge_label": "SUPPORTED", ' + '"judge_model": "groq openai/gpt-oss-120b"}' + ) + path.write_text( + '[{"claim_id": 1, "gold": "Supported", "predicted": "REFUTED", "states": ["ok"], ' + f'"urls": ["https://a.example/one"], "sources": [{source}]}}]', + encoding="utf-8", + ) + (row,) = eval_averitec._load_rows(path) + (loaded,) = row.sources + assert loaded.escalation == "escalated" + assert not loaded.probed and loaded.dropped is None + judged = score([row], judge=True, probes=True).judged + assert judged is not None and judged.probed is not None + assert judged.accuracy_3way == judged.probed.accuracy_3way == pytest.approx(1.0) + + +def test_the_opinions_table_counts_the_nei_probes_by_kind() -> None: + judged = score(PROBE_ROWS, judge=True, probes=True).judged + assert judged is not None + # Rows 0 and 1 are answered probes; row 4 is an answered escalation. The two + # dropped opinions are not opinions. + assert judged.opinions == {(GROQ, "nei"): 2, (GROQ, "escalated"): 1} + text = render_report( + score(PROBE_ROWS, judge=True, probes=True), date="2026-10-05", limit=6, judge=GROQ + ) + assert _md_line(GROQ, "nei", 2) in text + assert _md_line(GROQ, "escalated", 1) in text + + +def test_no_fallback_without_judge_is_a_usage_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + # Guarded: a ``main`` that did not refuse would otherwise start a live run. + monkeypatch.setattr(eval_averitec, "cache_dir", lambda: tmp_path) + monkeypatch.setattr( + eval_averitec.averitec, + "ensure_downloaded", + lambda _cache: pytest.fail("--no-fallback without --judge was not refused"), + ) + with pytest.raises(SystemExit) as exc: + eval_averitec.main(["--no-fallback"]) + assert exc.value.code == 2 + assert "--no-fallback" in capsys.readouterr().err + + +def test_every_quote_drop_reason_is_read_and_other_notes_are_not() -> None: + from proofpath import judge as judge_mod + + skipped = [ + f"s0: {judge_mod.QUOTE_MISSING}", + f"s1: {judge_mod.QUOTE_TOO_SHORT}", + f"s2: {judge_mod.QUOTE_NOT_FOUND}", + "s3 came back with no rationale", + "the judge's answer was not JSON; 1 items (s4)", + ] + assert eval_averitec._drops(skipped) == { + "s0": eval_averitec.DROP_NO_QUOTE, + "s1": eval_averitec.DROP_TOO_SHORT, + "s2": eval_averitec.DROP_NOT_IN_PASSAGE, + } diff --git a/tests/test_judge.py b/tests/test_judge.py index cffddfe..d891b04 100644 --- a/tests/test_judge.py +++ b/tests/test_judge.py @@ -694,13 +694,24 @@ def _item(ident: str, *, size: int = 238, tier: Tier = "low") -> judge.JudgeItem ) -def _answers(label: str = "NEI", rationale: str = 'the passage says "faster"') -> Any: - """Answer every id the request actually carries, so packing decides the shape.""" +def _answers( + label: str = "NEI", + rationale: str = 'the passage says "faster"', + quote: str = "The method is faster", +) -> Any: + """Answer every id the request actually carries, so packing decides the shape. + + The default quote is the opening of every ``_item`` passage, so an asserting + label passes the quote check (OPEN-ITEMS 20.9) unless a test says otherwise.""" def respond(request: httpx.Request) -> httpx.Response: sent = json.loads(request.content) ids = re.findall(r"id: (\S+)", sent["messages"][-1]["content"]) - payload = {"opinions": [{"id": i, "label": label, "rationale": rationale} for i in ids]} + payload = { + "opinions": [ + {"id": i, "label": label, "rationale": rationale, "quote": quote} for i in ids + ] + } return _ok(json.dumps(payload)) return respond @@ -742,8 +753,12 @@ def test_review_sends_two_messages_and_a_strict_schema() -> None: assert sent["response_format"]["json_schema"]["schema"]["properties"]["opinions"] # The per-item field the template names is `verdict`: the local label and tier. assert "verdict: NEI (low)" in sent["messages"][-1]["content"] + # An NEI owes no quote; this one quoted the passage, so it keeps it. assert opinions["c1"] == judge.JudgeOpinion( - label=Label.NEI, rationale='the passage says "faster"', model="ollama qwen3.5:9b" + label=Label.NEI, + rationale='the passage says "faster"', + model="ollama qwen3.5:9b", + quote="The method is faster", ) assert reviewer.cost.calls == 1 assert not reviewer.unavailable and reviewer.skipped == [] @@ -821,11 +836,13 @@ def test_lenient_parsing_keeps_the_good_opinions_and_notes_the_rest() -> None: respx.post(OLLAMA_URL).mock( return_value=_ok( '```json\n{"opinions": [' - '{"id": "c1", "label": "SUPPORTED", "rationale": "it says so"},' - '{"id": "nobody", "label": "NEI", "rationale": "x"},' - '{"id": "c2", "label": "MAYBE", "rationale": "x"},' - '{"id": "c3", "label": "NEI", "rationale": " "},' - '{"id": "c1", "label": "REFUTED", "rationale": "second helping"}' + '{"id": "c1", "label": "SUPPORTED", "rationale": "it says so",' + ' "quote": "The method is faster"},' + '{"id": "nobody", "label": "NEI", "rationale": "x", "quote": ""},' + '{"id": "c2", "label": "MAYBE", "rationale": "x", "quote": ""},' + '{"id": "c3", "label": "NEI", "rationale": " ", "quote": ""},' + '{"id": "c1", "label": "REFUTED", "rationale": "second helping",' + ' "quote": "The method is faster"}' "]}\n```" ) ) @@ -854,7 +871,7 @@ def test_becoming_unavailable_mid_way_keeps_what_was_already_gathered() -> None: """The judge is an extra opinion: losing it mid-run must not lose the run.""" respx.post(OLLAMA_URL).mock( side_effect=[ - _ok('{"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent"}]}'), + _ok('{"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent", "quote": ""}]}'), httpx.Response(401, json={"error": {"message": "Invalid API Key gsk_secret"}}), ] ) @@ -876,7 +893,7 @@ def test_a_second_review_starts_from_a_clean_state() -> None: respx.post(OLLAMA_URL).mock( side_effect=[ httpx.Response(401, json={"error": {"message": "Invalid API Key"}}), - _ok('{"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent"}]}'), + _ok('{"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent", "quote": ""}]}'), ] ) reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None), batch_size=1) @@ -1063,3 +1080,284 @@ def test_queries_leave_a_reasoning_model_room_to_think_and_answer_five_claims() def test_the_queries_budget_grows_with_the_batch_and_stays_under_its_ceiling() -> None: assert judge._queries_budget(1) < judge._queries_budget(5) assert judge._queries_budget(1000) == judge._QUERIES_MAX_TOKENS + + +# --- the quote check (OPEN-ITEMS 20.9, spec 2026-10-05 §1) ------------------------- + +# Curly quotes, an en dash, a double space and a line break: the typography a model +# does not reproduce when it quotes. Written as escapes so the characters the test is +# about are visible in the source. +QUOTED_PASSAGE = ( + "The vaccine\u2019s efficacy was 95 % \u2013 measured in \u201cphase 3\u201d trials.\n" + "Its final results came later." +) +NUMBERS = "Unemployment fell to 3.5% in March, and 15 million jobs were added." + + +@pytest.mark.parametrize( + ("quote", "passage"), + [ + # Exact. + ("The vaccine\u2019s efficacy was 95 %", QUOTED_PASSAGE), + # Case, straight quotes for curly ones, a hyphen for the en dash, one space for + # the run of two, and the quote's own wrapping quote marks and full stop gone. + ('"the vaccine\'s efficacy was 95 % - measured in "phase 3" trials."', QUOTED_PASSAGE), + # An ellipsis splits the quote; every part occurs, in order. + ("The vaccine\u2019s efficacy ... in \u201cphase 3\u201d trials", QUOTED_PASSAGE), + ("efficacy was 95 %\u2026final results came later", QUOTED_PASSAGE), + # Numbers at word boundaries. + ("15 million", NUMBERS), + ("3.5% in March", NUMBERS), + ("(3.5% in March)", NUMBERS), # brackets are the quote's, not the passage's + # A soft hyphen, a zero-width space and a no-break space in the passage are + # invisible on the page; a model quoting it types none of them. + ("unemployment fell to 3.5%", "Unem\u00adployment fell\u200b to 3.5%."), + ("15 million jobs", "and 15\u00a0million jobs"), + # The fraction slash is a slash. + ("1/2 cup of sugar", "Add 1\u20442 cup of sugar."), + # A superscript stays a superscript. + ("10\u2076 cells", "a dose of 10\u2076 cells per kilogram"), + # A negative figure keeps its sign, typed with a hyphen for the minus. + ("was \u22123.2%", "growth was -3.2%"), + # A documented limit of casefolding: units that differ only in case match. + ("output of 5 MW", "an output of 5 mW"), + ], +) +def test_a_quote_found_in_the_passage_passes(quote: str, passage: str) -> None: + assert judge.quote_problem(quote, passage) is None, (quote, passage) + + +@pytest.mark.parametrize( + ("quote", "passage"), + [ + # A paraphrase says the same thing in words the passage does not use. + ("the vaccine was 95 percent effective", QUOTED_PASSAGE), + # The parts of an ellipsis exist, but out of order. + ("final results ... the vaccine\u2019s efficacy", QUOTED_PASSAGE), + # Words from memory, not from the passage. + ("the trial enrolled 43,000 people", QUOTED_PASSAGE), + # Two words, but not at a word boundary: "5 million" is inside "15 million". + ("5 million", NUMBERS), + # NFKC would make the superscript a digit and "10⁶" the number 106. + ("106 cells", "a dose of 10\u2076 cells per kilogram"), + ("10\u2076 cells", "a dose of 106 cells per kilogram"), + # The sign is the claim: neither the minus nor the percent is trimmed away. + ("growth was \u22123.2%", "growth was 3.2%"), + # A number's own separators and sign glue it to its neighbours (re-review 1): + # the tail of "1.2 million" is not "2 million", nor of "43,000" "000". + ("2 million people", "about 1.2 million people"), + ("000 people", "about 43,000 people"), + ("3.2% growth", "a -3.2% growth rate"), + ("growth of 1", "growth of 1.5 percent"), + ("rose by 3", "rose by 3% last year"), + ], +) +def test_a_quote_not_in_the_passage_fails(quote: str, passage: str) -> None: + assert judge.quote_problem(quote, passage) == judge.QUOTE_NOT_FOUND, (quote, passage) + + +@pytest.mark.parametrize( + "quote", + [ + "e", # a letter of "Unemployment" + "the", # one word that is in nearly every passage + "ploy", # inside "Unemployment", and one token + "u ... 1 ... 9", # every part one token + "Unemployment fell ... March", # a single-word part decides nothing either + # A number is one word however it is written (re-review 2). + "3.5%", + "15 ... 3.5", + ], +) +def test_a_quote_part_of_fewer_than_two_words_is_too_short(quote: str) -> None: + assert judge.quote_problem(quote, NUMBERS) == judge.QUOTE_TOO_SHORT, quote + + +@pytest.mark.parametrize( + ("quote", "passage"), + [ + ("43,000 people", "the trial enrolled 43,000 people"), + ("1.2 million people", "about 1.2 million people"), + ("-3.2% growth", "a -3.2% growth rate"), + ("it's effective", "they said it's effective"), + # Re-review 3: a sentence-final full stop, a number at either end of the + # passage, and the second year of a dashed range all still match. + ("rose by 3", "Sales rose by 3. Then they fell."), + ("12 people", "12 people came"), + ("came to 12", "the total came to 12"), + ("2020 season", "the 2019\u20132020 season"), + ], +) +def test_a_number_or_contraction_quoted_whole_still_matches(quote: str, passage: str) -> None: + assert judge.quote_problem(quote, passage) is None, (quote, passage) + + +@pytest.mark.parametrize("quote", ["43,000", "3.2", "it's"]) +def test_a_number_or_contraction_alone_is_one_word(quote: str) -> None: + passage = "the trial enrolled 43,000 people; growth was 3.2; it's done" + assert judge.quote_problem(quote, passage) == judge.QUOTE_TOO_SHORT, quote + + +@pytest.mark.parametrize("quote", ["", " ", "...", "\u2026", '""', "\u201c\u201d", " - "]) +def test_an_empty_quote_is_no_quote(quote: str) -> None: + assert judge.quote_problem(quote, QUOTED_PASSAGE) == judge.QUOTE_MISSING, repr(quote) + + +def test_the_review_schema_and_prompt_ask_for_the_quote() -> None: + item = judge._REVIEW_SCHEMA["properties"]["opinions"]["items"] + assert item["properties"]["quote"] == {"type": "string"} + # Strict structured output needs every property required. + assert set(item["required"]) == set(item["properties"]) + text = judge.load_prompt("review").substitute(items="ITEMS") + assert '"quote"' in text + assert "exact words" in text + + +def _one_answer(label: str, quote: str | None) -> httpx.Response: + entry: dict[str, str] = {"id": "c1", "label": label, "rationale": "one sentence"} + if quote is not None: + entry["quote"] = quote + return _ok(json.dumps({"opinions": [entry]})) + + +@respx.mock +@pytest.mark.parametrize("label", ["SUPPORTED", "REFUTED"]) +def test_an_asserting_opinion_whose_quote_is_not_in_the_passage_is_dropped(label: str) -> None: + respx.post(OLLAMA_URL).mock(return_value=_one_answer(label, "the method is slower")) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + assert reviewer.review([_item("c1")]) == {} + # The run already prints ``skipped``; this is the line the reader sees. + assert reviewer.skipped == ["c1: the quoted words are not in the passage; opinion dropped"] + assert not reviewer.unavailable + + +@respx.mock +@pytest.mark.parametrize("quote", ["", None]) +@pytest.mark.parametrize("label", ["SUPPORTED", "REFUTED"]) +def test_an_asserting_opinion_with_no_quote_is_dropped(label: str, quote: str | None) -> None: + respx.post(OLLAMA_URL).mock(return_value=_one_answer(label, quote)) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + assert reviewer.review([_item("c1")]) == {} + assert reviewer.skipped == [f"c1: {judge.QUOTE_MISSING}"] + + +@respx.mock +@pytest.mark.parametrize("quote", ["", None]) +def test_an_nei_opinion_may_quote_nothing(quote: str | None) -> None: + """NEI asserts nothing, so it owes no passage (spec §1.2).""" + respx.post(OLLAMA_URL).mock(return_value=_one_answer("NEI", quote)) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + opinions = reviewer.review([_item("c1")]) + + assert opinions["c1"].label is Label.NEI + assert reviewer.skipped == [] + + +@respx.mock +def test_an_asserting_opinion_that_quotes_the_passage_is_kept() -> None: + respx.post(OLLAMA_URL).mock( + return_value=_one_answer("SUPPORTED", "\u201cthe method is faster.\u201d") + ) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + opinions = reviewer.review([_item("c1")]) + + # The quote it passed with is kept for the report, without its wrapping marks. + assert opinions["c1"] == judge.JudgeOpinion( + label=Label.SUPPORTED, + rationale="one sentence", + model="ollama qwen3.5:9b", + quote="the method is faster", + ) + assert reviewer.skipped == [] + + +@respx.mock +def test_the_quote_is_checked_against_its_own_items_passage() -> None: + """Two items in one batch: a quote from the other item's passage is not this one's.""" + respx.post(OLLAMA_URL).mock( + return_value=_ok( + json.dumps( + { + "opinions": [ + {"id": "c1", "label": "SUPPORTED", "rationale": "r", "quote": "cats purr"}, + {"id": "c2", "label": "SUPPORTED", "rationale": "r", "quote": "cats purr"}, + ] + } + ) + ) + ) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + items = [ + judge.JudgeItem("c1", "claim", "Dogs bark. Cats purr.", Label.NEI, "low"), + judge.JudgeItem("c2", "claim", "Fish swim.", Label.NEI, "low"), + ] + + assert set(reviewer.review(items)) == {"c1"} + assert reviewer.skipped == [f"c2: {judge.QUOTE_NOT_FOUND}"] + + +@respx.mock +def test_an_nei_opinion_keeps_only_a_quote_that_passed() -> None: + """NEI owes no quote, but a quote shown in the report is always a checked one.""" + respx.post(OLLAMA_URL).mock( + return_value=_ok( + json.dumps( + { + "opinions": [ + {"id": "c1", "label": "NEI", "rationale": "r", "quote": "it is silent"}, + {"id": "c2", "label": "NEI", "rationale": "r", "quote": "The method is"}, + ] + } + ) + ) + ) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + opinions = reviewer.review([_item("c1"), _item("c2")]) + + assert opinions["c1"].quote == "" + assert opinions["c2"].quote == "The method is" + assert reviewer.skipped == [] + + +@respx.mock +@pytest.mark.parametrize("label", ["SUPPORTED", "REFUTED"]) +def test_a_too_short_quote_drops_an_asserting_opinion(label: str) -> None: + respx.post(OLLAMA_URL).mock(return_value=_one_answer(label, "faster")) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None)) + + assert reviewer.review([_item("c1")]) == {} + assert reviewer.skipped == [f"c1: {judge.QUOTE_TOO_SHORT}"] + + +def test_the_item_cap_leaves_half_the_answer_budget_unspent() -> None: + """Every opinion now carries a quote. A typical one is ~50 tokens (estimated + below); ``_OPINION_TOKENS`` budgets double that for a long quote, and a full batch + of them has to fit in half of ``_REVIEW_TOKENS``: the other half is the reasoning + model's thinking, which it charges to the same budget.""" + typical = json.dumps( + { + "id": "c19", + "label": "SUPPORTED", + "rationale": "The passage gives the same rate for the same month as the claim does.", + "quote": "unemployment fell to 3.5% in March, the lowest rate since 1969", + } + ) + assert judge.estimate_tokens(typical) <= judge._OPINION_TOKENS // 2 + 10 + assert judge.BATCH_SIZE * judge._OPINION_TOKENS <= judge._REVIEW_TOKENS // 2 + + +@respx.mock +def test_a_default_judge_splits_a_batch_at_the_item_cap() -> None: + route = respx.post(OLLAMA_URL).mock(side_effect=_answers()) + reviewer = judge.Judge(judge.JudgeClient(OLLAMA, None), token_cap=10**6) + + opinions = reviewer.review([_item(f"c{n}", size=10) for n in range(judge.BATCH_SIZE + 1)]) + + assert route.call_count == 2 + assert len(opinions) == judge.BATCH_SIZE + 1 + assert [len(re.findall(r"id: (\S+)", p)) for p in _prompts(route)] == [judge.BATCH_SIZE, 1] diff --git a/tests/test_judge_fallback.py b/tests/test_judge_fallback.py index d908f64..a0b5263 100644 --- a/tests/test_judge_fallback.py +++ b/tests/test_judge_fallback.py @@ -54,7 +54,9 @@ {"role": "user", "content": "- id: c1\n claim: x\n passage: y\n verdict: NEI (low)"}, ] SCHEMA = {"type": "object", "properties": {"opinions": {"type": "array"}}} -OPINION = json.dumps({"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent"}]}) +OPINION = json.dumps( + {"opinions": [{"id": "c1", "label": "NEI", "rationale": "silent", "quote": ""}]} +) LIMIT = f"Groq limit reached — judging with local ollama {LOCAL}" # A 429 without ``Retry-After`` cools down for the default 60 s (cooldown spec 2.1), # so the switch it causes says it is temporary. diff --git a/tests/test_pipeline_closest.py b/tests/test_pipeline_closest.py new file mode 100644 index 0000000..8a0a474 --- /dev/null +++ b/tests/test_pipeline_closest.py @@ -0,0 +1,85 @@ +"""``pipeline.closest`` and ``decide_closest`` (OPEN-ITEMS 20.13, spec 2026-10-05 §2.1). + +The closest passage is the one ``aggregate`` already picks and, for an NEI verdict, +then drops. These tests pin that the helper returns that very passage and that the +product's verdicts do not move: ``tests/test_pipeline.py`` is left as it was, so it +keeps saying the same thing about ``aggregate`` and ``decide`` that it always did. +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from proofpath.models import Label, Passage +from proofpath.pipeline import Thresholds, aggregate, closest, decide, decide_closest +from proofpath.retrieval import Hit +from tests.fakes import FakeEmbedder, TableScorer + +PASSAGES = [ + Passage("cats purr", "d", 0), + Passage("dogs bark", "d", 1), + Passage("fish swim", "d", 2), +] +THRESHOLDS = Thresholds(decide=0.5, high=0.9, medium=0.7) + + +def _hits(*passages: Passage) -> list[Hit]: + return [Hit(passage, 1.0) for passage in passages] + + +def test_closest_is_the_passage_with_the_strongest_supported_or_refuted_probability() -> None: + probs = np.array([(0.30, 0.10, 0.60), (0.05, 0.42, 0.53)], dtype=np.float32) + best = closest(_hits(PASSAGES[0], PASSAGES[1]), probs) + assert best.label is Label.REFUTED + assert best.score == pytest.approx(0.42) + assert best.passage == PASSAGES[1] + + +def test_closest_is_the_passage_aggregate_keeps_when_it_decides() -> None: + probs = np.array([(0.95, 0.02, 0.03), (0.2, 0.1, 0.7)], dtype=np.float32) + hits = _hits(PASSAGES[0], PASSAGES[1]) + verdict = aggregate(hits, probs, thresholds=THRESHOLDS) + best = closest(hits, probs) + assert verdict.label is best.label is Label.SUPPORTED + assert verdict.passage == best.passage == PASSAGES[0] + assert verdict.score == pytest.approx(best.score) + + +def test_an_nei_verdict_still_drops_its_passage_while_closest_keeps_it() -> None: + probs = np.array([(0.45, 0.1, 0.45), (0.3, 0.3, 0.4)], dtype=np.float32) + hits = _hits(PASSAGES[0], PASSAGES[1]) + verdict = aggregate(hits, probs, thresholds=THRESHOLDS) + assert verdict.label is Label.NEI and verdict.passage is None # unchanged + assert verdict.score == pytest.approx(0.45) + assert closest(hits, probs).passage == PASSAGES[0] + + +def test_the_first_hit_wins_a_tie_as_aggregate_always_kept_it() -> None: + probs = np.array([(0.4, 0.1, 0.5), (0.4, 0.1, 0.5)], dtype=np.float32) + assert closest(_hits(PASSAGES[0], PASSAGES[1]), probs).passage == PASSAGES[0] + + +def test_no_signal_at_all_has_no_closest_passage() -> None: + probs = np.array([(0.0, 0.0, 1.0)], dtype=np.float32) + best = closest(_hits(PASSAGES[0]), probs) + assert best.label is Label.NEI and best.passage is None and best.score == 0.0 + + +def test_decide_closest_returns_decides_verdict_and_the_closest_passage() -> None: + table = {"cats purr": (0.45, 0.1, 0.45), "dogs bark": (0.3, 0.3, 0.4)} + verdict = decide( + "a cat purring", PASSAGES, FakeEmbedder(), TableScorer(table), k=2, thresholds=THRESHOLDS + ) + both = decide_closest( + "a cat purring", PASSAGES, FakeEmbedder(), TableScorer(table), k=2, thresholds=THRESHOLDS + ) + assert both == (verdict, PASSAGES[0]) + assert verdict.label is Label.NEI and verdict.passage is None + + +def test_decide_closest_with_no_passages_has_nothing_to_offer() -> None: + verdict, passage = decide_closest( + "anything", [], FakeEmbedder(), TableScorer({}), k=3, thresholds=THRESHOLDS + ) + assert verdict.label is Label.NEI and passage is None diff --git a/tests/test_report.py b/tests/test_report.py index 69c2b62..98719ac 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -1355,6 +1355,19 @@ def test_judge_detail_is_one_wording_for_every_surface() -> None: assert judge_detail(OPINION).startswith(JUDGE_DETAIL_PREFIX) +def test_judge_detail_shows_the_verified_quote_before_the_rationale() -> None: + """The quote passed the check against the passage (OPEN-ITEMS 20.9), so it is + shown as the passage's words, quoted, the way a verdict's ``source:`` line is.""" + quoted = dataclasses.replace(OPINION, quote="we observed a 4-8% improvement") + assert judge_detail(quoted) == ( + 'judge (groq openai/gpt-oss-120b): SUPPORTED \u2014 "we observed a 4-8% improvement" ' + '\u2014 the passage says "we observed a 4-8% improvement"' + ) + # An NEI that quoted nothing shows nothing where the quote would be. + silent = dataclasses.replace(OPINION, label=Label.NEI, rationale="it is silent") + assert judge_detail(silent) == "judge (groq openai/gpt-oss-120b): NEI \u2014 it is silent" + + def test_a_judged_finding_prints_its_own_judge_line_not_a_note() -> None: item = dataclasses.replace( finding(Kind.NOT_SUPPORTED, line=112, page=4), diff --git a/tests/test_verify.py b/tests/test_verify.py index dcda3e7..8e8ca5c 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -2265,15 +2265,21 @@ def close(self) -> None: def _agreeing(user: str) -> str: - ids = re.findall(r"id: (\S+)", user) + """SUPPORTED for every item, quoting each item's own passage whole: the quote + check (OPEN-ITEMS 20.9) is passed honestly, not switched off.""" + items = re.findall(r"id: (\S+)\n claim: .*\n passage: (.*)\n", user) agreed = [ - {"id": ident, "label": "SUPPORTED", "rationale": "the passage says so"} for ident in ids + {"id": ident, "label": "SUPPORTED", "rationale": "the passage says so", "quote": passage} + for ident, passage in items ] return json.dumps({"opinions": agreed}) -def _opinion(ident: str, label: str, rationale: str = "the passage says so") -> str: - return json.dumps({"opinions": [{"id": ident, "label": label, "rationale": rationale}]}) +def _opinion( + ident: str, label: str, rationale: str = "the passage says so", *, quote: str = "" +) -> str: + entry = {"id": ident, "label": label, "rationale": rationale, "quote": quote} + return json.dumps({"opinions": [entry]}) # 0.455 sits between ``decide`` (0.45) and ``medium`` (0.457948): decided, but only @@ -2323,7 +2329,14 @@ def test_only_a_low_tier_verdict_with_a_passage_is_escalated() -> None: def test_the_judges_opinion_is_attached_and_alters_nothing() -> None: """Spec section 11.1: the judge gives an opinion, never a verdict.""" client = FakeJudgeClient( - [_opinion("c0", "SUPPORTED", 'it says "improved translation quality"')] + [ + _opinion( + "c0", + "SUPPORTED", + 'it says "improved translation quality"', + quote="improved translation quality", + ) + ] ) report = judged(client) @@ -2343,7 +2356,13 @@ def test_the_judges_opinion_is_attached_and_alters_nothing() -> None: def test_an_agreeing_judge_leaves_the_finding_without_a_judge_line() -> None: """A second opinion worth printing is one that differs; agreement is recorded on the finding but does not add a line saying the same thing twice.""" - client = FakeJudgeClient([_opinion("c0", "REFUTED", "it contradicts the claim")]) + client = FakeJudgeClient( + [ + _opinion( + "c0", "REFUTED", "it contradicts the claim", quote="improved translation quality" + ) + ] + ) report = judged(client) finding = next(f for f in report.findings if f.kind is Kind.NOT_SUPPORTED) diff --git a/tests/test_verify_search.py b/tests/test_verify_search.py index d215c2c..119735a 100644 --- a/tests/test_verify_search.py +++ b/tests/test_verify_search.py @@ -519,7 +519,7 @@ def test_a_translated_claims_judgement_is_cached_under_the_english_hash(tmp_path {"items": [{"id": 0, "english": ENGLISH, "queries": ["OpenAI bankruptcy"]}]} ) # A low-tier REFUTED (spec section 9 step 8): a real passage, so it escalates. - opinion = _opinion("c0", "REFUTED") + opinion = _opinion("c0", "REFUTED", quote=BANKRUPT) searcher = StubSearcher({"bankruptcy": [PAGE_A]}) with Cache(tmp_path / "c.sqlite3") as db: built = engine( @@ -549,11 +549,18 @@ def test_a_translated_claims_judgement_is_cached_under_the_english_hash(tmp_path assert not_supported.source_id is not None assert not_supported.judge is not None - stored = db.get_judgement(claim_hash(ENGLISH), not_supported.source_id, judge.name) + stored = db.get_judgement( + claim_hash(ENGLISH), not_supported.source_id, judge.name, passage=BANKRUPT + ) assert stored is not None # Never under the reader's own sentence: that would reuse this opinion for a # differently translated run of the same Turkish claim later on. - assert db.get_judgement(claim_hash(TURKISH), not_supported.source_id, judge.name) is None + assert ( + db.get_judgement( + claim_hash(TURKISH), not_supported.source_id, judge.name, passage=BANKRUPT + ) + is None + ) # --- found pages are read as pages (final review, Critical 1) ------------------------