diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md new file mode 100644 index 000000000..ac8e51956 --- /dev/null +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -0,0 +1,20 @@ +### Fixed + +- **A capped novelty run says how far the cap reached** — extraction stops at + the atom cap in document order, so a capped run measures the input up to the span + the cap stopped at; the input is the text that was segmented, which leaves out + the bundled files a run did not select. The cap warning of `novelty analyze` now names the span it + stopped at, the input's spans up to it and their share of the input's + tokens, instead of calling the atoms a sample, and `novelty atoms` now + prints it too; `extraction.reached` records the same in `novelty.json` and + `atoms.json` (`null` when the cap did not bind); the `novelty.md` header + states it; and on a capped run the headline's `coverage:` line says the + readable words went into the input and the capped atoms come from the first + N% of the input's tokens, instead of saying they were measured (as does the + warning printed under it when half or less of the artifact's readable words + went into the input). A cap met on the input's last span says instead that + the atoms reach it and any atoms past the cap in the last reply were dropped, in the warning, the header and the + `coverage:` line. The tier forfeit (`added` is a rate over the atoms the + cap kept, not the whole input) and the `harmful_atoms` gate reason (0 + conflicts among the capped atoms) no longer call the capped atoms a sample + either, and hold however far the cap reached. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index b40065add..8a09a582f 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -83,7 +83,9 @@ self-judge; `arms run` does not check, so name the same skill was 1,900 calls, about $14.50. - **At triage** with `--include-resources`, the 25-atom cap takes atoms in document order, and the entry point comes first: the flag reaches - `references/` only if `SKILL.md` has fewer than 25 atoms. For such skills, + `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap + warning prints how far it reached through the input, the entry point and the files the budgets selected (span, + spans and token share). For such skills, use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for a full sweep; run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 2ac08ef34..0894076c7 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -134,7 +134,7 @@ marked. Read it as a review queue: extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which is a sample when + Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -893,11 +893,36 @@ Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the -sampled-rate ones. `--estimate-only` cannot know the atom count yet, so it lists them: +cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and +that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on +how far the cap reached (`--estimate-only` lists them before it is known), and neither does +the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). +`--estimate-only` cannot know the atom count yet, so it lists them: > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording +The cap is not a sample: extraction walks the spans in document order and stops at it, so a +capped run measures the input up to the span it stopped at. The input is the text that was segmented: the +entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file +left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the +artifact went into the input). The warning says how far the cap reached, as in `atom +extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens): +rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`, +`extraction.reached` records it, and the headline's `coverage:` line then says the words +went into the input and the capped atoms come from the first 23% of the input's tokens (the +warning under that line, when half or less of the words went in, says "went into the input" +too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its +`atoms.json`, do not repeat it. + +A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms +past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at +span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last +reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the +atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and +the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped +any atoms past it`; none of them says "the opening". + That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. @@ -964,6 +989,7 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | +| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index c22fdf419..4c87c3b31 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -140,13 +140,9 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None: def cmd_novelty_atoms(args: argparse.Namespace) -> None: """Extract atoms and spans only — the expensive stage, cached for reuse.""" - from dokimi.novelty.atoms import ( - ExtractionError, - ExtractionReport, - extract_atoms, - save_atoms, - verify_atoms, - ) + from dokimi.novelty.atoms import (ExtractionError, ExtractionReport, extract_atoms, save_atoms, + verify_atoms) + from dokimi.novelty.report import cap_warning from dokimi.novelty.spans import segment, span_summary artifact_bundle = _bundle_from_args(args) @@ -159,7 +155,9 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: atoms = extract_atoms(spans, client=extractor, cap=cap, report=extraction) except ExtractionError as exc: raise SystemExit(str(exc)) from None - for warning in extraction.warnings(len(spans)): + # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it. + reach = [cap_warning(cap, extraction.reached)] if extraction.reached else [] + for warning in extraction.warnings(len(spans)) + reach: print(f"WARNING {warning}", file=sys.stderr) if settings.values["verify_faithfulness"]: verify_atoms(atoms, spans, client=extractor) @@ -1151,20 +1149,23 @@ def _print_headline(artifact: dict) -> None: f"degraded: {paired['lenient']['degraded']}, " f"dead tokens: {artifact['spans'].get('frac_carrying_nothing')}" ) - _print_coverage(artifact.get("bundle") or {}) + _print_coverage(artifact.get("bundle") or {}, artifact.get("extraction")) for warning in artifact.get("warnings", []): print(f"WARNING {warning}", file=sys.stderr) -def _print_coverage(bundle: dict) -> None: +def _print_coverage(bundle: dict, extraction: dict | None = None) -> None: """How much of the artifact's readable words this run read, on stdout. The share is stated on stdout because the headline is what a user copies. A run under the floor also says so on stdout: a stderr-only warning is one a user piping the headline never sees, and a rate quoted off half-read skill is not a property of it. + A capped run says the words went into the input, and how far its atoms reach. """ from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage + from dokimi.novelty.report import coverage_reach, floored_percent + went, reach = coverage_reach(extraction) coverage = Coverage.from_record(bundle) if coverage is None or not coverage.denominator("readable"): return @@ -1174,7 +1175,7 @@ def _print_coverage(bundle: dict) -> None: rerun = "" if coverage.include_resources else "; pass --include-resources to measure the bundled files" if share is None: print("coverage: unknown (a bundled file did not decode, so its words are not " - f"counted){scope}") + f"counted){scope}{reach}") # The share is unknowable, but a floor on what was left unread is not: for an # entry-point run it is the stderr warning's figure, and a capped bundle run has one too. if coverage.counted_unmeasured_share >= 1 - MEASURED_SHARE_WARN_AT: @@ -1182,11 +1183,8 @@ def _print_coverage(bundle: dict) -> None: print(f"WARNING: {part} half or less of the artifact's readable words, so every " f"rate here describes that part, not the artifact{rerun}") return - # Floored on the 4-place share, in integers: a float product floors 0.29 to 28, and - # 49.99% must not print as "50%" beside a "below 50%" warning. A true 49.996% is already - # 0.5 by the time it is stored. - pct = "<1%" if 0 < share < 0.01 else f"{round(share * 10000) // 100}%" - print(f"coverage: {pct} of the artifact's readable words were measured{scope}") + pct = floored_percent(share) + print(f"coverage: {pct} of the artifact's readable words {went}{scope}{reach}") if share <= MEASURED_SHARE_WARN_AT: - print(f"WARNING: only {pct} of the artifact's readable words were measured, so every " + print(f"WARNING: only {pct} of the artifact's readable words {went}, so every " f"rate here describes that part, not the artifact{rerun}") diff --git a/dokimi/dokimi/gate/novelty_gate.py b/dokimi/dokimi/gate/novelty_gate.py index 4e4315f6c..1bb0a0b94 100644 --- a/dokimi/dokimi/gate/novelty_gate.py +++ b/dokimi/dokimi/gate/novelty_gate.py @@ -196,7 +196,7 @@ def _conflict_reading(artifact: dict, cfg: dict[str, Any]) -> tuple[int, int, bo elif capped: reason = ( f"the atom set was capped ({spans.get('n_atoms_indexed')} indexed), so 0 conflicts " - "among the sampled atoms is not a cleared check of the artifact" + "among the capped atoms is not a cleared check of the artifact" ) else: reason = None diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index aa5afab29..b87522c07 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -182,11 +182,8 @@ def analyze( sampling = ( f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" ) - if config.max_atoms is not None and len(atom_list) >= config.max_atoms: - warnings.append( - f"atom extraction hit the cap of {config.max_atoms}; rates describe the " - "sampled atoms, not the whole artifact" - ) + if extraction.reached and config.max_atoms is not None: # reached is set only under a cap + warnings.append(report.cap_warning(config.max_atoms, extraction.reached)) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index b858d2888..330b5e2b7 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -198,6 +198,8 @@ class ExtractionReport: call_error: The last failed call on a span that stayed unread. Errors that splitting recovered from are not recorded: they did not lose anything, and quoting one would blame the wrong cause. + reached: How far through the input a capped atom set reached (see + :func:`_cap_reach`); ``None`` when the cap did not bind. """ unread_span_ids: list[str] = field(default_factory=list) @@ -212,6 +214,7 @@ class ExtractionReport: parse_error: str = "" call_error: str = "" unusable_error: str = "" + reached: dict | None = None @property def last_error(self) -> str: @@ -287,6 +290,7 @@ def as_dict(self) -> dict: "calls": self.calls, "batches": self.batches, "last_error": self.last_error, + "reached": self.reached, } @@ -318,8 +322,9 @@ def extract_atoms( affordable; too large and the model starts skipping spans. cap: Stop after this many atoms. Recorded in the artifact as the sampling rule, so a capped run is never quoted as a full sweep. - report: Filled in place with what could not be read and what re-asking cost. - Spans no call could read are also marked ``unread`` in place. + report: Filled in place with what could not be read and what re-asking cost, + and how far a binding cap reached (``reached``). Spans no call could read + are also marked ``unread`` in place. Returns: Atoms in extraction order, de-duplicated by id. @@ -340,6 +345,7 @@ def parse(reply: str) -> list: for span in spans: span.unread = False seen: set[str] = set() + sent: set[str] = set() # span ids some call was shown, which bound the reach atoms: list[Atom] = [] lost: Exception | None = None # (spans, is a half): a half was counted in ``batches`` with the batch it came from. @@ -351,6 +357,7 @@ def parse(reply: str) -> list: if cap is not None and len(atoms) >= cap: break batch, is_half = pending.pop(0) + sent.update(span.id for span in batch) if not is_half: report.batches += 1 prompt = _EXTRACT_PROMPT.format( @@ -406,9 +413,41 @@ def parse(reply: str) -> list: # Only when nothing was read: spans that were read and asserted nothing are a # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost + report.reached = _cap_reach(spans, atoms, cap, sent) return atoms +def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, + sent: set[str]) -> dict | None: + """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap. + + Extraction walks the spans in document order and stops at the cap, so a capped set + covers the input up to the span it stopped at, not a sample of it. The input is the text segmented, + not the artifact: a bundled file it left out counts in no span. The reach is the last span any + kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, + but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span + of the document its call was never shown, and that made-up id must not move the reach + (an atom set citing nothing sent reaches the last span sent). ``spans`` counts the spans + up to and including it, ``of`` all of them, and ``tokens_share`` is their share of the + span tokens. + """ + if cap is None or not atoms or len(atoms) < cap: + return None + order = {span.id: index for index, span in enumerate(spans) if span.id in sent} + last = max((order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order), + default=max(order.values())) + reached = sum(span.tokens for span in spans[:last + 1]) + total = sum(span.tokens for span in spans) + # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0 + share = reached * 10000 // total + return { + "last_span": spans[last].id, + "spans": last + 1, + "of": len(spans), + "tokens_share": (max(share, 1) if reached else 0) / 10000, + } + + def _mark_unread(batch: list[Span], report: ExtractionReport) -> None: for span in batch: span.unread = True diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 57590833a..b1c7d5a4b 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -229,6 +229,71 @@ def _render_items(artifact: dict) -> list[str]: return lines +def floored_percent(share: float) -> str: + """A 0-1 share as a whole percent, ``<1%`` for a share above 0 and under 1%. + + Floored on the 4-place share, in integers: a float product floors 0.29 to 28, and + 49.99% must not print as "50%" beside a "below 50%" warning, nor a cap that stopped + short of the end as "100%". A true 49.996% is already 0.5 by the time it is stored. + """ + return "<1%" if 0 < share < 0.01 else f"{round(share * 10000) // 100}%" + + +def describe_reach(reached: dict) -> str: + """Where a capped extraction stopped, as ``span s0012 (12 of the input's 61 spans, 23% of + its tokens)``. + + The spans are the input's, the text that was segmented: the entry point, plus on a bundle + run the files its roles and budgets selected. Not the artifact's, whose other files are + not in it. + """ + return (f"span {reached['last_span']} ({reached['spans']} of the input's {reached['of']} " + f"spans, {floored_percent(reached['tokens_share'])} of its tokens)") + + +def reach_is_end(reached: dict) -> bool: + """Whether a binding cap stopped on the input's last span: the atoms then reach its end, and what the + cap cut is the atoms past it in the last reply, not a stretch of the input.""" + return reached["spans"] == reached["of"] + + +def cap_warning(cap: int, reached: dict) -> str: + """The warning for a binding cap: where extraction stopped, and what that leaves out.""" + left_out = ("any atoms past the cap in its last reply were dropped" if reach_is_end(reached) + else "rates describe the opening of the input, not all of it") + return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " + "pass --all-atoms for the full sweep") + + +def coverage_reach(extraction: dict | None) -> tuple[str, str]: + """The verb and tail of the headline's coverage line, from the ``extraction`` record. + + The coverage share counts the words that went into the input. Under a binding cap the + atoms come from those words up to the span the cap stopped at, so the line must not say + they were measured, and it says how far the atoms reach. + """ + reached = (extraction or {}).get("reached") + if not reached: + return "were measured", "" + if reach_is_end(reached): + return ("went into the input", "; the capped atoms reach the input's last span, and the " + "cap dropped any atoms past it") + return ("went into the input", "; the capped atoms come from the first " + f"{floored_percent(reached['tokens_share'])} of the input's tokens") + + +def _capped_at(artifact: dict) -> str: + """The report header's sentence on where a binding cap stopped extraction, or ``""``.""" + reached = (artifact.get("extraction") or {}).get("reached") + if not reached: + return "" + if reach_is_end(reached): + return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms reach the " + "input's last span, and the cap dropped any atoms past it in the last reply.") + return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms " + "come from the opening of the input, not all of it.") + + def _render_bundle(artifact: dict) -> list[str]: """Which file of the artifact carried the knowledge — and what was not read at all. @@ -384,7 +449,7 @@ def render_novelty_markdown(artifact: dict) -> str: f"Model **{artifact['model']}** ({artifact['provider']}), judged by " f"**{artifact['judge_model']}**. Artifact: {artifact['artifact']['tokens']} tokens, " f"{artifact['atoms']['n']} atoms ({artifact['atoms']['sampling']}), " - f"{artifact['atoms']['n_probed']} probed." + f"{artifact['atoms']['n_probed']} probed.{_capped_at(artifact)}" ) lines.append("") diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 35bf3fc41..052481887 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -24,15 +24,13 @@ from dataclasses import dataclass, field from typing import Any -#: What a run gives up, keyed by the setting that is off ("sampled": an atom cap the -#: artifact reached). The forfeits of a run are derived from its resolved settings and, -#: once known, its atom count -- not fixed per tier -- so `--no-verify` adds the -#: faithfulness forfeit, and `--all-atoms` or an atom set under the cap drops the -#: sampled-rate ones. +#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap; the +#: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a sampled rate over the atom cap: usable as a sign, not as a figure, " - "and not comparable with another artifact's; pass --all-atoms for the full sweep", + "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " + "sign, not as a figure, and not comparable with another artifact's; pass " + "--all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " "as 0.743 on the study's largest artifact)", ], @@ -217,7 +215,7 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, rows, yield an atom each), capped at ``max_atoms``. Without ``n_tokens`` only the span rule applies. The forfeits assume the cap is reached, since the atom count is a guess until extraction runs: an artifact with fewer atoms than the cap drops the - sampled-rate forfeits. + cap's forfeits. """ cap = self.values["max_atoms"] expected = max(1, n_spans // 2) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index b4f737923..9bd471137 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -71,7 +71,7 @@ }, "sampling": { "type": "string", - "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a sampled run is never read as a full sweep." + "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a capped run is never read as a full sweep." }, "faithfulness": { "type": ["object", "null"], @@ -335,7 +335,18 @@ "retried_batches": {"type": "integer", "minimum": 0, "description": "Batches whose reply did not parse, or had no usable atom, and were split and re-asked."}, "calls": {"type": "integer", "minimum": 0, "description": "Extraction calls made, splits included. An n-span batch that never parses costs up to 2n-1 calls."}, "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: batches the run reached (a capped run stops early), each counted once whatever its halves cost."}, - "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."} + "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, + "reached": { + "type": ["object", "null"], + "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", + "properties": { + "last_span": {"type": "string"}, + "spans": {"type": "integer", "minimum": 1}, + "of": {"type": "integer", "minimum": 1}, + "tokens_share": {"type": "number", "minimum": 0, "maximum": 1} + }, + "required": ["last_span", "spans", "of", "tokens_share"] + } } }, "attribution": { diff --git a/dokimi/scripts/growth_baseline.json b/dokimi/scripts/growth_baseline.json index 7ad83253b..9a5240a2f 100644 --- a/dokimi/scripts/growth_baseline.json +++ b/dokimi/scripts/growth_baseline.json @@ -1,6 +1,6 @@ { "files": { - "dokimi/commands/novelty.py": {"lines": 1192, "rationale": "estimate-only projection and W1 routing cost output (#164) from main, plus the #172 routing field, --all-vs-all, tied/unscored caveat and blind-skill warning, --repeats refused under --all-vs-all; file split tracked separately"}, + "dokimi/commands/novelty.py": {"lines": 1190, "rationale": "estimate-only projection and W1 routing cost output (#164) from main, plus the #172 routing field, --all-vs-all, tied/unscored caveat and blind-skill warning, --repeats refused under --all-vs-all; file split tracked separately"}, "dokimi/harnesses/benchflow.py": {"lines": 869}, "dokimi/harnesses/docker_solver.py": {"lines": 899, "rationale": "pre-existing when the gate was introduced"}, "dokimi/llm.py": {"lines": 1118, "rationale": "grew on main in #180 (replay a stored retry) after #93 seeded this baseline; split it in its own PR"}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 09f34c127..2a456e0ca 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -6,6 +6,7 @@ from __future__ import annotations import json +import re import pytest @@ -577,16 +578,98 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): judge=judge).artifact assert artifact["run_config"]["tier"]["name"] == "triage" forfeits = [w for w in artifact["warnings"] if w.startswith("tier `triage` forfeits:")] - # Two atoms under a cap of 25: the run is not sampled, so the sampled-rate forfeits - # the tier states up front are dropped from what it records. + # Two atoms under a cap of 25: the cap did not bind, so the cap's forfeits the tier + # states up front are dropped from what it records. assert artifact["atoms"]["n"] < 25 assert len(forfeits) == len(TIERS["triage"]["forfeits"]) - len(FORFEITS["sampled"]) assert sorted(artifact["run_config"]["tier"]["forfeits"]) == sorted( w.split("forfeits: ", 1)[1] for w in forfeits) - assert not any("sampled rate" in w for w in forfeits) + assert not any("rate over the atoms the cap kept" in w for w in forfeits) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) +def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path): + # One extraction batch sends all five spans; the cap keeps the first atom, from s0002. + artifact = _analyze(tmp_path, max_atoms=1).artifact + # s0001 and s0002 are 15 of the body's 42 span tokens. + reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + assert artifact["extraction"]["reached"] == reached + warning = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " + "of its tokens): rates describe the opening of the input, not all of it; pass " + "--all-atoms for the full sweep") + assert warning in artifact["warnings"] + assert not any("sampled atoms" in w for w in artifact["warnings"]) + assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " + "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " + "opening of the input, not all of it.") in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + # The renderers read every key of the reach, so a partial one does not validate. + partial = dict(artifact["extraction"], reached={"spans": 3}) + with pytest.raises(ArtifactValidationError, match="extraction/reached: 'last_span' is a required"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) + + +def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): + # A cap the last atom meets binds, and reaches that atom's span; one above it does not. + artifact = _analyze(tmp_path, max_atoms=2).artifact + assert artifact["atoms"]["n"] == 2 and artifact["spans"]["capped_atom_set"] is True + assert artifact["extraction"]["reached"]["last_span"] == "s0003" + uncapped = _analyze(tmp_path / "again", max_atoms=3).artifact + assert uncapped["extraction"]["reached"] is None + assert not any("hit the cap" in w for w in uncapped["warnings"]) + assert "atom cap stopped" not in render_novelty_markdown(uncapped) + validate_artifact("novelty", dict(uncapped, schema_version="novelty.v1")) + + +@pytest.mark.parametrize("cap, header, tail", [ + (2, "s0002 (2 of the input's 8 spans, 25% of its tokens): the atoms come from the opening " + "of the input, not all of it.", "come from the first 25% of the input's tokens"), + (8, "s0008 (8 of the input's 8 spans, 100% of its tokens): the atoms reach the input's last " + "span, and the cap dropped any atoms past it in the last reply.", + "reach the input's last span, and the cap dropped any atoms past it"), +], ids=["mid-input", "last-span"]) +def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path, capsys, cap, + header, tail): + from dokimi.commands import novelty as novelty_cmd + from dokimi.novelty.bundle import load_bundle + + root = tmp_path / "accel-tuning" + (root / "references").mkdir(parents=True) + (root / "SKILL.md").write_text(SKILL, encoding="utf-8") + (root / "references" / "tuning.md").write_text( + "# Tuning\n\nSet `--threads` to the core count.\n", encoding="utf-8") + # 500 words over the 100-word file budget: on disk, in no span the reach counts. + (root / "references" / "zlarge.md").write_text( + " ".join(f"word{i}" for i in range(500)), encoding="utf-8") + bundle = load_bundle(f"skill:{root}", include_resources=True, max_file_words=100) + assert [f.path for f in bundle.files] == ["SKILL.md", "references/tuning.md"] + scripted = _scripted("informed") + + def per_span(prompt, system): + if "Extract the checkable knowledge" not in prompt: + return scripted(prompt, system) + return json.dumps([dict(_ATOMS[0], statement=f"{sid} holds.", question=f"What of {sid}?", + span_ids=[sid]) + for sid in re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)]), None + + config = AnalyzeConfig( + artifact_ref=f"skill:{root}", artifact_kind=bundle.kind, name=bundle.name, + description=bundle.description, frontmatter=bundle.frontmatter, concurrency=1, + include_resources=True, max_atoms=cap, + ) + _, answerer, judge = _clients() + extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) + artifact = analyze(bundle.text, config, extractor=extractor, answerer=answerer, + judge=judge, artifact_bundle=bundle).artifact + assert artifact["extraction"]["reached"]["of"] == 8 + assert f"The atom cap stopped extraction at span {header}" in render_novelty_markdown(artifact) + novelty_cmd._print_coverage(artifact["bundle"], artifact["extraction"]) + # 6% of the artifact went into the input; the reach is a share of the input alone. + assert capsys.readouterr().out.startswith( + "coverage: 6% of the artifact's readable words went into the input; the capped atoms " + f"{tail}\n") + + def test_skipping_the_faithfulness_check_is_recorded_as_a_warning(tmp_path): result = _analyze(tmp_path, verify_faithfulness=False) assert result.artifact["atoms"]["faithfulness"] == {} diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 62e758833..8851908e5 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -7,11 +7,13 @@ import json import re +from types import SimpleNamespace import pytest -from dokimi.novelty.atoms import ExtractionError, ExtractionReport, extract_atoms +from dokimi.novelty.atoms import ExtractionError, ExtractionReport, _cap_reach, extract_atoms from dokimi.novelty.client import LLMClient +from dokimi.novelty.report import floored_percent from dokimi.novelty.spans import segment DOC = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 11)) @@ -221,6 +223,69 @@ def three_per_span_cut_above_13(prompt, system): ) in report.warnings(len(spans)) +def _three_per_span(prompt, system): + rows = [dict(row, statement=f"{row['statement']} ({k})", + question=f"q{k} {row['span_ids'][0]}") + for row in _rows(prompt) for k in range(3)] + return json.dumps(rows), None + + +def test_a_capped_extraction_records_the_last_span_its_atoms_reached(tmp_path): + """60 spans, 3 atoms each, 10 per batch: the 25th atom is span 9's, mid-batch.""" + from dokimi.novelty.atoms import save_atoms + + doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61)) + spans = segment(doc) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + batch_spans=10, cap=25, report=report) + assert len(atoms) == 25 and report.batches == 1 + # The batch sent s0001-s0010; the cap cut its reply inside s0009's atoms. + # Nine spans of 8 tokens, of 480 tokens in all: 72 / 480. + assert [s.tokens for s in spans[:9]] == [8] * 9 and sum(s.tokens for s in spans) == 480 + assert report.reached == {"last_span": "s0009", "spans": 9, "of": 60, "tokens_share": 0.15} + path = tmp_path / "atoms.json" + save_atoms(atoms, spans, path, cap=25, extraction=report) + assert json.loads(path.read_text())["extraction"]["reached"] == report.reached + + +def test_a_span_id_no_call_was_shown_does_not_move_the_reach(): + """The cap binds on the first batch's last span (s0010); one atom also cites s0055.""" + def made_up(prompt, system): + # Two atoms for each of s0001-s0005 and three for s0006-s0010: 25 in all. + rows = [dict(row, statement=f"{row['statement']} ({k})", question=f"q{k} {row['span_ids'][0]}") + for row in _rows(prompt) for k in range(2 if row["span_ids"][0] <= "s0005" else 3)] + rows[4]["span_ids"] = ["s0003", "s0055"] + return json.dumps(rows), None + + spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61))) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=made_up), + batch_spans=10, cap=25, report=report) + assert "s0055" in atoms[4].span_ids and report.batches == 1 + assert report.reached == {"last_span": "s0010", "spans": 10, "of": 60, + "tokens_share": 0.1666} + # An atom set that cites nothing it was shown reaches the last span sent. + first_ten = {span.id for span in spans[:10]} + assert _cap_reach(spans, [SimpleNamespace(span_ids=["s0055"])], 1, first_ten)["spans"] == 10 + + +@pytest.mark.parametrize("cap, reached", [ + (None, None), + (31, None), + # Met on the last span: still a capped set, and its reach is the whole artifact. + (30, {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0}), +]) +def test_an_extraction_the_cap_did_not_stop_reaches_nothing_or_the_end(cap, reached): + """No cap, a cap above the 30 atoms, and a cap the last atom meets exactly.""" + spans = segment(DOC) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + cap=cap, report=report) + assert len(atoms) == 30 and len(spans) == 10 + assert report.reached == report.as_dict()["reached"] == reached + + def _half_lost(n: int = 50, silent: frozenset[int] = frozenset()): """``n`` one-sentence spans, the first batch lost; every read span carries an atom except those at the ``silent`` indexes, which are read and assert nothing.""" @@ -500,3 +565,14 @@ def transport(prompt, system): "unparseable or unusable batches as halves" in w and w.endswith("avoids the extra calls") for w in report.warnings(len(spans)) ) + + +@pytest.mark.parametrize("first, share, shown", [ + (29_999, 0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% + (1, 0.0001, "<1%"), # a reach above zero is never 0% +]) +def test_a_cap_reach_share_is_floored_not_rounded(first, share, shown): + spans = [SimpleNamespace(id="s0001", tokens=first), SimpleNamespace(id="s0002", tokens=30_000 - first)] + reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1, sent={"s0001", "s0002"}) + assert reach == {"last_span": "s0001", "spans": 1, "of": 2, "tokens_share": share} + assert floored_percent(reach["tokens_share"]) == shown diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index 9db901a78..f6878035a 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -840,6 +840,38 @@ def test_a_tiny_non_zero_share_prints_as_under_one_percent(capsys): assert "WARNING: only <1% of the artifact's readable words" in out +def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(capsys): + # The entry point is every readable word, but the atoms stopped at 22.97% of its tokens. + reached = {"last_span": "s0012", "spans": 12, "of": 61, "tokens_share": 0.2297} + record = _readable_record(resource_words_on_disk=0) + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=record, + extraction={"reached": reached})) + out = capsys.readouterr().out + assert ("coverage: 100% of the artifact's readable words went into the input (entry point " + "only); the capped atoms come from the first 22% of the input's tokens\n") in out + assert "measured" not in out + # The unknown share keeps the reach too; a run the cap did not stop keeps its wording. + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=dict(record, unreadable_files=1), + extraction={"reached": reached})) + assert "counted) (entry point only); the capped atoms come from the first 22% of the " \ + "input's tokens\n" in capsys.readouterr().out + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=record, + extraction={"reached": None})) + assert "coverage: 100% of the artifact's readable words were measured (entry point only)\n" \ + in capsys.readouterr().out + + +def test_a_capped_run_under_the_floor_warns_in_the_coverage_line_s_words(capsys): + # 10 of the 40 readable words went into the input, and the atoms stopped short of them. + reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=_readable_record(), + extraction={"reached": reached})) + out = capsys.readouterr().out + assert ("WARNING: only 25% of the artifact's readable words went into the input, so every " + "rate here describes that part, not the artifact") in out + assert "measured" not in out + + def test_a_bundle_run_capped_by_the_budget_warns_without_advising_the_flag_it_passed(capsys): # --include-resources was passed; the budget left most of the reference unread. The # stderr warning list is silent about this, the stdout line is not, and it must not tell diff --git a/dokimi/tests/test_novelty_gate.py b/dokimi/tests/test_novelty_gate.py index 4df05c88d..fe41554ab 100644 --- a/dokimi/tests/test_novelty_gate.py +++ b/dokimi/tests/test_novelty_gate.py @@ -469,6 +469,9 @@ def test_a_capped_atom_set_does_not_clear_harmful_atoms(): capped = _named(evaluate_novelty(_artifact(spans=spans, conflict_coverage=full)), "harmful_atoms") assert capped.measured is False and "the atom set was capped (30 indexed)" in capped.message + # Not a sample, and not always an opening: a cap met on the last span reaches the end. + assert ("so 0 conflicts among the capped atoms is not a cleared check of the artifact;" + in capped.message) found = _named(evaluate_novelty(_artifact( spans=spans, conflict_coverage=full, categories={"harmful_conflicting": 2})), diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 24a5994ef..99881e1e2 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -5,7 +5,8 @@ import pytest -from dokimi.novelty.report import render_novelty_markdown +from dokimi.gate.novelty_gate import uncleared_conflicts +from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach, render_novelty_markdown def _artifact(**overrides): @@ -69,3 +70,41 @@ def test_found_conflicts_lead_whatever_the_coverage(): ], ids=["live", "replayed"]) def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line): assert line in render_novelty_markdown(_artifact(cost=cost)).splitlines() + + +def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): + end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} + assert coverage_reach({"reached": end}) == ( + "went into the input", "; the capped atoms reach the input's last span, and the cap dropped any atoms " + "past it") + assert "opening" not in _capped_at({"extraction": {"reached": end}}) + assert cap_warning(30, end) == ( + "atom extraction hit the cap of 30 at span s0010 (10 of the input's 10 spans, 100% of its tokens): " + "any atoms past the cap in its last reply were dropped; pass --all-atoms for the full sweep") + + +def test_a_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): + inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + assert coverage_reach({"reached": inside}) == ( + "went into the input", "; the capped atoms come from the first 35% of the input's tokens") + assert _capped_at({"extraction": {"reached": inside}}) == ( + " The atom cap stopped extraction at span s0002 (2 of the input's 5 spans, 35% of its tokens): the " + "atoms come from the opening of the input, not all of it.") + assert cap_warning(1, inside) == ( + "atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% of its tokens): rates " + "describe the opening of the input, not all of it; pass --all-atoms for the full sweep") + + +def test_a_cap_met_on_the_last_span_leaves_the_zero_uncleared_without_naming_an_opening(): + # Every pair compared and no harmful atom: the cap alone leaves the 0 uncleared. + end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} + artifact = _artifact(spans={"capped_atom_set": True, "n_atoms_indexed": 30}, + extraction={"reached": end}) + assert uncleared_conflicts(artifact) == ( + "the atom set was capped (30 indexed), so 0 conflicts among the capped atoms is not a " + "cleared check of the artifact") + markdown = render_novelty_markdown(artifact) + assert "the atoms reach the input's last span" in markdown + assert ("> **Conflicts not cleared.** The atom set was capped (30 indexed), so 0 conflicts " + "among the capped atoms") in markdown + assert "opening" not in markdown diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 03fea006e..7d727e7af 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -135,12 +135,9 @@ def _headline_coverage(text: str, work: Path) -> dict[str, Any]: """The coverage lines stdout printed, beside the ``bundle`` figures in the record.""" printed = [line for line in text.splitlines() if line.startswith(COVERAGE_LINES)] record = work / "out" / "novelty.json" - bundle = ( - json.loads(record.read_text(encoding="utf-8")).get("bundle") or {} - if record.exists() - else {} - ) - return {"printed": printed, "measured_share": bundle.get("measured_share")} + saved = json.loads(record.read_text(encoding="utf-8")) if record.exists() else {} + return {"printed": printed, "measured_share": (saved.get("bundle") or {}).get("measured_share"), + "reached": (saved.get("extraction") or {}).get("reached")} def _check_headline(coverage: dict[str, Any]) -> None: @@ -153,8 +150,10 @@ def _check_headline(coverage: dict[str, Any]) -> None: # An integer floor of the decimal share, not a float product (0.29 * 100 < 29). pct = f"{int(Decimal(str(share)).scaleb(2).to_integral_value(ROUND_FLOOR))}%" assert printed, "a run with a record must print its coverage line" + # A capped run's words went into the input; its atoms do not cover all of them. + went = "went into the input" if coverage["reached"] else "were measured" assert printed[0].startswith( - f"coverage: {pct} of the artifact's readable words were measured" + f"coverage: {pct} of the artifact's readable words {went}" ), printed warned = [line.startswith(f"WARNING: only {pct} ") for line in printed[1:]] assert warned == ([True] if share <= 0.5 else []), printed @@ -241,11 +240,25 @@ def _kept_estimate(step: dict) -> dict: # Runs main refuses only after the extraction calls it had already made. REFUSED_BEFORE_ANY_CALL = {"run/arms-without-full"} +# `novelty atoms` runs whose cap binds. +CAPPED_ATOMS = {"atoms/max-atoms-1"} +# Intended change: the cap warning says how far the cap reached instead of calling the +# atoms a sample. In the warnings and on stderr alike. +CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " + "not the whole artifact") +CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " + "of its tokens): rates describe the opening of the input, not all of it; pass " + "--all-atoms for the full sweep") +# Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier +# record, the warnings and the estimate's stderr. +CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" +CAP_FORFEIT = "`added` is a rate over the atoms the cap kept, not the whole input: usable" @pytest.mark.parametrize("name", list(CASES)) def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): - golden = json.loads(GOLDEN.read_text(encoding="utf-8")) + golden = json.loads(json.dumps(json.loads(GOLDEN.read_text(encoding="utf-8"))) + .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT)) steps = _capture(name, tmp_path) coverage = [step.pop("coverage") for step in steps] for step_coverage in coverage: @@ -255,11 +268,15 @@ def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): assert [_kept_estimate(step) for step in steps] == [ _kept_estimate(step) for step in golden[name]] else: - expected = golden[name] + expected = json.loads(json.dumps(golden[name]).replace(CAP_WARNING_ON_MAIN, CAP_WARNING)) if name in REFUSED_BEFORE_ANY_CALL: # Intended change: the arm and placebo checks now run before the clients are # built, so the refusal makes no model call. Same status and output as on main. expected = [dict(step, calls=[]) for step in expected] + if name in CAPPED_ATOMS: + # Intended change: `novelty atoms` prints the cap warning `analyze` records. + expected = [dict(step, stderr=f"{step['stderr']}WARNING {CAP_WARNING}\n") + for step in expected] assert steps == expected diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 52aeb6ba1..8133eee3d 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -226,7 +226,7 @@ def test_forfeits_follow_the_resolved_settings(): assert sampled in forfeits(standard) assert sampled not in forfeits(standard, n_atoms=15) assert sampled in forfeits(standard, n_atoms=60) - # --all-atoms drops the sampled-rate forfeit, triage included. + # --all-atoms drops the cap's forfeit, triage included. args = _parse("novelty", "analyze", "--artifact", "skill:x", "--tier", "triage", "--all-atoms") assert sampled not in resolve_settings(args.tier, _flags(args)).forfeits() @@ -276,10 +276,10 @@ def test_shortening_under_a_reached_cap_says_it_saw_only_the_opening(): on = {"max_atoms": 60, "shorten_artifact": True, "paraphrases": 1, "reversals": True, "verify_faithfulness": True, "classify_content": True, "confirm_carriers": True} - assert any("opening of the artifact" in f for f in forfeits(on, n_atoms=60)) - assert not any("opening of the artifact" in f for f in forfeits(on, n_atoms=10)) - assert not any("opening of the artifact" in f for f in forfeits(dict(on, max_atoms=None))) - assert not any("opening of the artifact" in f + assert any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=60)) + assert not any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=10)) + assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, max_atoms=None))) + assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, shorten_artifact=False), n_atoms=60))