From fa453d875f0d8dc01aace3b1969aa08e74929210 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Thu, 29 Oct 2026 17:18:37 +0000 Subject: [PATCH 01/16] fix(novelty): a capped run says how far the cap reached Atom extraction walks the spans in document order and stops at the cap, so a capped run measures the opening of the artifact. On main the run did not say how far that was. Found with `novelty analyze --tier triage` on two AGENTS.md files: the 25 atoms came from spans s0002-s0012 of 61 (about a quarter of the tokens) and s0003-s0013 of 32, yet the cap warning called them "the sampled atoms", novelty.json had only capped_atom_set and n_atoms_indexed, and the headline printed "coverage: 100% of the artifact's readable words were measured". This settles both points: the reach is recorded, and the warning and the headline say it. What a run measures is unchanged. - extract_atoms records `reached` in the extraction report when the cap binds: the last span any kept atom cites, the spans up to and including it, all spans, and those spans' share of the span tokens. It comes from the kept atoms, not the batches sent, since a batch the cap cut mid-reply was sent whole. It is null when the cap did not bind, and is written to both novelty.json and atoms.json. - novelty.v1.json's extraction block gains `reached` as an optional property. atoms.json has no schema and its loader does not read it. - The cap warning in analyze names the span, the span count and the token share, says the rates describe the opening of the artifact, and points at --all-atoms. - The headline's coverage line stays the bundle figure; on a capped run it says the words went into the input and the capped atoms come from the first N% of its tokens. The percent formatter moves to novelty.report so both figures floor the same way. - The novelty.md header says where the cap stopped extraction. - docs: the extraction field table, the tier/forfeit paragraph and the triage note in reviewing.md. - The settings parity test maps main's cap warning to the new one, and its headline check accepts the capped wording on a capped run. Signed-off-by: Anton Volkov --- ...029_120000_antonwolfy_novelty-cap-reach.md | 11 +++++ dokimi/docs/reviewing.md | 3 +- dokimi/docs/skill-novelty.md | 8 ++++ dokimi/dokimi/commands/novelty.py | 16 +++---- dokimi/dokimi/novelty/analyze.py | 7 ++-- dokimi/dokimi/novelty/atoms.py | 32 +++++++++++++- dokimi/dokimi/novelty/report.py | 41 +++++++++++++++++- dokimi/dokimi/schemas/novelty.v1.json | 12 +++++- dokimi/tests/test_novelty_analyze.py | 29 +++++++++++++ dokimi/tests/test_novelty_atoms.py | 42 +++++++++++++++++++ dokimi/tests/test_novelty_cli.py | 21 ++++++++++ dokimi/tests/test_novelty_settings_parity.py | 22 ++++++---- 12 files changed, 220 insertions(+), 24 deletions(-) create mode 100644 dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md new file mode 100644 index 000000000..d75b14110 --- /dev/null +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -0,0 +1,11 @@ +### Fixed + +- **A capped novelty run says how far the cap reached** — extraction stops at + the atom cap in document order, so a capped run measures the opening of the + artifact. The cap warning of `novelty analyze` now names the span it stopped + at, the spans up to it and their share of the tokens, instead of calling the + atoms a sample; `extraction.reached` records the same in `novelty.json` and + `atoms.json` (`null` when the cap did not bind); the `novelty.md` header + states it; and on a capped run the headline's `coverage:` line says the + readable words went into the input and the capped atoms come from the first + N% of its tokens, instead of saying they were measured. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 8a0f8982b..3c6f26778 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -71,7 +71,8 @@ the scores. `novelty` refuses a self-judge; `arms run` does not check, so name the same skill was 1,900 calls, about $14.50. - **At triage** with `--include-resources`, the 25-atom cap takes atoms in document order, and the entry point comes first: the flag reaches - `references/` only if `SKILL.md` has fewer than 25 atoms. For such skills, + `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap + warning prints how far it reached (span, spans and token share). For such skills, use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for a full sweep; run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 2ac08ef34..ee613e5a8 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -898,6 +898,13 @@ sampled-rate ones. `--estimate-only` cannot know the atom count yet, so it lists > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording +The cap is not a sample: extraction walks the spans in document order and stops at it, so a +capped run measures the opening of the artifact. Its warning says how far that is, as in +`atom extraction hit the cap of 25 at span s0012 (12 of 61 spans, 23% of the tokens): rates +describe the opening of the artifact, not the whole; pass --all-atoms for the full sweep`, +`extraction.reached` records it, and the headline's `coverage:` line then says the words +went into the input and the capped atoms come from the first 23% of its tokens. + That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. @@ -964,6 +971,7 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | +| `reached` | on a capped run, how far the cap reached: `last_span`, the last span any kept atom cites (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom); `spans`, the spans up to and including it, `of` all spans; `tokens_share`, those spans' share of the span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index c22fdf419..259f1fb3e 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -1151,20 +1151,23 @@ def _print_headline(artifact: dict) -> None: f"degraded: {paired['lenient']['degraded']}, " f"dead tokens: {artifact['spans'].get('frac_carrying_nothing')}" ) - _print_coverage(artifact.get("bundle") or {}) + _print_coverage(artifact.get("bundle") or {}, artifact.get("extraction")) for warning in artifact.get("warnings", []): print(f"WARNING {warning}", file=sys.stderr) -def _print_coverage(bundle: dict) -> None: +def _print_coverage(bundle: dict, extraction: dict | None = None) -> None: """How much of the artifact's readable words this run read, on stdout. The share is stated on stdout because the headline is what a user copies. A run under the floor also says so on stdout: a stderr-only warning is one a user piping the headline never sees, and a rate quoted off half-read skill is not a property of it. + A capped run says the words went into the input, and how far its atoms reach. """ from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage + from dokimi.novelty.report import coverage_reach, floored_percent + went, reach = coverage_reach(extraction) coverage = Coverage.from_record(bundle) if coverage is None or not coverage.denominator("readable"): return @@ -1174,7 +1177,7 @@ def _print_coverage(bundle: dict) -> None: rerun = "" if coverage.include_resources else "; pass --include-resources to measure the bundled files" if share is None: print("coverage: unknown (a bundled file did not decode, so its words are not " - f"counted){scope}") + f"counted){scope}{reach}") # The share is unknowable, but a floor on what was left unread is not: for an # entry-point run it is the stderr warning's figure, and a capped bundle run has one too. if coverage.counted_unmeasured_share >= 1 - MEASURED_SHARE_WARN_AT: @@ -1182,11 +1185,8 @@ def _print_coverage(bundle: dict) -> None: print(f"WARNING: {part} half or less of the artifact's readable words, so every " f"rate here describes that part, not the artifact{rerun}") return - # Floored on the 4-place share, in integers: a float product floors 0.29 to 28, and - # 49.99% must not print as "50%" beside a "below 50%" warning. A true 49.996% is already - # 0.5 by the time it is stored. - pct = "<1%" if 0 < share < 0.01 else f"{round(share * 10000) // 100}%" - print(f"coverage: {pct} of the artifact's readable words were measured{scope}") + pct = floored_percent(share) + print(f"coverage: {pct} of the artifact's readable words {went}{scope}{reach}") if share <= MEASURED_SHARE_WARN_AT: print(f"WARNING: only {pct} of the artifact's readable words were measured, so every " f"rate here describes that part, not the artifact{rerun}") diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index aa5afab29..702d72a46 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -182,10 +182,11 @@ def analyze( sampling = ( f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" ) - if config.max_atoms is not None and len(atom_list) >= config.max_atoms: + if extraction.reached: warnings.append( - f"atom extraction hit the cap of {config.max_atoms}; rates describe the " - "sampled atoms, not the whole artifact" + f"atom extraction hit the cap of {config.max_atoms} at " + f"{report.describe_reach(extraction.reached)}: rates describe the opening of " + "the artifact, not the whole; pass --all-atoms for the full sweep" ) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index b858d2888..b6e8f5879 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -198,6 +198,8 @@ class ExtractionReport: call_error: The last failed call on a span that stayed unread. Errors that splitting recovered from are not recorded: they did not lose anything, and quoting one would blame the wrong cause. + reached: How far through the artifact a capped atom set reached (see + :func:`_cap_reach`); ``None`` when the cap did not bind. """ unread_span_ids: list[str] = field(default_factory=list) @@ -212,6 +214,7 @@ class ExtractionReport: parse_error: str = "" call_error: str = "" unusable_error: str = "" + reached: dict | None = None @property def last_error(self) -> str: @@ -287,6 +290,7 @@ def as_dict(self) -> dict: "calls": self.calls, "batches": self.batches, "last_error": self.last_error, + "reached": self.reached, } @@ -318,8 +322,9 @@ def extract_atoms( affordable; too large and the model starts skipping spans. cap: Stop after this many atoms. Recorded in the artifact as the sampling rule, so a capped run is never quoted as a full sweep. - report: Filled in place with what could not be read and what re-asking cost. - Spans no call could read are also marked ``unread`` in place. + report: Filled in place with what could not be read and what re-asking cost, + and how far a binding cap reached (``reached``). Spans no call could read + are also marked ``unread`` in place. Returns: Atoms in extraction order, de-duplicated by id. @@ -406,9 +411,32 @@ def parse(reply: str) -> list: # Only when nothing was read: spans that were read and asserted nothing are a # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost + report.reached = _cap_reach(spans, atoms, cap) return atoms +def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None) -> dict | None: + """How far through the artifact a capped atom set reached; ``None`` under the cap. + + Extraction walks the spans in document order and stops at the cap, so a capped set + covers the opening of the artifact, not a sample of it. The reach is the last span any + kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, + but its later spans gave no atom. ``spans`` counts the spans up to and including it, + ``of`` all of them, and ``tokens_share`` is their share of the span tokens. + """ + if cap is None or not atoms or len(atoms) < cap: + return None + order = {span.id: index for index, span in enumerate(spans)} + last = max(order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order) + reached = sum(span.tokens for span in spans[:last + 1]) + return { + "last_span": spans[last].id, + "spans": last + 1, + "of": len(spans), + "tokens_share": round(reached / sum(span.tokens for span in spans), 4), + } + + def _mark_unread(batch: list[Span], report: ExtractionReport) -> None: for span in batch: span.unread = True diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 57590833a..594e16188 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -229,6 +229,45 @@ def _render_items(artifact: dict) -> list[str]: return lines +def floored_percent(share: float) -> str: + """A 0-1 share as a whole percent, ``<1%`` for a share above 0 and under 1%. + + Floored on the 4-place share, in integers: a float product floors 0.29 to 28, and + 49.99% must not print as "50%" beside a "below 50%" warning, nor a cap that stopped + short of the end as "100%". A true 49.996% is already 0.5 by the time it is stored. + """ + return "<1%" if 0 < share < 0.01 else f"{round(share * 10000) // 100}%" + + +def describe_reach(reached: dict) -> str: + """Where a capped extraction stopped: ``span s0012 (12 of 61 spans, 23% of the tokens)``.""" + return (f"span {reached['last_span']} ({reached['spans']} of {reached['of']} spans, " + f"{floored_percent(reached['tokens_share'])} of the tokens)") + + +def coverage_reach(extraction: dict | None) -> tuple[str, str]: + """The verb and tail of the headline's coverage line, from the ``extraction`` record. + + The coverage share counts the words that went into the input. Under a binding cap the + atoms come from the opening of those words only, so the line must not say they were + measured, and it says how far the atoms reach. + """ + reached = (extraction or {}).get("reached") + if not reached: + return "were measured", "" + return ("went into the input", "; the capped atoms come from the first " + f"{floored_percent(reached['tokens_share'])} of its tokens") + + +def _capped_at(artifact: dict) -> str: + """The report header's sentence on where a binding cap stopped extraction, or ``""``.""" + reached = (artifact.get("extraction") or {}).get("reached") + if not reached: + return "" + return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms " + "come from the opening of the artifact, not the whole.") + + def _render_bundle(artifact: dict) -> list[str]: """Which file of the artifact carried the knowledge — and what was not read at all. @@ -384,7 +423,7 @@ def render_novelty_markdown(artifact: dict) -> str: f"Model **{artifact['model']}** ({artifact['provider']}), judged by " f"**{artifact['judge_model']}**. Artifact: {artifact['artifact']['tokens']} tokens, " f"{artifact['atoms']['n']} atoms ({artifact['atoms']['sampling']}), " - f"{artifact['atoms']['n_probed']} probed." + f"{artifact['atoms']['n_probed']} probed.{_capped_at(artifact)}" ) lines.append("") diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index b4f737923..394913e5d 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -335,7 +335,17 @@ "retried_batches": {"type": "integer", "minimum": 0, "description": "Batches whose reply did not parse, or had no usable atom, and were split and re-asked."}, "calls": {"type": "integer", "minimum": 0, "description": "Extraction calls made, splits included. An n-span batch that never parses costs up to 2n-1 calls."}, "batches": {"type": "integer", "minimum": 0, "description": "Calls the same extraction would make with no splitting: batches the run reached (a capped run stops early), each counted once whatever its halves cost."}, - "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."} + "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, + "reached": { + "type": ["object", "null"], + "description": "How far through the artifact a capped atom set reached; null when the cap did not bind. Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the artifact, not a sample of it. last_span is the last span any kept atom cites (not the last batch sent: a batch the cap cut mid-reply was sent whole), spans the count of spans up to and including it, of all spans, and tokens_share those spans' share of the span tokens. Absent from artifacts written before it existed.", + "properties": { + "last_span": {"type": "string"}, + "spans": {"type": "integer", "minimum": 1}, + "of": {"type": "integer", "minimum": 1}, + "tokens_share": {"type": "number", "minimum": 0, "maximum": 1} + } + } } }, "attribution": { diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 09f34c127..83c682028 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -587,6 +587,35 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) +def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path): + # One extraction batch sends all five spans; the cap keeps the first atom, from s0002. + artifact = _analyze(tmp_path, max_atoms=1).artifact + # s0001 and s0002 are 15 of the body's 42 span tokens. + reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + assert artifact["extraction"]["reached"] == reached + warning = ("atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the " + "tokens): rates describe the opening of the artifact, not the whole; pass " + "--all-atoms for the full sweep") + assert warning in artifact["warnings"] + assert not any("sampled atoms" in w for w in artifact["warnings"]) + assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " + "s0002 (2 of 5 spans, 35% of the tokens): the atoms come from the opening of the " + "artifact, not the whole.") in render_novelty_markdown(artifact) + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + + +def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): + # A cap the last atom meets binds, and reaches that atom's span; one above it does not. + artifact = _analyze(tmp_path, max_atoms=2).artifact + assert artifact["atoms"]["n"] == 2 and artifact["spans"]["capped_atom_set"] is True + assert artifact["extraction"]["reached"]["last_span"] == "s0003" + uncapped = _analyze(tmp_path / "again", max_atoms=3).artifact + assert uncapped["extraction"]["reached"] is None + assert not any("hit the cap" in w for w in uncapped["warnings"]) + assert "atom cap stopped" not in render_novelty_markdown(uncapped) + validate_artifact("novelty", dict(uncapped, schema_version="novelty.v1")) + + def test_skipping_the_faithfulness_check_is_recorded_as_a_warning(tmp_path): result = _analyze(tmp_path, verify_faithfulness=False) assert result.artifact["atoms"]["faithfulness"] == {} diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 62e758833..739510d02 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -221,6 +221,48 @@ def three_per_span_cut_above_13(prompt, system): ) in report.warnings(len(spans)) +def _three_per_span(prompt, system): + rows = [dict(row, statement=f"{row['statement']} ({k})", + question=f"q{k} {row['span_ids'][0]}") + for row in _rows(prompt) for k in range(3)] + return json.dumps(rows), None + + +def test_a_capped_extraction_records_the_last_span_its_atoms_reached(tmp_path): + """60 spans, 3 atoms each, 10 per batch: the 25th atom is span 9's, mid-batch.""" + from dokimi.novelty.atoms import save_atoms + + doc = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61)) + spans = segment(doc) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + batch_spans=10, cap=25, report=report) + assert len(atoms) == 25 and report.batches == 1 + # The batch sent s0001-s0010; the cap cut its reply inside s0009's atoms. + # Nine spans of 8 tokens, of 480 tokens in all: 72 / 480. + assert [s.tokens for s in spans[:9]] == [8] * 9 and sum(s.tokens for s in spans) == 480 + assert report.reached == {"last_span": "s0009", "spans": 9, "of": 60, "tokens_share": 0.15} + path = tmp_path / "atoms.json" + save_atoms(atoms, spans, path, cap=25, extraction=report) + assert json.loads(path.read_text())["extraction"]["reached"] == report.reached + + +@pytest.mark.parametrize("cap, reached", [ + (None, None), + (31, None), + # Met on the last span: still a capped set, and its reach is the whole artifact. + (30, {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0}), +]) +def test_an_extraction_the_cap_did_not_stop_reaches_nothing_or_the_end(cap, reached): + """No cap, a cap above the 30 atoms, and a cap the last atom meets exactly.""" + spans = segment(DOC) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=_three_per_span), + cap=cap, report=report) + assert len(atoms) == 30 and len(spans) == 10 + assert report.reached == report.as_dict()["reached"] == reached + + def _half_lost(n: int = 50, silent: frozenset[int] = frozenset()): """``n`` one-sentence spans, the first batch lost; every read span carries an atom except those at the ``silent`` indexes, which are read and assert nothing.""" diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index 9db901a78..21f449e96 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -840,6 +840,27 @@ def test_a_tiny_non_zero_share_prints_as_under_one_percent(capsys): assert "WARNING: only <1% of the artifact's readable words" in out +def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(capsys): + # The entry point is every readable word, but the atoms stopped at 22.97% of its tokens. + reached = {"last_span": "s0012", "spans": 12, "of": 61, "tokens_share": 0.2297} + record = _readable_record(resource_words_on_disk=0) + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=record, + extraction={"reached": reached})) + out = capsys.readouterr().out + assert ("coverage: 100% of the artifact's readable words went into the input (entry point " + "only); the capped atoms come from the first 22% of its tokens\n") in out + assert "measured" not in out + # The unknown share keeps the reach too; a run the cap did not stop keeps its wording. + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=dict(record, unreadable_files=1), + extraction={"reached": reached})) + assert "counted) (entry point only); the capped atoms come from the first 22% of its " \ + "tokens\n" in capsys.readouterr().out + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=record, + extraction={"reached": None})) + assert "coverage: 100% of the artifact's readable words were measured (entry point only)\n" \ + in capsys.readouterr().out + + def test_a_bundle_run_capped_by_the_budget_warns_without_advising_the_flag_it_passed(capsys): # --include-resources was passed; the budget left most of the reference unread. The # stderr warning list is silent about this, the stdout line is not, and it must not tell diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 03fea006e..4354e8c99 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -135,12 +135,9 @@ def _headline_coverage(text: str, work: Path) -> dict[str, Any]: """The coverage lines stdout printed, beside the ``bundle`` figures in the record.""" printed = [line for line in text.splitlines() if line.startswith(COVERAGE_LINES)] record = work / "out" / "novelty.json" - bundle = ( - json.loads(record.read_text(encoding="utf-8")).get("bundle") or {} - if record.exists() - else {} - ) - return {"printed": printed, "measured_share": bundle.get("measured_share")} + saved = json.loads(record.read_text(encoding="utf-8")) if record.exists() else {} + return {"printed": printed, "measured_share": (saved.get("bundle") or {}).get("measured_share"), + "reached": (saved.get("extraction") or {}).get("reached")} def _check_headline(coverage: dict[str, Any]) -> None: @@ -153,8 +150,10 @@ def _check_headline(coverage: dict[str, Any]) -> None: # An integer floor of the decimal share, not a float product (0.29 * 100 < 29). pct = f"{int(Decimal(str(share)).scaleb(2).to_integral_value(ROUND_FLOOR))}%" assert printed, "a run with a record must print its coverage line" + # A capped run's words went into the input; its atoms do not cover all of them. + went = "went into the input" if coverage["reached"] else "were measured" assert printed[0].startswith( - f"coverage: {pct} of the artifact's readable words were measured" + f"coverage: {pct} of the artifact's readable words {went}" ), printed warned = [line.startswith(f"WARNING: only {pct} ") for line in printed[1:]] assert warned == ([True] if share <= 0.5 else []), printed @@ -241,6 +240,13 @@ def _kept_estimate(step: dict) -> dict: # Runs main refuses only after the extraction calls it had already made. REFUSED_BEFORE_ANY_CALL = {"run/arms-without-full"} +# Intended change: the cap warning says how far the cap reached instead of calling the +# atoms a sample. In the warnings and on stderr alike. +CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " + "not the whole artifact") +CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the " + "tokens): rates describe the opening of the artifact, not the whole; pass " + "--all-atoms for the full sweep") @pytest.mark.parametrize("name", list(CASES)) @@ -255,7 +261,7 @@ def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): assert [_kept_estimate(step) for step in steps] == [ _kept_estimate(step) for step in golden[name]] else: - expected = golden[name] + expected = json.loads(json.dumps(golden[name]).replace(CAP_WARNING_ON_MAIN, CAP_WARNING)) if name in REFUSED_BEFORE_ANY_CALL: # Intended change: the arm and placebo checks now run before the clients are # built, so the refusal makes no model call. Same status and output as on main. From 92d41c756586779b776810502bb9506016f2603a Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Thu, 29 Oct 2026 19:34:34 -0700 Subject: [PATCH 02/16] fix(novelty): floor the reach share; a cap that met the last span says so _cap_reach rounded tokens_share to 4 places before floored_percent floored it, so a cap one span short of the end (29,999 of 30,000 tokens) printed "100%". The share is now floored in integers, and a reach above zero stays "<1%". When the cap stopped on the last span, the warning, the report header and the coverage tail still said the rates describe "the opening of the artifact". The reach is then the whole artifact; what the cap cut is the atoms past it in the last reply. cap_warning now words the warning per case, and the header and coverage tail follow reach_is_end. Signed-off-by: Nikolay Petrov --- dokimi/dokimi/novelty/analyze.py | 6 +----- dokimi/dokimi/novelty/atoms.py | 5 ++++- dokimi/dokimi/novelty/report.py | 20 ++++++++++++++++++++ dokimi/tests/test_novelty_atoms.py | 15 ++++++++++++++- dokimi/tests/test_novelty_report.py | 25 ++++++++++++++++++++++++- 5 files changed, 63 insertions(+), 8 deletions(-) diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index 702d72a46..fb0b1cb1a 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -183,11 +183,7 @@ def analyze( f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" ) if extraction.reached: - warnings.append( - f"atom extraction hit the cap of {config.max_atoms} at " - f"{report.describe_reach(extraction.reached)}: rates describe the opening of " - "the artifact, not the whole; pass --all-atoms for the full sweep" - ) + warnings.append(report.cap_warning(config.max_atoms, extraction.reached)) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index b6e8f5879..9a2c0da46 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -429,11 +429,14 @@ def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None) -> dict | order = {span.id: index for index, span in enumerate(spans)} last = max(order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order) reached = sum(span.tokens for span in spans[:last + 1]) + total = sum(span.tokens for span in spans) + # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0 + share = reached * 10000 // total return { "last_span": spans[last].id, "spans": last + 1, "of": len(spans), - "tokens_share": round(reached / sum(span.tokens for span in spans), 4), + "tokens_share": (max(share, 1) if reached else 0) / 10000, } diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 594e16188..9a3879063 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -245,6 +245,20 @@ def describe_reach(reached: dict) -> str: f"{floored_percent(reached['tokens_share'])} of the tokens)") +def reach_is_end(reached: dict) -> bool: + """Whether a binding cap stopped on the artifact's last span: the atoms then reach its end, and what the + cap cut is the atoms past it in the last reply, not a stretch of the artifact.""" + return reached["spans"] == reached["of"] + + +def cap_warning(cap: int, reached: dict) -> str: + """The warning for a binding cap: where extraction stopped, and what that leaves out.""" + left_out = ("atoms past the cap in its last reply were dropped" if reach_is_end(reached) + else "rates describe the opening of the artifact, not the whole") + return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " + "pass --all-atoms for the full sweep") + + def coverage_reach(extraction: dict | None) -> tuple[str, str]: """The verb and tail of the headline's coverage line, from the ``extraction`` record. @@ -255,6 +269,9 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: reached = (extraction or {}).get("reached") if not reached: return "were measured", "" + if reach_is_end(reached): + return ("went into the input", "; the capped atoms reach its last span, and the cap " + "dropped the atoms past it") return ("went into the input", "; the capped atoms come from the first " f"{floored_percent(reached['tokens_share'])} of its tokens") @@ -264,6 +281,9 @@ def _capped_at(artifact: dict) -> str: reached = (artifact.get("extraction") or {}).get("reached") if not reached: return "" + if reach_is_end(reached): + return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms reach the " + "artifact's last span, and the cap dropped the atoms past it in the last reply.") return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms " "come from the opening of the artifact, not the whole.") diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 739510d02..89fa4f498 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -7,11 +7,13 @@ import json import re +from types import SimpleNamespace import pytest -from dokimi.novelty.atoms import ExtractionError, ExtractionReport, extract_atoms +from dokimi.novelty.atoms import ExtractionError, ExtractionReport, _cap_reach, extract_atoms from dokimi.novelty.client import LLMClient +from dokimi.novelty.report import floored_percent from dokimi.novelty.spans import segment DOC = "\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 11)) @@ -542,3 +544,14 @@ def transport(prompt, system): "unparseable or unusable batches as halves" in w and w.endswith("avoids the extra calls") for w in report.warnings(len(spans)) ) + + +@pytest.mark.parametrize("first, share, shown", [ + (29_999, 0.9999, "99%"), # one span short of the end: floored, never rounded up to 100% + (1, 0.0001, "<1%"), # a reach above zero is never 0% +]) +def test_a_cap_reach_share_is_floored_not_rounded(first, share, shown): + spans = [SimpleNamespace(id="s0001", tokens=first), SimpleNamespace(id="s0002", tokens=30_000 - first)] + reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1) + assert reach == {"last_span": "s0001", "spans": 1, "of": 2, "tokens_share": share} + assert floored_percent(reach["tokens_share"]) == shown diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 24a5994ef..81b2fda11 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -5,7 +5,7 @@ import pytest -from dokimi.novelty.report import render_novelty_markdown +from dokimi.novelty.report import _capped_at, coverage_reach, render_novelty_markdown def _artifact(**overrides): @@ -69,3 +69,26 @@ def test_found_conflicts_lead_whatever_the_coverage(): ], ids=["live", "replayed"]) def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line): assert line in render_novelty_markdown(_artifact(cost=cost)).splitlines() + + +def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): + end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} + assert coverage_reach({"reached": end}) == ( + "went into the input", "; the capped atoms reach its last span, and the cap dropped the atoms past it") + assert "opening" not in _capped_at({"extraction": {"reached": end}}) + from dokimi.novelty.report import cap_warning + + assert cap_warning(30, end) == ( + "atom extraction hit the cap of 30 at span s0010 (10 of 10 spans, 100% of the tokens): atoms past " + "the cap in its last reply were dropped; pass --all-atoms for the full sweep") + + +def test_a_cap_that_stopped_inside_the_artifact_still_says_the_rates_describe_the_opening(): + inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + assert coverage_reach({"reached": inside}) == ( + "went into the input", "; the capped atoms come from the first 35% of its tokens") + from dokimi.novelty.report import cap_warning + + assert cap_warning(1, inside) == ( + "atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the tokens): rates describe " + "the opening of the artifact, not the whole; pass --all-atoms for the full sweep") From f5a21b70fff2e0a26fdb24b2ba49c9adf11fc176 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Thu, 29 Oct 2026 20:03:27 -0700 Subject: [PATCH 03/16] fix(novelty): a cap that met the last span says any atoms past it were dropped Signed-off-by: Nikolay Petrov --- dokimi/dokimi/novelty/report.py | 6 +++--- dokimi/tests/test_novelty_report.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 9a3879063..c4698c4a3 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -253,7 +253,7 @@ def reach_is_end(reached: dict) -> bool: def cap_warning(cap: int, reached: dict) -> str: """The warning for a binding cap: where extraction stopped, and what that leaves out.""" - left_out = ("atoms past the cap in its last reply were dropped" if reach_is_end(reached) + left_out = ("any atoms past the cap in its last reply were dropped" if reach_is_end(reached) else "rates describe the opening of the artifact, not the whole") return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " "pass --all-atoms for the full sweep") @@ -271,7 +271,7 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: return "were measured", "" if reach_is_end(reached): return ("went into the input", "; the capped atoms reach its last span, and the cap " - "dropped the atoms past it") + "dropped any atoms past it") return ("went into the input", "; the capped atoms come from the first " f"{floored_percent(reached['tokens_share'])} of its tokens") @@ -283,7 +283,7 @@ def _capped_at(artifact: dict) -> str: return "" if reach_is_end(reached): return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms reach the " - "artifact's last span, and the cap dropped the atoms past it in the last reply.") + "artifact's last span, and the cap dropped any atoms past it in the last reply.") return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms " "come from the opening of the artifact, not the whole.") diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 81b2fda11..a67ab9a7c 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -74,12 +74,12 @@ def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line) def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} assert coverage_reach({"reached": end}) == ( - "went into the input", "; the capped atoms reach its last span, and the cap dropped the atoms past it") + "went into the input", "; the capped atoms reach its last span, and the cap dropped any atoms past it") assert "opening" not in _capped_at({"extraction": {"reached": end}}) from dokimi.novelty.report import cap_warning assert cap_warning(30, end) == ( - "atom extraction hit the cap of 30 at span s0010 (10 of 10 spans, 100% of the tokens): atoms past " + "atom extraction hit the cap of 30 at span s0010 (10 of 10 spans, 100% of the tokens): any atoms past " "the cap in its last reply were dropped; pass --all-atoms for the full sweep") From 27dac86e8f44d0930078e80df0f5abb544b6dca1 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:00:18 +0000 Subject: [PATCH 04/16] fix(novelty): the cap reach counts only spans a call was sent Answers review comment 4418984223. The reach took the largest span id any kept atom cites, and a cited id is checked only against the whole document, so one atom citing a span no call was shown moved the reach to it: 60 spans, 10 per batch, cap 25, the cap binding in the first batch and one atom also citing s0055 recorded s0055, 55 of 60 spans, 91% of the tokens. extract_atoms now keeps the ids of the spans it sent, and _cap_reach takes the last cited span among those (the last span sent when no kept atom cites one). The same case now reaches s0010, 10 of 60. The docstring, the extraction table in skill-novelty.md and the schema description of extraction.reached say so. Signed-off-by: Anton Volkov --- dokimi/docs/skill-novelty.md | 2 +- dokimi/dokimi/novelty/atoms.py | 19 +++++++++++++------ dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_atoms.py | 23 ++++++++++++++++++++++- 4 files changed, 37 insertions(+), 9 deletions(-) diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index ee613e5a8..a1b90ede4 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -971,7 +971,7 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | -| `reached` | on a capped run, how far the cap reached: `last_span`, the last span any kept atom cites (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom); `spans`, the spans up to and including it, `of` all spans; `tokens_share`, those spans' share of the span tokens. `null` when the cap did not bind | +| `reached` | on a capped run, how far the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all spans; `tokens_share`, those spans' share of the span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index 9a2c0da46..be630457f 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -345,6 +345,7 @@ def parse(reply: str) -> list: for span in spans: span.unread = False seen: set[str] = set() + sent: set[str] = set() # span ids some call was shown, which bound the reach atoms: list[Atom] = [] lost: Exception | None = None # (spans, is a half): a half was counted in ``batches`` with the batch it came from. @@ -356,6 +357,7 @@ def parse(reply: str) -> list: if cap is not None and len(atoms) >= cap: break batch, is_half = pending.pop(0) + sent.update(span.id for span in batch) if not is_half: report.batches += 1 prompt = _EXTRACT_PROMPT.format( @@ -411,23 +413,28 @@ def parse(reply: str) -> list: # Only when nothing was read: spans that were read and asserted nothing are a # result, and a recovered parse error is not a failure. raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost - report.reached = _cap_reach(spans, atoms, cap) + report.reached = _cap_reach(spans, atoms, cap, sent) return atoms -def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None) -> dict | None: +def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, + sent: set[str]) -> dict | None: """How far through the artifact a capped atom set reached; ``None`` under the cap. Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the artifact, not a sample of it. The reach is the last span any kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, - but its later spans gave no atom. ``spans`` counts the spans up to and including it, - ``of`` all of them, and ``tokens_share`` is their share of the span tokens. + but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span + of the document its call was never shown, and that made-up id must not move the reach + (an atom set citing nothing sent reaches the last span sent). ``spans`` counts the spans + up to and including it, ``of`` all of them, and ``tokens_share`` is their share of the + span tokens. """ if cap is None or not atoms or len(atoms) < cap: return None - order = {span.id: index for index, span in enumerate(spans)} - last = max(order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order) + order = {span.id: index for index, span in enumerate(spans) if span.id in sent} + last = max((order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order), + default=max(order.values())) reached = sum(span.tokens for span in spans[:last + 1]) total = sum(span.tokens for span in spans) # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0 diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index 394913e5d..acee71a36 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -338,7 +338,7 @@ "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, "reached": { "type": ["object", "null"], - "description": "How far through the artifact a capped atom set reached; null when the cap did not bind. Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the artifact, not a sample of it. last_span is the last span any kept atom cites (not the last batch sent: a batch the cap cut mid-reply was sent whole), spans the count of spans up to and including it, of all spans, and tokens_share those spans' share of the span tokens. Absent from artifacts written before it existed.", + "description": "How far through the artifact a capped atom set reached; null when the cap did not bind. Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the artifact, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of spans up to and including it, of all spans, and tokens_share those spans' share of the span tokens. Absent from artifacts written before it existed.", "properties": { "last_span": {"type": "string"}, "spans": {"type": "integer", "minimum": 1}, diff --git a/dokimi/tests/test_novelty_atoms.py b/dokimi/tests/test_novelty_atoms.py index 89fa4f498..8851908e5 100644 --- a/dokimi/tests/test_novelty_atoms.py +++ b/dokimi/tests/test_novelty_atoms.py @@ -249,6 +249,27 @@ def test_a_capped_extraction_records_the_last_span_its_atoms_reached(tmp_path): assert json.loads(path.read_text())["extraction"]["reached"] == report.reached +def test_a_span_id_no_call_was_shown_does_not_move_the_reach(): + """The cap binds on the first batch's last span (s0010); one atom also cites s0055.""" + def made_up(prompt, system): + # Two atoms for each of s0001-s0005 and three for s0006-s0010: 25 in all. + rows = [dict(row, statement=f"{row['statement']} ({k})", question=f"q{k} {row['span_ids'][0]}") + for row in _rows(prompt) for k in range(2 if row["span_ids"][0] <= "s0005" else 3)] + rows[4]["span_ids"] = ["s0003", "s0055"] + return json.dumps(rows), None + + spans = segment("\n\n".join(f"Setting number {i} defaults to {i * 10}." for i in range(1, 61))) + report = ExtractionReport() + atoms = extract_atoms(spans, client=LLMClient(model="x", transport=made_up), + batch_spans=10, cap=25, report=report) + assert "s0055" in atoms[4].span_ids and report.batches == 1 + assert report.reached == {"last_span": "s0010", "spans": 10, "of": 60, + "tokens_share": 0.1666} + # An atom set that cites nothing it was shown reaches the last span sent. + first_ten = {span.id for span in spans[:10]} + assert _cap_reach(spans, [SimpleNamespace(span_ids=["s0055"])], 1, first_ten)["spans"] == 10 + + @pytest.mark.parametrize("cap, reached", [ (None, None), (31, None), @@ -552,6 +573,6 @@ def transport(prompt, system): ]) def test_a_cap_reach_share_is_floored_not_rounded(first, share, shown): spans = [SimpleNamespace(id="s0001", tokens=first), SimpleNamespace(id="s0002", tokens=30_000 - first)] - reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1) + reach = _cap_reach(spans, [SimpleNamespace(span_ids=["s0001"])], cap=1, sent={"s0001", "s0002"}) assert reach == {"last_span": "s0001", "spans": 1, "of": 2, "tokens_share": share} assert floored_percent(reach["tokens_share"]) == shown From e089ea3c9e7019f1a331b4eeed61f157ca6231df Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:02:49 +0000 Subject: [PATCH 05/16] fix(novelty): the cap forfeit and gate reason stop calling atoms a sample Answers review comment 4418984232. The cap warning no longer calls the capped atoms a sample, but the tier forfeit printed beside it still said "`added` is a sampled rate over the atom cap", and the harmful_atoms gate reason said "0 conflicts among the sampled atoms". The forfeit now reads "`added` is a rate over the opening of the artifact, up to the atom cap: ...", and the gate reason "0 conflicts among the atoms of the artifact's opening". The FORFEITS key "sampled" and the atoms.sampling values stay: code and stored runs name them. skill-novelty.md (conflict coverage, the tier forfeits) and the schema description of atoms.sampling drop the same claim. The settings-parity test maps main's forfeit to the new one, as it does for the cap warning; the tiers test that found the shortening forfeit by "opening of the artifact" now matches its own wording. Changelog fragment updated. Signed-off-by: Anton Volkov --- .../20261029_120000_antonwolfy_novelty-cap-reach.md | 5 ++++- dokimi/docs/skill-novelty.md | 6 ++++-- dokimi/dokimi/gate/novelty_gate.py | 2 +- dokimi/dokimi/novelty/tiers.py | 10 ++++++---- dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_analyze.py | 6 +++--- dokimi/tests/test_novelty_gate.py | 2 ++ dokimi/tests/test_novelty_settings_parity.py | 7 ++++++- dokimi/tests/test_novelty_tiers.py | 10 +++++----- 9 files changed, 32 insertions(+), 18 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index d75b14110..cb674d48b 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -8,4 +8,7 @@ `atoms.json` (`null` when the cap did not bind); the `novelty.md` header states it; and on a capped run the headline's `coverage:` line says the readable words went into the input and the capped atoms come from the first - N% of its tokens, instead of saying they were measured. + N% of its tokens, instead of saying they were measured. The tier forfeit + (`added` is a rate over the opening of the artifact, up to the atom cap) and + the `harmful_atoms` gate reason (0 conflicts among the atoms of the + artifact's opening) no longer call the capped atoms a sample either. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index a1b90ede4..3c4b1e748 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -134,7 +134,7 @@ marked. Read it as a review queue: extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which is a sample when + Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which cover only the artifact's opening when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -893,7 +893,9 @@ Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the -sampled-rate ones. `--estimate-only` cannot know the atom count yet, so it lists them: +cap's ones (that `added` is a rate over the opening of the artifact, up to the atom cap, and +that `frac_carrying_nothing` is not interpretable on a capped atom set). +`--estimate-only` cannot know the atom count yet, so it lists them: > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording diff --git a/dokimi/dokimi/gate/novelty_gate.py b/dokimi/dokimi/gate/novelty_gate.py index 4e4315f6c..dc3aa6b55 100644 --- a/dokimi/dokimi/gate/novelty_gate.py +++ b/dokimi/dokimi/gate/novelty_gate.py @@ -196,7 +196,7 @@ def _conflict_reading(artifact: dict, cfg: dict[str, Any]) -> tuple[int, int, bo elif capped: reason = ( f"the atom set was capped ({spans.get('n_atoms_indexed')} indexed), so 0 conflicts " - "among the sampled atoms is not a cleared check of the artifact" + "among the atoms of the artifact's opening is not a cleared check of the artifact" ) else: reason = None diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 35bf3fc41..436293a7e 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -28,11 +28,13 @@ #: artifact reached). The forfeits of a run are derived from its resolved settings and, #: once known, its atom count -- not fixed per tier -- so `--no-verify` adds the #: faithfulness forfeit, and `--all-atoms` or an atom set under the cap drops the -#: sampled-rate ones. +#: cap's ones. The key's name is old: the cap is not a sample, extraction stops at it +#: in document order. FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a sampled rate over the atom cap: usable as a sign, not as a figure, " - "and not comparable with another artifact's; pass --all-atoms for the full sweep", + "`added` is a rate over the opening of the artifact, up to the atom cap: usable as " + "a sign, not as a figure, and not comparable with another artifact's; pass " + "--all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " "as 0.743 on the study's largest artifact)", ], @@ -217,7 +219,7 @@ def estimate(self, n_spans: int, *, n_tokens: int | None = None, rows, yield an atom each), capped at ``max_atoms``. Without ``n_tokens`` only the span rule applies. The forfeits assume the cap is reached, since the atom count is a guess until extraction runs: an artifact with fewer atoms than the cap drops the - sampled-rate forfeits. + cap's forfeits. """ cap = self.values["max_atoms"] expected = max(1, n_spans // 2) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index acee71a36..b277ba1fb 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -71,7 +71,7 @@ }, "sampling": { "type": "string", - "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a sampled run is never read as a full sweep." + "description": "How the probed subset was chosen (e.g. cap-per-artifact:120). Present so a capped run is never read as a full sweep." }, "faithfulness": { "type": ["object", "null"], diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 83c682028..01109b066 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -577,13 +577,13 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): judge=judge).artifact assert artifact["run_config"]["tier"]["name"] == "triage" forfeits = [w for w in artifact["warnings"] if w.startswith("tier `triage` forfeits:")] - # Two atoms under a cap of 25: the run is not sampled, so the sampled-rate forfeits - # the tier states up front are dropped from what it records. + # Two atoms under a cap of 25: the cap did not bind, so the cap's forfeits the tier + # states up front are dropped from what it records. assert artifact["atoms"]["n"] < 25 assert len(forfeits) == len(TIERS["triage"]["forfeits"]) - len(FORFEITS["sampled"]) assert sorted(artifact["run_config"]["tier"]["forfeits"]) == sorted( w.split("forfeits: ", 1)[1] for w in forfeits) - assert not any("sampled rate" in w for w in forfeits) + assert not any("rate over the opening" in w for w in forfeits) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) diff --git a/dokimi/tests/test_novelty_gate.py b/dokimi/tests/test_novelty_gate.py index 4df05c88d..3c7287fa8 100644 --- a/dokimi/tests/test_novelty_gate.py +++ b/dokimi/tests/test_novelty_gate.py @@ -469,6 +469,8 @@ def test_a_capped_atom_set_does_not_clear_harmful_atoms(): capped = _named(evaluate_novelty(_artifact(spans=spans, conflict_coverage=full)), "harmful_atoms") assert capped.measured is False and "the atom set was capped (30 indexed)" in capped.message + # The cap keeps the artifact's opening, in document order: not a sample of it. + assert "0 conflicts among the atoms of the artifact's opening" in capped.message found = _named(evaluate_novelty(_artifact( spans=spans, conflict_coverage=full, categories={"harmful_conflicting": 2})), diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 4354e8c99..061ecab6c 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -247,11 +247,16 @@ def _kept_estimate(step: dict) -> dict: CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the " "tokens): rates describe the opening of the artifact, not the whole; pass " "--all-atoms for the full sweep") +# Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier +# record, the warnings and the estimate's stderr. +CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" +CAP_FORFEIT = "`added` is a rate over the opening of the artifact, up to the atom cap: usable" @pytest.mark.parametrize("name", list(CASES)) def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): - golden = json.loads(GOLDEN.read_text(encoding="utf-8")) + golden = json.loads(json.dumps(json.loads(GOLDEN.read_text(encoding="utf-8"))) + .replace(CAP_FORFEIT_ON_MAIN, CAP_FORFEIT)) steps = _capture(name, tmp_path) coverage = [step.pop("coverage") for step in steps] for step_coverage in coverage: diff --git a/dokimi/tests/test_novelty_tiers.py b/dokimi/tests/test_novelty_tiers.py index 52aeb6ba1..8133eee3d 100644 --- a/dokimi/tests/test_novelty_tiers.py +++ b/dokimi/tests/test_novelty_tiers.py @@ -226,7 +226,7 @@ def test_forfeits_follow_the_resolved_settings(): assert sampled in forfeits(standard) assert sampled not in forfeits(standard, n_atoms=15) assert sampled in forfeits(standard, n_atoms=60) - # --all-atoms drops the sampled-rate forfeit, triage included. + # --all-atoms drops the cap's forfeit, triage included. args = _parse("novelty", "analyze", "--artifact", "skill:x", "--tier", "triage", "--all-atoms") assert sampled not in resolve_settings(args.tier, _flags(args)).forfeits() @@ -276,10 +276,10 @@ def test_shortening_under_a_reached_cap_says_it_saw_only_the_opening(): on = {"max_atoms": 60, "shorten_artifact": True, "paraphrases": 1, "reversals": True, "verify_faithfulness": True, "classify_content": True, "confirm_carriers": True} - assert any("opening of the artifact" in f for f in forfeits(on, n_atoms=60)) - assert not any("opening of the artifact" in f for f in forfeits(on, n_atoms=10)) - assert not any("opening of the artifact" in f for f in forfeits(dict(on, max_atoms=None))) - assert not any("opening of the artifact" in f + assert any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=60)) + assert not any("shortening candidates are scored only" in f for f in forfeits(on, n_atoms=10)) + assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, max_atoms=None))) + assert not any("shortening candidates are scored only" in f for f in forfeits(dict(on, shorten_artifact=False), n_atoms=60)) From 115edbc6d6111f3f770c39aed00685aa0e262632 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:04:51 +0000 Subject: [PATCH 06/16] fix(novelty): novelty atoms prints the cap warning analyze records Answers review comment 4418984237. `novelty atoms` saved extraction.reached in atoms.json but printed only the unread-span warnings and the span summary, and probe and shorten do not repeat it, so in the atoms -> probe flow the reach was visible only in the JSON. cmd_novelty_atoms now prints report.cap_warning on stderr when the cap binds, the same text analyze adds to its warnings. The settings-parity test expects it on atoms/max-atoms-1, where main printed nothing. Folding the atoms import onto two lines keeps commands/novelty.py under its growth baseline, which drops to 1190. skill-novelty.md and the changelog fragment say so. Signed-off-by: Anton Volkov --- ...20261029_120000_antonwolfy_novelty-cap-reach.md | 2 +- dokimi/docs/skill-novelty.md | 4 +++- dokimi/dokimi/commands/novelty.py | 14 ++++++-------- dokimi/scripts/growth_baseline.json | 2 +- dokimi/tests/test_novelty_settings_parity.py | 6 ++++++ 5 files changed, 17 insertions(+), 11 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index cb674d48b..cf1cd7e60 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -4,7 +4,7 @@ the atom cap in document order, so a capped run measures the opening of the artifact. The cap warning of `novelty analyze` now names the span it stopped at, the spans up to it and their share of the tokens, instead of calling the - atoms a sample; `extraction.reached` records the same in `novelty.json` and + atoms a sample, and `novelty atoms` now prints it too; `extraction.reached` records the same in `novelty.json` and `atoms.json` (`null` when the cap did not bind); the `novelty.md` header states it; and on a capped run the headline's `coverage:` line says the readable words went into the input and the capped atoms come from the first diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 3c4b1e748..3ba92610c 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -905,7 +905,9 @@ capped run measures the opening of the artifact. Its warning says how far that i `atom extraction hit the cap of 25 at span s0012 (12 of 61 spans, 23% of the tokens): rates describe the opening of the artifact, not the whole; pass --all-atoms for the full sweep`, `extraction.reached` records it, and the headline's `coverage:` line then says the words -went into the input and the capped atoms come from the first 23% of its tokens. +went into the input and the capped atoms come from the first 23% of its tokens. `novelty +atoms` prints the same warning on stderr; `probe` and `shorten`, which read its +`atoms.json`, do not repeat it. That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index 259f1fb3e..206c77b69 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -140,13 +140,9 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None: def cmd_novelty_atoms(args: argparse.Namespace) -> None: """Extract atoms and spans only — the expensive stage, cached for reuse.""" - from dokimi.novelty.atoms import ( - ExtractionError, - ExtractionReport, - extract_atoms, - save_atoms, - verify_atoms, - ) + from dokimi.novelty.atoms import (ExtractionError, ExtractionReport, extract_atoms, save_atoms, + verify_atoms) + from dokimi.novelty.report import cap_warning from dokimi.novelty.spans import segment, span_summary artifact_bundle = _bundle_from_args(args) @@ -159,7 +155,9 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None: atoms = extract_atoms(spans, client=extractor, cap=cap, report=extraction) except ExtractionError as exc: raise SystemExit(str(exc)) from None - for warning in extraction.warnings(len(spans)): + # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it. + reach = [cap_warning(cap, extraction.reached)] if extraction.reached else [] + for warning in extraction.warnings(len(spans)) + reach: print(f"WARNING {warning}", file=sys.stderr) if settings.values["verify_faithfulness"]: verify_atoms(atoms, spans, client=extractor) diff --git a/dokimi/scripts/growth_baseline.json b/dokimi/scripts/growth_baseline.json index 7ad83253b..9a5240a2f 100644 --- a/dokimi/scripts/growth_baseline.json +++ b/dokimi/scripts/growth_baseline.json @@ -1,6 +1,6 @@ { "files": { - "dokimi/commands/novelty.py": {"lines": 1192, "rationale": "estimate-only projection and W1 routing cost output (#164) from main, plus the #172 routing field, --all-vs-all, tied/unscored caveat and blind-skill warning, --repeats refused under --all-vs-all; file split tracked separately"}, + "dokimi/commands/novelty.py": {"lines": 1190, "rationale": "estimate-only projection and W1 routing cost output (#164) from main, plus the #172 routing field, --all-vs-all, tied/unscored caveat and blind-skill warning, --repeats refused under --all-vs-all; file split tracked separately"}, "dokimi/harnesses/benchflow.py": {"lines": 869}, "dokimi/harnesses/docker_solver.py": {"lines": 899, "rationale": "pre-existing when the gate was introduced"}, "dokimi/llm.py": {"lines": 1118, "rationale": "grew on main in #180 (replay a stored retry) after #93 seeded this baseline; split it in its own PR"}, diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 061ecab6c..20db510e3 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -240,6 +240,8 @@ def _kept_estimate(step: dict) -> dict: # Runs main refuses only after the extraction calls it had already made. REFUSED_BEFORE_ANY_CALL = {"run/arms-without-full"} +# `novelty atoms` runs whose cap binds. +CAPPED_ATOMS = {"atoms/max-atoms-1"} # Intended change: the cap warning says how far the cap reached instead of calling the # atoms a sample. In the warnings and on stderr alike. CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " @@ -271,6 +273,10 @@ def test_no_setting_bill_or_tier_record_differs_from_main(name, tmp_path): # Intended change: the arm and placebo checks now run before the clients are # built, so the refusal makes no model call. Same status and output as on main. expected = [dict(step, calls=[]) for step in expected] + if name in CAPPED_ATOMS: + # Intended change: `novelty atoms` prints the cap warning `analyze` records. + expected = [dict(step, stderr=f"{step['stderr']}WARNING {CAP_WARNING}\n") + for step in expected] assert steps == expected From 8e133a5c9e11003161e59ca34aa168a1788af2fc Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:05:39 +0000 Subject: [PATCH 07/16] fix(novelty): the floor warning on a capped run says the words went in Answers review comment 4418984241. On a capped run the coverage line says the readable words "went into the input", but the floor warning printed under it still said "only X% of the artifact's readable words were measured". A capped entry-point run with 10 of 40 readable words printed both. The warning now takes the same verb as the coverage line, so a capped run prints "WARNING: only 25% of the artifact's readable words went into the input, so every rate here describes that part, not the artifact"; an uncapped run keeps "were measured". A test pins it with a capped run under the floor. skill-novelty.md and the changelog fragment say so. Signed-off-by: Anton Volkov --- .../20261029_120000_antonwolfy_novelty-cap-reach.md | 3 ++- dokimi/docs/skill-novelty.md | 3 ++- dokimi/dokimi/commands/novelty.py | 2 +- dokimi/tests/test_novelty_cli.py | 11 +++++++++++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index cf1cd7e60..1d4066fff 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -8,7 +8,8 @@ `atoms.json` (`null` when the cap did not bind); the `novelty.md` header states it; and on a capped run the headline's `coverage:` line says the readable words went into the input and the capped atoms come from the first - N% of its tokens, instead of saying they were measured. The tier forfeit + N% of its tokens, instead of saying they were measured (as does the warning + printed under it when that share is half or less). The tier forfeit (`added` is a rate over the opening of the artifact, up to the atom cap) and the `harmful_atoms` gate reason (0 conflicts among the atoms of the artifact's opening) no longer call the capped atoms a sample either. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 3ba92610c..8ffcb5e4a 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -905,7 +905,8 @@ capped run measures the opening of the artifact. Its warning says how far that i `atom extraction hit the cap of 25 at span s0012 (12 of 61 spans, 23% of the tokens): rates describe the opening of the artifact, not the whole; pass --all-atoms for the full sweep`, `extraction.reached` records it, and the headline's `coverage:` line then says the words -went into the input and the capped atoms come from the first 23% of its tokens. `novelty +went into the input and the capped atoms come from the first 23% of its tokens (the warning +under that line, when half or less of the words went in, says "went into the input" too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its `atoms.json`, do not repeat it. diff --git a/dokimi/dokimi/commands/novelty.py b/dokimi/dokimi/commands/novelty.py index 206c77b69..4c87c3b31 100644 --- a/dokimi/dokimi/commands/novelty.py +++ b/dokimi/dokimi/commands/novelty.py @@ -1186,5 +1186,5 @@ def _print_coverage(bundle: dict, extraction: dict | None = None) -> None: pct = floored_percent(share) print(f"coverage: {pct} of the artifact's readable words {went}{scope}{reach}") if share <= MEASURED_SHARE_WARN_AT: - print(f"WARNING: only {pct} of the artifact's readable words were measured, so every " + print(f"WARNING: only {pct} of the artifact's readable words {went}, so every " f"rate here describes that part, not the artifact{rerun}") diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index 21f449e96..8172ad7d6 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -861,6 +861,17 @@ def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(ca in capsys.readouterr().out +def test_a_capped_run_under_the_floor_warns_in_the_coverage_line_s_words(capsys): + # 10 of the 40 readable words went into the input, and the atoms stopped short of them. + reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} + novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=_readable_record(), + extraction={"reached": reached})) + out = capsys.readouterr().out + assert ("WARNING: only 25% of the artifact's readable words went into the input, so every " + "rate here describes that part, not the artifact") in out + assert "measured" not in out + + def test_a_bundle_run_capped_by_the_budget_warns_without_advising_the_flag_it_passed(capsys): # --include-resources was passed; the budget left most of the reference unread. The # stderr warning list is silent about this, the stdout line is not, and it must not tell From b9bf52b73b6ef6d780db67b7644656117a732b88 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:06:12 +0000 Subject: [PATCH 08/16] fix(schemas): extraction.reached requires all four of its keys Answers review comments 4418984247 and 4420750589. The reached object had no required list, so {} or {"spans": 3} validated, while describe_reach, reach_is_end and coverage_reach read every key and raise KeyError on a partial one. reached now requires last_span, spans, of and tokens_share. The key itself stays optional and nullable, and it is new in this PR, so no stored artifact carries a partial one; schema_compat against the base reports the narrowing as a warning on a property no fixture writes. The capped analyze test pins that {"spans": 3} is rejected. Signed-off-by: Anton Volkov --- dokimi/dokimi/schemas/novelty.v1.json | 3 ++- dokimi/tests/test_novelty_analyze.py | 4 ++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index b277ba1fb..43ec6ab52 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -344,7 +344,8 @@ "spans": {"type": "integer", "minimum": 1}, "of": {"type": "integer", "minimum": 1}, "tokens_share": {"type": "number", "minimum": 0, "maximum": 1} - } + }, + "required": ["last_span", "spans", "of", "tokens_share"] } } }, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 01109b066..a29de14cb 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -602,6 +602,10 @@ def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path "s0002 (2 of 5 spans, 35% of the tokens): the atoms come from the opening of the " "artifact, not the whole.") in render_novelty_markdown(artifact) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) + # The renderers read every key of the reach, so a partial one does not validate. + partial = dict(artifact["extraction"], reached={"spans": 3}) + with pytest.raises(ArtifactValidationError, match="extraction/reached: 'last_span' is a required"): + validate_artifact("novelty", dict(artifact, schema_version="novelty.v1", extraction=partial)) def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): From 041a31114bd63e246d511d34c494c21fe70ae2c8 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:06:44 +0000 Subject: [PATCH 09/16] docs(novelty): describe the cap met on the artifact's last span Answers review comment 4420750586. skill-novelty.md and the changelog fragment gave only the "opening of the artifact" wording, while a cap met on the last span (report.reach_is_end) words the warning, the novelty.md header and the coverage: line differently. skill-novelty.md now quotes that case: the warning ending "any atoms past the cap in its last reply were dropped", the header saying the atoms reach the artifact's last span and the cap dropped any atoms past it in the last reply, and the coverage: tail. The changelog fragment names the case too. Signed-off-by: Anton Volkov --- ...029_120000_antonwolfy_novelty-cap-reach.md | 21 +++++++++++-------- dokimi/docs/skill-novelty.md | 12 +++++++++-- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index 1d4066fff..f3cda3070 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -4,12 +4,15 @@ the atom cap in document order, so a capped run measures the opening of the artifact. The cap warning of `novelty analyze` now names the span it stopped at, the spans up to it and their share of the tokens, instead of calling the - atoms a sample, and `novelty atoms` now prints it too; `extraction.reached` records the same in `novelty.json` and - `atoms.json` (`null` when the cap did not bind); the `novelty.md` header - states it; and on a capped run the headline's `coverage:` line says the - readable words went into the input and the capped atoms come from the first - N% of its tokens, instead of saying they were measured (as does the warning - printed under it when that share is half or less). The tier forfeit - (`added` is a rate over the opening of the artifact, up to the atom cap) and - the `harmful_atoms` gate reason (0 conflicts among the atoms of the - artifact's opening) no longer call the capped atoms a sample either. + atoms a sample, and `novelty atoms` now prints it too; `extraction.reached` + records the same in `novelty.json` and `atoms.json` (`null` when the cap did + not bind); the `novelty.md` header states it; and on a capped run the + headline's `coverage:` line says the readable words went into the input and + the capped atoms come from the first N% of its tokens, instead of saying + they were measured (as does the warning printed under it when that share is + half or less). A cap met on the last span says instead that the atoms reach + it and any atoms past the cap in the last reply were dropped, in the + warning, the header and the `coverage:` line. The tier forfeit (`added` is + a rate over the opening of the artifact, up to the atom cap) and the + `harmful_atoms` gate reason (0 conflicts among the atoms of the artifact's + opening) no longer call the capped atoms a sample either. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 8ffcb5e4a..935ae9667 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -906,10 +906,18 @@ capped run measures the opening of the artifact. Its warning says how far that i describe the opening of the artifact, not the whole; pass --all-atoms for the full sweep`, `extraction.reached` records it, and the headline's `coverage:` line then says the words went into the input and the capped atoms come from the first 23% of its tokens (the warning -under that line, when half or less of the words went in, says "went into the input" too). `novelty -atoms` prints the same warning on stderr; `probe` and `shorten`, which read its +under that line, when half or less of the words went in, says "went into the input" too). +`novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its `atoms.json`, do not repeat it. +A cap met on the artifact's last span leaves no stretch of it out: what it cut is any atoms +past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at +span s0010 (10 of 10 spans, 100% of the tokens): any atoms past the cap in its last reply were +dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the atoms reach the +artifact's last span and the cap dropped any atoms past it in the last reply, and the +`coverage:` line ends `the capped atoms reach its last span, and the cap dropped any atoms past +it`; none of them says "the opening". + That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. From fa5347ae5a39485a193f79c2f9bb96ecefe40e37 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:07:05 +0000 Subject: [PATCH 10/16] test(novelty): import cap_warning once; pin the mid-artifact header Answers review comment 4420750597. test_novelty_report.py imported cap_warning inside two test bodies though it already imports from dokimi.novelty.report at the top; it now comes in with the other names. The test of a cap that stopped inside the artifact now also asserts the novelty.md header sentence _capped_at gives it ("the atoms come from the opening of the artifact, not the whole"), which only test_novelty_analyze covered before. Signed-off-by: Anton Volkov --- dokimi/tests/test_novelty_report.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index a67ab9a7c..d0bd84e63 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -5,7 +5,7 @@ import pytest -from dokimi.novelty.report import _capped_at, coverage_reach, render_novelty_markdown +from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach, render_novelty_markdown def _artifact(**overrides): @@ -76,8 +76,6 @@ def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_th assert coverage_reach({"reached": end}) == ( "went into the input", "; the capped atoms reach its last span, and the cap dropped any atoms past it") assert "opening" not in _capped_at({"extraction": {"reached": end}}) - from dokimi.novelty.report import cap_warning - assert cap_warning(30, end) == ( "atom extraction hit the cap of 30 at span s0010 (10 of 10 spans, 100% of the tokens): any atoms past " "the cap in its last reply were dropped; pass --all-atoms for the full sweep") @@ -87,8 +85,9 @@ def test_a_cap_that_stopped_inside_the_artifact_still_says_the_rates_describe_th inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} assert coverage_reach({"reached": inside}) == ( "went into the input", "; the capped atoms come from the first 35% of its tokens") - from dokimi.novelty.report import cap_warning - + assert _capped_at({"extraction": {"reached": inside}}) == ( + " The atom cap stopped extraction at span s0002 (2 of 5 spans, 35% of the tokens): the atoms " + "come from the opening of the artifact, not the whole.") assert cap_warning(1, inside) == ( "atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the tokens): rates describe " "the opening of the artifact, not the whole; pass --all-atoms for the full sweep") From b3e103257f8abd918466d04c81c21011ce32f89c Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 11:14:01 +0100 Subject: [PATCH 11/16] fix(novelty): narrow max_atoms before the cap warning, for mypy analyze passed config.max_atoms (int | None) to report.cap_warning, which takes an int, so mypy failed on the branch. extraction.reached is set only when the cap binds, so the guard changes no output. Signed-off-by: Anton Volkov --- dokimi/dokimi/novelty/analyze.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dokimi/dokimi/novelty/analyze.py b/dokimi/dokimi/novelty/analyze.py index fb0b1cb1a..b87522c07 100644 --- a/dokimi/dokimi/novelty/analyze.py +++ b/dokimi/dokimi/novelty/analyze.py @@ -182,7 +182,7 @@ def analyze( sampling = ( f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full" ) - if extraction.reached: + if extraction.reached and config.max_atoms is not None: # reached is set only under a cap warnings.append(report.cap_warning(config.max_atoms, extraction.reached)) if not atom_list: warnings.append("no atoms with span provenance were extracted; nothing to measure") From 689c109cca24302269f27d85f579b0455fa6e087 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 10:44:13 +0000 Subject: [PATCH 12/16] style(novelty): keep the FORFEITS comment to two lines The static job's comment-block check from #298 (scripts/check_comment_blocks.py) failed this branch once main was merged in: the six-line #: block above FORFEITS in novelty/tiers.py has a line this PR edits, and a block of three or more lines fails once a PR touches it. The block now says in two lines what the code does not: forfeits are keyed by the setting that is off, and "sampled" is a reached atom cap, an old name kept although the cap takes atoms in document order, with a link to #299 for the backstory. How forfeits() derives them from the settings and the atom count is left to the function. No behaviour changes. Signed-off-by: Anton Volkov --- dokimi/dokimi/novelty/tiers.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index 436293a7e..a59aa6692 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -24,12 +24,8 @@ from dataclasses import dataclass, field from typing import Any -#: What a run gives up, keyed by the setting that is off ("sampled": an atom cap the -#: artifact reached). The forfeits of a run are derived from its resolved settings and, -#: once known, its atom count -- not fixed per tier -- so `--no-verify` adds the -#: faithfulness forfeit, and `--all-atoms` or an atom set under the cap drops the -#: cap's ones. The key's name is old: the cap is not a sample, extraction stops at it -#: in document order. +#: What a run gives up, keyed by the setting that is off. "sampled" is a reached atom cap; the +#: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 FORFEITS: dict[str, list[str]] = { "sampled": [ "`added` is a rate over the opening of the artifact, up to the atom cap: usable as " From 3e576e0d3cdf780b3d7ee7f00d9d5c1fc7a66bcd Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 11:06:16 +0000 Subject: [PATCH 13/16] fix(novelty): word the cap reach as the input's, not the artifact's Answers review comment 4422880472. extraction.reached counts the spans of the text that was segmented: the entry point, plus on a bundle run the files its roles and budgets selected. A bundled file left out is in no span, yet the cap warning, the novelty.md header and the headline's coverage: line said the atoms covered the opening of the artifact or reached the artifact's last span. The reach now names the input in both branches: "(12 of the input's 61 spans, 23% of its tokens)", "rates describe the opening of the input, not all of it", "come from the first 23% of the input's tokens" and "reach the input's last span". The coverage: line keeps saying how much of the artifact went into the input. The keys and values of extraction.reached are unchanged; its schema description, the docstrings, docs/skill-novelty.md, docs/reviewing.md and the changelog fragment now say the input. A capped bundle run with a reference over the file budget pins the header and the coverage: line, mid-input and on the last span; the tests and settings-parity mapping quoting the old words follow. Signed-off-by: Anton Volkov --- ...029_120000_antonwolfy_novelty-cap-reach.md | 30 +++++----- dokimi/docs/reviewing.md | 3 +- dokimi/docs/skill-novelty.md | 29 +++++----- dokimi/dokimi/novelty/atoms.py | 7 ++- dokimi/dokimi/novelty/report.py | 28 +++++---- dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_analyze.py | 58 +++++++++++++++++-- dokimi/tests/test_novelty_cli.py | 6 +- dokimi/tests/test_novelty_report.py | 19 +++--- dokimi/tests/test_novelty_settings_parity.py | 4 +- 10 files changed, 125 insertions(+), 61 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index f3cda3070..d6ae50e72 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -2,17 +2,19 @@ - **A capped novelty run says how far the cap reached** — extraction stops at the atom cap in document order, so a capped run measures the opening of the - artifact. The cap warning of `novelty analyze` now names the span it stopped - at, the spans up to it and their share of the tokens, instead of calling the - atoms a sample, and `novelty atoms` now prints it too; `extraction.reached` - records the same in `novelty.json` and `atoms.json` (`null` when the cap did - not bind); the `novelty.md` header states it; and on a capped run the - headline's `coverage:` line says the readable words went into the input and - the capped atoms come from the first N% of its tokens, instead of saying - they were measured (as does the warning printed under it when that share is - half or less). A cap met on the last span says instead that the atoms reach - it and any atoms past the cap in the last reply were dropped, in the - warning, the header and the `coverage:` line. The tier forfeit (`added` is - a rate over the opening of the artifact, up to the atom cap) and the - `harmful_atoms` gate reason (0 conflicts among the atoms of the artifact's - opening) no longer call the capped atoms a sample either. + input: the text that was segmented, which leaves out the bundled files a run + did not select. The cap warning of `novelty analyze` now names the span it + stopped at, the input's spans up to it and their share of the input's + tokens, instead of calling the atoms a sample, and `novelty atoms` now + prints it too; `extraction.reached` records the same in `novelty.json` and + `atoms.json` (`null` when the cap did not bind); the `novelty.md` header + states it; and on a capped run the headline's `coverage:` line says the + readable words went into the input and the capped atoms come from the first + N% of the input's tokens, instead of saying they were measured (as does the + warning printed under it when that share is half or less). A cap met on the + input's last span says instead that the atoms reach it and any atoms past + the cap in the last reply were dropped, in the warning, the header and the + `coverage:` line. The tier forfeit (`added` is a rate over the opening of + the artifact, up to the atom cap) and the `harmful_atoms` gate reason (0 + conflicts among the atoms of the artifact's opening) no longer call the + capped atoms a sample either. diff --git a/dokimi/docs/reviewing.md b/dokimi/docs/reviewing.md index 10e3da38b..8a09a582f 100644 --- a/dokimi/docs/reviewing.md +++ b/dokimi/docs/reviewing.md @@ -84,7 +84,8 @@ self-judge; `arms run` does not check, so name - **At triage** with `--include-resources`, the 25-atom cap takes atoms in document order, and the entry point comes first: the flag reaches `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap - warning prints how far it reached (span, spans and token share). For such skills, + warning prints how far it reached through the input, the entry point and the files the budgets selected (span, + spans and token share). For such skills, use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for a full sweep; run `--estimate-only` first, it costs more. - Step 5 needs evaluation content: a golden question set for 5a, executable diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index 935ae9667..ab22b1cdf 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -901,22 +901,25 @@ that `frac_carrying_nothing` is not interpretable on a capped atom set). > against the model matching the artifact's own wording The cap is not a sample: extraction walks the spans in document order and stops at it, so a -capped run measures the opening of the artifact. Its warning says how far that is, as in -`atom extraction hit the cap of 25 at span s0012 (12 of 61 spans, 23% of the tokens): rates -describe the opening of the artifact, not the whole; pass --all-atoms for the full sweep`, +capped run measures the opening of the input. The input is the text that was segmented: the +entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file +left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the +artifact went into the input). The warning says how far the cap reached, as in `atom +extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens): +rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`, `extraction.reached` records it, and the headline's `coverage:` line then says the words -went into the input and the capped atoms come from the first 23% of its tokens (the warning -under that line, when half or less of the words went in, says "went into the input" too). -`novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its +went into the input and the capped atoms come from the first 23% of the input's tokens (the +warning under that line, when half or less of the words went in, says "went into the input" +too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its `atoms.json`, do not repeat it. -A cap met on the artifact's last span leaves no stretch of it out: what it cut is any atoms +A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at -span s0010 (10 of 10 spans, 100% of the tokens): any atoms past the cap in its last reply were -dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the atoms reach the -artifact's last span and the cap dropped any atoms past it in the last reply, and the -`coverage:` line ends `the capped atoms reach its last span, and the cap dropped any atoms past -it`; none of them says "the opening". +span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last +reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the +atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and +the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped +any atoms past it`; none of them says "the opening". That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a sweep is not. @@ -984,7 +987,7 @@ lost to a failed call (with its error). The | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked | | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting | | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded | -| `reached` | on a capped run, how far the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all spans; `tokens_share`, those spans' share of the span tokens. `null` when the cap did not bind | +| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind | `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index be630457f..1ec62a3ff 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -198,7 +198,7 @@ class ExtractionReport: call_error: The last failed call on a span that stayed unread. Errors that splitting recovered from are not recorded: they did not lose anything, and quoting one would blame the wrong cause. - reached: How far through the artifact a capped atom set reached (see + reached: How far through the input a capped atom set reached (see :func:`_cap_reach`); ``None`` when the cap did not bind. """ @@ -419,10 +419,11 @@ def parse(reply: str) -> list: def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, sent: set[str]) -> dict | None: - """How far through the artifact a capped atom set reached; ``None`` under the cap. + """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap. Extraction walks the spans in document order and stops at the cap, so a capped set - covers the opening of the artifact, not a sample of it. The reach is the last span any + covers the opening of the input, not a sample of it. The input is the text segmented, + not the artifact: a bundled file it left out counts in no span. The reach is the last span any kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span of the document its call was never shown, and that made-up id must not move the reach diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index c4698c4a3..685dd839d 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -240,21 +240,27 @@ def floored_percent(share: float) -> str: def describe_reach(reached: dict) -> str: - """Where a capped extraction stopped: ``span s0012 (12 of 61 spans, 23% of the tokens)``.""" - return (f"span {reached['last_span']} ({reached['spans']} of {reached['of']} spans, " - f"{floored_percent(reached['tokens_share'])} of the tokens)") + """Where a capped extraction stopped, as ``span s0012 (12 of the input's 61 spans, 23% of + its tokens)``. + + The spans are the input's, the text that was segmented: the entry point, plus on a bundle + run the files its roles and budgets selected. Not the artifact's, whose other files are + not in it. + """ + return (f"span {reached['last_span']} ({reached['spans']} of the input's {reached['of']} " + f"spans, {floored_percent(reached['tokens_share'])} of its tokens)") def reach_is_end(reached: dict) -> bool: - """Whether a binding cap stopped on the artifact's last span: the atoms then reach its end, and what the - cap cut is the atoms past it in the last reply, not a stretch of the artifact.""" + """Whether a binding cap stopped on the input's last span: the atoms then reach its end, and what the + cap cut is the atoms past it in the last reply, not a stretch of the input.""" return reached["spans"] == reached["of"] def cap_warning(cap: int, reached: dict) -> str: """The warning for a binding cap: where extraction stopped, and what that leaves out.""" left_out = ("any atoms past the cap in its last reply were dropped" if reach_is_end(reached) - else "rates describe the opening of the artifact, not the whole") + else "rates describe the opening of the input, not all of it") return (f"atom extraction hit the cap of {cap} at {describe_reach(reached)}: {left_out}; " "pass --all-atoms for the full sweep") @@ -270,10 +276,10 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: if not reached: return "were measured", "" if reach_is_end(reached): - return ("went into the input", "; the capped atoms reach its last span, and the cap " - "dropped any atoms past it") + return ("went into the input", "; the capped atoms reach the input's last span, and the " + "cap dropped any atoms past it") return ("went into the input", "; the capped atoms come from the first " - f"{floored_percent(reached['tokens_share'])} of its tokens") + f"{floored_percent(reached['tokens_share'])} of the input's tokens") def _capped_at(artifact: dict) -> str: @@ -283,9 +289,9 @@ def _capped_at(artifact: dict) -> str: return "" if reach_is_end(reached): return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms reach the " - "artifact's last span, and the cap dropped any atoms past it in the last reply.") + "input's last span, and the cap dropped any atoms past it in the last reply.") return (f" The atom cap stopped extraction at {describe_reach(reached)}: the atoms " - "come from the opening of the artifact, not the whole.") + "come from the opening of the input, not all of it.") def _render_bundle(artifact: dict) -> list[str]: diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index 43ec6ab52..dedc0f4d7 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -338,7 +338,7 @@ "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, "reached": { "type": ["object", "null"], - "description": "How far through the artifact a capped atom set reached; null when the cap did not bind. Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the artifact, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of spans up to and including it, of all spans, and tokens_share those spans' share of the span tokens. Absent from artifacts written before it existed.", + "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the input, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", "properties": { "last_span": {"type": "string"}, "spans": {"type": "integer", "minimum": 1}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index a29de14cb..86fe5337f 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -6,6 +6,7 @@ from __future__ import annotations import json +import re import pytest @@ -593,14 +594,14 @@ def test_a_capped_run_says_how_far_through_the_artifact_the_cap_reached(tmp_path # s0001 and s0002 are 15 of the body's 42 span tokens. reached = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} assert artifact["extraction"]["reached"] == reached - warning = ("atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the " - "tokens): rates describe the opening of the artifact, not the whole; pass " + warning = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " + "of its tokens): rates describe the opening of the input, not all of it; pass " "--all-atoms for the full sweep") assert warning in artifact["warnings"] assert not any("sampled atoms" in w for w in artifact["warnings"]) assert ("1 atoms (cap-per-artifact:1), 1 probed. The atom cap stopped extraction at span " - "s0002 (2 of 5 spans, 35% of the tokens): the atoms come from the opening of the " - "artifact, not the whole.") in render_novelty_markdown(artifact) + "s0002 (2 of the input's 5 spans, 35% of its tokens): the atoms come from the " + "opening of the input, not all of it.") in render_novelty_markdown(artifact) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) # The renderers read every key of the reach, so a partial one does not validate. partial = dict(artifact["extraction"], reached={"spans": 3}) @@ -620,6 +621,55 @@ def test_the_reach_is_null_only_when_the_cap_did_not_bind(tmp_path): validate_artifact("novelty", dict(uncapped, schema_version="novelty.v1")) +@pytest.mark.parametrize("cap, header, tail", [ + (2, "s0002 (2 of the input's 8 spans, 25% of its tokens): the atoms come from the opening " + "of the input, not all of it.", "come from the first 25% of the input's tokens"), + (8, "s0008 (8 of the input's 8 spans, 100% of its tokens): the atoms reach the input's last " + "span, and the cap dropped any atoms past it in the last reply.", + "reach the input's last span, and the cap dropped any atoms past it"), +], ids=["mid-input", "last-span"]) +def test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact(tmp_path, capsys, cap, + header, tail): + from dokimi.commands import novelty as novelty_cmd + from dokimi.novelty.bundle import load_bundle + + root = tmp_path / "accel-tuning" + (root / "references").mkdir(parents=True) + (root / "SKILL.md").write_text(SKILL, encoding="utf-8") + (root / "references" / "tuning.md").write_text( + "# Tuning\n\nSet `--threads` to the core count.\n", encoding="utf-8") + # 500 words over the 100-word file budget: on disk, in no span the reach counts. + (root / "references" / "zlarge.md").write_text( + " ".join(f"word{i}" for i in range(500)), encoding="utf-8") + bundle = load_bundle(f"skill:{root}", include_resources=True, max_file_words=100) + assert [f.path for f in bundle.files] == ["SKILL.md", "references/tuning.md"] + scripted = _scripted("informed") + + def per_span(prompt, system): + if "Extract the checkable knowledge" not in prompt: + return scripted(prompt, system) + return json.dumps([dict(_ATOMS[0], statement=f"{sid} holds.", question=f"What of {sid}?", + span_ids=[sid]) + for sid in re.findall(r"^\[(s\d+)\]", prompt, flags=re.M)]), None + + config = AnalyzeConfig( + artifact_ref=f"skill:{root}", artifact_kind=bundle.kind, name=bundle.name, + description=bundle.description, frontmatter=bundle.frontmatter, concurrency=1, + include_resources=True, max_atoms=cap, + ) + _, answerer, judge = _clients() + extractor = LLMClient(model="extractor-m", role="extractor", transport=per_span) + artifact = analyze(bundle.text, config, extractor=extractor, answerer=answerer, + judge=judge, artifact_bundle=bundle).artifact + assert artifact["extraction"]["reached"]["of"] == 8 + assert f"The atom cap stopped extraction at span {header}" in render_novelty_markdown(artifact) + novelty_cmd._print_coverage(artifact["bundle"], artifact["extraction"]) + # 6% of the artifact went into the input; the reach is a share of the input alone. + assert capsys.readouterr().out.startswith( + "coverage: 6% of the artifact's readable words went into the input; the capped atoms " + f"{tail}\n") + + def test_skipping_the_faithfulness_check_is_recorded_as_a_warning(tmp_path): result = _analyze(tmp_path, verify_faithfulness=False) assert result.artifact["atoms"]["faithfulness"] == {} diff --git a/dokimi/tests/test_novelty_cli.py b/dokimi/tests/test_novelty_cli.py index 8172ad7d6..f6878035a 100644 --- a/dokimi/tests/test_novelty_cli.py +++ b/dokimi/tests/test_novelty_cli.py @@ -848,13 +848,13 @@ def test_a_capped_run_s_headline_does_not_say_the_whole_artifact_was_measured(ca extraction={"reached": reached})) out = capsys.readouterr().out assert ("coverage: 100% of the artifact's readable words went into the input (entry point " - "only); the capped atoms come from the first 22% of its tokens\n") in out + "only); the capped atoms come from the first 22% of the input's tokens\n") in out assert "measured" not in out # The unknown share keeps the reach too; a run the cap did not stop keeps its wording. novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=dict(record, unreadable_files=1), extraction={"reached": reached})) - assert "counted) (entry point only); the capped atoms come from the first 22% of its " \ - "tokens\n" in capsys.readouterr().out + assert "counted) (entry point only); the capped atoms come from the first 22% of the " \ + "input's tokens\n" in capsys.readouterr().out novelty_cmd._print_headline(dict(_minimal_artifact(), bundle=record, extraction={"reached": None})) assert "coverage: 100% of the artifact's readable words were measured (entry point only)\n" \ diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index d0bd84e63..0bf30bc95 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -74,20 +74,21 @@ def test_the_cost_line_names_replayed_calls_only_when_there_are_some(cost, line) def test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening(): end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} assert coverage_reach({"reached": end}) == ( - "went into the input", "; the capped atoms reach its last span, and the cap dropped any atoms past it") + "went into the input", "; the capped atoms reach the input's last span, and the cap dropped any atoms " + "past it") assert "opening" not in _capped_at({"extraction": {"reached": end}}) assert cap_warning(30, end) == ( - "atom extraction hit the cap of 30 at span s0010 (10 of 10 spans, 100% of the tokens): any atoms past " - "the cap in its last reply were dropped; pass --all-atoms for the full sweep") + "atom extraction hit the cap of 30 at span s0010 (10 of the input's 10 spans, 100% of its tokens): " + "any atoms past the cap in its last reply were dropped; pass --all-atoms for the full sweep") -def test_a_cap_that_stopped_inside_the_artifact_still_says_the_rates_describe_the_opening(): +def test_a_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_opening(): inside = {"last_span": "s0002", "spans": 2, "of": 5, "tokens_share": 0.3571} assert coverage_reach({"reached": inside}) == ( - "went into the input", "; the capped atoms come from the first 35% of its tokens") + "went into the input", "; the capped atoms come from the first 35% of the input's tokens") assert _capped_at({"extraction": {"reached": inside}}) == ( - " The atom cap stopped extraction at span s0002 (2 of 5 spans, 35% of the tokens): the atoms " - "come from the opening of the artifact, not the whole.") + " The atom cap stopped extraction at span s0002 (2 of the input's 5 spans, 35% of its tokens): the " + "atoms come from the opening of the input, not all of it.") assert cap_warning(1, inside) == ( - "atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the tokens): rates describe " - "the opening of the artifact, not the whole; pass --all-atoms for the full sweep") + "atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% of its tokens): rates " + "describe the opening of the input, not all of it; pass --all-atoms for the full sweep") diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 20db510e3..df77e9e1e 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -246,8 +246,8 @@ def _kept_estimate(step: dict) -> dict: # atoms a sample. In the warnings and on stderr alike. CAP_WARNING_ON_MAIN = ("atom extraction hit the cap of 1; rates describe the sampled atoms, " "not the whole artifact") -CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of 5 spans, 35% of the " - "tokens): rates describe the opening of the artifact, not the whole; pass " +CAP_WARNING = ("atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% " + "of its tokens): rates describe the opening of the input, not all of it; pass " "--all-atoms for the full sweep") # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. From 72bab67d693560f5bbfae3f578b75e1d4676ee2f Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 11:16:14 +0000 Subject: [PATCH 14/16] fix(novelty): the cap forfeit and gate reason name the input's opening Follows up review comment 4422880472, answered in 3e576e0 for the reach texts (the cap warning, the novelty.md header, the coverage: line). The cap's tier forfeit and the harmful_atoms gate reason, written earlier in this PR, still placed the capped atoms in the artifact's opening, though on a bundle run the input that was segmented leaves out the files its roles and budgets did not select. The forfeit now reads "`added` is a rate over the opening of the input, up to the atom cap: ...", and the gate reason "0 conflicts among the atoms of the input's opening is not a cleared check of the artifact". skill-novelty.md (pairs_total under a capped atom set, the tier forfeits) and the changelog fragment say the same. The settings-parity mapping (CAP_FORFEIT) and the gate test quote the new words. The shortening forfeit, older than this PR, is unchanged. Signed-off-by: Anton Volkov --- .../20261029_120000_antonwolfy_novelty-cap-reach.md | 4 ++-- dokimi/docs/skill-novelty.md | 4 ++-- dokimi/dokimi/gate/novelty_gate.py | 2 +- dokimi/dokimi/novelty/tiers.py | 2 +- dokimi/tests/test_novelty_gate.py | 4 ++-- dokimi/tests/test_novelty_settings_parity.py | 2 +- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index d6ae50e72..17a301af5 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -15,6 +15,6 @@ input's last span says instead that the atoms reach it and any atoms past the cap in the last reply were dropped, in the warning, the header and the `coverage:` line. The tier forfeit (`added` is a rate over the opening of - the artifact, up to the atom cap) and the `harmful_atoms` gate reason (0 - conflicts among the atoms of the artifact's opening) no longer call the + the input, up to the atom cap) and the `harmful_atoms` gate reason (0 + conflicts among the atoms of the input's opening) no longer call the capped atoms a sample either. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index ab22b1cdf..d548ee282 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -134,7 +134,7 @@ marked. Read it as a review queue: extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which cover only the artifact's opening when + Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which cover only the input's opening when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -893,7 +893,7 @@ Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the -cap's ones (that `added` is a rate over the opening of the artifact, up to the atom cap, and +cap's ones (that `added` is a rate over the opening of the input, up to the atom cap, and that `frac_carrying_nothing` is not interpretable on a capped atom set). `--estimate-only` cannot know the atom count yet, so it lists them: diff --git a/dokimi/dokimi/gate/novelty_gate.py b/dokimi/dokimi/gate/novelty_gate.py index dc3aa6b55..3c2cd2b8e 100644 --- a/dokimi/dokimi/gate/novelty_gate.py +++ b/dokimi/dokimi/gate/novelty_gate.py @@ -196,7 +196,7 @@ def _conflict_reading(artifact: dict, cfg: dict[str, Any]) -> tuple[int, int, bo elif capped: reason = ( f"the atom set was capped ({spans.get('n_atoms_indexed')} indexed), so 0 conflicts " - "among the atoms of the artifact's opening is not a cleared check of the artifact" + "among the atoms of the input's opening is not a cleared check of the artifact" ) else: reason = None diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index a59aa6692..c6857fee3 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -28,7 +28,7 @@ #: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a rate over the opening of the artifact, up to the atom cap: usable as " + "`added` is a rate over the opening of the input, up to the atom cap: usable as " "a sign, not as a figure, and not comparable with another artifact's; pass " "--all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " diff --git a/dokimi/tests/test_novelty_gate.py b/dokimi/tests/test_novelty_gate.py index 3c7287fa8..83d9653d0 100644 --- a/dokimi/tests/test_novelty_gate.py +++ b/dokimi/tests/test_novelty_gate.py @@ -469,8 +469,8 @@ def test_a_capped_atom_set_does_not_clear_harmful_atoms(): capped = _named(evaluate_novelty(_artifact(spans=spans, conflict_coverage=full)), "harmful_atoms") assert capped.measured is False and "the atom set was capped (30 indexed)" in capped.message - # The cap keeps the artifact's opening, in document order: not a sample of it. - assert "0 conflicts among the atoms of the artifact's opening" in capped.message + # The cap keeps the input's opening, in document order: not a sample of it. + assert "0 conflicts among the atoms of the input's opening" in capped.message found = _named(evaluate_novelty(_artifact( spans=spans, conflict_coverage=full, categories={"harmful_conflicting": 2})), diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index df77e9e1e..605d9eabe 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -252,7 +252,7 @@ def _kept_estimate(step: dict) -> dict: # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" -CAP_FORFEIT = "`added` is a rate over the opening of the artifact, up to the atom cap: usable" +CAP_FORFEIT = "`added` is a rate over the opening of the input, up to the atom cap: usable" @pytest.mark.parametrize("name", list(CASES)) From 50c27110971726b36104ce53af16b601cc59364d Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 11:27:43 +0000 Subject: [PATCH 15/16] fix(novelty): the cap forfeit and gate reason hold for any cap reach Review comment 4423042474: a cap met on the input's last span reaches the end of the input, yet the harmful_atoms gate reason still said the 0 conflicts were among "the atoms of the input's opening", and novelty.md quoted it under a header saying the atoms reach the last span. The gate reason now says "0 conflicts among the capped atoms", and the cap's tier forfeit, which --estimate-only lists before the reach is known, says `added` is a rate over the atoms the cap kept, not the whole input. The general statements that a capped run covers "the opening of the input" (skill-novelty.md, the changelog fragment, the extraction.reached schema description and two docstrings) now say it covers the input up to the span the cap stopped at; the mid-input warning and header keep "the opening". Tests: the gate assertion takes the new reason, a new report test checks an end-reach run's gate reason and novelty.md never say "opening", and the settings-parity forfeit text follows. Signed-off-by: Anton Volkov --- ...261029_120000_antonwolfy_novelty-cap-reach.md | 14 +++++++------- dokimi/docs/skill-novelty.md | 10 ++++++---- dokimi/dokimi/gate/novelty_gate.py | 2 +- dokimi/dokimi/novelty/atoms.py | 2 +- dokimi/dokimi/novelty/report.py | 4 ++-- dokimi/dokimi/novelty/tiers.py | 4 ++-- dokimi/dokimi/schemas/novelty.v1.json | 2 +- dokimi/tests/test_novelty_analyze.py | 2 +- dokimi/tests/test_novelty_gate.py | 5 +++-- dokimi/tests/test_novelty_report.py | 16 ++++++++++++++++ dokimi/tests/test_novelty_settings_parity.py | 2 +- 11 files changed, 41 insertions(+), 22 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index 17a301af5..b0f43c03d 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -1,9 +1,9 @@ ### Fixed - **A capped novelty run says how far the cap reached** — extraction stops at - the atom cap in document order, so a capped run measures the opening of the - input: the text that was segmented, which leaves out the bundled files a run - did not select. The cap warning of `novelty analyze` now names the span it + the atom cap in document order, so a capped run measures the input up to the span + the cap stopped at; the input is the text that was segmented, which leaves out + the bundled files a run did not select. The cap warning of `novelty analyze` now names the span it stopped at, the input's spans up to it and their share of the input's tokens, instead of calling the atoms a sample, and `novelty atoms` now prints it too; `extraction.reached` records the same in `novelty.json` and @@ -14,7 +14,7 @@ warning printed under it when that share is half or less). A cap met on the input's last span says instead that the atoms reach it and any atoms past the cap in the last reply were dropped, in the warning, the header and the - `coverage:` line. The tier forfeit (`added` is a rate over the opening of - the input, up to the atom cap) and the `harmful_atoms` gate reason (0 - conflicts among the atoms of the input's opening) no longer call the - capped atoms a sample either. + `coverage:` line. The tier forfeit (`added` is a rate over the atoms the + cap kept, not the whole input) and the `harmful_atoms` gate reason (0 + conflicts among the capped atoms) no longer call the capped atoms a sample + either, and hold however far the cap reached. diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index d548ee282..0894076c7 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -134,7 +134,7 @@ marked. Read it as a review queue: extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared unless a call fails or its reply has to be split. The artifact records the share as `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of - Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which cover only the input's opening when + Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped; otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A @@ -893,15 +893,17 @@ Each tier carries the **claims it forfeits**, and those strings are written into `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the run used and the atoms it extracted, not the tier's name: `--no-verify` adds the faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the -cap's ones (that `added` is a rate over the opening of the input, up to the atom cap, and -that `frac_carrying_nothing` is not interpretable on a capped atom set). +cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and +that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on +how far the cap reached (`--estimate-only` lists them before it is known), and neither does +the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`). `--estimate-only` cannot know the atom count yet, so it lists them: > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected > against the model matching the artifact's own wording The cap is not a sample: extraction walks the spans in document order and stops at it, so a -capped run measures the opening of the input. The input is the text that was segmented: the +capped run measures the input up to the span it stopped at. The input is the text that was segmented: the entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the artifact went into the input). The warning says how far the cap reached, as in `atom diff --git a/dokimi/dokimi/gate/novelty_gate.py b/dokimi/dokimi/gate/novelty_gate.py index 3c2cd2b8e..1bb0a0b94 100644 --- a/dokimi/dokimi/gate/novelty_gate.py +++ b/dokimi/dokimi/gate/novelty_gate.py @@ -196,7 +196,7 @@ def _conflict_reading(artifact: dict, cfg: dict[str, Any]) -> tuple[int, int, bo elif capped: reason = ( f"the atom set was capped ({spans.get('n_atoms_indexed')} indexed), so 0 conflicts " - "among the atoms of the input's opening is not a cleared check of the artifact" + "among the capped atoms is not a cleared check of the artifact" ) else: reason = None diff --git a/dokimi/dokimi/novelty/atoms.py b/dokimi/dokimi/novelty/atoms.py index 1ec62a3ff..330b5e2b7 100644 --- a/dokimi/dokimi/novelty/atoms.py +++ b/dokimi/dokimi/novelty/atoms.py @@ -422,7 +422,7 @@ def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None, """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap. Extraction walks the spans in document order and stops at the cap, so a capped set - covers the opening of the input, not a sample of it. The input is the text segmented, + covers the input up to the span it stopped at, not a sample of it. The input is the text segmented, not the artifact: a bundled file it left out counts in no span. The reach is the last span any kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole, but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span diff --git a/dokimi/dokimi/novelty/report.py b/dokimi/dokimi/novelty/report.py index 685dd839d..b1c7d5a4b 100644 --- a/dokimi/dokimi/novelty/report.py +++ b/dokimi/dokimi/novelty/report.py @@ -269,8 +269,8 @@ def coverage_reach(extraction: dict | None) -> tuple[str, str]: """The verb and tail of the headline's coverage line, from the ``extraction`` record. The coverage share counts the words that went into the input. Under a binding cap the - atoms come from the opening of those words only, so the line must not say they were - measured, and it says how far the atoms reach. + atoms come from those words up to the span the cap stopped at, so the line must not say + they were measured, and it says how far the atoms reach. """ reached = (extraction or {}).get("reached") if not reached: diff --git a/dokimi/dokimi/novelty/tiers.py b/dokimi/dokimi/novelty/tiers.py index c6857fee3..052481887 100644 --- a/dokimi/dokimi/novelty/tiers.py +++ b/dokimi/dokimi/novelty/tiers.py @@ -28,8 +28,8 @@ #: name is old, the cap takes atoms in document order: https://github.com/intel/dokimi/pull/299 FORFEITS: dict[str, list[str]] = { "sampled": [ - "`added` is a rate over the opening of the input, up to the atom cap: usable as " - "a sign, not as a figure, and not comparable with another artifact's; pass " + "`added` is a rate over the atoms the cap kept, not the whole input: usable as a " + "sign, not as a figure, and not comparable with another artifact's; pass " "--all-atoms for the full sweep", "`frac_carrying_nothing` is not interpretable on a capped atom set (0.064 read " "as 0.743 on the study's largest artifact)", diff --git a/dokimi/dokimi/schemas/novelty.v1.json b/dokimi/dokimi/schemas/novelty.v1.json index dedc0f4d7..9bd471137 100644 --- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -338,7 +338,7 @@ "last_error": {"type": "string", "description": "The last error on a span that stayed unread; errors a split recovered from are not recorded."}, "reached": { "type": ["object", "null"], - "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the opening of the input, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", + "description": "How far through the input a capped atom set reached; null when the cap did not bind. The input is the text that was segmented into spans: the entry point, plus on a bundle run the files its roles and budgets selected; bundled files it left out are in no span, so none of these figures counts them (bundle.measured_share says how much of the artifact went into the input). Extraction walks the spans in document order and stops at the cap, so a capped set covers the input up to the span it stopped at, not a sample of it. last_span is the last span any kept atom cites among the spans some call was sent (not the last batch sent: a batch the cap cut mid-reply was sent whole; nor a cited span no call was sent), spans the count of the input's spans up to and including it, of all the input's spans, and tokens_share those spans' share of the input's span tokens. Absent from artifacts written before it existed.", "properties": { "last_span": {"type": "string"}, "spans": {"type": "integer", "minimum": 1}, diff --git a/dokimi/tests/test_novelty_analyze.py b/dokimi/tests/test_novelty_analyze.py index 86fe5337f..2a456e0ca 100644 --- a/dokimi/tests/test_novelty_analyze.py +++ b/dokimi/tests/test_novelty_analyze.py @@ -584,7 +584,7 @@ def test_a_tier_writes_its_forfeits_into_the_run_it_produced(tmp_path): assert len(forfeits) == len(TIERS["triage"]["forfeits"]) - len(FORFEITS["sampled"]) assert sorted(artifact["run_config"]["tier"]["forfeits"]) == sorted( w.split("forfeits: ", 1)[1] for w in forfeits) - assert not any("rate over the opening" in w for w in forfeits) + assert not any("rate over the atoms the cap kept" in w for w in forfeits) validate_artifact("novelty", dict(artifact, schema_version="novelty.v1")) diff --git a/dokimi/tests/test_novelty_gate.py b/dokimi/tests/test_novelty_gate.py index 83d9653d0..fe41554ab 100644 --- a/dokimi/tests/test_novelty_gate.py +++ b/dokimi/tests/test_novelty_gate.py @@ -469,8 +469,9 @@ def test_a_capped_atom_set_does_not_clear_harmful_atoms(): capped = _named(evaluate_novelty(_artifact(spans=spans, conflict_coverage=full)), "harmful_atoms") assert capped.measured is False and "the atom set was capped (30 indexed)" in capped.message - # The cap keeps the input's opening, in document order: not a sample of it. - assert "0 conflicts among the atoms of the input's opening" in capped.message + # Not a sample, and not always an opening: a cap met on the last span reaches the end. + assert ("so 0 conflicts among the capped atoms is not a cleared check of the artifact;" + in capped.message) found = _named(evaluate_novelty(_artifact( spans=spans, conflict_coverage=full, categories={"harmful_conflicting": 2})), diff --git a/dokimi/tests/test_novelty_report.py b/dokimi/tests/test_novelty_report.py index 0bf30bc95..99881e1e2 100644 --- a/dokimi/tests/test_novelty_report.py +++ b/dokimi/tests/test_novelty_report.py @@ -5,6 +5,7 @@ import pytest +from dokimi.gate.novelty_gate import uncleared_conflicts from dokimi.novelty.report import _capped_at, cap_warning, coverage_reach, render_novelty_markdown @@ -92,3 +93,18 @@ def test_a_cap_that_stopped_inside_the_input_still_says_the_rates_describe_the_o assert cap_warning(1, inside) == ( "atom extraction hit the cap of 1 at span s0002 (2 of the input's 5 spans, 35% of its tokens): rates " "describe the opening of the input, not all of it; pass --all-atoms for the full sweep") + + +def test_a_cap_met_on_the_last_span_leaves_the_zero_uncleared_without_naming_an_opening(): + # Every pair compared and no harmful atom: the cap alone leaves the 0 uncleared. + end = {"last_span": "s0010", "spans": 10, "of": 10, "tokens_share": 1.0} + artifact = _artifact(spans={"capped_atom_set": True, "n_atoms_indexed": 30}, + extraction={"reached": end}) + assert uncleared_conflicts(artifact) == ( + "the atom set was capped (30 indexed), so 0 conflicts among the capped atoms is not a " + "cleared check of the artifact") + markdown = render_novelty_markdown(artifact) + assert "the atoms reach the input's last span" in markdown + assert ("> **Conflicts not cleared.** The atom set was capped (30 indexed), so 0 conflicts " + "among the capped atoms") in markdown + assert "opening" not in markdown diff --git a/dokimi/tests/test_novelty_settings_parity.py b/dokimi/tests/test_novelty_settings_parity.py index 605d9eabe..7d727e7af 100644 --- a/dokimi/tests/test_novelty_settings_parity.py +++ b/dokimi/tests/test_novelty_settings_parity.py @@ -252,7 +252,7 @@ def _kept_estimate(step: dict) -> dict: # Intended change: the cap's forfeit no longer calls `added` a sampled rate, in the tier # record, the warnings and the estimate's stderr. CAP_FORFEIT_ON_MAIN = "`added` is a sampled rate over the atom cap: usable" -CAP_FORFEIT = "`added` is a rate over the opening of the input, up to the atom cap: usable" +CAP_FORFEIT = "`added` is a rate over the atoms the cap kept, not the whole input: usable" @pytest.mark.parametrize("name", list(CASES)) From d6116ef33cca457fd91706c4e5d5d1fe2583ad26 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Fri, 30 Oct 2026 12:43:20 +0100 Subject: [PATCH 16/16] docs(changelog): say which share triggers the coverage warning The fragment said the warning under the coverage: line prints "when that share is half or less", which reads as the cap's token reach. It is the share of the artifact's readable words that went into the input, as docs/skill-novelty.md says. Review comment 4423182482. Signed-off-by: Anton Volkov --- .../20261029_120000_antonwolfy_novelty-cap-reach.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md index b0f43c03d..ac8e51956 100644 --- a/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md @@ -11,9 +11,9 @@ states it; and on a capped run the headline's `coverage:` line says the readable words went into the input and the capped atoms come from the first N% of the input's tokens, instead of saying they were measured (as does the - warning printed under it when that share is half or less). A cap met on the - input's last span says instead that the atoms reach it and any atoms past - the cap in the last reply were dropped, in the warning, the header and the + warning printed under it when half or less of the artifact's readable words + went into the input). A cap met on the input's last span says instead that + the atoms reach it and any atoms past the cap in the last reply were dropped, in the warning, the header and the `coverage:` line. The tier forfeit (`added` is a rate over the atoms the cap kept, not the whole input) and the `harmful_atoms` gate reason (0 conflicts among the capped atoms) no longer call the capped atoms a sample