fix(novelty): a capped run says how far the cap reached by antonwolfy · Pull Request #299 · intel/dokimi

antonwolfy · 2026-10-29T18:23:22Z

Summary

Found while evaluating IntelPython/dpnp#3099 (AGENTS.md guidance) with dokimi: a novelty
triage (novelty analyze --tier triage) of the PR's root AGENTS.md and dpnp/tests/AGENTS.md
measured only the opening of each file, and nothing the run printed or recorded said so.

Extraction walks the spans in document order and stops at the atom cap. On the root file the
first 25-span batch hit the provider's 4096-token output ceiling and was split; its first half
(12 spans) gave the 25 atoms. So every atom came from spans s0002-s0012 of 61, the first 23%
of the tokens; on the tests file, from s0003-s0013 of 32, the first 36%. The
changelog format, the CI table, ## Reviewing and the directory guides were never measured.
On main the same run said:

where	what `main` says
warning	`atom extraction hit the cap of 25; rates describe the sampled atoms, not the whole artifact` (reads as a representative sample)
headline	`coverage: 100% of the artifact's readable words were measured (entry point only)` (the bundle figure: which files' words went into the input)
`novelty.json`	`spans.capped_atom_set: true`, `n_atoms_indexed: 25`; nothing on which spans the cap reached

A reader saw "100% measured" next to a cap that covered 23% of the file.

This PR changes nothing about what a run measures. Extraction still stops at the cap in
document order. The run now records how far it got and says so.

Related: #293 proposes spreading the cap across the artifact instead of its opening; this PR only says where it stopped.

Changes

novelty/atoms.py: extract_atoms fills ExtractionReport.reached when the cap binds:
last_span (the last span any kept atom cites, among the spans some call was sent), spans
(spans up to and including it), of (all spans) and tokens_share (those spans' share of the
span tokens, floored to 4 places). The spans are the input's, the text that was segmented: the
entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file
the run left out is in none of them. It is taken from the kept atoms, not from the batches sent,
because a batch the cap cut mid-reply was sent whole but its later spans gave no atom. A cited
span id no call was sent does not count, so an atom citing a span its call never saw does not
move the reach (an atom set citing nothing sent reaches the last span sent). It is null when
the cap did not bind (a cap the last atom meets exactly binds, and its reach is the whole
input). The extraction block of novelty.json and of atoms.json (novelty atoms)
carries it.
schemas/novelty.v1.json: extraction.reached added as an optional property, with a
description, and not in required; inside it, all four keys are required, so a partial
reach such as {"spans": 3} does not validate. atoms.json has no schema, and its loader does not
read the key, so probe reads files written before this change as it did.
novelty/analyze.py: the cap warning names the reach, e.g. atom extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens): rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep. A cap met on the last span
says instead ...: any atoms past the cap in its last reply were dropped; pass --all-atoms for the full sweep.
commands/novelty.py (cmd_novelty_atoms): novelty atoms prints the same cap warning on
stderr. probe and shorten, which read its atoms.json, do not repeat it.
commands/novelty.py (_print_coverage): the coverage: line still gives the bundle
figure. On a capped run it reads coverage: 100% of the artifact's readable words went into the input (entry point only); the capped atoms come from the first 23% of the input's tokens (the root file above).
On the last span the tail is the capped atoms reach the input's last span, and the cap dropped any atoms past it.
The unknown-share line gets the same tail, and the warning under the line when the share is
half or less says "went into the input" too. A run the cap did not stop keeps its wording.
The percent formatter moves to novelty/report.py (floored_percent) so both figures
floor the same way and a cap short of the end never prints as 100%.
novelty/report.py: the novelty.md header adds The atom cap stopped extraction at span … : the atoms come from the opening of the input, not all of it. (on the last span: the
atoms reach the input's last span, and the cap dropped any atoms past it in the last
reply). Each reach text names the input, not the artifact, because a bundle run's input
leaves out the files it did not select; the coverage: line says how much of the artifact
went into it. The warning above also reaches the report through its warnings list.
novelty/tiers.py, gate/novelty_gate.py: the cap's tier forfeit reads "added is a rate
over the atoms the cap kept, not the whole input: …" in place of "a sampled rate over the
atom cap", and the harmful_atoms reason "0 conflicts among the capped atoms" in place of
"among the sampled atoms". Neither names an opening: both hold however far the cap
reached, and --estimate-only lists the forfeit before the reach is known. The FORFEITS key "sampled" and the
atoms.sampling values are unchanged.
Docs: docs/skill-novelty.md gets reached in the extraction field table and a paragraph
in the tier/forfeit section, with the last-span case; its conflict-coverage note and forfeit
paragraph no longer call a capped set a sample, and say a capped run measures the input up
to the span the cap stopped at. docs/reviewing.md's triage note says the warning prints how
far the cap reached through the input.
tests/test_novelty_settings_parity.py: the golden recorded on main has the old cap
warning and forfeit. The test now maps them to the new ones, and expects the cap warning on
novelty atoms --max-atoms 1, as it already does for its other intended changes; its headline
check accepts the capped wording when the record has a reach.
scripts/growth_baseline.json: commands/novelty.py drops from 1192 to 1190 lines.
Changelog fragment (Fixed).

Risk

The cap warning text changes for every capped run, and so does the coverage: line on a
capped run (went into the input … in place of were measured, in the floor warning too).
The cap's tier forfeit changes wherever it prints (--estimate-only with a cap, and runs the
cap stopped), and so does the harmful_atoms reason on a capped set; novelty atoms gains a
stderr line when the cap binds. Anything that greps for the old wording will no longer match.
novelty.json and atoms.json gain one key in extraction. Artifacts written before this
change still validate and load. A run the cap did not stop prints the same as before, apart
from reached: null in the record. Numbers are unchanged.

Testing

On a Linux review host, at 50c2711 (on main 83f18fb):

make ci-local: 28 passed, 2 failed, 7 skipped (external-catalog-smoke: needs docker; test-latest, docs.yml:deploy and
pip-audit.yml:report: job conditions false on a PR; cli-surface / Package the result:
an expression the local runner does not evaluate; conda-forge-deps: no micromamba;
semgrep: not on PATH). 10655 tests passed, 4 skipped; changed-line coverage 100% (49
lines). The two failures are gitleaks: its download returned HTTP 504 (Install gitleaks), so Scan commits had nothing to run; a gitleaks already on the host scanned
the branch's commits instead: no leaks found.
The comment-block check, the CLI snapshot check, mypy, schema_compat and pip-audit pass.
New tests:
- test_novelty_atoms.py: 60 spans, 3 atoms each, 10 per batch, cap 25. The cap cuts the
  first reply inside s0009's atoms, and reached is s0009, 9 of 60, 0.15. Without a cap,
  under it, or exactly at it on the last span, reached is null / the whole artifact.
  The record survives atoms.json.
- test_novelty_analyze.py: an analyze with max_atoms=1 gives the reach, the warning
  and the novelty.md header, and validates. An uncapped run has reached: null and no cap
  warning.
- test_novelty_cli.py: the headline on a capped run, its unknown-share variant, and the
  uncapped wording.
On main's code, with these test files copied in, all seven fail: the five above,
plus the parity test's two capped cases (run/max-atoms-1, run/triage-max-atoms-1).
Mutants, each run on a copy of fa453d8, each failing at least one new test: reach
never recorded; reach taken from the first cited span (min for max); reach taken
from the end of the batch sent; a cap met exactly treated as not binding (<=);
token share without the last span; reached left out of as_dict; main's old cap
warning restored; the token share dropped from the reach text; the headline given no
extraction record; the capped headline saying "were measured"; the unknown-share
line without the reach; the percent rounded instead of floored; the novelty.md
header sentence inverted. (The min for max mutant passes the one-atom analyze
test, where both are the same span, and fails the other two.)
Review rounds:
- 92d41c7 and f5a21b7 floor the stored share (pinned by
  test_a_cap_reach_share_is_floored_not_rounded: 29,999 of 30,000 tokens prints "99%")
  and word the cap met on the last span.
- 27dac86..fa5347a answer the rest. New tests: test_a_span_id_no_call_was_shown_does_not_move_the_reach
  (60 spans, 10 per batch, cap 25, one atom citing s0055: reach s0010, 10 of 60);
  test_a_capped_run_under_the_floor_warns_in_the_coverage_line_s_words; {"spans": 3}
  rejected by the schema; the new gate reason; the mid-artifact novelty.md sentence; the
  parity test's mapped forfeit and the warning from novelty atoms.
- On f5a21b7 these fail: the reach, floor-warning, schema and gate tests and 16 parity cases.
- Nine mutants on fa5347a each fail their test: the sent-span filter dropped, the
  fallback reach set to the first span, only each batch's first span recorded as sent,
  main's forfeit restored, main's gate reason restored, no warning from novelty atoms,
  "were measured" in the floor warning, required dropped from reached, and the
  opening dropped from the mid-artifact header.
- b3e1032 fixes the mypy error 92d41c7 brought in (analyze.py passed max_atoms,
  int | None, to cap_warning(cap: int, …)): it narrows max_atoms first, which changes no
  output, since reached is set only under a cap.
- eb34b3f merges main. 689c109 cuts the #: comment above FORFEITS in
  novelty/tiers.py from six lines to two for the comment-block check from ci: fail comment blocks over two lines that a PR adds or edits #298, which failed
  there at eb34b3f ("6 comment lines; keep it to 2 ...") and passes at 689c109. The CLI
  snapshot from test(cli): flag registry, ratchet contract test, CLI snapshot (#94 F1a) #254 is unchanged: tests/test_cli_flags.py passes and a fresh dump matches it.
- 3e576e0 words every reach text as the input's, not the artifact's, in both branches. The input is the entry point plus the files a bundle run selected, and the reach counts only its spans. Examples: "(12 of the input's 61 spans, 23% of its tokens)", "rates describe the opening of the input, not all of it", "come from the first 23% of the input's tokens", "reach the input's last span". extraction.reached keeps its keys and values; its schema description, the docs and the fragment follow. New test: test_a_capped_bundle_run_reaches_through_its_input_not_the_artifact, a capped --include-resources run with a reference over max_file_words, mid-input (cap 2: 2 of 8 spans, 25%) and on the last span (cap 8). It pins the novelty.md header and the coverage: line (6% of the artifact's readable words went into the input; ...). Both cases fail at 689c109, with four existing tests and three parity cases that quote the old words. Five mutants on 3e576e0 each fail it: the header's mid-input and last-span sentences back to "the artifact", the coverage: tail's mid-input and last-span wording back to "its", and the reach parenthetical back to "(2 of 8 spans, 25% of the tokens)".
- 72bab67 makes the cap's tier forfeit ("added is a rate over the opening of the input, up to the atom cap: …") and the harmful_atoms reason ("0 conflicts among the atoms of the input's opening") name the input too, with docs/skill-novelty.md and the fragment. The gate test and the parity mapping quote the new words. At 3e576e0 they fail (the gate test and 16 parity cases), and two mutants on 72bab67 that restore each old text each fail them.
- 50c2711 stops the harmful_atoms reason and the cap's tier forfeit from naming an opening, which a cap met on the input's last span does not leave: "0 conflicts among the capped atoms" and "added is a rate over the atoms the cap kept, not the whole input: …". docs/skill-novelty.md, the fragment, the extraction.reached schema description and two docstrings now say a capped run covers the input up to the span the cap stopped at; the mid-input warning and header keep "the opening". New test: test_a_cap_met_on_the_last_span_leaves_the_zero_uncleared_without_naming_an_opening, an end-reach reading (s0010, 10 of 10 spans) with every pair compared and no harmful atom: neither the gate reason nor novelty.md says "opening". At 72bab67 it fails, along with the gate test and 16 parity cases, and two mutants on 50c2711 that restore each old text each fail them.
Merged together on main f0f95a2 with the other PRs from this evaluation (fix(repo): scan says why it is incomplete #295 at 4fa7783, feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296 at cad52dd, feat(repo): replay grades record each credited hit; repo report --hits lists them #297 at 0a7539a, fix(novelty): a capped run says how far the cap reached #299 at fa453d8), on a workstation: the full suite has 10482 passed and 57 skipped, and the same 5 failures and 1 collection error as main alone there (10456 passed). All six come from that machine's setup: no docker, no hypothesis, and two tests that depend on its git and shell settings. That run has not been repeated since; a run at this head is still to come.
The dpnp figures: segmenting the two files at d8fb283 with novelty.spans.segment gives
the run's 61 spans / 1,736 tokens and 32 spans / 1,129 tokens; s0012 ends 23.1% of the root
file's tokens, s0013 36.8% of the tests file's (floored to 23% and 36%, as printed). The run
itself was not repeated: it needs a real model.

Revert

git revert the commits. Artifacts written meanwhile keep extraction.reached, which the
schema allows, so they still load.

Checklist

Tests added/updated and pytest passes
ruff, mypy, and schema checks pass
Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
Self-reviewed per "Before opening a PR" in AGENTS.md
No ad-hoc run artifacts committed
New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

Atom extraction walks the spans in document order and stops at the cap, so a capped run measures the opening of the artifact. On main the run did not say how far that was. Found with `novelty analyze --tier triage` on two AGENTS.md files: the 25 atoms came from spans s0002-s0012 of 61 (about a quarter of the tokens) and s0003-s0013 of 32, yet the cap warning called them "the sampled atoms", novelty.json had only capped_atom_set and n_atoms_indexed, and the headline printed "coverage: 100% of the artifact's readable words were measured". This settles both points: the reach is recorded, and the warning and the headline say it. What a run measures is unchanged. - extract_atoms records `reached` in the extraction report when the cap binds: the last span any kept atom cites, the spans up to and including it, all spans, and those spans' share of the span tokens. It comes from the kept atoms, not the batches sent, since a batch the cap cut mid-reply was sent whole. It is null when the cap did not bind, and is written to both novelty.json and atoms.json. - novelty.v1.json's extraction block gains `reached` as an optional property. atoms.json has no schema and its loader does not read it. - The cap warning in analyze names the span, the span count and the token share, says the rates describe the opening of the artifact, and points at --all-atoms. - The headline's coverage line stays the bundle figure; on a capped run it says the words went into the input and the capped atoms come from the first N% of its tokens. The percent formatter moves to novelty.report so both figures floor the same way. - The novelty.md header says where the cap stopped extraction. - docs: the extraction field table, the tier/forfeit paragraph and the triage note in reviewing.md. - The settings parity test maps main's cap warning to the new one, and its headline check accepts the capped wording on a capped run. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

codecov-commenter · 2026-10-29T18:30:30Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.61%. Comparing base (83f18fb) to head (d6116ef).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #299      +/-   ##
==========================================
+ Coverage   93.52%   93.61%   +0.08%     
==========================================
  Files         274      275       +1     
  Lines       35625    35741     +116     
  Branches     6052     6058       +6     
==========================================
+ Hits        33320    33459     +139     
+ Misses       1489     1468      -21     
+ Partials      816      814       -2

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty.py	`95.43% <100.00%> (+0.03%)`	⬆️
dokimi/dokimi/gate/novelty_gate.py	`98.98% <ø> (ø)`
dokimi/dokimi/novelty/analyze.py	`94.82% <100.00%> (ø)`
dokimi/dokimi/novelty/atoms.py	`98.20% <100.00%> (+0.08%)`	⬆️
dokimi/dokimi/novelty/report.py	`98.67% <100.00%> (+5.05%)`	⬆️
dokimi/dokimi/novelty/tiers.py	`100.00% <ø> (ø)`

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

vchamarthi · 2026-10-29T18:36:01Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s).

Review details

Commit reviewed: fa453d8 against merge base e966422
Coverage: 12 files in 4 groups
Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
Usage: 302k tokens (210k from cache), 40s
Tool calls: 23 (12 code_search, 10 file_read, 1 file_read_diff)
Docs and changelog for novelty cap reach: 3 files
Novelty core: analysis, atoms, and CLI command: 3 files
Novelty report and JSON schema: 2 files
Novelty tests: 4 files

vlad-perevezentsev

Review at fa453d8. The change is right in the main case: a cap that binds mid-artifact now names the span it stopped at, the spans and the token share, in the warning, novelty.json, atoms.json, the novelty.md header and the headline. The edge cases (a cap met on the last span, a made-up span id, rounding to 100%) and a few places that still call the cap a sample are in the line comments.

What I ran, at fa453d8:

make ci-local --offline: 0 failed, 17 passed, 13 skipped (network and Docker jobs).
A fake-extractor script over 60 spans: the cap met on the last span gives "60 of 60 spans, 100% of the tokens ... the opening of the artifact, not the whole"; one atom citing an unsent span moves the reach from 10 sent spans to 55; a lost first batch gives a reach of 19 spans with the unread warning beside it, which reads right.

Checked, no issues

Mid-batch cut: the reach is the last span a kept atom cites, not the last batch sent, so a reply the cap cut inside span 9 reports 9 spans, as the docs say.
Schema: reached is a new optional key of extraction, so the in-place edit only widens novelty.v1; schema_compat passes.
Scores: no rate, default or prompt changes; atoms.sampling stays cap-per-artifact:N, so old runs stay comparable.
Tests: the new tests read report.reached and the new warning text, which do not exist on the base, so they fail there.
Docs and changelog: the extraction table, the tier section of skill-novelty.md and reviewing.md match the code; the fragment is under Fixed and says what a user now sees.

exai-bot · 2026-10-30T01:48:11Z

✅ OpenCodeReview: Review complete: 0 finding(s) across 18 selected item(s).

Review details

Commit reviewed: d6116ef against merge base 83f18fb
Coverage: 18 files in 4 groups
Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
Usage: 244k tokens (110k from cache), 25s
Tool calls: 30 (27 file_read, 2 code_search, 1 file_read_diff)
Novelty cap/reach documentation and changelog: 3 files
Novelty analysis, atoms, tiers, and settings parity: 7 files
Novelty reporting, schema, CLI, and gate integration: 7 files
Growth baseline update: 1 file

…s so _cap_reach rounded tokens_share to 4 places before floored_percent floored it, so a cap one span short of the end (29,999 of 30,000 tokens) printed "100%". The share is now floored in integers, and a reach above zero stays "<1%". When the cap stopped on the last span, the warning, the report header and the coverage tail still said the rates describe "the opening of the artifact". The reach is then the whole artifact; what the cap cut is the atoms past it in the last reply. cap_warning now words the warning per case, and the header and coverage tail follow reach_is_end. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…e dropped Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Copilot

🟢 Approval recommended

No active review issues remain for the changes clarifying capped novelty evaluation coverage.

No open issues 🎉

What changed in this PR

0 new findings; nothing to fix.

This PR makes capped novelty evaluations report how far their retained atoms reach, without changing extraction or scoring. It clarifies coverage reporting; spreading atoms across the artifact remains separate work in #293.

Changes:

Records the last cited span, span count, and token share in extraction.reached.
Updates warnings, CLI coverage, and Markdown reports, distinguishing partial reach from reaching the last span.
Adds schema support, regression tests, documentation, and a changelog entry.

File	Description
`dokimi/tests/test_novelty_settings_parity.py`	Updates expected cap warnings and coverage wording.
`dokimi/tests/test_novelty_report.py`	Tests partial-reach and last-span messages.
`dokimi/tests/test_novelty_cli.py`	Tests capped, uncapped, and unknown-share headlines.
`dokimi/tests/test_novelty_atoms.py`	Tests reach calculation, persistence, and percentage boundaries.
`dokimi/tests/test_novelty_analyze.py`	Checks recorded reach, warnings, reports, and schema validation.
`dokimi/dokimi/schemas/novelty.v1.json`	Adds optional, nullable extraction reach metadata.
`dokimi/dokimi/novelty/report.py`	Centralizes percentage formatting and reach messages.
`dokimi/dokimi/novelty/atoms.py`	Calculates and serializes reach from retained atoms.
`dokimi/dokimi/novelty/analyze.py`	Uses recorded reach in cap warnings.
`dokimi/dokimi/commands/novelty.py`	Distinguishes input coverage from capped atom reach.
`dokimi/docs/skill-novelty.md`	Documents cap reach and its recorded fields.
`dokimi/docs/reviewing.md`	Explains reach warnings for triage runs.
`dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md`	Announces the reporting correction.

_{Review effort: Balanced}

napetrov · 2026-10-30T05:39:04Z

Two of the open threads are handled in 92d41c75 and f5a21b70 on this PR's branch (pushed).

novelty/report.py:239 (blocking, "100%" for a cap one span short): the share is now floored in integers in _cap_reach (atoms.py), before floored_percent, instead of rounded to 4 places first. A reach above zero stays "<1%". Test: test_a_cap_reach_share_is_floored_not_rounded (29,999 of 30,000 tokens prints "99%"; red on the previous head).
novelty/analyze.py:190 (exact-cap wording, first thread): when the cap stops on the last span (spans == of), the warning, the novelty.md header and the coverage tail no longer say the rates describe "the opening" of the artifact; they say the atoms past the cap in the last reply were dropped. The warning text is built by report.cap_warning. Tests: test_a_cap_that_stopped_on_the_last_span_does_not_say_the_atoms_come_from_the_opening and test_a_cap_that_stopped_inside_the_artifact_still_says_the_rates_describe_the_opening (tests/test_novelty_report.py).

Not in this push: the second analyze.py:190 thread (the FORFEITS["sampled"] and gate wording still say "sampled"), atoms.py:431 (a failed batch counted in the reach), atoms.py:414 (novelty atoms does not print the reach), commands/novelty.py:1192 (the "only X% measured" floor warning) and the novelty.v1.json required nit. Say if you want any of them in this PR.

Answers review comment 4418984223. The reach took the largest span id any kept atom cites, and a cited id is checked only against the whole document, so one atom citing a span no call was shown moved the reach to it: 60 spans, 10 per batch, cap 25, the cap binding in the first batch and one atom also citing s0055 recorded s0055, 55 of 60 spans, 91% of the tokens. extract_atoms now keeps the ids of the spans it sent, and _cap_reach takes the last cited span among those (the last span sent when no kept atom cites one). The same case now reaches s0010, 10 of 60. The docstring, the extraction table in skill-novelty.md and the schema description of extraction.reached say so. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

…mple Answers review comment 4418984232. The cap warning no longer calls the capped atoms a sample, but the tier forfeit printed beside it still said "`added` is a sampled rate over the atom cap", and the harmful_atoms gate reason said "0 conflicts among the sampled atoms". The forfeit now reads "`added` is a rate over the opening of the artifact, up to the atom cap: ...", and the gate reason "0 conflicts among the atoms of the artifact's opening". The FORFEITS key "sampled" and the atoms.sampling values stay: code and stored runs name them. skill-novelty.md (conflict coverage, the tier forfeits) and the schema description of atoms.sampling drop the same claim. The settings-parity test maps main's forfeit to the new one, as it does for the cap warning; the tiers test that found the shortening forfeit by "opening of the artifact" now matches its own wording. Changelog fragment updated. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comment 4418984237. `novelty atoms` saved extraction.reached in atoms.json but printed only the unread-span warnings and the span summary, and probe and shorten do not repeat it, so in the atoms -> probe flow the reach was visible only in the JSON. cmd_novelty_atoms now prints report.cap_warning on stderr when the cap binds, the same text analyze adds to its warnings. The settings-parity test expects it on atoms/max-atoms-1, where main printed nothing. Folding the atoms import onto two lines keeps commands/novelty.py under its growth baseline, which drops to 1190. skill-novelty.md and the changelog fragment say so. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comment 4418984241. On a capped run the coverage line says the readable words "went into the input", but the floor warning printed under it still said "only X% of the artifact's readable words were measured". A capped entry-point run with 10 of 40 readable words printed both. The warning now takes the same verb as the coverage line, so a capped run prints "WARNING: only 25% of the artifact's readable words went into the input, so every rate here describes that part, not the artifact"; an uncapped run keeps "were measured". A test pins it with a capped run under the floor. skill-novelty.md and the changelog fragment say so. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comments 4418984247 and 4420750589. The reached object had no required list, so {} or {"spans": 3} validated, while describe_reach, reach_is_end and coverage_reach read every key and raise KeyError on a partial one. reached now requires last_span, spans, of and tokens_share. The key itself stays optional and nullable, and it is new in this PR, so no stored artifact carries a partial one; schema_compat against the base reports the narrowing as a warning on a property no fixture writes. The capped analyze test pins that {"spans": 3} is rejected. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comment 4420750586. skill-novelty.md and the changelog fragment gave only the "opening of the artifact" wording, while a cap met on the last span (report.reach_is_end) words the warning, the novelty.md header and the coverage: line differently. skill-novelty.md now quotes that case: the warning ending "any atoms past the cap in its last reply were dropped", the header saying the atoms reach the artifact's last span and the cap dropped any atoms past it in the last reply, and the coverage: tail. The changelog fragment names the case too. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comment 4420750597. test_novelty_report.py imported cap_warning inside two test bodies though it already imports from dokimi.novelty.report at the top; it now comes in with the other names. The test of a cap that stopped inside the artifact now also asserts the novelty.md header sentence _capped_at gives it ("the atoms come from the opening of the artifact, not the whole"), which only test_novelty_analyze covered before. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

analyze passed config.max_atoms (int | None) to report.cap_warning, which takes an int, so mypy failed on the branch. extraction.reached is set only when the cap binds, so the guard changes no output. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy · 2026-10-30T10:19:25Z

Review round in 27dac86..b3e1032, on top of 92d41c7 and f5a21b7, which fixed the rounding to "100%" and the wording of a cap met on the last span.

The reach counts only spans a call was sent, so a made-up span id no longer moves it.
The tier forfeit and the harmful_atoms gate reason no longer call the capped atoms a sample; the stored key "sampled" and atoms.sampling are unchanged.
novelty atoms prints the cap warning, and the floor warning on a capped run says "went into the input".
extraction.reached requires all four of its keys.
The docs and the fragment describe the last-span case.
b3e1032 fixes the mypy error from 92d41c7 (max_atoms is narrowed before cap_warning; no output changes).

Every thread is answered and resolved; the Testing section has the mutants and make ci-local.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

The static job's comment-block check from intel#298 (scripts/check_comment_blocks.py) failed this branch once main was merged in: the six-line #: block above FORFEITS in novelty/tiers.py has a line this PR edits, and a block of three or more lines fails once a PR touches it. The block now says in two lines what the code does not: forfeits are keyed by the setting that is off, and "sampled" is a reached atom cap, an old name kept although the cap takes atoms in document order, with a link to intel#299 for the backstory. How forfeits() derives them from the settings and the atom count is left to the function. No behaviour changes. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Answers review comment 4422880472. extraction.reached counts the spans of the text that was segmented: the entry point, plus on a bundle run the files its roles and budgets selected. A bundled file left out is in no span, yet the cap warning, the novelty.md header and the headline's coverage: line said the atoms covered the opening of the artifact or reached the artifact's last span. The reach now names the input in both branches: "(12 of the input's 61 spans, 23% of its tokens)", "rates describe the opening of the input, not all of it", "come from the first 23% of the input's tokens" and "reach the input's last span". The coverage: line keeps saying how much of the artifact went into the input. The keys and values of extraction.reached are unchanged; its schema description, the docstrings, docs/skill-novelty.md, docs/reviewing.md and the changelog fragment now say the input. A capped bundle run with a reference over the file budget pins the header and the coverage: line, mid-input and on the last span; the tests and settings-parity mapping quoting the old words follow. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Follows up review comment 4422880472, answered in 3e576e0 for the reach texts (the cap warning, the novelty.md header, the coverage: line). The cap's tier forfeit and the harmful_atoms gate reason, written earlier in this PR, still placed the capped atoms in the artifact's opening, though on a bundle run the input that was segmented leaves out the files its roles and budgets did not select. The forfeit now reads "`added` is a rate over the opening of the input, up to the atom cap: ...", and the gate reason "0 conflicts among the atoms of the input's opening is not a cleared check of the artifact". skill-novelty.md (pairs_total under a capped atom set, the tier forfeits) and the changelog fragment say the same. The settings-parity mapping (CAP_FORFEIT) and the gate test quote the new words. The shortening forfeit, older than this PR, is unchanged. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

Review comment 4423042474: a cap met on the input's last span reaches the end of the input, yet the harmful_atoms gate reason still said the 0 conflicts were among "the atoms of the input's opening", and novelty.md quoted it under a header saying the atoms reach the last span. The gate reason now says "0 conflicts among the capped atoms", and the cap's tier forfeit, which --estimate-only lists before the reach is known, says `added` is a rate over the atoms the cap kept, not the whole input. The general statements that a capped run covers "the opening of the input" (skill-novelty.md, the changelog fragment, the extraction.reached schema description and two docstrings) now say it covers the input up to the span the cap stopped at; the mid-input warning and header keep "the opening". Tests: the gate assertion takes the new reason, a new report test checks an end-reach run's gate reason and novelty.md never say "opening", and the settings-parity forfeit text follows. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

vlad-perevezentsev

Review at 72bab67. Every point from my first round is fixed, and the reach texts now name the input rather than the artifact. I agree with @exai-bot's open point on the gate reason (it still says "the input's opening" when the atoms reach the input's last span); the tier forfeit has the same gap, see my reply in that thread.

What I ran, at 72bab67:

make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker jobs).
My first-round fake-extractor script at 689c109: 60 of 60 spans reads "any atoms past the cap in its last reply were dropped"; an atom from the first batch citing s0055 leaves the reach at s0009 (9 of 60 spans, 15%).

Checked, no issues

Input, not artifact: describe_reach, cap_warning, the novelty.md header and the coverage: tail all say "the input's" spans and tokens, which is what segment(artifact_bundle.text) counts; extraction.reached keeps its keys and values.
Reach edge cases: an atom set that cites no sent span falls back to the last span sent, and a reach above zero still prints as <1%, not 0%.
Schema: reached is new in this PR, so requiring its keys rejects no stored artifact; schema_compat passes.
Docs and changelog: the fragment, skill-novelty.md and reviewing.md follow the new wording.

The fragment said the warning under the coverage: line prints "when that share is half or less", which reads as the cap's token reach. It is the share of the artifact's readable words that went into the input, as docs/skill-novelty.md says. Review comment 4423182482. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy requested review from jharlow-intel, napetrov, vchamarthi and xaleryb as code owners October 29, 2026 18:23

This was referenced Oct 29, 2026

fix(repo): scan says why it is incomplete #295

Open

feat(repo): compare records the start context, filters tasks, and flags inconclusive rows #296

Open

feat(repo): replay grades record each credited hit; repo report --hits lists them #297

Open

antonwolfy self-assigned this Oct 29, 2026

vlad-perevezentsev reviewed Oct 29, 2026

View reviewed changes

napetrov added 2 commits October 29, 2026 19:34

fix(novelty): a cap that met the last span says any atoms past it wer…

f5a21b7

…e dropped Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov requested a review from a team as a code owner October 30, 2026 05:29

Copilot AI balanced review requested due to automatic review settings October 30, 2026 05:29

Copilot started reviewing on behalf of antonwolfy October 30, 2026 05:33

View session

exai-bot reviewed Oct 30, 2026

View reviewed changes

Comment thread dokimi/docs/skill-novelty.md Outdated

Comment thread dokimi/dokimi/schemas/novelty.v1.json Outdated

Comment thread dokimi/tests/test_novelty_report.py Outdated

Copilot AI reviewed Oct 30, 2026

View reviewed changes

antonwolfy added 8 commits October 30, 2026 10:00

antonwolfy added 2 commits October 30, 2026 11:39

Merge main into fix/novelty-cap-reach

eb34b3f

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

exai-bot reviewed Oct 30, 2026

View reviewed changes

Comment thread dokimi/dokimi/novelty/report.py Outdated

antonwolfy added 2 commits October 30, 2026 11:06

exai-bot reviewed Oct 30, 2026

View reviewed changes

Comment thread dokimi/dokimi/gate/novelty_gate.py Outdated

vlad-perevezentsev reviewed Oct 30, 2026

View reviewed changes

exai-bot reviewed Oct 30, 2026

View reviewed changes

Comment thread dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md Outdated

Conversation

antonwolfy commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Risk

Testing

Revert

Checklist

Uh oh!

codecov-commenter commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vchamarthi commented Oct 29, 2026

Uh oh!

vlad-perevezentsev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

exai-bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

napetrov commented Oct 30, 2026

Uh oh!

antonwolfy commented Oct 30, 2026

Uh oh!

Uh oh!

Uh oh!

vlad-perevezentsev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

antonwolfy commented Oct 29, 2026 •

edited

Loading

codecov-commenter commented Oct 29, 2026 •

edited

Loading

exai-bot commented Oct 30, 2026 •

edited

Loading