fix(novelty): a capped run says how far the cap reached

dokimi/changelog.d/20261029_120000_antonwolfy_novelty-cap-reach.md

@@ -0,0 +1,20 @@
+### Fixed
+
+- **A capped novelty run says how far the cap reached** — extraction stops at
+  the atom cap in document order, so a capped run measures the input up to the span
+  the cap stopped at; the input is the text that was segmented, which leaves out
+  the bundled files a run did not select. The cap warning of `novelty analyze` now names the span it
+  stopped at, the input's spans up to it and their share of the input's
+  tokens, instead of calling the atoms a sample, and `novelty atoms` now
+  prints it too; `extraction.reached` records the same in `novelty.json` and
+  `atoms.json` (`null` when the cap did not bind); the `novelty.md` header
+  states it; and on a capped run the headline's `coverage:` line says the
+  readable words went into the input and the capped atoms come from the first
+  N% of the input's tokens, instead of saying they were measured (as does the
+  warning printed under it when half or less of the artifact's readable words
+  went into the input). A cap met on the input's last span says instead that
+  the atoms reach it and any atoms past the cap in the last reply were dropped, in the warning, the header and the
+  `coverage:` line. The tier forfeit (`added` is a rate over the atoms the
+  cap kept, not the whole input) and the `harmful_atoms` gate reason (0
+  conflicts among the capped atoms) no longer call the capped atoms a sample
+  either, and hold however far the cap reached.

dokimi/docs/reviewing.md

@@ -83,7 +83,9 @@ self-judge; `arms run` does not check, so name
   the same skill was 1,900 calls, about $14.50.
 - **At triage** with `--include-resources`, the 25-atom cap takes atoms in
   document order, and the entry point comes first: the flag reaches
-  `references/` only if `SKILL.md` has fewer than 25 atoms. For such skills,
+  `references/` only if `SKILL.md` has fewer than 25 atoms, and the run's cap
+  warning prints how far it reached through the input, the entry point and the files the budgets selected (span,
+  spans and token share). For such skills,
   use `--tier standard` (60 atoms), or add `--all-atoms` to the tier command for
   a full sweep; run `--estimate-only` first, it costs more.
 - Step 5 needs evaluation content: a golden question set for 5a, executable

dokimi/docs/skill-novelty.md

@@ -134,7 +134,7 @@ marked. Read it as a review queue:
   extracted atom set (at most `max_atoms`, 120 by default) in one call, so every pair is compared
   unless a call fails or its reply has to be split. The artifact records the share as
   `conflict_coverage` (`pairs_compared`, `pairs_total`) and warns `the conflict check compared X of
-  Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which is a sample when
+  Y atom pairs` when it is short. `pairs_total` is over the extracted atoms, which are only the ones the cap kept when
   `spans.capped_atom_set` is true. The `harmful_atoms` gate check reads a count of 0 as cleared only
   when the share reaches `min_conflict_coverage` (1.0 by default) and the atom set was not capped;
   otherwise it is `measured: false`, as it is when `conflict_coverage` is present but malformed. A
@@ -893,11 +893,36 @@ Each tier carries the **claims it forfeits**, and those strings are written into
 `run_config.tier` *and* appended to the artifact's warnings. They follow the settings the
 run used and the atoms it extracted, not the tier's name: `--no-verify` adds the
 faithfulness forfeit, and `--all-atoms` or a skill with fewer atoms than the cap drops the
-sampled-rate ones. `--estimate-only` cannot know the atom count yet, so it lists them:
+cap's ones (that `added` is a rate over the atoms the cap kept, not the whole input, and
+that `frac_carrying_nothing` is not interpretable on a capped atom set). They do not depend on
+how far the cap reached (`--estimate-only` lists them before it is known), and neither does
+the `harmful_atoms` gate reason (`0 conflicts among the capped atoms`).
+`--estimate-only` cannot know the atom count yet, so it lists them:
 
 > tier `triage` forfeits: no paraphrase probe, so a `redundant` verdict is unprotected
 > against the model matching the artifact's own wording
 
+The cap is not a sample: extraction walks the spans in document order and stops at it, so a
+capped run measures the input up to the span it stopped at. The input is the text that was segmented: the
+entry point, plus on a bundle run the files its roles and budgets selected, so a bundled file
+left out is in none of the reach's spans or tokens (the `coverage:` line says how much of the
+artifact went into the input). The warning says how far the cap reached, as in `atom
+extraction hit the cap of 25 at span s0012 (12 of the input's 61 spans, 23% of its tokens):
+rates describe the opening of the input, not all of it; pass --all-atoms for the full sweep`,
+`extraction.reached` records it, and the headline's `coverage:` line then says the words
+went into the input and the capped atoms come from the first 23% of the input's tokens (the
+warning under that line, when half or less of the words went in, says "went into the input"
+too). `novelty atoms` prints the same warning on stderr; `probe` and `shorten`, which read its
+`atoms.json`, do not repeat it.
+
+A cap met on the input's last span leaves no stretch of the input out: what it cut is any atoms
+past the cap in that last reply. The warning then reads `atom extraction hit the cap of 30 at
+span s0010 (10 of the input's 10 spans, 100% of its tokens): any atoms past the cap in its last
+reply were dropped; pass --all-atoms for the full sweep`, the `novelty.md` header says the
+atoms reach the input's last span and the cap dropped any atoms past it in the last reply, and
+the `coverage:` line ends `the capped atoms reach the input's last span, and the cap dropped
+any atoms past it`; none of them says "the opening".
+
 That is the point of naming tiers at all. The cheap run is useful; a cheap run quoted as a
 sweep is not.
 
@@ -964,6 +989,7 @@ lost to a failed call (with its error). The
 | `retried_batches` | batches whose reply did not parse, or had no usable atom, and were split and re-asked |
 | `calls`, `batches` | calls made, splits included, against the calls the run would make with no splitting |
 | `last_error` | the error behind the unread spans (a failed call first); errors a split recovered from are not recorded |
+| `reached` | on a capped run, how far through the input (the spans segmented, not the files the bundle left out) the cap reached: `last_span`, the last span any kept atom cites among the spans some call was sent (a batch the cap cut mid-reply was sent whole, but its later spans gave no atom; a cited span no call was sent does not count); `spans`, the spans up to and including it, `of` all the input's spans; `tokens_share`, those spans' share of the input's span tokens. `null` when the cap did not bind |
 
 `probe`, `shorten` and `view` read `atoms.json` and repeat the warning. Unread spans are left out
 of `frac_carrying_nothing` (numerator and denominator) and dead weight, reported as

dokimi/dokimi/commands/novelty.py

@@ -140,13 +140,9 @@ def cmd_novelty_analyze(args: argparse.Namespace) -> None:
 
 def cmd_novelty_atoms(args: argparse.Namespace) -> None:
     """Extract atoms and spans only — the expensive stage, cached for reuse."""
-    from dokimi.novelty.atoms import (
-        ExtractionError,
-        ExtractionReport,
-        extract_atoms,
-        save_atoms,
-        verify_atoms,
-    )
+    from dokimi.novelty.atoms import (ExtractionError, ExtractionReport, extract_atoms, save_atoms,
+                                      verify_atoms)
+    from dokimi.novelty.report import cap_warning
     from dokimi.novelty.spans import segment, span_summary
 
     artifact_bundle = _bundle_from_args(args)
@@ -159,7 +155,9 @@ def cmd_novelty_atoms(args: argparse.Namespace) -> None:
         atoms = extract_atoms(spans, client=extractor, cap=cap, report=extraction)
     except ExtractionError as exc:
         raise SystemExit(str(exc)) from None
-    for warning in extraction.warnings(len(spans)):
+    # The cap warning analyze records; probe and shorten, which read atoms.json, do not repeat it.
+    reach = [cap_warning(cap, extraction.reached)] if extraction.reached else []
+    for warning in extraction.warnings(len(spans)) + reach:
         print(f"WARNING {warning}", file=sys.stderr)
     if settings.values["verify_faithfulness"]:
         verify_atoms(atoms, spans, client=extractor)
@@ -1151,20 +1149,23 @@ def _print_headline(artifact: dict) -> None:
         f"degraded: {paired['lenient']['degraded']}, "
         f"dead tokens: {artifact['spans'].get('frac_carrying_nothing')}"
     )
-    _print_coverage(artifact.get("bundle") or {})
+    _print_coverage(artifact.get("bundle") or {}, artifact.get("extraction"))
     for warning in artifact.get("warnings", []):
         print(f"WARNING {warning}", file=sys.stderr)
 
 
-def _print_coverage(bundle: dict) -> None:
+def _print_coverage(bundle: dict, extraction: dict | None = None) -> None:
     """How much of the artifact's readable words this run read, on stdout.
 
     The share is stated on stdout because the headline is what a user copies. A run under
     the floor also says so on stdout: a stderr-only warning is one a user piping the
     headline never sees, and a rate quoted off half-read skill is not a property of it.
+    A capped run says the words went into the input, and how far its atoms reach.
     """
     from dokimi.novelty.bundle import MEASURED_SHARE_WARN_AT, Coverage
+    from dokimi.novelty.report import coverage_reach, floored_percent
 
+    went, reach = coverage_reach(extraction)
     coverage = Coverage.from_record(bundle)
     if coverage is None or not coverage.denominator("readable"):
         return
@@ -1174,19 +1175,16 @@ def _print_coverage(bundle: dict) -> None:
     rerun = "" if coverage.include_resources else "; pass --include-resources to measure the bundled files"
     if share is None:
         print("coverage: unknown (a bundled file did not decode, so its words are not "
-              f"counted){scope}")
+              f"counted){scope}{reach}")
         # The share is unknowable, but a floor on what was left unread is not: for an
         # entry-point run it is the stderr warning's figure, and a capped bundle run has one too.
         if coverage.counted_unmeasured_share >= 1 - MEASURED_SHARE_WARN_AT:
             part = "what this run read is" if coverage.include_resources else "the entry point is"
             print(f"WARNING: {part} half or less of the artifact's readable words, so every "
                   f"rate here describes that part, not the artifact{rerun}")
         return
-    # Floored on the 4-place share, in integers: a float product floors 0.29 to 28, and
-    # 49.99% must not print as "50%" beside a "below 50%" warning. A true 49.996% is already
-    # 0.5 by the time it is stored.
-    pct = "<1%" if 0 < share < 0.01 else f"{round(share * 10000) // 100}%"
-    print(f"coverage: {pct} of the artifact's readable words were measured{scope}")
+    pct = floored_percent(share)
+    print(f"coverage: {pct} of the artifact's readable words {went}{scope}{reach}")
     if share <= MEASURED_SHARE_WARN_AT:
-        print(f"WARNING: only {pct} of the artifact's readable words were measured, so every "
+        print(f"WARNING: only {pct} of the artifact's readable words {went}, so every "
               f"rate here describes that part, not the artifact{rerun}")

dokimi/dokimi/gate/novelty_gate.py

@@ -196,7 +196,7 @@ def _conflict_reading(artifact: dict, cfg: dict[str, Any]) -> tuple[int, int, bo
     elif capped:
         reason = (
             f"the atom set was capped ({spans.get('n_atoms_indexed')} indexed), so 0 conflicts "
-            "among the sampled atoms is not a cleared check of the artifact"
+            "among the capped atoms is not a cleared check of the artifact"
         )
     else:
         reason = None

dokimi/dokimi/novelty/analyze.py

@@ -182,11 +182,8 @@ def analyze(
     sampling = (
         f"cap-per-artifact:{config.max_atoms}" if config.max_atoms is not None else "full"
     )
-    if config.max_atoms is not None and len(atom_list) >= config.max_atoms:
-        warnings.append(
-            f"atom extraction hit the cap of {config.max_atoms}; rates describe the "
-            "sampled atoms, not the whole artifact"
-        )
+    if extraction.reached and config.max_atoms is not None:  # reached is set only under a cap
+        warnings.append(report.cap_warning(config.max_atoms, extraction.reached))
     if not atom_list:
         warnings.append("no atoms with span provenance were extracted; nothing to measure")
 

dokimi/dokimi/novelty/atoms.py

@@ -198,6 +198,8 @@ class ExtractionReport:
         call_error: The last failed call on a span that stayed unread. Errors that
             splitting recovered from are not recorded: they did not lose anything, and
             quoting one would blame the wrong cause.
+        reached: How far through the input a capped atom set reached (see
+            :func:`_cap_reach`); ``None`` when the cap did not bind.
     """
 
     unread_span_ids: list[str] = field(default_factory=list)
@@ -212,6 +214,7 @@ class ExtractionReport:
     parse_error: str = ""
     call_error: str = ""
     unusable_error: str = ""
+    reached: dict | None = None
 
     @property
     def last_error(self) -> str:
@@ -287,6 +290,7 @@ def as_dict(self) -> dict:
             "calls": self.calls,
             "batches": self.batches,
             "last_error": self.last_error,
+            "reached": self.reached,
         }
 
 
@@ -318,8 +322,9 @@ def extract_atoms(
             affordable; too large and the model starts skipping spans.
         cap: Stop after this many atoms. Recorded in the artifact as the sampling
             rule, so a capped run is never quoted as a full sweep.
-        report: Filled in place with what could not be read and what re-asking cost.
-            Spans no call could read are also marked ``unread`` in place.
+        report: Filled in place with what could not be read and what re-asking cost,
+            and how far a binding cap reached (``reached``). Spans no call could read
+            are also marked ``unread`` in place.
 
     Returns:
         Atoms in extraction order, de-duplicated by id.
@@ -340,6 +345,7 @@ def parse(reply: str) -> list:
     for span in spans:
         span.unread = False
     seen: set[str] = set()
+    sent: set[str] = set()  # span ids some call was shown, which bound the reach
     atoms: list[Atom] = []
     lost: Exception | None = None
     # (spans, is a half): a half was counted in ``batches`` with the batch it came from.
@@ -351,6 +357,7 @@ def parse(reply: str) -> list:
         if cap is not None and len(atoms) >= cap:
             break
         batch, is_half = pending.pop(0)
+        sent.update(span.id for span in batch)
         if not is_half:
             report.batches += 1
         prompt = _EXTRACT_PROMPT.format(
@@ -406,9 +413,41 @@ def parse(reply: str) -> list:
         # Only when nothing was read: spans that were read and asserted nothing are a
         # result, and a recovered parse error is not a failure.
         raise ExtractionError(f"no atoms: every extraction call failed ({lost})") from lost
+    report.reached = _cap_reach(spans, atoms, cap, sent)
     return atoms
 
 
+def _cap_reach(spans: list[Span], atoms: list[Atom], cap: int | None,
+               sent: set[str]) -> dict | None:
+    """How far through the input (``spans``) a capped atom set reached; ``None`` under the cap.
+
+    Extraction walks the spans in document order and stops at the cap, so a capped set
+    covers the input up to the span it stopped at, not a sample of it. The input is the text segmented,
+    not the artifact: a bundled file it left out counts in no span. The reach is the last span any
+    kept atom cites, not the last batch sent: a batch the cap cut mid-reply was sent whole,
+    but its later spans gave no atom. Only the ``sent`` spans count: an atom may cite a span
+    of the document its call was never shown, and that made-up id must not move the reach
+    (an atom set citing nothing sent reaches the last span sent). ``spans`` counts the spans
+    up to and including it, ``of`` all of them, and ``tokens_share`` is their share of the
+    span tokens.
+    """
+    if cap is None or not atoms or len(atoms) < cap:
+        return None
+    order = {span.id: index for index, span in enumerate(spans) if span.id in sent}
+    last = max((order[span_id] for atom in atoms for span_id in atom.span_ids if span_id in order),
+               default=max(order.values()))
+    reached = sum(span.tokens for span in spans[:last + 1])
+    total = sum(span.tokens for span in spans)
+    # Floored in integers, as floored_percent prints it: a cap one span short of the end must not round to 1.0
+    share = reached * 10000 // total
+    return {
+        "last_span": spans[last].id,
+        "spans": last + 1,
+        "of": len(spans),
+        "tokens_share": (max(share, 1) if reached else 0) / 10000,
+    }
+
+
 def _mark_unread(batch: list[Span], report: ExtractionReport) -> None:
     for span in batch:
         span.unread = True