Repository navigation
Fix the three Codex findings on #2 - #4
Merged
Merged
Conversation
run_measurement copied the id and revision of the frozen wic-50 v1 catalog into every run, but it hashed the scenarios that actually ran. A run with --scenarios therefore claimed to be wic-50 v1. Default runs made the same claim, because the embedded set no longer matches the v1 hash since 964f6e5. A run now keeps the catalog id and revision only when its corpus hash equals the catalog hash. Any other corpus records id "custom" and its hash as the revision. The M7 spec states the rule under T1.2. The evidence test pinned the old label for a one-scenario corpus. It now expects "custom". A new test runs the frozen catalog scenarios and expects wic-50 v1. The CLI test checks the full corpus block of a --scenarios run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate deserialized a result with serde and checked only the totals. Serde accepts values that the published schema rejects, so validate printed "valid" for a failure_class of null, a failure.http_status of 0, or a preflight_override with forced false. The same gap existed for v1 and v2: a v2 failure_class of "bogus" and a v1 run_id passed too. The parser now checks the raw document against the embedded schema for its version, after serde so the serde messages stay the same. The check covers validate, site, annotate and rescore, because all of them parse through it. The hand-written v2 property check is removed, because the v2 schema covers it. jsonschema moves from a dev-dependency to a dependency. Cargo.lock does not change. annotate now refuses a v1 result. The v1 schema has no cause field, so the edited file always failed its own schema. All 187 tracked result files still validate, and the generated site is byte-identical. The release binary grows from 7.9 MB to 12.3 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser accepted any string in evidence_path. The site then wrote it verbatim into results.json and results.csv, and linked it from the index and the appendix. A result with /Users/alice/... therefore published the author's home directory. annotate --reference had the same defect: it accepted /Users/alice/notes.md, and validate and site both passed the edited file. validate_result and validate_measurement now check both fields. Every parse runs one of them, and every write runs them first. An evidence_path must be a relative path under evidence/. A cause reference must be an http(s) URL or a relative repository path. Each path segment must be non-empty, must not start with '.', and may use only ASCII letters, digits, '.', '_' and '-'. The check does not use std::path, because a Unix host does not see C:\ as absolute. All 187 tracked result files pass. One site fixture had a leading '/' on its evidence path. It is fixed, and the site golden does not change. The leading '/' trims in evidence_url and reference_url are removed, because no such value reaches them now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An endpoint can answer with a status from 600 to 999. The runner recorded that status in failure.http_status, but the v3 schema allows 100 to 599 only. Since the readers check the schema, run wrote a file that validate, site, annotate and rescore then refused. RFC 9110 defines status codes 100 to 599 only. The runner now records http_status only in that range. The failure code stays http_error, and the evidence keeps the raw status. Both result writers now check the serialized document the way the readers do, before they write it. A writer and the schema can no longer drift apart without a loud error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reference check applied the path segment rule to the whole value. It refused a case-study link with a section anchor, such as docs/case-studies/x.md#section, and a line anchor such as runner.rs#L770. The base accepted both and rendered working links. The check now splits off one '#' fragment. The path part keeps the segment rule. The fragment allows ASCII letters, digits, '.', '_' and '-' only, so it cannot carry a local path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Codex review on #2 reported three defects. All three are still on
main.--scenariosrecords the corpus aswic-50revisionv1. Two different scenario sets therefore share one label.validateaccepts documents that the published schemas reject. Onmain,validateexits 0 for each of these ten documents:failure_class: null,cause: null,http_status: 0,forced: false,ports: []orparameter_count_b: -1evidence_pathfailure_class: "bogus"run_idor acauseevidence_pathandcause.referenceaccept absolute local paths. The site copies them intoresults.json,results.csvand the HTML links.Fix
wic-50andv1. Any other scenario set gets idcustom, and its revision is the corpus SHA-256. The measurement contract spec states this rule under T1.2.jsonschemafrom dev-dependencies to dependencies.Cargo.lockdoes not change.annotaterefuses a v1 result. The v1 schema has nocausefield, so an annotated v1 file fails its own schema.evidence_pathmust be a relative path underevidence/.cause.referencemust be anhttp://orhttps://URL or a relative repository path, with an optional#fragment. Each path segment must be non-empty, must not start with., and must use only ASCII letters, digits,.,_and-. The fragment uses the same characters./, because the validator now refuses one. One site fixture had a leading/in a reference. I removed it. The golden output does not change.failure.http_statusonly from 100 to 599, as RFC 9110 defines. An endpoint can answer 600 to 999. The v3 schema refuses those values, so without this changerunwrites a file that the new check refuses. The failure code stayshttp_error, and the evidence keeps the raw status.Testing
cargo fmt --checkandcargo clippy --workspace --all-targets -- -D warningsare clean.cargo test --workspace: 150 tests pass.mainhas 141.mainand exits 2 on this branch.willitcall validate.willitcall site --results results/gives output that is byte-identical tomain.resultsjob (PythonDraft202012Validator), thetoolsunittest and the redaction ledger check pass. The ledger check verifies 6 runs and 300 transcripts.validateaccepts.jsonschemais now a runtime dependency.siteoverresults/takes 0.05 s.maintakes 0.04 s.