Repository navigation
Keep every local model path out of the site and the endpoint id - #5
Merged
Merged
Conversation
run_measurement copied the id and revision of the frozen wic-50 v1 catalog into every run, but it hashed the scenarios that actually ran. A run with --scenarios therefore claimed to be wic-50 v1. Default runs made the same claim, because the embedded set no longer matches the v1 hash since 964f6e5. A run now keeps the catalog id and revision only when its corpus hash equals the catalog hash. Any other corpus records id "custom" and its hash as the revision. The M7 spec states the rule under T1.2. The evidence test pinned the old label for a one-scenario corpus. It now expects "custom". A new test runs the frozen catalog scenarios and expects wic-50 v1. The CLI test checks the full corpus block of a --scenarios run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate deserialized a result with serde and checked only the totals. Serde accepts values that the published schema rejects, so validate printed "valid" for a failure_class of null, a failure.http_status of 0, or a preflight_override with forced false. The same gap existed for v1 and v2: a v2 failure_class of "bogus" and a v1 run_id passed too. The parser now checks the raw document against the embedded schema for its version, after serde so the serde messages stay the same. The check covers validate, site, annotate and rescore, because all of them parse through it. The hand-written v2 property check is removed, because the v2 schema covers it. jsonschema moves from a dev-dependency to a dependency. Cargo.lock does not change. annotate now refuses a v1 result. The v1 schema has no cause field, so the edited file always failed its own schema. All 187 tracked result files still validate, and the generated site is byte-identical. The release binary grows from 7.9 MB to 12.3 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser accepted any string in evidence_path. The site then wrote it verbatim into results.json and results.csv, and linked it from the index and the appendix. A result with /Users/alice/... therefore published the author's home directory. annotate --reference had the same defect: it accepted /Users/alice/notes.md, and validate and site both passed the edited file. validate_result and validate_measurement now check both fields. Every parse runs one of them, and every write runs them first. An evidence_path must be a relative path under evidence/. A cause reference must be an http(s) URL or a relative repository path. Each path segment must be non-empty, must not start with '.', and may use only ASCII letters, digits, '.', '_' and '-'. The check does not use std::path, because a Unix host does not see C:\ as absolute. All 187 tracked result files pass. One site fixture had a leading '/' on its evidence path. It is fixed, and the site golden does not change. The leading '/' trims in evidence_url and reference_url are removed, because no such value reaches them now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An endpoint can answer with a status from 600 to 999. The runner recorded that status in failure.http_status, but the v3 schema allows 100 to 599 only. Since the readers check the schema, run wrote a file that validate, site, annotate and rescore then refused. RFC 9110 defines status codes 100 to 599 only. The runner now records http_status only in that range. The failure code stays http_error, and the evidence keeps the raw status. Both result writers now check the serialized document the way the readers do, before they write it. A writer and the schema can no longer drift apart without a loud error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reference check applied the path segment rule to the whole value. It refused a case-study link with a section anchor, such as docs/case-studies/x.md#section, and a line anchor such as runner.rs#L770. The base accepted both and rendered working links. The check now splits off one '#' fragment. The path part keeps the segment rule. The fragment allows ASCII letters, digits, '.', '_' and '-' only, so it cannot carry a local path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The site sanitizers used Path::is_absolute. On Unix that misses C:\..., \\server\..., ~/... and file://, so a result recorded on Windows or with a home path put the user name into index.html, outcomes.html, results.csv and the JSON export. The site now uses wic_core::registry::is_local_absolute_path, the same rule that the registry applies to selectors, and shows only the last path component. One helper in site/data.rs replaces the three copies. The 32 checked-in results give a byte-identical site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
is_local_absolute_path matched only a leading '/', '~/', '~\', a drive letter with a separator, '\\' and 'file://'. It missed a rooted Windows path (\Users\...), a drive-relative path (C:Users\...), a '~user' path and a 'file:' URI in another case or with one slash. llama-server -m accepts each of these forms, so the site and a resolved endpoint id kept the user name. The rule now matches a leading '\', any leading '~', a case-insensitive 'file:' and a drive letter with a separator later in the value. An Ollama tag such as 'q:8b' has no separator, so it stays a model id. The registry schema mirrors the rule. None of the 32 published selectors changes. A registry test covers each form through resolve, the schema and the loader. The site test now covers each form in model_id and a v3 artifact source_id, which reaches the appendix and the JSON export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The site sanitizers use
Path::is_absolute. On Unix, that check does not match a Windows path, a~path or afile:URI. A result with such a model path puts the full path intoindex.html,outcomes.html,appendix.html,results.csvandresults.json. A Windows path and a~userpath carry the user name.The registry rule
is_local_absolute_pathalso misses four path forms.willitcall run --modeltakes any string, so each form can reachresolve:\Users\...)C:Users\...)~userpathfile:URI with one slash or in upper caseFor these forms,
resolvekeeps the full path as the endpoint id.This PR is stacked on #4. Merge #4 first.
Fix
safe_local_valueinsite/data.rs, replaces the three copies indata.rs,export.rsandhtml.rs. It useswic_core::registry::is_local_absolute_path, which is nowpub.is_local_absolute_pathnow matches a leading/,\or~, a case-insensitivefile:, and a drive letter with a separator later in the value. An Ollama tag such asq:8bhas no separator, so it stays a model id.safe_selectorpatterns inschemas/model-registry-v1.schema.jsonmirror the new rule. None of the 32 published selectors changes.Testing
cargo fmt --checkandcargo clippy --workspace --all-targets -- -D warningsare clean.cargo test --workspace: 151 tests pass. Fix the three Codex findings on #2 #4 has 150.site_never_renders_absolute_model_pathswrites each path form as amodel_idand as a v3 artifactsource_id. It checks five generated files for the user name.every_local_path_form_is_an_unsafe_selector_and_keeps_only_its_file_namesends each path form throughresolve, the schema and the registry loader. It also checks that five model ids, such asq:8bandfile-model:latest, stay valid selectors.!generated.contains("someone").is_local_absolute_pathrule: both new tests fail.source_idinhtml.rs, or inexport.rs: the site test fails.willitcall site --results results/gives output that is byte-identical to Fix the three Codex findings on #2 #4.toolsunittest and the redaction ledger check pass.This PR and the
wic-50v2 PR change the same import block insite/data.rs. The second one to merge needsmainmerged in first. The merged state of both passes all 155 tests.