Skip to content

feat(check): name the registries that declined to answer - #141

Open
justin13888 wants to merge 9 commits into
feat/111-advance-forced-versionsfrom
feat/112-per-registry-unreachable
Open

feat(check): name the registries that declined to answer#141
justin13888 wants to merge 9 commits into
feat/111-advance-forced-versionsfrom
feat/112-per-registry-unreachable

Conversation

@justin13888

@justin13888 justin13888 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

What this changes

ManifestCheck::registry_unreachable was one boolean per manifest, set at a single site
in Checker::fetch_all when any lookup failed for a reason other than NotFound. A
--fail-on gate could refuse to certify a run, but could only say "the registry did not
answer" — never which one. In a polyglot monorepo where proxy.golang.org timed out
while npm, PyPI and crates.io answered every request, the refusal named nothing anyone
could go and check.

Per-manifest is not the routing unit either. Checker::route_item produces three
distinct routes inside Ecosystem::Npm alone — the default registry, JSR, and a
per-scope private registry from .npmrc — and a Cargo.toml reaches crates.io plus
every alternate registry its dependencies name. Each already has its own cache key.

So fetch_all now records the route behind each failed lookup, keyed by that cache
key, and ManifestCheck reports them:

  • New public dependable_fetch::UnreachableRegistry { ecosystem, root }, built with
    UnreachableRegistry::new, with label() rendering the name a message prints.
  • New public ManifestCheck::unreachable_registries: Vec<UnreachableRegistry>, sorted
    by ecosystem then root and deduplicated per registry.
  • ManifestCheck::registry_unreachable stays, derived from that collection at the
    one site that builds both, so the two cannot disagree. Additive, not breaking.
  • The CLI gate unions the collections across every ManifestReport, re-sorts, and
    names them.

The refusal reads:

failing registries sentence
one error: cannot honour --fail-on: the Go registry did not answer
two error: cannot honour --fail-on: the Go and npm registries did not answer
three error: cannot honour --fail-on: the Go, JVM and npm registries did not answer

A root that is not the ecosystem's own default is named beside it — npm (jsr.io)
which is what tells JSR apart from npm inside one deno.json. The root is reduced to
host and port before it is printed: .npmrc interpolates ${VAR} into its registry
lines and .dependable.toml roots are hand-written, so https://ci:secret@nexus.internal/repo
renders as nexus.internal rather than putting a credential into CI job output.

Unchanged by construction: the predicate that decides what counts as a registry
declining to answer is character-identical (!matches!(e, FetchError::NotFound(_))), an
empty collection pushes no reason at all, and the FailOn::Any and FailOn::None early
exits are untouched. The existing 404/410 carve-out tests pass without edits.

Closes #112

Decisions taken

1. Deliverable boundary — how far the routing key goes

  • Taken: carry the routing key through the library so ManifestCheck reports WHICH
    registries did not answer, and have the gate's refusal name them. The
    partial-certification opt-in the issue also proposes is filed, not built.
  • Rejected: deriving the answer in the CLI from ManifestReport::ecosystem without
    touching dependable-fetch — it is cheaper, and per-ecosystem granularity really is
    derivable today, but it adds no routing key anywhere and the issue's own words are that
    ErrorOrigin::Unanswered "already carries the per-dependency half; what is missing is
    the routing key alongside it". It is also wrong at the margin: a deno.json failing
    only against JSR would report "npm did not answer", which is the same class of
    imprecision the issue was filed about, one level down.
  • Rejected: also building the opt-in that certifies a run over the ecosystems that
    DID answer — that changes what a gate PROMISES, not its granularity.
    gate_is_answerable's doc comment is an argued position that a gate whose inputs are
    missing must fail, and extending it needs that argument made, not a flag added. It also
    carries an unresolved semantic: which results are certified when one Cargo.toml
    routes to two registries and only one answered.
  • Rejected: exposing the set in check --format json — no integrity data appears in
    ANY output document today, so this would be the first, and it would set a precedent
    about what belongs in the check document as a side effect of a granularity change.
  • Reverses: delete the collection from ManifestCheck and derive the message from
    ManifestReport::ecosystem in gate_is_answerable.
  • Filed: the partial-certification opt-in, and the machine-readable exposure.

2. What the routing key IS

  • Taken: ecosystem plus an optional registry root — a struct, not a bare enum and not
    a bare string. The message prints the ecosystem name, and appends the root only when it
    differs from that ecosystem's default registry, reusing the exact discrimination
    default_cache_key already makes for deciding whether a cache key needs scoping.
  • Rejected: Ecosystem alone — it is provably not the routing unit. route_item
    produces three distinct routes within Ecosystem::Npm alone: the default registry,
    JSR, and any per-scope private registry from .npmrc. The issue names exactly those
    cases as why per-manifest is the wrong unit, and per-ecosystem reproduces the same
    collapse one level down.
  • Rejected: a bare registry-root string — RegistryFetcher::registry_root() has a
    trait default returning None, so a third-party fetcher yields no label and the type
    is (Ecosystem, Option<String>) in practice anyway. Structuring it that way from the
    start is honest about the fallback rather than patching around it.
  • Rejected: the cache_key verbatim — it is already at the failure site and costs
    nothing, but default_cache_key renders a non-default root as
    {osv_name}-{16 hex digits}, so a corporate Maven mirror would print
    maven-9f3c1a2b4d5e6f70. It also freezes an internal cache-namespacing convention as a
    public contract.
  • Reverses: reduce the struct to its ecosystem field and drop the root from the
    message.

3. How ManifestCheck::registry_unreachable changes

  • Taken: ADD the collection and keep the existing boolean, derived from it AT
    CONSTRUCTION in one place so the two cannot disagree. The commit is feat:, not
    feat!:.
  • Rejected: replacing the boolean outright — it is the cleaner shape, but
    dependable-fetch is a published crate and the repository CANNOT establish from within
    whether a version carrying registry_unreachable has shipped (the field originates in
    fix: stabilization pass — twelve correctness fixes across core, fetch, cli and report #99, still open). A ! commit resting on an unverifiable assumption, in a stack four
    pull requests deep, is the wrong risk — especially as feat(cli): implement --ecosystem rather than leaving it removed #105 documents that this stack has
    already mis-signalled one CLI break through a fix: commit.
  • Rejected: keeping the boolean public and exposing the collection through a method
    backed by a private field — ManifestCheck is plain data with every field public and no
    private state anywhere; introducing one private field breaks that shape for a struct
    callers pattern-match on.
  • Reverses: delete the boolean and retype the field, as a feat!: commit, once the
    crate's published surface can be established.

4. What the refusal message may print

  • Taken: reduce a registry root to scheme-less authority — host and port only —
    dropping userinfo, path, query and fragment.
    https://ci:secret@nexus.internal/repo/npm renders as nexus.internal.
  • Rejected: printing the root verbatim — registry roots come from .dependable.toml
    and .npmrc, expand_env interpolates ${VAR} into .npmrc content, and the refusal
    goes to stderr, which CI captures as job output. A user may legitimately write
    https://ci:${NPM_TOKEN}@nexus.internal/repo, and printing it would put a credential in
    a log. The repository is otherwise careful here.
  • Rejected: never printing a root at all — it gives up the actionability that
    motivated the routing key, and cannot distinguish two npm registries in one run, which
    is the JSR case the issue cites.
  • Reverses: print root unmodified, or drop it from the message and keep it in the
    data only.

5. Where the label is rendered (taken inside the manifest, not in the record above)

  • Taken: UnreachableRegistry::label() and the private authority_of live in
    dependable-fetch/src/check.rs, beside the type. The CLI calls label().
  • Rejected: rendering in runner.rs — the credential reduction is the security-
    relevant part of this change, and a second consumer of UnreachableRegistry (an IDE,
    the TUI) would have to reimplement it correctly to avoid printing a token. One
    implementation, tested where the type lives, is the safer shape.
  • Reverses: move label and authority_of into runner.rs and reduce the public
    type to its two fields.

6. Silent degradation to the bare ecosystem name

  • Taken: when the root cannot be reduced with confidence, print the ecosystem name
    alone and say nothing about the host. https://nexus.internal/repository/@scope/npm
    an ordinary Nexus npm-proxy path — therefore reads npm, indistinguishable from a
    fetcher that names no root at all.
  • Rejected: printing a distinguishable marker such as npm (registry root withheld)
    — it tells an attacker reading a public CI log that a private registry is configured,
    which is itself a disclosure, and it tells the operator nothing they can act on.
  • Rejected: attempting a smarter parse to recover the host from a path containing @
    — every additional heuristic is another way to be wrong in the direction that prints a
    secret. The failure mode must stay "say less".
  • Reverses: emit a distinct marker in the None branch of label().

7. The redundant boolean's lifetime

  • Taken: keep registry_unreachable underived-but-derived, without #[deprecated].
    It is derived at the single construction site so the two cannot disagree.
  • Rejected: marking it #[deprecated] now — the crate's published surface cannot be
    established from within the repository, so a deprecation warning would fire for
    consumers who may never have seen the field, and the retype it points at is not
    scheduled.
  • Reverses: add #[deprecated(note = "use unreachable_registries")] once the
    published surface is known.

8. Nested comma lists in the refusal

  • Taken: accepted. Three registries plus two other reasons reads the vulnerability scan did not complete, the Go, JVM and npm registries did not answer and 2 dependencies could not be evaluated.
  • Rejected: a separate stderr line for the registry list — the refusal is one sentence
    answering one question, and splitting it would make the exit-2 reason arrive in two
    places.
  • Reverses: emit the registry list on its own line beneath the refusal.

9. Label ordering is ASCII, so JVM and PHP precede npm

  • Taken: accepted. Deterministic is the property that matters; the ordering is stable
    across machines and runs.
  • Rejected: case-insensitive ordering — it is cosmetic, and a second sort key is
    another thing to keep in step with display_name.
  • Reverses: sort with a case-insensitive comparator.

Also settled inside the manifest

  • UnreachableRegistry::new is public. #[non_exhaustive] otherwise makes the type
    unconstructible outside dependable-fetch, and the CLI's own unit tests must build one.
  • ScanIntegrity loses Copy (it kept Clone, which is all the code used) because its
    registry_unreachable field is now a Vec. It is private to a binary-only crate, so
    this is not a semver event.
  • The failure site falls back to UnreachableRegistry::new(ecosystem, None) if the route
    lookup ever misses. Every task contributes a route so it cannot, but a lost entry must
    degrade to a less precise report, never to a certified run.

Not done

  • No CLI flag, config key, JSON/SARIF field, HTML-report field, exit code, or schema
    version. No url dependency — authority_of is hand-rolled over &str and tested
    against userinfo, ports, IPv6 literals, missing schemes and empty input.
  • What a 404 means, the unresolved/unevaluated counters, the --fail-on any exemption,
    OSV and vulnerability_scan_failed are all untouched.
  • A warm run still cannot report an unreachable registry: a cache hit issues no request,
    so nothing declines to answer. Pre-existing, documented on the field, deliberately not
    changed here.
  • NpmFetcher::resolve routes an @scope package to a per-scope registry that
    registry_root() does not name, so a scoped failure is reported under the fetcher's
    default root. Documented as a limitation on the type; filed separately.

Follow-ups filed

Findings raised against this diff before it was reviewed by anyone else

The first commit was audited adversarially before the branch was finished. Four defects
were found and are fixed in fix(check): stop a registry root's credentials reaching the refusal:

  1. High — the redaction leaked credentials. authority_of split the path off before
    stripping userinfo, so a password containing / survived as the "host":
    https://ci:AbC/dEf@nexus.internal/npm reduced to ci:AbC, and the gate printed a
    username and a token prefix on stderr. A standard-alphabet base64 token contains /
    about 40% of the time at 32 characters, and such a root makes every request fail, so
    the label prints on every run rather than rarely. ? and # took the same path. It
    now fails towards saying less: an @ surviving outside the authority candidate means
    the string cannot be split with confidence, so the root is dropped and the ecosystem
    named alone. \ joins the separators, because WHATWG resolves it as / for special
    schemes and reading it as authority named a host that was never contacted.
  2. Low — one registry could be named twice. Labels were deduplicated by adjacency
    under a sort keyed on the raw root, but several roots reduce to one label and need not
    be adjacent — http://nexus.corp/a, http://other.host/x, https://nexus.corp/b
    sort in that order. Labels are now sorted before deduplication.
  3. Low — the published collection deduplicated per route, not per registry, against
    its own doc: two alternate-registry aliases naming one index URL reached a library
    consumer as two identical entries. The CLI masked it.
  4. Nit — two doc comments claimed more than the code did, about what an
    authority-only label distinguishes and about a nesting name_unanswered in fact
    produces. Both now say what is true.

Found and not fixed here — needs its own issue

FetchError::Http's Display appends " for url ({url})", and a url::Url serialises
its user:password@ userinfo. So a run whose registry root carries credentials prints
them in the per-dependency error cellerror: http error: error sending request for url (https://ci:secret@nexus.internal/express) — rendered by
crates/dependable/src/output/table.rs and output/github.rs. That is pre-existing and
untouched by this change, and both files are outside this change's manifest, so the
README wording here was narrowed to describe the refusal line only rather than the run.
This is a real credential leak into CI output and should be filed and fixed
separately.

Findings raised in a second review round, and fixed here

  1. High — authority_of still leaked a credential fragment. The check for an @
    surviving past the authority was a match guard on the None arm of the userinfo
    split, so it was consulted only when the authority candidate held no @ of its own. A
    root whose userinfo and whose path both contain one took the Some arm instead, and
    rsplit_once returned the text between the embedded @ and the first delimiter —
    credential material, and a host the run never contacted. Such a root makes every
    request to it fail, so the refusal printed it on every gated run. Seven shapes leaked;
    hoisting the check above the split closes all seven and moves no other row:

    root before after
    https://user:p@ss/word@nexus.internal/x ss (dropped)
    https://ci:AbC@dEf/ghi@nexus.internal/npm dEf (dropped)
    https://ci:p@s?s@nexus.internal/x s (dropped)
    https://ci:p@s#s@nexus.internal/x s (dropped)
    https://ci:p@s\s@nexus.internal/x s (dropped)
    https://ci:tok@en/@nexus.internal/npm en (dropped)
    https://user@host.example:pw/x@real.internal/y host.example:pw (dropped)

    Confirmed against a 40-input corpus covering every row this branch already asserts on:
    exactly those seven rows change, and the other 33 are byte-identical. All seven are now
    regression cases. Whenever authority_of returns Some, every @ in the string lies
    inside the authority, so what survives is a host and port and never userinfo.
    authority_of's doc comment and the README line are corrected to claim only that —
    the README now also states that the line degrades to the bare ecosystem name rather
    than guessing (decision 6).

  2. Low — the public root field carried credentials with no warning. label()'s doc
    explains at length that a root must never be printed whole, but a consumer reading the
    field it is handed got no signal. Decision 5 rests on the argument that a second
    consumer must not have to reimplement the redaction; the field now says so, and names
    label() as the only safe rendering.

  3. Low — fetch_all's sort and dedup were untested at the production site.
    unreachable_registries_sort_by_ecosystem_then_root re-applied both in its own body
    over a hand-built vector, and the one integration test reaching fetch_all yielded a
    single entry, so deleting both lines from fetch_all left the whole suite green — the
    CLI re-sorts downstream, and that half is covered. The library's published ordering,
    which an embedding consumer reads straight off the value, was untested. A new
    integration test has both of a deno.json's registries decline and asserts the
    returned order directly; both routes are Ecosystem::Npm, so only the root can order
    them. Because the outcomes are gathered in a HashMap whose iteration order is seeded
    per map, the check is repeated concurrently through fresh Checkers, which costs one
    round of retry backoff. Measured: 30/30 runs pass as written, 30/30 fail with the
    sort deleted from fetch_all.
    The unit test keeps its coverage of the key and the
    dedup, with its doc comment corrected to claim only that.

Validation

Command Outcome
cargo test -p dependable-fetch --lib pass — 96 passed, 0 failed
cargo test -p dependable-fetch --test checker pass — 28 passed, 0 failed
cargo test -p dependable --test cli_gate pass — 13 passed, 0 failed; the 404/410 carve-out tests pass on their existing assertions
cargo test -p dependable --bin dependable pass — 117 passed, 0 failed
mise run test pass — whole workspace, 0 failures
mise run fmt:check pass — no output
mise run lint pass — clippy --workspace --all-targets -D warnings, clean
convco check origin/feat/111-advance-forced-versions..HEAD pass — "no errors in 7 commits"

No failure was observed in this branch, so nothing needed classifying as caused,
pre-existing, flaky or unavailable.

The refusal was also read off the built binary rather than only off the tests, against
two loopback servers answering 500:

error: cannot honour --fail-on: the Go (127.0.0.1:19501) registry did not answer
error: cannot honour --fail-on: the Go (127.0.0.1:19501) and npm (127.0.0.1:19502) registries did not answer

Base

Stacked on #122 (feat/111-advance-forced-versions), the tip of
master ← #99#106#108#116#122 ← this.

`ManifestCheck::registry_unreachable` was one boolean per manifest, so a
`--fail-on` gate could refuse to certify a run but never say which registry
had declined. A manifest is not the routing unit either: `route_item` sends a
`deno.json` to npm and to JSR, and a `Cargo.toml` to crates.io and to every
alternate registry its dependencies name, so a per-manifest — or even a
per-ecosystem — answer collapses routes that failed independently.

`fetch_all` now records the route behind each failed lookup, keyed by the
cache key that already distinguishes those routes, and returns them sorted so
`buffer_unordered`'s arrival order cannot reach the message. `ManifestCheck`
gains `unreachable_registries`; the boolean stays, derived from the collection
at the one site that builds both, because this crate is published and the
field is part of its surface.

The refusal now names them: "the Go registry did not answer", "the Go and npm
registries did not answer". A root that is not the ecosystem's own default is
named beside it — `npm (jsr.io)` — reduced to host and port, because a
configured root may carry credentials (`.npmrc` interpolates `${VAR}`) and the
refusal is printed on stderr, which CI captures as job output.

The 404 carve-out is untouched: the predicate that decides what counts as a
registry declining to answer is character-identical, and an empty collection
pushes no reason at all.

Refs #112
Two end-to-end tests the old boolean could not express.

`checker.rs` drives a `deno.json` against two servers: npm answers the
packument, JSR answers 500. Both routes are `Ecosystem::Npm`, so this is the
case that separates a per-registry answer from a per-ecosystem one — the
unreachable set names JSR alone, by JSR's own host, while the npm import is
evaluated in full.

`cli_gate.rs` runs the real binary with npm pointed at one loopback registry
and Go at another, and asserts exit 2, that the refusal names the Go proxy,
and — the assertion the issue is actually about — that it does not name npm,
which answered every request.

The 404/410 carve-out tests are unchanged apart from the pinned wording they
assert against.

Refs #112
The exit-code section said a run that could not reach a registry exits 2, but
not what it prints. It now shows the one- and two-registry sentences, says
that a non-default root is named beside its ecosystem and reduced to host and
port so a credential in a configured root cannot reach CI output, and repeats
that a 404 or 410 is an answer rather than an outage.

Refs #112
Four defects raised against the previous commit before review.

`authority_of` split the path off before stripping userinfo, so a password
containing `/` — a base64 token contains one about 40% of the time at 32
characters — survived as the "host": `https://ci:AbC/dEf@nexus.internal/npm`
reduced to `ci:AbC`, and the gate printed a username and a token prefix on
stderr, where CI captures it as job output. The same held for `?` and `#`.
This is a redaction, so it now fails towards saying less: the authority
candidate is taken first, and an `@` surviving outside it means the string
cannot be split into userinfo and host with any confidence, so the root is
dropped and the ecosystem named alone. `\` joins the separators because
WHATWG resolves it as `/` for special schemes, so reading it as part of the
authority named a host the run never contacted.

`unanswered_registries` deduplicated labels by adjacency under a sort keyed on
the raw root, but several roots reduce to one label and need not be adjacent:
three Rust registries at `http://nexus.corp/a`, `http://other.host/x` and
`https://nexus.corp/b` sort in exactly that order, and the sentence named
`nexus.corp` twice. Labels are now sorted before they are deduplicated, which
also makes the printed order the order a reader sees.

`ManifestCheck::unreachable_registries` promised deduplication per registry
but deduplicated per route, so two alternate-registry aliases naming one index
URL reached a library consumer as two identical entries. The CLI masked it.

`label()` now says what an authority-only name cannot distinguish — two paths
on one host — and `name_unanswered` no longer claims to have avoided a nesting
it in fact produces.

Refs #112
`authority_of` asked whether an `@` survived past the authority as a match
guard on the `None` arm of the userinfo split, so the question was put only
when the authority candidate held no `@` of its own. A root whose userinfo and
whose path both contain one took the `Some` arm instead and reduced to the text
between the embedded `@` and the first delimiter — a fragment of the token, and
a host the run never contacted:

    https://ci:AbC@dEf/ghi@nexus.internal/npm  ->  dEf

Such a root makes every request to it fail, so the refusal printed it on every
gated run, into CI job output. Seven shapes leaked, across `/`, `?`, `#` and
`\\` as the delimiter.

Hoist the check above the split so it applies unconditionally. Whenever
`authority_of` now returns `Some`, every `@` in the string lies inside the
authority, so what survives is a host and port and never userinfo. Verified over
a 40-input corpus: the seven leaking rows become `None` and no other row moves,
including every row this branch already asserted on.

The seven are added as regression cases, and `authority_of`'s doc comment and
the README line describing the refusal are corrected — the README now also says
that the line degrades to the bare ecosystem name rather than guessing.
…edentials

`label()`'s doc explains at length that a root may carry credentials and must
never be printed whole, but a consumer reading the field it is handed got no
signal at all — its doc said only where the root came from. The argument that a
second consumer must not have to reimplement the redaction needs the field
itself to say so, and to name `label()` as the only safe rendering.
…roduces it

`unreachable_registries_sort_by_ecosystem_then_root` re-applied
`sort_by(unreachable_sort_key)` and `dedup()` in its own body over a hand-built
vector, so it exercised the key and not `fetch_all`'s use of it. The one
integration test that reached the production site yielded a single entry, so it
passed either way: deleting both lines from `fetch_all` left the whole suite
green, because the CLI re-sorts and re-dedups downstream and that half is
covered. The library's published ordering — which an embedding consumer reads
straight off the value — was untested.

Add an integration test in which both of a `deno.json`'s registries decline, and
assert the returned `unreachable_registries` order directly. Both routes are
`Ecosystem::Npm`, so only the root can order them. The outcomes are gathered in
a `HashMap`, whose iteration order is seeded per map, so one check would agree
by luck half the time; the check is repeated concurrently through fresh
`Checker`s, which costs one round of retry backoff and drives an unsorted
`fetch_all`'s chance of passing to 1 in 256. Measured: 30/30 runs pass as
written, 30/30 fail with the sort deleted from `fetch_all`.

The unit test keeps its coverage of the key and the dedup, with its doc comment
corrected to claim only that.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant