diff --git a/.github/actions/dependable-check/README.md b/.github/actions/dependable-check/README.md index e0269b0..f003cd3 100644 --- a/.github/actions/dependable-check/README.md +++ b/.github/actions/dependable-check/README.md @@ -5,7 +5,7 @@ A composite action that installs the released `dependable` binary and runs ```yaml - uses: actions/checkout@v4 -- uses: getkono/dependable/.github/actions/dependable-check@v0.1.4 +- uses: getkono/dependable/.github/actions/dependable-check@v0.1.3 with: fail-on: vulnerable ``` @@ -37,7 +37,7 @@ single valid document while the annotations still reach the pull request. | `fail-on` | `vulnerable` | `none`, `outdated`, `vulnerable`, or `any`. | | `format` | `table` | `table`, `json`, `text`, or `sarif` — what goes to stdout. | | `annotations` | `auto` | `auto`, `always`, or `never`. `never` also turns off the job summary. | -| `version` | `latest` | A release tag such as `v0.1.4`, or `latest`. | +| `version` | `latest` | A release tag such as `v0.1.3`, or `latest`. | | `args` | `''` | Extra arguments appended to `dependable check` verbatim. | **`fail-on` defaults to `vulnerable`, deviating from the CLI's `none`.** This is @@ -59,7 +59,7 @@ Counts are deliberately not outputs: they would duplicate the JSON schema and drift from it. ```yaml -- uses: getkono/dependable/.github/actions/dependable-check@v0.1.4 +- uses: getkono/dependable/.github/actions/dependable-check@v0.1.3 id: deps continue-on-error: true with: @@ -83,7 +83,7 @@ Releases are tagged `v{version}` only — there is **no floating `v1` tag** — pin a full tag: ```yaml -uses: getkono/dependable/.github/actions/dependable-check@v0.1.4 +uses: getkono/dependable/.github/actions/dependable-check@v0.1.3 ``` `version: latest` resolves the newest release at run time; pinning `version:` to diff --git a/.github/actions/dependable-check/action.yml b/.github/actions/dependable-check/action.yml index 8584dd0..45b3542 100644 --- a/.github/actions/dependable-check/action.yml +++ b/.github/actions/dependable-check/action.yml @@ -40,7 +40,7 @@ inputs: default: 'auto' version: description: >- - Release tag to install, such as v0.1.4, or latest. Pinning a tag skips the + Release tag to install, such as v0.1.3, or latest. Pinning a tag skips the releases API call, so a workflow with no contents read permission must pin. required: false default: 'latest' diff --git a/.github/workflows/live.yml b/.github/workflows/live.yml new file mode 100644 index 0000000..be4fb22 --- /dev/null +++ b/.github/workflows/live.yml @@ -0,0 +1,35 @@ +# The `#[ignore]`d network tests, on a schedule. +# +# `mise run test` is hermetic by design, so the entire network-facing surface — +# the crates.io sparse index, npm, PyPI, Packagist, pub.dev, NuGet, Hex, JSR and +# OSV — is never exercised by CI. Those are also the least-covered modules in the +# workspace, and the ones whose correctness depends on a schema this repository +# does not control: when a registry changes a response shape, nothing here notices +# until a user reports a wrong answer. +# +# Deliberately not part of the PR gate. These tests fail for reasons a contributor +# cannot fix — an outage, a rate limit — and a gate that cries wolf gets ignored, +# including on the day it is right. +name: Live + +on: + schedule: + # 05:00 UTC daily, off the top of the hour to avoid the cron stampede. + - cron: "17 5 * * *" + workflow_dispatch: + +env: + CARGO_TERM_COLOR: always + +jobs: + live: + name: Live registry + OSV smoke tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Materialize Rust toolchain + run: rustup show active-toolchain || rustup show + - uses: Swatinem/rust-cache@v2 + - uses: jdx/mise-action@v2 + - name: Run the live tests + run: mise run test:live diff --git a/AGENTS.md b/AGENTS.md index e8f5f2a..dd354aa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,11 @@ # dependable Open-source CLI + Rust library for checking dependency versions and known -vulnerabilities. V1 scope is **Rust / Crates.io only**; see [`docs/SCOPE.md`](docs/SCOPE.md) -for what is deferred and why. +vulnerabilities. Eleven ecosystems ship — Rust, npm, PyPI, Go, Deno/JSR, pnpm, +Packagist, pub.dev, NuGet, Hex and Kotlin/Java on Maven Central — with Rust, npm and +Python marked stable and the rest experimental. The table in +[`README.md`](README.md) is authoritative for what ships and at what maturity; see +[`docs/SCOPE.md`](docs/SCOPE.md) for what is deferred and why. ## Workspace @@ -12,6 +15,10 @@ for what is deferred and why. public end-to-end entry point: the `Checker` (parse → fetch → evaluate → OSV scan) plus async IO (crates.io sparse index, OSV client, moka cache). Depends on and re-exports `dependable-core`, so external consumers (e.g. an IDE) need only this crate. +- **`dependable-report`** (`crates/dependable-report`) — the report and policy layer: + HTML rendering (minijinja), SARIF v2.1.0, the SPDX license evaluator, and the + `[policy]` engine. Pure like the core: it takes the finished report model and + returns bytes, so it holds no IO of its own. A library only — it ships no binary. - **`dependable-tui`** (`crates/dependable-tui`) — the interactive terminal UI (ratatui). Holds no IO of its own: it drives `dependable-fetch`. Its `App` state machine is free of both IO and ratatui, which is what makes navigation, search, @@ -21,6 +28,9 @@ for what is deferred and why. `tree` command renders the workspace dependency graph offline via `dependable_fetch::build_workspace_graph` (no `Checker`, no network). +Five crates, published to crates.io in dependency order: +core → report → fetch → tui → dependable. + ## Quality Validate changes before committing: diff --git a/Cargo.lock b/Cargo.lock index e1eb3f4..4acd44d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -556,6 +556,7 @@ dependencies = [ "owo-colors", "serde", "serde_json", + "tempfile", "tokio", "toml_edit", "tracing", diff --git a/README.md b/README.md index 8c03cb2..5c7f681 100644 --- a/README.md +++ b/README.md @@ -550,6 +550,41 @@ skipped and transitive deps are never fetched), and the public API is forward-compatible: enums are `#[non_exhaustive]` and the registry layer routes per ecosystem, so future registries (npm, PyPI, Go, …) are additive. +## Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | The run completed and no armed gate was tripped. | +| `1` | A gate was tripped: `--fail-on` matched, or a `[policy]` rule was violated. | +| `2` | The tool could not do the job — a config it cannot read, a policy it cannot enforce, or **a gate it cannot answer**. | + +That last case matters for CI. If you arm `--fail-on` (or a `[policy]` severity +rule) and the run cannot establish what the gate needs — the vulnerability scan did +not complete, or a registry never answered — `dependable` exits `2` and says so, +rather than exiting `0`. A gate that reports success on the run it could not perform +is worse than no gate at all. With no gate armed, an unreachable registry is still +reported per dependency and the run exits `0`, because nothing was promised. + +A registry that *did* answer is a different thing. A package it reports as +non-existent — an unpublished internal package, one served by a registry this run +does not route to, a deleted package — is a permanent per-dependency fact, not a +failure of the run: it appears in the table and in `--format json`, the run says on +stderr how many were skipped, and the exit code is `0`. The same goes for a +dependency whose declared version this run could not read, which is reported +`undetermined`: `--fail-on vulnerable` promises something about vulnerabilities and +`--fail-on outdated` something about staleness, and neither is a promise that every +constraint was parseable. `--fail-on any` is that promise, and it fails on both. + +Arming a gate the run cannot enforce is a configuration error, caught before any +network access: `--fail-on vulnerable` with vulnerability scanning switched off +(`--no-vuln`, or `[vulnerability] enabled = false`) exits `2`, as a `[policy]` +severity rule already did. Every advisory list would be empty and the gate could +never fail — a gate is either enforceable or it is a mistake. + +`.dependable.toml` is validated: an unknown key or a wrong-typed value is an error, +not a silent fallback to defaults. One mistyped character used to reset +`[global] fail_on` to `none` and disarm the gate with nothing on stderr. + ## Development | Command | Description | @@ -568,7 +603,10 @@ per ecosystem, so future registries (npm, PyPI, Go, …) are additive. - **`dependable-fetch`** — the high-level library: `Checker` ties parsing to async registry + OSV fetching and caching. The public end-to-end entry point for other tools; re-exports the core types so consumers need only this crate. -- **`dependable`** — the CLI binary; a thin wrapper over `dependable-fetch`. +- **`dependable-report`** — HTML, SARIF, and the `[policy]` engine, over the finished + report model. A library only. +- **`dependable-tui`** — the interactive terminal UI (ratatui), driving `dependable-fetch`. +- **`dependable`** — the CLI binary; a thin wrapper over the crates above. ## Git Hooks @@ -583,7 +621,7 @@ is a composite action that installs the released binary and runs the check: ```yaml - uses: actions/checkout@v4 -- uses: getkono/dependable/.github/actions/dependable-check@v0.1.4 +- uses: getkono/dependable/.github/actions/dependable-check@v0.1.3 with: fail-on: vulnerable ``` @@ -609,7 +647,7 @@ both the annotations and the job summary. Annotations go to **stderr**, so stdout. This repository's own GitHub Actions workflow runs format checks, linting, and -tests on pushes to `main` and on pull requests, plus a coverage job that uploads +tests on pushes to `master` and on pull requests, plus a coverage job that uploads an `lcov.info` artifact. ## License diff --git a/crates/dependable-core/src/graph.rs b/crates/dependable-core/src/graph.rs index 9a49676..0dcab67 100644 --- a/crates/dependable-core/src/graph.rs +++ b/crates/dependable-core/src/graph.rs @@ -177,6 +177,57 @@ pub struct WalkOptions<'a> { /// A node this rejects is skipped along with its whole subtree, but its /// siblings keep their slot indices. pub include: Option>, + /// Maximum number of node appearances to emit before the walk stops. `None` = + /// unlimited, which is only safe when the caller bounds the walk some other way. + /// + /// The walk enumerates *paths*, not nodes. With [`dedupe`](Self::dedupe) on, each + /// node expands at most once and the count is bounded by the graph; with it off, + /// a dependency graph shaped like a ladder has a number of distinct simple paths + /// exponential in its size, and a real lockfile does not finish. The budget is what + /// makes an unbounded walk terminate rather than appear to hang. + pub max_visits: Option, +} + +/// Default appearance budget for one walk — far above any real dependency forest, and +/// far below the point at which an exponential walk stops looking like a hang. +/// +/// # Where the number comes from +/// The two bounds it has to sit between. Above: the largest lockfiles in the wild run +/// to a few tens of thousands of packages, and a deduped walk emits one appearance per +/// edge, so a forest an order of magnitude past the worst real case still finishes +/// whole — no honest tree is ever cut by this. Below: an appearance is a few pointer +/// derefs and a `Vec` push, so a million of them is well under a second, which keeps a +/// walk that *is* exponential looking like a bounded operation rather than a hang. +/// A user cannot override it: raising it only buys a longer wait before the same +/// prefix, and the prefix is reported as one ([`Tree::truncated`]). +pub const DEFAULT_MAX_VISITS: usize = 1_000_000; + +/// Hard recursion ceiling, independent of [`WalkOptions::max_depth`]. +/// +/// The walk is recursive, so depth costs stack. No real dependency chain approaches +/// this; a cyclic graph cannot reach it (back-edges are cut), but a synthesized or +/// corrupt lockfile can, and overflowing the stack aborts the process. +/// +/// # Where the number comes from +/// A stack budget, not a graph property. One frame here carries the node index, the +/// child iterator, and the path bookkeeping — on the order of a hundred bytes — so 512 +/// frames is tens of kilobytes, comfortably inside the smallest stack this walk runs +/// on (a non-main thread's default, which is where a checker's tasks execute). Real +/// dependency chains are an order of magnitude shorter: the deepest transitive chains +/// observed in published lockfiles are in the low tens. Overrunning the ceiling is +/// therefore evidence of a corrupt or synthesized lockfile, and the walk reports the +/// prefix ([`Tree::truncated`]) rather than aborting the process. Not user-tunable, +/// because raising it trades a reported prefix for a stack overflow. +pub const MAX_WALK_DEPTH: usize = 512; + +/// What one [`DependencyGraph::walk`] did. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct WalkStats { + /// Node appearances emitted. + pub visits: usize, + /// Whether the walk stopped early on [`WalkOptions::max_visits`] or + /// [`MAX_WALK_DEPTH`], so the result is a prefix of the forest rather than all of it. + pub truncated: bool, } impl Default for WalkOptions<'_> { @@ -188,6 +239,7 @@ impl Default for WalkOptions<'_> { prefix: &[], expand: None, include: None, + max_visits: Some(DEFAULT_MAX_VISITS), } } } @@ -197,6 +249,12 @@ impl Default for WalkOptions<'_> { pub struct Tree { /// The root nodes, each with their expanded subtree. pub roots: Vec, + /// Whether the walk stopped on its appearance budget or depth ceiling, so this is a + /// prefix of the forest rather than all of it. + /// + /// A truncated tree that does not say so is a wrong answer wearing a complete one's + /// clothes; a renderer is expected to tell the reader. + pub truncated: bool, } /// One node in a [`Tree`], referencing a graph node by index. @@ -308,10 +366,14 @@ impl DependencyGraph { &self.nodes } - /// The direct dependencies of node `idx`. + /// The direct dependencies of node `idx`, or an empty slice if `idx` is not a node + /// in this graph. + /// + /// Indexing directly would panic, and this is public API on a library whose stated + /// audience is other tools holding indices they got from somewhere else. #[must_use] pub fn deps_of(&self, idx: usize) -> &[usize] { - &self.edges[idx] + self.edges.get(idx).map_or(&[], Vec::as_slice) } /// The root node indices. @@ -361,9 +423,10 @@ impl DependencyGraph { ..WalkOptions::default() }; let mut builder = TreeBuilder::default(); - self.walk(&walk, &mut builder); + let stats = self.walk(&walk, &mut builder); Tree { roots: builder.roots, + truncated: stats.truncated, } } @@ -377,19 +440,22 @@ impl DependencyGraph { /// Only subtrees that [`WalkOptions::expand`] admits are descended into, so /// a caller showing a mostly-closed tree pays for what it shows rather than /// for what the graph holds. - pub fn walk(&self, opts: &WalkOptions<'_>, visitor: &mut dyn Visitor) { + pub fn walk(&self, opts: &WalkOptions<'_>, visitor: &mut dyn Visitor) -> WalkStats { let mut walker = Walker { graph: self, opts, path: opts.prefix.to_vec(), expanded: HashSet::new(), on_path: HashSet::new(), + budget: opts.max_visits.unwrap_or(usize::MAX), + stats: WalkStats::default(), }; for (slot, &root) in self.roots.iter().enumerate() { walker.path.push(slot); walker.visit(root, 0, visitor); walker.path.pop(); } + walker.stats } } @@ -403,15 +469,26 @@ struct Walker<'a> { expanded: HashSet, /// Nodes on the path from a root to here, for cutting cycles. on_path: HashSet, + /// Appearances still allowed before the walk stops. + budget: usize, + stats: WalkStats, } impl Walker<'_> { fn visit(&mut self, node: usize, depth: usize, visitor: &mut dyn Visitor) { + // Both guards sit above `visitor.enter` so that a stopped walk never leaves an + // `enter` without its `leave` — the builder's stack discipline depends on it. + if self.budget == 0 || depth >= MAX_WALK_DEPTH { + self.stats.truncated = true; + return; + } if let Some(include) = self.opts.include && !include(&self.path) { return; } + self.budget -= 1; + self.stats.visits += 1; // Copied out of `self` so the child list stays borrowed from the graph // rather than from the walker, which the recursion below borrows anew. let deps: &[usize] = &self.graph.edges[node]; @@ -835,7 +912,8 @@ source = "registry+https://x" flatten( &g, &Tree { - roots: vec![b_root.clone()] + roots: vec![b_root.clone()], + truncated: false, } ), vec![("b", "0.1.0", false), ("serde", "1.0.0", false)], @@ -1085,4 +1163,107 @@ source = "registry+https://x" ] ); } + + /// The walk recurses, so depth costs stack. A chain longer than the ceiling is + /// truncated rather than allowed to overflow and abort the process. + #[test] + fn a_very_deep_chain_terminates_and_says_it_was_truncated() { + let depth = MAX_WALK_DEPTH + 50; + let mut nodes = Vec::new(); + let mut edges: Vec> = Vec::new(); + for n in 0..depth { + nodes.push(Node { + name: format!("c{n}"), + version: "1.0.0".to_string(), + kind: NodeKind::Registry, + }); + edges.push(if n + 1 < depth { vec![n + 1] } else { vec![] }); + } + let g = DependencyGraph { + root_slots: std::iter::once((0, 0)).collect(), + nodes, + edges, + roots: vec![0], + }; + let tree = g.tree(&TreeOptions::default()); + assert!( + tree.truncated, + "a chain past the ceiling must report truncation" + ); + } + + /// With `dedupe` off the walk enumerates simple paths, and a ladder-shaped graph has + /// exponentially many. This used to run until it was killed. + #[test] + fn an_exponential_walk_is_bounded_by_its_budget() { + // 40 layers of two nodes each: 2^40 distinct root-to-leaf paths. + let layers = 40usize; + let mut nodes = Vec::new(); + let mut edges: Vec> = Vec::new(); + for layer in 0..layers { + for side in 0..2 { + nodes.push(Node { + name: format!("n{layer}_{side}"), + version: "1.0.0".to_string(), + kind: NodeKind::Registry, + }); + let next = (layer + 1) * 2; + edges.push(if layer + 1 < layers { + vec![next, next + 1] + } else { + vec![] + }); + } + } + let g = DependencyGraph { + root_slots: std::iter::once((0, 0)).collect(), + nodes, + edges, + roots: vec![0], + }; + let opts = WalkOptions { + dedupe: false, + collapse_roots: false, + max_visits: Some(10_000), + ..WalkOptions::default() + }; + let mut builder = TreeBuilder::default(); + let stats = g.walk(&opts, &mut builder); + assert!(stats.truncated); + assert!( + stats.visits <= 10_000, + "emitted {} appearances", + stats.visits + ); + } + + /// A node index from outside this graph must not panic the library. + #[test] + fn deps_of_an_unknown_index_is_empty() { + let g = DependencyGraph { + root_slots: std::collections::HashMap::new(), + nodes: Vec::new(), + edges: Vec::new(), + roots: Vec::new(), + }; + assert!(g.deps_of(0).is_empty()); + assert!(g.deps_of(usize::MAX).is_empty()); + } + + /// A self-edge is a cycle of length one; it is cut like any other back-edge. + #[test] + fn a_self_edge_terminates() { + let g = DependencyGraph { + root_slots: std::iter::once((0, 0)).collect(), + nodes: vec![Node { + name: "a".to_string(), + version: "1.0.0".to_string(), + kind: NodeKind::Registry, + }], + edges: vec![vec![0]], + roots: vec![0], + }; + let tree = g.tree(&TreeOptions::default()); + assert!(!tree.truncated, "a self-edge is cut, not budgeted away"); + } } diff --git a/crates/dependable-core/src/item.rs b/crates/dependable-core/src/item.rs index 89dafc6..6c46580 100644 --- a/crates/dependable-core/src/item.rs +++ b/crates/dependable-core/src/item.rs @@ -74,9 +74,16 @@ impl Item { /// on a line worth reporting, but records a *zero-width* span, and writing a version /// into it would produce `numpy1.5.0`. `--fix` gates on this; the reporters, which /// only ever read the line, gate on `has_position`. + /// + /// The width is checked directly rather than inferred from the constraint. A parser + /// that knows its offsets have drifted — a JSON value whose escapes make the source + /// span and the decoded value disagree — collapses the span to signal exactly that, + /// and the constraint it parsed is still perfectly non-empty. #[must_use] pub fn is_rewritable(&self) -> bool { - self.has_position() && !self.version_constraint.is_empty() + self.has_position() + && !self.version_constraint.is_empty() + && self.version_col_end > self.version_col_start } } @@ -114,6 +121,12 @@ pub enum DependencyKind { /// A transitive dependency the manifest records explicitly (`go.mod`'s /// `// indirect`). Not a direct dependency of the module. Indirect, + /// A version this manifest forces onto the tree regardless of what asked for it — + /// npm `overrides`, Yarn `resolutions`, `pnpm.overrides`. + /// + /// Worth reporting precisely because it is how a vulnerable transitive dependency is + /// remediated: the pin is the fix, and a stale pin is the fix having quietly expired. + Override, } impl DependencyKind { @@ -128,15 +141,20 @@ impl DependencyKind { Self::Peer => "peer", Self::Workspace => "workspace", Self::Indirect => "indirect", + Self::Override => "override", } } /// Whether this is a dependency the package itself pulls in — everything except a - /// central declaration ([`Workspace`](Self::Workspace)) and a recorded transitive - /// ([`Indirect`](Self::Indirect)). + /// central declaration ([`Workspace`](Self::Workspace)), a recorded transitive + /// ([`Indirect`](Self::Indirect)), and a forced version ([`Override`](Self::Override)). + /// + /// An override names a package somewhere in the tree, not one this manifest depends + /// on, so counting it as direct would inflate the inventory with packages the + /// manifest never asked for. #[must_use] pub fn is_direct(self) -> bool { - !matches!(self, Self::Workspace | Self::Indirect) + !matches!(self, Self::Workspace | Self::Indirect | Self::Override) } } @@ -162,6 +180,15 @@ pub enum PackageSource { /// does need IO, and is [`resolve_workspace_inheritance`](crate::resolve_workspace_inheritance) /// applied by the caller that has the root in hand. Inherited, + /// A real registry package whose declared version this manifest does not, on its + /// own, resolve to a range: an npm `"$name"` override naming a dependency the + /// manifest never declares. + /// + /// Not [`Local`](Self::Local) — the package is published and the entry is real — + /// and not a parse error: the manifest is valid, its intent simply cannot be read + /// from what is written. Nothing is fetched for it, and the checker reports it as + /// [`DependencyStatus::Undetermined`](crate::result::DependencyStatus::Undetermined). + Unresolved, } #[cfg(test)] diff --git a/crates/dependable-core/src/lib.rs b/crates/dependable-core/src/lib.rs index a31f285..3f30f33 100644 --- a/crates/dependable-core/src/lib.rs +++ b/crates/dependable-core/src/lib.rs @@ -19,7 +19,7 @@ pub use ecosystem::Ecosystem; pub use error::ParseError; pub use graph::{ DependencyGraph, Node, NodeKind, PathPredicate, Placement, Tree, TreeNode, TreeOptions, Visit, - Visitor, WalkOptions, + Visitor, WalkOptions, WalkStats, }; pub use item::{DependencyKind, Item, PackageSource}; pub use lockfiles::{ @@ -41,5 +41,8 @@ pub use parsers::{ RequirementsTxtParser, WorkspaceDecl, parse, parse_cargo_config, parse_package_manifest, parse_package_name, parse_project, parse_workspace, resolve_workspace_inheritance, }; -pub use result::{CheckResult, DependencyStatus}; -pub use semver::{Evaluation, UnstableFilter, check_version, is_prerelease, to_semver_constraint}; +pub use result::{CheckResult, DependencyStatus, ErrorOrigin}; +pub use semver::{ + Evaluation, UnstableFilter, check_version, check_version_for, is_prerelease, + to_semver_constraint, try_to_semver_constraint, +}; diff --git a/crates/dependable-core/src/lockfiles/package_lock_graph.rs b/crates/dependable-core/src/lockfiles/package_lock_graph.rs index 0e6b237..e064885 100644 --- a/crates/dependable-core/src/lockfiles/package_lock_graph.rs +++ b/crates/dependable-core/src/lockfiles/package_lock_graph.rs @@ -43,13 +43,24 @@ struct Entry { /// key is a local workspace package. /// /// # Errors -/// Never fails: a lockfile that does not parse yields no packages, which callers -/// treat as "no resolved graph" rather than an error that hides the project. +/// Returns [`ParseError::Structural`] for a lockfile v1 (npm 6) document, whose graph +/// lives under a top-level `dependencies` tree this parser does not read. Returning an +/// empty graph for one made "this format is not supported" indistinguishable from "this +/// project has no dependencies"; the caller reports the former and falls back. pub fn parse_package_lock_graph(content: &str) -> Result { let mut entries: HashMap = HashMap::new(); let mut order: Vec = Vec::new(); + // A v1 lockfile records resolved versions under `dependencies..version`; v2/v3 + // keep that tree too, but always alongside a `packages` object. + let mut legacy_versions = false; for entry in scan_strings(content) { + let Some((section, _key, rest)) = split_path(&entry.path) else { + continue; + }; + if section == "dependencies" && matches!(rest, [field] if field == "version") { + legacy_versions = true; + } let Some(("packages", key, rest)) = split_path(&entry.path) else { continue; }; @@ -69,6 +80,13 @@ pub fn parse_package_lock_graph(content: &str) -> Result = order .iter() @@ -104,6 +122,7 @@ pub fn parse_package_lock_graph(content: &str) -> Result Option { { return Some(resolved.to_owned()); } + // A workspace link stub lives under `node_modules/` but is the member, not an + // install from the registry; calling it one puts a versionless npm package in the + // graph beside the real member. + if is_link_stub(key, entry) { + return None; + } // The root ("") and workspace packages ("packages/app") are local. key.contains("node_modules/").then(|| NPM_SOURCE.to_owned()) } +/// Whether this entry is npm's `node_modules/` stub for a workspace member. +/// +/// npm records a member twice: the stub, whose `resolved` is the member's path in the +/// repository and which carries no version of its own, and the member itself under that +/// path. `link: true` marks it, but the scan yields only string values and that is a +/// boolean — a relative `resolved` with no version identifies the same thing. +fn is_link_stub(key: &str, entry: &Entry) -> bool { + if !key.contains("node_modules/") || entry.version.is_some() { + return false; + } + entry.resolved.as_deref().is_some_and(|resolved| { + !resolved.contains("://") && !resolved.starts_with("git+") && !resolved.is_empty() + }) +} + +/// Resolve a link stub to the workspace member it points at. +/// +/// The stub declares no dependencies, so an edge that stops there severs the member's +/// entire subtree from the graph — and because the stub has no version, the edge was +/// emitted as a bare name, which resolves to whichever candidate came first in document +/// order. That is the stub, every time: npm writes `node_modules/*` before `packages/*`. +fn follow_link( + i: usize, + order: &[String], + entries: &HashMap, + index: &HashMap<&str, usize>, +) -> usize { + let key = &order[i]; + let Some(entry) = entries.get(key) else { + return i; + }; + if !is_link_stub(key, entry) { + return i; + } + let Some(resolved) = entry.resolved.as_deref() else { + return i; + }; + let path = resolved.trim_start_matches("./"); + index.get(path).copied().unwrap_or(i) +} + /// The package name implied by a `packages` key, for entries that declare none. fn package_name(key: &str) -> Option { if key.is_empty() { @@ -360,4 +426,50 @@ mod tests { let resolved = parse_package_lock_graph(r#"{"lockfileVersion": 1}"#).unwrap(); assert!(resolved.packages.is_empty()); } + + /// A v1 lockfile keeps its graph somewhere this parser does not read. Reporting an + /// empty graph made "unsupported format" look exactly like "no dependencies", so the + /// caller had nothing to tell the user and nothing to fall back from. + #[test] + fn a_v1_lockfile_is_reported_as_unsupported() { + let lock = r#"{ + "name": "app", + "lockfileVersion": 1, + "dependencies": { + "lodash": { "version": "4.17.21", "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz" } + } +}"#; + assert!(parse_package_lock_graph(lock).is_err()); + } + + /// npm writes a workspace member twice: a versionless stub under `node_modules/` and + /// the member itself. Edges used to stop at the stub — which has no dependencies — + /// so the member's whole subtree vanished from the graph. + #[test] + fn a_workspace_link_stub_resolves_to_the_member() { + let lock = r#"{ + "lockfileVersion": 3, + "packages": { + "": { "name": "root", "version": "1.0.0", "dependencies": { "app": "*" } }, + "node_modules/app": { "resolved": "packages/app", "link": true }, + "node_modules/lodash": { "version": "4.17.21" }, + "packages/app": { "name": "app", "version": "2.0.0", "dependencies": { "lodash": "^4.0.0" } } + } +}"#; + let resolved = parse_package_lock_graph(lock).expect("v3 lockfile"); + let root = resolved + .packages + .iter() + .find(|p| p.name == "root") + .expect("root"); + // The root's edge must reach the member at its real version, not the stub. + assert_eq!(root.dependencies, vec!["app 2.0.0".to_string()]); + + let app = resolved + .packages + .iter() + .find(|p| p.name == "app" && p.version == "2.0.0") + .expect("member"); + assert_eq!(app.dependencies, vec!["lodash 4.17.21".to_string()]); + } } diff --git a/crates/dependable-core/src/parsers/cargo_toml.rs b/crates/dependable-core/src/parsers/cargo_toml.rs index 338c9e6..cc1b06d 100644 --- a/crates/dependable-core/src/parsers/cargo_toml.rs +++ b/crates/dependable-core/src/parsers/cargo_toml.rs @@ -37,6 +37,26 @@ impl Parser for CargoTomlParser { } } + // [target..dependencies] and its dev/build siblings. + // + // A cfg-gated table is an ordinary dependency table that Cargo only applies to + // some targets; the crates in it are fetched, resolved, and vulnerable exactly + // like any other. Skipping them reported a manifest whose only dependency sits + // under `[target.'cfg(unix)'.dependencies]` as having none at all — silence that + // reads as a clean bill of health. + if let Some(targets) = root.get("target").and_then(|i| i.as_table_like()) { + for (_predicate, entry) in targets.iter() { + let Some(entry) = entry.as_table_like() else { + continue; + }; + for &(section, kind) in DEP_SECTIONS { + if let Some(table) = entry.get(section).and_then(|i| i.as_table_like()) { + collect_dependencies(table, kind, &starts, &mut items); + } + } + } + } + // [workspace.dependencies] if let Some(deps) = root .get("workspace") @@ -435,4 +455,44 @@ mod tests { // No `[registries]` table at all. assert!(parse_cargo_config("[net]\nretry = 2\n", None).is_empty()); } + + /// A cfg-gated table is a real dependency table. The manifest below used to parse to + /// zero items, so `check` printed "nothing to check" for a project with dependencies. + #[test] + fn target_gated_dependency_tables_are_collected() { + let content = concat!( + "[dependencies]\n", + "serde = \"1\"\n\n", + "[target.'cfg(unix)'.dependencies]\n", + "nix = \"0.29\"\n\n", + "[target.'cfg(windows)'.dependencies]\n", + "windows-sys = \"0.59\"\n\n", + "[target.x86_64-pc-windows-msvc.dev-dependencies]\n", + "winapi = \"0.3\"\n\n", + "[target.'cfg(target_os = \"linux\")'.build-dependencies]\n", + "cc = \"1\"\n", + ); + let m = CargoTomlParser.parse(content).expect("valid TOML"); + + for name in ["serde", "nix", "windows-sys", "winapi", "cc"] { + let it = find(&m, name); + assert_eq!(it.name, name); + } + assert_eq!(find(&m, "winapi").kind, DependencyKind::Dev); + assert_eq!(find(&m, "cc").kind, DependencyKind::Build); + assert_eq!(find(&m, "nix").kind, DependencyKind::Normal); + } + + /// The recorded span has to point at the cfg-gated line, not at a top-level one, or + /// `--fix` would rewrite the wrong dependency. + #[test] + fn a_target_gated_version_records_its_own_position() { + let content = + "[dependencies]\nserde = \"1\"\n\n[target.'cfg(unix)'.dependencies]\nnix = \"0.29\"\n"; + let m = CargoTomlParser.parse(content).expect("valid TOML"); + let nix = find(&m, "nix"); + assert!(nix.is_rewritable()); + let line = content.lines().nth(nix.version_line).unwrap(); + assert_eq!(&line[nix.version_col_start..nix.version_col_end], "0.29"); + } } diff --git a/crates/dependable-core/src/parsers/deno_json.rs b/crates/dependable-core/src/parsers/deno_json.rs index 71ab1df..fc0da61 100644 --- a/crates/dependable-core/src/parsers/deno_json.rs +++ b/crates/dependable-core/src/parsers/deno_json.rs @@ -57,13 +57,20 @@ fn build_item(entry: &JsonStringValue, starts: &[usize]) -> Option { let global_start = entry.content_start + version_offset; let (line, col_start) = offset_to_line_col(starts, global_start); + // See `package_json::build_item`: an escaped value's decoded offsets do not map onto + // the source span, so the span is withheld and the import is reported but not fixed. + let col_end = if entry.escaped { + col_start + } else { + col_start + entry.content_end.saturating_sub(global_start) + }; Some(Item { name, version_constraint: constraint, source, version_line: line, version_col_start: col_start, - version_col_end: col_start + entry.content_end.saturating_sub(global_start), + version_col_end: col_end, registry: None, locked_version: None, // `imports`/`scopes` are one flat map with no dev/build distinction to read. diff --git a/crates/dependable-core/src/parsers/json_scan.rs b/crates/dependable-core/src/parsers/json_scan.rs index 2eae9a5..b2dd2c6 100644 --- a/crates/dependable-core/src/parsers/json_scan.rs +++ b/crates/dependable-core/src/parsers/json_scan.rs @@ -6,6 +6,10 @@ //! comments — neither of which `serde_json` provides. This single pass covers all //! of it: object keys build the path, array elements use their index, and `//` //! and `/* */` comments are skipped. +//! +//! The scan is total: every reachable input either yields values or stops. Malformed +//! input yields whatever was scanned up to the error, and never panics or loops — +//! manifests arrive half-written from editors often enough that both were reachable. /// A string value found in a JSON(C) document. #[derive(Debug, Clone, PartialEq, Eq)] @@ -18,6 +22,15 @@ pub struct JsonStringValue { pub content_start: usize, /// Byte offset just past the last content byte (the closing quote). pub content_end: usize, + /// Whether the raw text carried backslash escapes, so [`value`](Self::value) is + /// shorter than — and not byte-aligned with — the `content_start..content_end` + /// span. + /// + /// Callers map offsets found in `value` back onto the source span. That mapping is + /// only valid when the two agree byte for byte, so an escaped string must not be + /// rewritten in place; the flag is what lets a caller decline rather than splice at + /// an offset that has drifted. + pub escaped: bool, } /// Scan JSON or JSONC `src`, returning every string value with its path, in @@ -35,6 +48,14 @@ pub fn scan_strings(src: &str) -> Vec { scanner.out } +/// One parsed string: its unescaped content, its raw span, and whether the two differ. +struct ParsedString { + value: String, + start: usize, + end: usize, + escaped: bool, +} + struct Scanner<'a> { bytes: &'a [u8], src: &'a str, @@ -43,20 +64,28 @@ struct Scanner<'a> { } impl Scanner<'_> { + /// The bytes from the cursor on, or empty once the cursor has passed the end. + /// + /// The cursor is advanced past a delimiter by several callers, so it can sit one + /// beyond the input; slicing `bytes[i..]` directly panics there. + fn rest(&self) -> &[u8] { + self.bytes.get(self.i..).unwrap_or(&[]) + } + /// Skip whitespace and `//` line / `/* */` block comments. fn skip_trivia(&mut self) { loop { while self.i < self.bytes.len() && self.bytes[self.i].is_ascii_whitespace() { self.i += 1; } - if self.bytes[self.i..].starts_with(b"//") { + if self.rest().starts_with(b"//") { self.i += 2; while self.i < self.bytes.len() && self.bytes[self.i] != b'\n' { self.i += 1; } - } else if self.bytes[self.i..].starts_with(b"/*") { + } else if self.rest().starts_with(b"/*") { self.i += 2; - while self.i < self.bytes.len() && !self.bytes[self.i..].starts_with(b"*/") { + while self.i < self.bytes.len() && !self.rest().starts_with(b"*/") { self.i += 1; } self.i = (self.i + 2).min(self.bytes.len()); @@ -73,12 +102,13 @@ impl Scanner<'_> { Some(b'{') => self.parse_object(path), Some(b'[') => self.parse_array(path), Some(b'"') => { - if let Some((value, start, end)) = self.parse_string() { + if let Some(s) = self.parse_string() { self.out.push(JsonStringValue { path: path.to_vec(), - value, - content_start: start, - content_end: end, + value: s.value, + content_start: s.start, + content_end: s.end, + escaped: s.escaped, }); } } @@ -91,32 +121,44 @@ impl Scanner<'_> { loop { self.skip_trivia(); match self.bytes.get(self.i) { - Some(b'}') | None => { + Some(b'}') => { self.i += 1; return; } + // End of input: stop *without* advancing. Stepping past the end here is + // what left the cursor at `len + 1`, so the enclosing container's next + // `skip_trivia` sliced out of range. + None => return, Some(b',') => { self.i += 1; continue; } Some(b'"') => {} + // A stray `]` closes the array we are nested in, not this object; leave + // it for that frame rather than consuming it. + Some(b']') => return, _ => { - // Unexpected; bail to avoid looping forever. + // Unexpected; skip it rather than looping forever. self.i += 1; continue; } } - let Some((key, ..)) = self.parse_string() else { + let before = self.i; + let Some(key) = self.parse_string() else { return; }; self.skip_trivia(); if self.bytes.get(self.i) != Some(&b':') { + // Guarantee progress even if `parse_string` consumed nothing. + if self.i == before { + self.i += 1; + } continue; } self.i += 1; // consume ':' self.skip_trivia(); let mut child = path.to_vec(); - child.push(key); + child.push(key.value); self.parse_value(&child); } } @@ -127,27 +169,37 @@ impl Scanner<'_> { loop { self.skip_trivia(); match self.bytes.get(self.i) { - Some(b']') | None => { + Some(b']') => { self.i += 1; return; } + None => return, Some(b',') => { self.i += 1; continue; } + // A stray `}` closes the object we are nested in. `parse_value` would + // route it to `skip_scalar`, which breaks on `}` without advancing — + // an infinite loop. Hand it back to the enclosing frame instead. + Some(b'}') => return, _ => {} } let mut child = path.to_vec(); child.push(index.to_string()); + let before = self.i; self.parse_value(&child); + // Nothing below is allowed to stall: a scalar that begins with a byte + // `skip_scalar` treats as a terminator would otherwise spin here forever. + if self.i == before { + self.i += 1; + } index += 1; } } - /// Parse a string at the cursor (which must be on the opening quote), - /// returning `(content, content_start, content_end)` and leaving the cursor - /// just past the closing quote. - fn parse_string(&mut self) -> Option<(String, usize, usize)> { + /// Parse a string at the cursor (which must be on the opening quote), leaving the + /// cursor just past the closing quote. + fn parse_string(&mut self) -> Option { debug_assert_eq!(self.bytes.get(self.i), Some(&b'"')); let content_start = self.i + 1; let mut j = content_start; @@ -156,17 +208,28 @@ impl Scanner<'_> { match self.bytes[j] { b'\\' => { escaped = true; + // A trailing backslash, or one before a multi-byte character, must + // not carry `j` past the end or onto a continuation byte — the + // `src[..j]` slice below would panic on either. j += 2; + while j < self.bytes.len() && !self.src.is_char_boundary(j) { + j += 1; + } } b'"' => { - let raw = &self.src[content_start..j]; + let raw = self.src.get(content_start..j)?; let value = if escaped { unescape(raw) } else { raw.to_string() }; self.i = j + 1; - return Some((value, content_start, j)); + return Some(ParsedString { + value, + start: content_start, + end: j, + escaped, + }); } _ => j += 1, } @@ -187,27 +250,77 @@ impl Scanner<'_> { } } -/// Unescape the common JSON string escapes (enough for package names, versions, -/// and URLs). +/// Unescape a JSON string body. +/// +/// `\uXXXX` is decoded, including surrogate pairs, because npm and Deno both emit +/// escaped scoped names (`@scope/pkg`) and a mangled key silently fails to match +/// the dependency it names. fn unescape(raw: &str) -> String { let mut out = String::with_capacity(raw.len()); - let mut chars = raw.chars(); + let mut chars = raw.chars().peekable(); while let Some(c) = chars.next() { - if c == '\\' { - match chars.next() { - Some('n') => out.push('\n'), - Some('t') => out.push('\t'), - Some('r') => out.push('\r'), - Some(other) => out.push(other), // \" \\ \/ and the rest - None => {} - } - } else { + if c != '\\' { out.push(c); + continue; + } + match chars.next() { + Some('n') => out.push('\n'), + Some('t') => out.push('\t'), + Some('r') => out.push('\r'), + Some('b') => out.push('\u{8}'), + Some('f') => out.push('\u{c}'), + Some('u') => match take_hex4(&mut chars) { + // A high surrogate is only meaningful paired with the low surrogate + // that follows it; either half alone is not a character. + Some(hi @ 0xD800..=0xDBFF) => { + let low = take_surrogate_escape(&mut chars); + match low { + Some(lo @ 0xDC00..=0xDFFF) => { + let combined = 0x1_0000 + + ((u32::from(hi) - 0xD800) << 10) + + (u32::from(lo) - 0xDC00); + out.push(char::from_u32(combined).unwrap_or('\u{FFFD}')); + } + _ => out.push('\u{FFFD}'), + } + } + Some(unit) => out.push(char::from_u32(u32::from(unit)).unwrap_or('\u{FFFD}')), + None => out.push('\u{FFFD}'), + }, + Some(other) => out.push(other), // \" \\ \/ and the rest + None => {} } } out } +/// Read exactly four hex digits as a UTF-16 code unit, or `None` if they are not there. +fn take_hex4(chars: &mut std::iter::Peekable>) -> Option { + let mut unit: u16 = 0; + for _ in 0..4 { + let digit = chars.peek().copied()?.to_digit(16)?; + chars.next(); + unit = unit * 16 + u16::try_from(digit).ok()?; + } + Some(unit) +} + +/// Read a following `\uXXXX` escape, used to complete a surrogate pair. +fn take_surrogate_escape(chars: &mut std::iter::Peekable>) -> Option { + if chars.peek() != Some(&'\\') { + return None; + } + let mut lookahead = chars.clone(); + lookahead.next(); + if lookahead.peek() != Some(&'u') { + return None; + } + lookahead.next(); + let unit = take_hex4(&mut lookahead)?; + *chars = lookahead; + Some(unit) +} + #[cfg(test)] mod tests { use super::*; @@ -260,6 +373,121 @@ mod tests { assert!(paths(&values).contains(&(vec!["imports", "lodash"], "npm:lodash@^4"))); } + /// Truncated input used to carry the cursor to `len + 1`, and the next + /// `skip_trivia` sliced `bytes[len + 1..]` — a panic on a half-written manifest. + #[test] + fn truncated_containers_terminate_without_panicking() { + for src in [ + "{", + "[", + "[{", + r#"{"a":{"#, + r#"{"dependencies":{"#, + r#"{"dependencies":{"react""#, + r#"{"dependencies":{"react":"#, + r#"{"a":["#, + r#"{"a":"unterminated"#, + "", + ] { + let _ = scan_strings(src); + } + } + + /// A stray closer inside the other kind of container reached `skip_scalar`, which + /// breaks on `}` and `]` *without* advancing — the scan spun forever. + #[test] + fn stray_closers_terminate() { + for src in ["[}", "{]", r#"{"a":[}]}"#, r#"[}{]"#, "[[}]]", r#"{"a":}"#] { + let _ = scan_strings(src); + } + } + + /// Everything scanned *before* a malformation is still returned. The scanner + /// promises to stop at the error, not to resynchronise past it, so this pins the + /// half it does guarantee — truncation is the common case and the prefix is what a + /// half-written manifest has to offer. + #[test] + fn values_before_a_malformation_are_kept() { + let src = r#"{"dependencies": {"react": "^18.0.0"}, "bad": [}"#; + let values = scan_strings(src); + let got = paths(&values); + assert!( + got.contains(&(vec!["dependencies", "react"], "^18.0.0")), + "got {got:?}" + ); + } + + #[test] + fn decodes_unicode_escapes_including_surrogate_pairs() { + // `@scope/pkg` is how a scoped name arrives from generated manifests; the + // old unescape dropped the backslash and yielded `u0040scope/pkg`. + let src = r#"{"dependencies":{"@scope\/pkg":"^1.0.0"}}"#; + let values = scan_strings(src); + let got = paths(&values); + assert!( + got.contains(&(vec!["dependencies", "@scope/pkg"], "^1.0.0")), + "got {got:?}" + ); + + // A surrogate pair is one character, not two replacement chars. + let src = r#"{"a":"😀"}"#; + let v = scan_strings(src); + assert_eq!(v[0].value, "\u{1F600}"); + + // A lone high surrogate has no completion and must not panic. + let src = r#"{"a":"\uD83D"}"#; + let v = scan_strings(src); + assert_eq!(v[0].value, "\u{FFFD}"); + + // A truncated escape is not four hex digits. + let src = r#"{"a":"\u00"}"#; + let v = scan_strings(src); + assert_eq!(v[0].value, "\u{FFFD}"); + } + + /// An escaped value's decoded offsets do not line up with its source span, so the + /// scanner flags it and the parsers withhold the rewrite span. + #[test] + fn escapes_are_flagged_and_plain_values_are_not() { + let plain = scan_strings(r#"{"a":"^1.0.0"}"#); + assert!(!plain[0].escaped); + let escaped = scan_strings(r#"{"a":"^1.0.0\/x"}"#); + assert!(escaped[0].escaped); + } + + /// Every span the scanner reports must slice back out of the source. A backslash + /// before a multi-byte character used to land the cursor mid-UTF-8. + #[test] + fn spans_stay_on_character_boundaries_with_multibyte_content() { + for src in [ + r#"{"名前":"^1.0.0","déps":{"café":"~2.0"}}"#, + r#"{"a":"café","b":"naïve"}"#, + r#"{"a":"\é"}"#, + r#"{"a":"x\"#, + ] { + for v in scan_strings(src) { + assert!( + src.get(v.content_start..v.content_end).is_some(), + "span {}..{} does not slice {src:?}", + v.content_start, + v.content_end + ); + } + } + } + + /// Non-ASCII keys shift every later byte offset; the recorded span must follow the + /// bytes, not the characters. + #[test] + fn spans_are_byte_offsets_after_multibyte_keys() { + let src = r#"{"café": { "react": "^18.0.0" } }"#; + let v = scan_strings(src) + .into_iter() + .find(|v| v.path == ["café", "react"]) + .unwrap(); + assert_eq!(&src[v.content_start..v.content_end], "^18.0.0"); + } + #[test] fn handles_arrays_with_indices() { let src = r#"{ "project": { "dependencies": ["flask>=2.0", "requests"] } }"#; diff --git a/crates/dependable-core/src/parsers/package_json.rs b/crates/dependable-core/src/parsers/package_json.rs index b6dc778..40e3396 100644 --- a/crates/dependable-core/src/parsers/package_json.rs +++ b/crates/dependable-core/src/parsers/package_json.rs @@ -4,6 +4,8 @@ //! positions, then resolves npm version aliases. Only the version portion of an //! alias is recorded for `--fix` (so `npm:left-pad@1.3.0` rewrites just `1.3.0`). +use std::collections::HashMap; + use super::Parser; use super::json_scan::{JsonStringValue, scan_strings}; use super::position::{line_starts, offset_to_line_col}; @@ -26,10 +28,12 @@ pub struct PackageJsonParser; impl Parser for PackageJsonParser { fn parse(&self, content: &str) -> Result { let starts = line_starts(content); + let entries = scan_strings(content); + let declared = direct_dependencies(&entries); let mut items = Vec::new(); - for entry in scan_strings(content) { + for entry in &entries { if let Some((key, kind)) = dependency_key(&entry.path) { - items.push(build_item(key, kind, &entry, &starts)); + items.push(build_item(key, kind, entry, &starts, &declared)); } } Ok(ParsedManifest { @@ -40,6 +44,22 @@ impl Parser for PackageJsonParser { } } +/// The constraint each `*dependencies` section declares, by package name. +/// +/// The lookup table an npm `"$name"` override value is resolved against; a later +/// section wins, which is the order npm itself reads them in. +fn direct_dependencies(entries: &[JsonStringValue]) -> HashMap<&str, &str> { + entries + .iter() + .filter_map(|entry| match entry.path.as_slice() { + [section, dep] if DEP_SECTIONS.iter().any(|(name, _)| name == section) => { + Some((dep.as_str(), entry.value.as_str())) + } + _ => None, + }) + .collect() +} + /// Return the dependency name and its kind if `path` points at a dependency entry: a /// member of a `*dependencies`/`catalog` map, or a `catalogs..` entry. /// @@ -51,19 +71,185 @@ fn dependency_key(path: &[String]) -> Option<(&str, DependencyKind)> { .iter() .find(|(name, _)| name == section) .map(|(_, kind)| (dep.as_str(), *kind)) + .or_else(|| (section == "catalog").then_some((dep.as_str(), DependencyKind::Workspace))) .or_else(|| { - (section == "catalog").then_some((dep.as_str(), DependencyKind::Workspace)) + is_override_section(section) + .then(|| override_name(dep)) + .flatten() + .map(|name| (name, DependencyKind::Override)) }), [section, _catalog, dep] if section == "catalogs" => { Some((dep.as_str(), DependencyKind::Workspace)) } + // `pnpm.overrides`, and npm's nested form where an override is scoped to the + // parent that pulls the package in: `overrides.parent.child`. + [outer, inner, dep] if outer == "pnpm" && inner == "overrides" => { + override_name(dep).map(|name| (name, DependencyKind::Override)) + } + [section, _parent, dep] if is_override_section(section) => { + override_name(dep).map(|name| (name, DependencyKind::Override)) + } _ => None, } } +/// Whether `section` is one of the maps that force a version onto the resolved tree. +fn is_override_section(section: &str) -> bool { + matches!(section, "overrides" | "resolutions") +} + +/// The package an override key names. +/// +/// Yarn `resolutions` keys carry a path (`parent/child`, `**/lodash`) and npm's nested +/// form uses `"."` to mean "the parent entry itself", which names no new package. +/// +/// pnpm scopes an override to the parent that pulls the package in by joining the two +/// with `>`: `"foo@2>bar"` forces a version onto **bar**, not onto `foo`. The overridden +/// package is therefore the *last* `>`-separated segment; reading the first named an +/// unrelated package, which `--fix` then rewrote the pin to that package's latest +/// version. +/// +/// Both pnpm and Yarn also allow a **range** in the key, and a range can contain `>` +/// (`"lodash@>=1.0.0"`). Splitting on every `>` read that comparator as a parent +/// separator and renamed the package to a fragment of its own range — `=1.0.0` — so the +/// split is bounded by [`is_parent_separator`]. +fn override_name(key: &str) -> Option<&str> { + if key == "." { + return None; + } + let key = key.trim(); + // The parent selectors in front of the last *separating* `>` scope the override; + // only the segment after it names the package being overridden. + let key = match key + .char_indices() + .rfind(|&(at, c)| c == '>' && is_parent_separator(key, at)) + { + Some((at, _)) => key[at + 1..].trim(), + None => key, + }; + // Segment first, then strip the version. Doing it the other way round cut `**/@scope/pkg` + // at the scope's own `@`, because that `@` is not at the start of the *key*. + // + // The package is the last segment — or the last *two* when the one before it is a + // scope, because `@scope/pkg` is one name that happens to contain a slash. + let mut start = key.rfind('/').map_or(0, |slash| slash + 1); + if start > 0 { + let head = &key[..start - 1]; + if let Some(prev) = head.rfind('/') { + if key[prev + 1..].starts_with('@') { + start = prev + 1; + } + } else if head.starts_with('@') { + start = 0; + } + } + let name = &key[start..]; + // A trailing `@version` selects a range, not part of the name. A *leading* `@` is the + // scope, so the offset has to be past the start. + let name = match name.rfind('@') { + Some(at) if at > 0 => &name[..at], + _ => name, + }; + (!name.is_empty() && name != "*" && name != "**").then_some(name) +} + +/// Whether the `>` at byte offset `at` joins a parent selector to the package it scopes, +/// rather than being a comparator inside a version range. +/// +/// A comparator **opens** a clause, so it follows what can open one: the `@` that +/// introduces the range, a clause delimiter, or another operator character (`>=`, `||>`). +/// A separator follows the end of a name or of a version, which none of those can be. +/// That is the whole distinction between `"lodash@>=1.0.0"` (one package, a range) and +/// `"quux@1>bar"` (a parent and the package it scopes). +fn is_parent_separator(key: &str, at: usize) -> bool { + let Some(prev) = key[..at].chars().next_back() else { + // A leading `>` opens a range that names no parent at all. + return false; + }; + !prev.is_whitespace() + && !matches!( + prev, + '@' | ',' | '|' | '(' | '[' | '<' | '>' | '=' | '!' | '~' | '^' + ) +} + +/// An npm override value that references one of this manifest's own direct +/// dependencies rather than stating a range. +/// +/// `{"dependencies": {"semver": "^7.5.0"}, "overrides": {"semver": "$semver"}}` is +/// npm's documented way to say "force the version I already depend on". It is not a +/// constraint, and handing it to the version checker produced +/// `unparseable constraint: unexpected character '$'` on a perfectly valid manifest — +/// the same shape `csproj.rs` already declines to read as a version in `$(MSBuildProp)`. +/// +/// Returns the referenced package name, or `None` when the value is a reference that +/// names nothing usable — a bare `"$"`, or `"$a b"`. Those are still references and are +/// still not constraints: handing them to the checker produced the same +/// `unparseable constraint: unexpected character '$'` hard failure, so +/// [`build_item`] routes every `$`-prefixed override value that does not resolve to +/// [`PackageSource::Unresolved`], exactly as a `$name` naming an undeclared dependency +/// already was. +fn override_reference(value: &str) -> Option<&str> { + let name = value.strip_prefix('$')?; + (!name.is_empty() && !name.contains(char::is_whitespace)).then_some(name) +} + /// Build an [`Item`] for one dependency entry, resolving aliases and recording the /// version sub-span for `--fix`. -fn build_item(key: &str, kind: DependencyKind, entry: &JsonStringValue, starts: &[usize]) -> Item { +fn build_item( + key: &str, + kind: DependencyKind, + entry: &JsonStringValue, + starts: &[usize], + declared: &HashMap<&str, &str>, +) -> Item { + // Every `$`-prefixed override value is a reference, including the ones that name + // nothing (`"$"`, `"$a b"`). Guarding on a *well-formed* reference let those two fall + // through to the checker as literal constraints and hard-fail on the `$`. + if kind == DependencyKind::Override && entry.value.starts_with('$') { + // Resolved, the reference *is* the referenced dependency's constraint, so the + // entry is checked against exactly the version the manifest forces. Unresolved, + // the manifest names a dependency it does not declare: real package, unreadable + // version, and nothing to ask a registry for. + return match override_reference(&entry.value).and_then(|r| declared.get(r)) { + // `declared` holds the referenced entry's *raw* manifest value, which is the + // whole of what `package.json` allows a dependency to be — an `npm:`/`jsr:` + // alias, a `workspace:`/`file:`/`catalog:` spec, a git or tarball URL — and + // not merely a registry range. Reading it as one hardcoded + // `PackageSource::Registry` constraint sent `"my-lib": "workspace:*"` to npm + // as the literal range `workspace:*`; resolving it here is the same + // resolution the referenced entry itself gets, so the two agree. + Some(constraint) => match resolve(key, constraint) { + Resolved::Skip(source) => skip_item(key, source, kind), + Resolved::Dep { + name, + constraint, + source, + .. + } => { + let (line, col) = offset_to_line_col(starts, entry.content_start); + Item { + name, + version_constraint: constraint, + source, + // The span holds `$semver`, not the constraint being checked, so + // it reports its position and declines its width — the same way + // an escaped value does — and no rewriter can splice a version + // over the reference. `version_offset` is deliberately dropped + // with it: it indexes the *referent's* value, which is not the + // text at this position at all. + version_line: line, + version_col_start: col, + version_col_end: col, + registry: None, + locked_version: None, + kind, + } + } + }, + None => skip_item(key, PackageSource::Unresolved, kind), + }; + } let value = &entry.value; match resolve(key, value) { Resolved::Skip(source) => skip_item(key, source, kind), @@ -76,13 +262,21 @@ fn build_item(key: &str, kind: DependencyKind, entry: &JsonStringValue, starts: let global_start = entry.content_start + version_offset; let global_end = entry.content_end; let (line, col_start) = offset_to_line_col(starts, global_start); + // `version_offset` indexes the *decoded* value. That only lands on the right + // source byte while the two are identical, so an escaped string reports its + // line and declines the span rather than handing `--fix` a drifted offset. + let col_end = if entry.escaped { + col_start + } else { + col_start + global_end.saturating_sub(global_start) + }; Item { name, version_constraint: constraint, source, version_line: line, version_col_start: col_start, - version_col_end: col_start + global_end.saturating_sub(global_start), + version_col_end: col_end, registry: None, locked_version: None, kind, @@ -284,4 +478,257 @@ mod tests { assert!(reacts.contains(&"^18.0.0")); assert!(reacts.contains(&"^17.0.0")); } + + /// `overrides` and `resolutions` are how a vulnerable transitive dependency is + /// pinned to a patched version. Not reading them meant the pin was invisible, so a + /// stale one could never be reported. + #[test] + fn overrides_and_resolutions_are_collected() { + let content = r#"{ + "dependencies": { "express": "^4.0.0" }, + "overrides": { "minimist": "1.2.6" }, + "resolutions": { "lodash": "4.17.21", "parent/debug": "4.3.4" }, + "pnpm": { "overrides": { "glob-parent": ">=5.1.2" } } + }"#; + let m = PackageJsonParser.parse(content).expect("valid JSON"); + let by_name = |n: &str| m.items.iter().find(|i| i.name == n).cloned(); + + for name in ["minimist", "lodash", "debug", "glob-parent"] { + let it = by_name(name).unwrap_or_else(|| panic!("{name} missing")); + assert_eq!(it.kind, DependencyKind::Override, "{name}"); + } + assert_eq!(by_name("minimist").unwrap().version_constraint, "1.2.6"); + assert_eq!(by_name("express").unwrap().kind, DependencyKind::Normal); + } + + /// npm's nested form scopes an override to the parent that pulls the package in, and + /// spells "the parent itself" as `"."`, which names no new package. + #[test] + fn nested_override_keys_resolve_to_the_package_they_name() { + let content = r#"{ "overrides": { "foo": { ".": "1.0.0", "bar": "2.0.0" } } }"#; + let m = PackageJsonParser.parse(content).expect("valid JSON"); + let names: Vec<&str> = m.items.iter().map(|i| i.name.as_str()).collect(); + assert!(names.contains(&"bar"), "got {names:?}"); + assert!(!names.contains(&"."), "got {names:?}"); + } + + #[test] + fn resolution_key_paths_and_globs_name_the_last_segment() { + assert_eq!(override_name("parent/child"), Some("child")); + assert_eq!(override_name("**/lodash"), Some("lodash")); + assert_eq!(override_name("@scope/pkg"), Some("@scope/pkg")); + assert_eq!(override_name("lodash@^4"), Some("lodash")); + assert_eq!(override_name("**/@scope/pkg"), Some("@scope/pkg")); + assert_eq!(override_name("parent/@scope/pkg"), Some("@scope/pkg")); + assert_eq!(override_name("@scope/pkg@^1"), Some("@scope/pkg")); + assert_eq!(override_name("."), None); + } + + /// npm's documented `$name` override value means "use the version of my own direct + /// dependency", not a version range. It reached the version checker verbatim and + /// came back `unparseable constraint: unexpected character '$'` — a hard error on a + /// valid manifest. + #[test] + fn a_dollar_override_resolves_to_the_dependency_it_names() { + let content = r#"{ + "dependencies": { "semver": "^7.5.0" }, + "overrides": { "semver": "$semver" } +}"#; + let m = parse(content); + let overridden = m + .items + .iter() + .find(|i| i.kind == DependencyKind::Override) + .expect("an override item"); + assert_eq!(overridden.name, "semver"); + assert_eq!(overridden.version_constraint, "^7.5.0"); + assert_eq!(overridden.source, PackageSource::Registry); + // The recorded span holds `$semver`, not the constraint, so it must not be + // offered to `--fix` as a place to write a version. + assert!(!overridden.is_rewritable()); + } + + /// A referent is whatever `package.json` lets a dependency be, not necessarily a + /// registry range. + /// + /// The reference resolved to the referent's **raw** manifest value under a hardcoded + /// `PackageSource::Registry`, so an ordinary pnpm/npm workspace shape — + /// `{"dependencies":{"my-lib":"workspace:*"},"overrides":{"my-lib":"$my-lib"}}` — + /// produced a checkable dependency whose constraint was the literal string + /// `workspace:*`, which npm was asked about and `VersionReq` could not read. An + /// aliased referent was checked under the wrong name, and a `jsr:` referent was + /// fetched from the wrong registry. + #[test] + fn a_dollar_override_resolves_its_referent_the_same_way_the_referent_is_resolved() { + let content = r#"{ + "dependencies": { + "my-lib": "workspace:*", + "local": "file:../local", + "forked": "github:org/repo#v1", + "lodash": "npm:lodash-es@^4.17.21", + "path": "jsr:@std/path@^1.0.0" + }, + "overrides": { + "my-lib": "$my-lib", + "local": "$local", + "forked": "$forked", + "lodash": "$lodash", + "path": "$path" + } +}"#; + let m = parse(content); + let overrides: Vec<&Item> = m + .items + .iter() + .filter(|i| i.kind == DependencyKind::Override) + .collect(); + assert_eq!(overrides.len(), 5); + let by_source = |source: PackageSource| -> Vec<&Item> { + overrides + .iter() + .copied() + .filter(|i| i.source == source) + .collect() + }; + // A workspace, file, or git referent is not a registry dependency at either + // spelling, so the override is skipped exactly as the referent is. + assert_eq!(by_source(PackageSource::Local).len(), 2); + assert_eq!(by_source(PackageSource::Git).len(), 1); + for item in by_source(PackageSource::Local) + .into_iter() + .chain(by_source(PackageSource::Git)) + { + assert!(!item.is_checkable(), "{} was sent to a registry", item.name); + } + // An alias names the package actually published, under the registry that + // publishes it. + let aliased = by_source(PackageSource::Registry); + assert_eq!(aliased.len(), 1); + assert_eq!(aliased[0].name, "lodash-es"); + assert_eq!(aliased[0].version_constraint, "^4.17.21"); + let jsr = by_source(PackageSource::Jsr); + assert_eq!(jsr.len(), 1); + assert_eq!(jsr[0].name, "@std/path"); + assert_eq!(jsr[0].version_constraint, "^1.0.0"); + // Whatever it resolved to, the span still holds `$name` and declines its width. + for item in &overrides { + assert!(!item.is_rewritable(), "{}", item.name); + } + } + + /// A reference to something the manifest never declares is unresolvable, not a + /// parse error: the package is real, its intended version simply cannot be read. + #[test] + fn a_dollar_override_naming_nothing_declared_is_unresolvable() { + let content = r#"{ "overrides": { "semver": "$semver" } }"#; + let m = parse(content); + let overridden = m.items.first().expect("an override item"); + assert_eq!(overridden.name, "semver"); + assert_eq!(overridden.source, PackageSource::Unresolved); + assert!(!overridden.is_checkable()); + } + + /// The reference form is npm's, and only inside an override map. A `$` elsewhere is + /// left exactly as it was read. + #[test] + fn a_dollar_is_only_a_reference_inside_an_override() { + assert_eq!(override_reference("$semver"), Some("semver")); + assert_eq!(override_reference("$@scope/pkg"), Some("@scope/pkg")); + assert_eq!(override_reference("$"), None); + assert_eq!(override_reference("^7.5.0"), None); + + let content = r#"{ "dependencies": { "weird": "$semver" } }"#; + let m = parse(content); + assert_eq!(find(&m, "weird").version_constraint, "$semver"); + } + + /// A reference that names nothing usable is still a reference. `"$"` and `"$a b"` + /// failed the well-formedness guard and fell through to the checker as literal + /// constraints, which hard-failed on the `$` — the exact failure the reference form + /// exists to prevent. + #[test] + fn a_reference_naming_nothing_is_unresolved_rather_than_a_constraint() { + let content = r#"{ "overrides": { "minimist": "$", "foo": "$a b", "bar": "$nope" } }"#; + let m = parse(content); + for name in ["minimist", "foo", "bar"] { + let item = find(&m, name); + assert_eq!(item.source, PackageSource::Unresolved, "{name}"); + assert!( + item.version_constraint.is_empty(), + "{name} kept `{}` as a constraint", + item.version_constraint + ); + assert!(!item.is_checkable(), "{name} was sent to a registry"); + } + } + + /// A pnpm override key scoped to a parent (`foo@2>bar`) pins **bar**. Reading the + /// first segment named `foo`, so the entry was checked against an unrelated + /// package's version list — and `fix --all` would then have rewritten a pin on `bar` + /// to whatever `foo`'s newest release happened to be. + #[test] + fn a_scoped_override_key_names_the_package_after_the_last_arrow() { + assert_eq!(override_name("foo@2>bar"), Some("bar")); + assert_eq!(override_name("foo>bar"), Some("bar")); + assert_eq!(override_name("a>b>c"), Some("c")); + assert_eq!( + override_name("@scope/pkg@1>@scope/other"), + Some("@scope/other") + ); + assert_eq!(override_name("foo"), Some("foo")); + } + + /// The same key read end to end through the parser, so the defect is falsified where + /// it was observed rather than only at the helper. + #[test] + fn a_scoped_pnpm_override_is_checked_as_the_package_it_pins() { + let content = r#"{ "pnpm": { "overrides": { "foo@2>bar": "3.0.0" } } }"#; + let m = parse(content); + let names: Vec<&str> = m.items.iter().map(|i| i.name.as_str()).collect(); + assert_eq!(names, vec!["bar"], "got {names:?}"); + assert_eq!(find(&m, "bar").kind, DependencyKind::Override); + } + + /// pnpm and Yarn both allow a range in the override key, and a range contains `>`. + /// Splitting on every `>` cut the key inside its own range: `lodash@>=1.0.0` was + /// read as a package called `=1.0.0`, which no registry has, so the entry reported + /// as a 404 — and once a 404 stopped failing the gate, silently. + #[test] + fn a_range_in_an_override_key_is_not_a_parent_separator() { + assert_eq!(override_name("lodash@>=1.0.0"), Some("lodash")); + assert_eq!(override_name("foo>bar@>=1"), Some("bar")); + assert_eq!(override_name("foo@>1.0.0"), Some("foo")); + assert_eq!(override_name("a@>b"), Some("a")); + assert_eq!(override_name("lodash@>=1.0.0,>=1.1"), Some("lodash")); + assert_eq!(override_name("@scope/pkg@>=1.0.0"), Some("@scope/pkg")); + assert_eq!(override_name("**/lodash@>=1.0.0"), Some("lodash")); + // A separator still separates when a range sits on either side of it. + assert_eq!(override_name("quux@1>bar@^2.1.0"), Some("bar")); + assert_eq!(override_name("foo@>=1.0.0>bar"), Some("bar")); + assert_eq!( + override_name("@scope/pkg@1>@scope/other"), + Some("@scope/other") + ); + } + + /// The whole reported key list, read through the parser: every one of these named a + /// fragment of its own range before, so the manifest was checked against four + /// packages that do not exist. + #[test] + fn override_keys_carrying_ranges_name_their_own_packages() { + let content = r#"{ "overrides": { + "lodash@>=1.0.0": "4.17.21", + "foo>bar@>=1": "2.0.0", + "foo@>1.0.0": "1.0.0", + "a@>b": "1.0.0", + "quux@1>baz@^2.1.0": "2.0.0" + } }"#; + let m = parse(content); + let names: Vec<&str> = m.items.iter().map(|i| i.name.as_str()).collect(); + assert_eq!( + names, + vec!["lodash", "bar", "foo", "a", "baz"], + "got {names:?}" + ); + } } diff --git a/crates/dependable-core/src/parsers/pyproject_toml.rs b/crates/dependable-core/src/parsers/pyproject_toml.rs index 27e2826..d223088 100644 --- a/crates/dependable-core/src/parsers/pyproject_toml.rs +++ b/crates/dependable-core/src/parsers/pyproject_toml.rs @@ -80,10 +80,42 @@ impl Parser for PyprojectTomlParser { } } - // pixi: top-level [dependencies] (name = "version-spec"). + // pixi: top-level [dependencies] (name = "version-spec"), and the same tables + // nested under [tool.pixi] in a pyproject.toml. if let Some(t) = root.get("dependencies").and_then(TomlItem::as_table_like) { collect_table_deps(t, DependencyKind::Normal, &starts, &mut items); } + for path in [ + ["tool", "pixi", "dependencies"], + ["tool", "pixi", "pypi-dependencies"], + ] { + if let Some(t) = nav(root, &path).and_then(TomlItem::as_table_like) { + collect_table_deps(t, DependencyKind::Normal, &starts, &mut items); + } + } + + // PEP 518 build requirements. Cargo's `[build-dependencies]` are read, and these + // are the same thing for Python: real packages, resolved and installed, and just + // as capable of carrying an advisory. + if let Some(arr) = nav(root, &["build-system", "requires"]).and_then(TomlItem::as_array) { + collect_pep508_array(arr, DependencyKind::Build, &starts, &mut items); + } + + // uv and PDM development groups. + if let Some(arr) = + nav(root, &["tool", "uv", "dev-dependencies"]).and_then(TomlItem::as_array) + { + collect_pep508_array(arr, DependencyKind::Dev, &starts, &mut items); + } + if let Some(t) = + nav(root, &["tool", "pdm", "dev-dependencies"]).and_then(TomlItem::as_table_like) + { + for (_group, value) in t.iter() { + if let Some(arr) = value.as_array() { + collect_pep508_array(arr, DependencyKind::Dev, &starts, &mut items); + } + } + } Ok(ParsedManifest { kind: ManifestKind::PyprojectToml, @@ -139,6 +171,37 @@ fn parse_table_dep( starts, )); } + // Poetry's multiple-constraint form: an array of tables, each with its own marker + // (`foo = [{version = "^1.0", python = "<3.8"}, {version = "^2.0", python = ">=3.8"}]`). + // Markers are not evaluated here, so the first entry that declares a version stands + // for the dependency — which reports it, where dropping the whole array did not. + if let Some(array) = item.as_array() { + for value in array.iter() { + let Some(table) = value.as_inline_table() else { + continue; + }; + if table.contains_key("path") || table.contains_key("url") { + return Some(skip_item(name, PackageSource::Local, kind)); + } + if table.contains_key("git") { + return Some(skip_item(name, PackageSource::Git, kind)); + } + if let Some(version_value) = table.get("version") + && let Some(version) = version_value.as_str() + && let Some(span) = version_value.span() + { + return Some(make_item( + name, + version, + span, + PackageSource::Registry, + kind, + starts, + )); + } + } + return None; + } // Inline table / `[tool.poetry.dependencies.x]`. if let Some(table) = item.as_table_like() { if table.contains_key("path") || table.contains_key("url") { @@ -291,4 +354,60 @@ mod tests { assert_eq!(find(&m, "pytest").version_constraint, ">=7.0"); assert_eq!(sliced(content, find(&m, "coverage")), ">=6.0"); } + + /// PEP 518 build requirements are packages like any other. Cargo's + /// `[build-dependencies]` are read; the Python equivalent was not. + #[test] + fn build_system_requires_are_collected() { + let content = "[build-system]\nrequires = [\"setuptools>=61\", \"wheel\"]\nbuild-backend = \"setuptools.build_meta\"\n"; + let m = PyprojectTomlParser.parse(content).expect("valid TOML"); + let setuptools = m + .items + .iter() + .find(|i| i.name == "setuptools") + .expect("setuptools"); + assert_eq!(setuptools.kind, DependencyKind::Build); + assert_eq!(setuptools.version_constraint, ">=61"); + assert!(m.items.iter().any(|i| i.name == "wheel")); + } + + #[test] + fn uv_pdm_and_pixi_tables_are_collected() { + let content = concat!( + "[tool.uv]\n", + "dev-dependencies = [\"pytest>=8\"]\n\n", + "[tool.pdm.dev-dependencies]\n", + "test = [\"coverage>=7\"]\n\n", + "[tool.pixi.dependencies]\n", + "numpy = \">=1.26\"\n\n", + "[tool.pixi.pypi-dependencies]\n", + "requests = \">=2.31\"\n", + ); + let m = PyprojectTomlParser.parse(content).expect("valid TOML"); + for name in ["pytest", "coverage", "numpy", "requests"] { + assert!(m.items.iter().any(|i| i.name == name), "{name} missing"); + } + assert_eq!( + m.items.iter().find(|i| i.name == "pytest").unwrap().kind, + DependencyKind::Dev + ); + } + + /// Poetry's array-of-tables form used to fall through every branch and vanish — no + /// item, no error, no way to tell it apart from a manifest that never named it. + #[test] + fn poetry_multiple_constraint_arrays_are_not_dropped() { + let content = concat!( + "[tool.poetry.dependencies]\n", + "foo = [{version = \"^1.0\", python = \"<3.8\"}, {version = \"^2.0\", python = \">=3.8\"}]\n", + ); + let m = PyprojectTomlParser.parse(content).expect("valid TOML"); + let foo = m + .items + .iter() + .find(|i| i.name == "foo") + .expect("foo missing"); + assert_eq!(foo.version_constraint, "^1.0"); + assert!(foo.is_rewritable()); + } } diff --git a/crates/dependable-core/src/parsers/requirements_txt.rs b/crates/dependable-core/src/parsers/requirements_txt.rs index b1c5cd5..7e54192 100644 --- a/crates/dependable-core/src/parsers/requirements_txt.rs +++ b/crates/dependable-core/src/parsers/requirements_txt.rs @@ -114,6 +114,21 @@ pub(crate) fn parse_pep508_spec(spec: &str) -> Option<(String, &str, usize)> { return None; } + // PEP 508 allows the specifier in parentheses — `flask (>=2.0)` — which is the form + // PEP 621 metadata round-trips into. Left in place the parens ended up inside the + // constraint, which `VersionReq` then rejected. + if bytes.get(pos) == Some(&b'(') + && let Some(close) = spec[pos + 1..].find(')') + { + let inner = &spec[pos + 1..pos + 1 + close]; + let lead = inner.len() - inner.trim_start().len(); + let constraint = inner.trim(); + if constraint.is_empty() { + return Some((name.to_string(), "", spec.len())); + } + return Some((name.to_string(), constraint, pos + 1 + lead)); + } + // Constraint runs to a `;` environment marker or the end. let rest = &spec[pos..]; let region = rest.split(';').next().unwrap_or(rest); @@ -181,4 +196,29 @@ mod tests { assert_eq!(find(&m, "numpy").version_constraint, ""); assert!(find(&m, "numpy").is_checkable()); } + + /// PEP 508 permits the specifier in parentheses. The parens used to land inside the + /// constraint, producing `(>=2.0)` — which does not parse, so the dependency was + /// misreported rather than checked. + #[test] + fn parenthesized_specifiers_are_unwrapped() { + let (name, constraint, offset) = parse_pep508_spec("flask (>=2.0)").unwrap(); + assert_eq!(name, "flask"); + assert_eq!(constraint, ">=2.0"); + assert_eq!(&"flask (>=2.0)"[offset..offset + constraint.len()], ">=2.0"); + + let (name, constraint, _) = parse_pep508_spec("requests[security] (>=2.0,<3)").unwrap(); + assert_eq!(name, "requests"); + assert_eq!(constraint, ">=2.0,<3"); + + // Parens with an environment marker outside them. + let (_, constraint, _) = + parse_pep508_spec("flask (>=2.0) ; python_version < \"3.8\"").unwrap(); + assert_eq!(constraint, ">=2.0"); + + // Empty parens are a bare requirement. + let (name, constraint, _) = parse_pep508_spec("flask ()").unwrap(); + assert_eq!(name, "flask"); + assert_eq!(constraint, ""); + } } diff --git a/crates/dependable-core/src/result.rs b/crates/dependable-core/src/result.rs index 732032a..11ade39 100644 --- a/crates/dependable-core/src/result.rs +++ b/crates/dependable-core/src/result.rs @@ -46,6 +46,12 @@ pub struct CheckResult { /// leaves this `None` everywhere; only a caller that asked for license /// collection sees it filled in. pub license: Option, + /// Where an [`Error`](DependencyStatus::Error) status came from. + /// + /// [`ErrorOrigin::None`] for every other status. Carried from the typed fetch error + /// rather than re-derived from the message, because the message cannot answer it and + /// a gate turns on the answer. + pub error_origin: ErrorOrigin, } impl CheckResult { @@ -55,7 +61,6 @@ impl CheckResult { pub fn new(item: Item, status: DependencyStatus) -> Self { Self { item, - status, latest_compatible: None, latest_available: None, patch_available: false, @@ -63,6 +68,27 @@ impl CheckResult { all_vulnerabilities: HashMap::new(), advisories: Vec::new(), license: None, + // Unrecorded provenance is not evidence that a registry answered, and the + // safe reading of "we do not know" is the one that keeps a gate failing. + error_origin: match &status { + DependencyStatus::Error(_) => ErrorOrigin::Local, + _ => ErrorOrigin::None, + }, + status, + } + } + + /// A result for a dependency that has no status, carrying **where** the failure + /// came from — see [`ErrorOrigin`]. + /// + /// [`Self::new`] with an `Error` status records [`ErrorOrigin::Local`], the answer + /// that keeps a gate failing; only a caller that knows a registry answered may say + /// so, and it says so here. + #[must_use] + pub fn errored(item: Item, message: impl Into, origin: ErrorOrigin) -> Self { + Self { + error_origin: origin, + ..Self::new(item, DependencyStatus::Error(message.into())) } } @@ -72,7 +98,6 @@ impl CheckResult { pub fn from_evaluation(item: Item, eval: Evaluation) -> Self { Self { item, - status: eval.status, latest_compatible: eval.latest_compatible, latest_available: eval.latest_available, patch_available: eval.patch_available, @@ -80,6 +105,11 @@ impl CheckResult { all_vulnerabilities: HashMap::new(), advisories: Vec::new(), license: None, + error_origin: match &eval.status { + DependencyStatus::Error(_) => ErrorOrigin::Local, + _ => ErrorOrigin::None, + }, + status: eval.status, } } @@ -108,6 +138,45 @@ impl CheckResult { } } +/// Where a [`DependencyStatus::Error`] came from. +/// +/// A `--fail-on` gate has to tell these apart and the message cannot tell it: "not +/// found" and "unparseable constraint" are both prose, and reading provenance out of +/// prose is what folded them together. A registry answering that a package does not +/// exist is a permanent per-dependency fact that must not fail a whole build — gating on +/// it turned every repository with one unpublished internal package into exit 2. Input +/// that is not a version requirement at all reached no registry: nothing was +/// established, and certifying the build over it is the gate lying. +/// +/// The line is drawn at what the *manifest* says, not at what this crate happens to +/// support. A constraint that is a perfectly good declaration in a dialect with no +/// front-end here is [`DependencyStatus::Undetermined`] and carries no origin at all — +/// failing a build over that punishes the user for a gap that is ours, and +/// `--fail-on any` already fails on it for anyone who wants every constraint +/// established. An `Error` is reserved for what nobody could read as a requirement. +/// +/// `#[non_exhaustive]`: match with a wildcard arm so new origins are additive. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[non_exhaustive] +pub enum ErrorOrigin { + /// Not an error: the dependency has a real status. + #[default] + None, + /// The registry answered by name: no such package. A private or internal package, + /// one served by a registry this run does not route to, a deleted package. + NotFound, + /// A registry request that produced no answer — a timeout, a refused connection, a + /// 5xx, an undecodable response, a document listing no versions at all. + Unanswered, + /// A failure this run reached without a registry ever being asked: input that is not + /// a version requirement at all (`^^^bogus` — operators announcing a range that is + /// never spelled), or a fetch whose result never arrived. + /// + /// Deliberately *not* a constraint this crate merely has no front-end for. That is + /// [`DependencyStatus::Undetermined`], which is outside the gate. + Local, +} + /// The status of a single dependency. /// /// `#[non_exhaustive]`: match with a wildcard arm so new statuses are additive. @@ -120,6 +189,24 @@ pub enum DependencyStatus { Outdated, Vulnerable, Error(String), + /// A real package whose declared version this run could not read: the constraint is + /// written in a dialect that did not translate, it names a channel or a branch + /// instead of a range, or it refers to something the manifest never declares. + /// + /// Deliberately distinct from [`Self::UpToDate`]: an unreadable constraint is not + /// evidence that a dependency is current, and reporting it as current is what + /// disarms `--fail-on outdated`. + /// + /// Deliberately distinct from [`Self::Error`] too, and the distinction is a policy, + /// not a shade of meaning. `Error` is the registry failing, the fetch failing, or + /// input nobody could read as a requirement — a run that could not do its job. + /// `Undetermined` is a constraint the *manifest* got right and this crate has no + /// front-end for, which is a gap here rather than a defect there. So it does not + /// count toward the run's unevaluated tally, does not trip `--fail-on vulnerable` + /// (which promises something about vulnerabilities) or `--fail-on outdated` (which + /// promises something about staleness), and does trip `--fail-on any`, which is the + /// setting that promises everything was established. + Undetermined, Local, Git, } @@ -135,6 +222,7 @@ impl DependencyStatus { DependencyStatus::Outdated => "outdated", DependencyStatus::Vulnerable => "vulnerable", DependencyStatus::Error(_) => "error", + DependencyStatus::Undetermined => "undetermined", DependencyStatus::Local => "local", DependencyStatus::Git => "git", } @@ -150,6 +238,7 @@ impl DependencyStatus { DependencyStatus::Outdated => "OUTDATED", DependencyStatus::Vulnerable => "VULN", DependencyStatus::Error(_) => "ERROR", + DependencyStatus::Undetermined => "UNDETERMINED", DependencyStatus::Local => "LOCAL", DependencyStatus::Git => "GIT", } diff --git a/crates/dependable-core/src/semver/checker.rs b/crates/dependable-core/src/semver/checker.rs index 061c177..8de0727 100644 --- a/crates/dependable-core/src/semver/checker.rs +++ b/crates/dependable-core/src/semver/checker.rs @@ -2,7 +2,8 @@ use ::semver::{Version, VersionReq}; -use super::normalize::normalize_constraint; +use super::normalize::{is_dialect_tag, normalize_range_constraint, try_to_semver_constraint}; +use crate::ecosystem::Ecosystem; use crate::result::DependencyStatus; /// The outcome of evaluating one constraint against a set of available versions. @@ -21,10 +22,15 @@ pub struct Evaluation { /// Parse a version-requirement string into a [`semver::VersionReq`]. /// +/// The string is put through [`normalize_range_constraint`] first, so the ordinary +/// npm / Composer / Dart spellings of a range — a `||` union, space-separated +/// comparators, a hyphen range, a stability flag, Dart's `any` — are accepted rather +/// than rejected as malformed. +/// /// # Errors /// Returns an error if the (normalized) constraint is not a valid requirement. pub fn to_version_req(constraint: &str) -> Result { - VersionReq::parse(&normalize_constraint(constraint)) + VersionReq::parse(&normalize_range_constraint(constraint)) } /// Whether `constraint` is an npm dist-tag that tracks the newest release @@ -33,6 +39,37 @@ fn is_latest_tag(constraint: &str) -> bool { constraint.trim() == "latest" } +/// Classify a dependency whose constraint is written in `ecosystem`'s dialect. +/// +/// The wrapper [`check_version`] cannot see the difference between an author who +/// declared no constraint and a constraint this crate failed to translate, because +/// the failing translators hand it the same empty string for both — so an +/// untranslatable range became `*`, resolved to the newest release, and reported +/// `up to date`. Translating here keeps that distinction: a failed translation is +/// [`DependencyStatus::Undetermined`], which claims nothing about currency. +#[must_use] +pub fn check_version_for( + constraint: &str, + ecosystem: Ecosystem, + versions: &[String], + locked_at: Option<&str>, +) -> Evaluation { + match try_to_semver_constraint(constraint, ecosystem) { + Some(translated) => check_version(&translated, versions, locked_at), + None => Evaluation { + status: DependencyStatus::Undetermined, + latest_compatible: None, + latest_available: newest(versions).map(|v| v.to_string()), + patch_available: false, + }, + } +} + +/// The newest parseable version in `versions`, if any. +fn newest(versions: &[String]) -> Option { + versions.iter().filter_map(|v| Version::parse(v).ok()).max() +} + /// Classify a dependency. /// /// `versions` may be in any order; `locked_at` is the resolved version from a @@ -62,17 +99,49 @@ pub fn check_version(constraint: &str, versions: &[String], locked_at: Option<&s // parse and being misreported. `--fix` still never rewrites the tag (see the // fix layer), so the manifest keeps tracking the channel. let req = match to_version_req(constraint) { - Ok(req) => Some(req), - Err(_) if is_latest_tag(constraint) => Some(VersionReq::STAR), - Err(_) => None, + Ok(req) => req, + // An absent requirement means "any version" — a bare `numpy` in a requirements + // file, or a manifest entry that names no range. `VersionReq` rejects the empty + // string, so the intent has to be spelled out. + Err(_) if constraint.trim().is_empty() => VersionReq::STAR, + Err(_) if is_latest_tag(constraint) => VersionReq::STAR, + // A constraint that names a channel or a branch rather than a range — npm's + // `next`, Composer's `dev-master` — is a declaration this crate has no front-end + // for, not input the author got wrong. It is `Undetermined`: nothing is claimed + // about the dependency's currency, the run says so on stderr, and `--fail-on any` + // still fails on it, but a gate about vulnerabilities or staleness is not failed + // because of a dialect gap that is ours. `latest` never reaches here; the ranges + // npm and Composer document (`^1 || ^2`, `>=1.0.0 <2.0.0`, `1.2.3 - 2.3.4`, + // `2.8.*@dev`) are translated by `normalize_range_constraint` and do not either. + Err(_) if is_dialect_tag(constraint) => { + return Evaluation { + status: DependencyStatus::Undetermined, + latest_compatible: None, + latest_available: Some(latest_available.to_string()), + patch_available: false, + }; + } + // What is left is not a range at all — `^^^bogus`, a constraint whose operators + // say it means to be one and is not. It used to fall through as + // `UpdateAvailable`, which reads as "a newer version is waiting for you" — the + // one message a dependency whose requirement was never understood must not send. + // Wildcards never land here: `1.x` and `1.*` are requirements the crate parses, + // so they stay real evaluations and reach the fix layer, where the wildcard guard + // declines to pin them. + Err(e) => { + return Evaluation { + status: DependencyStatus::Error(format!("unparseable constraint: {e}")), + latest_compatible: None, + latest_available: Some(latest_available.to_string()), + patch_available: false, + }; + } }; - let latest_compatible = req - .as_ref() - .and_then(|r| parsed.iter().rev().find(|v| r.matches(v)).cloned()); + let latest_compatible = parsed.iter().rev().find(|v| req.matches(v)).cloned(); let locked = locked_at.and_then(|s| Version::parse(s).ok()); // A locked version that no longer satisfies the declared constraint. - if let (Some(req), Some(locked)) = (req.as_ref(), locked.as_ref()) + if let Some(locked) = locked.as_ref() && !req.matches(locked) { return Evaluation { @@ -90,7 +159,15 @@ pub fn check_version(constraint: &str, versions: &[String], locked_at: Option<&s None => DependencyStatus::UpdateAvailable, Some(cur) if *cur >= latest_available => DependencyStatus::UpToDate, Some(cur) => match latest_compatible.as_ref() { - Some(lc) if lc > cur && lc.major == cur.major && lc.minor == cur.minor => { + // Under semver's 0.x rules the leftmost non-zero component is the breaking + // axis, so on `0.0.z` every bump is breaking and nothing there is a patch. + // Calling it one would hand `--fix` a green light it has not earned. + Some(lc) + if lc > cur + && lc.major == cur.major + && lc.minor == cur.minor + && !(cur.major == 0 && cur.minor == 0) => + { DependencyStatus::PatchAvailable } _ => DependencyStatus::UpdateAvailable, @@ -115,6 +192,84 @@ mod tests { list.iter().map(|s| (*s).to_string()).collect() } + /// A constraint whose dialect did not translate must not be reported as current. + /// + /// Maven, NuGet, and PEP 440 all signal a failed translation by returning the empty + /// string, which the checker read as "no constraint declared" and turned into `*` — + /// so `latest_compatible` became the newest release and the status became + /// `UpToDate`, disarming `--fail-on outdated` at every ecosystem that translates. + /// One case per affected ecosystem, each taken from a manifest that parses. + #[test] + fn an_untranslatable_constraint_is_undetermined_not_up_to_date() { + // A Gradle version catalog with an unclosed Maven interval. + let jvm = check_version_for( + "[4.0,4.9", + Ecosystem::Jvm, + &vers(&["4.0.0", "4.9.0", "5.5.0"]), + None, + ); + assert_eq!(jvm.status, DependencyStatus::Undetermined, "{jvm:?}"); + assert_eq!(jvm.latest_compatible, None); + assert_eq!(jvm.latest_available.as_deref(), Some("5.5.0")); + + // Maven's `LATEST` keyword, which is a version *selector*, not a range. + let latest_keyword = + check_version_for("LATEST", Ecosystem::Jvm, &vers(&["2.10.0", "2.14.0"]), None); + assert_eq!(latest_keyword.status, DependencyStatus::Undetermined); + + // A NuGet interval missing its closing bracket. The newest release is *outside* + // the range the author meant, so `UpToDate` here was actively wrong. + let nuget = check_version_for( + "[12.0.0,13.0.0", + Ecosystem::CSharp, + &vers(&["12.0.3", "13.0.4"]), + None, + ); + assert_eq!(nuget.status, DependencyStatus::Undetermined, "{nuget:?}"); + + // PEP 440 `!=`, which has no `semver::VersionReq` spelling, so every clause is + // dropped and nothing is left to compare against. + let python = check_version_for( + "!=2.31.0", + Ecosystem::Python, + &vers(&["2.31.0", "2.34.2"]), + None, + ); + assert_eq!(python.status, DependencyStatus::Undetermined, "{python:?}"); + } + + /// The other half of the same distinction: an author who declared *no* constraint + /// still means "any version", and must keep evaluating as `*` rather than becoming + /// undetermined. + #[test] + fn an_absent_constraint_still_means_any_version() { + for ecosystem in [ + Ecosystem::Python, + Ecosystem::CSharp, + Ecosystem::Jvm, + Ecosystem::Elixir, + Ecosystem::Rust, + ] { + let e = check_version_for("", ecosystem, &vers(&["1.0.0", "2.0.0"]), Some("2.0.0")); + assert_eq!(e.status, DependencyStatus::UpToDate, "{ecosystem:?}: {e:?}"); + assert_eq!(e.latest_compatible.as_deref(), Some("2.0.0")); + } + } + + /// A constraint the ecosystem *can* translate keeps its real evaluation, so the + /// guard above is not simply refusing to answer. + #[test] + fn a_translatable_constraint_is_still_evaluated() { + let e = check_version_for( + "[1.0,2.0)", + Ecosystem::Jvm, + &vers(&["1.0.0", "1.9.0", "2.5.0"]), + None, + ); + assert_eq!(e.status, DependencyStatus::UpdateAvailable, "{e:?}"); + assert_eq!(e.latest_compatible.as_deref(), Some("1.9.0")); + } + #[test] fn up_to_date_when_constraint_allows_latest() { let e = check_version("1", &vers(&["1.0.0", "1.2.0", "1.5.0"]), None); @@ -179,6 +334,133 @@ mod tests { assert!(matches!(e.status, DependencyStatus::Error(_))); } + /// A requirement nobody could parse is not an upgrade recommendation. It used to + /// come back as `UpdateAvailable` — indistinguishable from a real available upgrade. + /// + /// `Error` is now reserved for input that is not a range at all: operators that + /// announce a range and then do not spell one. The npm and Composer ranges that used + /// to land here are translated instead (see + /// [`the_ordinary_npm_and_composer_ranges_translate`]), and a name — a dist-tag, a + /// branch alias — is `Undetermined`. + #[test] + fn a_constraint_that_is_not_a_range_at_all_is_an_error_not_an_upgrade() { + let versions = vec!["1.0.0".to_string(), "2.0.0".to_string()]; + for constraint in ["^^^bogus", "workspace:^", ">=<1.0.0", "~~"] { + let ev = check_version(constraint, &versions, None); + assert!( + matches!(ev.status, DependencyStatus::Error(_)), + "{constraint} yielded {:?}", + ev.status + ); + assert!(ev.latest_compatible.is_none(), "{constraint}"); + // The registry answered, so what it said is still worth reporting. + assert_eq!( + ev.latest_available.as_deref(), + Some("2.0.0"), + "{constraint}" + ); + } + } + + /// The ranges npm and Composer document are *valid* constraints this crate merely + /// had no front-end for, and every one of them used to reach `VersionReq::parse` + /// verbatim, fail, and be recorded as a dependency the run could not evaluate — which + /// the new `unevaluated` gate turns into exit 2 for the whole repository under the + /// shipped Action's default `fail-on: vulnerable`. + #[test] + fn the_ordinary_npm_and_composer_ranges_translate() { + let versions = vec![ + "15.4.0".to_string(), + "16.8.0".to_string(), + "17.0.2".to_string(), + "19.0.0".to_string(), + ]; + // (constraint, the newest release it admits out of `versions`) + for (constraint, compatible) in [ + (">=16.8.0 <19.0.0", "17.0.2"), + ("^15.0.0 || ^16.0.0", "16.8.0"), + ("15.4.0 - 17.0.2", "17.0.2"), + ("16.8.0 - 17", "17.0.2"), + (">= 16.8.0", "19.0.0"), + ("any", "19.0.0"), + ("@dev", "19.0.0"), + ("16.*@dev", "16.8.0"), + ] { + let ev = check_version(constraint, &versions, None); + assert!( + !matches!( + ev.status, + DependencyStatus::Error(_) | DependencyStatus::Undetermined + ), + "{constraint} yielded {:?}", + ev.status + ); + assert_eq!( + ev.latest_compatible.as_deref(), + Some(compatible), + "{constraint}" + ); + } + } + + /// A name is not a malformed range. npm's `next`, Composer's `dev-master` and a Git + /// branch alias are ordinary declarations in dialects this crate has no front-end + /// for, so they claim nothing rather than failing the run: `Undetermined` keeps + /// `--fail-on any` failing while leaving `--fail-on vulnerable` and + /// `--fail-on outdated` — which promise nothing about parseability — alone. + #[test] + fn a_channel_or_branch_name_is_undetermined_not_an_error() { + let versions = vec!["1.0.0".to_string(), "2.0.0".to_string()]; + for constraint in ["next", "beta", "canary", "dev-master", "not-a-range"] { + let ev = check_version(constraint, &versions, None); + assert_eq!( + ev.status, + DependencyStatus::Undetermined, + "{constraint} yielded {:?}", + ev.status + ); + assert!(ev.latest_compatible.is_none(), "{constraint}"); + assert_eq!( + ev.latest_available.as_deref(), + Some("2.0.0"), + "{constraint}" + ); + } + } + + /// An empty constraint is `*`, not an error — a bare `numpy` in a requirements file + /// is a legitimate declaration and must keep resolving. + #[test] + fn an_empty_constraint_still_resolves() { + let versions = vec!["1.0.0".to_string(), "2.0.0".to_string()]; + let ev = check_version("", &versions, None); + assert!( + !matches!(ev.status, DependencyStatus::Error(_)), + "{:?}", + ev.status + ); + assert_eq!(ev.latest_compatible.as_deref(), Some("2.0.0")); + } + + /// Under semver's 0.x rules the leftmost non-zero component is the breaking axis, so + /// on `0.0.z` there is no compatible axis left and nothing is a patch. + #[test] + fn zero_zero_versions_have_no_patch_axis() { + let versions = vec!["0.0.3".to_string(), "0.0.4".to_string()]; + let ev = check_version("^0.0.3", &versions, Some("0.0.3")); + assert_eq!( + ev.status, + DependencyStatus::UpdateAvailable, + "0.0.3 -> 0.0.4 is a breaking bump" + ); + assert!(!ev.patch_available); + + // `0.2.z` still has one: the patch component floats under `^0.2.3`. + let versions = vec!["0.2.3".to_string(), "0.2.9".to_string()]; + let ev = check_version("^0.2.3", &versions, Some("0.2.3")); + assert_eq!(ev.status, DependencyStatus::PatchAvailable); + } + /// The reachability witness for issue #87: a wildcard constraint really does /// reach the fix layer as an upgradable dependency, so the guard there is not /// dead code. `1.x` allows `1.9.0` but not `2.0.0`, so the newest release sits diff --git a/crates/dependable-core/src/semver/elixir.rs b/crates/dependable-core/src/semver/elixir.rs index ce37e26..b831c9c 100644 --- a/crates/dependable-core/src/semver/elixir.rs +++ b/crates/dependable-core/src/semver/elixir.rs @@ -3,21 +3,61 @@ //! Hex *versions* are already semver, so only *constraints* need translating. The //! `~>` operator differs from semver's `~`: `~> 2.1` means `>=2.1.0, <3.0.0` (only //! the last given component is bounded), whereas `~> 2.1.3` means `>=2.1.3, -//! <2.2.0`. The comparison operators (`>=`, `>`, `<=`, `<`, `==`) map directly, a -//! bare version is exact, and `or` (union) is not expressible in `semver::VersionReq` -//! — we keep the last (newest-allowing) clause, which is lossy only for multi-range -//! disjunctions. +//! <2.2.0`. The comparison operators (`>=`, `>`, `<=`, `<`, `==`) map directly and a +//! bare version is exact. +//! +//! `and` (intersection) is semver's comma. `or` (union) has no semver spelling, so one +//! clause has to be chosen; we take the one with the highest lower bound rather than +//! whichever was written last, because Hex does not require clauses in ascending order +//! and `~> 2.0 or ~> 1.0` would otherwise resolve to the 1.x range and report every 2.x +//! release as out of range. /// Convert a Hex version requirement into a `semver::VersionReq`-compatible string. +/// +/// A constraint that cannot be translated returns the **empty string**, which is the one +/// signal [`try_to_semver_constraint`](crate::semver::try_to_semver_constraint) reads as +/// a failed translation — so the dependency is reported `undetermined` and claims +/// nothing about its own currency. Returning the constraint verbatim instead, as this +/// did, hid the failure from that guard entirely: a non-empty result is taken as a +/// successful translation, so `!= 1.0.0` was passed on as a range and hard-failed the +/// whole run rather than being recorded as a dialect this crate cannot express. #[must_use] pub fn hex_constraint_to_semver(constraint: &str) -> String { - // `A or B` is a union; semver can't express it, so keep the last clause. - let clause = constraint - .rsplit(" or ") - .next() - .unwrap_or(constraint) - .trim(); - convert_clause(clause).unwrap_or_default() + let unions: Vec<&str> = constraint.split(" or ").map(str::trim).collect(); + let mut best: Option<(::semver::Version, String)> = None; + for union in unions { + // `and` is an intersection, which semver writes as a comma-separated list. + let Some(converted) = union + .split(" and ") + .map(|clause| convert_clause(clause.trim())) + .collect::>>() + .map(|parts| parts.join(", ")) + else { + continue; + }; + let bound = lower_bound(&converted); + if best.as_ref().is_none_or(|(b, _)| bound > *b) { + best = Some((bound, converted)); + } + } + best.map_or_else(String::new, |(_, converted)| converted) +} + +/// The lowest version a converted clause admits, used only to rank union branches. +fn lower_bound(converted: &str) -> ::semver::Version { + let zero = ::semver::Version::new(0, 0, 0); + converted + .split(',') + .filter_map(|part| { + let p = part.trim(); + let rest = p + .strip_prefix(">=") + .or_else(|| p.strip_prefix('=')) + .or_else(|| p.strip_prefix('>'))?; + ::semver::Version::parse(rest.trim()).ok() + }) + .max() + .unwrap_or(zero) } fn convert_clause(clause: &str) -> Option { @@ -50,12 +90,14 @@ fn tilde(version: &str) -> Option { return None; } let lower = to_semver_version(version)?; + // Components are parsed straight from a manifest, so `u64::MAX` is reachable input + // and a plain `+ 1` panics in debug and wraps in release. let upper = if nums.len() >= 3 { // Bound the minor: only the patch may float. - format!("{}.{}.0", nums[0], nums[1] + 1) + format!("{}.{}.0", nums[0], nums[1].saturating_add(1)) } else { // Bound the major: the minor may float. - format!("{}.0.0", nums[0] + 1) + format!("{}.0.0", nums[0].saturating_add(1)) }; Some(format!(">={lower}, <{upper}")) } @@ -106,4 +148,55 @@ mod tests { ">=2.0.0, <3.0.0" ); } + + /// An untranslatable constraint must come back as the empty string — the one signal + /// `try_to_semver_constraint` reads as a failed translation, which makes the + /// dependency `undetermined`. + /// + /// Returning the constraint verbatim, as this used to, is invisible to that guard: a + /// non-empty result is taken as a *successful* translation, so `!= 1.0.0` — an + /// ordinary Hex exclusion — was passed on as though it were a semver range and hard + /// failed the whole run instead. + #[test] + fn an_untranslatable_constraint_signals_failure_rather_than_echoing_itself() { + for constraint in ["~> not.a.version", "@@@", ">= banana", "!= 1.0.0"] { + assert_eq!( + hex_constraint_to_semver(constraint), + "", + "{constraint} was not reported as a failed translation" + ); + } + } + + /// `and` is an intersection, which semver writes as a comma. + #[test] + fn intersections_become_comma_separated_bounds() { + let converted = hex_constraint_to_semver(">= 1.0.0 and < 2.0.0"); + let req = ::semver::VersionReq::parse(&converted).expect(&converted); + assert!(req.matches(&::semver::Version::parse("1.5.0").unwrap())); + assert!(!req.matches(&::semver::Version::parse("2.0.0").unwrap())); + assert!(!req.matches(&::semver::Version::parse("0.9.0").unwrap())); + } + + /// Hex does not require union clauses in ascending order, so taking the last one + /// could pick the *older* range and report every newer release as out of range. + #[test] + fn a_union_picks_the_newest_clause_regardless_of_order() { + for constraint in ["~> 2.0 or ~> 1.0", "~> 1.0 or ~> 2.0"] { + let converted = hex_constraint_to_semver(constraint); + let req = ::semver::VersionReq::parse(&converted).expect(&converted); + assert!( + req.matches(&::semver::Version::parse("2.3.0").unwrap()), + "{constraint} -> {converted} rejected 2.3.0" + ); + } + } + + /// Version components come straight from a manifest, so `u64::MAX` is reachable and + /// the `+ 1` that bounds a `~>` range must not panic on it. + #[test] + fn an_enormous_version_component_does_not_overflow() { + let _ = hex_constraint_to_semver("~> 18446744073709551615.0.0"); + let _ = hex_constraint_to_semver("~> 18446744073709551615.18446744073709551615"); + } } diff --git a/crates/dependable-core/src/semver/maven.rs b/crates/dependable-core/src/semver/maven.rs index e8c79ae..cf63a40 100644 --- a/crates/dependable-core/src/semver/maven.rs +++ b/crates/dependable-core/src/semver/maven.rs @@ -101,6 +101,13 @@ pub fn maven_to_semver(version: &str) -> Option { /// /// A union (`(,1.0],[1.2,)`) is not expressible in `semver::VersionReq`; the last /// (newest-allowing) interval is kept, matching the Hex translation. +/// +/// Anything else returns the **empty string**, the signal +/// [`try_to_semver_constraint`](crate::semver::try_to_semver_constraint) reads as a +/// failed translation. A `+` wildcard in a shape this does not recognise used to widen +/// to `"*"`, which matches every version and so reports the newest release as satisfying +/// a constraint nobody read — a confident `up to date` is the worst available answer for +/// a constraint that was never understood. #[must_use] pub fn maven_constraint_to_semver(constraint: &str) -> String { let c = constraint.trim(); @@ -112,7 +119,7 @@ pub fn maven_constraint_to_semver(constraint: &str) -> String { return "*".to_string(); } if c.contains('+') { - return wildcard_range(c).unwrap_or_else(|| "*".to_string()); + return wildcard_range(c).unwrap_or_default(); } if c.starts_with('[') || c.starts_with('(') { return interval_range(c).unwrap_or_default(); diff --git a/crates/dependable-core/src/semver/mod.rs b/crates/dependable-core/src/semver/mod.rs index 054f9a8..f969062 100644 --- a/crates/dependable-core/src/semver/mod.rs +++ b/crates/dependable-core/src/semver/mod.rs @@ -7,7 +7,8 @@ pub mod normalize; pub mod nuget; pub mod python; -pub use checker::{Evaluation, check_version, to_version_req}; +pub use checker::{Evaluation, check_version, check_version_for, to_version_req}; pub use normalize::{ - UnstableFilter, is_prerelease, normalize_constraint, normalize_version, to_semver_constraint, + UnstableFilter, is_dialect_tag, is_prerelease, normalize_constraint, + normalize_range_constraint, normalize_version, to_semver_constraint, try_to_semver_constraint, }; diff --git a/crates/dependable-core/src/semver/normalize.rs b/crates/dependable-core/src/semver/normalize.rs index 132e14e..c404118 100644 --- a/crates/dependable-core/src/semver/normalize.rs +++ b/crates/dependable-core/src/semver/normalize.rs @@ -81,24 +81,37 @@ const PYTHON_PRERELEASE: &[&str] = &[ ".experimental", ".canary", ".pre", - ".post", ]; /// Whether `version` looks like a pre-release / unstable version for `ecosystem`. /// -/// Uses a case-insensitive substring match against a marker set, plus Python's -/// implicit forms (`1.0a1`, `1.0b2`, `1.0rc1`) and the JVM's, which no substring -/// list can cover. +/// A version that parses as semver answers for itself — that is the definition, and it +/// is exact in both directions. The substring test alone was wrong both ways: +/// `1.0.0-M1` and `1.0.0-unstable.3` are pre-releases carrying no listed marker, and +/// `1.2.3+build-rc` is a *stable* release whose build metadata happens to contain one. /// -/// The marker list is hyphen-prefixed, which is the spelling every ecosystem that -/// publishes semver uses. Maven does not: it separates a qualifier with a dot as -/// readily as with a hyphen (`6.0.0.M1`, `8.0.0.Beta1`, `5.3.0.RC1`) and -/// abbreviates the word (`2.0-M1`, `2.0-CR1`, `2.0-a1`). `-SNAPSHOT` is the one -/// form the universal list happens to catch, so before -/// [`maven::is_prerelease`](crate::semver::maven::is_prerelease) was consulted the -/// default `Exclude` filter offered a beta as the latest stable release. +/// The marker list is the fallback for the many ecosystem versions that are *not* +/// semver (PEP 440, NuGet's four-part versions, Go's `v` prefix), where a substring is +/// the best available signal, plus Python's implicit forms (`1.0a1`, `1.0b2`, `1.0rc1`). +/// +/// The JVM needs both exceptions. Maven separates a qualifier with a dot as readily as +/// with a hyphen (`6.0.0.M1`, `8.0.0.Beta1`, `5.3.0.RC1`) and abbreviates the word +/// (`2.0-M1`, `2.0-CR1`, `2.0-a1`), none of which the marker list spells, so +/// [`maven::is_prerelease`](crate::semver::maven::is_prerelease) is consulted; and it +/// treats a trailing word as a build variant rather than a preview, so `32.1.3-android` +/// — which *is* semver with a pre-release segment — is a release, and the semver +/// reading is skipped there. Before both, the default `Exclude` filter offered a beta +/// as the latest stable release. #[must_use] pub fn is_prerelease(version: &str, ecosystem: Ecosystem) -> bool { + // The semver reading is skipped for the JVM: `32.1.3-android` parses as semver with + // a pre-release segment, but under Maven's order that trailing word is a build + // variant of a release. Maven's tokenizer, consulted below, is the authority there. + if !matches!(ecosystem, Ecosystem::Jvm) + && let Ok(parsed) = ::semver::Version::parse(version.trim_start_matches('v')) + { + return !parsed.pre.is_empty(); + } let lower = version.to_ascii_lowercase(); if UNIVERSAL_PRERELEASE.iter().any(|m| lower.contains(m)) { return true; @@ -108,10 +121,9 @@ pub fn is_prerelease(version: &str, ecosystem: Ecosystem) -> bool { PYTHON_PRERELEASE.iter().any(|m| lower.contains(m)) || python_implicit_prerelease(&lower) } - // Maven's qualifiers are tokens, not suffixes, so they are recognized by - // the tokenizer that already models Maven's order rather than by substring - // — which would read `9.4.51.v20230217` (a dated build of a release) or - // `32.1.3-android` (a build variant) as unstable. + // Maven's qualifiers are tokens, not suffixes, so they are recognized by the + // tokenizer that already models Maven's order rather than by substring — which + // would read `9.4.51.v20230217` (a dated build of a release) as unstable. Ecosystem::Jvm => crate::semver::maven::is_prerelease(version), _ => false, } @@ -154,9 +166,186 @@ pub fn normalize_constraint(constraint: &str) -> String { } } +/// Composer's stability flags. They qualify which *stability* of a release the +/// constraint admits, never which versions, so `2.8.*@dev` admits exactly what +/// `2.8.*` admits and a bare `@dev` admits everything. +const STABILITY_FLAGS: &[&str] = &["dev", "alpha", "beta", "rc", "stable"]; + +/// Strip a trailing Composer stability flag, returning the range that carries it. +fn strip_stability_flag(constraint: &str) -> &str { + match constraint.rsplit_once('@') { + Some((range, flag)) + if STABILITY_FLAGS + .iter() + .any(|known| flag.eq_ignore_ascii_case(known)) => + { + range.trim() + } + _ => constraint, + } +} + +/// Translate a range written in the npm / Composer / Dart / Cargo dialect into a +/// `semver::VersionReq`-compatible string, returning the input unchanged when there +/// is nothing to translate. +/// +/// The `semver` crate parses Cargo's spelling of a range and only Cargo's, so the +/// ordinary forms the other three ecosystems document — a `||` union, comparators +/// separated by spaces rather than commas, npm's hyphen range, a Composer stability +/// flag, Dart's `any` — all reached `VersionReq::parse` verbatim and failed. Those are +/// *valid* constraints this crate simply had no front-end for, and treating them as +/// unreadable input made a dependency the manifest declares correctly the reason a +/// whole run could not answer its gate. +/// +/// Only a translation that itself parses is adopted. Anything else is handed back +/// untouched for [`check_version`](crate::semver::check_version) to classify, which is +/// what keeps a dist-tag (`latest`, `next`) and genuine garbage (`^^^bogus`) +/// distinguishable from each other downstream. +#[must_use] +pub fn normalize_range_constraint(constraint: &str) -> String { + let trimmed = constraint.trim(); + if trimmed.is_empty() { + return String::new(); + } + // A dist-tag names a channel rather than a range. `check_version` resolves the one + // that tracks the newest release; the rest have no range reading at all. + if trimmed == "latest" { + return trimmed.to_owned(); + } + let core = strip_stability_flag(trimmed); + // A bare `@dev` is "any version, dev stability" — the stability half is not a + // version constraint, and what is left constrains nothing. + if core.is_empty() { + return "*".to_owned(); + } + // Dart spells "no constraint" as `any`, and `pubspec.yaml` accepts it deliberately. + // `*` is the same statement in a spelling `VersionReq` reads. + if core.eq_ignore_ascii_case("any") || core == "x" || core == "X" { + return "*".to_owned(); + } + translate_union(core).unwrap_or_else(|| normalize_constraint(constraint)) +} + +/// Pick one branch of a `||` union: the one admitting the highest versions. +/// +/// `VersionReq` has no union, so a branch has to be chosen. The highest lower bound +/// is the same choice [`hex_constraint_to_semver`](crate::semver::elixir::hex_constraint_to_semver) +/// and [`maven_constraint_to_semver`](crate::semver::maven::maven_constraint_to_semver) +/// already make, and for the reason they give: a union is written to widen what is +/// accepted, so resolving it to the oldest branch reports every release above that +/// branch as out of range. +fn translate_union(core: &str) -> Option { + let mut best: Option<((u64, u64, u64), String)> = None; + for branch in core.split("||") { + let branch = branch.trim(); + if branch.is_empty() { + continue; + } + let Some(translated) = translate_conjunction(branch) else { + continue; + }; + let Ok(req) = ::semver::VersionReq::parse(&translated) else { + continue; + }; + let bound = lower_bound(&req); + if best + .as_ref() + .is_none_or(|(best_bound, _)| bound > *best_bound) + { + best = Some((bound, translated)); + } + } + best.map(|(_, translated)| translated) +} + +/// The lowest version a requirement admits, used only to rank union branches. +fn lower_bound(req: &::semver::VersionReq) -> (u64, u64, u64) { + use ::semver::Op; + req.comparators + .iter() + .filter(|c| { + matches!( + c.op, + Op::Greater | Op::GreaterEq | Op::Exact | Op::Caret | Op::Tilde + ) + }) + .map(|c| (c.major, c.minor.unwrap_or(0), c.patch.unwrap_or(0))) + .max() + .unwrap_or((0, 0, 0)) +} + +/// Translate one branch of a union: an intersection of comparators, which npm and +/// Composer separate with spaces and Cargo with commas, or npm's hyphen range. +fn translate_conjunction(branch: &str) -> Option { + if let Some((low, high)) = branch.split_once(" - ") { + return hyphen_range(low.trim(), high.trim()); + } + let mut parts: Vec = Vec::new(); + let mut pending_op: Option<&str> = None; + for token in branch.split([',', ' ', '\t']).filter(|t| !t.is_empty()) { + // `>= 1.2.3` writes the operator and the version as two tokens. + if token + .chars() + .all(|c| matches!(c, '>' | '<' | '=' | '^' | '~' | '!')) + { + pending_op = Some(token); + continue; + } + let version = normalize_constraint(token); + parts.push(match pending_op.take() { + Some(op) => format!("{op}{version}"), + None => version, + }); + } + (!parts.is_empty() && pending_op.is_none()).then(|| parts.join(", ")) +} + +/// npm's hyphen range: `1.2.3 - 2.3.4` admits both endpoints. +/// +/// A partial upper bound bounds the segment it stops at rather than padding with +/// zeros — npm reads `1.2.3 - 2.3` as every `2.3.x`, so padding it to `<=2.3.0` +/// would exclude releases the author accepted. +fn hyphen_range(low: &str, high: &str) -> Option { + if high.contains(char::is_whitespace) { + return None; + } + let low = normalize_version(low); + ::semver::Version::parse(&low).ok()?; + let high = high.strip_prefix(['v', 'V']).unwrap_or(high); + let nums: Vec = high + .split('.') + .map(|s| s.parse().ok()) + .collect::>()?; + let upper = match nums.as_slice() { + [major] => format!("<{}.0.0", major.checked_add(1)?), + [major, minor] => format!("<{major}.{}.0", minor.checked_add(1)?), + [major, minor, patch] => format!("<={major}.{minor}.{patch}"), + _ => return None, + }; + Some(format!(">={low}, {upper}")) +} + +/// Whether `constraint` names something — a channel, a branch alias — rather than +/// stating a range badly. +/// +/// The two have to be told apart because they call for opposite answers. `^^^bogus` +/// is not a constraint at all and the run should say so; npm's `next`, Composer's +/// `dev-master`, and a Git branch alias are ordinary declarations in dialects this +/// crate has no front-end for, and failing a build over one punishes the user for a +/// gap that is ours. A name is spelled the way a name is spelled: it opens with a +/// letter and carries no comparison operator anywhere. +#[must_use] +pub fn is_dialect_tag(constraint: &str) -> bool { + let c = constraint.trim(); + c.starts_with(|ch: char| ch.is_ascii_alphabetic()) + && c.chars() + .all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_' | '.' | '/')) +} + /// Convert a constraint into a `semver::VersionReq`-compatible string for the -/// given ecosystem. Python uses PEP 440 translation; every other ecosystem is -/// already semver-compatible and only needs [`normalize_constraint`]. +/// given ecosystem. Python, NuGet, Hex and Maven translate in dedicated modules; +/// every other ecosystem writes a dialect close enough to Cargo's that +/// [`normalize_range_constraint`] covers it. #[must_use] pub fn to_semver_constraint(constraint: &str, ecosystem: Ecosystem) -> String { match ecosystem { @@ -164,8 +353,42 @@ pub fn to_semver_constraint(constraint: &str, ecosystem: Ecosystem) -> String { Ecosystem::CSharp => crate::semver::nuget::nuget_constraint_to_semver(constraint), Ecosystem::Elixir => crate::semver::elixir::hex_constraint_to_semver(constraint), Ecosystem::Jvm => crate::semver::maven::maven_constraint_to_semver(constraint), - _ => normalize_constraint(constraint), + _ => normalize_range_constraint(constraint), + } +} + +/// Convert a constraint for `semver`, or `None` when the ecosystem's dialect could +/// not be expressed as a `semver::VersionReq`. +/// +/// All four dedicated translators signal failure the same way — by returning an empty +/// string. [`maven_constraint_to_semver`](crate::semver::maven::maven_constraint_to_semver) +/// and [`nuget_constraint_to_semver`](crate::semver::nuget::nuget_constraint_to_semver) +/// do so for an unreadable version, a malformed interval, or a wildcard shape they do +/// not recognise; [`pep440_constraint_to_semver`](crate::semver::python::pep440_constraint_to_semver) +/// does so once every clause has been dropped; and +/// [`hex_constraint_to_semver`](crate::semver::elixir::hex_constraint_to_semver) does so +/// when no union branch converted. An empty result is therefore ambiguous on its own: +/// it means "the author declared no constraint" *and* "we could not read the constraint +/// the author declared", and the checker treating the second as the first turned it +/// into `*` — which resolves to the newest release and reports `up to date`, the one +/// answer a constraint that was never understood must not give. +/// +/// The two are told apart by what went in: an empty result from a **non-empty** input +/// is a failed translation. That reading is only sound because every translator's +/// failure path is spelled this one way — three of them used to widen to `"*"` or echo +/// their input verbatim instead, which is a failure this guard cannot see and which +/// produces exactly the confident `up to date` it exists to prevent. +/// +/// The remaining ecosystems have no dedicated translator and never signal failure here: +/// [`normalize_range_constraint`] hands an untranslatable range back unchanged, and +/// [`check_version`](crate::semver::check_version) classifies it. +#[must_use] +pub fn try_to_semver_constraint(constraint: &str, ecosystem: Ecosystem) -> Option { + let translated = to_semver_constraint(constraint, ecosystem); + if translated.trim().is_empty() && !constraint.trim().is_empty() { + return None; } + Some(translated) } /// Normalize a concrete version string: strip a leading `v`/`V` and pad partial @@ -218,6 +441,198 @@ mod tests { list.iter().map(|s| (*s).to_string()).collect() } + /// Assert that `constraint` translates to something `VersionReq` actually accepts. + /// + /// `is_some()` alone was the false assurance that let this whole class of defect + /// ship: a translation is only a translation if the result parses, and every witness + /// below returned `Some` while failing `VersionReq::parse` downstream. + fn assert_translates(constraint: &str, ecosystem: Ecosystem) -> ::semver::VersionReq { + let translated = try_to_semver_constraint(constraint, ecosystem) + .unwrap_or_else(|| panic!("{ecosystem:?} `{constraint}` translated to nothing")); + ::semver::VersionReq::parse(&translated).unwrap_or_else(|e| { + panic!("{ecosystem:?} `{constraint}` translated to `{translated}`, which is not a requirement: {e}") + }) + } + + /// Every ecosystem's spelling of "any version", plus the ordinary forms around it. + /// A translator that drops one of these hands the checker an empty string, which + /// [`try_to_semver_constraint`] then reports as a constraint it could not read — + /// which is how `requests = "*"` became `undetermined` for every Poetry project + /// with an unpinned dependency. + #[test] + fn a_constraint_that_states_something_never_translates_to_nothing() { + let cases: &[(Ecosystem, &str)] = &[ + (Ecosystem::Python, "*"), + (Ecosystem::Python, ">=1.0"), + (Ecosystem::Python, "==1.2.3"), + (Ecosystem::Python, "~=1.4"), + (Ecosystem::Python, "==1.0.*"), + (Ecosystem::Python, "^1.2"), + (Ecosystem::Python, "~1.2"), + (Ecosystem::Python, ">=1.0,<2.0"), + (Ecosystem::Python, "1.2.3"), + (Ecosystem::Python, "===1.0"), + (Ecosystem::CSharp, "*"), + (Ecosystem::CSharp, "1.0.0"), + (Ecosystem::CSharp, "[1.0,2.0)"), + (Ecosystem::CSharp, "1.*"), + (Ecosystem::Jvm, "+"), + (Ecosystem::Jvm, "latest.release"), + (Ecosystem::Jvm, "1.+"), + (Ecosystem::Jvm, "[1.0,2.0)"), + (Ecosystem::Elixir, "~> 1.0"), + (Ecosystem::Elixir, ">= 1.0.0"), + (Ecosystem::Rust, "*"), + (Ecosystem::Npm, "*"), + (Ecosystem::Npm, "1.x"), + (Ecosystem::Php, "*"), + (Ecosystem::Dart, "any"), + ]; + for (ecosystem, constraint) in cases { + assert_translates(constraint, *ecosystem); + } + } + + /// The ranges npm, Composer and Dart document, none of which the `semver` crate + /// parses on its own. + /// + /// Every one of these used to reach `VersionReq::parse` verbatim and fail, which the + /// checker recorded as a dependency the run could not evaluate — and a single one of + /// those makes the whole repository unanswerable under the shipped Action's default + /// `fail-on: vulnerable`. They are valid constraints; the gap was ours. + #[test] + fn the_documented_range_dialects_translate_into_something_that_parses() { + let v = |s: &str| ::semver::Version::parse(s).expect(s); + // (ecosystem, constraint, versions it must admit, versions it must not) + let cases: &[(Ecosystem, &str, &[&str], &[&str])] = &[ + ( + Ecosystem::Npm, + ">=16.8.0 <19.0.0", + &["16.8.0", "18.3.1"], + &["16.7.0", "19.0.0"], + ), + ( + Ecosystem::Npm, + "^15.0.0 || ^16.0.0", + &["16.8.0"], + &["17.0.0"], + ), + ( + Ecosystem::Npm, + "1.2.3 - 2.3.4", + &["1.2.3", "2.0.0", "2.3.4"], + &["1.2.2", "2.3.5"], + ), + // A partial upper bound bounds the segment it stops at: `2.3` is every 2.3.x. + ( + Ecosystem::Npm, + "1.2.3 - 2.3", + &["2.3.9"], + &["1.2.2", "2.4.0"], + ), + (Ecosystem::Npm, ">= 1.2.3", &["1.2.3", "9.0.0"], &["1.2.2"]), + // Dart's `any` is "no constraint", which is what `*` says. + (Ecosystem::Dart, "any", &["0.0.1", "9.9.9"], &[]), + // A Composer stability flag qualifies stability, not versions. + (Ecosystem::Php, "2.8.*@dev", &["2.8.0", "2.8.9"], &["2.9.0"]), + (Ecosystem::Php, "@dev", &["0.1.0", "9.9.9"], &[]), + ( + Ecosystem::Php, + "^1.0 || ^2.0", + &["2.4.0"], + &["3.0.0", "0.9.0"], + ), + ]; + for (ecosystem, constraint, admits, rejects) in cases { + let req = assert_translates(constraint, *ecosystem); + for version in *admits { + assert!( + req.matches(&v(version)), + "`{constraint}` rejected {version}" + ); + } + for version in *rejects { + assert!( + !req.matches(&v(version)), + "`{constraint}` admitted {version}" + ); + } + } + } + + /// A dialect this crate cannot express must reach the checker as a failed + /// translation — an empty result — or as a name the checker recognises as a name. + /// Either way the dependency is `undetermined`, which claims nothing; what it must + /// never be is a range that silently means something else. + #[test] + fn a_dialect_with_no_front_end_is_never_mistaken_for_a_range() { + // Failed translations: the translator drops everything it could not read. + for (ecosystem, constraint) in [ + (Ecosystem::Elixir, "!= 1.0.0"), + (Ecosystem::Python, "!=2.31.0"), + ] { + assert_eq!( + try_to_semver_constraint(constraint, ecosystem), + None, + "{ecosystem:?} `{constraint}`" + ); + } + // Names: handed back untouched, and recognised as names rather than as ranges + // somebody got wrong. + for (ecosystem, constraint) in [ + (Ecosystem::Npm, "next"), + (Ecosystem::Npm, "beta"), + (Ecosystem::Npm, "canary"), + (Ecosystem::Php, "dev-master"), + ] { + let translated = try_to_semver_constraint(constraint, ecosystem) + .expect("a name is handed back, not dropped"); + assert!( + ::semver::VersionReq::parse(&translated).is_err(), + "{ecosystem:?} `{constraint}` must not be read as a range" + ); + assert!(is_dialect_tag(&translated), "{ecosystem:?} `{constraint}`"); + } + // `latest` is the one name with a range reading, and the checker owns it. + assert_eq!( + try_to_semver_constraint("latest", Ecosystem::Npm).as_deref(), + Some("latest") + ); + } + + /// The other side of [`is_dialect_tag`]: operators that announce a range and then do + /// not spell one are not names, and must stay a hard error. + #[test] + fn garbage_wearing_range_operators_is_not_a_name() { + for constraint in ["^^^bogus", ">=<1.0.0", "~~", "1.2.3", "*", "@dev"] { + assert!(!is_dialect_tag(constraint), "{constraint}"); + } + for constraint in ["next", "dev-master", "latest", "release/2.x"] { + assert!(is_dialect_tag(constraint), "{constraint}"); + } + } + + /// The other side of the same coin: a dialect semver genuinely cannot express must + /// keep coming back as a failed translation, or the checker resolves it to `*` and + /// reports `up to date` for a constraint nobody read. + #[test] + fn a_constraint_semver_cannot_express_stays_a_failed_translation() { + // Exclusion has no semver spelling; `$(Version)` is an MSBuild property, not a + // version; `LATEST`/`RELEASE` are Maven's server-resolved tags. + for (ecosystem, constraint) in [ + (Ecosystem::Python, "!=1.5"), + (Ecosystem::CSharp, "$(Version)"), + (Ecosystem::Jvm, "LATEST"), + (Ecosystem::Jvm, "RELEASE"), + ] { + assert_eq!( + try_to_semver_constraint(constraint, ecosystem), + None, + "{ecosystem:?} `{constraint}`" + ); + } + } + #[test] fn universal_prerelease_markers() { for v in ["1.0.0-alpha", "1.0.0-RC1", "2.0.0-beta.3", "1.0.0-SNAPSHOT"] { @@ -229,7 +644,7 @@ mod tests { #[test] fn python_specific_prereleases() { - for v in ["1.0a1", "1.0b2", "1.0rc1", "1.0.dev3", "1.0.post1"] { + for v in ["1.0a1", "1.0b2", "1.0rc1", "1.0.dev3"] { assert!(is_prerelease(v, Ecosystem::Python), "{v}"); } // The `[ab]\d` rule must not fire on non-Python ecosystems. @@ -238,6 +653,37 @@ mod tests { assert!(!is_prerelease("1.0.0", Ecosystem::Python)); } + /// PEP 440 orders `1.0 < 1.0.post1`: a post-release is a *later* release of the same + /// version, not a preview of it. Treating it as unstable hid it from the default + /// filter, so a project on `1.0` was told it was current. + #[test] + fn a_python_post_release_is_not_a_prerelease() { + for v in ["1.0.post1", "1.0.post2", "2.1.post0"] { + assert!(!is_prerelease(v, Ecosystem::Python), "{v}"); + } + // A post-release of a pre-release is still a pre-release. + assert!(is_prerelease("1.0rc1.post1", Ecosystem::Python)); + } + + /// The old substring test was wrong in both directions, and each direction cost + /// something: a missed pre-release is recommended as an upgrade, and a stable + /// release whose build metadata happens to read `-rc` is hidden from one. + #[test] + fn semver_versions_are_classified_by_parsing_not_by_substring() { + for v in [ + "1.0.0-M1", + "1.0.0-CR2", + "1.0.0-unstable.3", + "1.0.0-0", + "4.0.0-insiders", + ] { + assert!(is_prerelease(v, Ecosystem::Rust), "{v}"); + } + for v in ["1.2.3+build-rc", "1.2.3+alpha", "1.0.0", "10.20.30"] { + assert!(!is_prerelease(v, Ecosystem::Rust), "{v}"); + } + } + /// The marker list is hyphen-prefixed; Maven's qualifiers are not. Under the /// default `Exclude` filter this offered `8.0.0.Beta1` as Hibernate's latest /// release and `7.1.0.M1` as Spring's. diff --git a/crates/dependable-core/src/semver/nuget.rs b/crates/dependable-core/src/semver/nuget.rs index ed99c6f..28e9735 100644 --- a/crates/dependable-core/src/semver/nuget.rs +++ b/crates/dependable-core/src/semver/nuget.rs @@ -47,6 +47,13 @@ pub fn nuget_to_semver(version: &str) -> Option { /// Handles interval notation (`[1.0,2.0)`, `[1.0]`, `(1.0,)`, `(,2.0]`), floating /// wildcards (`*`, `1.*`, `1.0.*`), and a bare version (`1.0`), which NuGet reads /// as an inclusive minimum (`>=1.0`). +/// +/// Anything else returns the **empty string**, the signal +/// [`try_to_semver_constraint`](crate::semver::try_to_semver_constraint) reads as a +/// failed translation. A wildcard shape this does not recognise used to widen to `"*"`, +/// which matches every version and so reports the newest release as satisfying a +/// constraint nobody read — a confident `up to date` is the worst available answer for a +/// constraint that was never understood. #[must_use] pub fn nuget_constraint_to_semver(constraint: &str) -> String { let c = constraint.trim(); @@ -54,7 +61,7 @@ pub fn nuget_constraint_to_semver(constraint: &str) -> String { return String::new(); } if c.contains('*') { - return floating_range(c).unwrap_or_else(|| "*".to_string()); + return floating_range(c).unwrap_or_default(); } if c.starts_with('[') || c.starts_with('(') { return interval_range(c).unwrap_or_default(); diff --git a/crates/dependable-core/src/semver/python.rs b/crates/dependable-core/src/semver/python.rs index eb7de21..b739e0d 100644 --- a/crates/dependable-core/src/semver/python.rs +++ b/crates/dependable-core/src/semver/python.rs @@ -33,20 +33,38 @@ pub fn pep440_to_semver(version: &str) -> Option { let patch = nums.get(2).copied().unwrap_or("0"); let core = format!("{major}.{minor}.{patch}"); - let pre = convert_suffix(suffix); - if pre.is_empty() { - Some(core) - } else { - Some(format!("{core}-{pre}")) + // PEP 440 orders `1.0 < 1.0.post1 < 1.0.1`. Semver has no identifier that sorts + // *above* a release, so a post-release becomes build metadata: it compares equal to + // its base rather than — as a pre-release identifier — below it. Equal is the + // closest semver can get, and it is the safe side: `1.0.post1` is no longer hidden + // by the pre-release filter, and no longer makes `1.0` look like the newer release. + let (pre, post) = convert_suffix(suffix); + let mut out = core; + if !pre.is_empty() { + out.push('-'); + out.push_str(&pre); } + if !post.is_empty() { + out.push('+'); + out.push_str(&post); + } + Some(out) } -/// Convert a PEP 440 pre/post/dev suffix into a semver pre-release identifier -/// (`a1` → `alpha.1`, `rc1` → `rc.1`, `.dev2` → `dev.2`, `.post1` → `post.1`). -fn convert_suffix(suffix: &str) -> String { +/// Split a PEP 440 suffix into its semver pre-release and build-metadata halves. +/// +/// `a1` → (`alpha.1`, ``), `rc1` → (`rc.1`, ``), `.dev2` → (`dev.2`, ``), +/// `.post1` → (``, `post.1`), `rc1.post2` → (`rc.1`, `post.2`). +/// +/// `dev` and the `a`/`b`/`rc` family sort below the release and are pre-release +/// identifiers; `post` sorts above it and has no pre-release spelling. +fn convert_suffix(suffix: &str) -> (String, String) { let lower = suffix.to_ascii_lowercase(); let bytes = lower.as_bytes(); - let mut parts: Vec = Vec::new(); + let mut pre: Vec = Vec::new(); + let mut post: Vec = Vec::new(); + // Digits belong to the token they follow, so the group is carried across segments. + let mut in_post = false; let mut i = 0; while i < bytes.len() { let c = bytes[i]; @@ -63,18 +81,19 @@ fn convert_suffix(suffix: &str) -> String { "dev" => "dev", other => other, }; - parts.push(canon.to_string()); + in_post = canon == "post"; + if in_post { &mut post } else { &mut pre }.push(canon.to_string()); } else if c.is_ascii_digit() { let start = i; while i < bytes.len() && bytes[i].is_ascii_digit() { i += 1; } - parts.push(lower[start..i].to_string()); + if in_post { &mut post } else { &mut pre }.push(lower[start..i].to_string()); } else { i += 1; // separators: . - _ } } - parts.join(".") + (pre.join("."), post.join(".")) } /// Convert a PEP 440 (or Poetry) constraint into a `semver::VersionReq` string. @@ -97,6 +116,15 @@ pub fn pep440_constraint_to_semver(constraint: &str) -> String { const OPERATORS: &[&str] = &["===", "==", "~=", "!=", ">=", "<=", "^", "~", ">", "<", "="]; fn convert_clause(clause: &str) -> Option { + // PEP 440's — and Poetry's — explicit "any version", and the most common way to + // write an unpinned dependency. It matches no operator and holds no numeric release, + // so it used to be dropped, leaving an empty translation from a non-empty input; + // once an empty translation meant "we could not read this", `requests = "*"` became + // `undetermined` — excluded from `--fail-on outdated` and failing `--fail-on any`. + // The NuGet and Maven translators already map their own wildcards to `*`. + if clause == "*" { + return Some("*".to_string()); + } for op in OPERATORS { if let Some(rest) = clause.strip_prefix(op) { return convert_op(op, rest.trim()); @@ -118,7 +146,10 @@ fn convert_op(op: &str, version: &str) -> Option { "=" => pep440_to_semver(version).map(|v| format!("={v}")), "~=" => compatible_release(version), "!=" => None, // exclusion is not expressible in semver - "^" | "~" => Some(format!("{op}{version}")), // Poetry / semver-native + // Poetry / semver-native. The operand still has to be normalized: Poetry accepts + // a full PEP 440 version here, and `^1.0.post1` handed through verbatim is not a + // requirement `semver` can parse. + "^" | "~" => pep440_to_semver(version).map(|v| format!("{op}{v}")), ">=" | "<=" | ">" | "<" => pep440_to_semver(version).map(|v| format!("{op}{v}")), _ => None, } @@ -203,7 +234,9 @@ fn pad_to_semver(nums: &[u64]) -> String { fn bump_last(nums: &[u64]) -> Option { let (last, head) = nums.split_last()?; let mut out: Vec = head.to_vec(); - out.push(last + 1); + // Segments come straight from a manifest, so `u64::MAX` is reachable input and a + // plain `+ 1` panics in debug and wraps in release. + out.push(last.saturating_add(1)); Some(pad_to_semver(&out)) } @@ -242,9 +275,35 @@ mod tests { assert_eq!(pep440_to_semver("1.0b2").as_deref(), Some("1.0.0-beta.2")); assert_eq!(pep440_to_semver("1.0rc1").as_deref(), Some("1.0.0-rc.1")); assert_eq!(pep440_to_semver("1.0.dev3").as_deref(), Some("1.0.0-dev.3")); + } + + /// A post-release must not become a pre-release identifier: semver sorts any + /// pre-release *below* its base version, which is the exact inverse of PEP 440 and + /// made `1.0.post1` look older than `1.0`. + #[test] + fn a_post_release_never_sorts_below_its_base() { assert_eq!( pep440_to_semver("1.0.post1").as_deref(), - Some("1.0.0-post.1") + Some("1.0.0+post.1") + ); + assert_eq!( + pep440_to_semver("1.0.rev2").as_deref(), + Some("1.0.0+post.2") + ); + + let base = ::semver::Version::parse("1.0.0").unwrap(); + let post = ::semver::Version::parse(&pep440_to_semver("1.0.post1").unwrap()).unwrap(); + assert!(post >= base, "{post} sorted below {base}"); + + // A pre-release carrying a post segment keeps both, in the right halves. + assert_eq!( + pep440_to_semver("1.0rc1.post2").as_deref(), + Some("1.0.0-rc.1+post.2") + ); + let rc = ::semver::Version::parse("1.0.0-rc.1+post.2").unwrap(); + assert!( + rc < base, + "a post-release of an rc is still below the release" ); } @@ -270,11 +329,46 @@ mod tests { assert_eq!(pep440_constraint_to_semver("~=1.4.2"), ">=1.4.2, <1.5.0"); } + /// Poetry's unpinned dependency. This translated to the empty string, which the + /// failed-translation heuristic then read as a constraint it could not parse. + #[test] + fn a_bare_wildcard_is_any_version() { + assert_eq!(pep440_constraint_to_semver("*"), "*"); + assert_eq!(pep440_constraint_to_semver(" * "), "*"); + assert_eq!( + pep440_constraint_to_semver("* ; python_version < \"3.8\""), + "*" + ); + assert!(::semver::VersionReq::parse("*").is_ok()); + } + #[test] fn passes_through_poetry_operators_and_drops_exclusions() { assert_eq!(pep440_constraint_to_semver("^1.2.3"), "^1.2.3"); - assert_eq!(pep440_constraint_to_semver("~1.2"), "~1.2"); + assert_eq!(pep440_constraint_to_semver("~1.2"), "~1.2.0"); // `!=` is dropped, leaving the expressible clauses. assert_eq!(pep440_constraint_to_semver(">=1.0,!=1.5"), ">=1.0.0"); } + + /// Every other operator normalizes its operand; `^`/`~` used to hand theirs through + /// verbatim, so a full PEP 440 version behind one produced a requirement `semver` + /// cannot parse — which the checker then reported as an available upgrade. + #[test] + fn poetry_caret_and_tilde_normalize_their_operand() { + for constraint in ["^1.0.post1", "~1.0a1", "^1.0", "~2"] { + let converted = pep440_constraint_to_semver(constraint); + assert!( + ::semver::VersionReq::parse(&converted).is_ok(), + "{constraint} -> {converted} does not parse" + ); + } + } + + /// Reachable straight from a requirements file; `+ 1` on a parsed segment panicked + /// in debug and wrapped in release. + #[test] + fn an_enormous_version_component_does_not_overflow() { + let _ = pep440_constraint_to_semver("==18446744073709551615.*"); + let _ = pep440_constraint_to_semver("~=18446744073709551615.0"); + } } diff --git a/crates/dependable-fetch/src/check.rs b/crates/dependable-fetch/src/check.rs index 8446710..af05511 100644 --- a/crates/dependable-fetch/src/check.rs +++ b/crates/dependable-fetch/src/check.rs @@ -7,14 +7,15 @@ //! [`crate::OsvClient`]) remain public for callers who want to compose by hand. use std::collections::{HashMap, HashSet}; +use std::hash::{DefaultHasher, Hash, Hasher}; use std::path::{Path, PathBuf}; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; use dependable_core::{ - CheckResult, DependencyStatus, Ecosystem, Evaluation, Item, LockfileKind, ManifestKind, - PackageSource, UnstableFilter, apply_lockfile, check_version, parse, parse_lockfile_kind, - resolve_workspace_inheritance, to_semver_constraint, + CheckResult, DependencyStatus, Ecosystem, ErrorOrigin, Evaluation, Item, LockfileKind, + ManifestKind, PackageSource, UnstableFilter, apply_lockfile, check_version_for, parse, + parse_lockfile_kind, resolve_workspace_inheritance, }; use futures::stream::{self, StreamExt}; use semver::Version as SemverVersion; @@ -97,6 +98,23 @@ pub struct ManifestCheck { pub results: Vec, /// Non-fatal degradations (e.g. an OSV outage that skipped vulnerability data). pub warnings: Vec, + /// Whether the vulnerability scan was requested but did not complete. + /// + /// Separate from [`warnings`](Self::warnings), and typed, because a caller has to be + /// able to *act* on it rather than parse prose: an empty advisory list means "nothing + /// was found" and "nothing was looked for" alike, and a `--fail-on vulnerable` gate + /// reading the first when the second is true reports a clean build it never checked. + pub vulnerability_scan_failed: bool, + /// Whether any registry lookup failed for a reason other than the package not + /// existing. + /// + /// The difference a gate turns on. A 404 is a per-dependency fact — a private or + /// internal package, one served by a registry this run does not route to, a deleted + /// package — and it says nothing about the dependencies that *were* resolved. A + /// timeout, a refused connection, a 5xx, or an undecodable response is the registry + /// declining to answer, and a `--fail-on` promise cannot be kept from answers that + /// were never given. + pub registry_unreachable: bool, /// The manifest whose `[workspace.dependencies]` govern this one — itself, when it /// declares its own `[workspace]`, else the nearest ancestor that does. /// @@ -190,10 +208,51 @@ struct FetchTask { } /// The result of one fetch task: `(name, cache_key, versions-or-error)`. -type FetchOutcome = (String, String, Result, String>); +/// +/// The error is still typed here, and only stringified once +/// [`Checker::fetch_all`] has read whether it was the registry answering "no such +/// package" or the registry not answering at all. +type FetchOutcome = (String, String, Result, FetchError>); + +/// One fetch that produced no versions, with the provenance a gate turns on kept +/// beside the message. +/// +/// The message alone cannot answer "did a registry say this package does not exist?", +/// and the answer is not cosmetic: a 404 is a permanent per-dependency fact that must +/// not fail a whole build, while an unanswered request leaves every dependency it +/// covered unfounded. Re-deriving either from prose is how they got merged. +#[derive(Debug, Clone)] +pub(crate) struct FetchFailure { + /// Where the failure came from. + pub(crate) origin: ErrorOrigin, + /// What to show the user. + pub(crate) message: String, +} + +impl From<&FetchError> for FetchFailure { + fn from(error: &FetchError) -> Self { + Self { + origin: match error { + FetchError::NotFound(_) => ErrorOrigin::NotFound, + // Everything else is a request that produced no usable answer: a + // timeout, a refused connection, a 5xx, an undecodable body, a document + // listing no versions at all. + _ => ErrorOrigin::Unanswered, + }, + message: error.to_string(), + } + } +} -/// Fetched versions (or a per-package error message) keyed by package name. -type FetchedMap = HashMap, String>>; +/// Fetched versions (or a per-package failure), keyed by `(cache_key, name)`. +/// +/// Keyed by the same pair the fetch tasks are deduplicated by, and for the same reason: +/// a name alone is not unique within a manifest. A `Cargo.toml` naming one crate from +/// crates.io and another of the same name from a private registry, or a `deno.json` +/// importing both `jsr:foo` and `npm:foo`, issues two tasks — and a name-keyed map +/// collapsed them into one slot, so whichever request finished last silently answered +/// for both. +type FetchedMap = HashMap<(String, String), Result, FetchFailure>>; impl Checker { /// Start configuring a checker. @@ -600,17 +659,22 @@ impl Checker { } } - let fetched = self.fetch_all(tasks).await; + let (fetched, registry_unreachable) = self.fetch_all(tasks).await; let mut results: Vec = parsed .items .iter() - .map(|item| evaluate_item(item, &fetched, ecosystem, self.unstable)) + .map(|item| { + let (_, cache_key) = self.route_item(item, &fetcher, ecosystem); + evaluate_item(item, &fetched, &cache_key, ecosystem, self.unstable) + }) .collect(); + let mut vulnerability_scan_failed = false; if let Some(osv) = &self.osv && let Err(e) = scan_vulnerabilities(osv, ecosystem, &mut results).await { warnings.push(format!("vulnerability scan skipped: {e}")); + vulnerability_scan_failed = true; } // License collection is a post-pass over the finished results, shaped @@ -628,6 +692,8 @@ impl Checker { ecosystem, results, warnings, + vulnerability_scan_failed, + registry_unreachable, workspace_root: workspace.map(|(root, _)| root), }; @@ -667,12 +733,23 @@ impl Checker { if let Some(alias) = &item.registry && let Some(fetcher) = self.rust_registries.get(alias) { - return ( - fetcher.clone(), - format!("{}::{alias}", ecosystem.osv_name()), - ); + // `:` is not a legal character in a Windows path component, and this key + // becomes a directory name under the disk-cache root. + return (fetcher.clone(), format!("{}-{alias}", ecosystem.osv_name())); } - (default.clone(), ecosystem.osv_name().to_string()) + ( + default.clone(), + default_cache_key(default.as_ref(), ecosystem), + ) + } + + /// Whether this checker reads and writes the on-disk registry cache. + /// + /// Exposed so a caller — and this crate's own tests — can assert that a checker it + /// did not explicitly opt in has no filesystem side effects. + #[must_use] + pub fn uses_disk_cache(&self) -> bool { + self.disk_cache.is_some() } /// Run every fetch task concurrently, serving and populating the in-process @@ -691,7 +768,14 @@ impl Checker { /// request. The concurrency inside a single manifest is safe — its tasks are /// already deduplicated by `(cache_key, name)` before they get here. Anyone /// parallelising the *manifest* loop must add coalescing here first. - async fn fetch_all(&self, tasks: Vec) -> FetchedMap { + /// Fetch every task's version list, returning the results and whether any lookup + /// failed for a reason other than the package not existing. + /// + /// A 404 is an answer: the package is private, internal, deleted, or served by a + /// registry this run does not route to. Anything else — a timeout, a refused + /// connection, a 5xx, a response that would not decode — is the registry declining + /// to answer, and a gate cannot be honoured from answers that were never given. + async fn fetch_all(&self, tasks: Vec) -> (FetchedMap, bool) { let total = tasks.len(); self.emit(ProgressEvent::Started { total }); @@ -700,14 +784,16 @@ impl Checker { for task in tasks { let key = (task.cache_key.clone(), task.name.clone()); if let Some(versions) = self.versions_cache.get(&key).await { - out.insert(task.name.clone(), Ok(versions)); + out.insert(key, Ok(versions)); } else if let Some(disk) = &self.disk_cache && let Some(versions) = disk.get(&task.cache_key, &task.name).await { // Disk hit: warm the in-process cache so sibling manifests in this // run hit moka instead of re-reading the file. - self.versions_cache.insert(key, versions.clone()).await; - out.insert(task.name.clone(), Ok(versions)); + self.versions_cache + .insert(key.clone(), versions.clone()) + .await; + out.insert(key, Ok(versions)); } else { to_fetch.push(task); } @@ -719,12 +805,10 @@ impl Checker { let progress = self.progress.clone(); let counter = counter.clone(); async move { - let result = task - .fetcher - .fetch_versions(&task.name) - .await - .map(|fetched| fetched.versions) - .map_err(|e| e.to_string()); + let result = + crate::retry::with_retry(|| task.fetcher.fetch_versions(&task.name)) + .await + .map(|fetched| fetched.versions); let done = counter.fetch_add(1, Ordering::Relaxed) + 1; if let Some(p) = &progress { p(ProgressEvent::Advanced { @@ -739,6 +823,7 @@ impl Checker { .collect() .await; + let mut registry_unreachable = false; for (name, cache_key, result) in fetched { if let Ok(versions) = &result { self.versions_cache @@ -748,11 +833,19 @@ impl Checker { disk.put(&cache_key, &name, versions).await; } } - out.insert(name, result); + if let Err(e) = &result + && !matches!(e, FetchError::NotFound(_)) + { + registry_unreachable = true; + } + out.insert( + (cache_key, name), + result.map_err(|e| FetchFailure::from(&e)), + ); } self.emit(ProgressEvent::Finished); - out + (out, registry_unreachable) } fn emit(&self, event: ProgressEvent) { @@ -791,20 +884,46 @@ fn undeclared_inheritance(items: &[Item], root: &Path) -> Vec { /// Evaluate one parsed item against the fetched version lists, applying the /// configured pre-release filter before classification. +/// The cache key for an ecosystem's default fetcher. +/// +/// A checker pointed at a private index or a mirror answers different questions than one +/// pointed at the public registry, and the on-disk cache records only `(key, name)`. With +/// the key naming the ecosystem alone, a run against a mirror wrote entries a later +/// public run read back as its own — and the entry's name guard cannot catch it, because +/// the name matches. Only a non-default root is scoped, so existing entries stay valid. +fn default_cache_key(fetcher: &dyn RegistryFetcher, ecosystem: Ecosystem) -> String { + let base = ecosystem.osv_name(); + match fetcher.registry_root() { + Some(root) + if root.trim_end_matches('/') != ecosystem.default_registry().trim_end_matches('/') => + { + let mut hasher = DefaultHasher::new(); + root.hash(&mut hasher); + format!("{base}-{:016x}", hasher.finish()) + } + _ => base.to_string(), + } +} + fn evaluate_item( item: &Item, fetched: &FetchedMap, + cache_key: &str, ecosystem: Ecosystem, unstable: UnstableFilter, ) -> CheckResult { if !item.is_checkable() { let status = match item.source { PackageSource::Git => DependencyStatus::Git, + // A real package whose declared version could not be read from the + // manifest. Nothing to fetch, and nothing that would justify calling it + // current. + PackageSource::Unresolved => DependencyStatus::Undetermined, _ => DependencyStatus::Local, }; return CheckResult::new(item.clone(), status); } - match fetched.get(&item.name) { + match fetched.get(&(cache_key.to_owned(), item.name.clone())) { Some(Ok(versions)) => { // The current version drives `IncludeIfCurrent`: the locked version if // known, else the declared constraint (its pre-release markers, if any, @@ -822,18 +941,27 @@ fn evaluate_item( .iter() .map(|(semver, _)| semver.clone()) .collect(); - let constraint = to_semver_constraint(&item.version_constraint, ecosystem); - let eval = check_version(&constraint, &candidates, item.locked_version.as_deref()); + // Translation happens inside `check_version_for`, which is what keeps a + // dialect this crate could not read (`[4.0,4.9`, `LATEST`, `!=2.31.0`) + // apart from a manifest that declared no constraint at all. The first is + // `Undetermined`; only the second is `*`. + let eval = check_version_for( + &item.version_constraint, + ecosystem, + &candidates, + item.locked_version.as_deref(), + ); CheckResult::from_evaluation( item.clone(), in_native_versions(eval, &translated, ecosystem), ) } - Some(Err(e)) => CheckResult::new(item.clone(), DependencyStatus::Error(e.clone())), - None => CheckResult::new( - item.clone(), - DependencyStatus::Error("not fetched".to_string()), - ), + Some(Err(failure)) => { + CheckResult::errored(item.clone(), failure.message.clone(), failure.origin) + } + // No entry at all: the task was never issued, or its result never arrived. No + // registry answered anything here. + None => CheckResult::errored(item.clone(), "not fetched", ErrorOrigin::Local), } } @@ -1010,14 +1138,25 @@ fn same_flavour_only( /// the two produce identical cache keys, and so the advisories describe the exact /// version that was flagged. fn osv_query_for(result: &CheckResult, ecosystem: Ecosystem) -> Option { - if !result.item.is_checkable() || matches!(result.status, DependencyStatus::Error(_)) { + if !result.item.is_checkable() { return None; } - let version = result - .item - .locked_version - .clone() - .or_else(|| result.latest_compatible.clone())?; + // A registry failure used to exclude the dependency from the scan entirely. But OSV + // needs no registry data — only a name and a version — and the lockfile already + // supplied one before any fetch happened. Skipping it meant a rate-limited package + // was not merely unresolved but *unaudited*, which is the more expensive half. + // + // `latest_compatible` is not a fallback here: on an errored fetch there is no version + // list behind it, so only a locked version is trustworthy. + let version = if matches!(result.status, DependencyStatus::Error(_)) { + result.item.locked_version.clone()? + } else { + result + .item + .locked_version + .clone() + .or_else(|| result.latest_compatible.clone())? + }; Some(OsvQuery { ecosystem: ecosystem.osv_name().to_string(), name: result.item.name.clone(), @@ -1076,7 +1215,8 @@ pub struct CheckerBuilder { concurrency: usize, read_lockfiles: bool, unstable: UnstableFilter, - disk_cache: bool, + /// `None` until a caller says either way; see [`CheckerBuilder::disk_cache`]. + disk_cache: Option, disk_cache_dir: Option, progress: Option, } @@ -1098,7 +1238,7 @@ impl Default for CheckerBuilder { concurrency: DEFAULT_CONCURRENCY, read_lockfiles: true, unstable: UnstableFilter::default(), - disk_cache: true, + disk_cache: None, disk_cache_dir: None, progress: None, } @@ -1216,14 +1356,25 @@ impl CheckerBuilder { /// Enable or disable the persistent on-disk registry cache (default: enabled). /// When enabled, registry version lists are cached under the OS cache directory /// with a short TTL so repeat and CI runs avoid re-fetching. Maps to `--no-cache`. + /// Turn the on-disk cache on or off explicitly. An explicit choice always wins, + /// whatever order the builder is called in. + /// + /// Unset, the cache is on only when [`CheckerBuilder::disk_cache_dir`] named a + /// directory. It used to default to on *with the shared OS cache directory*, so + /// merely constructing a checker gave it write access to a location every other run + /// on the machine reads — a side effect no library consumer asked for, and the one + /// that let this repository's own test suite write fabricated version lists into the + /// developer's real cache, where later runs read them back as registry truth. pub fn disk_cache(mut self, enabled: bool) -> Self { - self.disk_cache = enabled; + self.disk_cache = Some(enabled); self } - /// Override the on-disk cache directory (default: the OS cache directory). - /// Mainly for tests and embedders that want an isolated cache location; has no - /// effect when [`CheckerBuilder::disk_cache`] is disabled. + /// Use `dir` as the on-disk cache directory, and — absent an explicit + /// [`CheckerBuilder::disk_cache`] — enable the cache. + /// + /// Naming a directory is itself an opt-in: a caller that chose an isolated location + /// means to use it. An explicit `disk_cache(false)` still wins. pub fn disk_cache_dir(mut self, dir: impl Into) -> Self { self.disk_cache_dir = Some(dir.into()); self @@ -1284,8 +1435,10 @@ impl CheckerBuilder { // Resolve the disk cache: enabled + a usable directory (explicit override // or the OS default). If no directory resolves, the disk cache is simply off. - let disk_cache = self - .disk_cache + // An explicit choice wins; otherwise the cache is on only when a directory was + // named. Nothing falls back to the shared OS cache root without being asked. + let enabled = self.disk_cache.unwrap_or(self.disk_cache_dir.is_some()); + let disk_cache = enabled .then(|| self.disk_cache_dir.or_else(DiskCache::default_root)) .flatten() .map(|dir| Arc::new(DiskCache::new(dir, DISK_CACHE_TTL))); @@ -1312,6 +1465,110 @@ impl CheckerBuilder { #[cfg(test)] mod tests { use super::*; + + /// Every ecosystem's default fetcher has to scope its cache key when it is pointed + /// somewhere other than the public registry. + /// + /// The on-disk cache records only `(key, name)`, so a fetcher that reports no + /// registry root writes a mirror's version list under the public registry's key and + /// a later default run reads it back as its own — the name guard cannot catch it, + /// because the name matches. `MavenCentralFetcher` landed after the scoping did and + /// was the one impl that never got `registry_root`, so a `[jvm] registry` mirror's + /// answers were cached as Maven Central's. Driving all nine here means the tenth + /// implementation cannot repeat it. + #[test] + fn every_default_fetcher_scopes_a_non_default_registry() { + use crate::registries::{ + CratesIoFetcher, GoProxyFetcher, HexFetcher, MavenCentralFetcher, NpmFetcher, + NuGetFetcher, PackagistFetcher, PubDevFetcher, PyPiFetcher, + }; + + const MIRROR: &str = "https://nexus.corp.example/repository/proxy"; + let client = reqwest::Client::new(); + let mirrors: Vec<(Ecosystem, Arc)> = vec![ + ( + Ecosystem::Rust, + Arc::new(CratesIoFetcher::with_registry(client.clone(), MIRROR, None)), + ), + ( + Ecosystem::Go, + Arc::new(GoProxyFetcher::with_proxy(client.clone(), MIRROR)), + ), + ( + Ecosystem::Npm, + Arc::new(NpmFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::Python, + Arc::new(PyPiFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::Php, + Arc::new(PackagistFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::Dart, + Arc::new(PubDevFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::CSharp, + Arc::new(NuGetFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::Elixir, + Arc::new(HexFetcher::with_registry(client.clone(), MIRROR)), + ), + ( + Ecosystem::Jvm, + Arc::new(MavenCentralFetcher::with_registry(client.clone(), MIRROR)), + ), + ]; + for (ecosystem, fetcher) in &mirrors { + assert_ne!( + default_cache_key(fetcher.as_ref(), *ecosystem), + ecosystem.osv_name(), + "{ecosystem:?} caches a mirror's answers under the public registry's key" + ); + } + + // The default registry keeps the bare key, so existing cache entries stay valid. + let defaults: Vec<(Ecosystem, Arc)> = vec![ + ( + Ecosystem::Rust, + Arc::new(CratesIoFetcher::new(client.clone())), + ), + (Ecosystem::Go, Arc::new(GoProxyFetcher::new(client.clone()))), + (Ecosystem::Npm, Arc::new(NpmFetcher::new(client.clone()))), + ( + Ecosystem::Python, + Arc::new(PyPiFetcher::new(client.clone())), + ), + ( + Ecosystem::Php, + Arc::new(PackagistFetcher::new(client.clone())), + ), + ( + Ecosystem::Dart, + Arc::new(PubDevFetcher::new(client.clone())), + ), + ( + Ecosystem::CSharp, + Arc::new(NuGetFetcher::new(client.clone())), + ), + (Ecosystem::Elixir, Arc::new(HexFetcher::new(client.clone()))), + ( + Ecosystem::Jvm, + Arc::new(MavenCentralFetcher::new(client.clone())), + ), + ]; + for (ecosystem, fetcher) in &defaults { + assert_eq!( + default_cache_key(fetcher.as_ref(), *ecosystem), + ecosystem.osv_name(), + "{ecosystem:?} rescoped its own default registry" + ); + } + } use dependable_core::parse; /// The single item declared by `manifest`. Built through the parser because @@ -1364,14 +1621,30 @@ mod tests { assert!(osv_query_for(&result, Ecosystem::Rust).is_none()); } + /// A registry failure used to exclude the dependency from the vulnerability scan. + /// But the lockfile already named the version, and OSV needs nothing from the + /// registry — so a rate-limited package was not merely unresolved, it was unaudited. + /// The fixture is the real case: `time 0.2.7` carries RUSTSEC-2020-0071. #[test] - fn an_errored_result_is_never_queried() { + fn an_errored_result_is_still_audited_when_the_lockfile_named_a_version() { let mut declared = registry_item(); declared.locked_version = Some("0.2.7".to_string()); let result = CheckResult::new( declared, DependencyStatus::Error("registry unreachable".to_string()), ); + let query = osv_query_for(&result, Ecosystem::Rust).expect("the locked version is known"); + assert_eq!(query.version, "0.2.7"); + } + + /// Without a lockfile there is no version to ask about: `latest_compatible` is not a + /// fallback here, because an errored fetch has no version list behind it. + #[test] + fn an_errored_result_with_no_locked_version_is_not_queried() { + let result = CheckResult::new( + registry_item(), + DependencyStatus::Error("registry unreachable".to_string()), + ); assert!(osv_query_for(&result, Ecosystem::Rust).is_none()); } @@ -1420,12 +1693,15 @@ mod tests { ecosystem: Ecosystem, unstable: UnstableFilter, ) -> CheckResult { + // The map is keyed by (cache_key, name); with one item under test the key + // only has to match what `evaluate_item` is told to look up. + const CACHE_KEY: &str = "test"; let mut fetched: FetchedMap = HashMap::new(); fetched.insert( - item.name.clone(), + (CACHE_KEY.to_string(), item.name.clone()), Ok(versions.iter().map(|v| (*v).to_string()).collect()), ); - evaluate_item(item, &fetched, ecosystem, unstable) + evaluate_item(item, &fetched, CACHE_KEY, ecosystem, unstable) } /// Every version this tool reports has to be one the registry publishes. The diff --git a/crates/dependable-fetch/src/error.rs b/crates/dependable-fetch/src/error.rs index d9fe8f0..5fed114 100644 --- a/crates/dependable-fetch/src/error.rs +++ b/crates/dependable-fetch/src/error.rs @@ -17,9 +17,49 @@ pub enum FetchError { #[error("registry returned status {code} for `{package}`")] Status { code: u16, package: String }, + /// A successful response that carried no versions at all. + /// + /// Distinct from [`NotFound`](Self::NotFound), which is the registry saying the + /// package does not exist. A `200` with an empty version list is not that answer: a + /// Nexus or Artifactory group repository whose upstream proxy is down serves exactly + /// such a locally-merged document for a package that certainly does exist. Treating + /// it as a 404 exempted it from a `--fail-on` gate, certifying a build against a + /// dependency nothing was ever known about. + #[error("registry listed no versions for `{package}`")] + EmptyVersionList { package: String }, + #[error("failed to decode response for `{package}`: {detail}")] Decode { package: String, detail: String }, #[error("OSV query failed: {0}")] Osv(String), + + #[error("OSV returned status {code}")] + OsvStatus { code: u16 }, +} + +impl FetchError { + /// Whether retrying might succeed. + /// + /// Rate limits and server faults are the registry saying "not now"; a timeout or a + /// refused connection is the network doing the same. A 404 is an answer, and a + /// decode failure is a response we will parse identically next time — retrying + /// either only spends the user's time reaching the same conclusion. + #[must_use] + pub fn is_transient(&self) -> bool { + match self { + Self::Status { code, .. } | Self::OsvStatus { code } => { + *code == 429 || (500..600).contains(code) + } + Self::Http(error) => error.is_timeout() || error.is_connect(), + // An empty-but-well-formed document parses the same way next time, so a + // retry only spends the user's time reaching the same conclusion — the same + // reasoning as a decode failure. It is still not a 404, so the gate refuses + // to certify through it. + Self::NotFound(_) + | Self::EmptyVersionList { .. } + | Self::Decode { .. } + | Self::Osv(_) => false, + } + } } diff --git a/crates/dependable-fetch/src/lib.rs b/crates/dependable-fetch/src/lib.rs index 70e06f3..098d04d 100644 --- a/crates/dependable-fetch/src/lib.rs +++ b/crates/dependable-fetch/src/lib.rs @@ -44,6 +44,7 @@ pub mod discover; pub mod error; pub mod osv; pub mod registries; +mod retry; pub mod tree; // High-level entry point (recommended for embedding). @@ -77,10 +78,10 @@ pub use registries::{ // parsers, `check_version`, ...). pub use dependable_core as core; pub use dependable_core::{ - CheckResult, DependencyGraph, DependencyKind, DependencyStatus, Ecosystem, Evaluation, Item, - LockfileKind, ManifestKind, Node, NodeKind, PackageSource, ParseError, ParsedManifest, - PathPredicate, Placement, Tree, TreeNode, TreeOptions, UnstableFilter, Visit, Visitor, - WalkOptions, WorkspaceDecl, resolve_workspace_inheritance, + CheckResult, DependencyGraph, DependencyKind, DependencyStatus, Ecosystem, ErrorOrigin, + Evaluation, Item, LockfileKind, ManifestKind, Node, NodeKind, PackageSource, ParseError, + ParsedManifest, PathPredicate, Placement, Tree, TreeNode, TreeOptions, UnstableFilter, Visit, + Visitor, WalkOptions, WalkStats, WorkspaceDecl, resolve_workspace_inheritance, }; /// One-import convenience for consumers: `use dependable_fetch::prelude::*;`. @@ -106,5 +107,9 @@ pub fn build_client() -> Result { std::env::consts::OS )) .timeout(Duration::from_secs(10)) + // Separate from the total timeout: a host that accepts the connection and then + // stalls is a different failure from one that never answers at all, and without + // this a black-holed address spends the entire request budget on the handshake. + .connect_timeout(Duration::from_secs(5)) .build() } diff --git a/crates/dependable-fetch/src/osv/client.rs b/crates/dependable-fetch/src/osv/client.rs index a3988e6..e7b722a 100644 --- a/crates/dependable-fetch/src/osv/client.rs +++ b/crates/dependable-fetch/src/osv/client.rs @@ -161,14 +161,30 @@ impl OsvClient { .collect(), }; - let resp = self.client.post(&self.batch_url).json(&body).send().await?; - if !resp.status().is_success() { - return Err(FetchError::Osv(format!("status {}", resp.status()))); + let parsed: BatchResponse = crate::retry::with_retry(|| async { + let resp = self.client.post(&self.batch_url).json(&body).send().await?; + if !resp.status().is_success() { + return Err(FetchError::OsvStatus { + code: resp.status().as_u16(), + }); + } + resp.json::() + .await + .map_err(|e| FetchError::Osv(e.to_string())) + }) + .await?; + + // One result per query, in order, is the API's contract. A short body used + // to leave the unanswered slots empty — recorded as "no vulnerabilities" and + // written into the cache for ten minutes, so even a retry in the same process + // could not recover. A truncated answer is an error, not a clean bill. + if parsed.results.len() < chunk.len() { + return Err(FetchError::Osv(format!( + "querybatch answered {} of {} queries", + parsed.results.len(), + chunk.len() + ))); } - let parsed: BatchResponse = resp - .json() - .await - .map_err(|e| FetchError::Osv(e.to_string()))?; for (slot, &i) in chunk.iter().enumerate() { let ids: Vec = parsed @@ -236,7 +252,9 @@ impl OsvClient { }; let resp = self.client.post(&self.query_url).json(&body).send().await?; if !resp.status().is_success() { - return Err(FetchError::Osv(format!("status {}", resp.status()))); + return Err(FetchError::OsvStatus { + code: resp.status().as_u16(), + }); } let parsed: DetailResponse = resp .json() diff --git a/crates/dependable-fetch/src/registries/crates_io.rs b/crates/dependable-fetch/src/registries/crates_io.rs index efb925e..0453cdb 100644 --- a/crates/dependable-fetch/src/registries/crates_io.rs +++ b/crates/dependable-fetch/src/registries/crates_io.rs @@ -188,6 +188,10 @@ impl ApiOwner { } impl RegistryFetcher for CratesIoFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, @@ -210,7 +214,14 @@ impl RegistryFetcher for CratesIoFetcher { }); } let body = resp.text().await?; - Ok(parse_index(&body)) + parse_index(&body).map_err(|error| match error { + // The package name is only known here, at the call site. + FetchError::Decode { detail, .. } => FetchError::Decode { + package: name.to_owned(), + detail, + }, + other => other, + }) } .boxed() } @@ -302,20 +313,37 @@ impl RegistryFetcher for CratesIoFetcher { /// Parse the newline-delimited JSON index body into versions, newest-first, with /// yanked releases filtered out. The newest version's declared feature flags are /// attached for `list --features`. -fn parse_index(body: &str) -> FetchedVersions { +fn parse_index(body: &str) -> Result { + let mut malformed = 0usize; let mut entries: Vec = body .lines() .filter(|line| !line.trim().is_empty()) - .filter_map(|line| serde_json::from_str::(line).ok()) + .filter_map(|line| match serde_json::from_str::(line) { + Ok(entry) => Some(entry), + Err(_) => { + malformed += 1; + None + } + }) .filter(|line| !line.yanked) .collect(); + // A partially corrupt body is the dangerous case, not a wholly broken one: dropping + // the bad lines leaves a plausible-looking but *short* version list, and if the + // newest release was among them the dependency reports up to date. A body we cannot + // read in full is an error, so the caller retries or reports rather than believing it. + if malformed > 0 { + return Err(FetchError::Decode { + package: String::new(), + detail: format!("{malformed} malformed line(s) in the sparse index response"), + }); + } entries.sort_by(|a, b| cmp_vers_desc(&a.vers, &b.vers)); let features = entries .first() .map(IndexLine::feature_names) .unwrap_or_default(); let versions: Vec = entries.into_iter().map(|line| line.vers).collect(); - FetchedVersions::new(versions).with_features(features) + Ok(FetchedVersions::new(versions).with_features(features)) } /// Order two version strings newest-first, falling back to reverse lexical order @@ -361,7 +389,7 @@ mod tests { "{\"name\":\"x\",\"vers\":\"1.1.0\",\"yanked\":true}\n", "{\"name\":\"x\",\"vers\":\"1.2.0\",\"yanked\":false}\n", ); - let fetched = parse_index(body); + let fetched = parse_index(body).expect("a well-formed index body"); assert_eq!(fetched.versions, vec!["1.2.0", "1.0.0"]); assert_eq!(fetched.latest_tag.as_deref(), Some("1.2.0")); assert!(fetched.features.is_empty()); // no features declared @@ -373,9 +401,32 @@ mod tests { "{\"name\":\"x\",\"vers\":\"1.0.0\",\"yanked\":false,\"features\":{\"legacy\":[]}}\n", "{\"name\":\"x\",\"vers\":\"2.0.0\",\"yanked\":false,\"features\":{\"default\":[\"std\"],\"derive\":[\"x-derive\"]},\"features2\":{\"rc\":[\"dep:rc\"]}}\n", ); - let fetched = parse_index(body); + let fetched = parse_index(body).expect("a well-formed index body"); assert_eq!(fetched.versions, vec!["2.0.0", "1.0.0"]); // Newest version (2.0.0) only, merging `features` + `features2`, sorted. assert_eq!(fetched.features, vec!["default", "derive", "rc"]); } + + /// A partially corrupt body is the dangerous case: dropping the unreadable lines + /// leaves a plausible-looking but short version list, and if the newest release was + /// among them the dependency reports up to date. + #[test] + fn a_malformed_index_line_is_an_error_not_a_shorter_list() { + let body = concat!( + "{\"name\":\"serde\",\"vers\":\"1.0.0\",\"yanked\":false}\n", + "{\"name\":\"serde\",\"vers\":\"1.2.0\",\"yank\n", + ); + assert!( + parse_index(body).is_err(), + "a truncated line was silently dropped" + ); + } + + /// Blank lines are not corruption; the index body ends with one. + #[test] + fn blank_lines_are_not_treated_as_corruption() { + let body = "{\"name\":\"serde\",\"vers\":\"1.0.0\",\"yanked\":false}\n\n"; + let fetched = parse_index(body).expect("a trailing newline is not a malformed line"); + assert_eq!(fetched.versions, vec!["1.0.0"]); + } } diff --git a/crates/dependable-fetch/src/registries/go_proxy.rs b/crates/dependable-fetch/src/registries/go_proxy.rs index ce38582..8f4deb6 100644 --- a/crates/dependable-fetch/src/registries/go_proxy.rs +++ b/crates/dependable-fetch/src/registries/go_proxy.rs @@ -44,6 +44,10 @@ impl GoProxyFetcher { } impl RegistryFetcher for GoProxyFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, @@ -62,7 +66,7 @@ impl RegistryFetcher for GoProxyFetcher { return Ok(FetchedVersions::new(versions)); } // Empty list (e.g. a module with only pseudo-versions): fall back. - } else if status != reqwest::StatusCode::NOT_FOUND { + } else if !is_absent(status) { return Err(FetchError::Status { code: status.as_u16(), package: name.to_string(), @@ -73,7 +77,7 @@ impl RegistryFetcher for GoProxyFetcher { let latest_url = format!("{}/{escaped}/@latest", self.base_url); let resp = self.client.get(&latest_url).send().await?; let status = resp.status(); - if status == reqwest::StatusCode::NOT_FOUND { + if is_absent(status) { return Err(FetchError::NotFound(name.to_string())); } if !status.is_success() { @@ -92,6 +96,23 @@ impl RegistryFetcher for GoProxyFetcher { } } +/// Whether a proxy response means "this module is not here", as opposed to the proxy +/// failing to answer. +/// +/// The Go module proxy protocol names **both** `404` and `410` as the not-found +/// responses, and `410 Gone` is the canonical `proxy.golang.org` answer for a module it +/// will not serve — the private or internal path a repository excludes with `GOPRIVATE`, +/// or one the proxy has been asked to forget. Reading it as a transport failure marked +/// the whole registry unreachable, and a `--fail-on` gate cannot be honoured through +/// that: one private module turned a passing run into exit 2, which is exactly the CI +/// break the 404 carve-out exists to prevent. +fn is_absent(status: reqwest::StatusCode) -> bool { + matches!( + status, + reqwest::StatusCode::NOT_FOUND | reqwest::StatusCode::GONE + ) +} + /// Parse the newline-delimited `@v/list` body into clean (no leading `v`) semver /// versions, newest-first, dropping anything unparseable. fn parse_list(body: &str) -> Vec { @@ -156,6 +177,17 @@ mod tests { assert_eq!(parse_list(body), vec!["1.0.0"]); } + /// The protocol's two not-found answers, and nothing else. A `410` reaching the + /// `Status` arm made a private module unreachable-registry rather than absent. + #[test] + fn both_not_found_statuses_mean_absent() { + assert!(is_absent(reqwest::StatusCode::NOT_FOUND)); + assert!(is_absent(reqwest::StatusCode::GONE)); + assert!(!is_absent(reqwest::StatusCode::FORBIDDEN)); + assert!(!is_absent(reqwest::StatusCode::TOO_MANY_REQUESTS)); + assert!(!is_absent(reqwest::StatusCode::INTERNAL_SERVER_ERROR)); + } + #[test] fn escapes_uppercase_letters() { assert_eq!( diff --git a/crates/dependable-fetch/src/registries/hex.rs b/crates/dependable-fetch/src/registries/hex.rs index 922f085..78c9d8a 100644 --- a/crates/dependable-fetch/src/registries/hex.rs +++ b/crates/dependable-fetch/src/registries/hex.rs @@ -95,6 +95,10 @@ fn hex_repository(links: &std::collections::HashMap) -> Option Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/jsr.rs b/crates/dependable-fetch/src/registries/jsr.rs index 5e6b658..8deb2dc 100644 --- a/crates/dependable-fetch/src/registries/jsr.rs +++ b/crates/dependable-fetch/src/registries/jsr.rs @@ -53,6 +53,10 @@ impl JsrFetcher { } impl RegistryFetcher for JsrFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/maven_central.rs b/crates/dependable-fetch/src/registries/maven_central.rs index 57bb603..d315e04 100644 --- a/crates/dependable-fetch/src/registries/maven_central.rs +++ b/crates/dependable-fetch/src/registries/maven_central.rs @@ -50,6 +50,10 @@ impl MavenCentralFetcher { } impl RegistryFetcher for MavenCentralFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, @@ -124,8 +128,14 @@ fn parse_metadata(body: &str, package: &str) -> Result Option<&str> { + None + } } /// Fetch the declared license of each of `names` from one registry, concurrently. diff --git a/crates/dependable-fetch/src/registries/npm.rs b/crates/dependable-fetch/src/registries/npm.rs index 5854dd5..ad7d438 100644 --- a/crates/dependable-fetch/src/registries/npm.rs +++ b/crates/dependable-fetch/src/registries/npm.rs @@ -165,6 +165,10 @@ impl From for Owner { } impl RegistryFetcher for NpmFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/nuget.rs b/crates/dependable-fetch/src/registries/nuget.rs index aee7840..659a627 100644 --- a/crates/dependable-fetch/src/registries/nuget.rs +++ b/crates/dependable-fetch/src/registries/nuget.rs @@ -79,6 +79,10 @@ impl NuGetFetcher { } impl RegistryFetcher for NuGetFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/packagist.rs b/crates/dependable-fetch/src/registries/packagist.rs index d8e7fa6..832db60 100644 --- a/crates/dependable-fetch/src/registries/packagist.rs +++ b/crates/dependable-fetch/src/registries/packagist.rs @@ -105,6 +105,10 @@ struct Source { } impl RegistryFetcher for PackagistFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/pub_dev.rs b/crates/dependable-fetch/src/registries/pub_dev.rs index 57caf90..baf1892 100644 --- a/crates/dependable-fetch/src/registries/pub_dev.rs +++ b/crates/dependable-fetch/src/registries/pub_dev.rs @@ -51,6 +51,10 @@ impl PubDevFetcher { } impl RegistryFetcher for PubDevFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/registries/pypi.rs b/crates/dependable-fetch/src/registries/pypi.rs index 1502660..a57192b 100644 --- a/crates/dependable-fetch/src/registries/pypi.rs +++ b/crates/dependable-fetch/src/registries/pypi.rs @@ -319,6 +319,10 @@ fn repository_url(urls: &HashMap) -> Option { } impl RegistryFetcher for PyPiFetcher { + fn registry_root(&self) -> Option<&str> { + Some(&self.base_url) + } + fn fetch_versions<'a>( &'a self, name: &'a str, diff --git a/crates/dependable-fetch/src/retry.rs b/crates/dependable-fetch/src/retry.rs new file mode 100644 index 0000000..d87280d --- /dev/null +++ b/crates/dependable-fetch/src/retry.rs @@ -0,0 +1,46 @@ +//! Retry with exponential backoff for transient registry and OSV failures. +//! +//! Every fetcher previously treated 403, 429, 500 and a timeout identically: one +//! attempt, then a per-package error. With `concurrency` defaulting to 20 against +//! registries that rate-limit, a large monorepo reliably provoked 429s — and a +//! rate-limited package became `DependencyStatus::Error`, which `--fail-on vulnerable` +//! ignored and the vulnerability scan skipped entirely. A transient failure turned into +//! a silently unaudited dependency. + +use std::time::Duration; + +use crate::error::FetchError; + +/// How many times an operation is attempted in total. +const MAX_ATTEMPTS: u32 = 3; + +/// Delay before the second attempt; doubled for each one after. +const BASE_DELAY: Duration = Duration::from_millis(200); + +/// Run `operation`, retrying while it fails transiently. +/// +/// Only [`FetchError::is_transient`] failures are retried: a 404 is an answer, and +/// repeating it wastes the user's time to reach the same conclusion. +/// +/// The backoff is fixed rather than driven by `Retry-After`; the header is not currently +/// carried on the error, so a server asking for a longer pause than this gets the pause +/// this gives it. +pub(crate) async fn with_retry(mut operation: F) -> Result +where + F: FnMut() -> Fut, + Fut: Future>, +{ + let mut delay = BASE_DELAY; + for attempt in 1..MAX_ATTEMPTS { + match operation().await { + Err(error) if error.is_transient() => { + tracing::debug!(attempt, %error, "transient fetch failure; retrying"); + tokio::time::sleep(delay).await; + delay *= 2; + } + settled => return settled, + } + } + // The final attempt's result stands, transient or not. + operation().await +} diff --git a/crates/dependable-fetch/src/tree.rs b/crates/dependable-fetch/src/tree.rs index df04dac..5312041 100644 --- a/crates/dependable-fetch/src/tree.rs +++ b/crates/dependable-fetch/src/tree.rs @@ -350,7 +350,19 @@ pub fn build_project_graph( }); }; - let resolved = parser(&read(&lock_path)?)?; + // A lockfile we cannot read is not a reason to fail the command. The manifests still + // describe the direct dependencies, and `UnreadableLockfile` is how the caller tells + // the user that a lockfile is sitting there unread — which is the actionable half. + let resolved = match parser(&read(&lock_path)?) { + Ok(resolved) => resolved, + Err(_) => { + let graph = direct_graph(&root_name, &root_version, &direct, &workspace_names, &roots); + return Ok(WorkspaceGraph { + graph, + source: GraphSource::UnreadableLockfile, + }); + } + }; let resolved = with_root(resolved, &root_name, &root_version, direct); Ok(WorkspaceGraph { graph: DependencyGraph::from_resolved(&resolved, &workspace_names, &roots), diff --git a/crates/dependable-fetch/tests/checker.rs b/crates/dependable-fetch/tests/checker.rs index 07e9041..624765d 100644 --- a/crates/dependable-fetch/tests/checker.rs +++ b/crates/dependable-fetch/tests/checker.rs @@ -1150,3 +1150,137 @@ async fn an_inherited_name_the_root_never_declared_is_reported() { assert_eq!(result.status, DependencyStatus::Local); } } + +/// One name, two registries, one manifest. The fetch map used to be keyed by name +/// alone while the tasks were deduplicated by `(cache_key, name)`, so both routes +/// landed in the same slot and whichever request finished last answered for both — +/// non-deterministically, since they complete out of order. +#[tokio::test] +async fn a_name_published_to_two_registries_is_not_collapsed() { + let server = MockServer::start().await; + // The npm `foo` and the JSR `foo` are different packages with different versions. + Mock::given(method("GET")) + .and(path("/foo")) + .respond_with( + ResponseTemplate::new(200).set_body_string(r#"{"versions":{"1.0.0":{},"9.9.9":{}}}"#), + ) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/foo/meta.json")) + .respond_with( + ResponseTemplate::new(200) + .set_body_string(r#"{"latest":"2.0.0","versions":{"2.0.0":{}}}"#), + ) + .mount(&server) + .await; + + let client = build_client().unwrap(); + let checker = Checker::builder() + .http_client(client.clone()) + .registry( + Ecosystem::Npm, + Arc::new(NpmFetcher::with_registry(client.clone(), server.uri())), + ) + .jsr_registry(Arc::new(JsrFetcher::with_registry(client, server.uri()))) + .vulnerabilities(false) + .build() + .unwrap(); + + let manifest = r#"{ "imports": { "a": "npm:foo@^1.0.0", "b": "jsr:foo@^2.0.0" } }"#; + let check = checker + .check_manifest(ManifestKind::DenoJson, manifest, None) + .await + .unwrap(); + + let npm = check + .results + .iter() + .find(|r| r.item.name == "foo" && r.item.source == PackageSource::Registry) + .expect("npm foo"); + let jsr = check + .results + .iter() + .find(|r| r.item.name == "foo" && r.item.source == PackageSource::Jsr) + .expect("jsr foo"); + + assert_eq!(npm.latest_available.as_deref(), Some("9.9.9")); + assert_eq!(jsr.latest_available.as_deref(), Some("2.0.0")); +} + +/// A private index and the public registry publish different version lists for the same +/// name. The on-disk entry records only `(key, name)`, so with the key naming just the +/// ecosystem, one run's answers were served to the other — and the entry's name guard +/// cannot catch it, because the name matches. +#[tokio::test] +async fn a_private_registry_does_not_share_disk_cache_entries_with_the_public_one() { + let private = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/se/rd/serde")) + .respond_with( + ResponseTemplate::new(200) + .set_body_string("{\"name\":\"serde\",\"vers\":\"0.0.1\",\"yanked\":false}\n"), + ) + .mount(&private) + .await; + + let public = MockServer::start().await; + mount_index(&public).await; + + let dir = tempfile::tempdir().unwrap(); + + let build = |uri: String| { + Checker::builder() + .http_client(build_client().unwrap()) + .rust_registry(uri, None) + .vulnerabilities(false) + .disk_cache_dir(dir.path()) + .build() + .unwrap() + }; + + // Populate the cache from the private index first. + let from_private = build(private.uri()) + .check_manifest(ManifestKind::CargoToml, MANIFEST, Some(LOCK)) + .await + .unwrap(); + let private_serde = from_private + .results + .iter() + .find(|r| r.item.name == "serde") + .expect("serde"); + assert_eq!(private_serde.latest_available.as_deref(), Some("0.0.1")); + + // A public run sharing the same cache directory must ask the public index, not read + // the private index's answer back out of the cache. + let from_public = build(public.uri()) + .check_manifest(ManifestKind::CargoToml, MANIFEST, Some(LOCK)) + .await + .unwrap(); + let public_serde = from_public + .results + .iter() + .find(|r| r.item.name == "serde") + .expect("serde"); + assert_ne!( + public_serde.latest_available.as_deref(), + Some("0.0.1"), + "the public run was served the private index's version list" + ); + assert!(!public.received_requests().await.unwrap().is_empty()); +} + +/// The leak that started all of this: constructing a checker must not, on its own, give +/// it write access to the cache directory shared by every run on the machine. +#[test] +fn a_default_checker_writes_to_no_shared_cache() { + let checker = Checker::builder() + .http_client(build_client().unwrap()) + .vulnerabilities(false) + .build() + .unwrap(); + assert!( + !checker.uses_disk_cache(), + "the disk cache must be opted into, not inherited" + ); +} diff --git a/crates/dependable-fetch/tests/http.rs b/crates/dependable-fetch/tests/http.rs index b091617..6abd6ed 100644 --- a/crates/dependable-fetch/tests/http.rs +++ b/crates/dependable-fetch/tests/http.rs @@ -148,6 +148,13 @@ async fn live_pypi_flask_has_versions() { let fetcher = PyPiFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("flask").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -206,6 +213,13 @@ async fn live_go_proxy_has_versions() { let fetcher = GoProxyFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("golang.org/x/text").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -305,6 +319,13 @@ async fn live_packagist_monolog_has_versions() { let fetcher = PackagistFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("monolog/monolog").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -330,6 +351,13 @@ async fn live_pub_dev_http_has_versions() { let fetcher = PubDevFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("http").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -375,6 +403,13 @@ async fn live_nuget_newtonsoft_has_versions() { let fetcher = NuGetFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("Newtonsoft.Json").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -400,6 +435,13 @@ async fn live_hex_phoenix_has_versions() { let fetcher = HexFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("phoenix").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -486,6 +528,13 @@ async fn live_npm_react_has_versions() { let fetcher = NpmFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("react").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -494,6 +543,13 @@ async fn live_jsr_std_path_has_versions() { let fetcher = JsrFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("@std/path").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] @@ -502,6 +558,13 @@ async fn live_crates_io_serde_has_versions() { let fetcher = CratesIoFetcher::new(build_client().unwrap()); let fetched = fetcher.fetch_versions("serde").await.unwrap(); assert!(!fetched.versions.is_empty()); + // A registry that answers must answer with something parseable: an empty-ish list + // satisfied the old assertion no matter what shape the response had. + assert!( + fetched.versions.iter().all(|v| !v.trim().is_empty()), + "blank version in {:?}", + fetched.versions + ); } #[tokio::test] diff --git a/crates/dependable-fetch/tests/osv.rs b/crates/dependable-fetch/tests/osv.rs index d450653..1a848f1 100644 --- a/crates/dependable-fetch/tests/osv.rs +++ b/crates/dependable-fetch/tests/osv.rs @@ -294,3 +294,44 @@ async fn live_query_detail_enriches_a_known_advisory() { } } } + +/// The API answers one result per query, in order. A short body used to leave the +/// unanswered slots empty — recorded as "no vulnerabilities" *and cached as clean* for +/// ten minutes, so even a retry in the same process could not recover. +#[tokio::test] +async fn a_short_querybatch_response_is_an_error_not_a_clean_bill() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/v1/querybatch")) + // Two queries go out; one result comes back. + .respond_with( + ResponseTemplate::new(200) + .set_body_string(r#"{"results":[{"vulns":[{"id":"RUSTSEC-2020-0071"}]}]}"#), + ) + .mount(&server) + .await; + + let client = OsvClient::with_url( + build_client().unwrap(), + format!("{}/v1/querybatch", server.uri()), + false, + ); + let queries = vec![ + OsvQuery { + ecosystem: "crates.io".into(), + name: "time".into(), + version: "0.2.7".into(), + }, + OsvQuery { + ecosystem: "crates.io".into(), + name: "serde".into(), + version: "1.0.0".into(), + }, + ]; + + let result = client.query_batch(&queries).await; + assert!( + result.is_err(), + "a truncated batch response was accepted as a complete answer" + ); +} diff --git a/crates/dependable-report/Cargo.toml b/crates/dependable-report/Cargo.toml index 63dc273..405817b 100644 --- a/crates/dependable-report/Cargo.toml +++ b/crates/dependable-report/Cargo.toml @@ -7,17 +7,6 @@ rust-version.workspace = true license.workspace = true repository.workspace = true -[features] -default = ["bin"] -# The `dependable-report` binary. On by default so the workspace's clippy and -# test runs (which build default features only) keep compiling it. -bin = [] - -[[bin]] -name = "dependable-report" -path = "src/main.rs" -required-features = ["bin"] - [dependencies] dependable-core.workspace = true minijinja.workspace = true diff --git a/crates/dependable-report/src/main.rs b/crates/dependable-report/src/main.rs deleted file mode 100644 index ce9e544..0000000 --- a/crates/dependable-report/src/main.rs +++ /dev/null @@ -1,20 +0,0 @@ -//! The `dependable-report` binary. -//! -//! Its eventual job is to render a report from the JSON that -//! `dependable check --format json` writes, so CI can produce an HTML or SARIF -//! artifact from a check that already ran, without re-fetching anything. -//! -//! Nothing is rendered yet: the crate is a scaffold, so this reports that and -//! exits `2` (the CLI's "could not do the job" code), leaving stdout empty so a -//! pipeline reading it as data never receives a message instead. - -use std::process::ExitCode; - -fn main() -> ExitCode { - eprintln!( - "dependable-report v{} is a scaffold.", - dependable_report::VERSION - ); - eprintln!("error: report rendering is not implemented yet"); - ExitCode::from(2) -} diff --git a/crates/dependable-report/src/policy.rs b/crates/dependable-report/src/policy.rs index bf489f7..c7e516a 100644 --- a/crates/dependable-report/src/policy.rs +++ b/crates/dependable-report/src/policy.rs @@ -726,14 +726,51 @@ pub fn evaluate(report: &Report, policy: &Policy) -> PolicyOutcome { } // 5/6. CVSS gate, then the unrated advisories it could not see. - if result.advisories.is_empty() { - continue; - } let Some(threshold_score) = threshold_score else { continue; }; + + // A withdrawn advisory has been retracted by its publisher. The rest of the + // codebase already knows this — the summary counts them, the HTML report + // flags them — and only the gate that fails the build was blind to it. + let live: Vec = result + .advisories + .iter() + .filter(|a| !a.is_withdrawn()) + .cloned() + .collect(); + + if live.is_empty() { + // `advisories` is enrichment; `current_vulnerabilities` is the finding. + // Treating an empty enrichment as "nothing to gate on" let a dependency + // that is *known vulnerable* — but whose enrichment failed, or was never + // requested — pass every severity gate in silence. That is the package + // the gate exists to catch. It has no score, so it is unrated, and the + // `unrated_advisories` knob decides, exactly as for a scored-but-unrated + // advisory. + if result.current_vulnerabilities.is_empty() { + continue; + } + let level = match policy.unrated_advisories { + UnratedPolicy::Ignore => continue, + UnratedPolicy::Warn => Level::Warning, + UnratedPolicy::Fail => Level::Violation, + }; + push( + level, + cvss_rule, + Detail::Unrated { + count: result.current_vulnerabilities.len(), + advisories: result.current_vulnerabilities.clone(), + best_known: None, + }, + None, + ); + continue; + } + let mut violated = false; - if let Some(score) = result.max_cvss() + if let Some(score) = Advisory::max_cvss(&live) && score >= threshold_score { violated = true; @@ -743,12 +780,12 @@ pub fn evaluate(report: &Report, policy: &Policy) -> PolicyOutcome { Detail::Cvss { score, threshold: threshold_score, - advisories: over_score(&result.advisories, threshold_score), + advisories: over_score(&live, threshold_score), }, None, ); } else if let (Some(band), Some(threshold_band)) = - (result.max_severity(), threshold_band) + (Advisory::max_severity(&live), threshold_band) && band >= threshold_band { // A published band with no scorable vector. Reported as a band @@ -760,13 +797,13 @@ pub fn evaluate(report: &Report, policy: &Policy) -> PolicyOutcome { Detail::Severity { band, threshold: threshold_band, - advisories: over_band(&result.advisories, threshold_band), + advisories: over_band(&live, threshold_band), }, None, ); } - let unrated = unrated_ids(&result.advisories); + let unrated = unrated_ids(&live); if violated || unrated.is_empty() { continue; } @@ -781,7 +818,7 @@ pub fn evaluate(report: &Report, policy: &Policy) -> PolicyOutcome { Detail::Unrated { count: unrated.len(), advisories: unrated, - best_known: result.max_cvss(), + best_known: Advisory::max_cvss(&live), }, None, ); @@ -829,17 +866,32 @@ fn parse_version(raw: &str, ecosystem: Ecosystem) -> Option { /// How many breaking releases separate `current` from `latest`. /// -/// Under `0.x` the **minor** is the breaking axis — which is how the version -/// checker already classifies compatibility — so `0.1 → 0.9` is eight breaking -/// releases behind, not zero. Counting it as zero would make the gate blind to -/// exactly the churn it exists to catch. +/// Under `0.x` the **minor** is the breaking axis — which is how the version checker +/// already classifies compatibility — so `0.1 -> 0.9` is eight breaking releases behind, +/// not zero. Counting it as zero would make the gate blind to exactly the churn it +/// exists to catch. +/// +/// Crossing out of `0.x` counts the majors past `0` alone, so `0.1 -> 1.0` is one and +/// `0.1 -> 3.0` is three. +/// +/// # Limitation +/// The measure is **not monotonic across that crossing**, and this rewrite did not make +/// it so: it is a restatement of the same truth table, identical on every input. The +/// 0.x releases skipped on the way out of the line are not counted, because the set of +/// published versions is not available here — only the two endpoints are. So a +/// dependency at `0.1.0` scores 8 against an upstream `0.9.0` and 1 against `1.0.0`, and +/// a `max_major_behind = 2` gate it had been failing starts passing the moment upstream +/// ships `1.0.0` — further behind, measured as closer. Repairing that needs the +/// published version set this function is not given, which is a design change rather +/// than a repair; it is filed rather than attempted here. fn major_distance(current: &semver::Version, latest: &semver::Version) -> u64 { - if current.major != latest.major { - latest.major.saturating_sub(current.major) - } else if current.major == 0 { - latest.minor.saturating_sub(current.minor) - } else { - 0 + match (current.major, latest.major) { + // Both on the 0.x line: the minor is the breaking axis. + (0, 0) => latest.minor.saturating_sub(current.minor), + // Leaving 0.x: the crossing is one breaking release, plus each major past 1.0. + (0, to) => to, + // Both past 1.0: the major is the breaking axis. + (from, to) => to.saturating_sub(from), } } @@ -1030,6 +1082,12 @@ reason = "CVE-2023-xxxx fix" Advisory::new(id).with_severity(AdvisorySeverity::from_score(score)) } + /// `withdrawn` is a plain field with no builder; this keeps the fixtures readable. + fn withdrawn(mut advisory: Advisory) -> Advisory { + advisory.withdrawn = Some("2024-01-01T00:00:00Z".to_owned()); + advisory + } + fn banded(id: &str, label: &str) -> Advisory { Advisory::new(id).with_severity(AdvisorySeverity::from_label(label)) } @@ -1763,4 +1821,121 @@ reason = "CVE-2023-xxxx fix" ); } } + + /// The gate's whole purpose is the package that is known vulnerable. Treating an + /// empty *enrichment* list as "nothing to gate on" let exactly that package through: + /// enrichment is opt-in and can fail, while `current_vulnerabilities` is the finding. + #[test] + fn a_known_vulnerable_dependency_without_enrichment_does_not_pass_silently() { + let mut result = checked("openssl = \"0.10\""); + result.status = DependencyStatus::Vulnerable; + result.current_vulnerabilities = vec!["RUSTSEC-2020-0071".to_owned()]; + assert!(result.advisories.is_empty(), "enrichment did not run"); + + // Under the default `warn`, it is reported rather than passing in silence. + let warned = evaluate(&rust(vec![result.clone()]), &policy("max_cvss = 7.0\n")); + assert!( + warned + .findings + .iter() + .any(|f| matches!(f.rule, Rule::MaxCvss | Rule::FailOnSeverity)), + "no finding: {:?}", + warned.findings + ); + + // Under `fail`, it fails the build. + let failed = evaluate( + &rust(vec![result]), + &policy("max_cvss = 7.0\nunrated_advisories = \"fail\"\n"), + ); + assert!(failed.has_violations()); + } + + /// A retracted advisory is not a finding. The summary and the HTML report both know + /// this already; only the gate that fails the build did not. + #[test] + fn a_withdrawn_advisory_does_not_fail_the_build() { + let mut result = checked("openssl = \"0.10\""); + result.status = DependencyStatus::Vulnerable; + result.current_vulnerabilities = vec!["RUSTSEC-2020-0071".to_owned()]; + result.advisories = vec![withdrawn(scored("RUSTSEC-2020-0071", 9.8))]; + + let outcome = evaluate( + &rust(vec![result]), + &policy("max_cvss = 7.0\nunrated_advisories = \"ignore\"\n"), + ); + assert!( + !outcome.has_violations(), + "a withdrawn advisory failed the build: {:?}", + outcome.findings + ); + } + + /// A live advisory alongside a withdrawn one still fails: filtering the retracted one + /// must not disarm the gate. + #[test] + fn a_live_advisory_beside_a_withdrawn_one_still_fails() { + let mut result = checked("openssl = \"0.10\""); + result.status = DependencyStatus::Vulnerable; + result.current_vulnerabilities = vec!["A".to_owned(), "B".to_owned()]; + result.advisories = vec![withdrawn(scored("A", 9.8)), scored("B", 8.1)]; + let outcome = evaluate(&rust(vec![result]), &policy("max_cvss = 7.0\n")); + assert!(outcome.has_violations()); + } + + /// The measure, stated as it actually behaves. Under `0.x` the minor is the breaking + /// axis; past `1.0` the major is; and across the crossing only the majors past `0` + /// are counted — so `0.1 -> 0.9` is 8 while `0.1 -> 1.0` is 1, and being further + /// behind measures as being closer. That is a known limitation, recorded on + /// [`major_distance`] and filed, not something this test claims is fixed. + #[test] + fn major_distance_counts_the_breaking_axis_of_each_version_line() { + let v = |s: &str| semver::Version::parse(s).unwrap(); + assert_eq!(major_distance(&v("0.1.0"), &v("0.9.0")), 8); + assert_eq!(major_distance(&v("0.1.0"), &v("1.0.0")), 1); + assert_eq!(major_distance(&v("0.1.0"), &v("3.0.0")), 3); + // Monotonic in the major once past 1.0 — and, as the limitation says, *not* + // across the crossing: 8 for `0.9.0` against 1 for the `1.0.0` that follows it. + assert!( + major_distance(&v("0.1.0"), &v("3.0.0")) > major_distance(&v("0.1.0"), &v("1.0.0")) + ); + assert!( + major_distance(&v("0.1.0"), &v("0.9.0")) > major_distance(&v("0.1.0"), &v("1.0.0")) + ); + assert_eq!(major_distance(&v("1.0.0"), &v("3.0.0")), 2); + assert_eq!(major_distance(&v("4.0.0"), &v("1.0.0")), 0); + } + + /// `cvss.rs` scores only v3.0 and v3.1 vectors, so a v4-only advisory carries no + /// score and no band and is therefore *unrated* — which under the default `warn` + /// passes a `max_cvss` gate. This pins that behaviour so the day v4 scoring lands, + /// or the default changes, it is a deliberate edit and not a silent drift. + #[test] + fn a_v4_only_advisory_is_unrated_and_the_default_only_warns() { + let mut result = checked("openssl = \"0.10\""); + result.status = DependencyStatus::Vulnerable; + result.current_vulnerabilities = vec!["CVE-2025-0001".to_owned()]; + result.advisories = vec![Advisory::new("CVE-2025-0001").with_severity( + AdvisorySeverity::default().with_vector( + "CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:H/VA:H", + dependable_core::result::CvssVersion::V4, + ), + )]; + + let warned = evaluate(&rust(vec![result.clone()]), &policy("max_cvss = 7.0\n")); + assert!(!warned.has_violations(), "default is warn, not fail"); + assert!( + warned + .findings + .iter() + .any(|f| matches!(f.rule, Rule::MaxCvss | Rule::FailOnSeverity)) + ); + + // `fail` is how an operator closes this gap today. + let failed = evaluate( + &rust(vec![result]), + &policy("max_cvss = 7.0\nunrated_advisories = \"fail\"\n"), + ); + assert!(failed.has_violations()); + } } diff --git a/crates/dependable-report/src/sarif.rs b/crates/dependable-report/src/sarif.rs index 65e8de8..048bbfa 100644 --- a/crates/dependable-report/src/sarif.rs +++ b/crates/dependable-report/src/sarif.rs @@ -60,7 +60,7 @@ //! lands a line short of the version it points at. use std::collections::BTreeMap; -use std::path::{Component, Path}; +use std::path::{Component, Path, Prefix}; use dependable_core::result::{Advisory, Severity}; use dependable_core::{CheckResult, DependencyStatus, Item}; @@ -423,34 +423,111 @@ fn fingerprint( // URIs // --------------------------------------------------------------------------- -/// A manifest path as a SARIF `artifactLocation.uri`: relative to the report -/// root where possible, `/`-joined on every platform, percent-encoded. +/// A manifest path as a SARIF `artifactLocation.uri`. /// -/// A path outside `root` falls back to the path as given. -/// [`Path::components`] normalizes `.` away, so `./crates/app/Cargo.toml` -/// yields `crates/app/Cargo.toml` whether or not the prefix stripped. No -/// filesystem access: nothing here canonicalizes or probes. +/// A path under `root` becomes a relative URI, `/`-joined on every platform and +/// percent-encoded — the form GitHub code scanning wants, since the log carries no +/// `uriBaseId`. [`Path::components`] normalizes `.` away, so `./crates/app/Cargo.toml` +/// yields `crates/app/Cargo.toml` whether or not the prefix stripped. +/// +/// A path *outside* `root` cannot be expressed relatively, and emitting it as a bare +/// path-absolute string produced a URI nothing resolves: consumers reject an absolute +/// path with no base, and a Windows path additionally had its drive letter +/// percent-encoded into `C%3A/Users/...`. Such a path becomes an absolute `file:` URI +/// instead, where a drive prefix is legal and keeps its colon. A *relative* path outside +/// the root stays relative — it is already the form a consumer can resolve. +/// +/// "Outside the root" is decided by [`Path::has_root`] and not [`Path::is_absolute`], +/// which on Windows are not the same question: `/elsewhere/Cargo.toml` is rooted but +/// carries no drive, so `is_absolute` is false there and the same input produced a +/// `file:` URI on Unix and a bare path-absolute string on Windows. A rooted path is +/// exactly as unresolvable without a base on one platform as on the other, and a SARIF +/// log should not describe the same manifest differently for the machine that rendered +/// it. A drive-relative path (`C:foo`) is rooted by neither test and stays relative. +/// +/// No filesystem access: nothing here canonicalizes or probes. fn uri_for(root: &Path, path: &Path) -> String { - let relative = path.strip_prefix(root).unwrap_or(path); - let mut absolute = false; + if let Ok(relative) = path.strip_prefix(root) { + return encode_uri(&join_components(relative)); + } + if path.has_root() { + return absolute_file_uri(path); + } + encode_uri(&join_components(path)) +} + +/// `/`-join a path's components, dropping `.` and any root — but **keeping** a prefix. +/// +/// A drive-relative path (`C:foo`) is rooted by neither [`Path::has_root`] nor +/// [`Path::is_absolute`], so it arrives here, and dropping its prefix turned +/// `C:crates\app` into `crates/app` — the same URI a path on any other drive produces. +/// The prefix is emitted as an ordinary segment and percent-encoded with the rest +/// (`C%3A/crates/app`), because outside a `file:` URI's leading position a bare colon +/// is not the drive separator it is there. +fn join_components(path: &Path) -> String { + let parts: Vec = path + .components() + .filter_map(|component| match component { + Component::RootDir | Component::CurDir => None, + Component::Prefix(p) => Some(p.as_os_str().to_string_lossy().into_owned()), + Component::ParentDir => Some("..".to_string()), + Component::Normal(part) => Some(part.to_string_lossy().into_owned()), + }) + .collect(); + parts.join("/") +} + +/// How one Windows path prefix is spelled in a `file:` URI: an authority, and a leading +/// path segment. +/// +/// Split out from [`absolute_file_uri`] because the four prefix forms are the whole +/// difficulty and this is the only way to test them off Windows — [`Path`] parses a +/// prefix only there, while [`Prefix`] itself is spellable everywhere. +/// +/// - A drive (`C:`, and the verbatim `\\?\C:` that [`std::fs::canonicalize`] hands +/// back) is a path segment. The colon is legal in a `file:` URI path and encoding it +/// yields `C%3A`, which resolves to nothing. +/// - A UNC share (`\\server\share`, and its verbatim spelling) has a real authority: +/// `file://server/share/...`. Emitting it as a path produced `file://///server/...`, +/// which names a different thing. +/// - The verbatim and device namespaces (`\\?\Volume{…}`, `\\.\COM1`) name no +/// authority, so they stay path segments — encoded, because `?` unencoded opens a URI +/// query and truncated the path at `file:////`. +fn prefix_uri_parts(prefix: Prefix<'_>) -> (Option, Option) { + let drive = |letter: u8| Some(format!("{}:", letter.to_ascii_uppercase() as char)); + match prefix { + Prefix::Disk(letter) | Prefix::VerbatimDisk(letter) => (None, drive(letter)), + Prefix::UNC(server, share) | Prefix::VerbatimUNC(server, share) => ( + Some(encode_uri(&server.to_string_lossy())), + Some(encode_uri(&share.to_string_lossy())), + ), + Prefix::Verbatim(name) | Prefix::DeviceNS(name) => { + (None, Some(encode_uri(&name.to_string_lossy()))) + } + } +} + +/// An absolute path as a `file:` URI, with each segment percent-encoded. +fn absolute_file_uri(path: &Path) -> String { + let mut authority = String::new(); let mut parts: Vec = Vec::new(); - for component in relative.components() { + for component in path.components() { match component { - Component::Prefix(prefix) => { - parts.push(prefix.as_os_str().to_string_lossy().into_owned()); + Component::Prefix(p) => { + let (server, segment) = prefix_uri_parts(p.kind()); + if let Some(server) = server { + authority = server; + } + if let Some(segment) = segment { + parts.push(segment); + } } - Component::RootDir => absolute = true, - Component::CurDir => {} + Component::RootDir | Component::CurDir => {} Component::ParentDir => parts.push("..".to_string()), - Component::Normal(part) => parts.push(part.to_string_lossy().into_owned()), + Component::Normal(part) => parts.push(encode_uri(&part.to_string_lossy())), } } - let joined = parts.join("/"); - encode_uri(&if absolute { - format!("/{joined}") - } else { - joined - }) + format!("file://{authority}/{}", parts.join("/")) } /// Percent-encode every byte outside the URI-safe set, leaving `/` as the path @@ -946,7 +1023,9 @@ mod tests { // No uriBaseId: it would carry the developer's absolute path. assert!(location["artifactLocation"].get("uriBaseId").is_none()); - // A path outside the root falls back to the path as given. + // An absolute path outside the root becomes a `file:` URI. A bare + // path-absolute string is not resolvable by a consumer that was given no + // `uriBaseId`, which this log deliberately omits. let outside = rendered(&report_at( PathBuf::from("/repo"), PathBuf::from("/elsewhere/Cargo.toml"), @@ -954,7 +1033,7 @@ mod tests { )); assert_eq!( results_of(&outside)[0]["locations"][0]["physicalLocation"]["artifactLocation"]["uri"], - "/elsewhere/Cargo.toml" + "file:///elsewhere/Cargo.toml" ); // `.` components are normalized away even when the prefix does not strip. @@ -1265,4 +1344,139 @@ mod tests { assert_eq!(build(1_700_000_000), build(0)); } + + /// A Windows path had its drive prefix percent-encoded into `C%3A/...`, which names + /// nothing. In a `file:` URI the colon is legal and must survive; the encoding still + /// applies to the segments, where a space is real. + /// + /// Windows-only: elsewhere a backslash is an ordinary character and `C:\...` is one + /// relative component, so there is no drive prefix to preserve. The CI matrix runs + /// the suite on `windows-latest`, which is where this bites. + #[cfg(windows)] + #[test] + fn a_windows_path_keeps_its_drive_and_encodes_its_segments() { + let uri = uri_for( + Path::new(r"D:\repo"), + Path::new(r"C:\Users\dev\my project\Cargo.toml"), + ); + assert!(!uri.contains("%3A"), "the drive colon was encoded: {uri}"); + assert_eq!(uri, "file:///C:/Users/dev/my%20project/Cargo.toml"); + + // A rooted path carrying no drive is `is_absolute() == false` on Windows and + // true everywhere else. It is unresolvable without a base on both, so it takes + // the same `file:` form on both — a log must not describe one manifest two ways + // depending on the machine that rendered it. + assert_eq!( + uri_for(Path::new(r"D:\repo"), Path::new("/elsewhere/Cargo.toml")), + "file:///elsewhere/Cargo.toml" + ); + + // A drive-relative path (`C:foo`) is rooted by neither test: it resolves against + // that drive's working directory, so it stays relative — but it keeps naming its + // drive. This assertion used to read `crates/app/Cargo.toml`, which is the URI a + // path on *any* drive produces: the correction restores information the renderer + // was dropping, it does not relax the check. + assert_eq!( + uri_for(Path::new(r"D:\repo"), Path::new(r"C:crates\app\Cargo.toml")), + "C%3A/crates/app/Cargo.toml" + ); + + // A UNC share has a real authority. Rendering it as a path produced + // `file://///server/share/...`, which names a different thing. + assert_eq!( + uri_for( + Path::new(r"D:\repo"), + Path::new(r"\\server\share\repo\Cargo.toml") + ), + "file://server/share/repo/Cargo.toml" + ); + + // The verbatim form `std::fs::canonicalize` returns, and which `discover.rs` + // deliberately preserves. Emitting the prefix unencoded left `file:////?/C:/...`, + // where the `?` opens a URI query and truncates the path at `file:////`. + assert_eq!( + uri_for(Path::new(r"D:\repo"), Path::new(r"\\?\C:\repo\Cargo.toml")), + "file:///C:/repo/Cargo.toml" + ); + assert_eq!( + uri_for( + Path::new(r"D:\repo"), + Path::new(r"\\?\UNC\server\share\repo\Cargo.toml") + ), + "file://server/share/repo/Cargo.toml" + ); + } + + /// The prefix forms, off Windows. + /// + /// [`Path`] parses a prefix only on Windows, so the end-to-end assertions above run + /// on one platform in the CI matrix — which is how the UNC and verbatim spellings + /// went unnoticed. [`Prefix`] itself is spellable everywhere, so the decision each + /// form drives is checked on every platform the suite runs on. + #[test] + fn every_windows_prefix_form_has_a_uri_spelling() { + use std::ffi::OsStr; + + // A drive is a path segment, and its colon must survive: `C%3A` in the leading + // position of a `file:` URI resolves to nothing. + assert_eq!( + prefix_uri_parts(Prefix::Disk(b'C')), + (None, Some("C:".to_owned())) + ); + // `\?\C:\...` is what `std::fs::canonicalize` returns and what `simplified()` + // preserves; it names the same drive. + assert_eq!( + prefix_uri_parts(Prefix::VerbatimDisk(b'C')), + (None, Some("C:".to_owned())) + ); + + // A UNC share is an authority plus a first segment, not four extra slashes. + assert_eq!( + prefix_uri_parts(Prefix::UNC(OsStr::new("server"), OsStr::new("share"))), + (Some("server".to_owned()), Some("share".to_owned())) + ); + assert_eq!( + prefix_uri_parts(Prefix::VerbatimUNC( + OsStr::new("server"), + OsStr::new("share") + )), + (Some("server".to_owned()), Some("share".to_owned())) + ); + + // The verbatim and device namespaces name no authority, and every byte of them + // is encoded — an unencoded `?` opens a URI query and truncates the path. + let (authority, segment) = prefix_uri_parts(Prefix::Verbatim(OsStr::new("Volume{1}"))); + assert_eq!(authority, None); + let segment = segment.expect("a verbatim namespace names a segment"); + assert!(!segment.contains('{'), "{segment}"); + assert!(!segment.contains('}'), "{segment}"); + + assert_eq!( + prefix_uri_parts(Prefix::DeviceNS(OsStr::new("COM1"))), + (None, Some("COM1".to_owned())) + ); + } + + /// A space in a directory name still has to be encoded, in both forms. + #[test] + fn spaces_are_encoded_in_relative_and_absolute_uris() { + assert_eq!( + uri_for(Path::new("/repo"), Path::new("/repo/my app/Cargo.toml")), + "my%20app/Cargo.toml" + ); + assert_eq!( + uri_for(Path::new("/repo"), Path::new("/other dir/Cargo.toml")), + "file:///other%20dir/Cargo.toml" + ); + } + + /// A relative path that does not sit under the root is already resolvable; turning it + /// into a `file:` URI would invent a base it never had. + #[test] + fn a_relative_path_outside_the_root_stays_relative() { + assert_eq!( + uri_for(Path::new("/repo"), Path::new("crates/app/Cargo.toml")), + "crates/app/Cargo.toml" + ); + } } diff --git a/crates/dependable-report/src/summary.rs b/crates/dependable-report/src/summary.rs index e635b05..07335f7 100644 --- a/crates/dependable-report/src/summary.rs +++ b/crates/dependable-report/src/summary.rs @@ -39,6 +39,9 @@ pub struct Summary { pub vulnerable: usize, /// [`DependencyStatus::Error`] count. pub error: usize, + /// [`DependencyStatus::Undetermined`] count: real packages whose declared + /// version this run could not read. + pub undetermined: usize, /// [`DependencyStatus::Local`] count. pub local: usize, /// [`DependencyStatus::Git`] count. @@ -207,6 +210,7 @@ impl Report { DependencyStatus::Outdated => summary.outdated += 1, DependencyStatus::Vulnerable => summary.vulnerable += 1, DependencyStatus::Error(_) => summary.error += 1, + DependencyStatus::Undetermined => summary.undetermined += 1, DependencyStatus::Local => summary.local += 1, DependencyStatus::Git => summary.git += 1, // `DependencyStatus` is `#[non_exhaustive]`; an unrecognized diff --git a/crates/dependable-report/tests/bin.rs b/crates/dependable-report/tests/bin.rs deleted file mode 100644 index 6ae7ad9..0000000 --- a/crates/dependable-report/tests/bin.rs +++ /dev/null @@ -1,27 +0,0 @@ -//! The scaffolded binary, exercised the way a pipeline would invoke it. - -use std::process::Command; - -#[test] -fn the_binary_reports_that_it_is_a_scaffold_and_exits_2() { - let out = Command::new(env!("CARGO_BIN_EXE_dependable-report")) - .output() - .expect("run dependable-report"); - - assert_eq!( - out.status.code(), - Some(2), - "a scaffold must not look like a successful render" - ); - assert!( - out.stdout.is_empty(), - "stdout is the report; nothing else may be written there: {}", - String::from_utf8_lossy(&out.stdout) - ); - let stderr = String::from_utf8_lossy(&out.stderr); - assert!( - stderr.contains(dependable_report::VERSION), - "the version is how a user tells which scaffold they hit: {stderr}" - ); - assert!(stderr.contains("not implemented yet"), "{stderr}"); -} diff --git a/crates/dependable/Cargo.toml b/crates/dependable/Cargo.toml index 7922248..04965a8 100644 --- a/crates/dependable/Cargo.toml +++ b/crates/dependable/Cargo.toml @@ -19,6 +19,9 @@ name = "dependable" path = "src/main.rs" [dependencies] +# The `fix` command writes through a temporary file in the manifest's own directory +# and renames it into place, so an interrupted write cannot truncate a manifest. +tempfile.workspace = true dependable-fetch.workspace = true dependable-tui.workspace = true dependable-report = { workspace = true, optional = true } diff --git a/crates/dependable/src/cli.rs b/crates/dependable/src/cli.rs index bb2b6cb..f3223e0 100644 --- a/crates/dependable/src/cli.rs +++ b/crates/dependable/src/cli.rs @@ -75,7 +75,7 @@ pub struct CheckArgs { /// `include-if-current`. Overrides `[global] unstable`. #[arg(long, value_enum)] pub unstable: Option, - /// Ignore `Cargo.lock`. + /// Ignore the lockfile, checking declared constraints only.. #[arg(long)] pub no_lock_file: bool, /// Skip vulnerability scanning. @@ -90,9 +90,14 @@ pub struct CheckArgs { /// Output format. #[arg(long, value_enum, default_value_t = CheckFormat::Table)] pub format: CheckFormat, - /// Exit non-zero when results match this level. - #[arg(long, value_enum, default_value_t = FailOn::None)] - pub fail_on: FailOn, + /// Exit non-zero when results match this level. Overrides `[global] fail_on`. + /// + /// `Option` so that an explicit `--fail-on none` is distinguishable from the flag + /// being absent. With a plain default the two were the same value, so the documented + /// precedence — CLI over config — silently inverted for that one setting: a config + /// saying `fail_on = "any"` could not be turned off from the command line. + #[arg(long, value_enum)] + pub fail_on: Option, /// GitHub Actions annotations and job summary: `auto` (default, on under /// the runner), `always`, or `never`. #[arg(long, value_enum, default_value_t = AnnotationMode::Auto)] @@ -109,9 +114,6 @@ pub struct CheckArgs { /// Verbose logging (HTTP request details). #[arg(short, long)] pub verbose: bool, - /// Restrict to ecosystem(s) (reserved; V1 only checks Rust). - #[arg(long)] - pub ecosystem: Option, } #[derive(Args)] @@ -198,6 +200,12 @@ pub struct FixArgs { /// Print what would change without writing. #[arg(long)] pub dry_run: bool, + /// Ignore the on-disk registry cache (always fetch fresh). + #[arg(long)] + pub no_cache: bool, + /// Skip vulnerability scanning, so a vulnerable dependency is not upgraded for it. + #[arg(long)] + pub no_vuln: bool, #[arg(long, default_value_t = 3)] pub depth: usize, #[arg(long)] diff --git a/crates/dependable/src/config.rs b/crates/dependable/src/config.rs index fff8158..3654796 100644 --- a/crates/dependable/src/config.rs +++ b/crates/dependable/src/config.rs @@ -17,6 +17,7 @@ use dependable_report::policy::Policy; /// The full configuration, with sane defaults when the file is absent. #[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] pub struct Config { #[serde(default)] pub global: GlobalConfig, @@ -46,9 +47,25 @@ pub struct Config { /// By construction this is the same value [`load_policy`] returns — same /// figment, same key — so there is one schema and no way for the two to /// disagree. + /// + /// Declared in every build, and typed as the policy schema only where the + /// `report` feature can enforce it. `deny_unknown_fields` on this struct means an + /// absent field is a *rejected* field: with the declaration behind the feature, a + /// `--no-default-features` build failed to load any config carrying `[policy]` at + /// all, exiting 2 on "unknown field: found `policy`" — and the warning path that + /// exists precisely to say "this build cannot enforce your policy" was never + /// reached. The feature gates what is done with the block, not whether it is a + /// known key. #[cfg(feature = "report")] #[serde(default)] pub policy: Policy, + /// The `[policy]` block, unread. + /// + /// See the `report` build's field above: the key stays known so the file still + /// loads, and [`crate::runner`] warns that the gate is not enforced. + #[cfg(not(feature = "report"))] + #[serde(default)] + pub policy: figment::value::Dict, } impl Config { @@ -78,7 +95,7 @@ impl Config { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct GlobalConfig { pub concurrency: usize, pub include_ghsa: bool, @@ -100,7 +117,7 @@ impl Default for GlobalConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct RustConfig { pub enabled: bool, pub registry: String, @@ -116,7 +133,7 @@ impl Default for RustConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct GoConfig { pub enabled: bool, pub registry: String, @@ -132,7 +149,7 @@ impl Default for GoConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct NpmConfig { pub enabled: bool, pub registry: String, @@ -151,7 +168,7 @@ impl Default for NpmConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct PythonConfig { pub enabled: bool, pub registry: String, @@ -167,7 +184,7 @@ impl Default for PythonConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct PhpConfig { pub enabled: bool, pub registry: String, @@ -183,7 +200,7 @@ impl Default for PhpConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct DartConfig { pub enabled: bool, pub registry: String, @@ -199,7 +216,7 @@ impl Default for DartConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct CsharpConfig { pub enabled: bool, pub registry: String, @@ -215,7 +232,7 @@ impl Default for CsharpConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct ElixirConfig { pub enabled: bool, pub registry: String, @@ -231,7 +248,7 @@ impl Default for ElixirConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct JvmConfig { pub enabled: bool, pub registry: String, @@ -249,7 +266,7 @@ impl Default for JvmConfig { } #[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(default)] +#[serde(default, deny_unknown_fields)] pub struct VulnConfig { pub enabled: bool, pub osv_batch_url: String, @@ -266,14 +283,22 @@ impl Default for VulnConfig { /// Load configuration: defaults overlaid with `path` (if present). /// -/// A missing file is not an error — defaults are used. A malformed file falls -/// back to defaults as well (the runner surfaces nothing fatal for config). -#[must_use] -pub fn load_config(path: &Path) -> Config { +/// A missing file is not an error — defaults are used. +/// +/// # Errors +/// A file that is present but cannot be read into the schema is an error. It used to +/// fall back to `Config::default()`, which silently reset `[global] fail_on` to `none`: +/// one mistyped value anywhere in the file disarmed the CI gate, and the run then +/// exited 0 with nothing on stderr to say why. Unknown keys are rejected for the same +/// reason — `fail-on` with a hyphen was accepted and dropped — matching `[policy]`, +/// which has always rejected its own typos. +pub fn load_config(path: &Path) -> Result> { Figment::from(Serialized::defaults(Config::default())) .merge(Toml::file(path)) .extract() - .unwrap_or_default() + // Boxed: `figment::Error` is 200-odd bytes, and this sits on the hot success + // path of every subcommand. + .map_err(Box::new) } /// Where a `[policy]` block came from — or why there is none. @@ -341,6 +366,45 @@ pub fn has_policy_table(path: &Path) -> bool { Figment::from(Toml::file(path)).find_value("policy").is_ok() } +#[cfg(test)] +mod schema_tests { + use super::*; + + fn write(name: &str, content: &str) -> std::path::PathBuf { + let dir = std::env::temp_dir() + .join("dependable-config-schema-tests") + .join(name); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("create the scratch directory"); + let path = dir.join("dependable.toml"); + std::fs::write(&path, content).expect("write the config"); + path + } + + /// `[policy]` is a known key in every build, enforced or not. + /// + /// `deny_unknown_fields` turns "not declared" into "rejected", so declaring the + /// field only under the `report` feature made a `--no-default-features` build exit 2 + /// on any config carrying a policy block — including the configs the very warning + /// about unenforced policies exists to serve. + #[test] + fn a_policy_block_loads_whether_or_not_this_build_enforces_it() { + let path = write("policy_present", "[policy]\nmax_cvss = 7.0\n"); + let config = load_config(&path).expect("a config carrying `[policy]` must load"); + // The rest of the file is still read, so this is not a blanket "ignore + // everything" escape hatch. + assert!(config.rust.enabled); + } + + /// The other half of `deny_unknown_fields`: a key nothing declares is still a hard + /// error, in both builds. + #[test] + fn an_undeclared_key_is_still_rejected() { + let path = write("unknown_key", "[nonsense]\nvalue = 1\n"); + assert!(load_config(&path).is_err()); + } +} + #[cfg(all(test, feature = "report"))] mod tests { use std::fs; @@ -418,7 +482,7 @@ mod tests { }; assert_eq!(policy.allowed_licenses, vec!["MIT", "Apache-2.0"]); assert!(policy.requires_licenses()); - assert_eq!(load_config(&path).policy, policy); + assert_eq!(load_config(&path).expect("a valid config").policy, policy); } #[test] @@ -427,7 +491,51 @@ mod tests { // leaving a gate that looks configured and enforces nothing. let path = write("wrong_type", "[policy]\nmax_cvss = \"high\"\n"); assert!(load_policy(&path).is_err()); - assert_eq!(load_config(&path).policy, Policy::default()); + // `load_config` used to return `Policy::default()` here — the same silent + // fallback, one layer down. It now refuses the file outright. + assert!(load_config(&path).is_err()); + } + + /// One mistyped value used to reset the *whole* config to defaults, which meant + /// `[global] fail_on` silently became `none` and the CI gate was disarmed — with + /// nothing on stderr to say so. + #[test] + fn a_wrong_typed_value_does_not_silently_disarm_the_gate() { + let path = write( + "wrong_typed_global", + "[global]\nfail_on = \"vulnerable\"\nconcurrency = \"twenty\"\n", + ); + assert!( + load_config(&path).is_err(), + "a bad value must not become defaults" + ); + } + + /// `[policy]` has always rejected its own typos; `[global]` accepted and dropped + /// them, so `fail-on` with a hyphen left the gate off and looked configured. + #[test] + fn an_unknown_key_is_rejected_rather_than_ignored() { + for (name, body) in [ + ("hyphen_key", "[global]\nfail-on = \"vulnerable\"\n"), + ("typo_key", "[global]\nconcurency = 4\n"), + ("typo_table", "[globl]\nfail_on = \"any\"\n"), + ( + "typo_rust", + "[rust]\nregistery = \"https://example.test\"\n", + ), + ] { + let path = write(name, body); + assert!(load_config(&path).is_err(), "{name} was accepted"); + } + } + + /// A file that is simply absent is still not an error. + #[test] + fn a_missing_config_is_defaults() { + let path = scratch("missing_config").join("nope.toml"); + let cfg = load_config(&path).expect("a missing file is not an error"); + assert_eq!(cfg.global.fail_on, FailOn::None); + assert_eq!(cfg.global.concurrency, 20); } #[test] @@ -461,7 +569,7 @@ mod tests { let Ok(PolicySource::Configured(policy)) = load_policy(&path) else { panic!("expected a configured policy"); }; - assert_eq!(load_config(&path).policy, policy); + assert_eq!(load_config(&path).expect("a valid config").policy, policy); assert_eq!(policy.fail_on_severity, Some(Severity::High)); } } diff --git a/crates/dependable/src/fix.rs b/crates/dependable/src/fix.rs index ddc0c5f..b998a21 100644 --- a/crates/dependable/src/fix.rs +++ b/crates/dependable/src/fix.rs @@ -7,10 +7,11 @@ //! is not silently changed (e.g. an npm caret range is not turned into a pin). use std::collections::HashMap; +use std::io::Write as _; use std::path::Path; use anyhow::Context; -use dependable_fetch::{CheckResult, DependencyStatus}; +use dependable_fetch::{CheckResult, DependencyKind, DependencyStatus}; /// A single applied (or would-be-applied) version change. #[derive(Debug, Clone)] @@ -25,38 +26,103 @@ struct Edit { line: usize, start: usize, end: usize, + /// The text the span held when the plan was made. + /// + /// The span comes from a parse that happened before the network check, and the file + /// is read again at write time. If anything moved in between — an editor auto-save, a + /// `cargo add`, a concurrent `dependable fix` — the offsets now point somewhere else, + /// and splicing into them corrupts the manifest. Checking the text first is what + /// turns that into a refusal. + expected: String, replacement: String, } -/// Rewrite version constraints in `manifest` to the best available upgrade. +/// A manifest rewrite that has been computed but not yet written. +pub struct PlannedFix { + /// The manifest the rewrite applies to. + pub path: std::path::PathBuf, + /// The full new contents. + updated: String, + /// What changed, for reporting. + pub records: Vec, +} + +/// Compute the rewrite for `manifest` without touching it. /// /// Pinned (`=x.y.z`) deps are skipped unless `all` is set; multi-constraint forms /// (containing `,`) are skipped because they can't be rewritten to a single -/// version. With `dry_run`, nothing is written. +/// version. +/// +/// Planning is separated from writing so a multi-manifest run can compute every rewrite +/// before it writes any. Writing as it went left the tree half-rewritten when the third +/// of five manifests failed, with no record of the two already changed. /// /// # Errors -/// Returns an error if the manifest cannot be read or written. -pub fn apply_fixes( - manifest: &Path, - results: &[CheckResult], - all: bool, - dry_run: bool, -) -> anyhow::Result> { +/// Returns an error if the manifest cannot be read, or if a recorded span no longer +/// holds the constraint it was planned against. +pub fn plan(manifest: &Path, results: &[CheckResult], all: bool) -> anyhow::Result { let content = std::fs::read_to_string(manifest) .with_context(|| format!("reading {}", manifest.display()))?; - let (updated, records) = plan_fixes(&content, results, all); - if !dry_run && !records.is_empty() { - std::fs::write(manifest, updated) - .with_context(|| format!("writing {}", manifest.display()))?; + let (updated, records) = plan_fixes(&content, results, all) + .with_context(|| format!("rewriting {}", manifest.display()))?; + Ok(PlannedFix { + path: manifest.to_path_buf(), + updated, + records, + }) +} + +/// Write a planned rewrite, atomically. +/// +/// The new contents go to a temporary file in the manifest's own directory and are +/// renamed over it, so a crash, a full disk, or a `SIGINT` leaves the original intact. +/// `fs::write` truncates first, which meant an interrupted write left a manifest empty +/// or half-written and no backup to recover from. +/// +/// # Errors +/// Returns an error if the temporary file cannot be created, written, or renamed. +pub fn commit(planned: &PlannedFix) -> anyhow::Result<()> { + if planned.records.is_empty() { + return Ok(()); } - Ok(records) + let directory = planned.path.parent().unwrap_or_else(|| Path::new(".")); + let mut temp = tempfile::NamedTempFile::new_in(directory).with_context(|| { + format!( + "creating a temporary file beside {}", + planned.path.display() + ) + })?; + temp.write_all(planned.updated.as_bytes()) + .with_context(|| format!("writing {}", planned.path.display()))?; + // Flush to the filesystem before the rename, so the rename cannot publish a file + // whose contents are still only in memory. + temp.as_file() + .sync_all() + .with_context(|| format!("flushing {}", planned.path.display()))?; + // A manifest is usually 0644 while a temporary file is 0600; preserve what was there. + #[cfg(unix)] + if let Ok(metadata) = std::fs::metadata(&planned.path) { + use std::os::unix::fs::PermissionsExt as _; + let _ = temp + .as_file() + .set_permissions(std::fs::Permissions::from_mode( + metadata.permissions().mode(), + )); + } + temp.persist(&planned.path) + .with_context(|| format!("replacing {}", planned.path.display()))?; + Ok(()) } /// Compute the rewritten manifest and the applied records from `content` and the /// check `results`, with no filesystem IO (the file boundary lives in /// [`apply_fixes`]). Format-agnostic: it edits each recorded version span in place, /// so JSON, YAML, and TOML manifests are rewritten without reformatting. -fn plan_fixes(content: &str, results: &[CheckResult], all: bool) -> (String, Vec) { +fn plan_fixes( + content: &str, + results: &[CheckResult], + all: bool, +) -> anyhow::Result<(String, Vec)> { let mut edits: Vec = Vec::new(); let mut records = Vec::new(); for result in results { @@ -68,6 +134,14 @@ fn plan_fixes(content: &str, results: &[CheckResult], all: bool) -> (String, Vec if !item.is_rewritable() { continue; } + // An override is a version this manifest deliberately forces onto the resolved + // tree — very often a security pin holding a transitive dependency above a + // vulnerable release. Reporting that a newer version exists is useful; rewriting + // the pin to it defeats the reason the entry was written, so `fix` declines the + // whole kind rather than trying to guess which overrides are safe to move. + if item.kind == DependencyKind::Override { + continue; + } let updatable = matches!( result.status, DependencyStatus::PatchAvailable @@ -96,6 +170,7 @@ fn plan_fixes(content: &str, results: &[CheckResult], all: bool) -> (String, Vec line: item.version_line, start: item.version_col_start, end: item.version_col_end, + expected: item.version_constraint.clone(), replacement: new_constraint.clone(), }); records.push(FixRecord { @@ -108,9 +183,9 @@ fn plan_fixes(content: &str, results: &[CheckResult], all: bool) -> (String, Vec let updated = if edits.is_empty() { content.to_string() } else { - apply_edits(content, &edits) + apply_edits(content, &edits)? }; - (updated, records) + Ok((updated, records)) } /// Build a new constraint from `original`, preserving its leading operator/`v` @@ -186,7 +261,7 @@ fn is_wildcard(rest: &str) -> bool { /// Apply byte-range edits to `content`, operating per line. Edits on the same /// line are applied right-to-left so earlier offsets stay valid. -fn apply_edits(content: &str, edits: &[Edit]) -> String { +fn apply_edits(content: &str, edits: &[Edit]) -> anyhow::Result { let mut by_line: HashMap> = HashMap::new(); for edit in edits { by_line.entry(edit.line).or_default().push(edit); @@ -201,19 +276,78 @@ fn apply_edits(content: &str, edits: &[Edit]) -> String { sorted.sort_by_key(|edit| std::cmp::Reverse(edit.start)); let mut s = line.to_string(); for edit in sorted { - if edit.start <= edit.end && edit.end <= s.len() { - s.replace_range(edit.start..edit.end, &edit.replacement); - } + // A bounds check alone only proves the span is *inside* the file, not that it + // still points at the constraint. The content is re-read after the network + // check, so anything that edited the file in between shifts every later + // offset — and the splice would land on whatever now occupies them. + let found = s + .get(edit.start..edit.end) + .filter(|found| *found == edit.expected); + let Some(_) = found else { + anyhow::bail!( + "the manifest changed while it was being checked: expected `{}` at line {}, \ + found `{}` — nothing was written; re-run to pick up the new contents", + edit.expected, + edit.line + 1, + s.get(edit.start..edit.end).unwrap_or("") + ); + }; + s.replace_range(edit.start..edit.end, &edit.replacement); } out.push_str(&s); } - out + Ok(out) } #[cfg(test)] mod tests { use super::*; + /// An override forces a version onto the resolved tree — frequently a security pin. + /// `fix --all` used to rewrite it to the newest release, undoing the pin; combined + /// with a `>`-scoped pnpm key it rewrote it to an unrelated package's newest release. + #[test] + fn fix_all_leaves_an_override_alone() { + let content = r#"{ + "dependencies": { + "monolog": "^2.0" + }, + "overrides": { + "minimist": "1.2.6" + } +} +"#; + let results = results_for( + ManifestKind::PackageJson, + content, + &[("minimist", "1.2.8"), ("monolog", "2.9.1")], + ); + assert_eq!(results.len(), 2, "the fixture must produce two items"); + assert!( + results + .iter() + .any(|r| r.item.kind == DependencyKind::Override), + "the fixture must produce an override item" + ); + + let (updated, records) = plan_fixes(content, &results, true).expect("the plan applies"); + + // The override is declined; its non-override neighbour is still fixed, so this + // asserts the guard rather than a `--all` path that happens to do nothing. + assert_eq!( + records + .iter() + .map(|record| record.name.as_str()) + .collect::>(), + ["monolog"], + "{records:?}" + ); + assert!( + updated.contains(r#""minimist": "1.2.6""#), + "the override was rewritten: {updated}" + ); + } + #[test] fn rewrite_preserves_operator_prefix() { assert_eq!( @@ -397,9 +531,10 @@ mod tests { line: 1, start: 9, end: 13, + expected: "^1.0".to_string(), replacement: "^1.5.0".to_string(), }]; - let out = apply_edits(content, &edits); + let out = apply_edits(content, &edits).expect("the span still holds `^1.0`"); assert_eq!(out, "[dependencies]\nserde = \"^1.5.0\"\n"); } @@ -412,16 +547,18 @@ mod tests { line: 0, start: 2, end: 5, + expected: "1.0".to_string(), replacement: "1.9".to_string(), }, Edit { line: 0, start: 8, end: 11, + expected: "2.0".to_string(), replacement: "2.9".to_string(), }, ]; - let out = apply_edits(content, &edits); + let out = apply_edits(content, &edits).expect("both spans still hold their text"); assert_eq!(out, "a=1.9 b=2.9\n"); } @@ -475,7 +612,7 @@ mod tests { content, &[("react", "18.2.0"), ("typescript", "5.4.5")], ); - let (updated, records) = plan_fixes(content, &results, false); + let (updated, records) = plan_fixes(content, &results, false).expect("the plan applies"); assert_eq!( updated, @@ -513,7 +650,7 @@ mod tests { content, &[("monolog/monolog", "2.9.1")], ); - let (updated, records) = plan_fixes(content, &results, false); + let (updated, records) = plan_fixes(content, &results, false).expect("the plan applies"); assert_eq!( updated, @@ -539,7 +676,7 @@ mod tests { content, &[("http", "1.2.0"), ("provider", "6.1.0")], ); - let (updated, records) = plan_fixes(content, &results, false); + let (updated, records) = plan_fixes(content, &results, false).expect("the plan applies"); // Versions bumped, indentation and the trailing comment untouched. assert_eq!( @@ -583,7 +720,7 @@ mod tests { "the old guards would both have passed" ); - let (updated, records) = plan_fixes(member, &results, false); + let (updated, records) = plan_fixes(member, &results, false).expect("the plan applies"); assert!(records.is_empty(), "{records:?}"); assert_eq!( @@ -612,12 +749,59 @@ mod tests { ); assert_eq!(declaration.version_line, 1, "and the span points at it"); - let (updated, records) = plan_fixes(root, &results, false); + let (updated, records) = plan_fixes(root, &results, false).expect("the plan applies"); assert_eq!(records.len(), 1, "{records:?}"); assert_eq!(updated, "[workspace.dependencies]\nserde = \"1.0.219\"\n"); } + /// The span is computed from a parse that happened before the network check, and the + /// file is read again at write time. If it moved in between — an editor auto-save, a + /// `cargo add`, a concurrent `fix` — the offsets point at different bytes now. The + /// old bounds check only proved the span was inside the file, so the splice landed on + /// whatever now occupied it and the manifest was silently corrupted. + #[test] + fn a_span_that_no_longer_holds_its_constraint_is_refused() { + let content = "[dependencies]\nserde = \"^1.0\"\n"; + // The same span, against content where a line was inserted above it. + let shifted = "[dependencies]\n# a comment someone just added\nserde = \"^1.0\"\n"; + let edits = vec![Edit { + line: 1, + start: 9, + end: 13, + expected: "^1.0".to_string(), + replacement: "^1.5.0".to_string(), + }]; + + assert!( + apply_edits(content, &edits).is_ok(), + "the unshifted file still applies" + ); + + let err = apply_edits(shifted, &edits).expect_err("a moved span must be refused"); + let message = err.to_string(); + assert!( + message.contains("changed while it was being checked"), + "{message}" + ); + assert!(message.contains("nothing was written"), "{message}"); + } + + /// A span running past the end of its line is refused rather than silently skipped: + /// the old code's bounds check dropped such an edit and reported success for it. + #[test] + fn an_out_of_range_span_is_refused_not_skipped() { + let content = "a=1.0\n"; + let edits = vec![Edit { + line: 0, + start: 2, + end: 99, + expected: "1.0".to_string(), + replacement: "1.9".to_string(), + }]; + assert!(apply_edits(content, &edits).is_err()); + } + /// The end-to-end shape of issue #87, with no flags and no lockfile. /// /// `1.x` matches `1.0.0` and `1.9.0` but not `2.0.0`, so `check_version` reports @@ -641,7 +825,7 @@ mod tests { "the fixture must produce a checkable item" ); - let (updated, records) = plan_fixes(content, &results, false); + let (updated, records) = plan_fixes(content, &results, false).expect("the plan applies"); assert!(records.is_empty(), "{records:?}"); assert_eq!(updated, content, "the manifest must be byte-identical"); @@ -677,7 +861,7 @@ mod tests { "the `--all` branch reads `latest_available`, so the fixture must set it" ); - let (updated, records) = plan_fixes(content, &results, true); + let (updated, records) = plan_fixes(content, &results, true).expect("the plan applies"); // The wildcard is declined; its non-wildcard neighbour still gets fixed, so // this asserts the guard and not a `--all` path that simply does nothing. diff --git a/crates/dependable/src/output/github.rs b/crates/dependable/src/output/github.rs index b2e5e26..feba416 100644 --- a/crates/dependable/src/output/github.rs +++ b/crates/dependable/src/output/github.rs @@ -109,7 +109,11 @@ fn level_of(status: &DependencyStatus) -> Option { match status { DependencyStatus::Vulnerable => Some(Level::Error), DependencyStatus::Outdated | DependencyStatus::UpdateAvailable => Some(Level::Warning), - DependencyStatus::Error(_) => Some(Level::Notice), + // Both are "this dependency was not checked": one because the registry or the + // fetch failed, one because the declared version could not be read. A plain + // `Error` got a notice and `Undetermined` got nothing at all, so the status made + // honest elsewhere was the one status a pull request never heard about. + DependencyStatus::Error(_) | DependencyStatus::Undetermined => Some(Level::Notice), _ => None, } } @@ -374,6 +378,9 @@ fn message(finding: &Finding<'_>, level: Level) -> String { ), Level::Notice => match &result.status { DependencyStatus::Error(why) => format!("{name} could not be checked: {why}"), + DependencyStatus::Undetermined => { + format!("{name} could not be checked: its declared version could not be read") + } other => format!("{name}: {}", other.label()), }, }; @@ -811,6 +818,7 @@ mod tests { fn report(path: &str, results: Vec) -> ManifestReport { ManifestReport { + integrity: crate::output::ScanIntegrity::default(), path: PathBuf::from(path), ecosystem: Ecosystem::Rust, results, @@ -1134,4 +1142,22 @@ mod tests { assert_eq!(cell("a|b\nc"), "a\\|b c"); assert_eq!(code_cell("a`b`c"), "`abc`"); } + + /// Both statuses mean "this dependency was not checked", and a pull request has to + /// hear about both. `Undetermined` produced no annotation at all while a plain + /// `Error` got a notice, so the one status that says "we could not read this" was + /// the one nobody saw. + #[test] + fn a_dependency_that_could_not_be_checked_is_annotated_either_way() { + assert_eq!( + level_of(&DependencyStatus::Undetermined), + Some(Level::Notice) + ); + assert_eq!( + level_of(&DependencyStatus::Error("boom".to_owned())), + Some(Level::Notice) + ); + assert_eq!(level_of(&DependencyStatus::UpToDate), None); + assert_eq!(level_of(&DependencyStatus::PatchAvailable), None); + } } diff --git a/crates/dependable/src/output/json.rs b/crates/dependable/src/output/json.rs index eaf10d4..c708bd7 100644 --- a/crates/dependable/src/output/json.rs +++ b/crates/dependable/src/output/json.rs @@ -27,6 +27,8 @@ struct SummaryDto { outdated: usize, vulnerable: usize, error: usize, + /// Additive: real packages whose declared version this run could not read. + undetermined: usize, } #[derive(Serialize)] @@ -94,6 +96,7 @@ pub fn render(reports: &[ManifestReport]) -> anyhow::Result<()> { outdated: summary.outdated, vulnerable: summary.vulnerable, error: summary.error, + undetermined: summary.undetermined, }, results, }; diff --git a/crates/dependable/src/output/list.rs b/crates/dependable/src/output/list.rs index 04fe9e9..28399f8 100644 --- a/crates/dependable/src/output/list.rs +++ b/crates/dependable/src/output/list.rs @@ -270,6 +270,10 @@ fn source_token(source: PackageSource) -> &'static str { PackageSource::Local => "local", PackageSource::Git => "git", PackageSource::Inherited => "inherited", + // Its own token, not the catch-all: a dangling `$name` override is a real + // published package whose version this manifest does not state, and a consumer + // that cannot tell it from any other unknown source cannot report it. + PackageSource::Unresolved => "unresolved", _ => "unknown", } } @@ -304,7 +308,27 @@ fn annotation(item: &Item) -> &'static str { DependencyKind::Peer => " (peer)", DependencyKind::Workspace => " (declared)", DependencyKind::Indirect => " (indirect)", + DependencyKind::Override => " (override)", _ => "", }, } } + +#[cfg(test)] +mod tests { + use super::*; + + /// Every source a manifest can state gets its own token. A dangling `$name` override + /// is a real published package whose version this manifest does not state, and it + /// serialized as `"unknown"` — indistinguishable, to a consumer of + /// `list --format json`, from any other source the tool has no name for. + #[test] + fn every_known_source_has_its_own_token() { + assert_eq!(source_token(PackageSource::Registry), "registry"); + assert_eq!(source_token(PackageSource::Jsr), "jsr"); + assert_eq!(source_token(PackageSource::Local), "local"); + assert_eq!(source_token(PackageSource::Git), "git"); + assert_eq!(source_token(PackageSource::Inherited), "inherited"); + assert_eq!(source_token(PackageSource::Unresolved), "unresolved"); + } +} diff --git a/crates/dependable/src/output/mod.rs b/crates/dependable/src/output/mod.rs index 5f20b25..47df136 100644 --- a/crates/dependable/src/output/mod.rs +++ b/crates/dependable/src/output/mod.rs @@ -25,6 +25,47 @@ pub struct ManifestReport { /// The manifest whose `[workspace.dependencies]` supplied any inherited constraint. /// `None` outside a workspace. pub workspace_root: Option, + /// Whether this manifest's results are complete enough to gate a build on. + /// + /// A scan that could not run produces the same empty advisory lists as a scan that + /// found nothing, so without this the exit code cannot tell a clean project from an + /// unreachable OSV. + pub integrity: ScanIntegrity, +} + +/// How much of what a gate needs was actually established for one manifest. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct ScanIntegrity { + /// The vulnerability scan was asked for and did not complete. + pub vulnerability_scan_failed: bool, + /// A registry lookup failed for a reason other than the package not existing. + /// + /// This, and not the count below, is what a `--fail-on` gate cannot be honoured + /// through: the registry declined to answer, so the run has no facts about the + /// dependencies it asked about. + pub registry_unreachable: bool, + /// How many dependencies a registry answered about by name: no such package. + /// + /// Reported, never gated on. Each is a permanent fact about one dependency — a + /// private or internal package, one served by a registry this run does not route to, + /// a deleted package — and says nothing about the ones that did resolve, so it must + /// not turn a whole gate into a failure; it is said plainly on stderr instead. + /// + /// Counted from [`CheckResult::registry_not_found`], never from the error message: + /// the carve-out is only safe while it covers exactly the answers a registry gave. + /// + /// [`CheckResult::registry_not_found`]: dependable_fetch::CheckResult::registry_not_found + pub unresolved: usize, + /// How many dependencies this run failed to evaluate on its own. + /// + /// An `Error` that no registry ever produced: a constraint written in a dialect that + /// did not parse, a dependency whose fetch task never ran. Nothing was established + /// about the package, and unlike a 404 there is no fact to report in place of a + /// status — so a `--fail-on` gate cannot be honoured over it, exactly as it could + /// not before the 404 carve-out existed. Folding these in with the 404s let two + /// unparseable constraints pass `--fail-on vulnerable` under a note claiming the + /// registry had not found them. + pub unevaluated: usize, } /// Aggregate status counts across one or more reports. @@ -53,6 +94,9 @@ pub struct Summary { pub outdated: usize, pub vulnerable: usize, pub error: usize, + /// Real packages whose declared version this run could not read — an + /// untranslatable constraint, or a reference to something never declared. + pub undetermined: usize, pub local: usize, pub git: usize, } @@ -76,6 +120,7 @@ impl Summary { DependencyStatus::Outdated => s.outdated += 1, DependencyStatus::Vulnerable => s.vulnerable += 1, DependencyStatus::Error(_) => s.error += 1, + DependencyStatus::Undetermined => s.undetermined += 1, DependencyStatus::Local => s.local += 1, DependencyStatus::Git => s.git += 1, _ => {} @@ -170,6 +215,7 @@ mod tests { fn report(path: &str, declarations: &[(&str, DependencyStatus)]) -> ManifestReport { ManifestReport { + integrity: ScanIntegrity::default(), path: PathBuf::from(path), ecosystem: Ecosystem::Rust, results: declarations diff --git a/crates/dependable/src/output/sarif.rs b/crates/dependable/src/output/sarif.rs index 112c81b..13523e9 100644 --- a/crates/dependable/src/output/sarif.rs +++ b/crates/dependable/src/output/sarif.rs @@ -80,6 +80,7 @@ mod tests { fn report(path: &str) -> ManifestReport { ManifestReport { + integrity: crate::output::ScanIntegrity::default(), path: PathBuf::from(path), ecosystem: Ecosystem::Rust, results: Vec::new(), diff --git a/crates/dependable/src/output/table.rs b/crates/dependable/src/output/table.rs index d23b753..bb887d9 100644 --- a/crates/dependable/src/output/table.rs +++ b/crates/dependable/src/output/table.rs @@ -107,6 +107,8 @@ fn status_cell(result: &CheckResult) -> String { Style::new().yellow() } DependencyStatus::Outdated | DependencyStatus::Error(_) => Style::new().red(), + // Not green: an unreadable constraint is not evidence of currency. + DependencyStatus::Undetermined => Style::new().yellow(), DependencyStatus::Vulnerable => Style::new().red().bold(), DependencyStatus::Local | DependencyStatus::Git => Style::new().dimmed(), _ => Style::new(), @@ -144,6 +146,9 @@ fn print_totals(summary: &Summary) { if summary.error > 0 { parts.push(format!("{} error", summary.error)); } + if summary.undetermined > 0 { + parts.push(format!("{} undetermined", summary.undetermined)); + } let skipped = summary.local + summary.git; if skipped > 0 { parts.push(format!("{skipped} skipped")); diff --git a/crates/dependable/src/output/tree.rs b/crates/dependable/src/output/tree.rs index a3bbadf..8268654 100644 --- a/crates/dependable/src/output/tree.rs +++ b/crates/dependable/src/output/tree.rs @@ -50,6 +50,13 @@ fn ascii(graph: &DependencyGraph, opts: &TreeOptions) -> String { } write_node(&mut out, graph, root, "", true, true); } + // A tree that ran out of budget is a prefix, and a prefix that does not say so reads + // as the whole graph. `--no-dedupe` on a large lockfile is how a reader gets here. + if tree.truncated { + out.push_str( + "\n(tree truncated: too many paths to draw — narrow it with --depth, or drop --no-dedupe)\n", + ); + } out } @@ -134,6 +141,13 @@ struct FlatGraph { edges: Vec<(usize, usize)>, /// Compact ids of the roots. roots: Vec, + /// Whether the walk ran out of budget, so this graph is a prefix of the real one. + /// + /// The ASCII renderer prints a notice for this; JSON and DOT have to carry it too, + /// and they are the formats whose consumer cannot eyeball the difference. A + /// truncated machine-readable graph that does not say so is byte-indistinguishable + /// from a complete one. + truncated: bool, } fn flatten(graph: &DependencyGraph, opts: &TreeOptions) -> FlatGraph { @@ -171,6 +185,7 @@ fn flatten(graph: &DependencyGraph, opts: &TreeOptions) -> FlatGraph { order, edges, roots, + truncated: tree.truncated, } } @@ -189,6 +204,10 @@ struct GraphDto<'a> { roots: Vec, nodes: Vec>, edges: Vec, + /// Additive: `true` when the walk ran out of budget and this graph is a prefix of + /// the real one. Always present, so a consumer can require it rather than infer + /// completeness from its absence. + truncated: bool, } #[derive(Serialize)] @@ -230,6 +249,7 @@ fn json(graph: &DependencyGraph, opts: &TreeOptions) -> anyhow::Result { roots: flat.roots, nodes, edges, + truncated: flat.truncated, }; Ok(serde_json::to_string_pretty(&dto)?) } @@ -240,6 +260,13 @@ fn dot(graph: &DependencyGraph, opts: &TreeOptions) -> String { let mut out = String::from( "digraph dependencies {\n rankdir=LR;\n node [shape=box, fontname=\"monospace\"];\n", ); + // A graph attribute rather than only a comment, so a tool reading the DOT — and not + // just a person reading the file — can see that this is a prefix of the real graph. + if flat.truncated { + out.push_str( + " // tree truncated: too many paths to draw — narrow it with --depth, or drop --no-dedupe\n dependable_truncated=true;\n", + ); + } for (id, &orig) in flat.order.iter().enumerate() { let n = &graph.nodes()[orig]; let label = if n.version.is_empty() { @@ -369,6 +396,62 @@ source = "registry+https://x" assert!(!out.contains("serde")); } + /// A chain longer than the walk's hard recursion ceiling, so every renderer sees a + /// tree that stopped short. + fn deep_chain() -> DependencyGraph { + const DEPTH: usize = 600; + let mut lock = String::new(); + for n in 0..DEPTH { + lock.push_str("[[package]]\n"); + let _ = writeln!(lock, "name = \"c{n}\""); + lock.push_str("version = \"1.0.0\"\n"); + if n > 0 { + lock.push_str("source = \"registry+https://x\"\n"); + } + if n + 1 < DEPTH { + let _ = writeln!(lock, "dependencies = [\"c{}\"]", n + 1); + } + lock.push('\n'); + } + let resolved = parse_cargo_lock_graph(&lock).unwrap(); + let names = ["c0".to_owned()].into_iter().collect(); + DependencyGraph::from_resolved(&resolved, &names, &["c0".to_owned()]) + } + + /// The ASCII renderer says a truncated walk is truncated. So must the machine + /// formats — they are the ones whose consumer cannot see the difference, and a JSON + /// graph that stopped short used to be byte-indistinguishable from a complete one. + #[test] + fn every_format_reports_a_truncated_walk() { + let graph = deep_chain(); + let opts = TreeOptions::default(); + assert!( + flatten(&graph, &opts).truncated, + "the fixture must actually truncate, or the assertions below prove nothing" + ); + + let ascii = ascii(&graph, &opts); + assert!(ascii.contains("(tree truncated"), "{ascii}"); + + let json = json(&graph, &opts).unwrap(); + assert!(json.contains("\"truncated\": true"), "{json}"); + + let dot = dot(&graph, &opts); + assert!(dot.contains("dependable_truncated=true;"), "{dot}"); + assert!(dot.contains("// tree truncated"), "{dot}"); + } + + /// And a complete walk says so too, rather than leaving the key out — a consumer + /// must be able to require the flag, not infer completeness from its absence. + #[test] + fn a_complete_walk_reports_itself_complete() { + let json = json(&sample(), &TreeOptions::default()).unwrap(); + assert!(json.contains("\"truncated\": false"), "{json}"); + + let dot = dot(&sample(), &TreeOptions::default()); + assert!(!dot.contains("dependable_truncated"), "{dot}"); + } + #[test] fn json_has_nodes_edges_and_roots() { let out = json(&sample(), &TreeOptions::default()).unwrap(); diff --git a/crates/dependable/src/runner.rs b/crates/dependable/src/runner.rs index 73ba766..cb0b082 100644 --- a/crates/dependable/src/runner.rs +++ b/crates/dependable/src/runner.rs @@ -15,11 +15,11 @@ use dependable_fetch::core::{ parse_cargo_config, parse_npmrc, parse_project, parse_workspace, resolve_workspace_inheritance, }; use dependable_fetch::{ - CheckError, Checker, DependencyStatus, Ecosystem, GoProxyFetcher, GraphSource, HexFetcher, - Item, JsrFetcher, ManifestKind, MavenCentralFetcher, NpmFetcher, NuGetFetcher, PackageSource, - PackagistFetcher, ParseError, ProgressEvent, PubDevFetcher, PyPiFetcher, ScopedRegistry, - TreeOptions, UnstableFilter, WorkspaceGraphOptions, build_client, build_workspace_graph, - nearest_workspace_root, workspace_source, + CheckError, Checker, DependencyStatus, Ecosystem, ErrorOrigin, GoProxyFetcher, GraphSource, + HexFetcher, Item, JsrFetcher, ManifestKind, MavenCentralFetcher, NpmFetcher, NuGetFetcher, + PackageSource, PackagistFetcher, ParseError, ProgressEvent, PubDevFetcher, PyPiFetcher, + ScopedRegistry, TreeOptions, UnstableFilter, WorkspaceGraphOptions, build_client, + build_workspace_graph, nearest_workspace_root, workspace_source, }; use dependable_tui::TuiOptions; use globset::{GlobBuilder, GlobSet, GlobSetBuilder}; @@ -31,7 +31,7 @@ use crate::config::{Config, load_config}; use crate::config::{PolicySource, load_policy}; use crate::fix; use crate::output::list::ProjectReport; -use crate::output::{self, ManifestReport}; +use crate::output::{self, ManifestReport, ScanIntegrity}; /// Effective settings after layering CLI flags over env vars over config. struct Settings { @@ -50,10 +50,33 @@ struct Settings { osv_url: String, } +/// Read a `DEPENDABLE_*` switch by its **value**, not merely by its presence. +/// +/// `std::env::var_os(..).is_some()` reads `DEPENDABLE_NO_VULN=0`, `=false`, and the +/// empty string an unset GitHub Actions expression expands to as "yes, turn the scan +/// off" — so a workflow writing `DEPENDABLE_NO_VULN: ${{ inputs.no-vuln }}` disabled +/// vulnerability scanning on every run, silently, whether or not the input was set. A +/// switch that cannot be spelled `off` is a switch that cannot be left alone. +/// +/// Unset, empty, `0`, `false`, `no` and `off` are all off; anything else is on. +fn env_flag(name: &str) -> bool { + flag_is_on(std::env::var(name).ok().as_deref()) +} + +/// [`env_flag`]'s decision, without the environment — the half that is testable. +fn flag_is_on(raw: Option<&str>) -> bool { + raw.is_some_and(|raw| { + !matches!( + raw.trim().to_ascii_lowercase().as_str(), + "" | "0" | "false" | "no" | "off" + ) + }) +} + fn resolve_check_settings(args: &CheckArgs, cfg: &Config) -> Settings { - let env_no_vuln = std::env::var_os("DEPENDABLE_NO_VULN").is_some(); - let env_no_cache = std::env::var_os("DEPENDABLE_NO_CACHE").is_some(); - let env_ghsa = std::env::var_os("DEPENDABLE_INCLUDE_GHSA").is_some(); + let env_no_vuln = env_flag("DEPENDABLE_NO_VULN"); + let env_no_cache = env_flag("DEPENDABLE_NO_CACHE"); + let env_ghsa = env_flag("DEPENDABLE_INCLUDE_GHSA"); let env_concurrency = std::env::var("DEPENDABLE_CONCURRENCY") .ok() .and_then(|s| s.parse::().ok()); @@ -61,11 +84,11 @@ fn resolve_check_settings(args: &CheckArgs, cfg: &Config) -> Settings { .ok() .and_then(|s| FailOn::from_env(&s)); - let fail_on = if args.fail_on != FailOn::None { - args.fail_on - } else { - env_fail_on.unwrap_or(cfg.global.fail_on) - }; + // Documented precedence, honoured for every value: CLI, then env, then config. + // The old test `args.fail_on != FailOn::None` could not tell an explicit + // `--fail-on none` from clap's default, so a config `fail_on` could not be turned + // off from the command line at all. + let fail_on = args.fail_on.or(env_fail_on).unwrap_or(cfg.global.fail_on); Settings { concurrency: args @@ -78,6 +101,9 @@ fn resolve_check_settings(args: &CheckArgs, cfg: &Config) -> Settings { check_vuln: cfg.vulnerability.enabled && !args.no_vuln && !env_no_vuln, licenses: policy_requires_licenses(cfg), cache: !args.no_cache && !env_no_cache, + // `--include-ghsa` is a flag, so absence is indistinguishable from `false` and + // it can only ever widen the scan. OR-ing is therefore the whole contract: any + // layer asking for GHSA gets it, and no layer can silently take it away. include_ghsa: args.include_ghsa || cfg.global.include_ghsa || env_ghsa, fail_on, unstable: args @@ -257,11 +283,28 @@ impl Engine { for warning in &check.warnings { eprintln!("warning: {} — {warning}", path.display()); } + // Split by provenance, not by status: a 404 is the registry answering, + // and anything else that produced an `Error` is this run failing to + // evaluate the dependency. Only the first is exempt from the gate. + let count = |origin| { + check + .results + .iter() + .filter(|r| r.error_origin == origin) + .count() + }; + let integrity = ScanIntegrity { + vulnerability_scan_failed: check.vulnerability_scan_failed, + registry_unreachable: check.registry_unreachable, + unresolved: count(ErrorOrigin::NotFound), + unevaluated: count(ErrorOrigin::Local), + }; Ok(Some(ManifestReport { path: path.to_path_buf(), ecosystem: check.ecosystem, results: check.results, workspace_root: check.workspace_root, + integrity, })) } Err(CheckError::UnsupportedEcosystem(eco)) => { @@ -334,8 +377,10 @@ fn progress_sink() -> Arc { /// `dependable check` pub async fn run_check(args: CheckArgs) -> anyhow::Result { - let cfg = load_config(&args.config); + let cfg = + load_config(&args.config).with_context(|| format!("reading {}", args.config.display()))?; let settings = resolve_check_settings(&args, &cfg); + check_fail_on_is_enforceable(settings.fail_on, settings.check_vuln)?; // Both policy steps run before discovery, so a misconfigured gate costs a // parse rather than a full network check. #[cfg(feature = "report")] @@ -384,6 +429,21 @@ pub async fn run_check(args: CheckArgs) -> anyhow::Result { // user explicitly asked for. #[cfg(feature = "report")] if let Some(policy) = &policy { + // The static check above proved the gate *could* be enforced; this one proves it + // *was*. A CVSS rule reads advisory lists, and a scan that never ran leaves those + // empty — indistinguishable from a project with no advisories, so the gate would + // pass vacuously on exactly the run that could not check it. + if policy.requires_cvss() + && reports + .iter() + .any(|r| r.integrity.vulnerability_scan_failed) + { + eprintln!( + "error: `[policy]` gates on advisory severity, but the vulnerability scan did not complete" + ); + eprintln!(" refusing to pass a policy that was never evaluated"); + return Ok(ExitCode::from(2)); + } let root = args.path.clone().unwrap_or_else(|| PathBuf::from(".")); let outcome = dependable_report::policy::evaluate(&build_report(root, &reports), policy); report_policy(&outcome); @@ -391,7 +451,7 @@ pub async fn run_check(args: CheckArgs) -> anyhow::Result { return Ok(ExitCode::from(1)); } } - Ok(exit_code(&reports, fail_on)) + Ok(exit_code(&reports, fail_on, args.quiet)) } /// Assemble the neutral report model the policy engine consumes from the CLI's @@ -460,7 +520,37 @@ fn check_policy_is_enforceable( "fail_on_severity" }; anyhow::bail!( - "`[policy] {key}` requires vulnerability scanning, which is disabled; drop `--no-vuln` (or re-enable `[vulnerability] enabled`), or remove the CVSS rule" + "`[policy] {key}` requires vulnerability scanning, which is disabled; drop \ + `--no-vuln` (or re-enable `[vulnerability] enabled`), or remove the CVSS rule" + ); + } + Ok(()) +} + +/// `--fail-on vulnerable` needs vulnerability scanning to mean anything. +/// +/// The whole content of that gate is the advisory lists, and with scanning off every one +/// of them is empty — indistinguishable from "nothing was found". No result is ever +/// `Vulnerable`, so the gate cannot fail, and the run exits 0 with nothing on stderr: +/// `[vulnerability] enabled = false` (or a stray `DEPENDABLE_NO_VULN`) silently disarms +/// the setting the shipped Action defaults to. This is +/// [`check_policy_is_enforceable`]'s rule applied to the other gate — a gate is either +/// enforceable or it is a configuration error — and the message keeps its shape. +/// +/// `--fail-on any` is deliberately *not* covered. Its promise is not the advisory list: +/// it fails on every status that is not up to date, so it is refutable by the run that +/// was actually performed, and refusing it would reject the ordinary offline +/// `--no-vuln --fail-on any` freshness check. It is the narrower gate — the one whose +/// entire subject matter is the scan — that goes vacuous. +/// +/// # Errors +/// +/// When `--fail-on vulnerable` is armed and vulnerability scanning is disabled. +fn check_fail_on_is_enforceable(fail_on: FailOn, check_vuln: bool) -> anyhow::Result<()> { + if fail_on == FailOn::Vulnerable && !check_vuln { + anyhow::bail!( + "`--fail-on vulnerable` requires vulnerability scanning, which is disabled; drop \ + `--no-vuln` (or re-enable `[vulnerability] enabled`), or gate on something else" ); } Ok(()) @@ -481,8 +571,9 @@ fn resolve_policy(config: &Path) -> anyhow::Result( fn warn_policy_ignored(config: &Path) { if crate::config::has_policy_table(config) { eprintln!( - "warning: {} declares `[policy]`, but this build has no `report` feature; the policy is not enforced", + "warning: {} declares `[policy]`, but this build has no `report` feature; the policy is not enforced", config.display() ); } @@ -585,7 +676,11 @@ pub async fn run_list(args: ListArgs) -> anyhow::Result { // warnings discovery emits are advice to *enable* something — and telling // someone to enable an ecosystem they switched off is noise `check`, `fix`, and // `report` already know not to make. - let cfg = load_config(&args.config); + // Fallible for the same reason as every other command: the file decides which + // ecosystems are on, so a config that cannot be read must not be silently + // replaced by defaults that enable all of them. + let cfg = + load_config(&args.config).with_context(|| format!("reading {}", args.config.display()))?; let manifests = collect_manifests( args.manifest.as_deref(), args.path.as_deref(), @@ -756,7 +851,8 @@ fn relative_to(root: &Path, manifest: &Path) -> PathBuf { /// Returns an error if the checker cannot be built or the terminal cannot be /// configured. pub async fn run_tui(args: TuiArgs) -> anyhow::Result { - let cfg = load_config(&args.config); + let cfg = + load_config(&args.config).with_context(|| format!("reading {}", args.config.display()))?; let settings = tui_settings(&cfg); // No progress bar: the UI draws its own screen. let engine = Engine::new(&settings, &cfg, false)?; @@ -823,15 +919,21 @@ pub fn run_tree(args: TreeArgs) -> anyhow::Result { /// `dependable fix` pub async fn run_fix(args: FixArgs) -> anyhow::Result { - let cfg = load_config(&args.config); + let cfg = + load_config(&args.config).with_context(|| format!("reading {}", args.config.display()))?; let settings = Settings { concurrency: args.concurrency.unwrap_or(cfg.global.concurrency).max(1), depth: args.depth, check_lockfile: cfg.global.lock_file, - check_vuln: false, + // A vulnerable-but-current dependency is exactly the one worth upgrading, and + // `fix.rs` has always had a `Vulnerable` arm — it was simply unreachable. + check_vuln: cfg.vulnerability.enabled && !args.no_vuln, licenses: false, - cache: true, - include_ghsa: false, + // `fix` writes to the user's manifests, so it must be able to refuse a cached + // answer. Without this it decided what to write from an hour-old cache with no + // way to bypass it. + cache: !args.no_cache, + include_ghsa: cfg.global.include_ghsa, fail_on: FailOn::None, unstable: cfg.global.unstable.into(), registry: cfg.rust.registry.clone(), @@ -850,22 +952,34 @@ pub async fn run_fix(args: FixArgs) -> anyhow::Result { } let engine = Engine::new(&settings, &cfg, true)?; - let mut total = 0; + + // Plan every manifest before writing any of them. Writing as it went left the tree + // half-rewritten when a later manifest failed — and because the report was printed + // *after* each write, the failing iteration also destroyed the record of what had + // already changed. + let mut planned = Vec::new(); for manifest in &manifests { let Some(report) = engine.check_manifest(manifest).await? else { continue; }; report_inherited_skips(manifest, &report); - let records = fix::apply_fixes(manifest, &report.results, args.all, args.dry_run)?; - if records.is_empty() { + planned.push(fix::plan(manifest, &report.results, args.all)?); + } + + let mut total = 0; + for plan in &planned { + if plan.records.is_empty() { continue; } + if !args.dry_run { + fix::commit(plan)?; + } println!( "{}{}", - manifest.display(), + plan.path.display(), if args.dry_run { " (dry run)" } else { "" } ); - for record in &records { + for record in &plan.records { println!(" {} {} → {}", record.name, record.from, record.to); total += 1; } @@ -1040,7 +1154,8 @@ fn load_template_overrides(root: &Path) -> anyhow::Result anyhow::Result { use std::io::Write; - let cfg = load_config(&args.config); + let cfg = + load_config(&args.config).with_context(|| format!("reading {}", args.config.display()))?; let settings = resolve_report_settings(&args, &cfg); let root = args.path.clone().unwrap_or_else(|| PathBuf::from(".")); @@ -1315,7 +1430,157 @@ fn expand_env(content: &str) -> String { out } -fn exit_code(reports: &[ManifestReport], fail_on: FailOn) -> ExitCode { +/// Whether a gate can be honoured from what this run actually established. +/// +/// `FailOn::None` gates on nothing, so nothing can be missing. Every other setting is a +/// promise not to pass a build with a particular property, and a run that failed to +/// *look* cannot keep it. +/// +/// The question is about the run, not about any one dependency. A registry that never +/// answered, or an advisory scan that did not complete, leaves the whole result set +/// unfounded — every dependency it covered is reported non-vulnerable because nothing +/// was asked. A registry that answered "no such package" left nothing unfounded: that +/// is a permanent, per-dependency fact about a private, internal, or deleted package, +/// visible in the table and in `--format json`, and gating the whole build on it turned +/// one unpublished internal package into a hard exit 2 for every repository that has +/// one — including every consumer of the shipped Action, which defaults to +/// `--fail-on vulnerable`. +/// +/// The carve-out is for that answer alone. A dependency this run failed to evaluate by +/// itself — a value nobody could read as a version requirement — reached no registry, so +/// there is no fact standing in for its status and the gate is as unanswerable as it +/// ever was. Exempting those too let `{"lodash": "^^^bogus"}` pass `--fail-on vulnerable` +/// under a note blaming a registry that was never asked. +/// +/// A constraint this crate simply has no front-end for is not one of those. It is +/// [`DependencyStatus::Undetermined`], it carries no [`ErrorOrigin`], and it is counted +/// nowhere here: an ordinary `"react": "^15 || ^16"` or a Dart `any` is the manifest +/// being right and this tool being incomplete, and failing a build over that punishes +/// the user for our gap. [`note_undetermined`] says so on stderr, and `--fail-on any` +/// still fails on it. +fn gate_is_answerable(reports: &[ManifestReport], fail_on: FailOn) -> Result<(), String> { + if fail_on == FailOn::None { + return Ok(()); + } + let mut reasons: Vec = Vec::new(); + if reports + .iter() + .any(|r| r.integrity.vulnerability_scan_failed) + { + reasons.push("the vulnerability scan did not complete".to_owned()); + } + // `FailOn::Any` fails on `DependencyStatus::Error`, and both an unanswering registry + // and an unreadable constraint produce exactly that — so there the promise is kept + // rather than missed, and the run exits 1 on the errors themselves. The other + // settings match specific statuses and skip errors entirely, which is where a run + // that established nothing could still be reported as clean. + if fail_on != FailOn::Any { + if reports.iter().any(|r| r.integrity.registry_unreachable) { + reasons.push("the registry did not answer".to_owned()); + } + let unevaluated: usize = reports.iter().map(|r| r.integrity.unevaluated).sum(); + if unevaluated > 0 { + reasons.push(format!( + "{unevaluated} dependenc{} could not be evaluated", + if unevaluated == 1 { "y" } else { "ies" } + )); + } + } + if reasons.is_empty() { + return Ok(()); + } + Err(join_reasons(&reasons)) +} + +/// `a`, `a and b`, `a, b and c` — the gate's reasons read as a sentence. +fn join_reasons(reasons: &[String]) -> String { + match reasons { + [] => String::new(), + [only] => only.clone(), + [rest @ .., last] => format!("{} and {last}", rest.join(", ")), + } +} + +/// Say on stderr how many dependencies the registry reported as non-existent. +/// +/// Such a dependency has no status to gate on and the gate no longer stops for it, so +/// the run says plainly that it was not covered — otherwise a passing +/// `--fail-on vulnerable` reads as "every dependency here is clean" when one of them was +/// never checked. +/// +/// Silent for `FailOn::None` (nothing was gated on) and for `FailOn::Any` (which fails +/// on these results, so they *were* gated on), and silent under `--quiet`, whose help +/// says "Only print errors" — a note about what was skipped is not one. +/// +/// Counted from the registry's own answer ([`ScanIntegrity::unresolved`]), never from +/// every `Error`: an unreadable constraint reached no registry, and saying it was "not +/// found in its registry" reported a cause that never happened. +fn note_unresolved(reports: &[ManifestReport], fail_on: FailOn, quiet: bool) { + if quiet || matches!(fail_on, FailOn::None | FailOn::Any) { + return; + } + let unresolved: usize = reports.iter().map(|r| r.integrity.unresolved).sum(); + if unresolved == 0 { + return; + } + eprintln!( + "note: {unresolved} dependenc{} not found in {} registry, so {} not gated on", + if unresolved == 1 { "y was" } else { "ies were" }, + if unresolved == 1 { "its" } else { "their" }, + if unresolved == 1 { "it is" } else { "they are" }, + ); +} + +/// Say on stderr how many dependencies this run could not read a version out of. +/// +/// `Undetermined` is a real package whose declared constraint this run could not +/// translate, and the status was made honest without saying so anywhere: it trips no +/// `--fail-on outdated` gate, produces no SARIF result, and left a passing run reading +/// as "everything here is current" when a dependency had never been evaluated. +/// +/// Mirrors [`note_unresolved`] exactly — same silences, same shape. Deliberately *not* a +/// gate: adding `Undetermined` to `--fail-on outdated` would change what that setting +/// promises, and `--fail-on any` already fails on it. +fn note_undetermined(reports: &[ManifestReport], fail_on: FailOn, quiet: bool) { + if quiet || matches!(fail_on, FailOn::None | FailOn::Any) { + return; + } + let undetermined = reports + .iter() + .flat_map(|report| &report.results) + .filter(|result| matches!(result.status, DependencyStatus::Undetermined)) + .count(); + if undetermined == 0 { + return; + } + eprintln!( + "note: {undetermined} dependenc{} a declared version this run could not read, so {} not \ + gated on", + if undetermined == 1 { + "y has" + } else { + "ies have" + }, + if undetermined == 1 { + "it is" + } else { + "they are" + }, + ); +} + +fn exit_code(reports: &[ManifestReport], fail_on: FailOn, quiet: bool) -> ExitCode { + // A gate whose inputs are missing must fail, not pass. `--fail-on vulnerable` with an + // unreachable OSV used to exit 0 while printing the errors that explain why it could + // not know — a green build that had never been checked, which is the one outcome a + // gate exists to prevent. + if let Err(reason) = gate_is_answerable(reports, fail_on) { + eprintln!("error: cannot honour --fail-on: {reason}"); + eprintln!(" refusing to report a clean run that was never completed"); + return ExitCode::from(2); + } + note_unresolved(reports, fail_on, quiet); + note_undetermined(reports, fail_on, quiet); let triggered = reports .iter() .flat_map(|report| &report.results) @@ -1344,6 +1609,44 @@ fn exit_code(reports: &[ManifestReport], fail_on: FailOn) -> ExitCode { mod tests { use super::*; + /// A `DEPENDABLE_*` switch is read by its value. Presence alone let + /// `DEPENDABLE_NO_VULN: ${{ inputs.no-vuln }}` — an expression that expands to the + /// empty string when the input is unset — turn vulnerability scanning off for every + /// run of the shipped Action, with nothing said about it anywhere. + #[test] + fn a_kill_switch_can_be_spelled_off() { + for raw in ["1", "true", "yes", "on", "anything"] { + assert!(flag_is_on(Some(raw)), "{raw}"); + } + for raw in ["", " ", "0", "false", "FALSE", "no", "off", " off "] { + assert!(!flag_is_on(Some(raw)), "{raw:?}"); + } + assert!(!flag_is_on(None)); + } + + /// `--fail-on vulnerable` with scanning off cannot fail: every advisory list is + /// empty, nothing is ever `Vulnerable`, and the run exits 0 having checked nothing. + /// That is the shipped Action's default gate, disarmed by a config key. + #[test] + fn a_vulnerability_gate_with_no_scan_is_a_configuration_error() { + let err = check_fail_on_is_enforceable(FailOn::Vulnerable, false) + .expect_err("an unenforceable gate must not be accepted"); + assert!( + format!("{err}").contains("requires vulnerability scanning"), + "{err}" + ); + // With the scan on it is an ordinary gate. + assert!(check_fail_on_is_enforceable(FailOn::Vulnerable, true).is_ok()); + // The other settings promise things the run can still establish without a scan, + // so an offline freshness check keeps working. + for fail_on in [FailOn::None, FailOn::Outdated, FailOn::Any] { + assert!( + check_fail_on_is_enforceable(fail_on, false).is_ok(), + "{fail_on:?}" + ); + } + } + #[test] fn cargo_home_prefers_explicit_env_then_dot_cargo() { // An explicit `$CARGO_HOME` is used verbatim. @@ -1448,4 +1751,274 @@ mod tests { // An unterminated `${` is emitted verbatim. assert_eq!(expand_env("a=${OPEN"), "a=${OPEN"); } + + fn fixture_item() -> dependable_fetch::Item { + dependable_fetch::core::parse( + dependable_fetch::ManifestKind::CargoToml, + "[dependencies]\nserde = \"1\"\n", + ) + .expect("fixture manifest") + .items + .into_iter() + .next() + .expect("one dependency") + } + + /// A result the **registry** produced by name: no such package. Built through + /// [`CheckResult::not_found`] rather than by hand, because the provenance — not the + /// wording of the message — is what the gate reads. + fn not_found_result() -> dependable_fetch::CheckResult { + dependable_fetch::CheckResult::errored( + fixture_item(), + "package `@acme/internal` not found", + ErrorOrigin::NotFound, + ) + } + + fn report_of( + integrity: ScanIntegrity, + results: Vec, + ) -> ManifestReport { + ManifestReport { + path: PathBuf::from("Cargo.toml"), + ecosystem: dependable_fetch::Ecosystem::Rust, + results, + workspace_root: None, + integrity, + } + } + + fn report_with(integrity: ScanIntegrity, statuses: &[DependencyStatus]) -> ManifestReport { + report_of( + integrity, + statuses + .iter() + .map(|s| dependable_fetch::CheckResult::new(fixture_item(), s.clone())) + .collect(), + ) + } + + /// The defect this exists to prevent: OSV unreachable, `--fail-on vulnerable` armed, + /// every result left non-vulnerable because nothing was ever asked — and the run + /// exiting 0, certifying a build it had not checked. + #[test] + fn a_failed_scan_cannot_pass_a_vulnerability_gate() { + let reports = vec![report_with( + ScanIntegrity { + vulnerability_scan_failed: true, + registry_unreachable: false, + unresolved: 0, + unevaluated: 0, + }, + &[DependencyStatus::UpToDate], + )]; + assert!(gate_is_answerable(&reports, FailOn::Vulnerable).is_err()); + assert!(gate_is_answerable(&reports, FailOn::Outdated).is_err()); + assert!(gate_is_answerable(&reports, FailOn::Any).is_err()); + // Nothing was gated on, so nothing can be missing. + assert!(gate_is_answerable(&reports, FailOn::None).is_ok()); + } + + /// A registry that never answered leaves every dependency it covered unfounded, so + /// the gate still cannot be honoured — the half of the guard that has to survive the + /// narrowing below. + #[test] + fn an_unreachable_registry_cannot_pass_a_status_gate() { + let reports = vec![report_with( + ScanIntegrity { + vulnerability_scan_failed: false, + registry_unreachable: true, + unresolved: 0, + unevaluated: 0, + }, + &[DependencyStatus::Error("registry unreachable".to_owned())], + )]; + assert!(gate_is_answerable(&reports, FailOn::Vulnerable).is_err()); + assert!(gate_is_answerable(&reports, FailOn::Outdated).is_err()); + assert_eq!( + exit_code(&reports, FailOn::Vulnerable, false), + ExitCode::from(2) + ); + // `Any` fails on the `Error` statuses an unanswering registry produces, so its + // promise is kept — that is the gate working, not a hole. + assert!(gate_is_answerable(&reports, FailOn::Any).is_ok()); + assert_eq!(exit_code(&reports, FailOn::Any, false), ExitCode::from(1)); + // Nothing was gated on, so nothing can be missing. + assert!(gate_is_answerable(&reports, FailOn::None).is_ok()); + } + + /// A registry that answered "no such package" answered. A private or internal + /// package, one served by a registry this run does not route to, or a deleted one is + /// a permanent per-dependency fact: it is reported, and it does not turn every + /// `--fail-on` setting into exit 2 for the dependencies that *did* resolve. + /// + /// This corrects an assertion that pinned the opposite. `--fail-on vulnerable` is + /// the shipped Action's default, so under the old rule every repository containing + /// one unpublished internal package went from a passing step to a hard failure, with + /// no escape short of dropping the gate. + #[test] + fn a_package_the_registry_says_does_not_exist_does_not_break_the_gate() { + let not_found = not_found_result(); + assert_eq!( + not_found.error_origin, + ErrorOrigin::NotFound, + "the carve-out has to be reached through the provenance, not through the message" + ); + let reports = vec![report_of( + ScanIntegrity { + vulnerability_scan_failed: false, + registry_unreachable: false, + unresolved: 1, + unevaluated: 0, + }, + vec![ + dependable_fetch::CheckResult::new(fixture_item(), DependencyStatus::UpToDate), + not_found, + ], + )]; + assert!(gate_is_answerable(&reports, FailOn::Vulnerable).is_ok()); + assert!(gate_is_answerable(&reports, FailOn::Outdated).is_ok()); + assert!(gate_is_answerable(&reports, FailOn::Any).is_ok()); + assert_eq!( + exit_code(&reports, FailOn::Vulnerable, false), + ExitCode::SUCCESS + ); + assert_eq!( + exit_code(&reports, FailOn::Outdated, false), + ExitCode::SUCCESS + ); + // `Any` still fails on the error itself — that is the gate working, and it is + // the setting that asks to hear about anything less than a clean answer. + assert_eq!(exit_code(&reports, FailOn::Any, false), ExitCode::from(1)); + } + + /// The other half of the same distinction, and the regression the carve-out + /// introduced: an unparseable constraint never reaches a registry, so nothing was + /// established about the dependency at all. Exempting it alongside the 404s let + /// `{"lodash": "^^^bogus"}` pass `--fail-on vulnerable` under a note saying the + /// registry had not found it — a gate certifying a build it had not evaluated. + #[test] + fn a_dependency_this_run_could_not_evaluate_still_breaks_the_gate() { + let error = dependable_fetch::CheckResult::new( + fixture_item(), + DependencyStatus::Error("unparseable constraint: unexpected character '^'".to_owned()), + ); + assert_eq!( + error.error_origin, + ErrorOrigin::Local, + "no registry was ever asked about this dependency" + ); + let reports = vec![report_of( + ScanIntegrity { + vulnerability_scan_failed: false, + registry_unreachable: false, + unresolved: 0, + unevaluated: 1, + }, + vec![ + dependable_fetch::CheckResult::new(fixture_item(), DependencyStatus::UpToDate), + error, + ], + )]; + assert_eq!( + gate_is_answerable(&reports, FailOn::Vulnerable).unwrap_err(), + "1 dependency could not be evaluated" + ); + assert!(gate_is_answerable(&reports, FailOn::Outdated).is_err()); + assert_eq!( + exit_code(&reports, FailOn::Vulnerable, false), + ExitCode::from(2) + ); + // `Any` fails on the `Error` itself, so its promise is kept. + assert!(gate_is_answerable(&reports, FailOn::Any).is_ok()); + assert_eq!(exit_code(&reports, FailOn::Any, false), ExitCode::from(1)); + // Nothing was gated on, so nothing can be missing. + assert!(gate_is_answerable(&reports, FailOn::None).is_ok()); + } + + /// Every unanswerable reason at once, read as one sentence rather than as a + /// hand-written combination per pair. + #[test] + fn the_gate_names_every_reason_it_could_not_be_honoured() { + let reports = vec![report_of( + ScanIntegrity { + vulnerability_scan_failed: true, + registry_unreachable: true, + unresolved: 0, + unevaluated: 2, + }, + vec![], + )]; + assert_eq!( + gate_is_answerable(&reports, FailOn::Vulnerable).unwrap_err(), + "the vulnerability scan did not complete, the registry did not answer and 2 \ + dependencies could not be evaluated" + ); + } + + /// A complete run still gates on what it found, and still passes when it finds + /// nothing — the guard must not turn every check into a failure. + #[test] + fn a_complete_run_gates_on_its_findings_as_before() { + let clean = vec![report_with( + ScanIntegrity::default(), + &[DependencyStatus::UpToDate], + )]; + assert!(gate_is_answerable(&clean, FailOn::Vulnerable).is_ok()); + assert_eq!( + exit_code(&clean, FailOn::Vulnerable, false), + ExitCode::SUCCESS + ); + + let vulnerable = vec![report_with( + ScanIntegrity::default(), + &[DependencyStatus::Vulnerable], + )]; + assert_eq!( + exit_code(&vulnerable, FailOn::Vulnerable, false), + ExitCode::from(1) + ); + } + + /// Parse a real command line, so the test exercises the same `Option` clap produces + /// rather than a hand-built struct that could disagree with it. + fn check_args(argv: &[&str]) -> crate::cli::CheckArgs { + use clap::Parser as _; + let cli = crate::cli::Cli::try_parse_from(argv).expect("a valid command line"); + match cli.command { + Some(crate::cli::Command::Check(args)) => args, + _ => panic!("expected the check subcommand"), + } + } + + /// Documented precedence is CLI over env over config. `fail_on` inverted it: the + /// guard compared against `FailOn::None`, which is also clap's default, so an + /// explicit `--fail-on none` was indistinguishable from the flag being absent and a + /// config that armed the gate could not be disarmed from the command line. + #[test] + fn an_explicit_fail_on_none_beats_the_config() { + let mut cfg = Config::default(); + cfg.global.fail_on = FailOn::Any; + + let explicit = resolve_check_settings( + &check_args(&["dependable", "check", "--fail-on", "none"]), + &cfg, + ); + assert_eq!( + explicit.fail_on, + FailOn::None, + "the command line was ignored" + ); + + // Absent, the config still governs. + let absent = resolve_check_settings(&check_args(&["dependable", "check"]), &cfg); + assert_eq!(absent.fail_on, FailOn::Any); + + // And a non-default flag still wins, as it always did. + let vulnerable = resolve_check_settings( + &check_args(&["dependable", "check", "--fail-on", "vulnerable"]), + &cfg, + ); + assert_eq!(vulnerable.fail_on, FailOn::Vulnerable); + } } diff --git a/crates/dependable/tests/cli_fix.rs b/crates/dependable/tests/cli_fix.rs new file mode 100644 index 0000000..5cdc7d8 --- /dev/null +++ b/crates/dependable/tests/cli_fix.rs @@ -0,0 +1,149 @@ +//! End-to-end coverage for `dependable fix` — the only command that writes to the +//! user's files, and the one that had no test asserting the bytes it produces. +//! +//! Hermetic: every fixture declares path dependencies only, so no registry request is +//! made. That is enough to exercise the write path, which is what these cover. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn workdir(name: &str) -> PathBuf { + let dir = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(name); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).expect("create the scratch directory"); + dir +} + +fn run(dir: &Path, args: &[&str]) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_dependable")); + command.arg("fix").arg(dir).args(args); + command.env_remove("DEPENDABLE_FAIL_ON"); + command.output().expect("run dependable fix") +} + +/// A manifest whose only dependencies are local paths: nothing to fetch, nothing to +/// rewrite, so `fix` must leave the file byte-identical. +const LOCAL_ONLY: &str = "[package]\nname = \"app\"\nversion = \"0.1.0\"\n\n[dependencies]\nhelper = { path = \"../helper\" }\n"; + +#[test] +fn a_run_with_nothing_to_change_leaves_the_manifest_byte_identical() { + let dir = workdir("fix_no_change"); + let manifest = dir.join("Cargo.toml"); + fs::write(&manifest, LOCAL_ONLY).unwrap(); + + let output = run(&dir, &[]); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_eq!( + fs::read_to_string(&manifest).unwrap(), + LOCAL_ONLY, + "fix rewrote a manifest it had nothing to change" + ); +} + +/// Comments, ordering, and formatting are not `fix`'s to touch; it replaces one span. +#[test] +fn formatting_and_comments_survive_a_run() { + let dir = workdir("fix_formatting"); + let manifest = dir.join("Cargo.toml"); + let original = "# a leading comment\n[package]\nname = \"app\"\nversion = \"0.1.0\"\n\n[dependencies]\n# why this dep exists\nhelper = { path = \"../helper\" } # trailing\n"; + fs::write(&manifest, original).unwrap(); + + let output = run(&dir, &[]); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_eq!(fs::read_to_string(&manifest).unwrap(), original); +} + +/// `--dry-run` must not write. This is the flag users reach for before trusting the +/// command, so it is the one that must never be wrong. +#[test] +fn a_dry_run_writes_nothing() { + let dir = workdir("fix_dry_run"); + let manifest = dir.join("Cargo.toml"); + fs::write(&manifest, LOCAL_ONLY).unwrap(); + let before = fs::metadata(&manifest).unwrap().modified().unwrap(); + + let output = run(&dir, &["--dry-run"]); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + + assert_eq!(fs::read_to_string(&manifest).unwrap(), LOCAL_ONLY); + assert_eq!(fs::metadata(&manifest).unwrap().modified().unwrap(), before); +} + +/// A manifest that cannot be parsed must not be rewritten, and must not abort the run +/// with a half-written tree behind it. +#[test] +fn an_unparseable_manifest_is_left_alone() { + let dir = workdir("fix_unparseable"); + let manifest = dir.join("Cargo.toml"); + let broken = "[package\nname = \"app\"\n"; + fs::write(&manifest, broken).unwrap(); + + let _ = run(&dir, &[]); + assert_eq!( + fs::read_to_string(&manifest).unwrap(), + broken, + "a manifest that could not be parsed was written to anyway" + ); +} + +/// The write goes through a temporary file in the manifest's own directory and is +/// renamed into place. Nothing may be left behind on success. +#[test] +fn no_temporary_files_are_left_beside_the_manifest() { + let dir = workdir("fix_no_temp_files"); + fs::write(dir.join("Cargo.toml"), LOCAL_ONLY).unwrap(); + + let output = run(&dir, &[]); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + + let entries: Vec = fs::read_dir(&dir) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + assert_eq!( + entries, + vec!["Cargo.toml".to_string()], + "stray files: {entries:?}" + ); +} + +/// A read-only manifest must fail loudly rather than truncating it. `fs::write` opens +/// with `O_TRUNC`, so the pre-atomic path destroyed the file before discovering it +/// could not write. +#[cfg(unix)] +#[test] +fn a_read_only_manifest_is_not_destroyed() { + use std::os::unix::fs::PermissionsExt as _; + + let dir = workdir("fix_read_only"); + let manifest = dir.join("Cargo.toml"); + fs::write(&manifest, LOCAL_ONLY).unwrap(); + let mut permissions = fs::metadata(&manifest).unwrap().permissions(); + permissions.set_mode(0o444); + fs::set_permissions(&manifest, permissions).unwrap(); + + let _ = run(&dir, &[]); + + assert_eq!( + fs::read_to_string(&manifest).unwrap(), + LOCAL_ONLY, + "a read-only manifest was truncated" + ); +} diff --git a/crates/dependable/tests/cli_gate.rs b/crates/dependable/tests/cli_gate.rs new file mode 100644 index 0000000..afa13cd --- /dev/null +++ b/crates/dependable/tests/cli_gate.rs @@ -0,0 +1,670 @@ +//! End-to-end coverage for the `--fail-on` gate, driven against a throwaway HTTP +//! registry on loopback. +//! +//! Every defect these falsify shipped past a review because the suite had no way to make +//! a registry *answer*: `cli_policy.rs` and `cli_sarif.rs` stay hermetic by declaring +//! path dependencies, so no fetch is ever built and no status code is ever seen. A gate +//! that turns on the difference between "the registry said no such package", "the +//! registry did not answer", and "this run could not read the constraint" cannot be +//! tested that way at all. +//! +//! So these run the real binary against a single-shot server on `127.0.0.1:0`, in the +//! shape `cli_fix.rs` established: hermetic, no dev-dependency, and the real fetch path +//! rather than a stub of it. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn workdir(name: &str) -> PathBuf { + let dir = PathBuf::from(env!("CARGO_TARGET_TMPDIR")).join(name); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).expect("create the scratch directory"); + dir +} + +/// One canned response: status code, `Content-Type`, body. +type Response = (u16, &'static str, String); + +fn json(body: impl Into) -> Response { + (200, "application/json", body.into()) +} + +fn text(body: impl Into) -> Response { + (200, "text/plain", body.into()) +} + +fn xml(body: impl Into) -> Response { + (200, "application/xml", body.into()) +} + +/// A status with no body — the shape a proxy's `404`/`410` takes. +fn status(code: u16) -> Response { + (code, "text/plain", String::new()) +} + +/// A single-shot registry: a path-to-response table served on loopback. +/// +/// Deliberately minimal rather than a mock-server crate — `dependable` has no +/// dev-dependencies at all. Unlike the fixture in `cli_fix.rs` this one carries the +/// status code and content type per route, because the defects here are *about* status +/// codes and about documents that are not JSON. Every response closes the connection, so +/// no keep-alive state has to be modelled. An unrouted path is a plain `404`. +fn registry(routes: Vec<(String, Response)>) -> String { + use std::io::{BufRead as _, BufReader, Write as _}; + use std::net::TcpListener; + + let mut routes = routes; + routes.push((OSV_BATCH_PATH.to_string(), clean_advisory_batch())); + let listener = TcpListener::bind("127.0.0.1:0").expect("bind a loopback port"); + let addr = listener.local_addr().expect("read the bound port"); + std::thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { continue }; + let routes = routes.clone(); + std::thread::spawn(move || { + let mut reader = BufReader::new(stream.try_clone().expect("clone the socket")); + let mut request = String::new(); + if reader.read_line(&mut request).is_err() { + return; + } + // Drain the headers so the client is never left writing into a socket + // nobody is reading, which some stacks report as a reset rather than as + // the response we are about to send. + let mut line = String::new(); + while reader.read_line(&mut line).is_ok_and(|n| n > 2) { + line.clear(); + } + let path = request.split_whitespace().nth(1).unwrap_or("").to_string(); + let (code, content_type, body) = routes + .iter() + .find(|(route, _)| *route == path) + .map(|(_, response)| response.clone()) + .unwrap_or_else(|| status(404)); + let reason = match code { + 200 => "OK", + 404 => "Not Found", + 410 => "Gone", + _ => "Status", + }; + let response = format!( + "HTTP/1.1 {code} {reason}\r\nContent-Type: {content_type}\r\nContent-Length: \ + {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + let _ = stream.write_all(response.as_bytes()); + let _ = stream.flush(); + }); + } + }); + format!("http://{addr}") +} + +/// Where the fixture serves OSV's `querybatch`. +const OSV_BATCH_PATH: &str = "/v1/querybatch"; + +/// A `querybatch` answer in which no version is affected. +/// +/// The scan has to actually *run* here: `--fail-on vulnerable` is unenforceable with +/// scanning off, so a fixture that turned it off could not exercise the gate at all — +/// and turning it off is precisely the configuration error the guard now rejects. +/// +/// `querybatch` promises one result per query and the client rejects a short body as a +/// truncated answer rather than a clean bill, so the fixture serves more empty results +/// than any manifest here has dependencies; the extras are ignored. +fn clean_advisory_batch() -> Response { + json(format!("{{\"results\":[{}]}}", ["{}"; 64].join(","))) +} + +/// An npm abbreviated packument: the version keys and the `latest` dist-tag are all the +/// version checker reads. +fn packument(name: &str, versions: &[&str], latest: &str) -> Response { + let entries: Vec = versions + .iter() + .map(|v| format!("\"{v}\":{{\"name\":\"{name}\",\"version\":\"{v}\"}}")) + .collect(); + json(format!( + "{{\"name\":\"{name}\",\"dist-tags\":{{\"latest\":\"{latest}\"}},\"versions\":{{{}}}}}", + entries.join(",") + )) +} + +/// Point every ecosystem fetcher at `base` and switch OSV off, so a run touches nothing +/// but the loopback registry. +fn write_config(dir: &Path, base: &str) -> PathBuf { + let config = dir.join(".dependable.toml"); + fs::write( + &config, + format!( + "[npm]\nregistry = \"{base}\"\n\n[python]\nregistry = \"{base}/pypi\"\n\n\ + [go]\nregistry = \"{base}\"\n\n[jvm]\nregistry = \"{base}\"\n\n\ + [dart]\nregistry = \"{base}\"\n\n\ + [vulnerability]\nenabled = true\nosv_batch_url = \"{base}{OSV_BATCH_PATH}\"\n" + ), + ) + .unwrap(); + config +} + +fn check(dir: &Path, config: &Path, args: &[&str]) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_dependable")); + command + .arg("check") + .arg(dir) + .arg("--config") + .arg(config) + .arg("--no-cache") + .args(args); + command.env_remove("DEPENDABLE_FAIL_ON"); + command.env_remove("DEPENDABLE_NO_VULN"); + // A user `.npmrc` would override the configured registry and send the run at the + // real npm. + command.env("HOME", dir); + command.current_dir(dir); + command.output().expect("run dependable check") +} + +/// `(stdout, stderr, exit code)`. +fn outcome(output: &Output) -> (String, String, i32) { + ( + String::from_utf8_lossy(&output.stdout).into_owned(), + String::from_utf8_lossy(&output.stderr).into_owned(), + output.status.code().unwrap_or(-1), + ) +} + +// --------------------------------------------------------------------------- +// A registry that answers "no such module" with `410 Gone` +// --------------------------------------------------------------------------- + +/// The Go module proxy protocol names **both** `404` and `410` as the not-found +/// responses, and `410 Gone` is `proxy.golang.org`'s answer for a module it will not +/// serve — the private path the 404 carve-out exists for. Reading it as a transport +/// failure marked the whole registry unreachable, so one private module took a Go +/// repository from a passing `--fail-on vulnerable` (the shipped Action's default) to +/// exit 2. +#[test] +fn a_go_module_the_proxy_answers_410_for_does_not_break_the_gate() { + let dir = workdir("gate_go_gone"); + let base = registry(vec![ + ("/github.com/acme/private/@v/list".to_string(), status(410)), + ("/github.com/acme/private/@latest".to_string(), status(410)), + ( + "/github.com/stretchr/testify/@v/list".to_string(), + text("v1.8.0\nv1.9.0\n"), + ), + ]); + let config = write_config(&dir, &base); + fs::write( + dir.join("go.mod"), + "module example.com/app\n\ngo 1.22\n\nrequire (\n\tgithub.com/acme/private v0.1.0\n\t\ + github.com/stretchr/testify v1.8.0\n)\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stdout.contains("package `github.com/acme/private` not found"), + "a 410 was not read as an absent module:\n{stdout}" + ); + assert!( + !stderr.contains("the registry did not answer"), + "one private module marked the whole registry unreachable:\n{stderr}" + ); + assert!( + stderr.contains("note: 1 dependency was not found in its registry"), + "stderr: {stderr}" + ); + // The module that *did* resolve is still evaluated. + assert!(stdout.contains("update available"), "stdout: {stdout}"); +} + +// --------------------------------------------------------------------------- +// The 404 carve-out covers a 404, and nothing else +// --------------------------------------------------------------------------- + +/// The regression the carve-out introduced. An unparseable constraint never reaches a +/// registry, so nothing is established about the dependency at all — but it produced the +/// same `DependencyStatus::Error` as a 404 and was exempted with them. Two dependencies +/// were left unevaluated, the note blamed a registry that was never asked, and +/// `--fail-on vulnerable` certified the build. +#[test] +fn an_unreadable_constraint_still_refuses_to_certify_the_build() { + let dir = workdir("gate_unreadable_constraint"); + let base = registry(vec![ + ( + "/lodash".to_string(), + packument("lodash", &["4.17.20", "4.17.21"], "4.17.21"), + ), + ( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + ), + ]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"lodash\":\"^^^bogus\",\"express\":\"^4.19.0\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 2, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stderr.contains("error: cannot honour --fail-on: 1 dependency could not be evaluated"), + "stderr: {stderr}" + ); + assert!( + !stderr.contains("not found in its registry"), + "the note blamed a registry that was never asked:\n{stderr}" + ); +} + +/// The other half, which must survive the repair: a registry that answers `404` answered. +/// A private or internal package is a permanent per-dependency fact, reported and not +/// gated on, and it must not turn `--fail-on vulnerable` into exit 2 for the +/// dependencies that did resolve. +#[test] +fn a_package_the_registry_answers_404_for_does_not_break_the_gate() { + let dir = workdir("gate_404_carve_out"); + let base = registry(vec![( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"@acme/internal\":\"^1.0.0\",\"express\":\ + \"^4.19.0\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stderr.contains("note: 1 dependency was not found in its registry, so it is not gated on"), + "stderr: {stderr}" + ); + // `--quiet` says "Only print errors"; a note about what was skipped is not one. + let quiet = check(&dir, &config, &["--fail-on", "vulnerable", "-q"]); + let (_, quiet_stderr, quiet_code) = outcome(&quiet); + assert_eq!(quiet_code, 0); + assert!( + !quiet_stderr.contains("not found in its registry"), + "`-q` still printed the note: {quiet_stderr}" + ); +} + +// --------------------------------------------------------------------------- +// A `>` inside an override key's range +// --------------------------------------------------------------------------- + +/// pnpm and Yarn both allow a range in an override key, and a range contains `>`. +/// Splitting on every `>` cut each key inside its own range, so the run asked the +/// registry for `=1.0.0`, `=1`, `1.0.0` and `b` — and, with a 404 no longer failing the +/// gate, said nothing about it. +#[test] +fn an_override_key_carrying_a_range_is_checked_as_its_own_package() { + let dir = workdir("gate_override_range"); + let base = registry(vec![ + ( + "/lodash".to_string(), + packument("lodash", &["4.17.21"], "4.17.21"), + ), + ("/bar".to_string(), packument("bar", &["2.0.0"], "2.0.0")), + ("/foo".to_string(), packument("foo", &["1.0.0"], "1.0.0")), + ("/a".to_string(), packument("a", &["1.0.0"], "1.0.0")), + ]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"overrides\":{\"lodash@>=1.0.0\":\"4.17.21\",\"foo>bar@>=1\":\ + \"2.0.0\",\"foo@>1.0.0\":\"1.0.0\",\"a@>b\":\"1.0.0\",\"quux@1>bar@^2.1.0\":\"2.0.0\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + for fragment in ["=1.0.0", "not found"] { + assert!( + !stdout.contains(fragment), + "a range fragment was read as a package name:\n{stdout}" + ); + } + assert!( + stdout.contains("lodash") && stdout.contains("bar") && stdout.contains("foo"), + "stdout: {stdout}" + ); + assert!(stdout.contains("5 up to date"), "stdout: {stdout}"); +} + +// --------------------------------------------------------------------------- +// Poetry's `"*"` +// --------------------------------------------------------------------------- + +/// `*` is PEP 440's and Poetry's explicit "any version", and the most common way to +/// write an unpinned dependency. It translated to the empty string, which the +/// failed-translation heuristic then read as a constraint nobody could parse — so every +/// Poetry project with an unpinned dependency was excluded from `--fail-on outdated` and +/// failed `--fail-on any`. +#[test] +fn a_poetry_wildcard_resolves_instead_of_going_undetermined() { + let dir = workdir("gate_poetry_wildcard"); + let base = registry(vec![( + "/pypi/requests/json".to_string(), + json("{\"releases\":{\"2.31.0\":[],\"2.32.3\":[]}}"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("pyproject.toml"), + "[tool.poetry.dependencies]\nrequests = \"*\"\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "any"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stdout.contains("up to date") && !stdout.contains("undetermined"), + "stdout: {stdout}" + ); +} + +// --------------------------------------------------------------------------- +// `Undetermined` says so +// --------------------------------------------------------------------------- + +/// `Undetermined` was gated on by nothing and noted by nothing, so a run that could not +/// read two constraints printed a clean `--fail-on outdated` and said nothing at all +/// about them. The status stays out of the gate — `--fail-on any` already fails on it — +/// but the run now says what it could not evaluate. +#[test] +fn a_dependency_whose_version_could_not_be_read_is_noted() { + let dir = workdir("gate_undetermined_note"); + let base = registry(vec![( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"express\":\"^4.19.0\"},\"overrides\":\ + {\"lodash\":\"$nope\",\"minimist\":\"$\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "outdated"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + // `"$"` names nothing, and used to reach the checker as a literal constraint and + // hard-fail on the `$`. + assert!( + !stdout.contains("unparseable constraint"), + "a dangling reference was read as a constraint:\n{stdout}" + ); + assert!(stdout.contains("2 undetermined"), "stdout: {stdout}"); + assert!( + stderr.contains( + "note: 2 dependencies have a declared version this run could not read, so they are \ + not gated on" + ), + "stderr: {stderr}" + ); + + let quiet = check(&dir, &config, &["--fail-on", "outdated", "-q"]); + let (_, quiet_stderr, quiet_code) = outcome(&quiet); + assert_eq!(quiet_code, 0); + assert!( + !quiet_stderr.contains("could not read"), + "`-q` still printed the note: {quiet_stderr}" + ); +} + +// --------------------------------------------------------------------------- +// A 200 that lists no versions +// --------------------------------------------------------------------------- + +/// A `maven-metadata.xml` that parses but names no version is not an authoritative "this +/// artifact does not exist" — a Nexus or Artifactory group repository whose upstream +/// proxy is down serves exactly such a locally-merged document. Reporting it as a 404 +/// exempted it from the gate, certifying a build against a dependency nothing was ever +/// known about. +#[test] +fn a_metadata_document_listing_no_versions_is_not_exempt_from_the_gate() { + let dir = workdir("gate_empty_metadata"); + let base = registry(vec![( + "/com/acme/thing/maven-metadata.xml".to_string(), + xml( + "com.acmething\ + ", + ), + )]); + let config = write_config(&dir, &base); + fs::create_dir_all(dir.join("gradle")).unwrap(); + fs::write( + dir.join("gradle/libs.versions.toml"), + "[libraries]\nthing = { module = \"com.acme:thing\", version = \"1.0.0\" }\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 2, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stderr.contains("error: cannot honour --fail-on: the registry did not answer"), + "stderr: {stderr}" + ); + assert!( + !stderr.contains("not found in its registry"), + "an answered-but-empty document was reported as a 404:\n{stderr}" + ); +} + +// --------------------------------------------------------------------------- +// The ranges npm, Composer and Dart document +// --------------------------------------------------------------------------- + +/// A union, a space-separated range, a hyphen range and a stability flag are ordinary, +/// valid declarations. Every one of them reached `VersionReq::parse` verbatim, failed, +/// and was recorded as a dependency the run could not evaluate — and the new +/// `unevaluated` gate turns a single one of those into exit 2 for the whole repository, +/// under the `fail-on: vulnerable` the shipped Action defaults to. On `master` the same +/// manifest exited 0. +#[test] +fn the_ordinary_npm_ranges_no_longer_make_a_repository_unanswerable() { + let dir = workdir("gate_npm_range_dialects"); + let base = registry(vec![ + ( + "/react".to_string(), + packument("react", &["16.8.0", "16.14.0", "18.3.1"], "18.3.1"), + ), + ( + "/lodash".to_string(), + packument("lodash", &["4.17.20", "4.17.21"], "4.17.21"), + ), + ( + "/express".to_string(), + packument("express", &["4.18.2", "4.19.2"], "4.19.2"), + ), + ( + "/symfony".to_string(), + packument("symfony", &["6.4.0"], "6.4.0"), + ), + ]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"react\":\"^15.0.0 || ^16.0.0\",\"lodash\":\ + \">=4.17.20 <5.0.0\",\"express\":\"4.18.2 - 4.19.2\",\"symfony\":\"6.4.*@dev\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + !stderr.contains("could not be evaluated"), + "a documented range was read as unevaluable:\n{stderr}" + ); + assert!( + !stdout.contains("unparseable constraint") && !stdout.contains("undetermined"), + "stdout: {stdout}" + ); + // All four are real evaluations. The union resolves to its highest branch, so + // `react`'s newest admissible release is 16.14.0 and the 18.3.1 outside it is an + // available update; the other three are already at the newest release they admit. + assert!(stdout.contains("Totals: 3 up to date"), "stdout: {stdout}"); + assert!( + stdout.contains("^15.0.0 || ^16.0.0") && stdout.contains("update available"), + "stdout: {stdout}" + ); +} + +/// Dart spells "no constraint" as `any`, and `pubspec.yaml` carries an explicit clause to +/// accept it — so the tool read a value in one module and hard-failed the whole run on it +/// in another. +#[test] +fn a_dart_any_constraint_is_read_as_no_constraint() { + let dir = workdir("gate_dart_any"); + let base = registry(vec![( + "/api/packages/meta".to_string(), + json("{\"versions\":[{\"version\":\"1.15.0\"},{\"version\":\"1.16.0\"}]}"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("pubspec.yaml"), + "name: my_app\ndependencies:\n meta: any\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + !stderr.contains("could not be evaluated"), + "`any` was read as unevaluable:\n{stderr}" + ); + assert!(stdout.contains("up to date"), "stdout: {stdout}"); +} + +/// A dist-tag names a channel, not a range. It is `undetermined` — noted, outside the +/// vulnerability and staleness gates, and still failing `--fail-on any`, which is the +/// setting that promises every constraint was established. +#[test] +fn a_dist_tag_is_undetermined_rather_than_unevaluated() { + let dir = workdir("gate_dist_tag"); + let base = registry(vec![( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"express\":\"next\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!(stdout.contains("undetermined"), "stdout: {stdout}"); + assert!( + stderr.contains("note: 1 dependency has a declared version this run could not read"), + "stderr: {stderr}" + ); + + // `--fail-on any` is where "everything must be established" lives, and it still fires. + let strict = check(&dir, &config, &["--fail-on", "any"]); + let (_, strict_stderr, strict_code) = outcome(&strict); + assert_eq!(strict_code, 1, "stderr: {strict_stderr}"); +} + +// --------------------------------------------------------------------------- +// A gate whose entire subject matter was switched off +// --------------------------------------------------------------------------- + +/// `--fail-on vulnerable` is the whole of a claim about advisories, and with scanning off +/// every advisory list is empty — so nothing is ever `Vulnerable`, the gate cannot fail, +/// and the run exits 0 having checked nothing. `[policy]` already refuses this exact +/// configuration one function away; the `--fail-on` gate did not. +#[test] +fn a_vulnerability_gate_with_no_scan_is_refused_rather_than_passed() { + let dir = workdir("gate_vacuous_vulnerable"); + let base = registry(vec![( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"express\":\"^4.19.0\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable", "--no-vuln"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 2, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + stderr.contains("requires vulnerability scanning, which is disabled"), + "stderr: {stderr}" + ); + + // An offline freshness check promises nothing about advisories and still runs. + let offline = check(&dir, &config, &["--fail-on", "outdated", "--no-vuln"]); + let (_, offline_stderr, offline_code) = outcome(&offline); + assert_eq!(offline_code, 0, "stderr: {offline_stderr}"); +} + +// --------------------------------------------------------------------------- +// An override that references a workspace dependency +// --------------------------------------------------------------------------- + +/// A `$name` override takes the referenced entry's raw manifest value, which is whatever +/// `package.json` allows a dependency to be. Reading it as a registry range sent +/// `"my-lib": "workspace:*"` to npm as the literal constraint `workspace:*`, which +/// `VersionReq` cannot parse — so an ordinary pnpm workspace shape took the whole run to +/// exit 2. +#[test] +fn an_override_referencing_a_workspace_dependency_is_not_sent_to_a_registry() { + let dir = workdir("gate_override_workspace_referent"); + let base = registry(vec![( + "/express".to_string(), + packument("express", &["4.19.2"], "4.19.2"), + )]); + let config = write_config(&dir, &base); + fs::write( + dir.join("package.json"), + "{\"name\":\"app\",\"dependencies\":{\"my-lib\":\"workspace:*\",\"express\":\ + \"^4.19.0\"},\"overrides\":{\"my-lib\":\"$my-lib\"}}\n", + ) + .unwrap(); + + let output = check(&dir, &config, &["--fail-on", "vulnerable"]); + let (stdout, stderr, code) = outcome(&output); + + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!( + !stdout.contains("unparseable constraint") && !stdout.contains("not found"), + "a workspace referent was asked of npm:\n{stdout}" + ); + assert!( + !stderr.contains("could not be evaluated"), + "stderr: {stderr}" + ); +} diff --git a/crates/dependable/tests/cli_policy.rs b/crates/dependable/tests/cli_policy.rs index 0732c52..2a47057 100644 --- a/crates/dependable/tests/cli_policy.rs +++ b/crates/dependable/tests/cli_policy.rs @@ -1,6 +1,13 @@ //! End-to-end: the `[policy]` block of `.dependable.toml` gates `dependable //! check`'s exit code. //! +//! Every case here asserts *enforcement*, which is what the `report` feature builds, +//! so the file states that requirement rather than failing a `--no-default-features` +//! run for the absence of a subcommand it never compiled. That a `[policy]` block +//! still **loads** without the feature is asserted in `config::schema_tests`, which +//! runs in both builds. +#![cfg(feature = "report")] +//! //! Hermetic. The fixture declares nothing but path dependencies, so no registry //! fetch task and no OSV query is built — yet every declared dependency still //! yields a result for the policy engine to judge. @@ -125,14 +132,21 @@ fn an_unknown_severity_band_fails_the_run_and_lists_the_valid_ones() { } #[test] -fn an_unrelated_config_typo_still_falls_back_to_defaults() { - // `load_config` stays lenient: only `[policy]` is strict, so tightening the - // gate did not tighten everything else. +fn a_typo_outside_the_policy_block_is_also_an_error() { + // This used to assert the opposite: only `[policy]` was strict, and a typo anywhere + // else was silently dropped. That leniency was not a smaller version of the same + // safety — it was the hole. A dropped `[global]` key resets the table it is in, so + // one mistyped character put `fail_on` back to `none` and disarmed the CI gate, + // with nothing on stderr to say why. let config = config("policy_other_typo", "[global]\nconcurrencyy = 4\n"); let output = check(&config, &["--no-vuln"], &[]); + let stderr = stderr(&output); - assert_eq!(code(&output), 0, "stderr: {}", stderr(&output)); + assert_eq!(code(&output), 2, "stderr: {stderr}"); + // The message names the offending key and the ones that would have worked. + assert!(stderr.contains("concurrencyy"), "stderr: {stderr}"); + assert!(stderr.contains("concurrency"), "stderr: {stderr}"); } #[test] diff --git a/crates/dependable/tests/cli_sarif.rs b/crates/dependable/tests/cli_sarif.rs index 2d36658..2191375 100644 --- a/crates/dependable/tests/cli_sarif.rs +++ b/crates/dependable/tests/cli_sarif.rs @@ -3,6 +3,11 @@ //! Hermetic. The fixture's only dependency is a `path = "..."` one, which is a //! `Local` item and so fails `Item::is_checkable()` — no registry request is ever //! made — and `--no-vuln` skips OSV. Nothing here touches the network. +//! +//! SARIF rendering lives in `dependable-report`, so the format only exists in a build +//! carrying the `report` feature; the file states that rather than failing a +//! `--no-default-features` run for the absence of a renderer it never compiled. +#![cfg(feature = "report")] use std::fs; use std::path::PathBuf; diff --git a/docs/INTEGRATIONS.md b/docs/INTEGRATIONS.md index c307aba..708d98c 100644 --- a/docs/INTEGRATIONS.md +++ b/docs/INTEGRATIONS.md @@ -73,9 +73,9 @@ what exists today vs. what is roadmapped (tracked as GitHub issues; see | **Interactive UI (`dependable`)** | Running `dependable` in a terminal opens a TUI over the resolved dependency graph: browse every project in the repository, descend into sub-dependencies, search by glob, and read each package's public metadata, freshness, and advisories. The graph is built offline from lockfiles; the network is touched only for the package on screen. | **Shipping** | | **Library (`dependable-fetch::Checker`)** | The recommended embedding point. `check_manifest(kind, &str, Option<&str>)` accepts in-memory content — ideal for **unsaved editor buffers** — while `check_path` reads from disk. Emits `ProgressEvent`s for UI progress; a `RegistryFetcher` trait makes new ecosystems purely additive; public types are `#[non_exhaustive]` for forward-compatibility. | **Shipping** | | **CLI JSON output** (`--format json`) | A stable machine schema: a `summary` object plus a `results` array with `status` tokens (`OK`/`PATCH`/`UPDATE`/`OUTDATED`/`VULN`/`ERROR`/`LOCAL`/`GIT`). The generic path for scripting and non-GitHub CI. | **Shipping** | -| **CI exit codes** (`--fail-on none\|outdated\|vulnerable\|any`) | `0` = clean / threshold not met, `1` = threshold met, `2` = tool/fatal error. Also settable via `.dependable.toml` and `DEPENDABLE_FAIL_ON`. | **Shipping** | +| **CI exit codes** (`--fail-on none\|outdated\|vulnerable\|any`) | `0` = clean / threshold not met, `1` = threshold met, `2` = tool/fatal error — which includes a gate the run could not answer (an incomplete vulnerability scan, or dependencies that could not be resolved), so an outage fails the gate rather than passing it. Also settable via `.dependable.toml` and `DEPENDABLE_FAIL_ON`. | **Shipping** | | **`fix`** (`--dry-run`, `--all`) | In-place, position-preserving version rewrites of the **manifest only** — no lockfile edits, no PRs, no scheduled runs. The local, opt-in alternative to a bot's automated PR. | **Shipping** | -| **SARIF output** (`--format sarif`) | SARIF v2.1.0 (`DEP001` outdated / `DEP002` vulnerable, with locations + CVSS) so results upload into the GitHub Security tab and VS Code. | **Roadmap — V2** (#16) | +| **SARIF output** (`--format sarif`) | SARIF v2.1.0 (`DEP001` outdated / `DEP002` vulnerable, with locations + CVSS) so results upload into the GitHub Security tab and VS Code. | **Shipping** | | **GitHub Actions** | A composite action (`.github/actions/dependable-check`) plus `--annotations auto\|always\|never`: PR annotations (`::error file=…,line=…::`) written to **stderr**, so they compose with every `--format`, and a job summary appended to `GITHUB_STEP_SUMMARY`. | **Shipping** | | **GitLab Code Quality** | Code Quality JSON report format. | **Roadmap — V2** (#19) | | **First-party editor integration** | An official **LSP server and/or VSCode extension** built on `dependable-fetch` — inline outdated/vulnerable hints and quick-fixes, powered by the same engine as the CLI. | **Roadmap — target V2** (new) | diff --git a/docs/SCOPE.md b/docs/SCOPE.md index a3026d4..e9a1f78 100644 --- a/docs/SCOPE.md +++ b/docs/SCOPE.md @@ -23,8 +23,11 @@ end-to-end and establishes the type model + traits that later ecosystems plug in | `dependable-tui` | library | The interactive terminal UI (ratatui): the dependency forest, recursive drill-down, glob search, and the package detail pane. Holds no IO of its own — it drives `dependable-fetch`. | | `dependable` | application | CLI: `check` / `list` / `tree` / `fix` / `tui`, with `table` / `json` / `text` output, `.dependable.toml` + `DEPENDABLE_*` config, and `--fail-on` CI exit codes. A bare `dependable` in a terminal opens the TUI. | -`dependable-report` (the V2 crate) is scaffolded — the crate and its module slots -exist, but it renders nothing yet and ships no user-visible command. +`dependable-report` **ships**: HTML reports, SARIF v2.1.0, the SPDX license +evaluator, and the `[policy]` engine, all reachable from the CLI (`dependable report`, +`check --format sarif`, and a `[policy]` block in `.dependable.toml`). It is a library +only — the scaffolded `dependable-report` binary was removed rather than published as +a command that printed "not implemented yet" and exited 2. ### 1a. Dependency tree (`tree`) — Rust, offline @@ -170,7 +173,10 @@ Cross-cutting enablers (also V1.1): extend `Ecosystem`/`ManifestKind` enums + `d | License visibility + allowlist | `list --licenses`; `[policy] allowed_licenses` over a documented SPDX subset. Go, JSR, NuGet, and pub.dev publish no registry license | registry fields | §6.4, §8 D6 | | PDF export | `--pdf` via headless chromium (HTML stays self-contained) — **V3** | system chromium | §6.1, §8 D4 | -### A3 — V1.1 / future polish (non-goals that are real future work) +### A3 — V1.1 polish (**all shipped**) + +Kept for the record of what each item covered; every row below is in the product +today. | Item | What it covers | PRD | |---|---|---| diff --git a/release-plz.toml b/release-plz.toml index 327e538..4a80a5c 100644 --- a/release-plz.toml +++ b/release-plz.toml @@ -9,7 +9,8 @@ # crates.io publishing is enabled: every crate is published so both library # consumers (dependable-core, dependable-fetch) and `cargo install dependable` # work. release-plz publishes in dependency order -# (core → fetch → tui → dependable). +# (core → report → fetch → tui → dependable) — `dependable` depends on +# `dependable-report` by version, so that crate has to exist on crates.io first. # # Auth is crates.io Trusted Publishing (OIDC) only — the `release` job carries # `id-token: write` and release-plz mints a short-lived token per run. We never