Skip to content

ci: fail security audits against the committed lockfile - #1228

Open
loonghao wants to merge 2 commits into
mainfrom
fix/committed-lock-audit-gate
Open

loonghao wants to merge 2 commits into
mainfrom
fix/committed-lock-audit-gate

Conversation

@loonghao

@loonghao loonghao commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Audit the committed Cargo.lock without regenerating or reselecting dependencies.
  • Propagate cargo-audit's nonzero exit status instead of suppressing it in the just recipe.
  • Route the scheduled audit through the same recipe as PR CI.
  • Run script regressions when either audit workflow or the justfile changes.

Why

The current audit job reports success even when its log says 8 vulnerabilities found and 8 denied warnings found. It also attempts a lock regeneration that fails because the zip ^8.1 compatible releases are yanked. The leading - in the just recipe and || true in the scheduled workflow hide both errors.

Concrete existing-head audit evidence.

Verification

  • Added five stdlib Python regression tests: six assertions fail against unchanged main, all five tests pass against this patch.
  • The tests execute the actual recipe's shell command with a controlled vx process to verify exit codes 0, 1, 2 and 101 propagate, the exact audit arguments are used, and existing lock bytes remain unchanged.
  • The workflow's actual CI Success shell gate is executed with successful, failed and cancelled audit results.
  • Static contracts cover the non-suppressing just recipe, shared scheduled/PR entrypoint, unchanged warning denial, and both push/PR regression triggers.
  • Python compilation and trailing-whitespace checks pass.
  • Local source snapshot only: vx/just/Rust tooling is not installed in this cloud workspace. No local Rust build or live cargo-audit execution is claimed. Hosted terminal validation is recorded below.

Scope and expected result

This is a truthful reporting fix, not a dependency remediation. With the current dependencies the real audit is expected to fail, and CI must reflect that. No Cargo.toml/Cargo.lock change, advisory exclusion, warning relaxation, permission, credential, branch-protection or release-setting change.

The compatible dependency patches remain separately in #1227. Remaining dependencies and the existing advisory inline-test warning are separate work. This PR remains Draft; no merge or release is included.

Final-head verification

Final head a245ee621e25f768b0ce116a4d87e487e86364be includes the independent-review test hardening.

  • Script Tests passed all three jobs. The unit-test log records 154 tests passed, including all five new audit tests; Python syntax and shell lint also passed.
  • Independent review found no blocking regression and reran the five tests. Two intentionally broken variants (|| : and a disabled scheduled audit step) now both fail the contract test.
  • Real Security Audit job executed vx just security-audit-ci → vx cargo audit --deny warnings. It scanned 573 committed dependencies, reported 13 vulnerabilities and 9 denied warnings, and failed with exit code 1, including the failing recipe and GitHub step. There is no generate-lockfile invocation in this log.
  • This newer result is against unchanged main's dependency lock and the advisory database fetched on 2026-10-17. It must not be conflated with the historical fix(deps): patch four compatible locked dependencies #1227 candidate's 8/8 result or described as a security remediation.
  • Full CI 39010222418 completed on 2026-10-17. Its only failed jobs are the real Security Audit and the CI Success aggregate, which received security-audit=failure and exited 1. This verifies end-to-end failure propagation; it is not an infrastructure error or a clean security audit.
  • Raw full-workspace summaries: Linux 4289 passed / 128 skipped; macOS 4292 passed / 128 skipped; Windows 4313 passed / 130 skipped. All three logs report 0 failures absorbed by nextest retries. Windows marks vx::integration_tests tool_specific_tests::test_go_tool_help as LEAK (1 leaky passing test); this is not a claim of warning-free tests.
  • All four native release build jobs, GNU ARM64 cross-build, provider static checks and three installer jobs passed. Extended-format installer summaries total 27 Linux, 27 macOS and 24 Windows tests, with 0 skipped.
  • Code Quality passed, including hakari (no operations to perform), formatting, clippy and docs. Its pre-existing inline-test checker still fails under || true; that advisory result remains explicitly out of scope and must not be called a fully clean quality gate.
  • All six other final-head workflows (Script Tests, Test Setup Action, Test Providers, Test Docker Images, Repo contract and CodeQL) completed successfully.
  • The audit failure remains visible. No suppressions or exclusions were added to make this PR green; dependency remediation and any merge decision remain separate.

Strict audit of merged main — 2026-10-17 08:03 UTC

The compatible lock patches in #1227 are now merged as 0388b75a24f28b376c5372c95a4d064ffdd581df. A separate read-only audit of that exact committed lock reports 11 vulnerabilities and 7 denied warnings, with exit code 1.

  • Tool: official cargo-audit 0.22.2 release, Linux musl archive SHA256 7fb9497f8594b389e5fce5ef9b92db08432996895b2e0c5a0167a69ed445c428, verified before execution.
  • Lock identity: 573 dependencies, Git blob bffb70410116a0e4dc754fcbe32559bc009c8e40, SHA256 dff420534c64eba856804ad49036002e8215bc87c1202ca9391206b3cfeeca2b. The blob matches merged main and its bytes were unchanged after both JSON and terminal audits.
  • Command: cargo-audit audit --deny warnings --file Cargo.lock --db advisory-db --json; terminal confirmation used the same database with --no-fetch. No ignore, target filtering, lock regeneration or dependency update was used.
  • Advisory DB: 64f92cdc64401fab47baa10e2e11fe382aa57155, 1334 advisories.
  • Remaining vulnerability reports: quinn-proto 0.11.14 (5), hickory-proto 0.25.2 (2), zip 8.6.0 (3), sevenz-rust 0.6.1 (1). Denied warnings: 6 unmaintained packages and lru 0.16.4 unsoundness. This is package-level advisory evidence, not a full application reachability assessment.
  • The automatic main audit instead regenerated a temporary lock containing 578 dependencies and swallowed a 6-vulnerability/7-warning failure. That result does not describe the committed 573-dependency lock and is not a clean audit.

Dependency remediation remains outstanding. The audit-gate fix in #1228 remains Draft and blocked by its failed required CI Success check; no protection or permission changes, bypass, or merge of #1228 was performed.

@loonghao
loonghao marked this pull request as ready for review November 17, 2026 10:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant