Skip to content

ci(patch): Recover the benchmark check's exit status and find the coverage executable - #9

Open
hassila wants to merge 1 commit into
mainfrom
ci/fix-coverage-export-and-benchmark-status
Open

hassila wants to merge 1 commit into
mainfrom
ci/fix-coverage-export-and-benchmark-status

Conversation

@hassila

@hassila hassila commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Two workflows fail on every pull request in this repository, independent of the change under review; #8 shows both.

Benchmark comparison always reports an unknown failure. The compare step in swift-benchmark-delta.yml runs under GitHub's default bash -e. swift package benchmark baseline check exits non-zero for any threshold deviation, so the step aborted at that line, before the stderr mapping that turns the result into same, improvement, regression, crash or missing baseline. exitStatus kept its initial value of 1 and the job failed with "unknown failure" whatever the benchmark actually said; the workflow has never passed in its eight-run history. A set +e / set -e guard around the check lets the mapping run. With the mapping reachable, a crash or a missing baseline now also gets a comment producer: the failure text step covers every status other than same, regression and improvement, with distinct wording for those two.

Coverage export finds no executable. swift-code-coverage.yml looked for .build/debug/<repository name>PackageTests.xctest, but the package is named FixedPointDecimal, so that path never existed and the job relied on a recursive find for any .xctest file. That worked until the swift:latest container moved to Swift 6.4, whose default Swift Build backend emits per-target -test-runner and .so artifacts on Linux and no .xctest at all. The find came back empty, and because the file test was unquoted, [ -f ] evaluated true and llvm-cov export ran without a file ("No filenames specified!"). The coverage run now uses --build-system native (supported, deprecated in 6.4), which keeps the single <Package>PackageTests.xctest executable, and the export step finds it by package name, keeps the recursive fallback, quotes the path, and lists the build directory before failing if nothing is found.

Verification. Shell control flow of both steps checked by hand against the follow-up steps; SwiftPM's behaviour for --build-system native and the profile path taken from the 6.4.0 release sources. No Linux Swift 6.4 build was run locally (no container runtime on the machine), so this PR's own CI run is the first real test of the coverage path. Codex (gpt-6-astra) blocked the first version of the coverage fix, which assumed a .swift-testing executable that Swift Build never produces; the rework above answers that and its two should-fix points, and its final pass found nothing blocking.

🤖 Generated with Claude Code

…erage executable

Two workflow defects hit every pull request. The benchmark comparison step runs under `bash -e`, so
a non-zero exit from `benchmark baseline check` aborted the step before the exit-code mapping and
every threshold deviation reported as an unknown failure; the workflow has failed on all eight runs
in its history. With the mapping reachable, a crash or a missing baseline also needs a comment
producer, so the failure text now covers every status that is not same, regression or improvement.

The coverage export looked for `<repository>PackageTests.xctest`, but the package is named
FixedPointDecimal, and since the `swift:latest` container moved to 6.4 the default Swift Build
backend emits per-target runner and `.so` artifacts, so the fallback `find` came back empty and
`llvm-cov` ran with no file. The coverage run now uses the native build system, whose single
`<Package>PackageTests.xctest` executable the export step finds by package name, and the step lists
the build directory when it finds nothing instead of invoking `llvm-cov` without a file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.83%. Comparing base (ab45157) to head (8c2ea99).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   96.79%   96.83%   +0.04%     
==========================================
  Files          35       35              
  Lines        4795     4795              
==========================================
+ Hits         4641     4643       +2     
+ Misses        154      152       -2     

see 2 files with indirect coverage changes
see 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab45157...8c2ea99. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

@hassila
hassila requested a review from freef4ll September 21, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant