Repository navigation
ci: require Python for the tests and measure their coverage with a floor - #26
Merged
Merged
Conversation
The fixtures, cli, timeout and compile_failure tests are Python scripts. Without an interpreter, configuration used to succeed and the suite ran without them, silently. Python 3 is now required when the tests are configured. The README also said CMake 3.16 where 3.21 has been required since the release archives. Part of #21.
-DCORETRACE_COVERAGE=ON instruments this project's own targets (the library, the CLI and the unit tests) at -O0 for source-based coverage; fetched dependencies are not. scripts/ci/coverage.sh runs the suite with LLVM_PROFILE_FILE set, merges the profiles, reports src/, include/ and main.cpp per file, writes an HTML report, adds the totals to the GitHub Actions job summary, and fails when the line coverage is below the floor it is given. The coverage job of the Build workflow runs it on Ubuntu 24.04 with clang-20 and uploads the report. Its floor starts at 100 so that the first run shows the value reached; the next commit sets it there. Locally on macOS (AppleClang, xcrun llvm-cov): 89.20% of lines, 100% of functions; process.cpp is the low file at 58%, the other platform's paths. Part of #21.
The first run of the coverage job on Ubuntu 24.04 with clang-20 measured 89.20% of lines and 100% of functions, the same as the local macOS run. process.cpp is the low file at 58%: its child-side code between fork and exec never reaches a profile, and its pipe, fork and waitpid failure paths are not provoked. The floor is set just under the value reached, to be raised as tests land. Part of #21.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 5 of #21, the last one: closes #21. Stacked on #25 (
test/cli-unit): merge #22 to #25 first.Change
build: fail configuration without Python 3. The fixtures, cli, timeout and compile_failure tests are Python scripts; without an interpreter, configuration used to succeed and the suite ran without them, silently.find_package(Python3 REQUIRED)when the tests are configured. The README also said CMake 3.16 where 3.21 has been required since the release archives.ci: measure the suite's coverage with llvm-cov and fail below a floor.-DCORETRACE_COVERAGE=ONinstruments this project's own targets (the library, the CLI, the unit tests) at-O0for source-based coverage, as ctrace does; fetched dependencies are not.scripts/ci/coverage.sh <build> <out> [<floor>]runs the suite withLLVM_PROFILE_FILEset (the Python tests spawn the CLI many times,%4mmerges per binary), merges the profiles, reportssrc/,include/andmain.cppper file, writes an HTML report, adds the totals to the job summary, and fails below the floor. Thecoveragejob of theBuildworkflow runs it on Ubuntu 24.04 with clang-20 and uploads the report.ci: set the coverage floor at 89% of linessets it just under, to be raised as tests land.Tests (written first)
-DPython3_EXECUTABLE=/nonexistent/python3, the tree used to configure with "Could NOT find Python3" and four tests fewer; it now fails at configuration.xcrun llvm-cov):src/cli_args.cppsrc/events.cppsrc/findings.cppsrc/process.cppsrc/runtime_analyzer.cppsrc/sarif.cppsrc/shipped_toolchain.cppprocess.cppreads the same on Linux: its child-side code betweenforkandexecnever reaches a profile, and its pipe, fork and waitpid failure paths are not provoked.Verification
ctest --test-dir build17/17 on the instrumented tree;./scripts/format-check.shclean.Buildworkflow. Thecoveragejob failed once at floor 100, as meant, and passes at 89.