Repository navigation
test: state each fixture's expectations in its header and check them - #23
Merged
Merged
Conversation
The findings test checked eleven fixtures from a table in the script, and
the sweeps only asked the 31 others to build and finish. Every fixture under
test/ now states what it proves in its header, one line per expectation:
// args: --ct-modules=alloc,vtable --ct-vtable-diag
// expect: rule=vtable-null-this cwe=CWE-476 level=error line=11
// expect: none
// expect: program-exit=3
// expect: output="auto-free ptr="
test/check_fixtures.py (the `fixtures` test) runs each one with its
arguments and matches the SARIF results against the declared findings,
faulting line and allocation site included, or none; the analyzer's exit
status follows. A fixture without an expectation fails, so nobody adds one
without saying what it proves. expect[linux] and expect[darwin] keep a line
to one platform: realloc(p, 0) frees on glibc and leaks on macOS, and sbrk
is only a real allocation on Linux.
The autofree fixtures get back the expectations of the old shell script:
auto-freed allocations leave no leak and print the auto-free line, the
escaping ones leak and exit 1. The five vtable fixtures outside the
diagnostics now run with the vtable module; ct_vtable_uaf.cpp is reported by
the alloc module, as the access faults before the vtable module sees the
freed vptr. The diagnostics fixtures' hand-written sites name the line of
the call itself.
The CLI checks of the old script (text output, batch log, exit 2) move to
test/cli/check_cli.py (the `cli` test). sweep_compiles goes: sweep_runs
builds the same tree and keeps the batch-mode heuristics covered. The
header lines shift heap_overflow_write.c by one line, followed by
check-analysis.sh and api_test.
Part of #21.
test/examples/test_*.py needed the ctestfw package and tested a `cc` compiler, run_autofree_tests.sh drove build/cc by hand, and test/scripts with test/docker built that. None was registered with CMake or run by CI; their expectations now live in the fixtures' headers. Part of #21.
This was referenced Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 2 of #21. Stacked on #22 (
test/api-consumer): merge that one first.Change
Every fixture states what it proves. The 43 sources under
test/carry a header, one line per expectation, read bytest/check_fixtures.py(thefixturestest):Each fixture runs with its arguments (
--ct-modules=alloc,boundsby default); the SARIF results must match the declared findings one to one, faulting line and allocation site in the fixture as compiled; the analyzer's exit status must be1with findings and0without.program-exitandoutputcheck the program itself.expect[linux]:andexpect[darwin]:keep a line to one platform. A fixture without an expectation fails, which is what the checker did on its first run, on all 43.What the fixtures now prove. The 7 memory and 5 vtable-diagnostic fixtures keep the expectations of the old table. The 14 autofree fixtures get back the expectations of
run_autofree_tests.sh: auto-freed allocations leave no leak and print the auto-free line, the three escaping ones leak and exit1. The five other vtable fixtures run with the vtable module and report nothing;ct_vtable_uaf.cppis reported by the alloc module, as the access faults before the vtable module sees the freed vptr (with the vtable module alone it segfaults with no finding).ct_realloc_zero.candct_autofree_sbrk.cdiffer by platform:realloc(p, 0)frees on glibc and leaks on macOS;sbrkis a real allocation on Linux only. The vtable-diagnostic fixtures' hand-written site strings now name the line of the call itself (they pointed two lines above it).test/cli/check_cli.py(theclitest) keeps the CLI checks of the old findings script: text output, the SARIF log of a batch, exit2for what cannot be analyzed.Deleted:
check_findings.py(replaced),sweep_compiles(sweep_runsbuilds the same tree and keeps the batch heuristics,-x c++and-DDEBUG, covered), and the template leftovers registered nowhere:test/examples/test_*.py,run_autofree_tests.sh,test/scripts/,test/docker/.The header lines shift
heap_overflow_write.cby one line:check-analysis.shandapi_test.cppfollow. README: the tests table and how to write a fixture.Tests (written first)
check_fixtures.pyran before any fixture had a header: 43 failures, "states no expectation". With the headers, 43 pass on macOS. WithParseFindingsdisabled inRun: 40 failures infixturesandclifails too.Verification
ctest --test-dir build14/14 (fixtures16 s,cli4 s);./scripts/format-check.shclean.Buildworkflow. The two platform-specific expectations are predictions for Linux; the workflow decides them.