Skip to content

test: state each fixture's expectations in its header and check them - #23

Merged
SizzleUnrlsd merged 2 commits into
mainfrom
test/fixture-expectations
Sep 27, 2026
Merged

SizzleUnrlsd merged 2 commits into
mainfrom
test/fixture-expectations

Conversation

@SizzleUnrlsd

Copy link
Copy Markdown
Contributor

Item 2 of #21. Stacked on #22 (test/api-consumer): merge that one first.

Change

  1. Every fixture states what it proves. The 43 sources under test/ carry a header, one line per expectation, read by test/check_fixtures.py (the fixtures test):

    // args: --ct-modules=alloc,vtable --ct-vtable-diag
    // expect: rule=vtable-null-this cwe=CWE-476 level=error line=11
    // expect: none
    // expect: program-exit=3
    // expect: output="auto-free ptr="

    Each fixture runs with its arguments (--ct-modules=alloc,bounds by default); the SARIF results must match the declared findings one to one, faulting line and allocation site in the fixture as compiled; the analyzer's exit status must be 1 with findings and 0 without. program-exit and output check the program itself. expect[linux]: and expect[darwin]: keep a line to one platform. A fixture without an expectation fails, which is what the checker did on its first run, on all 43.

  2. What the fixtures now prove. The 7 memory and 5 vtable-diagnostic fixtures keep the expectations of the old table. The 14 autofree fixtures get back the expectations of run_autofree_tests.sh: auto-freed allocations leave no leak and print the auto-free line, the three escaping ones leak and exit 1. The five other vtable fixtures run with the vtable module and report nothing; ct_vtable_uaf.cpp is reported by the alloc module, as the access faults before the vtable module sees the freed vptr (with the vtable module alone it segfaults with no finding). ct_realloc_zero.c and ct_autofree_sbrk.c differ by platform: realloc(p, 0) frees on glibc and leaks on macOS; sbrk is a real allocation on Linux only. The vtable-diagnostic fixtures' hand-written site strings now name the line of the call itself (they pointed two lines above it).

  3. test/cli/check_cli.py (the cli test) keeps the CLI checks of the old findings script: text output, the SARIF log of a batch, exit 2 for what cannot be analyzed.

  4. Deleted: check_findings.py (replaced), sweep_compiles (sweep_runs builds the same tree and keeps the batch heuristics, -x c++ and -DDEBUG, covered), and the template leftovers registered nowhere: test/examples/test_*.py, run_autofree_tests.sh, test/scripts/, test/docker/.

  5. The header lines shift heap_overflow_write.c by one line: check-analysis.sh and api_test.cpp follow. README: the tests table and how to write a fixture.

Tests (written first)

check_fixtures.py ran before any fixture had a header: 43 failures, "states no expectation". With the headers, 43 pass on macOS. With ParseFindings disabled in Run: 40 failures in fixtures and cli fails too.

Verification

  • macOS arm64: ctest --test-dir build 14/14 (fixtures 16 s, cli 4 s); ./scripts/format-check.sh clean.
  • Linux: this pull request's Build workflow. The two platform-specific expectations are predictions for Linux; the workflow decides them.

The findings test checked eleven fixtures from a table in the script, and
the sweeps only asked the 31 others to build and finish. Every fixture under
test/ now states what it proves in its header, one line per expectation:

    // args: --ct-modules=alloc,vtable --ct-vtable-diag
    // expect: rule=vtable-null-this cwe=CWE-476 level=error line=11
    // expect: none
    // expect: program-exit=3
    // expect: output="auto-free ptr="

test/check_fixtures.py (the `fixtures` test) runs each one with its
arguments and matches the SARIF results against the declared findings,
faulting line and allocation site included, or none; the analyzer's exit
status follows. A fixture without an expectation fails, so nobody adds one
without saying what it proves. expect[linux] and expect[darwin] keep a line
to one platform: realloc(p, 0) frees on glibc and leaks on macOS, and sbrk
is only a real allocation on Linux.

The autofree fixtures get back the expectations of the old shell script:
auto-freed allocations leave no leak and print the auto-free line, the
escaping ones leak and exit 1. The five vtable fixtures outside the
diagnostics now run with the vtable module; ct_vtable_uaf.cpp is reported by
the alloc module, as the access faults before the vtable module sees the
freed vptr. The diagnostics fixtures' hand-written sites name the line of
the call itself.

The CLI checks of the old script (text output, batch log, exit 2) move to
test/cli/check_cli.py (the `cli` test). sweep_compiles goes: sweep_runs
builds the same tree and keeps the batch-mode heuristics covered. The
header lines shift heap_overflow_write.c by one line, followed by
check-analysis.sh and api_test.

Part of #21.
test/examples/test_*.py needed the ctestfw package and tested a `cc`
compiler, run_autofree_tests.sh drove build/cc by hand, and test/scripts
with test/docker built that. None was registered with CMake or run by CI;
their expectations now live in the fixtures' headers.

Part of #21.
Base automatically changed from test/api-consumer to main September 27, 2026 15:13
@SizzleUnrlsd SizzleUnrlsd self-assigned this Sep 27, 2026
@SizzleUnrlsd
SizzleUnrlsd merged commit 1a9e472 into main Sep 27, 2026
8 of 9 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the test/fixture-expectations branch September 27, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant