Skip to content

test: check every CLI option and every invalid-argument path - #24

Merged
SizzleUnrlsd merged 2 commits into
mainfrom
test/cli-options
Sep 27, 2026
Merged

SizzleUnrlsd merged 2 commits into
mainfrom
test/cli-options

Conversation

@SizzleUnrlsd

Copy link
Copy Markdown
Contributor

Item 3 of #21. Stacked on #23 (test/fixture-expectations): merge #22 and #23 first.

Change

test/cli/check_cli.py (the cli test) proves every option as the README's table states it, on small programs written into the work directory (not under test/, which the sweeps run in full):

  • --run-arg values reach the program as arguments, in order; --env NAME=VALUE sets the variable, and nothing leaks into the program without it; --cwd runs the program in that directory, with a relative -o too.
  • --show-output prints the program's stdout and stderr after the summary, --show-events the CoreTrace event lines; neither is printed without the option.
  • --format text and --format sarif as a separate argument, --output=<path> with an equals sign, --help on stdout with exit 0.
  • Batch mode: a program exiting non-zero is [RUNTIME] on its line and counted in runtime_failures, but is not a finding (exit 0); --strict-test-exit raises the status to 1; --output-dir places the binaries; --test-dir is repeatable; a directory that does not exist gives exit 2 and a message.
  • Every ParseArgs error path exits 2 with its message and the usage on stderr: unknown option, -o/--run-arg/--env/--cwd/--timeout/--test-dir/--output-dir/--format without a value, --env without NAME= or with an empty name, a negative --timeout, no compiler arguments with or without --. -o given both to the analyzer and to the compiler is refused too.

README: the cli row of the tests table.

Tests (written first)

The checks are the deliverable, written from the README's option table before running them; all 48 passed on the first run, so the documented behaviour holds. To prove they bite, I removed the handlers of --run-arg, --env, --cwd, --strict-test-exit, --show-output and --show-events and the unknown-option message: nine checks fail, one or two per option, and none of the others.

Verification

  • macOS arm64: ctest --test-dir build 14/14 (cli 8 s); ./scripts/format-check.sh clean.
  • Linux: this pull request's Build workflow.

Half of the options had no test. check_cli.py now proves each one as the
README states it, on small programs written into the work directory:
--run-arg values reach the program in order, --env sets the variable and
nothing leaks without it, --cwd runs the program there (with a relative -o
too), --show-output and --show-events print after the summary and only when
asked, --format as a separate argument, --output=<path>, --help. In batch
mode: a non-zero program exit is [RUNTIME] and counted but not a finding,
--strict-test-exit raises the status to 1, --output-dir places the binaries,
--test-dir is repeatable, and a missing directory gives exit 2 with a
message. Every ParseArgs error path (unknown option, missing value, bad
--env, negative --timeout, bad --format, no compiler arguments) exits 2 with
its message and the usage on stderr; so does -o given twice.

With the option handlers removed one by one, nine checks fail.

Part of #21.
The check wrote cwd_relative and its .dSYM next to the fixtures, under
test/, which the sweeps run in full. The relative path now points into the
work directory.
Base automatically changed from test/fixture-expectations to main September 27, 2026 15:14
@SizzleUnrlsd SizzleUnrlsd self-assigned this Sep 27, 2026
@SizzleUnrlsd
SizzleUnrlsd merged commit 8f1e915 into main Sep 27, 2026
3 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the test/cli-options branch September 27, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant