Repository navigation
test: check every CLI option and every invalid-argument path - #24
Merged
Merged
Conversation
Half of the options had no test. check_cli.py now proves each one as the README states it, on small programs written into the work directory: --run-arg values reach the program in order, --env sets the variable and nothing leaks without it, --cwd runs the program there (with a relative -o too), --show-output and --show-events print after the summary and only when asked, --format as a separate argument, --output=<path>, --help. In batch mode: a non-zero program exit is [RUNTIME] and counted but not a finding, --strict-test-exit raises the status to 1, --output-dir places the binaries, --test-dir is repeatable, and a missing directory gives exit 2 with a message. Every ParseArgs error path (unknown option, missing value, bad --env, negative --timeout, bad --format, no compiler arguments) exits 2 with its message and the usage on stderr; so does -o given twice. With the option handlers removed one by one, nine checks fail. Part of #21.
The check wrote cwd_relative and its .dSYM next to the fixtures, under test/, which the sweeps run in full. The relative path now points into the work directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 3 of #21. Stacked on #23 (
test/fixture-expectations): merge #22 and #23 first.Change
test/cli/check_cli.py(theclitest) proves every option as the README's table states it, on small programs written into the work directory (not undertest/, which the sweeps run in full):--run-argvalues reach the program as arguments, in order;--env NAME=VALUEsets the variable, and nothing leaks into the program without it;--cwdruns the program in that directory, with a relative-otoo.--show-outputprints the program's stdout and stderr after the summary,--show-eventsthe CoreTrace event lines; neither is printed without the option.--format textand--format sarifas a separate argument,--output=<path>with an equals sign,--helpon stdout with exit0.[RUNTIME]on its line and counted inruntime_failures, but is not a finding (exit0);--strict-test-exitraises the status to1;--output-dirplaces the binaries;--test-diris repeatable; a directory that does not exist gives exit2and a message.ParseArgserror path exits2with its message and the usage on stderr: unknown option,-o/--run-arg/--env/--cwd/--timeout/--test-dir/--output-dir/--formatwithout a value,--envwithoutNAME=or with an empty name, a negative--timeout, no compiler arguments with or without--.-ogiven both to the analyzer and to the compiler is refused too.README: the
clirow of the tests table.Tests (written first)
The checks are the deliverable, written from the README's option table before running them; all 48 passed on the first run, so the documented behaviour holds. To prove they bite, I removed the handlers of
--run-arg,--env,--cwd,--strict-test-exit,--show-outputand--show-eventsand the unknown-option message: nine checks fail, one or two per option, and none of the others.Verification
ctest --test-dir build14/14 (cli8 s);./scripts/format-check.shclean.Buildworkflow.