Repository navigation
ci: let a cancelled job's test harness stop its VMs and remove its scratch - #975
Conversation
A cancelled CI step gets SIGINT and, if it still runs 7.5 s later, SIGTERM, and the runner kills its process tree 2.5 s after that. nvx.py turns SIGINT into KeyboardInterrupt, which unwinds the command through the context managers that stop its OpenVMM processes and remove its scratch directories. On SIGTERM, though, Python exits at once and runs none of them, so a harness whose cleanup outlasts the SIGINT window, or that something else ends with SIGTERM, such as timeout(1), leaves them behind (#946). main() now runs the command with SIGTERM handled as Python handles SIGINT, by signal.default_int_handler, so either signal interrupts the command, which then prints Interrupted and exits with status 130. As Python does for SIGINT, it leaves an inherited ignored SIGTERM ignored. It installs the handler only in the main thread of a POSIX process, and restores the default disposition once the command returns, so that in-process callers such as the tests keep their own. A CliTests test sends its own process SIGTERM from a command and checks that the command unwinds, that main() reports the interruption and restores the disposition, and that an ignored SIGTERM stays ignored. doc/usage.md documents the behavior, and a new Cancellation section of doc/ci.md describes how the runner cancels a step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
run_acceptance, which runs the sandbox-lifecycle and sandbox-proxy-only scenarios, stops and deprovisions the sandboxes of an acceptance that fails, but only on an Exception. Ctrl-C raises KeyboardInterrupt, and so will a cancelled CI job once its SIGINT reaches the harness (#946), and that skipped the cleanup: the sandbox's OpenVMM, which nvx-rs sandbox start detaches from the harness, kept running, and the fixture stayed in the temporary directory. Any exception now cleans up the acceptance. A failure still raises a RuntimeError that names the cleanup errors, among them a fixture that the cleanup preserved for recovery. An interruption propagates as it is, so that nvx.py reports it as one and the scenario duration report records it as interrupted, after a warning that names those errors. A test interrupts the fake sandbox CLI during the security probe and checks that the cleanup stops and deprovisions the running sandbox, removes the fixture, keeps the checks' evidence, and propagates the KeyboardInterrupt, and that a failed deprovision preserves the fixture with a warning that names it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OpenvmmProcess built the TimeAbiMonitor that scans OpenVMM's console only after it had started OpenVMM and its output reader, outside the try blocks that stop OpenVMM when the constructor fails (#720). The monitor only parses the command line, so it cannot fail on a valid command, but an error there would have left OpenVMM running with nothing to stop it (#946). The constructor now builds the monitor first, before it opens the transcript or starts OpenVMM, so that a failure there starts nothing. A test makes the monitor fail and checks that OpenVMM never starts and that no transcript is created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
A critical startup race can leave OpenVMM running if the harness dies before the child arms its parent-death signal; an owner-PID check after arming and regression coverage are needed.
1 open issue
What changed in this PR
This PR addresses #946 by improving cancellation handling in NVX’s Linux test harnesses, allowing cleanup to stop VMs and remove scratch directories.
Changes:
- Forward CI cancellation signals to the harness and run cleanup on interruption.
- Add parent-death handling for launched processes and initialize the time ABI monitor before starting OpenVMM.
- Remove leftover scratch directories during runner validation, with regression tests and documentation.
| File | Description |
|---|---|
scripts/test_nvx_tools.py |
Tests signal handling, forwarding, and scratch cleanup. |
scripts/test_microvm_tests.py |
Tests interrupted sandbox cleanup and monitor initialization failure. |
scripts/test_adversarial.py |
Tests parent-death handling and supervisor behavior. |
scripts/nvx.py |
Handles SIGTERM as an interruption on POSIX. |
scripts/nvx_tools/sandbox_lifecycle_tests.py |
Cleans up sandboxes when acceptance is interrupted. |
scripts/nvx_tools/process_supervisor.py |
Accepts the expected owner PID. |
scripts/nvx_tools/openvmm_process.py |
Initializes the monitor before launching OpenVMM. |
scripts/nvx_tools/benchmark.py |
Applies parent-death handling to direct launches. |
scripts/nvx_tools/adversarial_oracles.py |
Adds parent-death wrappers and supervisor cleanup. |
doc/usage.md |
Documents interruption and process cleanup behavior. |
doc/ci.md |
Explains cancellation handling and leftover cleanup. |
doc/benchmarks.md |
Documents the added Linux launch overhead. |
.github/workflows/run-nvx-microvm-tests.yml |
Executes Linux harnesses directly with exec. |
.github/actions/validate-runner/action.yml |
Removes scratch directories left by earlier jobs. |
.github/actions/run-benchmark/forward-signals.sh |
Forwards cancellation signals to background harnesses. |
.github/actions/run-benchmark/action.yml |
Uses signal forwarding or exec for Linux benchmarks. |
.github/actions/check-quality/action.yml |
Adds shell checks for the signal-forwarding helper. |
Review effort: Balanced
37c91b4 to
7781277
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Cancellation can be lost during harness teardown, and runner cleanup can kill the supervisor while leaving OpenVMM running without regression coverage for that scenario.
2 open issues
1 resolved since last review
Review effort: Balanced
When a CI job is cancelled, the runner's orphan cleanup kills the test harness outright, together with the job's processes that it listed. OpenVMM ran as a plain child of the harness, so one that the harness started after the listing kept running: a VM ran on azure-mshv-1 for 8 days (#768). A contained process tree, such as nvx-rs run and the OpenVMM that it starts, ran under a supervisor that had no parent-death signal either (#946). On Linux, InteractiveProcess now starts a command that it doesn't contain through util-linux's setpriv --pdeathsig KILL, which makes SIGKILL the command's parent-death signal (PR_SET_PDEATHSIG), and then through a shell that starts the command only if the harness is still its parent, since a harness that died before setpriv armed the signal sent none. The signal survives both execs, so the kernel kills OpenVMM when the harness exits. A preexec_fn that called prctl would avoid the extra execs, but it makes CPython fork the whole harness instead of using vfork: on a Xeon Silver 4114 host, it made Popen take 10.3 ms in a 75 MiB process and 37.7 ms in a 265 MiB one, against 0.64 ms, while setpriv and the shell added 2.1 ms to the time until the command ran with Dash as /bin/sh, and 3.5 ms with Bash. The benchmark's launch timings include them; on the KVM runners, the p50 of openvmm_snapshot_restore spans 51.7 to 63.8 ms over the last 20 commits of its history. A host without setpriv starts the command as before; the Azure Linux 3 runners have it in util-linux 2.40.2. ProcessTreeContainment now passes the supervisor the ID of the process that started it. The supervisor makes SIGTERM its parent-death signal, checks once that its parent still runs, since a parent that exited before then sends no signal, and only then starts the command. A signal that finds the supervisor with another parent kills the whole tree, descendants that left its session included, and the supervisor then ends by SIGTERM. Signals that arrive while its parent lives reach the command as before, and those that arrive before the command starts now reach it once it does. The orphan cleanup kills each process whose environment names the job in RUNNER_TRACKING_ID, so it could kill the supervisor right after the harness, before the supervisor ended a tree that the cleanup didn't list, such as an OpenVMM that started after the listing. The supervisor therefore runs without that ID and gives it back to its command, which the cleanup still tracks. The kernel sends a parent-death signal when the thread that started the process exits, not only the whole process, so both apply only to a process that the main thread starts, as nvx.py starts its VMs; other threads get the command lines of before. Tests check which commands get the wrapper and the supervisor its parent and environment, and on Linux, that killing an owner process outright ends the command that InteractiveProcess started directly and, with containment, the supervisor, the command, and a descendant in a session of its own; that killing the processes that an orphan cleanup listed ends a descendant that started after the listing; that neither a command nor a supervisor whose owner already exited starts; and that a supervisor forwards SIGTERM while its parent lives. doc/ci.md, doc/benchmarks.md, and doc/usage.md describe the behavior and its cost. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
7781277 to
efda0a8
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A critical startup race allows cancellation before the harness PID is recorded to exit without signalling or waiting for the harness, leaving it running and risking skipped interruption cleanup.
1 open issue
2 resolved since last review
Review effort: Balanced
When a job is cancelled, the runner sends SIGINT, and 7.5 s later SIGTERM, to each running step's shell alone, and Bash runs a trap only once the command in its foreground ends. The test harness, which the Linux steps ran in the foreground, never learned of the cancellation: it ran on until the runner's orphan cleanup killed it, and the scratch directories that its scenario had created stayed in /tmp. The ten Linux runners held 64 of them on 2026-10-28, and each of the 18 left since #779 was last modified during a cancelled pull-request job on its runner (#946). The Linux steps that run test-microvm now exec it, so that the harness is the step's process and gets the runner's signals. Three Linux steps of run-benchmark run the harness in a loop with the commands that validate and copy each attempt, so they source the new forward-signals.sh: its run_harness runs the harness in the background and passes SIGINT and SIGTERM on to it, and once the harness exits, it ends the step with the status of the signal, even when the signal came too late to stop the harness, as when it arrived as the harness exited. A signal that arrives between two harness runs ends the step at once instead of letting it start the next one. The device operation rates step execs its harness, its last command. With the previous commits, the harness then stops OpenVMM and removes its scratch directories before it exits. A background command of a non-interactive Bash starts with SIGINT ignored, which Python then leaves ignored, so run_harness restores it before it execs the harness. A SIGINT that reaches the harness before then is lost, as one passed on right after run_harness started the harness would be, so run_harness passes each signal on as SIGTERM, which the harness never ignores and which nvx.py handles as SIGINT. A signal that arrives once run_harness has started the harness, but before it has the harness's PID, is passed on once it has the PID, instead of ending the step and leaving the harness to run on. Replaying the runner's sequence against a stand-in harness on Linux, a foreground harness still ran 7.5 s after SIGINT, while an exec'd or forwarded one cleaned up and exited with status 130 within about a second of SIGINT or SIGTERM. A shell that signalled itself right after run_harness started a harness ended at once and left the harness running until run_harness deferred the signal; passed on as SIGINT there, the signal was lost in 5 of 20 runs, and passed on as SIGTERM, it stopped the harness in all 20. check-quality lints and formats forward-signals.sh with the other Bash files. Tests check that every Linux step that runs the harness execs it or runs it with run_harness after it sources forward-signals.sh, and on Linux, that SIGINT or SIGTERM sent to the step's shell alone reaches the harness, which cleans up, and ends the step with the signal's status; that a cancellation that doesn't stop the harness still ends the step once the harness exits; that run_harness otherwise returns the harness's status; that a SIGINT between two harness runs ends the step before the second one; and that a SIGINT or SIGTERM that a DEBUG trap sends the shell right after run_harness starts the harness stops the harness before the step ends. The tests that run the benchmark steps against a stubbed harness, which an exec would bypass, define a foreground run_harness instead. doc/ci.md describes the cancellation path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A test harness that something kills outright, as the runner's orphan cleanup killed those of cancelled jobs, leaves the scratch directories of its scenarios in the temporary directory, where nothing removed them. On 2026-10-28, the ten Linux runners held 64 /tmp/nvx-* and /tmp/openvmm-* directories, the oldest from 2026-10-12, when the runners booted (#946). The previous commits let a cancellation reach the harness, whose cleanup then removes them, but a harness that is killed outright still leaves them, and the ones already there stay. validate-runner now removes, after it has ended the OpenVMM processes that earlier jobs left, the account's nvx-* and openvmm-* directories in Python's temporary directory, where the harness creates them, with a notice that names each one and when it last changed. A runner runs one job at a time, and no step before validate-runner runs the harness, so such a directory is left from an earlier job. The step first makes each directory writable to its owner, since a scenario can leave one that even its owner can't empty, and warns about one that it still can't remove, but fails if it can't list them. It considers only directories, so neither a file nor a symbolic link that matches reaches rm. A test runs the step against a temporary directory of its own on Linux: it removes leftover harness directories, a read-only one included, keeps other directories, files, and links and their targets, says nothing without leftovers, warns about a directory that rm doesn't remove, and fails when find does. It also checks that every scratch directory that the harness creates in the temporary directory has a prefix that the step removes. doc/ci.md describes the step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
efda0a8 to
8b3dbca
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer validation is needed to confirm that the process-cleanup ownership changes preserve cancellation and complete VM-tree reaping on the self-hosted Azure KVM/MSHV runners.
No open issues 🎉
1 resolved since last review
Review effort: Balanced


When a CI job is cancelled, the test harness now learns of it, stops its OpenVMM processes, and removes its scratch directories. A harness that is killed outright takes its VMs with it, and the runner's next job removes the scratch directories that are still left. Fixes #946.
Root cause
The runner cancels a step by sending SIGINT, and 7.5 s later SIGTERM, to the step's shell alone. Bash runs a trap only once its foreground command ends, so the harness, which every Linux step ran in the foreground, never learned of the cancellation. It ran on until the runner's orphan cleanup killed it, together with the job's processes that the cleanup had listed, and none of its cleanup ran: the scratch directories of its scenarios stayed in
/tmp, 64 of them on the ten Linux runners on October 28, and a VM that it started after the listing ran for 8 days (#768).Changes
scripts: let SIGTERM interrupt nvx.py as SIGINT does:nvx.pyhandles SIGTERM withsignal.default_int_handlerwhile it runs a command, so either signal unwinds the command through its cleanup, which then printsInterruptedand exits with status 130. As Python does for SIGINT, an inherited ignored SIGTERM stays ignored.sandbox: stop an interrupted acceptance's sandboxes:run_acceptance, which runssandbox-lifecycleandsandbox-proxy-only, cleaned up only onException. It now stops and deprovisions the sandboxes on any exception, and an interruption still propagates as one.microvm: build the time ABI monitor before OpenVMM starts: as Track nvx-rs host VMM supervisor bugs: a non-UTF-8 process name at a reused PID blocks reconcile, reattach, and stop #720 noted,OpenvmmProcessbuilt the monitor outside its cleanup.scripts: end OpenVMM with the harness that started it on Linux:InteractiveProcessstarts a command that it doesn't contain through util-linux'ssetpriv --pdeathsig KILL, which the Azure Linux 3 runners have in util-linux 2.40.2. A shell then starts the command only if the harness is still its parent, because a harness that died beforesetprivarmed the signal sent none.nvx-rs runand the OpenVMM that it starts, now gets the PID of its owner. It makes SIGTERM its parent-death signal, checks that its owner still runs, and kills its whole subreaper tree once the owner exits.RUNNER_TRACKING_IDand gives it back to its command. The orphan cleanup kills the processes whose environment holds that ID, so it could otherwise kill the supervisor right after the harness, before the supervisor ended a tree whose OpenVMM started after the cleanup's listing.preexec_fncallingprctlwould avoid the extra execs, but it makes CPython fork the whole harness instead of using vfork. On prometheus32 (Xeon Silver 4114),Popenthen took 10.3 ms at 75 MiB of RSS and 37.7 ms at 265 MiB, against 0.64 ms.setprivand the shell add 2.1 ms to the time until the command runs with Dash as/bin/sh, and 3.5 ms with Bash, which the runners use.openvmm_snapshot_restorespans 51.7 to 63.8 ms, and the gate needs a regression of more than 40% and 5 ms.doc/benchmarks.mdstates the cost.ci: let a cancellation reach the test harness:test-microvmstepsexecthe harness.run-benchmark's looping Linux steps source the newforward-signals.sh. Itsrun_harnessruns the harness in the background, with the SIGINT that Bash ignores there restored, and passes SIGINT and SIGTERM on to it as SIGTERM, whichnvx.pyhandles as SIGINT and which, unlike SIGINT, the harness never ignores, not even before it has restored SIGINT. A signal that arrives asrun_harnessstarts the harness, before it has the harness's PID, reaches the harness once it has it. Once the harness exits, the step ends with the signal's status, even if the signal came too late to stop the harness. A signal that arrives between two harness runs ends the step at once.execs its harness.ci: remove the scratch directories that earlier jobs left: after it ends leftover OpenVMM processes,validate-runnerremoves the account'snvx-*andopenvmm-*directories in Python's temporary directory, with a notice for each one. A test checks that every scratch directory that the harness creates has one of those prefixes.Validation
execandrun_harness: it cleaned up and exited with 130 within about 1 s of SIGINT or SIGTERM.run_harnessstarts the harness, from a functraceDEBUGtrap, on prometheus32 (test_run_harness_passes_on_a_cancellation_that_arrives_as_it_starts)snapshot-coreon KVM on prometheus32, with the artifacts ofv0.1.0-dev.6843c5847198(this branch's base), run as the CI step does (bash -c 'exec python3 scripts/nvx.py test-microvm …'), signalled while OpenVMM ran in its scratch directory/tmp/nvx-snapshot-core-*removed. SIGKILL: OpenVMM gone within 0.02 s, directory left forvalidate-runner.l3-l4-egress-protocols(supervisor →nvx-rs run→ OpenVMM), signalled the same wayorigin/dev: OpenVMM died by SIGABRT 1.21 s later, after its console pty closed; the #768 VM shows that this doesn't always happen. This branch, with the shell's parent check: SIGKILL, its parent-death signal, after 0.04 s.setprivarmed the signal (test_a_command_whose_owner_exited_never_starts)RUNNER_TRACKING_ID, let the contained command start a descendant, then SIGKILL the listed processes in order (test_the_orphan_cleanup_leaves_a_contained_tree_to_its_supervisor)validate-runner's new step on the Azure runners, in this pull request's CI runs: 55 Linux jobs, on each of the ten runners/tmp/nvx-managed-lifecycle-i5cq39cjonazure-mshv-2and/tmp/nvx-smp-snapshot-m1xbd214onazure-mshv-3. Both were last modified at 23:09:54Z, as #977's MSHV microVM shards were cancelled on those runners (run 40784650296, whose head lacks this pull request): the failure that this pull request fixes. The step found nothing else; the 64 directories that #946 counted were gone before this pull request's first run.validate-nvxunit tests on Linux (prometheus32, Python 3.14)test_linux_runners_terminate_openvmm_left_by_earlier_jobs, which fails the same way onorigin/devon that host.test_setup_azurewasn't run: the host has no Azure SDK.test_adversarial.pyandtest_hosts.pyon Linuxvalidate-nvxunit tests on Windowstest_setup_azurefailures that an unmodified checkout also has on Windows, plus, on a loaded machine, 10 s timeouts of Git'ssh.exein twotest_microvm_testsnetwork-policy tests that pass when run alone.ruff check,ruff format --check,pyrightfor Linux and Windows, Markdown linksNot run locally: this pull request's CI matrix, which runs the changed steps on the KVM, MSHV, and WHP runners. Not yet seen on the Azure runners: a cancellation of this pull request's own harness, since none of its runs was cancelled while a Linux harness ran.
Not in this pull request
PhysicalHostName, which is independent of the harness.test-openvmmandtest-adversarialsteps, which run other harnesses, keep their foreground form.