Skip to content

ci: let a cancelled job's test harness stop its VMs and remove its scratch - #975

Merged
Pedro Henrique Penna (ppenna) merged 6 commits into
devfrom
agents/issue-946-harness-cancellation
Oct 29, 2026
Merged

Pedro Henrique Penna (ppenna) merged 6 commits into
devfrom
agents/issue-946-harness-cancellation

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

When a CI job is cancelled, the test harness now learns of it, stops its OpenVMM processes, and removes its scratch directories. A harness that is killed outright takes its VMs with it, and the runner's next job removes the scratch directories that are still left. Fixes #946.

Root cause

The runner cancels a step by sending SIGINT, and 7.5 s later SIGTERM, to the step's shell alone. Bash runs a trap only once its foreground command ends, so the harness, which every Linux step ran in the foreground, never learned of the cancellation. It ran on until the runner's orphan cleanup killed it, together with the job's processes that the cleanup had listed, and none of its cleanup ran: the scratch directories of its scenarios stayed in /tmp, 64 of them on the ten Linux runners on October 28, and a VM that it started after the listing ran for 8 days (#768).

Changes

  1. scripts: let SIGTERM interrupt nvx.py as SIGINT does: nvx.py handles SIGTERM with signal.default_int_handler while it runs a command, so either signal unwinds the command through its cleanup, which then prints Interrupted and exits with status 130. As Python does for SIGINT, an inherited ignored SIGTERM stays ignored.
  2. sandbox: stop an interrupted acceptance's sandboxes: run_acceptance, which runs sandbox-lifecycle and sandbox-proxy-only, cleaned up only on Exception. It now stops and deprovisions the sandboxes on any exception, and an interruption still propagates as one.
  3. microvm: build the time ABI monitor before OpenVMM starts: as Track nvx-rs host VMM supervisor bugs: a non-UTF-8 process name at a reused PID blocks reconcile, reattach, and stop #720 noted, OpenvmmProcess built the monitor outside its cleanup.
  4. scripts: end OpenVMM with the harness that started it on Linux:
    • InteractiveProcess starts a command that it doesn't contain through util-linux's setpriv --pdeathsig KILL, which the Azure Linux 3 runners have in util-linux 2.40.2. A shell then starts the command only if the harness is still its parent, because a harness that died before setpriv armed the signal sent none.
    • The containment supervisor, which runs nvx-rs run and the OpenVMM that it starts, now gets the PID of its owner. It makes SIGTERM its parent-death signal, checks that its owner still runs, and kills its whole subreaper tree once the owner exits.
    • The supervisor runs without the job's RUNNER_TRACKING_ID and gives it back to its command. The orphan cleanup kills the processes whose environment holds that ID, so it could otherwise kill the supervisor right after the harness, before the supervisor ended a tree whose OpenVMM started after the cleanup's listing.
    • Both apply only to processes that the main thread starts, because the kernel sends the signal when the starting thread exits.
    • A preexec_fn calling prctl would avoid the extra execs, but it makes CPython fork the whole harness instead of using vfork. On prometheus32 (Xeon Silver 4114), Popen then took 10.3 ms at 75 MiB of RSS and 37.7 ms at 265 MiB, against 0.64 ms. setpriv and the shell add 2.1 ms to the time until the command runs with Dash as /bin/sh, and 3.5 ms with Bash, which the runners use.
    • The benchmark's launch timings include those milliseconds. Over the last 20 commits, the KVM history of openvmm_snapshot_restore spans 51.7 to 63.8 ms, and the gate needs a regression of more than 40% and 5 ms. doc/benchmarks.md states the cost.
  5. ci: let a cancellation reach the test harness:
    • The Linux test-microvm steps exec the harness.
    • run-benchmark's looping Linux steps source the new forward-signals.sh. Its run_harness runs the harness in the background, with the SIGINT that Bash ignores there restored, and passes SIGINT and SIGTERM on to it as SIGTERM, which nvx.py handles as SIGINT and which, unlike SIGINT, the harness never ignores, not even before it has restored SIGINT. A signal that arrives as run_harness starts the harness, before it has the harness's PID, reaches the harness once it has it. Once the harness exits, the step ends with the signal's status, even if the signal came too late to stop the harness. A signal that arrives between two harness runs ends the step at once.
    • The device operation rates step execs its harness.
    • check-quality lints the new file.
  6. ci: remove the scratch directories that earlier jobs left: after it ends leftover OpenVMM processes, validate-runner removes the account's nvx-* and openvmm-* directories in Python's temporary directory, with a notice for each one. A test checks that every scratch directory that the harness creates has one of those prefixes.

Validation

Check Result
Runner's cancellation replayed against a stand-in harness on prometheus32 (bash 5.3, Linux 7.0) Foreground, as before: the harness still ran 7.5 s after SIGINT. exec and run_harness: it cleaned up and exited with 130 within about 1 s of SIGINT or SIGTERM.
A shell that signals itself right after run_harness starts the harness, from a functrace DEBUG trap, on prometheus32 (test_run_harness_passes_on_a_cancellation_that_arrives_as_it_starts) Without the deferral, the step ended at once and the harness ran on. With the deferral, a SIGINT passed on to the harness was lost in 5 of 20 runs, since the harness hadn't restored SIGINT yet; passed on as SIGTERM, the signal stopped the harness in all 20.
snapshot-core on KVM on prometheus32, with the artifacts of v0.1.0-dev.6843c5847198 (this branch's base), run as the CI step does (bash -c 'exec python3 scripts/nvx.py test-microvm …'), signalled while OpenVMM ran in its scratch directory SIGINT and SIGTERM: exit 130 within 0.11 and 0.16 s, OpenVMM gone, /tmp/nvx-snapshot-core-* removed. SIGKILL: OpenVMM gone within 0.02 s, directory left for validate-runner.
l3-l4-egress-protocols (supervisor → nvx-rs run → OpenVMM), signalled the same way SIGKILL: the whole tree gone within 1.27 s. SIGINT: exit 130 within 0.16 s, tree gone.
An orphaned OpenVMM: idle Alpine guest, harness killed with SIGKILL, OpenVMM reaped by the test as subreaper origin/dev: OpenVMM died by SIGABRT 1.21 s later, after its console pty closed; the #768 VM shows that this doesn't always happen. This branch, with the shell's parent check: SIGKILL, its parent-death signal, after 0.04 s.
A wrapped command started by a process other than its owner, as after an owner that died before setpriv armed the signal (test_a_command_whose_owner_exited_never_starts) Never starts; its owner starts it.
The orphan cleanup replayed on prometheus32: list the processes that hold the job's RUNNER_TRACKING_ID, let the contained command start a descendant, then SIGKILL the listed processes in order (test_the_orphan_cleanup_leaves_a_contained_tree_to_its_supervisor) The supervisor, which the cleanup doesn't list, ends the late descendant. With the supervisor still holding the ID, the test fails: "outlived the cleanup".
validate-runner's new step on the Azure runners, in this pull request's CI runs: 55 Linux jobs, on each of the ten runners In run 40787837720, it removed /tmp/nvx-managed-lifecycle-i5cq39cj on azure-mshv-2 and /tmp/nvx-smp-snapshot-m1xbd214 on azure-mshv-3. Both were last modified at 23:09:54Z, as #977's MSHV microVM shards were cancelled on those runners (run 40784650296, whose head lacks this pull request): the failure that this pull request fixes. The step found nothing else; the 64 directories that #946 counted were gone before this pull request's first run.
validate-nvx unit tests on Linux (prometheus32, Python 3.14) 1,321 tests: all pass except test_linux_runners_terminate_openvmm_left_by_earlier_jobs, which fails the same way on origin/dev on that host. test_setup_azure wasn't run: the host has no Azure SDK.
test_adversarial.py and test_hosts.py on Linux Pass (84 and 4 tests).
validate-nvx unit tests on Windows 1,371 tests: only the 18 test_setup_azure failures that an unmodified checkout also has on Windows, plus, on a loaded machine, 10 s timeouts of Git's sh.exe in two test_microvm_tests network-policy tests that pass when run alone.
Pinned shellcheck 0.11.0 and shfmt on the Bash files Pass.
ruff check, ruff format --check, pyright for Linux and Windows, Markdown links Pass.

Not run locally: this pull request's CI matrix, which runs the changed steps on the KVM, MSHV, and WHP runners. Not yet seen on the Azure runners: a cancellation of this pull request's own harness, since none of its runs was cancelled while a Linux harness ran.

Not in this pull request

A cancelled CI step gets SIGINT and, if it still runs 7.5 s later,
SIGTERM, and the runner kills its process tree 2.5 s after that. nvx.py
turns SIGINT into KeyboardInterrupt, which unwinds the command through
the context managers that stop its OpenVMM processes and remove its
scratch directories. On SIGTERM, though, Python exits at once and runs
none of them, so a harness whose cleanup outlasts the SIGINT window, or
that something else ends with SIGTERM, such as timeout(1), leaves them
behind (#946).

main() now runs the command with SIGTERM handled as Python handles
SIGINT, by signal.default_int_handler, so either signal interrupts the
command, which then prints Interrupted and exits with status 130. As
Python does for SIGINT, it leaves an inherited ignored SIGTERM ignored.
It installs the handler only in the main thread of a POSIX process, and
restores the default disposition once the command returns, so that
in-process callers such as the tests keep their own.

A CliTests test sends its own process SIGTERM from a command and checks
that the command unwinds, that main() reports the interruption and
restores the disposition, and that an ignored SIGTERM stays ignored.
doc/usage.md documents the behavior, and a new Cancellation section of
doc/ci.md describes how the runner cancels a step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
run_acceptance, which runs the sandbox-lifecycle and sandbox-proxy-only
scenarios, stops and deprovisions the sandboxes of an acceptance that
fails, but only on an Exception. Ctrl-C raises KeyboardInterrupt, and so
will a cancelled CI job once its SIGINT reaches the harness (#946), and
that skipped the cleanup: the sandbox's OpenVMM, which nvx-rs sandbox
start detaches from the harness, kept running, and the fixture stayed
in the temporary directory.

Any exception now cleans up the acceptance. A failure still raises a
RuntimeError that names the cleanup errors, among them a fixture that
the cleanup preserved for recovery. An interruption propagates as it
is, so that nvx.py reports it as one and the scenario duration report
records it as interrupted, after a warning that names those errors.

A test interrupts the fake sandbox CLI during the security probe and
checks that the cleanup stops and deprovisions the running sandbox,
removes the fixture, keeps the checks' evidence, and propagates the
KeyboardInterrupt, and that a failed deprovision preserves the fixture
with a warning that names it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OpenvmmProcess built the TimeAbiMonitor that scans OpenVMM's console
only after it had started OpenVMM and its output reader, outside the
try blocks that stop OpenVMM when the constructor fails (#720). The
monitor only parses the command line, so it cannot fail on a valid
command, but an error there would have left OpenVMM running with
nothing to stop it (#946).

The constructor now builds the monitor first, before it opens the
transcript or starts OpenVMM, so that a failure there starts nothing. A
test makes the monitor fail and checks that OpenVMM never starts and
that no transcript is created.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical startup race can leave OpenVMM running if the harness dies before the child arms its parent-death signal; an owner-PID check after arming and regression coverage are needed.

1 open issue
What changed in this PR

This PR addresses #946 by improving cancellation handling in NVX’s Linux test harnesses, allowing cleanup to stop VMs and remove scratch directories.

Changes:

  • Forward CI cancellation signals to the harness and run cleanup on interruption.
  • Add parent-death handling for launched processes and initialize the time ABI monitor before starting OpenVMM.
  • Remove leftover scratch directories during runner validation, with regression tests and documentation.
File Description
scripts/​test_nvx_tools.py Tests signal handling, forwarding, and scratch cleanup.
scripts/​test_microvm_tests.py Tests interrupted sandbox cleanup and monitor initialization failure.
scripts/​test_adversarial.py Tests parent-death handling and supervisor behavior.
scripts/​nvx.py Handles SIGTERM as an interruption on POSIX.
scripts/​nvx_tools/​sandbox_lifecycle_tests.py Cleans up sandboxes when acceptance is interrupted.
scripts/​nvx_tools/​process_supervisor.py Accepts the expected owner PID.
scripts/​nvx_tools/​openvmm_process.py Initializes the monitor before launching OpenVMM.
scripts/​nvx_tools/​benchmark.py Applies parent-death handling to direct launches.
scripts/​nvx_tools/​adversarial_oracles.py Adds parent-death wrappers and supervisor cleanup.
doc/​usage.md Documents interruption and process cleanup behavior.
doc/​ci.md Explains cancellation handling and leftover cleanup.
doc/​benchmarks.md Documents the added Linux launch overhead.
.github/​workflows/​run-nvx-microvm-tests.yml Executes Linux harnesses directly with exec.
.github/​actions/​validate-runner/​action.yml Removes scratch directories left by earlier jobs.
.github/​actions/​run-benchmark/​forward-signals.sh Forwards cancellation signals to background harnesses.
.github/​actions/​run-benchmark/​action.yml Uses signal forwarding or exec for Linux benchmarks.
.github/​actions/​check-quality/​action.yml Adds shell checks for the signal-forwarding helper.

Review effort: Balanced

Comment thread scripts/nvx_tools/adversarial_oracles.py Outdated
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-946-harness-cancellation branch from 37c91b4 to 7781277 Compare October 28, 2026 22:36
Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cancellation can be lost during harness teardown, and runner cleanup can kill the supervisor while leaving OpenVMM running without regression coverage for that scenario.

2 open issues
1 resolved since last review

Review effort: Balanced

Comment thread scripts/nvx_tools/adversarial_oracles.py
Comment thread .github/actions/run-benchmark/forward-signals.sh Outdated
When a CI job is cancelled, the runner's orphan cleanup kills the test
harness outright, together with the job's processes that it listed.
OpenVMM ran as a plain child of the harness, so one that the harness
started after the listing kept running: a VM ran on azure-mshv-1 for 8
days (#768). A contained process tree, such as nvx-rs run and the
OpenVMM that it starts, ran under a supervisor that had no parent-death
signal either (#946).

On Linux, InteractiveProcess now starts a command that it doesn't
contain through util-linux's setpriv --pdeathsig KILL, which makes
SIGKILL the command's parent-death signal (PR_SET_PDEATHSIG), and then
through a shell that starts the command only if the harness is still
its parent, since a harness that died before setpriv armed the signal
sent none. The signal survives both execs, so the kernel kills OpenVMM
when the harness exits. A preexec_fn that called prctl would avoid the
extra execs, but it makes CPython fork the whole harness instead of
using vfork: on a Xeon Silver 4114 host, it made Popen take 10.3 ms in
a 75 MiB process and 37.7 ms in a 265 MiB one, against 0.64 ms, while
setpriv and the shell added 2.1 ms to the time until the command ran
with Dash as /bin/sh, and 3.5 ms with Bash. The benchmark's launch
timings include them; on the KVM runners, the p50 of
openvmm_snapshot_restore spans 51.7 to 63.8 ms over the last 20
commits of its history. A host without setpriv starts the command as
before; the Azure Linux 3 runners have it in util-linux 2.40.2.

ProcessTreeContainment now passes the supervisor the ID of the process
that started it. The supervisor makes SIGTERM its parent-death signal,
checks once that its parent still runs, since a parent that exited
before then sends no signal, and only then starts the command. A
signal that finds the supervisor with another parent kills the whole
tree, descendants that left its session included, and the supervisor
then ends by SIGTERM. Signals that arrive while its parent lives reach
the command as before, and those that arrive before the command starts
now reach it once it does. The orphan cleanup kills each process whose
environment names the job in RUNNER_TRACKING_ID, so it could kill the
supervisor right after the harness, before the supervisor ended a tree
that the cleanup didn't list, such as an OpenVMM that started after the
listing. The supervisor therefore runs without that ID and gives it
back to its command, which the cleanup still tracks.

The kernel sends a parent-death signal when the thread that started the
process exits, not only the whole process, so both apply only to a
process that the main thread starts, as nvx.py starts its VMs; other
threads get the command lines of before.

Tests check which commands get the wrapper and the supervisor its
parent and environment, and on Linux, that killing an owner process
outright ends the command that InteractiveProcess started directly
and, with containment, the supervisor, the command, and a descendant in
a session of its own; that killing the processes that an orphan cleanup
listed ends a descendant that started after the listing; that neither a
command nor a supervisor whose owner already exited starts; and that a
supervisor forwards SIGTERM while its parent lives. doc/ci.md,
doc/benchmarks.md, and doc/usage.md describe the behavior and its cost.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-946-harness-cancellation branch from 7781277 to efda0a8 Compare October 28, 2026 22:58
Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical startup race allows cancellation before the harness PID is recorded to exit without signalling or waiting for the harness, leaving it running and risking skipped interruption cleanup.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread .github/actions/run-benchmark/forward-signals.sh
When a job is cancelled, the runner sends SIGINT, and 7.5 s later
SIGTERM, to each running step's shell alone, and Bash runs a trap only
once the command in its foreground ends. The test harness, which the
Linux steps ran in the foreground, never learned of the cancellation:
it ran on until the runner's orphan cleanup killed it, and the scratch
directories that its scenario had created stayed in /tmp. The ten Linux
runners held 64 of them on 2026-10-28, and each of the 18 left since
#779 was last modified during a cancelled pull-request job on its
runner (#946).

The Linux steps that run test-microvm now exec it, so that the harness
is the step's process and gets the runner's signals. Three Linux steps
of run-benchmark run the harness in a loop with the commands that
validate and copy each attempt, so they source the new
forward-signals.sh: its run_harness runs the harness in the background
and passes SIGINT and SIGTERM on to it, and once the harness exits, it
ends the step with the status of the signal, even when the signal came
too late to stop the harness, as when it arrived as the harness exited.
A signal that arrives between two harness runs ends the step at once
instead of letting it start the next one. The device operation rates
step execs its harness, its last command. With the previous commits,
the harness then stops OpenVMM and removes its scratch directories
before it exits.

A background command of a non-interactive Bash starts with SIGINT
ignored, which Python then leaves ignored, so run_harness restores it
before it execs the harness. A SIGINT that reaches the harness before
then is lost, as one passed on right after run_harness started the
harness would be, so run_harness passes each signal on as SIGTERM,
which the harness never ignores and which nvx.py handles as SIGINT. A
signal that arrives once run_harness has started the harness, but
before it has the harness's PID, is passed on once it has the PID,
instead of ending the step and leaving the harness to run on.

Replaying the runner's sequence against a stand-in harness on Linux, a
foreground harness still ran 7.5 s after SIGINT, while an exec'd or
forwarded one cleaned up and exited with status 130 within about a
second of SIGINT or SIGTERM. A shell that signalled itself right after
run_harness started a harness ended at once and left the harness
running until run_harness deferred the signal; passed on as SIGINT
there, the signal was lost in 5 of 20 runs, and passed on as SIGTERM,
it stopped the harness in all 20.

check-quality lints and formats forward-signals.sh with the other Bash
files. Tests check that every Linux step that runs the harness execs it
or runs it with run_harness after it sources forward-signals.sh, and on
Linux, that SIGINT or SIGTERM sent to the step's shell alone reaches
the harness, which cleans up, and ends the step with the signal's
status; that a cancellation that doesn't stop the harness still ends
the step once the harness exits; that run_harness otherwise returns the
harness's status; that a SIGINT between two harness runs ends the step
before the second one; and that a SIGINT or SIGTERM that a DEBUG trap
sends the shell right after run_harness starts the harness stops the
harness before the step ends. The tests that run the benchmark steps
against a stubbed harness, which an exec would bypass, define a
foreground run_harness instead. doc/ci.md describes the cancellation
path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A test harness that something kills outright, as the runner's orphan
cleanup killed those of cancelled jobs, leaves the scratch directories
of its scenarios in the temporary directory, where nothing removed
them. On 2026-10-28, the ten Linux runners held 64 /tmp/nvx-* and
/tmp/openvmm-* directories, the oldest from 2026-10-12, when the
runners booted (#946). The previous commits let a cancellation reach
the harness, whose cleanup then removes them, but a harness that is
killed outright still leaves them, and the ones already there stay.

validate-runner now removes, after it has ended the OpenVMM processes
that earlier jobs left, the account's nvx-* and openvmm-* directories
in Python's temporary directory, where the harness creates them, with
a notice that names each one and when it last changed. A runner runs
one job at a time, and no step before validate-runner runs the
harness, so such a directory is left from an earlier job. The step
first makes each directory writable to its owner, since a scenario can
leave one that even its owner can't empty, and warns about one that it
still can't remove, but fails if it can't list them. It considers only
directories, so neither a file nor a symbolic link that matches reaches
rm.

A test runs the step against a temporary directory of its own on
Linux: it removes leftover harness directories, a read-only one
included, keeps other directories, files, and links and their targets,
says nothing without leftovers, warns about a directory that rm
doesn't remove, and fails when find does. It also checks that every
scratch directory that the harness creates in the temporary directory
has a prefix that the step removes. doc/ci.md describes the step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-946-harness-cancellation branch from efda0a8 to 8b3dbca Compare October 28, 2026 23:30
Copilot AI balanced review requested due to automatic review settings October 28, 2026 23:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer validation is needed to confirm that the process-cleanup ownership changes preserve cancellation and complete VM-tree reaping on the self-hosted Azure KVM/MSHV runners.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) merged commit 213cbaf into dev Oct 29, 2026
85 of 87 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-946-harness-cancellation branch October 29, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: a cancelled job leaves its scratch directories behind and can leave OpenVMM running

2 participants