Repository navigation
scripts: keep concurrent egress script tests from deleting each other's probe files - #1021
Merged
Pedro Henrique Penna (ppenna) merged 2 commits intoOct 30, 2026
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the changes isolating temporary probe files in concurrent host-side egress tests.
No open issues 🎉
What changed in this PR
Fixes #1007 by isolating egress probe files for concurrent host-side tests, while preserving the microVM scripts’ default use of the guest’s private /tmp.
Changes:
- Route probe cleanup, writes, and reads through
${TMPDIR:-/tmp}in all three egress scripts. - Give each host-side script run its own temporary directory.
- Add assertions that stale results are removed and exactly the expected probe files remain.
| File | Description |
|---|---|
scripts/test_microvm_tests.py |
Isolates each run’s TMPDIR and verifies cleanup and expected result files. |
scripts/nvx_tools/microvm_test_scripts/l3-l4-egress-protocols.sh.in |
Uses TMPDIR for protocol probe results. |
scripts/nvx_tools/microvm_test_scripts/l3-l4-egress-port-ranges.sh.in |
Uses TMPDIR for port-range probe results. |
scripts/nvx_tools/microvm_test_scripts/ipv6-egress-policy.sh.in |
Uses TMPDIR for IPv6 policy probe results. |
Review effort: Balanced
l3-l4-egress-protocols.sh.in, l3-l4-egress-port-ranges.sh.in, and ipv6-egress-policy.sh.in start their probes in the background, and each probe writes its result to a fixed path, /tmp/nvx-egress-PROBE, which the script reads once they all end. Before it probes, the script removes every such file with rm -f /tmp/nvx-egress-*. A guest's /tmp is private, but scripts/test_microvm_tests.py also runs these scripts in the host's shell, where /tmp is shared, so that runs at once on one host remove and overwrite each other's results (#1007). Each script now keeps its results in TMPDIR, and in /tmp when TMPDIR is unset or empty. The runners stage the scripts with a quoted here-document, so the guest's shell expands TMPDIR only when it runs a script, and nothing in NVX's guests sets it: the scenarios keep using the guest's /tmp, while a test can give each run of a script a directory of its own. The three scenarios pass with this change on KVM, MSHV, and WHP, on test VMs and on bare metal, and their guests report every probe as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Several validate-nvx runs at once on one host failed the tests that run the egress scripts in the host's shell, with 70 != 0 and with "cat: /tmp/nvx-egress-tcp-inside: No such file or directory", on Linux and on Windows, where Git for Windows' sh maps /tmp to the user's temporary directory (#1007): one run's rm -f /tmp/nvx-egress-* removed the results of another between its probes and its report. On Windows, removing a result that another run holds open can also fail, and the script then exits with "rm: cannot remove '/tmp/nvx-egress-icmp': Permission denied". The scripts now keep their results in TMPDIR, which _probe_environment sets to a directory of each run's own: a temporary directory for the L3/L4 scripts, through the new _run_egress_script, and for ipv6-egress-policy the tmp of the fake guest root in which _run_ipv6_guest_script already gives the script its /proc and resolver. Each test also leaves a stale result in that directory and requires that the run replace it with exactly its probes' results, so a probe or a cleanup that went back to a shared /tmp fails the test even when it runs alone, as the scripts did before the previous commit. Before this change, every one of eight processes that each ran the three tests 15 times at once failed on Linux KVM and MSHV hosts, and every one of six processes that each ran them 10 times failed on Windows hosts. Four whole runs of scripts/test_microvm_tests.py at once failed the egress tests in most runs on the Linux hosts, and in two of three runs on a Windows host. With this change, the egress tests passed in every one of these runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/issue-1007-debug-test-fix-commit
branch
from
October 29, 2026 21:59
c7eb86d to
a79ca92
Compare
Pedro Henrique Penna (ppenna)
assigned
Huzaifa Danish (huzaifa-d) and unassigned
Huzaifa Danish (huzaifa-d)
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1007-debug-test-fix-commit
branch
October 30, 2026 02:04
CaffeeLake (CaffeeLake)
pushed a commit
to CaffeeLake/nvx
that referenced
this pull request
Oct 30, 2026
microsoft#952 changed how the summarize job of nvx-rs-mutants.yml finds the outcomes of the shards. Each shard now uploads its outcomes under its index, the download merges the artifacts of all the shards into shards/, and the merge counts a shard as reported when shards/INDEX/outcomes.json exists. The merge step also fails before it merges anything when the mutants job did not succeed, which it reads from MUTANTS_RESULT under set -u. MutantsWorkflowStepTests still ran the merge step with each shard's lists in a directory named after its artifact, and without MUTANTS_RESULT. On Linux, where these tests run, test_the_outcomes_of_every_shard_merge failed with "MUTANTS_RESULT: unbound variable", and test_shards_that_report_nothing_fail_the_merge did not find its "Only 1 of 3 shards reported" error. Both fail at b558d34, which merged microsoft#952, and pass at its parent, 29c6854; they failed NVX CLI / Linux in CI run 40958853538 of microsoft#1021. The tests now lay the outcomes out as the download does, under the index of each shard and with outcomes.json, and pass the mutants job's result. The test of missing shards also gives one shard lists without outcomes.json, which the merge must leave out of both the lists and the count. A new test requires that a mutants job that failed, was cancelled, or was skipped fail the merge before it merges or summarizes anything. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1007
l3-l4-egress-protocols.sh.in,l3-l4-egress-port-ranges.sh.in, andipv6-egress-policy.sh.inwrote each probe's result to a fixed path,/tmp/nvx-egress-<probe>, after they removed every such file withrm -f /tmp/nvx-egress-*. That is harmless in a guest, whose/tmpis private. Butscripts/test_microvm_tests.pyalso runs these scripts in the host's POSIX shell, where/tmpis shared, so concurrent runs on one host removed and overwrote each other's results. The tests then failed with70 != 0andcat: /tmp/nvx-egress-tcp-inside: No such file or directory. On Windows, where Git for Windows'shmaps/tmpto the user's temporary directory, a run can also fail to remove a result that another run holds open, and the script exits withrm: cannot remove '/tmp/nvx-egress-icmp': Permission denied.Changes
scripts: keep the egress probes' results in TMPDIR: each script keeps its results in${TMPDIR:-/tmp}. The runners stage the scripts with a quoted here-document, so the guest's shell expandsTMPDIRonly when it runs a script, and NVX's guests never set it. The scenarios therefore keep using the guest's/tmp.scripts: give each host run of an egress script a TMPDIR of its own:_probe_environmentsetsTMPDIRto a directory that belongs to one run. For the L3/L4 scripts that is a temporary directory, through the new_run_egress_script. Foripv6-egress-policyit is thetmpof the fake guest root in which_run_ipv6_guest_scriptalready gives the script its/procand resolver. Each test also leaves a stale result in that directory and requires that the run replace it with exactly its probes' results. A probe or a cleanup that went back to a shared/tmptherefore fails the test even when it runs alone, as the tests do against the scripts ofdev.The other scripts that these tests run in the host's shell already keep their files in per-test directories, or write none: the tests of
host-loopback-policyandprivate-network-ingressrewrite those scripts'/tmppaths into the test's root.Validation
I reproduced the race and validated the fix on every configured host, in isolated worktrees at
dev(29c68547b) and at this branch's scripts and tests:scripts/test_microvm_tests.pyat once, as concurrentvalidate-nvxblocks run it: 4 on Linux, 3 on Windows.devdevnvx-kvm-testlinux-kvm, VMnvx-mshv-testlinux-mshv, VMprometheus32linux-kvm, bare metal,shis dashprometheus30linux-mshv, bare metalnvx-whp-testwindows-whp, VMprometheus28windows-whp, bare metalEvery failure at
devwas in these three tests, with the messages above.On the guest side,
test-microvm --scenario l3-l4-egress-protocols --scenario l3-l4-egress-port-ranges --scenario ipv6-egress-policypasses on all six hosts (KVM, MSHV, and WHP). Those runs usednvx-rsbuilt from the branch and thev0.1.0-dev.320666c6c096packages, and the guests report every probe, as before.Locally on Windows:
ruff check,ruff format --check, andpyright --pythonplatform LinuxandWindowsare clean.dev: only the stale result remains inTMPDIR.host-loopback-policyandprivate-network-ingressscript subtests time out under Git'sshwith or without this change, as scripts: keep the guest console's echo out of microVM test markers #991 also noted. They pass on both remote Windows hosts.