runtime-adapter: a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used fails intermittently #1020

Activity

1. Pedro Henrique Penna (ppenna)

   self-assigned this

   on Oct 29, 2026
2. Pedro Henrique Penna (ppenna)

   added the

   Bug

   issue type on Oct 29, 2026
3. ppenna commented on Oct 30, 2026

   

   Pedro Henrique Penna (ppenna)

   on Oct 30, 2026

   Contributor

   More actions

   Third occurrence, and root cause confirmed by reproduction.

   Occurrence. Run 40949525134 (attempt 1, dev push of #963's merge, head 4d9e76d90), job NVX microVM tests / Linux / KVM / Guests and sandboxes, step Run runtime adapter scenarios on Linux: the same signature (the second start exited 0 with sandbox=started; left: Some(0), right: Some(8)). The same scenario passed on MSHV and WHP in that run.

   Root cause. After a clone's restore, its repair finishes through TimeAbi::restore_finish, which hands the time ABI's repair step 7 to the nvx-time daemon (daemon::queue_restore). The daemon runs step 7 in a child (start_restore → restore_work → grace_period_release, nvx-rs/src/guest/time/daemon.rs), which calls sys::enter_private_mount_namespace() (unshare(CLONE_NEWNS), then MS_REC | MS_PRIVATE) to force an RCU grace period. That child runs only once the guest goes idle, typically right after the clone's start and first probe, so its new namespace copies every mount, the clone's overlay and ext4 scratch included. While the copy exists, the worker's teardown (Root::teardown, a plain umount of each mount) succeeds, but its unmounts are not the last references: neither file system is shut down or synced, nothing else syncs, and nvx-exit powers the VM off through the control port. The earlier clone's writes, the overlay's upper and work entries included, stay in the guest's page cache, so the copy of its scratch is fresh and the next clone over it starts.

   Reproduction on a Linux/KVM VM host (nvx-kvm-test, Azure Linux 3), at 4d9e76d90 with this run's OpenVMM (c32c57d12) and guest artifacts:

   • The scenario alone failed once in 11 runs, with the CI signature.
   • With nvx-time instrumented (a patched binary appended to build/initramfs.cpio.gz as an extra newc archive; the runtime identity comes from the provenance digests), and a diagnostic probe that lists the processes in other mount namespaces, step 7 started only once the clone had started and served its first probes (16–32 ms after the diagnostic probe), entered its namespace, and finished its release 5–13 ms later.
   • Holding step 7's namespace for 5 s once the overlay is mounted made the scenario fail 3 of 3 times with the CI signature. Just before the stop, the probe showed nvx-time (pid 53, a child of the daemon, pid 42) in another mount namespace whose table held the clone's ext4 scratch and overlay.
   • Control, with the same wait and the same 5 s delay but entering the namespace after them: 3 of 3 passed.

   Candidate fix (prepared locally, not pushed yet): Root::teardown syncs an ext4 scratch's file system (syncfs on its mount point, through the root's Kernel seam) right before it unmounts the scratch, so a teardown that succeeds leaves every write on the scratch device whatever other namespace still holds the mounts. A scratch whose sync fails stays mounted and is reported (failed to sync the scratch: …). With the fix, on the same host:

   • The 5 s hold no longer fails the scenario: 3 of 3 runs of the diagnostic copy and 5 of 5 runs of the original test passed; 20 of 20 runs of the original test also passed with the original initramfs.
   • python3 scripts/nvx.py test-runtime-adapter --backend kvm passed: the 155 scenarios of 11 targets.
   • At dev 6b485c82c, the nvx-rs tests (with all features, the default ones, and none, and the musl guest tests of check-guest-binaries), documentation, MSRV check, guest-binary build and tests, and runtime adapter tests pass, except 3 nvx-rs guest tests that fail identically without the fix on that host's kernel 6.6 (runtime_binds_need_a_mount_root_unless_there_are_none and two …_ends_nothing_by_id). fmt and clippy pass for the Linux targets.

   The daemon's namespace copy may also briefly pin other mounts after any restore, such as those of NVX's own clones later; teardown durability no longer depends on it, and a separate change could keep grace_period_release from copying the workload's mounts.
4. Pedro Henrique Penna (ppenna)

   closed this as completedin #1031on Oct 30, 2026