[Enrique Saurez (esaurez)]

Chain: #951 → this PR → #957 → #959 → #960, on dev; #879, #881, #885, #886, #877, #932, #884, #887, #933, #880, #936, #937, #939, #940, #941, #888, #949, and #950 have merged.

Part of #844 (PR 5 of 5: clones in VMs through the test adapter).

What and why

With #949, #950, and #951, NVX plans a clone of a template's generation, restores the generation's snapshot in place as the clone's launch, activates the clone through Activate, Ready, and Start, attributes and records a clone's failures, and reattaches a clone once its caller restarted. Nothing runs a clone in a VM yet. This PR gives the runtime adapter's host program a path that starts clones of templates, and VM tests of the clones on a real hypervisor, which cover the VM cases that #842, #843, and #844 leave to it.

Starting a clone

runtime-adapter-host start starts a clone of a generation of a template store in a sandbox store, through nvx::host::template:

• The library's clone module holds what the options choose: the image, an EROFS layer of the template's placeholder's length; the upper, on an ext4 scratch image or in RAM; the memory target; the workload identity; the overlay mode; and the scratch's journal policy. Request::clone_request gives NVX's CloneRequest, with the bindings that the sandbox's store stages for the images, so the start hashes nothing.
• The template store leases the generation to the sandbox at the host's time, which may come from a clock fixture, RUNTIME_ADAPTER_HOST_TIME, for the tests of eligibility.
• NVX's ClonePlan checks the clone against the generation from the metadata of its files alone, under the sandbox's lock, so a clone that the template cannot serve fails before any VMM starts.
• The command records the clone in the sandbox's store as a launch record in the mode clone, with the template store and the generation, so that probe, exec, and a later stop find them.
• The restore sequence activates the clone through the adapter's clone order, host::CloneOrder: an Activate with the clone's record, a check of the guest agent's build in its Ready, and Start, with no mapping table. The order is built in one place, clone_order(), so that an optional export can join it later (nvx-rs: exports, MAPS and network templates for clones #846).

Failures

A clone that fails ends with NVX's CloneFailure. The command prints its record as failure.KEY lines, then:

• releases the lease, with the records, once the failure's teardown is resolved and nothing of the clone runs, and exits with 8;
• keeps the lease and the records for a later reattach while the teardown is unresolved, as when a VM of the clone may still run, or NVX could not record the failure before it ended the VM, and exits with 9;
• quarantines the generation through its tombstone, for a failure of the template alone, once it released the sandbox's lock.

A guest agent that doesn't answer the command once the clone's sandbox started fails the clone through Started::fail.

Reattach, stop, probe, and exec

reattach, stop, probe, and exec reattach to a clone with NVX's reattach_clone, in a new session in which the clone answers the Activate of its record again:

• a clone whose VM ended releases its lease;
• one that runs under another runtime identity ends through its verified identity, and fails;
• one whose state cannot be determined keeps its lease.

A stop releases the lease once its VM exited, and NVX's records of the clone told how it ended, since the release removes them: a VM that had exited when the stop began ended without its holder, which NVX's Vmm::stop_unless_exited tells by stopping nothing (stopped=exited) and keeping the runtime record of its launch, from which reattach_clone attributes its exit; and a status of the guest's that is evidence of a failure, such as its repair's, fails the clone, as a start's failure ends. reattach and stop take --template to release the lease of a start that ended before it recorded the clone, whose template store no record names. They also take the step hook, at a new step, found, at which they hold the VM that they found, and have sent its guest agent nothing more.

The guest agent

The guest agent gains probes of its memory, of the reads of its disks, of a read of each disk to its end, and of the guest's time state. A clone answers a probe of when the events of its activation and order happened after the repair's acknowledgement of the restore.

Not in this PR

• GPT clones, the fresh-scratch check, and a clone's export and network settings (nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845); exports, MAPS, and network templates (nvx-rs: exports, MAPS and network templates for clones #846).
• The adapter's template and clone operations (runtime adapter: template and clone operations #847).
• Checks from the host's side that no host input reaches a clone before its repair's acknowledgement, and that its restore reads no large content (runtime-adapter: prove two clone-restore invariants from the host side #965).

Tests

The VM tests, tests/vm_clone.rs, start clones on a real hypervisor, as CI's microVM jobs run the adapter's ignored tests on KVM, MSHV, and WHP. They cover the issue's cases:

• one generation reused across uppers and memory targets: one_generation_serves_clones_of_each_upper_and_memory_target, two clones at once, over a scratch image and in RAM, then a third at another memory target;
• images of other lengths and a geometry mismatch, refused at the plan before any VMM starts: a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts;
• a scratch that isn't ext4, and a record whose workload account the image lacks, refused as request faults before Ready; and no workload before Ready, since a ping, a start, or an execution before the Activate fails the clone: a_clone_whose_activation_fails_ends_before_it_is_ready;
• executions that run as the record's workload identity: a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready;
• same-boot and reboot eligibility, with clock fixtures: a_generation_is_eligible_by_the_clock_of_its_capture;
• no large-content read on restore, scratch included, with a read of each disk to its end as the positive control: a_clone_reads_no_more_of_its_disks_than_its_root_needs;
• deferred time work after Ready: a_clone_s_time_settles_after_it_is_ready;
• no host input before the acknowledgement: in each test that runs a clone, the clone's reset, the host's first message, its Ready, and its start came in that order, after the repair's acknowledgement of the restore, as the clone's activation probe reports them;
• restarts with reattach: each_command_reattaches_to_the_clone_from_a_new_process;
• a runtime mismatch: a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity;
• a clone whose VM ends on its own once a stop found it running, which fails by the loss of its status before the stop releases its lease: a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released;
• a leased clone of a tombstoned generation: a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease;
• the lease of a start that ended before it recorded its clone: the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store.

Without a VM:

• tests/clone_start.rs: the options that make no start, a generation that the store doesn't hold refused before anything else, the host time's text form, the clone's record, and the request's bindings and defaults;
• tests/guest_session.rs: a clone reports when the events of its activation happened after the acknowledgement;
• tests/step_hook.rs: the found step, and reattach and stop, like launch, refusing a step hook that they cannot read;
• unit tests of the activation probe, and of the disk statistics and time state that the new probes read.

Validation

With the OpenVMM submodule at its gitlink, 832d5bb4. This PR changes no nvx-rs file. Everything below ran at 21b9983, on #951's a06737d.

Windows (x86_64-pc-windows-msvc):

• nvx-rs: fmt; Clippy with -D warnings, with all features, --lib, no default features, each feature alone, and the guest feature for the musl target; tests with all features (1250 passed), default (1022), and no default features (265; with host, 944); tests/host_modules.rs (22); rustdoc with -D warnings; the MSRV (1.89) check; the OpenVMM vectors (3).
• tests/runtime-adapter: fmt, Clippy, build, tests, and MSRV in its protocol-only (47 passed) and host (108 passed, and the 70 VM tests ignored) configurations; Clippy in its musl guest configuration.
• The Markdown links.

Linux (x86_64-unknown-linux-gnu):

• nvx-rs: the same checks, with all features (1993 passed), default (1007), and no default features (265; with host, 930), its macOS check, and its guest binaries' checks.
• tests/runtime-adapter: the same configurations, protocol-only (47 passed) and host (107 passed, and the 70 VM tests ignored), with its musl guest tests (140 passed) and the MSRV check in each.
• As root: the adapter's musl guest tests, whose worker tests use cgroups only as root (132 passed), and nvx-rs's lib tests of the clone, platform, supervisor, store, sandbox, and sys modules (472 passed).

CI, ci.yml run 40825521178, at 21b9983: every job that ran passed. The adapter's VM suites passed in each "Guests and sandboxes" job, on KVM, MSHV, and WHP: vm_clone (12), vm_layered (43), vm_restart (3), vm_steps (3), and vm_template (9).

The same 70 VM tests passed on WHP and on MSHV, with NVX's development release v0.1.0-dev.2d14547b669f, whose OpenVMM is 832d5bb4, and the head's Alpine initramfs and guest agent.

[Copilot]

🟡 Changes recommended

Saved lease-path resolution can stop the wrong sandbox and release its lease, while missing or corrupt indexed-generation keys bypass typed failure reporting and quarantine.

2 open issues

• Prevent lease path reopening from switching sandboxes · New
• Route corrupt indexed generations through typed quarantine failures · New

What changed in this PR

This PR connects the runtime adapter to NVX’s template-clone lifecycle, completing the VM integration portion of #844 on top of the planning, restore, and recovery APIs.

Changes:

• Adds clone startup with staged images, configurable resources, failure reporting, and lease management.
• Extends reattach, stop, probe, and exec to manage clones.
• Adds guest diagnostics and automated coverage for clone activation, recovery, eligibility, and resource behavior.

File	Description
tests/​runtime-adapter/​tests/​vm_clone.rs	Adds real-hypervisor clone lifecycle scenarios.
tests/​runtime-adapter/​tests/​guest_session.rs	Tests activation timeline reporting.
tests/​runtime-adapter/​tests/​clone_start.rs	Tests clone options, records, bindings, and clock fixtures.
tests/​runtime-adapter/​src/​worker/​probes.rs	Adds disk-read, memory, and time probes.
tests/​runtime-adapter/​src/​worker/​capture.rs	Passes the repair acknowledgment time to the clone worker.
tests/​runtime-adapter/​src/​worker/​activation.rs	Records and reports clone activation events.
tests/​runtime-adapter/​src/​worker.rs	Accepts acknowledgment timing for clone diagnostics.
tests/​runtime-adapter/​src/​probe.rs	Shares fact-key validation within the crate.
tests/​runtime-adapter/​src/​lib.rs	Exposes the host-only clone module.
tests/​runtime-adapter/​src/​layered.rs	Shares staging errors and documents clone launch records.
tests/​runtime-adapter/​src/​clone.rs	Defines clone requests, records, and clock fixtures.
tests/​runtime-adapter/​src/​bin/​runtime-adapter-host.rs	Implements clone startup and clone-aware lifecycle commands.
tests/​runtime-adapter/​README.md	Documents clone commands, probes, and test scenarios.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Clone cleanup can erase repair-failure attribution before outcome classification, while test gaps leave acknowledgement gating and the no-large-content-read restore requirement unverified.

1 open issue

• Preserve clone records until failure attribution completes · New

2 previously missed issues

In code that hasn't changed since last review

Test input remains blocked until repair acknowledgement

tests/​runtime-adapter/​tests/​vm_clone.rs:599

These timestamps record when work_clone consumes input, but that worker starts only after repair returns, so input queued before the acknowledgement would still pass this assertion. The test therefore does not verify the required no-input-before-acknowledgement gate. Add a VM test that pauses repair before acknowledgement, attempts early host input, and observes at the transport/VMM boundary that delivery remains blocked until acknowledgement.

Test host-side reads during restore staging

tests/​runtime-adapter/​tests/​vm_clone.rs:1256

These counters measure only guest block requests, so hashing or copying the entire distro or scratch file in the host's start path would still pass this test and its guest-side positive control. Add host-side logical file-read observation around start, after staging, with a matching positive control alongside the guest counters. Include the VMM's reads and cached reads so the check covers the no-large-content-read restore requirement.

2 resolved since last review

• Prevent lease path reopening from switching sandboxes
• Route corrupt indexed generations through typed quarantine failures

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Adding Found to Step::ALL causes the launch-step test to wait for an unreachable pause and panic, so the VM suite fails until the test is restricted to the four launch steps.

1 open issue

• Restrict VM pause test to launch steps, not Found · New

1 resolved since last review

• Preserve clone records until failure attribution completes

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because a critical shutdown race can clear runtime state and release a lease without recording or reporting the clone’s lost-status failure.

No open issues 🎉

1 resolved since last review

• Restrict VM pause test to launch steps, not Found

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are reported for the runtime adapter’s template-clone lifecycle integration.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The documented example uses format 2, but the writer and reader require format 1, so records following the example cannot be read back.

1 open issue

• Use record format 1 in the example · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Clone restore, teardown, and lease release cross process and hypervisor boundaries, while exact-head KVM, MSHV, and WHP validation remains pending and cannot be established by static review.

No open issues 🎉

1 resolved since last review

• Use record format 1 in the example

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Exact-head validation on KVM, MSHV, and WHP remains pending for restoration, reattachment, lease handling, and teardown, leaving cross-process lifecycle guarantees unconfirmed.

No open issues 🎉

_{Review effort: Balanced}