[Enrique Saurez (esaurez)]

Stacked on #950; this PR retargets when its parent merges.
Chain: #888 → #949 → #950 → this PR, on dev; #879, #881, #885, #886, #877, #932, #884, #887, #933, #880, #936, #937, #939, #940, and #941 have merged.

Part of #844 (PR 4 of 5: failures, teardown, and reattach).

What and why

#950 runs a clone's restore sequence to its sandbox's start and the launch's commit. A failure there ends the launch with an error that says what failed, but not whose failure it is, and nothing of it outlives the caller. This PR attributes every failure of a clone, from its plan to its commit, records it in the clone's sandbox with the teardown that followed, and lets a caller that restarted take its clones over.

Attribution

A clone's plan, launch, and restore sequence fail with a CloneFailure, whose Evidence attributes the failure to an Outcome, at a Stage, once the clone got as far as its Progress:

• the request: a denied authorization; a stale binding; invalid targets, format, settings, Activate, or mappings; an image, a scratch, or a root that the template cannot serve; and a supplied file of the wrong identity, length, or geometry, which the host's staging takes instead once the caller proves it, CloneFailure::with_staging_proven;
• the host: an endpoint that cannot be bound, resource exhaustion, a host that cannot restore the template, another runtime, or a failure of its own operations;
• unknown: a lost status, a timeout, a reset, a forced kill, or a broken protocol;
• the template: a malformed generation, or a defect of its capture or repair that a diagnosis attributes to it. This is the only outcome that quarantines the generation, through its tombstone, CloneFailure::quarantine;
• the repair stage, for a status of the clone's repair alone.

OpenVMM's outcome report gives the status with which the guest ended the clone's VM, which status_evidence attributes: a status of the repair at any point, and one of the activation before the start was sent. It prevails over what the host or the caller saw of the same end, but not over evidence of the template, which shows the provenance that a status alone leaves to establish.

A VMM that exits before its restore-ready event, as OpenVMM does once it cannot restore the snapshot, leaves its fatal error in its log. The failure's message keeps that error, and its time ABI code attributes the failure to the host when it says that the host cannot present the template's CPU profile, TSC, or LAPIC timer. The wait for the event gives a VMM that closed the endpoint without sending it ten seconds to exit.

Records and teardown

From the launch on, a failure ends the clone's VMM through its verified identity. The sandbox keeps two more files, which a release removes, as a removal of the sandbox does, and which a cold launch never writes:

• clone-progress.log, the clone's progress log: a first line that names the launch and the boot of the host, then a line for each step from the spawn to the commit, without a sync. The launch writes the spawn before it returns, since a teardown of the launch removes its journal, which records the spawn too. It removes an earlier launch's log through a handle of the sandbox's directory, which its lock checked, as a release removes the clone's files, and creates its own log where none may exist: a path that names another sandbox's directory by then fails the launch, rather than lose that sandbox's log;
• clone-failure.json, the record of the failure, JSON in format FAILURE_FORMAT. It is written before the teardown, with the teardown pending, so that a crash of the caller during the teardown leaves it for a reattach to finish, and again once the teardown ended, each time durably: the write syncs the sandbox's directory once the record is in place. It holds the stage, the evidence and the status, the original outcome, the transaction, the generation, the progress, and the teardown: no VM, an exit or a termination that the host verified, or an unresolved one. It keeps at most 8192 characters of the failure's message, and 1024 of the reason of an unresolved teardown: of a longer text, its first and last ones, with an ellipsis between them. So a reattach reads every record that a failure writes, however long a caller's diagnosis.

An unresolved teardown keeps the launch's journal or runtime record, and with it the lease. So does a failure whose pending record cannot be written, as on a full disk: nothing that tells that the clone launched is removed before its failure is recorded. A failure of the restore sequence, or of the caller's, then ends the clone's VMM through its handle and keeps the launch's journal, Starting::end_keeping_journal, with its teardown unresolved; a launch that fails before it returns keeps its journal for recovery; and a reattach fails before it ends or forgets anything of the clone's VM, which it leaves as it found it. A record that a write put in place, but whose sync failed, may not be durable yet, so a reattach that finds it syncs the sandbox's directory before it ends or forgets anything of the clone's launch, and fails the same way if that sync fails. A crash of the caller before the record then leaves a launch that a reattach records as failed, rather than a clone that ended. A clone whose sandbox records a failure launches no more under its lease. A release ends the reservation of the lock that acquired the lease before it removes anything (#949), so one that fails after it removed the failure record leaves no launch to begin. A launch that spawned its VMM but could not record it in its journal records the failure as of the spawn, and then ends the VMM through its handle.

A step of the caller's own between Ready and the commit, such as one that maps a directory of the host into the clone, fails the clone with the caller's evidence through Activated::fail or Started::fail. These record the failure and end the launch, as the sequence's own failures do.

Reattach

reattach_clone takes a clone over once the caller of its launch restarted:

• A VMM of the clone's launch that runs under the caller's runtime identity is checked in a new session. The clone answers the byte-equal Activate of clone-activation.bin with the Ready that it answered once, and the reattach checks the transaction ID that it reports. ReattachedClone::Live returns the VM with the session.
• A clone of another runtime identity, or one that fails the check, is ended through its verified identity, and fails at the reattach stage.
• It records the failure of a launch that a crash interrupted, and of a VM that exited, which it classifies by one reading of its outcome report, before it ends the launch or forgets the VM. It finishes a failure that a crash left pending, and resolves the teardown of an earlier one with what the VMM left as it ended.
• A record of a failure that it cannot read, malformed or of another format, still tells that the clone launched and failed, though not how. The reattach takes it as the record of an earlier failure. If the store holds the journal or the runtime record of the clone's launch, which the reattach may remove, it first replaces the record with its own failure at the reattach stage, with the read error and the host's evidence, its teardown pending, which keeps how far the clone got for a reattach after a crash. It then ends a running VMM through its verified identity and, whatever it found of the VM, records the failure with its teardown, over whose evidence a status of the guest's in OpenVMM's outcome report prevails. It never returns the clone live, ended, or indeterminate.
• A record that it can read, but whose template store, lease, generation, key, or launch is not the binding's, is another clone's failure, which tells nothing of this clone. The reattach takes it as a record that it cannot read, with an error that names what differs, so the failure that it records and returns is this clone's, and a quarantine of it reaches only the generation that the clone leases.
• A progress log that is damaged, missing, or written during another boot of the host leaves how far the clone got uncertain, so a status that a workload may have given is no evidence of the activation.
• It reads nothing of the template store, so a tombstone or a retirement of the generation leaves the clone its lease.
• Its termination bound applies to a VMM whose launch committed. A launch that a crash interrupted ends as LockedStore::recover ends it, within the store's own wait for the VMM's termination.

CloneRecords::read reads what a sandbox keeps of a clone, for the operations that list, inspect, or remove clones. It fails for a failure record of another clone, as for one that cannot be read. It returns the outcome report of the clone's VM only once the clone got past its plan: the launch removes the report of an earlier VM of the sandbox, which the sandbox otherwise keeps until the next launch begins, before it creates the progress log.

Stops

A holder that checks whether its clone's VMM has exited, then stops the VM, would attribute to its stop a VMM that exits in between: Vmm::stop finds it exited, runs no step, reports StopOutcome::Graceful, and clears the runtime record, so a reattach finds no record, and ends the clone as one that its holder stopped, rather than with the loss of its status. Vmm::stop_unless_exited stops the VM as Vmm::stop does, unless its own check, before the graceful step, finds the VMM exited. It then returns None, runs no step, signals nothing, and keeps the runtime record, so that reattach_clone finds that the VM exited unheld, and records the loss of the clone's status, if no report of the guest's own end tells why, before it clears the record. Once that check found the VMM running, its exit is the stop's. Vmm::stop, Vmm::terminate, and their callers are unchanged.

Also fixes

Two fixes to code from #949, which its review found:

• The restore-ready socket gets mode 0600 right after the bind, whatever the umask, so that a VMM that is not root can connect to it (review thread).
• An acquisition opens the leased generation's payload as the release of the lease opens it, following no link from the template store's directory on, and checks the payload's owner and mode through that handle, so that it grants no lease that the release could not end (review thread).

Not in this PR

• Clones in VMs through the test adapter (PR 5).
• GPT clones and the fresh-scratch check (nvx-rs: activate a clone's root for GPT images and either upper, with its export, network settings and workload view #845); exports and network templates (nvx-rs: exports, MAPS and network templates for clones #846).

Validation

At this head, 0352bb9, with OpenVMM at d2338ea2, per commit: on Windows, fmt and Clippy with -D warnings for nvx-rs in CI's three configurations (--all-targets --all-features, --lib, and --lib --no-default-features), tests/host_modules.rs, the tests of the platform, template, store, sandbox, and supervisor modules, and the test adapter's fmt, Clippy, and host tests; and for the x86_64 Linux target, Clippy with -D warnings in the same three configurations, the aarch64-apple-darwin check, and the test adapter's Linux host and musl guest Clippy.

The gates, every step of check-nvx-rs on Windows and on Linux x86_64 with the host tests as root on Linux, and the test adapter's VM suites, vm_layered, vm_template, vm_restart, and vm_steps, on Windows (WHP) and Linux (MSHV), run at the final heads of the stack once its reviews are clean.

Tests added, with the fake VMM and the fake control:

• the restore-ready listener, which reads a client that closed before its connection was accepted to its end, on both platforms;
• the attribution table, row by row: the repair and activation statuses, the precedence of a status over what was seen, but not over the template's evidence, OpenVMM's fatal error and its codes, and resource exhaustion;
• the teardowns, and the records: progress logs that a crash cut short or that another launch wrote, failure records of another format, failure records of another clone, refused with what differs, and the line of OpenVMM's fatal error read from the end of its log;
• the request's refusals: unauthorized images, malformed generations, keys that the read refuses for what they are (a link, a directory on Linux, and a key that holds more than a key may), whose failures quarantine their generation, keys of another encoding version than the build reads, whose failures are the host's and quarantine nothing, and paths that no longer name their supplied files;
• failures recorded before their teardown and again once it ended, and launches that cannot be ended, which keep their records and their lease until a reattach ends them;
• reattaches: a committed clone in a new session; a clone that the reattach may not keep; launches that a crash interrupted; VMs that exited, classified by one reading of their reports; earlier failures finished; records of a failure that cannot be read, with a live VM, one under another runtime identity, VMs that exited with a status that is evidence, one that is not, or no report, launches that a crash interrupted before and after the recovery, an unbegun clone beside an earlier VM's report, and an indeterminate VM; a record of another clone's failure with the template's evidence, which the reattach replaces with this clone's failure, whose quarantine tombstones nothing; and progress logs that a reboot or a crash may have cut short;
• a caller that fails a ready or started clone with its own evidence;
• stops through stand-in VMMs: of a VMM that had exited before the stop began, whose runtime record, and with it the lease, only Vmm::stop_unless_exited keeps, and of running ones, which it stops as Vmm::stop does; and a clone whose VMM exits once its holder found it running, before the holder's stop: it fails at its run with the loss of its status, which quarantines nothing, while Vmm::stop ends it as stopped;
• inspections of a clone whose launch has not begun, which take no report that an earlier VM of the sandbox left, even once a reattach recorded a failure of its own, and launches, which remove that report before they record anything;
• inspections of clones whose launch began, which take their VM's report, including that of a VMM that the launch's journal did not record, once a reattach of the clone or a recovery of the sandbox ended the launch;
• a caller's diagnosis too long for a record: the record keeps its ends and its evidence through a reattach, and the reattached failure quarantines the generation;
• failure records that are in place when their directory is synced, and that keep both ends of a message or a reason that a read could not take whole;
• failures whose record cannot be written: a caller's diagnosis of the template keeps the launch's journal and the lease, then a reattach records the clone as failed; reattaches of an interrupted launch, of a VM that exited, of one under another runtime identity, and of one that fails its activation check end and forget nothing, then record the failure once they can; the supervisor keeps the journal of a launch, and the runtime record of a VM that exited, whose caller cannot record them; and a reattach makes the record of an earlier failure durable before it ends the launch, and ends nothing when it cannot;
• a progress log that replaces an earlier one only in the directory that its handle holds, while the sandbox's path names another sandbox's directory, whose log then fails the creation;
• a launch that its caller abandons before the clone's activation, whose progress log records the spawn, so that a reattach reads the outcome report of the clone's VM and records the failure that the guest's status gives;
• on Linux, the restore-ready socket's mode 0600 under a umask that withholds its owner's permission to write it, and an acquisition's refusal of a link in the place of the template store's directory, of the directory of the generations, or of the payload, leasing nothing.

These tests run no VM: clones' real restores come with PR 5's VM tests.

[Copilot]

🟡 Changes recommended

Lease release can remove the retry-blocking failure marker before revoking the reservation, allowing another launch under the same lease if cleanup fails.

3 open issues

• Revoke reservation before removing clone files · New
• Advance tracker when process recording fails after spawn · New
• Propagate termination bound into launch recovery · New

What changed in this PR

This PR extends NVX’s clone restore lifecycle with attributed failures, persistent failure and teardown records, and reattachment after the caller restarts. It implements the failure-handling and recovery portion of #844; real-VM integration remains for the next PR.

Changes:

• Adds typed failure attribution across clone planning, launch, activation, and commit.
• Records clone progress and teardown outcomes, with cleanup when the lease is released.
• Adds clone reattachment and fake-process tests for failures, recovery, and restore-ready connections.

File	Description
nvx-rs/​src/​host/​template/​store/​leases/​tests.rs	Tests release cleanup of progress and failure records.
nvx-rs/​src/​host/​template/​store/​leases.rs	Documents clone records removed during release.
nvx-rs/​src/​host/​template/​store.rs	Exposes directory validation for failure attribution.
nvx-rs/​src/​host/​template/​mod.rs	Exports and documents failure and reattachment APIs.
nvx-rs/​src/​host/​template/​clone/​tests/​restore.rs	Tests attribution, teardown records, and caller-reported failures.
nvx-rs/​src/​host/​template/​clone/​tests.rs	Tests planning failures and their evidence.
nvx-rs/​src/​host/​template/​clone/​record.rs	Attributes activation-record construction errors.
nvx-rs/​src/​host/​template/​clone/​reattach.rs	Implements clone reattachment and failure recovery.
nvx-rs/​src/​host/​template/​clone/​launch.rs	Tracks launch progress and records failures.
nvx-rs/​src/​host/​supervisor/​tests.rs	Adapts tests to reattachment reporting callbacks.
nvx-rs/​src/​host/​supervisor/​mod.rs	Adds callbacks before teardown and runtime-record removal.
nvx-rs/​src/​host/​store/​tests.rs	Extends clone-file cleanup coverage.
nvx-rs/​src/​host/​store/​mod.rs	Registers progress and failure files with the sandbox store.
nvx-rs/​src/​host/​store/​lifecycle/​tests.rs	Tests removal and validation of the new records.
nvx-rs/​src/​host/​sandbox/​scratch.rs	Exposes binding version for attribution checks.
nvx-rs/​src/​host/​sandbox/​distro.rs	Exposes binding version for attribution checks.
nvx-rs/​src/​host/​platform/​windows.rs	Tests restore-ready clients that disconnect before acceptance.
nvx-rs/​src/​host/​platform/​linux.rs	Tests restore-ready clients that disconnect before acceptance.
nvx-rs/​README.md	Documents failure attribution, records, teardown, and reattachment.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

An unreadable failure record is treated as absent, allowing a clone with unresolved teardown to become usable again under the same lease instead of preserving the error or safely terminating the VM.

1 open issue

• Do not treat unreadable failure records as absent · New

3 resolved since last review

• Revoke reservation before removing clone files
• Propagate termination bound into launch recovery
• Advance tracker when process recording fails after spawn

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The reattach wildcard branch discards clone-failure.json read errors and records a generic run-stage exit failure, losing diagnostics that should be preserved at Stage::Reattach.

1 open issue

• Preserve clone-failure diagnostics for committed VMM exits · New

1 resolved since last review

• Do not treat unreadable failure records as absent

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Cross-platform VMM termination and crash-recovery changes need human review because tests use fakes and real clone restore/reattach validation is deferred to PR 5.

No open issues 🎉

1 resolved since last review

• Preserve clone-failure diagnostics for committed VMM exits

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The spawn state is not durably recorded before Restoring is returned, so early teardown can cause reattachment to skip failure reporting and incorrectly report a failed guest repair as Ended.

1 open issue

• If Restoring is abandoned or dropped before activate(), teardown can remove the journal before… · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

A critical recovery bug can leave clone progress at Launching after a caller crash, causing reattach_clone to return Ended even for repair status 195 because the guard incorrectly requires a recorded spawn.

1 open issue

• Allow reattach repair for launched clones without recorded spawn · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human assessment is needed because recovery can overlook a valid activation record, causing a guest repair failure to go unrecorded and the clone to be reported as ended.

No open issues 🎉

1 resolved since last review

• Allow reattach repair for launched clones without recorded spawn

_{Review effort: Balanced}