From 52bb19a631e4c9064981862eba80f4d659b1c39c Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 25 Oct 2026 20:15:58 -0700 Subject: [PATCH 1/5] runtime-adapter: type the selections that a template store refuses A start whose template store refuses its selection now ends with the refusal's record, in the form of a clone failure's, at the stage `selection`, and with the exit status 10, before it leases or launches anything. A template store that does not exist, and a generation that the store never published or no longer holds, name no template; a generation whose capture the restore window of its time contract no longer admits at the host's time is expired; and one that a tombstone quarantines, or whose capture is after the host's time, is ineligible. The record's code is that of NVX's error, and the diagnostic stays on standard error. A selection that races a retirement, or follows one that a crash interrupted, ends as the store's refusal of the lease too: a retirement tombstones a generation before it deletes its payload, and deletes none that a lease keeps, and the start reads the generation's compatibility key before the lease only for a memory target that the request does not name, leaving a key that it cannot read to NVX's plan. The clone tests check each refusal through the record, including a generation that `remove` retired, and the unit tests the refusals of a selection that names no template, which need no hypervisor. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7d3db585-441c-495b-901f-ab7203c0f36b --- tests/runtime-adapter/README.md | 50 ++++-- .../src/bin/runtime-adapter-host.rs | 170 ++++++++++++++---- tests/runtime-adapter/tests/clone_start.rs | 33 ++-- tests/runtime-adapter/tests/vm_clone.rs | 134 +++++++++++--- 4 files changed, 297 insertions(+), 90 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index ee0b22032..72248b6e0 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -1517,6 +1517,18 @@ each, the clock as a manifest spells it, `linux-boottime` or It affects the store's check alone: OpenVMM's restore samples the host's own time. +A selection that the store refuses ends the start before it leases anything, +with the refusal's record, in the form of a failure's, at the stage +`selection`, without a VM, and the status 10: a template store that does not +exist, and a generation that the store never published, or no longer holds, +name no template, `no-template`, by the evidence `no-store`, `unpublished`, or +`retired`; a generation whose capture the restore window of its time contract +no longer admits at the host's time is `expired`; and one that a tombstone +quarantines, or whose capture is after the host's time, by the clock that +measures its downtime, is `ineligible`, by the evidence `tombstone` or +`early`. The record's code is the code of NVX's error: `not-found`, +`template-tombstoned`, or `template-ineligible`. + NVX's `ClonePlan` then checks the clone against the generation from the metadata of its files alone, under the sandbox's lock, which the lease holds: its network, which must be the one that the template's key holds, the guest's @@ -1764,7 +1776,10 @@ Each command of `runtime-adapter-host` but `stage`, `list`, `inspect`, and leaving the VM running, and the sandbox holding the lease. A clone that fails, such as one whose activation failed before it was ready, and whose VM ended with the status of the failure's class, from 50 to 58, ends as - [Clones](#clones) describes, with the record of its failure. + [Clones](#clones) describes, with the record of its failure. A selection + that the template store refuses, of no template, of an expired generation, + or of an ineligible one, ends the start before it leases anything, with the + refusal's record, as [Clones](#clones) describes. - `stop STORE [--template TEMPLATES]` reattaches and attaches as `reattach` does, then stops the VM: the adapter's `Stop`, which the guest agent confirms with `Stopped` before it @@ -1850,13 +1865,18 @@ commas, or `none`), `downtime` and `source` (the downtime of the leased generation's capture at the host's time, in nanoseconds, and the clock that measured it, `monotonic` or `utc`), `released` (the identity of the lease that a stop, or a start that -failed, released), `failure.KEY` (the record of a clone's failure: its +failed, released), `failure.KEY` (the record of a clone's failure, or of a +start's selection that the template store refused: its `outcome`, what the failure is attributed to, `request`, `staging`, `host`, -`unknown`, `template`, or `repair`; its `evidence`, what the host saw of it, -which attributes it, such as `supplied-length`; its `stage`, where the clone +`unknown`, `template`, or `repair`, or `no-template`, `expired`, or +`ineligible` for a refused selection; its `evidence`, what the host saw of it, +which attributes it, such as `supplied-length`, or `no-store`, `unpublished`, +`retired`, `expired`, `tombstone`, or `early` for a refused selection; its +`stage`, where the clone failed, `plan`, `launch`, `restore`, `activation`, `start`, `commit`, `run`, or -`reattach`; its `progress`, the step that the clone reached, as `progress` -spells it; its `status`, the status with which the guest ended the VM, as the +`reattach`, or `selection`; its `progress`, the step that the clone reached, as +`progress` spells it, or `none` for a refused selection; its `status`, the +status with which the guest ended the VM, as the VM's outcome report gives it, or `none`; its `teardown`, how the teardown that followed ended, `no-vm`, `exited`, `terminated`, or `unresolved`; and its `code`, the code of the failure's error, such as `runtime-mismatch`), @@ -1890,7 +1910,9 @@ failed a mapping of the table, and the VM stopped gracefully, with a success and a complete teardown, with 7 when `remove` found leases that keep the generation, which it tombstoned, with 8 when a clone failed, once nothing of it runs and the command released its lease, with 9 when a clone failed, and its -teardown is unresolved, so that the sandbox keeps the lease, and with 1, after +teardown is unresolved, so that the sandbox keeps the lease, with 10 when the +template store refused a start's selection, so that the start leased and +launched nothing, and with 1, after a diagnostic on standard error, when they fail. A value of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time is a usage error, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step @@ -2272,9 +2294,10 @@ Windows, without a hypervisor: other commands do not take, with a clock fixture that is not a host's time too; - a start whose template store does not exist, or does not hold the - generation, whose image the plan refuses, and whose shares it cannot export, - each with status 1 before it needs NVX's artifacts or creates the sandbox's - store, and the shares before it reads the template store; + generation, each with status 10 and the record of a selection that names no + template, and one whose image the plan refuses, or whose shares it cannot + export, with status 1, each before it needs NVX's artifacts or creates the + sandbox's store, and the shares before it reads the template store; - a start whose network cannot be read, or is one that the plan refuses, with status 1 before anything else: before its image, which does not exist, its shares, which it cannot export, and the template store, which does not exist; @@ -2688,19 +2711,19 @@ directory that the starts authorize: | --- | --- | --- | | `one_generation_serves_clones_of_each_upper_and_memory_target` | Two at once, over a scratch image at 256 MiB, and in RAM at 512 MiB with untrusted lowers, then one over a scratch image at 1 GiB | Each started, by the host's monotonic clock, with its own lease, which the store lists with the clone's sandbox, its memory target online, its upper, overlay mode, and image's contents, and the events of its activation in order; a stop releases each lease; the generation's snapshot unchanged, and its payload with nothing but the capture's files once the clones stopped | | `one_generation_serves_clones_of_either_format_over_either_upper` | Of a template whose placeholder is a GPT disk of two layers, four at once: the EROFS image over a scratch image and in RAM, a GPT disk of one layer in RAM, and one of three layers over a scratch image, with a bound that its first partition meets exactly; then the disk of one layer, with a bound one byte smaller than its first partition | Each started, with its upper and the overlay's options; the EROFS image's root, without a first partition; each GPT disk's layers, ext4 file systems mounted read-only once each, from the partitions that its own table names, stacked in order, and the bytes of its own first partition; then status 8, and the record of a failure of the request, at the activation, once the VM exited, by the evidence of an unusable root, with the guest's status 54; nothing quarantined, and the lease released, with the records | -| `a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts` | An image 4 KiB longer, one 4 KiB shorter, and one 100 bytes longer, than the placeholder, a scratch image of twice the template's scratch, memory targets of 2 GiB and of 320 MiB, unstaged images, and a generation that the store does not hold, then the generation once its key is gone | For those that NVX's plan refuses, status 8, with NVX's reason, and the record of a failure of the request, at the plan, without a VM, by the evidence of a length that the request supplied, of the image's geometry, which OpenVMM cannot attach, or of invalid targets, which quarantines nothing; for the others, status 1, without a record; any lease released, and no VMM's log or record in the sandbox; for the generation without its key, status 8, and the record of a failure of the template, at the plan, by the evidence of a malformed generation, which quarantines the generation, so that a later start fails without a lease | +| `a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts` | An image 4 KiB longer, one 4 KiB shorter, and one 100 bytes longer, than the placeholder, a scratch image of twice the template's scratch, memory targets of 2 GiB and of 320 MiB, unstaged images, and a generation that the store does not hold, then the generation once its key is gone | For those that NVX's plan refuses, status 8, with NVX's reason, and the record of a failure of the request, at the plan, without a VM, by the evidence of a length that the request supplied, of the image's geometry, which OpenVMM cannot attach, or of invalid targets, which quarantines nothing; for unstaged images, status 1, without a record; for the generation, status 10, with the record of a selection that names no template, which the store never published, without a lease; any lease released, and no VMM's log or record in the sandbox; for the generation without its key, status 8, and the record of a failure of the template, at the plan, by the evidence of a malformed generation, which quarantines the generation, so that the store refuses a later start's selection, with status 10, as `ineligible`, by the evidence of its tombstone, without a lease | | `a_clone_whose_activation_fails_ends_before_it_is_ready` | A scratch image of zeros, a journal that the record forbids, a workload identity whose account the image lacks, and, through NVX's restore sequence, a ping, a start, or the start of an execution before the `Activate` | Status 8, no `Ready`, and the record of a failure, at the activation, once the VM exited: of the request, by the evidence of an unusable root, with the guest's status 54, or of invalid settings, with 55, or, before the `Activate`, of an unknown cause, by the protocol's, with 51, which the sandbox records until the release; nothing quarantined, and the lease released, with the records | | `a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used` | Over a scratch image, in whose root the `contents` probe writes a file, then over a copy of that scratch image once the first clone stopped | The first started; then status 8, since the copy's upper is not fresh, and the record of a failure of the request, at the activation, once the VM exited, by the evidence of an unusable root, with the guest's status 54; nothing quarantined, and the lease released, with the records | | `clones_at_once_keep_their_own_uppers_and_machine_ids` | Three at once: over a scratch image and in RAM, each with a generated machine ID, and in RAM with the image's own; two executions in each, as user ID 0, that write a marker in the root, and try to write `/etc/machine-id` | In every execution, the hostname `runtime-adapter`, and no marker of another clone; a generated machine ID that differs between the clones, the same in each execution of a clone, held by one bind that is read-only, `nosuid`, and `nodev`, and that the write cannot change; with the image's own machine ID, none, since the image has none, no bind, and in the later execution, the ID that the first wrote; then each stop releases its lease | | `a_clone_whose_view_cannot_be_set_ends_before_it_is_ready` | Of a template of the EROFS image, a GPT disk whose layer holds a regular file `etc`, with the image's own machine ID, then with a generated one | The first started, with the disk's root; then status 8, since no bind can hold the machine ID below a file, and the record of a failure of the request, at the activation, once the VM exited, by the evidence of invalid settings, with the guest's status 55; nothing quarantined, and the lease released, with the records | | `a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready` | Over a scratch image, as nobody | An execution that runs as nobody, one that names an identity refused with `EINVAL`, and the first execution after the clone's start | -| `a_generation_is_eligible_by_the_clock_of_its_capture` | At the host's time, then with clock fixtures: UTC a day behind, another boot an hour after the capture by UTC, and the restore window passed by each clock, or UTC before the capture | The host's monotonic clock on the boot of the capture, whatever UTC says, UTC after a reboot, with the fixture's downtime, and the store's refusal without a lease outside the window | +| `a_generation_is_eligible_by_the_clock_of_its_capture` | At the host's time, then with clock fixtures: UTC a day behind, another boot an hour after the capture by UTC, and the restore window passed by each clock, or UTC before the capture | The host's monotonic clock on the boot of the capture, whatever UTC says, UTC after a reboot, with the fixture's downtime, and, outside the window, status 10, with the record of the store's refusal of the selection, `expired` once the window passed, and `ineligible`, by the evidence `early`, before the capture, without a lease | | `a_clone_reads_no_more_of_its_disks_than_its_root_needs` | Over a scratch image, and in RAM | Less than an eighth of each disk read once the clone is ready, and each disk's whole length once the guest read it to its end | | `a_clone_s_time_settles_after_it_is_ready` | In RAM | The time daemon's restore check `ok`, after the restore's generation, without a violation, after a downtime that the host's monotonic clock measured, and a clone that still serves | | `each_command_reattaches_to_the_clone_from_a_new_process` | Over a scratch image, then `reattach`, `probe`, `exec`, and `reattach` again, each from a new process | The launch's records, committed, with the clone's VM and no failure; each command `live`, with the clone's VMM, in a new session of the same broker, in a later epoch; the events of the activation as they were; an execution as nobody; and a stop that releases the lease, with the records | | `a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity` | In RAM, then a `reattach` with another guest agent's build | Status 8, `reattached=failed`, without a session, and the record of a failure of the host, at the reattach, by the evidence of the runtime's mismatch, once NVX terminated the VMM, which quarantines nothing; the lease released, with the records; then another clone that starts and stops | | `a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released` | In RAM, then a `stop` that the step hook pauses at `found`, while the test terminates the clone's VMM | Status 8, `reattached=live` and `stopped=exited`, and the record of a failure of an unknown cause, at the run, by the evidence of the clone's lost status, without a status of the guest's, once the VM exited, which quarantines nothing; the lease released, with the records; then another clone that starts and stops | -| `a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease` | Over a scratch image, then a tombstone of its generation, through NVX's store | A start that the tombstone refuses, without a lease; the running clone, which reattaches, probes, runs an execution, and stops, which releases its lease; and the generation's snapshot unchanged | +| `a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease` | Over a scratch image, then a tombstone of its generation, through NVX's store | A start whose selection the store refuses, with status 10 and the record of an `ineligible` generation, by the evidence of its tombstone, without a lease; the running clone, which reattaches, probes, runs an execution, and stops, which releases its lease; and the generation's snapshot unchanged | | `the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store` | In RAM, in a sandbox that NVX's store leased a generation to, as a start that ended before it recorded the clone leaves it: first moved, with another sandbox at its lease's path, which holds a lease of its own, then none; then in place, without a record of a clone, then with the record of an earlier clone, of another template | For the moved sandbox, a `stop` and a `reattach` that refuse it, since the sandbox at its lease's path does not hold its lease, and leave that sandbox's lease; in place, a stop that finds the clone `ended` at `planned`, and keeps the lease, whose template store no record names; a stop with another template store, which the release refuses; a `reattach` or a `stop` with `--template` naming the lease's own, which releases it; and a start and a stop in the sandbox after each | | `a_clone_maps_its_export_beneath_its_root_before_its_sandbox_starts` | Over a scratch image, sharing one host directory: the directory itself, read-only, a read-write directory inside it, at its target there, a read-only file, and a read-write directory, with a file hidden | Started, with four mappings, and a record that names the export, a directory, by the identity of its plan, with four mappings; on Linux, the VMM's one `--mount`, of the directory, read-write; the export mounted once, read-write, `nosuid`, and `nodev`, below a `hostfs` of mode 0700 that root owns; the mappings below the root, and nothing else, in the table's order, each with its access, `nosuid`, and `nodev`, showing what the host holds but for the hidden file, with the missing targets created empty in the upper; the same from an execution as user ID 0; and writes through the writable mappings alone, which the host's directory holds, before and after the stop | | `clones_at_once_see_only_their_own_exports` | Four at once: one with a read-write export of its own and a generated machine ID, two over the same read-only export, over a scratch image and in RAM, and one without an export | Each with an export started with one mapping, and a record that names it by the identity of its plan, the same for the clones of one export, and on Linux, one `--mount`, read-write or read-only, as the mapping needs; each export mounted read-write, even where its mapping is read-only; below the root, the mapping, beside the bind of a generated machine ID; each clone's own directory, with its access; without an export, no export in the record, no mapping, no `--mount` on Linux, no export mounted, and nothing below the root; the `entries` probe, whose helper writes the root's machine ID, failing with `EROFS` where the generated one is bound; the host's directories with what the writable export took alone; then each stop releases its lease | @@ -2711,6 +2734,7 @@ directory that the starts authorize: | `a_clone_binds_the_resolver_of_its_policy_s_dns_server_read_only` | Of a template of a network with egress allowed, two at once, with a generated machine ID, and with the image's own; then, once they stopped, a GPT disk whose layer holds a regular file `etc`, with the image's own machine ID | Each of the first two started, with a record that names the gateway's DNS server, one network device, and at `/etc/resolv.conf`, a file of mode 0444 that names the server, held by a bind that is read-only, `nosuid`, and `nodev`, after the bind of the generated machine ID, below the root, whose file system refuses a write as root, with nothing in the upper there; and a TCP connection to the server's DNS port, which the gateway's DNS proxy accepts without the host's resolver; then status 8, since no bind can hold the resolver configuration below a file, and the record of a failure of the request, at the activation, once the VM exited, by the evidence of invalid settings, with the guest's status 57; nothing quarantined, and the lease released, with the records | | `a_clone_s_executions_find_the_proxy_of_its_policy` | Of a template of a proxy-only network, one, with an execution as user ID 0 that runs `env`, then one that names its own `HTTPS_PROXY` | A record that names the proxy's URL, `http://192.0.2.1:PORT`, at the port of the proxy's listener, and no DNS server; the first execution with its own entry, then the URL in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`; the second with exactly its own environment; one network device, and the image's own `/etc/resolv.conf`, which no bind holds | | `a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver` | Of a template of a network with egress allowed, one with a generated machine ID, sharing a host directory read-write | Started, with one mapping, and a record that names the export and the gateway's DNS server; on Linux, the VMM's one `--mount`, read-write, and its arguments of the template's policy; below the root, the binds of the machine ID and of the resolver configuration, then the export's mapping; the export's directory, and a write through it, which the host's directory holds after the stop | +| `a_selection_that_names_no_template_leases_nothing` | A start that names a template store that does not exist, then, once `remove` retired the generation, which no lease kept, a start of it | Status 10, with the record of a selection that names no template, by the evidence `no-store`, then `retired`, without a lease or a VM; and a store that indexes no generation | Every network that the scenarios give a template is one that a template takes: a network that lets nothing through, egress allowed, with outbound diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index e7584fa47..31edf27ee 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -186,7 +186,12 @@ //! agent that does not answer the `Ping` or the `Echo` fails the clone so, through //! `Started::fail`, as the restore sequence attributes the failures of its sessions. A start //! that fails otherwise, at a step of the adapter's own before the clone's launch, releases the -//! lease. +//! lease. A selection that the template store refuses ends the start before it leases anything, +//! with the refusal's record, in the form of a failure's: a template store that does not exist, +//! and a generation that the store never published, or no longer holds, name no template; a +//! generation whose capture the restore window of its time contract no longer admits at the +//! host's time is expired; and one that a tombstone quarantines, or whose capture is after the +//! host's time, is ineligible. //! - `runtime-adapter-host stop STORE` reattaches and attaches as `reattach` does, then stops the //! VM: it asks the guest agent to stop, which first tears down a layered root, waits for the VM //! to exit, and checks the VM's outcome report. A VM whose guest agent does not confirm the stop @@ -300,18 +305,24 @@ //! in nanoseconds, and the clock that measured it: `monotonic`, the host's monotonic clock, on //! the host's boot that captured the template, or `utc` otherwise; //! - `released`: the identity of the lease that a stop, or a start that failed, released; -//! - `failure.KEY`: the record of a clone's failure, with these KEYs: +//! - `failure.KEY`: the record of a clone's failure, or of a start's selection that the template +//! store refused, with these KEYs: //! - `outcome`: what the failure is attributed to: `request`, `staging`, `host`, `unknown`, -//! `template`, or `repair`; -//! - `evidence`: what the host saw of it, which attributes it, such as `supplied-length`; +//! `template`, or `repair`, or, for a refused selection, `no-template`, `expired`, or +//! `ineligible`; +//! - `evidence`: what the host saw of it, which attributes it, such as `supplied-length`, or, +//! for a refused selection, `no-store`, `unpublished`, or `retired`, of no template, +//! `expired`, or `tombstone` or `early`, of an ineligible generation; //! - `stage`: where the clone failed: `plan`, `launch`, `restore`, `activation`, `start`, -//! `commit`, `run`, or `reattach`; -//! - `progress`: the step that the clone reached, as `progress` spells it; +//! `commit`, `run`, or `reattach`, or `selection` for a refused selection; +//! - `progress`: the step that the clone reached, as `progress` spells it, or `none` for a +//! refused selection; //! - `status`: the status with which the guest ended the VM, as OpenVMM's outcome report gives //! it, or `none`; //! - `teardown`: how the teardown that followed ended: `no-vm`, `exited`, `terminated`, or //! `unresolved`; -//! - `code`: the code of the failure's error, such as `runtime-mismatch`; +//! - `code`: the code of the failure's error, such as `runtime-mismatch`, or `not-found`, +//! `template-tombstoned`, or `template-ineligible` for a refused selection; //! - `quarantined`: whether the command quarantined the generation of a clone that failed: //! `yes`, for a failure of the template, or `no`; //! - `store`: the identity of the template store that `list` or `inspect` read, 32 hexadecimal @@ -355,10 +366,11 @@ //! a mapping of the table, and the VM stopped gracefully, with success and a complete teardown, //! with 7 when `remove` finds leases that keep the generation, which it tombstoned, //! with 8 when a clone failed, once nothing of it runs and the command released its lease, with 9 -//! when a clone failed, and its teardown is unresolved, so that the sandbox keeps the lease, and -//! with 1, after a diagnostic on standard error, when they fail. Anything else is a usage error, -//! with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, or of -//! `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time. +//! when a clone failed, and its teardown is unresolved, so that the sandbox keeps the lease, with +//! 10 when the template store refused a start's selection, so that the start leased and launched +//! nothing, and with 1, after a diagnostic on standard error, when they fail. Anything else is a +//! usage error, with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, +//! or of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time. use std::env; use std::error::Error; @@ -383,13 +395,13 @@ use nvx::host::store::{CallerFiles, IdentityPart, LOG_NAME, LaunchId, LockedStor use nvx::host::supervisor::{self, Reattached, Starting, StopOutcome, Vmm, VmmCommand}; use nvx::host::template::{ Acquired, CAPTURES_NAME, Capture, CloneControl, CloneFailure, ClonePlan, Created, - DowntimeSource, Eligibility, Evidence, FailureRecord, HostTime, Lease, ReattachedClone, - RestoreDeadlines, Retirement, Store, TRANSACTION_NAME, reattach_clone, + DowntimeSource, Eligibility, Evidence, FailureRecord, Generation, HostTime, Lease, + ReattachedClone, RestoreDeadlines, Retirement, Store, TRANSACTION_NAME, reattach_clone, }; use nvx::protocol::CAPABILITY_LEN; use nvx::protocol::root::ImageFormat; use nvx::uuid::Uuid; -use nvx::{ErrorKind, Hypervisor}; +use nvx::{ErrorCode, ErrorKind, Hypervisor}; use runtime_adapter::GUEST_BUILD; use runtime_adapter::capture::{ DEFAULT_CPU_PROFILE, DEFAULT_PROCESSORS, DEFAULT_SCRATCH_LENGTH, Request, @@ -454,6 +466,10 @@ const CLONE_FAILED: u8 = 8; /// VM of the clone may still run, or NVX could not record the failure before it ended the VM, and /// the sandbox keeps the clone's lease, and NVX's records of its launch, for a later reattach. const CLONE_UNRESOLVED: u8 = 9; +/// The exit status of a start whose template store refused its selection: the store does not +/// exist, or does not hold the generation, or the generation is expired or ineligible at the +/// host's time, so that the start leased and launched nothing. +const SELECTION_REFUSED: u8 = 10; // The timers and limits of `exec` below are values chosen for tests that make a valid // configuration, within the limits that the guest and the host allow: chunks of at most // `exec::MAX_CHUNK` bytes, at most `worker::MAX_EXECUTION_MEMORY` bytes held in the guest for @@ -2124,24 +2140,34 @@ fn start( network, ..clone.request.clone() }; - // A request that the adapter's plan refuses, shares that it cannot export, and a generation - // that the template store does not hold, fail the start before it needs NVX's artifacts or - // the sandbox's store. + // A request that the adapter's plan refuses, shares that it cannot export, and a selection + // that the template store refuses as it names no template, fail the start before it needs + // NVX's artifacts or the sandbox's store. let plan = request.plan().map_err(|error| describe(&error))?; let export = request.export().map_err(|error| describe(&error))?; - let templates = Store::open(&clone.templates).map_err(|error| describe(&error))?; - let generation = templates - .index() - .map_err(|error| describe(&error))? - .get(clone.generation) - .cloned() - .ok_or_else(|| { - format!( - "the template store {} holds no generation {}", - templates.dir().display(), - clone.generation - ) - })?; + let templates = match Store::open(&clone.templates) { + Ok(templates) => templates, + Err(error) if error.kind() == ErrorKind::NotFound => { + return Ok(selection_refused(Refusal::NoTemplate("no-store"), &error)); + } + Err(error) => return Err(describe(&error)), + }; + let index = templates.index().map_err(|error| describe(&error))?; + let Some(generation) = index.get(clone.generation).cloned() else { + // The store never assigns a store generation twice, so one before its next publication's + // was published, and retired since. + let evidence = if clone.generation < index.next_generation() { + "retired" + } else { + "unpublished" + }; + let error = nvx::Error::not_found(format!( + "the template store {} holds no generation {}", + templates.dir().display(), + clone.generation + )); + return Ok(selection_refused(Refusal::NoTemplate(evidence), &error)); + }; // The clone's memory target is the request's, or else the template's RAM, which the // generation's key gives. A key that cannot be read gives none: NVX's plan reads the key // again, once the store leased the generation, and attributes the failure, to the generation @@ -2160,15 +2186,22 @@ fn start( None => HostTime::now().map_err(|error| describe(&error))?, }; let sandbox = SandboxStore::create(store).map_err(|error| describe(&error))?; - let mut acquired = templates - .acquire( - generation.generation(), - sandbox.dir(), - &CallerFiles::none(), - &now, - Some(Instant::now() + LOCK_TIMEOUT), - ) - .map_err(|error| describe(&error))?; + let acquired = templates.acquire( + generation.generation(), + sandbox.dir(), + &CallerFiles::none(), + &now, + Some(Instant::now() + LOCK_TIMEOUT), + ); + let mut acquired = match acquired { + Ok(acquired) => acquired, + Err(error) => { + return match Refusal::of(&error, &generation, &now) { + Some(refusal) => Ok(selection_refused(refusal, &error)), + None => Err(describe(&error)), + }; + } + }; println!("generation={}", generation.generation()); println!("lease={}", acquired.lease.id()); if let Eligibility::Eligible(downtime) = generation.eligibility(&now) { @@ -2197,6 +2230,65 @@ fn start( } } +/// Why a template store refused the selection of a start, as the start reports it: its outcome, +/// with the evidence that attributes it. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Refusal { + /// The selection names no template: the store does not exist, `no-store`, or never published + /// the generation, `unpublished`, or retired it, `retired`. + NoTemplate(&'static str), + /// The restore window of the time contract of the generation's capture no longer admits it at + /// the host's time. + Expired, + /// The generation takes no new clone: a tombstone quarantines it, `tombstone`, or its capture + /// is after the host's time, `early`. + Ineligible(&'static str), +} + +impl Refusal { + /// Returns why the store refused, with `error`, to lease `generation` at the host's time `now`, + /// or `None` if the error is not a refusal of the selection: the store no longer holds the + /// generation, which it retired since the start read its index, a tombstone quarantines it, or + /// it is not eligible at `now`, as `Generation::eligibility` decides it. A sandbox that is not + /// provisioned, whose error is of the kind `NotFound` too, is not a refusal of the selection. + fn of(error: &nvx::Error, generation: &Generation, now: &HostTime) -> Option { + match error.code() { + ErrorCode::TemplateTombstoned => Some(Self::Ineligible("tombstone")), + ErrorCode::TemplateIneligible => Some(match generation.eligibility(now) { + Eligibility::Expired(_) => Self::Expired, + _ => Self::Ineligible("early"), + }), + ErrorCode::NotFound => Some(Self::NoTemplate("retired")), + _ => None, + } + } + + /// Returns the outcome and the evidence of the refusal, as its record spells them. + fn names(self) -> (&'static str, &'static str) { + match self { + Self::NoTemplate(evidence) => ("no-template", evidence), + Self::Expired => ("expired", "expired"), + Self::Ineligible(evidence) => ("ineligible", evidence), + } + } +} + +/// Ends a start whose template store refused its selection, as `refusal` says, with `error`, which +/// leased and launched nothing: prints the refusal's record, as [`print_clone_failure`] prints a +/// failure's, at the stage `selection`, and returns the exit status [`SELECTION_REFUSED`]. +fn selection_refused(refusal: Refusal, error: &nvx::Error) -> ExitCode { + let (outcome, evidence) = refusal.names(); + println!("failure.outcome={outcome}"); + println!("failure.evidence={evidence}"); + println!("failure.stage=selection"); + println!("failure.progress=none"); + println!("failure.status=none"); + println!("failure.teardown=no-vm"); + println!("failure.code={}", error.code()); + eprintln!("runtime-adapter-host: {}", describe(error)); + ExitCode::from(SELECTION_REFUSED) +} + /// How the launch of a clone failed, once the store leased its generation. enum Failed { /// The clone failed, as NVX attributes the failure, and records it from the clone's launch on. diff --git a/tests/runtime-adapter/tests/clone_start.rs b/tests/runtime-adapter/tests/clone_start.rs index 34f02872f..15563c31a 100644 --- a/tests/runtime-adapter/tests/clone_start.rs +++ b/tests/runtime-adapter/tests/clone_start.rs @@ -1,12 +1,12 @@ //! The host program's `start` refuses options that do not make a start, and a clock fixture that is -//! not a host's time, as usage errors, and a network that the plan refuses, shares that it cannot -//! export, a generation that the template store does not hold, or a template store that does not -//! exist, before it needs NVX's artifacts or creates the sandbox's store. A clone's record names its -//! template store and its generation, which a stop reads back, beside the plan that the other -//! commands read, with the clone's network, whose proxy an execution of the clone then finds at -//! the URL that NVX's activation record names, a clone's request carries the export of its shares, -//! and a host's time keeps its text form, the form of the clock fixture. The tests need no -//! hypervisor. +//! not a host's time, as usage errors, a network that the plan refuses and shares that it cannot +//! export, and a selection that names no template, of a template store that does not exist, or of +//! a generation that the store does not hold, with the refusal's record and its own status, before +//! it needs NVX's artifacts or creates the sandbox's store. A clone's record names its template +//! store and its generation, which a stop reads back, beside the plan that the other commands +//! read, with the clone's network, whose proxy an execution of the clone then finds at the URL +//! that NVX's activation record names, a clone's request carries the export of its shares, and a +//! host's time keeps its text form, the form of the clock fixture. The tests need no hypervisor. #![cfg(feature = "host")] use std::fs; @@ -300,8 +300,15 @@ fn a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_any ]; host(&arguments, None) }; - let failed = |output: &Output, problem: &str| { - assert_eq!(output.status.code(), Some(1), "{output:?}"); + let refused = |output: &Output, evidence: &str, problem: &str| { + assert_eq!(output.status.code(), Some(10), "{output:?}"); + let stdout = String::from_utf8_lossy(&output.stdout); + let record = format!( + "failure.outcome=no-template\nfailure.evidence={evidence}\nfailure.stage=selection\n\ + failure.progress=none\nfailure.status=none\nfailure.teardown=no-vm\n\ + failure.code=not-found\n" + ); + assert_eq!(stdout, record); let stderr = String::from_utf8_lossy(&output.stderr); assert!(stderr.contains(problem), "{stderr}"); assert!(!stderr.contains("usage:"), "{stderr}"); @@ -310,9 +317,9 @@ fn a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_any // No template store. let output = start(&templates, "1"); - failed(&output, "cannot open the template store"); + refused(&output, "no-store", "cannot open the template store"); - // A template store that holds no generation. + // A template store that never published a generation. Store::create(&templates).unwrap(); for generation in ["1", "2"] { let output = start(&templates, generation); @@ -320,7 +327,7 @@ fn a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_any "the template store {} holds no generation {generation}", Store::open(&templates).unwrap().dir().display() ); - failed(&output, &problem); + refused(&output, "unpublished", &problem); } // An image that the plan refuses fails the start before the store is read. diff --git a/tests/runtime-adapter/tests/vm_clone.rs b/tests/runtime-adapter/tests/vm_clone.rs index c90010283..e97c87daa 100644 --- a/tests/runtime-adapter/tests/vm_clone.rs +++ b/tests/runtime-adapter/tests/vm_clone.rs @@ -31,7 +31,8 @@ //! refuses, a memory target beyond its capacity or of a part of a block. The start then releases //! the lease, as it does once the adapter refuses images that the sandbox does not stage. A //! generation whose key is gone fails the plan too, as the template's failure, and the start -//! quarantines it through its tombstone. +//! quarantines it through its tombstone, so that the store refuses a later start's selection of +//! it, as ineligible. //! - A clone whose activation fails ends before it is ready, with the status of the failure's //! class, which attributes the failure, and the VMM's exit: a scratch image that holds no ext4 //! file system, a journal that the clone's record forbids, a workload identity whose account the @@ -74,9 +75,12 @@ //! - A clone whose VM ends without a word of its guest once a stop found it running fails, as NVX //! attributes the loss of its status, before the stop, which stops nothing, releases its lease: //! the template still serves clones. -//! - A tombstone quarantines the generation of a running clone: the generation takes no new lease, -//! while the clone keeps running, reattaches, stops, and releases its lease, and the generation's -//! snapshot stays as the capture left it. +//! - A tombstone quarantines the generation of a running clone: the store refuses the selection of +//! the generation for a new clone, as ineligible, while the clone keeps running, reattaches, +//! stops, and releases its lease, and the generation's snapshot stays as the capture left it. +//! - A selection that names no template leases nothing, with the record of the store's refusal: a +//! template store that does not exist, a generation that the store never published, and one that +//! `remove` retired. //! - A start that ended once the store leased the generation, before it recorded the clone, //! leaves a lease whose template store no record names, or only the record of an earlier clone, //! of another template: a stop keeps it, and only its own template store, which `--template` @@ -85,8 +89,8 @@ //! sandbox at the lease's path, which holds another lease, or none. //! - The store checks the generation's eligibility at the host's time: on the boot that captured //! the template, by the host's monotonic clock, whatever UTC says, and after a reboot by UTC, -//! which clock fixtures stand for. It refuses a generation that is no longer, or not yet, -//! eligible, without a lease. +//! which clock fixtures stand for. It refuses the selection of a generation that is no longer +//! eligible, as expired, or not yet, as ineligible, without a lease, with the refusal's record. //! - A clone reads no more of its disks on its restore than its root needs, which a read of each //! disk to its end shows the guest's statistics count. //! - A clone's time settles after it is ready, with no violation, after the downtime that the @@ -184,6 +188,9 @@ const FAILED: i32 = 1; /// The exit status of a command whose clone failed, once nothing of the clone runs, and the /// command released its lease. const CLONE_FAILED: i32 = 8; +/// The exit status of a start whose template store refused its selection, which leased and +/// launched nothing. +const SELECTION_REFUSED: i32 = 10; /// The guest agent's build of a host program of another runtime. const ANOTHER_BUILD: &str = "another-build"; /// Bytes in a mebibyte. @@ -861,7 +868,7 @@ impl Template { let generation = GENERATION.to_string(); let mut options = vec![ OsStr::new("--template"), - self.dir().as_os_str(), + start.templates.unwrap_or(self.dir()).as_os_str(), OsStr::new("--generation"), OsStr::new(&generation), OsStr::new("--distro"), @@ -947,6 +954,8 @@ enum Image<'a> { /// What a scenario asks of the start of a clone. #[derive(Default)] struct StartOf<'a> { + /// The template store, or the template's by default. + templates: Option<&'a Path>, /// The image, or the platform's image by default. image: Option>, /// The upper. @@ -1259,6 +1268,23 @@ impl Cloned { .map(|argument| String::from_utf8_lossy(argument).into_owned()) .collect() } + + /// Checks that the template store refused the start's selection before the start leased + /// anything, as the record that the start printed says: the refusal's `outcome` and + /// `evidence`, at the selection, without a VM, the code `code` of NVX's error, and no lease, + /// generation, or quarantine, and as [`assert_refused`](Self::assert_refused) checks it, with + /// a diagnostic that says `problem`. + fn assert_selection_refused(&self, outcome: &str, evidence: &str, code: &str, problem: &str) { + let started = &self.started; + assert_eq!( + failure(started), + [outcome, evidence, "selection", "none", "none", "no-vm"], + "{started:?}" + ); + assert_eq!(started["failure.code"], code, "{started:?}"); + assert_eq!(started.len(), 7, "{started:?}"); + self.assert_refused(problem); + } } /// What an execution of [`VIEW`] found in a clone's root. @@ -2089,8 +2115,19 @@ fn a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts() { OsStr::new("--upper"), OsStr::new("ram"), ]; - let printed = sandbox.run("start", &options, FAILED); - assert!(printed.is_empty(), "{printed:?}"); + // The store never published another generation: the selection names no template. + let printed = sandbox.run("start", &options, SELECTION_REFUSED); + let record = [ + "no-template", + "unpublished", + "selection", + "none", + "none", + "no-vm", + ]; + assert_eq!(failure(&printed), record, "{printed:?}"); + assert_eq!(printed["failure.code"], "not-found", "{printed:?}"); + assert_eq!(printed.len(), 7, "{printed:?}"); assert!(sandbox.stderr().contains("holds no generation 2")); sandbox.assert_unlaunched(); @@ -2121,11 +2158,13 @@ fn a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts() { refused.assert_refused(" is missing"); assert!(template.leases().is_empty()); assert_eq!(template.store().tombstones().unwrap(), [GENERATION]); - let quarantined = template.start("cl-r", StartOf::default(), FAILED); - assert!(quarantined.started.is_empty(), "{:?}", quarantined.started); - quarantined.assert_refused(&format!( - "a tombstone quarantines the generation {GENERATION} of the template store" - )); + let quarantined = template.start("cl-r", StartOf::default(), SELECTION_REFUSED); + quarantined.assert_selection_refused( + "ineligible", + "tombstone", + "template-tombstoned", + &format!("a tombstone quarantines the generation {GENERATION} of the template store"), + ); } #[test] @@ -2392,13 +2431,15 @@ fn a_generation_is_eligible_by_the_clock_of_its_capture() { assert_eq!(printed["source"], "utc", "{printed:?}"); assert_eq!(printed["downtime"], HOUR_NS.to_string()); - // A generation that is no longer eligible, or not yet, takes no lease. + // A generation that is no longer eligible, or not yet, takes no lease: the store refuses the + // selection of an expired generation, and of one whose capture is after the host's time. let cases = [ ( HostTime { monotonic_ns: contract.capture_monotonic_ns + MAX_DOWNTIME_NS + 1, ..now }, + ["expired", "expired"], "is no longer eligible: after a downtime of", "by the host's monotonic clock", ), @@ -2407,6 +2448,7 @@ fn a_generation_is_eligible_by_the_clock_of_its_capture() { utc_ns: contract.capture_utc_ns + MAX_DOWNTIME_NS + 1, ..rebooted }, + ["expired", "expired"], "is no longer eligible: after a downtime of", "by UTC", ), @@ -2415,19 +2457,19 @@ fn a_generation_is_eligible_by_the_clock_of_its_capture() { utc_ns: contract.capture_utc_ns - 1, ..rebooted }, + ["ineligible", "early"], "is not eligible yet: by UTC", "1 ns before the anchor of the capture", ), ]; - for (time, state, measured) in cases { + for (time, [outcome, evidence], state, measured) in cases { let start = StartOf { time: Some(time), ..StartOf::default() }; - let refused = template.start("cl-k", start, FAILED); - assert!(refused.started.is_empty(), "{:?}", refused.started); + let refused = template.start("cl-k", start, SELECTION_REFUSED); let generation = format!("the generation {GENERATION} of the template store"); - refused.assert_refused(&generation); + refused.assert_selection_refused(outcome, evidence, "template-ineligible", &generation); let stderr = refused.sandbox.stderr(); assert!( stderr.contains(state) && stderr.contains(measured), @@ -2723,12 +2765,15 @@ fn a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease() { store.tombstone(GENERATION, Some(deadline)).unwrap(); assert_eq!(store.tombstones().unwrap(), [GENERATION]); - // The generation takes no new lease. - let refused = template.start("cl-n", StartOf::default(), FAILED); - assert!(refused.started.is_empty(), "{:?}", refused.started); - refused.assert_refused(&format!( - "a tombstone quarantines the generation {GENERATION} of the template store" - )); + // The generation takes no new lease: the store refuses the selection of a quarantined + // generation as ineligible. + let refused = template.start("cl-n", StartOf::default(), SELECTION_REFUSED); + refused.assert_selection_refused( + "ineligible", + "tombstone", + "template-tombstoned", + &format!("a tombstone quarantines the generation {GENERATION} of the template store"), + ); // The clone keeps running on its lease: it reattaches, serves its order, and stops, which // releases the lease. @@ -2842,6 +2887,45 @@ fn the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_tem assert_payload(&another, false); } +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_selection_that_names_no_template_leases_nothing() { + let mut template = Template::create("cl-none"); + + // A template store that does not exist. + let sandbox = template + .sandbox + .as_mut() + .expect("the template keeps its sandbox"); + let missing = sandbox.root.join("missing"); + let start = StartOf { + templates: Some(&missing), + ..StartOf::default() + }; + let refused = template.start("cl-m", start, SELECTION_REFUSED); + refused.assert_selection_refused( + "no-template", + "no-store", + "not-found", + "cannot open the template store", + ); + assert!(!missing.exists()); + + // A generation that the store retired, once no lease kept it. + let sandbox = template + .sandbox + .as_mut() + .expect("the template keeps its sandbox"); + let removed = sandbox.run("remove", &[OsStr::new("1")], 0); + assert_eq!(removed["removed"], "retired", "{removed:?}"); + let refused = template.start("cl-m", StartOf::default(), SELECTION_REFUSED); + let problem = format!("holds no generation {GENERATION}"); + refused.assert_selection_refused("no-template", "retired", "not-found", &problem); + let store = template.store(); + assert!(store.index().unwrap().generations().is_empty()); + assert!(store.leases(GENERATION).unwrap().is_empty()); +} + /// Leases the generation of `template` to the store of `sandbox`, through NVX's store, as a start /// does before it records its clone, and returns the lease's identity. fn lease_to(template: &Template, sandbox: &Sandbox) -> String { From a35e5d1ae96ac92286159c0aa98f612ea571f061 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Sun, 25 Oct 2026 20:56:39 -0700 Subject: [PATCH 2/5] runtime-adapter: run one execution in a clone of a template `runtime-adapter-host run` runs one execution in a clone of a generation of a template store, then tears the clone down, in one command. - Its preflight stages the clone's images in the sandbox's store, as `stage` does, and prints `preflight=staged`, before the clone's start, which hashes nothing. A start that the guest would refuse fails the command, as a usage error, before it stages anything. - It starts the clone as `start` does, through the same code, and ends as a start ends once the template store refuses the selection (status 10) or the clone fails (8, or 9 while the teardown is unresolved). - It runs the execution in the clone's root, through `host::Executions`, in the session of the clone order in which the clone's sandbox started, as `exec` runs one: the program, its arguments, and its environment reach the guest in the execution's start, never in the template's boot tokens. - Whatever the execution did, it stops the clone as `stop` does, and releases the lease once the VM exited. It holds the sandbox's lock from the lease to the teardown. It exits 0 once the execution finished, whatever its status, and the clone stopped. - An execution that failed to run, as one whose session ended, fails the command after the teardown. Since the session may have ended with the VM, as when its VMM crashed, the run first waits up to 10 seconds for the VMM to exit, as a stop waits for one that it terminated: the stop then finds that the VM exited without it, and ends the clone as a reattach that finds it so does, with the record of its failure. The run exits with the stop's status once it is not a success (8, or 9, for a clone that failed, and 4 for a root that failed), and with 1 otherwise. The stop takes the same session, in which the guest may still send messages about an execution that the driver finished without them, as when it gave up waiting for its outcome. `Client::stop_skipping` stops the guest agent as `stop` does, but skips the messages that its caller accepts, and `Executions::is_about` accepts the guest's messages about the driver's own executions; `stop` keeps its behavior. The parsing of an execution's options, the checks of its starts, the run of the executions, and the stop of a VM are factored out of `exec` and `reattach`, which keep their behavior. The README gains the table of the template and clone operations, and documents `run`, its key, and its statuses. The unit tests cover run's command lines and its refused selections after the preflight, and a stop that skips the guest's late messages about the executions that the driver gave up on; the VM tests cover a run with one execution, an execution that the guest refuses, and runs whose clone does not start. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7d3db585-441c-495b-901f-ab7203c0f36b --- tests/runtime-adapter/README.md | 116 ++- .../src/bin/runtime-adapter-host.rs | 663 ++++++++++++++---- tests/runtime-adapter/src/host.rs | 37 +- tests/runtime-adapter/src/host/executions.rs | 18 +- tests/runtime-adapter/src/lib.rs | 7 +- tests/runtime-adapter/tests/clone_start.rs | 206 +++++- tests/runtime-adapter/tests/host_session.rs | 93 ++- tests/runtime-adapter/tests/vm_clone.rs | 269 ++++++- 8 files changed, 1205 insertions(+), 204 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 72248b6e0..7b8b14b42 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -936,7 +936,11 @@ transport to the session's host endpoint, such as the one that or shares one with the next message; - `ping`, `echo` and `stop` each send a request and check its answer: `Pong`, an `Echo` of the same bytes, and `Stopped`. `stop` also returns the failure - of a layered root's teardown, which the guest reports before it confirms; + of a layered root's teardown, which the guest reports before it confirms. + `stop_skipping` stops the guest agent as `stop` does, but skips the messages + before `Stopped` that its caller accepts, such as those that the guest may + still send about a driver's executions, which `Executions::is_about` + accepts; - `configure_root` sends a root configuration, and returns how the guest answered: `Ready`, with the guest agent's build, or the stage of the root that failed; @@ -999,7 +1003,9 @@ cancellation for each execution, at most a credit for each message that it reads, of output windows of at most 32 frames, so that the guest's messages ahead of its own credit stay few. The guest may still send messages about the executions of a run that gave up on one or failed, so a later batch takes an -attachment of its own, as each command of the host program does. +attachment of its own, as each command of the host program does, and a stop in +the same attachment skips them, with `Client::stop_skipping` and +`Executions::is_about`, as `runtime-adapter-host run` stops its clone. The guest ends its session on a second start of the same key, even in a later epoch of the session, so the keys of a driver's executions must be new to the @@ -1345,6 +1351,17 @@ the hook does nothing. ### Templates +The adapter's operations on templates, and on their clones, are commands of +`runtime-adapter-host`, which [Commands](#commands) details: + +| Operation | Command | What it does | +| --- | --- | --- | +| Create, refresh | `create`, `refresh` | Serves an eligible generation of the template's capture class that the store holds, or captures a new one, as `refresh` always does, and prints it once the store committed it, then evicts the generations that it supersedes; see [Eviction](#eviction) | +| List, inspect | `list`, `inspect` | Reports the generations, their keys, classes, states, eligibility and its reason, and leases, and the metadata of a capture, without loading VM state or hashing; see [Listings](#listings) | +| Remove, retire | `remove` | Tombstones a generation, so that it takes no new lease, reports busy, with the status 7, while leases remain, and deletes it once they drained, but never moves or deletes a leased generation | +| Start from a template | `start` | Starts a clone of an explicit, eligible generation, with its image, scratch, and settings, which the sandbox stages; reports a selection of no template, of an expired generation, or of an ineligible one, and a clone's failure, attributed to the request, the host, or the template, or whose teardown is unresolved; never captures, boots cold, rehashes, or falls back to another format; see [Clones](#clones) | +| Run from a template | `run` | Stages the clone's images in a preflight that it reports, starts a clone as `start` does, runs one execution in the clone's root, and tears the clone down; see [Running a clone](#running-a-clone) | + `runtime-adapter-host create` serves a template of the guest agent from an NVX template store, `nvx::host::template::Store`, with `Store::create_template`: a generation of the template's capture class that the store holds, and that is @@ -1631,6 +1648,40 @@ A tombstone of the generation refuses new leases of it, and leaves those that sandboxes hold to their clones, which reattach, stop, and release them as before. +#### Running a clone + +`runtime-adapter-host run` runs one execution in a clone of a generation, and +tears the clone down. Its preflight, which it reports with `preflight=staged`, +stages the clone's images in the sandbox's store, as `stage` stages them, +verifying the image, which hashes it only when a manifest is beside it, before +the clone's start, which hashes nothing. The command first checks that the +guest takes the execution's start, so a start that it would refuse fails the +command before it stages anything. It then starts the clone as `start` does, +at the host's time, or the one that `RUNTIME_ADAPTER_HOST_TIME` gives, and ends +as a start ends once the template store refuses the selection, or the clone +fails, with the record of the refusal or of the failure. Once the clone runs, +the command runs the execution in its root, the one that the clone's +activation assembled, through `host::Executions`, in the session of the +adapter's clone order in which the clone's sandbox started, as `exec` runs +one. The program, its arguments, and its environment reach the guest in the +execution's start, through the guest agent's execution control, never in the +template's boot tokens. Then, whatever the execution did, the command stops the +clone as `stop` does, and releases the lease once the store verified that the +VM exited: a run leaves no clone running, and no lease held, unless its stop +timed out. The stop takes the same session, in which it skips the guest's +messages about the execution that may still arrive once the command finished +it without them, as when it gave up waiting for its outcome. It holds the +sandbox's lock, which the lease took, from the start to +the teardown, so no other command reattaches to its clone meanwhile. A run +whose execution failed to run, such as one whose session ended, still tears the +clone down, then fails, after the execution's diagnostic. Since the session may +have ended with the VM, as when its VMM crashed, the run first waits up to 10 +seconds for the VMM to exit, as long as a stop waits for one that it +terminated: its stop then finds that the VM exited without it, and ends the +clone as a reattach that finds it so does, with the record of its failure, so +the run exits with 8, or 9 while the teardown is unresolved. It exits with 4 +once the guest reported that a stage of the root failed, and with 1 otherwise. + ### Commands Each command of `runtime-adapter-host` but `stage`, `list`, `inspect`, and @@ -1822,6 +1873,18 @@ Each command of `runtime-adapter-host` but `stage`, `list`, `inspect`, and number. The guest refuses a key that it heard of before, and the epochs of a broker instance only advance, while a broker restored from a snapshot has a new instance ID, so no command repeats the key of another's execution. +- `run STORE --template TEMPLATES --generation G --distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] EXECUTION` + runs one EXECUTION, one of `exec`'s, in a [clone](#clones) of the generation + G of the template store TEMPLATES, in the sandbox store STORE, which it + creates if needed, then tears the clone down, as + [running a clone](#running-a-clone) describes. The execution's options may + come among the start's, in any order, and its PROGRAM and ARGs are every + argument after `--`. Its preflight stages the clone's images in STORE, as + `stage` stages them, and prints `preflight=staged`. It then starts the clone + as `start` does, runs the execution in the clone's root as `exec` runs one, + and prints its output and outcome once it finished. Then it stops the clone + as `stop` does, whatever the execution did, which releases the lease once the + VM exited. `reattach`, `stop`, `probe`, and `exec` find a VM that booted cold under another runtime identity mismatched, and then signal nothing: the VM keeps @@ -1840,7 +1903,8 @@ that assembled the root), `stage` and `problem` (the stage of the root that failed, such as `assembly`, and the problem, or the problem of a mapping that failed), `table` (`applied` or `failed`: how the guest took the mapping table), `mapped` (how many of the table's mappings the guest applied), `sandbox` -(`started` once the guest started the sandbox), `PROBE.KEY` (the fact KEY that +(`started` once the guest started the sandbox), `preflight` (`staged` once +`run` staged the images of its clone), `PROBE.KEY` (the fact KEY that the probe named PROBE found), `exec.N.KEY` (what execution N did: its `launch`, `started`, `failed STAGE ERRNO`, or `none`; its `status`, `exit CODE`, `signal NUMBER`, or `none`; the `cause` for which a side ended it, such as @@ -1855,18 +1919,20 @@ completed, which the outcome report records apart from the root's), `created` (`captured` once a capture made the template's generation, as a refresh's always does, or `existing` once the store held it), `generation` (the generation's number in the template store, which `create` or `refresh` served, -`inspect` reports, or `start` leased), `key` and `class` (the identities of its +`inspect` reports, or `start` or `run` leased), `key` and `class` (the identities of its compatibility key and of its capture class, in hexadecimal), `retired` and `draining` (the generations that the eviction of a creation or a refresh retired: those of the class that are gone, with those whose interrupted retirement it completed, and those that leases keep, each list separated by commas, or `none`), -`lease` (the identity of the lease that `start` took, 32 hexadecimal digits), +`lease` (the identity of the lease that `start` or `run` took, 32 hexadecimal +digits), `downtime` and `source` (the downtime of the leased generation's capture at the host's time, in nanoseconds, and the clock that measured it, `monotonic` or -`utc`), `released` (the identity of the lease that a stop, or a start that -failed, released), `failure.KEY` (the record of a clone's failure, or of a -start's selection that the template store refused: its +`utc`), `released` (the identity of the lease that a stop, a start that failed, +or the teardown of a run released), `failure.KEY` (the record of a clone's +failure, or of the selection of a start or a run that the template store +refused: its `outcome`, what the failure is attributed to, `request`, `staging`, `host`, `unknown`, `template`, or `repair`, or `no-template`, `expired`, or `ineligible` for a refused selection; its `evidence`, what the host saw of it, @@ -1902,7 +1968,8 @@ leases that keep it). Values escape `\`, newlines, and carriage returns as `\\`, `\n`, and `\r`, and each byte of a program's output that is not part of UTF-8 text as `\xHH`. The commands exit with 0 once they did what they were asked, `exec` once every -execution finished, whatever its outcome, with 3 when they find a VM that +execution finished, whatever its outcome, and `run` once its execution finished +so, and the clone stopped as `stop` stops one, with 3 when they find a VM that booted cold mismatched, with 4 when the guest reported that a stage of the root failed, and the VM ended with the stage's status and a complete teardown, with 5 when the [step hook](#the-step-hook) ended a command, with 6 when the guest @@ -1911,10 +1978,13 @@ and a complete teardown, with 7 when `remove` found leases that keep the generation, which it tombstoned, with 8 when a clone failed, once nothing of it runs and the command released its lease, with 9 when a clone failed, and its teardown is unresolved, so that the sandbox keeps the lease, with 10 when the -template store refused a start's selection, so that the start leased and -launched nothing, and with 1, after +template store refused the selection of a start or a run, so that it leased +and launched nothing, and with 1, after a diagnostic on standard error, when they -fail. A value of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time is a +fail, as `run` does once its execution failed to run, after its teardown, +unless the teardown ended with 4, 8, or 9, which `run` then exits with, after +the diagnostic. A +value of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time is a usage error, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, with status 2. Without arguments, the program reports the contract that it was built against. @@ -2072,7 +2142,11 @@ Windows. They cover: the run; a timeout that fires while the guest says nothing, after which the execution is cancelled, then fenced; and a run whose deadline passes first, in a receive, which leaves the client usable, or in a send that waits for a - guest that stopped reading, which ends it. + guest that stopped reading, which ends it; and a stop, once the driver gave + up on an execution, that skips the guest's late messages about it, before + and after a teardown that failed, while `stop` takes the first for its + answer, and neither skips a message about an execution that the driver did + not start. Through a relay between its test broker and a pseudo-terminal, `tests/guest_session.rs` also checks that `guest::serve_terminal` and @@ -2301,6 +2375,17 @@ Windows, without a hypervisor: - a start whose network cannot be read, or is one that the plan refuses, with status 1 before anything else: before its image, which does not exist, its shares, which it cannot export, and the template store, which does not exist; +- the command lines of `run` that the host program refuses, each with status + 2: missing options, a missing `--` or program, the qualifiers of an identity + without one, an execution's start that the guest would refuse, invalid + values, arguments before `--`, and the options of other commands, with a + clock fixture that is not a host's time too; +- a run whose image the plan refuses, with status 1, before it stages anything + or creates the sandbox's store, and runs whose template store does not exist, + or does not hold the generation, each with status 10 and the preflight's + `staged` before the record of a selection that names no template, once the + preflight staged the clone's images, before it needs NVX's artifacts, and + without a lease; - the text form of a host's time, which `RUNTIME_ADAPTER_HOST_TIME` takes, written and read back on both clocks, and the texts that are none; - a clone's record: its JSON, with the plan that `layered::LaunchRecord` @@ -2691,7 +2776,8 @@ generations and payloads that the store then holds. ### Clone scenarios `tests/vm_clone.rs` starts [clones](#clones) of templates on a real hypervisor, -in the host-only configuration, through `runtime-adapter-host start`. Each test +in the host-only configuration, through `runtime-adapter-host start` and +`run`. Each test captures a template of its own, of 1 processor and 256 MiB of RAM with a capacity of 1 GiB, in a template store of its own, of a copy of NVX's Ubuntu EROFS image that a directory of the platform holds, which the clones stage as @@ -2735,6 +2821,8 @@ directory that the starts authorize: | `a_clone_s_executions_find_the_proxy_of_its_policy` | Of a template of a proxy-only network, one, with an execution as user ID 0 that runs `env`, then one that names its own `HTTPS_PROXY` | A record that names the proxy's URL, `http://192.0.2.1:PORT`, at the port of the proxy's listener, and no DNS server; the first execution with its own entry, then the URL in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`; the second with exactly its own environment; one network device, and the image's own `/etc/resolv.conf`, which no bind holds | | `a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver` | Of a template of a network with egress allowed, one with a generated machine ID, sharing a host directory read-write | Started, with one mapping, and a record that names the export and the gateway's DNS server; on Linux, the VMM's one `--mount`, read-write, and its arguments of the template's policy; below the root, the binds of the machine ID and of the resolver configuration, then the export's mapping; the export's directory, and a write through it, which the host's directory holds after the stop | | `a_selection_that_names_no_template_leases_nothing` | A start that names a template store that does not exist, then, once `remove` retired the generation, which no lease kept, a start of it | Status 10, with the record of a selection that names no template, by the evidence `no-store`, then `retired`, without a lease or a VM; and a store that indexes no generation | +| `a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down` | A run over a scratch image, as nobody, whose execution names its environment and its working directory, and ends with status 3, then a run in RAM, as nobody, whose execution names an identity of its own | Each with status 0, in one command: the preflight's `staged`, the clone started as a start starts one, the execution, and a stop that releases the lease, without a VM; the first execution's output, of the Ubuntu image's root, as nobody, in its working directory, with its environment, and its status; the second refused with `EINVAL`; no lease left, and the payload with nothing but the capture's files | +| `a_run_whose_clone_does_not_start_runs_nothing` | A run that names a template store that does not exist, one with a clock fixture past the restore window, and one over a scratch image of zeros | Once the preflight staged the images, status 10, with the record of a selection that names no template, by the evidence `no-store`, then of an `expired` generation, without a lease or a VM; then status 8, with the record of a failure of the request at the activation, once the VM exited, the lease released, and no execution or stop | Every network that the scenarios give a template is one that a template takes: a network that lets nothing through, egress allowed, with outbound diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 31edf27ee..9569b5c0c 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -228,6 +228,26 @@ //! the broker instance's ID, as a little-endian number. The guest refuses a key that it heard of //! before, and the epochs of a broker instance only advance, while a broker restored from a //! snapshot has a new instance ID, so no command repeats the key of another's execution. +//! - `runtime-adapter-host run STORE --template TEMPLATES --generation G --distro FORMAT:PATH +//! --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity +//! UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] EXECUTION` runs one execution in a +//! clone of the generation G of the template store TEMPLATES, in the sandbox store STORE, which +//! it creates if needed, then tears the clone down. The EXECUTION is one of `exec`'s, whose +//! options may come among the start's, in any order, and whose PROGRAM and ARGs are every +//! argument after `--`. The command first checks that the guest takes the execution's start. +//! Its preflight then stages the clone's images in STORE, as `stage` stages them, which verifies +//! the image, and hashes it only when a manifest is beside it, and prints that it did, before +//! the clone's start, which hashes nothing. It starts the clone as `start` does, and ends as a +//! start ends once the template store refuses the selection, or the clone fails. It runs the +//! execution in the clone's root, as `exec` runs one, through the guest agent's execution +//! control, in the session in which the clone's sandbox started, so that the program, its +//! arguments, and its environment reach the guest in the execution's start, never in the +//! template's boot tokens, and prints its output and outcome once it finished. Then it stops +//! the clone, as `stop` does, whatever the execution did, which releases the lease once the VM +//! exited. The stop takes the same session, in which it skips the guest's messages about the +//! execution that may still arrive once the command finished it without them, as when it gave +//! up waiting for its outcome. It holds the sandbox's lock, which the lease took, until then, so +//! no other command reattaches to the clone meanwhile. //! //! Each command but `stage`, `list`, `inspect`, and `remove`, which need no artifacts, takes NVX's //! artifacts from the directory that `NVX_ARTIFACTS_DIR` names, a repository checkout or an @@ -272,6 +292,7 @@ //! - `table`: how the guest took the mapping table: `applied` or `failed`; //! - `mapped`: how many of the table's mappings the guest applied; //! - `sandbox`: `started` once the guest started the sandbox; +//! - `preflight`: `staged` once `run` staged the images of its clone; //! - `PROBE.KEY`: the fact KEY that the probe named PROBE found; //! - `exec.N.KEY`: what execution N did, with these KEYs: //! - `launch`: `started`, `failed STAGE ERRNO`, with the number of the guest's stage that failed @@ -294,19 +315,20 @@ //! one that the store held; //! - `generation`, `key`, and `class`: the store generation of the template that `create` or //! `refresh` served, or that `inspect` reports, and the identities of its compatibility key and -//! of its capture class, each 64 hexadecimal digits, or the store generation that `start` -//! leased; +//! of its capture class, each 64 hexadecimal digits, or the store generation that `start` or +//! `run` leased; //! - `retired` and `draining`: the generations that the eviction of `create` or `refresh` retired, //! each list separated by commas, or `none`: those of the class that it deleted, or found gone //! already, with those whose interrupted retirement it completed, and those that leases keep, //! tombstoned; -//! - `lease`: the identity of the lease that `start` took, 32 hexadecimal digits; +//! - `lease`: the identity of the lease that `start` or `run` took, 32 hexadecimal digits; //! - `downtime` and `source`: the downtime of the leased generation's capture at the host's time, //! in nanoseconds, and the clock that measured it: `monotonic`, the host's monotonic clock, on //! the host's boot that captured the template, or `utc` otherwise; -//! - `released`: the identity of the lease that a stop, or a start that failed, released; -//! - `failure.KEY`: the record of a clone's failure, or of a start's selection that the template -//! store refused, with these KEYs: +//! - `released`: the identity of the lease that a stop, a start that failed, or the teardown of a +//! run released; +//! - `failure.KEY`: the record of a clone's failure, or of the selection of a start or a run that +//! the template store refused, with these KEYs: //! - `outcome`: what the failure is attributed to: `request`, `staging`, `host`, `unknown`, //! `template`, or `repair`, or, for a refused selection, `no-template`, `expired`, or //! `ineligible`; @@ -354,23 +376,27 @@ //! For tests, `launch`, `reattach`, and `stop` take a step hook from the variable //! `RUNTIME_ADAPTER_STEP_HOOK`, which pauses the command at a named step, for a time that a launch //! leaves out of its deadline, or exits it there without unwinding, with status 5; see -//! `runtime_adapter::hook`. `start` takes the +//! `runtime_adapter::hook`. `start` and `run` take the //! host's time, at which the store checks the generation's eligibility, from the variable //! `RUNTIME_ADAPTER_HOST_TIME`, if it is set, as a clock fixture, such as another boot's; see //! `runtime_adapter::clone::HOST_TIME_VARIABLE`. //! //! They exit with 0 once they did what they were asked, `exec` once every execution finished, -//! whatever its outcome, with 3 when reattach finds a VM that booted cold mismatched, with 4 when +//! whatever its outcome, and `run` once its execution finished so, and the clone stopped, as +//! `stop` stops one, with 3 when reattach finds a VM that booted cold mismatched, with 4 when //! the guest reported that a stage of the root failed, and the VM ended with the stage's status //! and a complete teardown, with 5 when the step hook ends a command, with 6 when the guest failed //! a mapping of the table, and the VM stopped gracefully, with success and a complete teardown, //! with 7 when `remove` finds leases that keep the generation, which it tombstoned, //! with 8 when a clone failed, once nothing of it runs and the command released its lease, with 9 //! when a clone failed, and its teardown is unresolved, so that the sandbox keeps the lease, with -//! 10 when the template store refused a start's selection, so that the start leased and launched -//! nothing, and with 1, after a diagnostic on standard error, when they fail. Anything else is a -//! usage error, with status 2, as is a value of `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, -//! or of `RUNTIME_ADAPTER_HOST_TIME` that is not a host's time. +//! 10 when the template store refused the selection of a start or a run, so that it leased and +//! launched nothing, and with 1, after a diagnostic on standard error, when they fail, as `run` +//! does once its execution failed to run, after the teardown, unless the teardown ended with 4, 8, +//! or 9, as when the clone's VM exited during the execution, which `run` then exits with, after +//! the diagnostic. Anything else is a usage error, with status 2, as is a value of +//! `RUNTIME_ADAPTER_STEP_HOOK` that sets no step hook, or of `RUNTIME_ADAPTER_HOST_TIME` that is +//! not a host's time. use std::env; use std::error::Error; @@ -390,6 +416,7 @@ use nvx::host::instance::generate_capability; use nvx::host::launch::{DEFAULT_MEMORY_MIB, LaunchSpec}; use nvx::host::nvxs::HostConnectionError; use nvx::host::report::{self, Report}; +use nvx::host::sandbox::SandboxPlan; use nvx::host::session; use nvx::host::store::{CallerFiles, IdentityPart, LOG_NAME, LaunchId, LockedStore, SandboxStore}; use nvx::host::supervisor::{self, Reattached, Starting, StopOutcome, Vmm, VmmCommand}; @@ -488,6 +515,15 @@ const EXEC_INPUT_FRAMES: u32 = 4; const EXEC_OUTPUT_FRAMES: u32 = 8; /// The argument that ends an execution's arguments, after which another execution may follow. const EXEC_END: &str = ";"; +/// The options of an execution, which `run` takes among those of a start. +const EXECUTION_OPTIONS: [&str; 6] = [ + "--identity", + "--groups", + "--capabilities", + "--numeric", + "--cwd", + "--env", +]; /// The command lines that the program accepts. const USAGE: &str = concat!( "usage: runtime-adapter-host\n", @@ -522,6 +558,16 @@ const USAGE: &str = concat!( " [--network FILE]\n", " [--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]...", " [--share-root DIR]...]\n", + " runtime-adapter-host run STORE --template TEMPLATES --generation G", + " --distro FORMAT:PATH\n", + " --distro-identity UUID --upper (scratch:PATH | ram) [--guest-build BUILD]", + " [--memory MIB]\n", + " [--workload-identity UID:GID] [--overlay-mode MODE]", + " [--scratch-journal POLICY]\n", + " [--first-partition-bound BYTES] [--machine-id (image | generated)]", + " [--network FILE]\n", + " [--share HOST:TARGET[:ro]... [--deny PATH]... [--reserved TARGET]...", + " [--share-root DIR]...] EXECUTION\n", " runtime-adapter-host probe STORE PROBE... [--guest-build BUILD]\n", " runtime-adapter-host exec STORE [--guest-build BUILD] EXECUTION [';' EXECUTION]...\n", " EXECUTION: [--identity UID:GID [--groups GID,...] [--capabilities N,...]", @@ -621,6 +667,20 @@ enum Command { /// The host's time, for tests, if `RUNTIME_ADAPTER_HOST_TIME` sets one. time: Option, }, + /// Start a clone of a generation of a template, run one execution in its root, and tear it + /// down. + Run { + /// The sandbox's store. + store: PathBuf, + /// The guest agent's build. + guest_build: String, + /// The clone. + request: Box, + /// The execution. + execution: Box, + /// The host's time, for tests, if `RUNTIME_ADAPTER_HOST_TIME` sets one. + time: Option, + }, /// Reattach to the VM, and run probes in its layered root. Probe { /// The sandbox's store. @@ -641,7 +701,7 @@ enum Command { }, } -/// An execution of `exec`, as its options and arguments give it. +/// An execution of `exec` or `run`, as its options and arguments give it. #[derive(Default)] struct Execution { /// `--identity`: the user ID and the group ID. @@ -864,7 +924,9 @@ fn main() -> ExitCode { format, identity, scratch, - } => stage(&store, &distro, format, identity, scratch.as_deref()), + } => { + stage(&store, &distro, format, identity, scratch.as_deref()).map(|()| ExitCode::SUCCESS) + } Command::Launch { store, guest_agent, @@ -917,6 +979,13 @@ fn main() -> ExitCode { request, time, } => start(&store, &guest_build, &request, time), + Command::Run { + store, + guest_build, + request, + execution, + time, + } => run(&store, &guest_build, &request, &execution, time), Command::Probe { store, guest_build, @@ -944,7 +1013,7 @@ fn parse(arguments: Vec) -> Result { let name = match name.to_str() { Some( name @ ("stage" | "launch" | "create" | "refresh" | "list" | "inspect" | "remove" - | "reattach" | "stop" | "start" | "probe" | "exec"), + | "reattach" | "stop" | "start" | "run" | "probe" | "exec"), ) => name.to_owned(), _ => return Err(format!("unknown command {name:?}")), }; @@ -957,6 +1026,9 @@ fn parse(arguments: Vec) -> Result { if name == "exec" { return parse_exec(store, arguments); } + if name == "run" { + return parse_run(store, arguments); + } if matches!(name.as_str(), "list" | "inspect" | "remove") { return parse_generations(&name, store, arguments); } @@ -980,6 +1052,7 @@ fn parse(arguments: Vec) -> Result { } let guest_build = options .guest_build + .take() .unwrap_or_else(|| GUEST_BUILD.to_owned()); Ok(match name.as_str() { "stage" => { @@ -1090,44 +1163,12 @@ fn parse(arguments: Vec) -> Result { }, } } - "start" => { - let templates = options.template.ok_or("start needs --template")?; - let generation = options.generation.ok_or("start needs --generation")?; - let (format, distro) = options.distro.ok_or("start needs --distro")?; - let identity = options.identity.ok_or("start needs --distro-identity")?; - let scratch = options.upper.ok_or("start needs --upper")?; - if scratch.is_none() && options.scratch_journal.is_some() { - return Err("--scratch-journal needs --upper scratch:PATH".to_owned()); - } - if format == ImageFormat::Erofs && options.first_partition_bound.is_some() { - return Err("--first-partition-bound needs --distro gpt:PATH".to_owned()); - } - check_shares(&options.sharing)?; - Command::Start { - store, - guest_build, - request: Box::new(CloneOptions { - templates, - generation, - request: runtime_adapter::clone::Request { - distro, - format, - identity, - scratch, - memory: options.memory, - workload_identity: options.workload_identity, - overlay_mode: options.overlay_mode, - scratch_journal: options.scratch_journal, - first_partition_bound: options.first_partition_bound, - machine_id: options.machine_id.unwrap_or(MachineIdMode::Image), - sharing: options.sharing, - network: None, - }, - network: options.network, - }), - time: host_time_from_env()?, - } - } + "start" => Command::Start { + store, + guest_build, + request: Box::new(clone_options(&name, options)?), + time: host_time_from_env()?, + }, "probe" if probes.is_empty() => return Err("probe needs a probe".to_owned()), "probe" => Command::Probe { store, @@ -1157,8 +1198,12 @@ fn parse_option( .into_string() .map_err(|value| format!("the value {value:?} of {option} is not Unicode")) }; - // A refresh takes the options of a creation. - let command = if name == "refresh" { "create" } else { name }; + // A refresh takes the options of a creation, and a run those of a start. + let command = match name { + "refresh" => "create", + "run" => "start", + name => name, + }; match (command, option) { ("launch" | "create", "--guest-agent") if options.guest_agent.is_none() => { options.guest_agent = Some(PathBuf::from(value)); @@ -1362,38 +1407,7 @@ fn parse_exec( .ok_or_else(|| format!("{option:?} needs a value"))??; match option.as_str() { "--guest-build" if guest_build.is_none() => guest_build = Some(value), - "--identity" if execution.identity.is_none() => { - let identity = ids(&value) - .ok_or_else(|| format!("--identity takes UID:GID, not {value:?}"))?; - execution.identity = Some(identity); - } - "--groups" if execution.groups.is_none() => { - let groups = - numbers(&value, |group| group.parse::().ok()).ok_or_else(|| { - format!("--groups takes GIDs, separated by commas, not {value:?}") - })?; - execution.groups = Some(groups); - } - "--capabilities" if execution.capabilities.is_none() => { - let capabilities = numbers(&value, |number| { - number.parse::().ok().filter(|number| *number < 64) - }) - .ok_or_else(|| { - format!( - "--capabilities takes capability numbers from 0 to 63, separated by \ - commas, not {value:?}" - ) - })?; - let set = capabilities - .into_iter() - .fold(0, |set, number| set | 1 << number); - execution.capabilities = Some(set); - } - "--cwd" if execution.working_directory.is_none() => { - execution.working_directory = Some(value); - } - "--env" => execution.environment.push(value), - _ => return Err(format!("unexpected option {option:?} of exec")), + _ => execution_option(&mut execution, "exec", &option, value)?, } } let mut more = false; @@ -1405,24 +1419,7 @@ fn parse_exec( } execution.arguments.push(argument); } - if execution.arguments.is_empty() { - return Err("exec needs a program after `--`".to_owned()); - } - if execution.identity.is_none() { - let qualifiers = [ - ("--groups", execution.groups.is_some()), - ("--capabilities", execution.capabilities.is_some()), - ("--numeric", execution.numeric), - ]; - if let Some((option, _)) = qualifiers.iter().find(|(_, given)| *given) { - return Err(format!("{option} needs --identity")); - } - } - // The guest ends its session on a start that it cannot decode, so the program refuses - // one that breaks the rules of the encoding, as a relative working directory or an - // environment entry without a name does, or that the framing does not carry, before it - // attaches. - check_start(&execution.start(Key::new(0, 0)))?; + check_execution(&execution, "exec")?; executions.push(execution); if !more { break; @@ -1443,6 +1440,177 @@ fn parse_exec( }) } +/// Parses the option `option` of an execution of the command `name`, `exec` or `run`, with +/// `value`, into `execution`, or says why it is not one of an execution's options that take a +/// value, or why its value is invalid. +fn execution_option( + execution: &mut Execution, + name: &str, + option: &str, + value: String, +) -> Result<(), String> { + match option { + "--identity" if execution.identity.is_none() => { + let identity = + ids(&value).ok_or_else(|| format!("--identity takes UID:GID, not {value:?}"))?; + execution.identity = Some(identity); + } + "--groups" if execution.groups.is_none() => { + let groups = numbers(&value, |group| group.parse::().ok()).ok_or_else(|| { + format!("--groups takes GIDs, separated by commas, not {value:?}") + })?; + execution.groups = Some(groups); + } + "--capabilities" if execution.capabilities.is_none() => { + let capabilities = numbers(&value, |number| { + number.parse::().ok().filter(|number| *number < 64) + }) + .ok_or_else(|| { + format!( + "--capabilities takes capability numbers from 0 to 63, separated by commas, \ + not {value:?}" + ) + })?; + let set = capabilities + .into_iter() + .fold(0, |set, number| set | 1 << number); + execution.capabilities = Some(set); + } + "--cwd" if execution.working_directory.is_none() => { + execution.working_directory = Some(value); + } + "--env" => execution.environment.push(value), + _ => return Err(format!("unexpected option {option:?} of {name}")), + } + Ok(()) +} + +/// Checks the execution `execution` of the command `name`, once its arguments are parsed: that it +/// names a program, that it names an identity if it qualifies one, and that the guest takes its +/// start, or says why not. +fn check_execution(execution: &Execution, name: &str) -> Result<(), String> { + if execution.arguments.is_empty() { + return Err(format!("{name} needs a program after `--`")); + } + if execution.identity.is_none() { + let qualifiers = [ + ("--groups", execution.groups.is_some()), + ("--capabilities", execution.capabilities.is_some()), + ("--numeric", execution.numeric), + ]; + if let Some((option, _)) = qualifiers.iter().find(|(_, given)| *given) { + return Err(format!("{option} needs --identity")); + } + } + // The guest ends its session on a start that it cannot decode, so the program refuses one + // that breaks the rules of the encoding, as a relative working directory or an environment + // entry without a name does, or that the framing does not carry, before it attaches. + check_start(&execution.start(Key::new(0, 0))) +} + +/// Parses the arguments of `run` after its sandbox store `store`: the options of `start`, and +/// those of one execution, as `exec` takes them, in any order, then `--`, and the execution's +/// program and its arguments, every one that follows, or says why they are not the command's. +fn parse_run( + store: PathBuf, + mut arguments: impl Iterator, +) -> Result { + let mut options = Options::default(); + let mut execution = Execution::default(); + loop { + let argument = arguments + .next() + .ok_or("run needs `-- PROGRAM` after its options")?; + let Some(option) = argument.to_str().filter(|option| option.starts_with("--")) else { + return Err(format!("unexpected argument {argument:?} of run")); + }; + match option { + "--" => break, + "--numeric" if !execution.numeric => { + execution.numeric = true; + continue; + } + _ => {} + } + let option = option.to_owned(); + let value = arguments + .next() + .ok_or_else(|| format!("{option:?} needs a value"))?; + if EXECUTION_OPTIONS.contains(&option.as_str()) { + let value = value + .into_string() + .map_err(|value| format!("the value {value:?} of {option} is not Unicode"))?; + execution_option(&mut execution, "run", &option, value)?; + } else { + parse_option(&mut options, "run", &option, value)?; + } + } + for argument in arguments { + let argument = argument + .into_string() + .map_err(|argument| format!("the argument {argument:?} of run is not Unicode"))?; + execution.arguments.push(argument); + } + check_execution(&execution, "run")?; + let guest_build = options + .guest_build + .take() + .unwrap_or_else(|| GUEST_BUILD.to_owned()); + Ok(Command::Run { + store, + guest_build, + request: Box::new(clone_options("run", options)?), + execution: Box::new(execution), + time: host_time_from_env()?, + }) +} + +/// Returns the clone that the options `options` of `start` or `run`, the command `name`, select +/// and describe, or says why they do not. +fn clone_options(name: &str, options: Options) -> Result { + let templates = options + .template + .ok_or_else(|| format!("{name} needs --template"))?; + let generation = options + .generation + .ok_or_else(|| format!("{name} needs --generation"))?; + let (format, distro) = options + .distro + .ok_or_else(|| format!("{name} needs --distro"))?; + let identity = options + .identity + .ok_or_else(|| format!("{name} needs --distro-identity"))?; + let scratch = options + .upper + .ok_or_else(|| format!("{name} needs --upper"))?; + if scratch.is_none() && options.scratch_journal.is_some() { + return Err("--scratch-journal needs --upper scratch:PATH".to_owned()); + } + if format == ImageFormat::Erofs && options.first_partition_bound.is_some() { + return Err("--first-partition-bound needs --distro gpt:PATH".to_owned()); + } + check_shares(&options.sharing)?; + Ok(CloneOptions { + templates, + generation, + request: runtime_adapter::clone::Request { + distro, + format, + identity, + scratch, + memory: options.memory, + workload_identity: options.workload_identity, + overlay_mode: options.overlay_mode, + scratch_journal: options.scratch_journal, + first_partition_bound: options.first_partition_bound, + machine_id: options.machine_id.unwrap_or(MachineIdMode::Image), + sharing: options.sharing, + network: None, + }, + network: options.network, + }) +} + /// Parses the arguments of `list`, `inspect`, or `remove`, the command `name`, after its template /// store `store`: none for `list`, and the store generation of a generation, a positive number, /// for `inspect` and `remove`, or says why they are not the command's. @@ -1515,7 +1683,7 @@ fn stage( format: ImageFormat, identity: Uuid, scratch: Option<&Path>, -) -> Result { +) -> Result<(), String> { // Verification may hash the image, so it runs before the sandbox is locked. let staged = Staged::stage(distro, format, identity, scratch).map_err(|error| describe(&error))?; @@ -1523,10 +1691,7 @@ fn stage( let _locked = store .lock(Some(Instant::now() + LOCK_TIMEOUT)) .map_err(|error| describe(&error))?; - staged - .write(store.dir()) - .map_err(|error| describe(&error))?; - Ok(ExitCode::SUCCESS) + staged.write(store.dir()).map_err(|error| describe(&error)) } /// Launches a VM of the guest agent `guest_agent` in the sandbox store `store`, on `hypervisor` @@ -2121,13 +2286,48 @@ impl Drop for TemporaryDirectory { /// Starts a clone of the generation of the template store that `clone` names, as it asks, in the /// sandbox store `store`, which it creates if needed, with the guest agent's build `guest_build`, -/// at the host's time `time`, or at the time that `HostTime::now` samples without one. +/// at the host's time `time`, or at the time that `HostTime::now` samples without one, as +/// [`start_clone`] does, and leaves it running. fn start( store: &Path, guest_build: &str, clone: &CloneOptions, time: Option, ) -> Result { + // The clone keeps running once the command lets go of its VM, its session, and its sandbox's + // lock, as once a launch committed. + Ok(match start_clone(store, guest_build, clone, time)? { + Ok(_running) => ExitCode::SUCCESS, + Err(status) => status, + }) +} + +/// A clone that [`start_clone`] started, whose launch it committed, and which runs. +struct Running { + /// The runtime of NVX's artifacts, under which the clone runs. + runtime: Runtime, + /// The clone's sandbox store. + sandbox: SandboxStore, + /// The sandbox's lock, which the clone's lease holds. + locked: LockedStore, + /// The clone's VM. + vmm: Vmm, + /// The session of the adapter's clone order in which the clone's sandbox started. + client: Client, +} + +/// Starts a clone of the generation of the template store that `clone` names, as it asks, in the +/// sandbox store `store`, which it creates if needed, with the guest agent's build `guest_build`, +/// at the host's time `time`, or at the time that `HostTime::now` samples without one, and prints +/// what it did. Returns the clone, which runs, once it committed the clone's launch, or the status +/// with which the command ends otherwise: once the template store refused the selection, or the +/// clone failed, as [`clone_failed`] ends it. +fn start_clone( + store: &Path, + guest_build: &str, + clone: &CloneOptions, + time: Option, +) -> Result, String> { // A network that the plan refuses fails the start before anything else, as it fails a launch. // NVX's plan of the clone then requires the template's network of it. let network = clone @@ -2148,7 +2348,10 @@ fn start( let templates = match Store::open(&clone.templates) { Ok(templates) => templates, Err(error) if error.kind() == ErrorKind::NotFound => { - return Ok(selection_refused(Refusal::NoTemplate("no-store"), &error)); + return Ok(Err(selection_refused( + Refusal::NoTemplate("no-store"), + &error, + ))); } Err(error) => return Err(describe(&error)), }; @@ -2166,7 +2369,10 @@ fn start( templates.dir().display(), clone.generation )); - return Ok(selection_refused(Refusal::NoTemplate(evidence), &error)); + return Ok(Err(selection_refused( + Refusal::NoTemplate(evidence), + &error, + ))); }; // The clone's memory target is the request's, or else the template's RAM, which the // generation's key gives. A key that cannot be read gives none: NVX's plan reads the key @@ -2197,7 +2403,7 @@ fn start( Ok(acquired) => acquired, Err(error) => { return match Refusal::of(&error, &generation, &now) { - Some(refusal) => Ok(selection_refused(refusal, &error)), + Some(refusal) => Ok(Err(selection_refused(refusal, &error))), None => Err(describe(&error)), }; } @@ -2218,8 +2424,16 @@ fn start( guest_build, }; match clone.run(&mut acquired) { - Ok(()) => Ok(ExitCode::SUCCESS), - Err(Failed::Clone(failure)) => clone_failed(Ok(&templates), acquired.locked, failure), + Ok((vmm, client)) => Ok(Ok(Running { + runtime, + sandbox, + locked: acquired.locked, + vmm, + client, + })), + Err(Failed::Clone(failure)) => { + clone_failed(Ok(&templates), acquired.locked, failure).map(Err) + } Err(Failed::Command(problem)) => { // The command failed before the clone's launch, so nothing of it runs. match release_lease(&templates, &mut acquired.locked) { @@ -2321,14 +2535,15 @@ impl CloneLaunch<'_> { /// Plans, records, and launches the clone in the sandbox of `acquired`, which leased the /// generation, activates it, maps its export's table, if it has one, starts its sandbox, /// exchanges a `Ping` and the longest `Echo` with its guest agent, and commits the launch, - /// leaving the clone running. + /// leaving the clone running. Returns the clone's VM, and the session of the adapter's order + /// in which its sandbox started. /// /// Fails with the clone's failure, as NVX's plan, launch, restore sequence, and commit return /// it, or as `Started::fail` records it for a guest agent that does not answer the command /// once the clone's sandbox started, which ended the launch, unless its teardown is /// unresolved, and with the problem of a step of the adapter's own before the launch /// otherwise. - fn run(self, acquired: &mut Acquired) -> Result<(), Failed> { + fn run(self, acquired: &mut Acquired) -> Result<(Vmm, Client), Failed> { let Self { templates, request, @@ -2389,8 +2604,7 @@ impl CloneLaunch<'_> { let (evidence, error) = unanswered(&order, error, deadline); return Err(Failed::Clone(started.fail(evidence, error))); } - started.commit().map_err(Failed::Clone)?; - Ok(()) + started.commit().map_err(Failed::Clone) } } @@ -2807,23 +3021,74 @@ fn reattach( } let Found { store, - mut locked, + locked, vmm, leased, .. } = found; + let clone = leased.then_some(CloneStop { + runtime: &runtime, + guest_build, + templates, + }); + stop_vm(&store, locked, vmm, clone, |deadline| { + attached? + .stop(deadline) + .map_err(|error| unconfirmed(&error)) + }) +} + +/// Returns the problem of a stop whose guest agent did not confirm it, as the client failed with +/// `error`. +fn unconfirmed(error: &ClientError) -> String { + format!( + "the guest agent did not confirm the stop: {}", + describe(error) + ) +} + +/// What a command that stops the VM of a clone, as [`stop_vm`] stops it, needs to end the clone: +/// the runtime and the guest agent's build under which it reattaches to the clone once its VM +/// exited, as [`reattach_leased`] does, and the template store that the command names, if any, in +/// place of the one that the clone's record names, through which it releases the clone's lease. +#[derive(Clone, Copy)] +struct CloneStop<'a> { + /// The runtime of NVX's artifacts. + runtime: &'a Runtime, + /// The guest agent's build. + guest_build: &'a str, + /// The template store that the command names, if any. + templates: Option<&'a Path>, +} + +/// Stops the VM `vmm` of the sandbox store `store`, whose lock `locked` the command holds, and +/// prints how it stopped. Its graceful step, `confirm`, asks the guest agent to stop by the +/// deadline that it takes, and returns, once the guest agent confirmed the stop, the failure of +/// the root's teardown that the guest agent reported first, if any. A VM whose guest agent cannot +/// confirm the stop is terminated. +/// +/// The VM of a clone, which `clone` ends, stops as [`reattach`] describes: with +/// `Vmm::stop_unless_exited`, after which the command reads how the clone ended from its records, +/// and releases its lease once the clone ended through the stop, or ends as [`conclude`] does +/// otherwise. A VM that booted cold stops with `Vmm::stop`. +/// +/// Returns success once the VM exited, as its guest agent confirmed, with the outcome `success 0` +/// and a complete teardown, and [`ROOT_FAILED`] once the guest agent reported the failure of its +/// root, which it prints, and fails otherwise. +fn stop_vm( + store: &SandboxStore, + mut locked: LockedStore, + vmm: Vmm, + clone: Option>, + confirm: impl FnOnce(Instant) -> Result, String>, +) -> Result { // A VM whose guest agent cannot confirm the stop is terminated. let mut failure = None; let graceful = |_: &Vmm, deadline: Instant| -> Result<(), String> { - failure = attached?.stop(deadline).map_err(|error| { - format!( - "the guest agent did not confirm the stop: {}", - describe(&error) - ) - })?; + failure = confirm(deadline)?; Ok(()) }; - let stopped = if leased { + let stopped = if clone.is_some() { // A clone's VM that had exited when the stop began ended without its holder: the stop // stops nothing then, and keeps the runtime record of its launch, which tells so. vmm.stop_unless_exited( @@ -2844,8 +3109,9 @@ fn reattach( .map_err(|error| describe(&error))?; let Some(stopped) = stopped else { println!("stopped=exited"); - let reattached = reattach_leased(&mut locked, &runtime, guest_build)?; - return conclude(reattached, locked, templates); + let clone = clone.expect("only the stop of a clone's VM stops nothing"); + let reattached = reattach_leased(&mut locked, clone.runtime, clone.guest_build)?; + return conclude(reattached, locked, clone.templates); }; println!( "stopped={}", @@ -2856,13 +3122,15 @@ fn reattach( } ); // The store verified that the VM exited, unless the stop timed out. - if leased && stopped.outcome != StopOutcome::TimedOut { - match reattach_leased(&mut locked, &runtime, guest_build)? { + if let Some(clone) = clone + && stopped.outcome != StopOutcome::TimedOut + { + match reattach_leased(&mut locked, clone.runtime, clone.guest_build)? { ReattachedClone::Ended { .. } => { - let templates = clone_templates(&locked, templates)?; + let templates = clone_templates(&locked, clone.templates)?; release_lease(&templates, &mut locked)?; } - reattached => return conclude(reattached, locked, templates), + reattached => return conclude(reattached, locked, clone.templates), } } if let Some(problem) = stopped.graceful_error { @@ -2873,12 +3141,12 @@ fn reattach( } match failure { None => { - check_outcome(&store, "success", 0)?; + check_outcome(store, "success", 0)?; Ok(ExitCode::SUCCESS) } Some(failure) => { print_failure(&failure); - check_outcome(&store, "guest-exit", failure.stage().exit_status().into())?; + check_outcome(store, "guest-exit", failure.stage().exit_status().into())?; Ok(ExitCode::from(ROOT_FAILED)) } } @@ -2926,10 +3194,21 @@ fn exec(store: &Path, guest_build: &str, executions: &[Execution]) -> Result) -> Result, String> { let mut starts = Vec::with_capacity(executions.len()); for (number, execution) in (1..).zip(executions) { let start = execution.start(Key::new(0, 0)); - let start = match &plan { + let start = match plan { Some(plan) => with_proxy(start, plan), None => start, }; @@ -2938,19 +3217,28 @@ fn exec(store: &Path, guest_build: &str, executions: &[Execution]) -> Result, + deadline: Instant, +) -> Result<(), String> { // The keys of the attachment's executions are new to the guest: see the module documentation. let identity = client.identity(); let (base, _) = identity.instance_id.split_at(8); let base = u64::from_le_bytes(base.try_into().expect("an instance ID has 16 bytes")); - let mut running = Executions::new(); let mut ids = Vec::new(); for (number, mut start) in (1..).zip(starts) { let key = Key::new(base.wrapping_add(number), identity.epoch); start.key = key; running - .start(&mut client, start, exec_timers(), deadline) + .start(client, start, exec_timers(), deadline) .map_err(|error| format!("cannot start execution {number}: {}", describe(&error)))?; ids.push(key.id()); } @@ -2958,13 +3246,102 @@ fn exec(store: &Path, guest_build: &str, executions: &[Execution]) -> Result, +) -> Result { + // A start that the guest would refuse fails the command before it stages anything. + let plan = clone.request.plan().map_err(|error| describe(&error))?; + let starts = starts_of(std::slice::from_ref(execution), Some(&plan))?; + let request = &clone.request; + stage( + store, + &request.distro, + request.format, + request.identity, + request.scratch.as_deref(), + )?; + println!("preflight=staged"); + let Running { + runtime, + sandbox, + locked, + vmm, + mut client, + } = match start_clone(store, guest_build, clone, time)? { + Ok(running) => running, + Err(status) => return Ok(status), + }; + let mut executions = Executions::new(); + let ran = run_executions( + &mut client, + &mut executions, + starts, + Instant::now() + SESSION_TIMEOUT, + ); + if ran.is_err() { + // The session may have ended with the VM, as when its VMM crashed. Once the VMM's exit is + // done, the stop finds that the VM exited without the command, and ends the clone as a + // reattach that finds it so does, rather than take the exit for its own. The stop checks + // the VMM itself, so a wait that fails changes nothing. + let _ = vmm.wait(Instant::now() + FORCED_TIMEOUT); + } + // The clone's record names the template store that releases its lease, as for `stop`. + let stop = CloneStop { + runtime: &runtime, + guest_build, + templates: None, + }; + let stopped = stop_vm(&sandbox, locked, vmm, Some(stop), |deadline| { + client + .stop_skipping(deadline, |message| executions.is_about(message)) + .map_err(|error| unconfirmed(&error)) + }); + match (ran, stopped) { + (Ok(()), stopped) => stopped, + // A stop that ended otherwise than in success, as one that found that the clone's VM + // exited during the execution, tells how the clone ended, so the command ends with its + // status, after the execution's diagnostic. + (Err(problem), Ok(status)) if status != ExitCode::SUCCESS => { + eprintln!("runtime-adapter-host: {problem}"); + Ok(status) + } + (Err(problem), Ok(_)) => Err(problem), + (Err(problem), Err(teardown)) => Err(format!("{problem}; {teardown}")), + } } /// Prints what the execution numbered `number`, `execution`, which finished, did. diff --git a/tests/runtime-adapter/src/host.rs b/tests/runtime-adapter/src/host.rs index e33a42b9c..9c601e917 100644 --- a/tests/runtime-adapter/src/host.rs +++ b/tests/runtime-adapter/src/host.rs @@ -364,18 +364,33 @@ impl Client { /// The guest agent of a VM with a layered root tears the root down first, and reports a /// teardown that failed with `RootFailed` before `Stopped`: the client returns that failure. pub fn stop(&mut self, deadline: Instant) -> Result, ClientError> { + self.stop_skipping(deadline, |_| false) + } + + /// Stops the guest agent as [`stop`](Self::stop) does, but skips the guest's messages before + /// `Stopped` that `late` accepts: those that the guest may still send, in this attachment, + /// about executions that a driver finished without them, as [`Executions::is_about`] accepts + /// them. Any other message answers the `Stop`, as it does for [`stop`](Self::stop). + pub fn stop_skipping( + &mut self, + deadline: Instant, + mut late: impl FnMut(&Message) -> bool, + ) -> Result, ClientError> { let request = Message::Stop; - let failure = match self.request(&request, deadline)? { - Message::Stopped => return Ok(None), - Message::RootFailed(body) => self.decoded(RootFailure::decode(&body))?, - reply => return Err(self.mismatched(&request, &reply)), - }; - match self - .receive(Some(deadline)) - .map_err(|error| self.fail(error))? - { - Message::Stopped => Ok(Some(failure)), - reply => Err(self.mismatched(&request, &reply)), + self.send(&request, Some(deadline))?; + let mut failure = None; + loop { + match self + .receive(Some(deadline)) + .map_err(|error| self.fail(error))? + { + Message::Stopped => return Ok(failure), + Message::RootFailed(body) if failure.is_none() => { + failure = Some(self.decoded(RootFailure::decode(&body))?); + } + reply if late(&reply) => {} + reply => return Err(self.mismatched(&request, &reply)), + } } } diff --git a/tests/runtime-adapter/src/host/executions.rs b/tests/runtime-adapter/src/host/executions.rs index bfd269eae..0a25a5c62 100644 --- a/tests/runtime-adapter/src/host/executions.rs +++ b/tests/runtime-adapter/src/host/executions.rs @@ -120,7 +120,9 @@ impl From for ExecutionsError { /// guest's messages to the executions, and theirs to the guest, until every execution finished. /// A driver runs one batch: it starts every execution before it runs them, and none after. The /// guest may still send messages about the executions of a run that gave up on one or failed, so -/// a later batch takes an attachment of its own, as each command of the host program does. +/// a later batch takes an attachment of its own, as each command of the host program does, and a +/// stop in the same attachment skips them, as [`Client::stop_skipping`] does with +/// [`is_about`](Self::is_about). /// /// One thread runs them all: each receive waits at most until the earliest timer of the /// executions, which so fires in time whatever the guest sends, and the guest's messages about one @@ -253,6 +255,20 @@ impl Executions { .all(|execution| execution.outcome().is_some()) } + /// Returns whether `message` is one of the guest's messages about an execution that the driver + /// started, of its key, whose body decodes: one that the guest may still send once the driver + /// finished the execution without it, as when it gave up waiting for the execution's outcome, + /// or once its run failed. + pub fn is_about(&self, message: &Message) -> bool { + let Some(Ok(decoded)) = GuestMessage::decode(message) else { + return false; + }; + let key = decoded.key(); + self.executions + .get(&key.id()) + .is_some_and(|execution| execution.start().key == key) + } + /// Hands the guest's `message` to the host's side of its execution at `now`. fn dispatch(&mut self, message: &Message, now: Instant) -> Result<(), ExecutionsError> { let decoded = GuestMessage::decode(message) diff --git a/tests/runtime-adapter/src/lib.rs b/tests/runtime-adapter/src/lib.rs index 16294802b..222f23096 100644 --- a/tests/runtime-adapter/src/lib.rs +++ b/tests/runtime-adapter/src/lib.rs @@ -55,9 +55,10 @@ //! `host` module's clone order is the framing through which the restore sequence of //! `nvx::host::template` activates a clone and starts its sandbox, and the host program's //! `runtime-adapter-host start` starts a clone of a generation in a sandbox of its own, which -//! leases the generation, in the `clone` module. The host program's `refresh` captures a new -//! generation, `list` and `inspect` report the store's generations, and `remove` retires one, in -//! the `generations` module, which also decides which generations the host program evicts. +//! leases the generation, in the `clone` module. Its `run` runs one execution in such a clone, +//! then tears the clone down. The host program's `refresh` captures a new generation, `list` and +//! `inspect` report the store's generations, and `remove` retires one, in the `generations` +//! module, which also decides which generations the host program evicts. #[cfg(feature = "host")] pub mod capture; diff --git a/tests/runtime-adapter/tests/clone_start.rs b/tests/runtime-adapter/tests/clone_start.rs index 15563c31a..afb4d78dc 100644 --- a/tests/runtime-adapter/tests/clone_start.rs +++ b/tests/runtime-adapter/tests/clone_start.rs @@ -2,11 +2,15 @@ //! not a host's time, as usage errors, a network that the plan refuses and shares that it cannot //! export, and a selection that names no template, of a template store that does not exist, or of //! a generation that the store does not hold, with the refusal's record and its own status, before -//! it needs NVX's artifacts or creates the sandbox's store. A clone's record names its template -//! store and its generation, which a stop reads back, beside the plan that the other commands -//! read, with the clone's network, whose proxy an execution of the clone then finds at the URL -//! that NVX's activation record names, a clone's request carries the export of its shares, and a -//! host's time keeps its text form, the form of the clock fixture. The tests need no hypervisor. +//! it needs NVX's artifacts or creates the sandbox's store. Its `run` refuses options that do not +//! make a run, an execution's included, as usage errors, an image that the plan refuses before it +//! stages anything, and a selection that names no template once its preflight staged the clone's +//! images, as a start refuses it. A clone's record names its template store and its generation, +//! which a stop reads back, beside the plan that the other commands read, with the clone's +//! network, whose proxy an execution of the clone then finds at the URL that NVX's activation +//! record names, a clone's request carries the export of its shares, +//! and a host's time keeps its text form, the form of the clock fixture. The tests need no +//! hypervisor. #![cfg(feature = "host")] use std::fs; @@ -19,7 +23,8 @@ use nvx::activation::{MachineIdMode, StoreGeneration}; use nvx::exec::{Key, Limits, Timers, Window}; use nvx::host::artifacts::Artifacts; use nvx::host::sandbox::SandboxNetwork; -use nvx::host::template::{HostClock, HostTime, Store}; +use nvx::host::store::SandboxStore; +use nvx::host::template::{HostClock, HostTime, Lease, Store}; use nvx::protocol::root::ImageFormat; use nvx::uuid::Uuid; use runtime_adapter::clone::{ @@ -354,6 +359,195 @@ fn a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_any assert!(!store.exists()); } +#[test] +fn the_host_program_refuses_options_that_do_not_make_a_run() { + let run = [ + "run", + "store", + "--template", + "templates", + "--generation", + "1", + "--distro", + "erofs:image", + "--distro-identity", + IDENTITY, + "--upper", + "ram", + ]; + let program = ["--", "/bin/true"]; + let with = |options: &[&'static str]| [&run[..], options, &program[..]].concat(); + let before = |end: usize| [&run[..end], &program[..]].concat(); + // The options of a start and of an execution come in any order before `--`, and the + // execution's program and its arguments are every argument after it. + let cases: Vec<(Vec<&str>, &str)> = vec![ + (vec!["run"], "run needs a sandbox store"), + ( + vec!["run", "store"], + "run needs `-- PROGRAM` after its options", + ), + (run.to_vec(), "run needs `-- PROGRAM` after its options"), + ( + [&run[..], &["--"][..]].concat(), + "run needs a program after `--`", + ), + (before(2), "run needs --template"), + (before(4), "run needs --generation"), + (before(6), "run needs --distro"), + (before(8), "run needs --distro-identity"), + (before(10), "run needs --upper"), + ( + with(&["--scratch-journal", "forbidden"]), + "--scratch-journal needs --upper scratch:PATH", + ), + ( + with(&["--generation", "2"]), + "unexpected option \"--generation\" of run", + ), + (with(&["--groups", "1"]), "--groups needs --identity"), + ( + with(&["--capabilities", "0"]), + "--capabilities needs --identity", + ), + (with(&["--numeric"]), "--numeric needs --identity"), + ( + with(&["--identity", "1000"]), + "--identity takes UID:GID, not \"1000\"", + ), + ( + with(&["--cwd", "tmp"]), + "names a working directory that is not absolute", + ), + ( + with(&["--env", "=value"]), + "has an environment entry that is not NAME=VALUE", + ), + ( + with(&["--cwd", "/a", "--cwd", "/b"]), + "unexpected option \"--cwd\" of run", + ), + (with(&["extra"]), "unexpected argument \"extra\" of run"), + (vec!["run", "store", "--env"], "\"--env\" needs a value"), + ( + vec!["start", "store", "--identity", "1000:1000"], + "unexpected option \"--identity\" of start", + ), + ]; + // Each option that a run does not take. + let others: [(&str, &str); 5] = [ + ("--guest-agent", "agent"), + ("--hypervisor", "whp"), + ("--runtime-dir", "/run/a"), + ("--cpu-profile", "auto"), + ("--memory-capacity", "512"), + ]; + let others = others.map(|(option, value)| { + ( + with(&[option, value]), + format!("unexpected option \"{option}\" of run"), + ) + }); + let cases = cases + .into_iter() + .map(|(arguments, problem)| (arguments, problem.to_owned())) + .chain(others); + for (arguments, problem) in cases { + let output = host(&arguments, None); + assert_eq!(output.status.code(), Some(2), "{arguments:?}: {output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&problem) && stderr.contains("usage: runtime-adapter-host"), + "{arguments:?}: {stderr}" + ); + } + + // A clock fixture that is not a host's time is a usage error too, of a run that the options + // make. + let output = host(&with(&[]), Some("now")); + assert_eq!(output.status.code(), Some(2), "{output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&format!("{HOST_TIME_VARIABLE} takes host=HOST,boot=BOOT")), + "{stderr}" + ); +} + +#[test] +fn a_run_refuses_a_selection_that_names_no_template_once_its_preflight_staged_the_images() { + // The run's preflight stages the clone's images in the sandbox's store, which it creates, + // then the start reads the template store's index before it needs NVX's artifacts, which the + // commands do not name. + let directory = directory("run-unindexed"); + let distro = image(directory.join("distro.erofs"), 4096); + let distro = format!("erofs:{}", distro.display()); + let templates = directory.join("templates"); + let store = directory.join("store"); + let run = |templates: &Path, generation: &str, distro: &str| { + let arguments = [ + "run", + store.to_str().unwrap(), + "--template", + templates.to_str().unwrap(), + "--generation", + generation, + "--distro", + distro, + "--distro-identity", + IDENTITY, + "--upper", + "ram", + "--identity", + "65534:65534", + "--", + "/bin/true", + ]; + host(&arguments, None) + }; + + // An image that the plan refuses fails the run before it stages anything. + let missing = format!("erofs:{}", directory.join("missing.erofs").display()); + let output = run(&templates, "1", &missing); + assert_eq!(output.status.code(), Some(1), "{output:?}"); + assert_eq!(String::from_utf8_lossy(&output.stdout), ""); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(!stderr.contains("template store"), "{stderr}"); + assert!(!store.exists()); + + let refused = |output: &Output, evidence: &str, problem: &str| { + assert_eq!(output.status.code(), Some(10), "{output:?}"); + let stdout = String::from_utf8_lossy(&output.stdout); + let record = format!( + "preflight=staged\nfailure.outcome=no-template\nfailure.evidence={evidence}\n\ + failure.stage=selection\nfailure.progress=none\nfailure.status=none\n\ + failure.teardown=no-vm\nfailure.code=not-found\n" + ); + assert_eq!(stdout, record); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains(problem), "{stderr}"); + assert!(!stderr.contains("usage:"), "{stderr}"); + let sandbox = SandboxStore::open(&store).unwrap(); + assert!(Staged::read(sandbox.dir()).unwrap().is_some()); + assert!(Lease::of(&sandbox).unwrap().is_none()); + assert!(sandbox.runtime().unwrap().is_none()); + }; + + // No template store. + let output = run(&templates, "1", &distro); + refused(&output, "no-store", "cannot open the template store"); + assert!(!templates.exists()); + + // A template store that never published a generation. + Store::create(&templates).unwrap(); + for generation in ["1", "2"] { + let output = run(&templates, generation, &distro); + let problem = format!( + "the template store {} holds no generation {generation}", + Store::open(&templates).unwrap().dir().display() + ); + refused(&output, "unpublished", &problem); + } +} + #[test] fn a_host_s_time_keeps_its_text_form() { let linux = host_time(HostClock::LinuxBoottime); diff --git a/tests/runtime-adapter/tests/host_session.rs b/tests/runtime-adapter/tests/host_session.rs index 05f261e21..213388e6a 100644 --- a/tests/runtime-adapter/tests/host_session.rs +++ b/tests/runtime-adapter/tests/host_session.rs @@ -16,8 +16,9 @@ //! after a later reset, the end of the stream, or a failure of the transport, whose successor sends //! the same record, and the failures that are final. The broker's guest also runs the runners of //! NVX's execution engine for the executions that the host program's driver runs, so that the tests -//! cover the driver's output, credit, outcomes, and timers, the refusals that end its run, and a -//! guest that stops reading. +//! cover the driver's output, credit, outcomes, and timers, the refusals that end its run, a guest +//! that stops reading, and a stop that skips the guest's late messages about the executions that +//! the driver gave up on. #![cfg(feature = "host")] use std::collections::{BTreeMap, VecDeque}; @@ -2645,3 +2646,91 @@ fn the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_ ); assert_ended(&mut client); } + +#[test] +fn a_stop_skips_the_guest_s_late_messages_about_the_executions_that_the_driver_gave_up_on() { + // The execution's timeout passes while the guest sends nothing, so the driver gives up on it, + // and finishes it without the guest's messages about it, which arrive once the host stops the + // guest agent in the same attachment. + let timed = Timers::new(Duration::from_millis(50), Duration::from_millis(100)) + .timeout(Duration::from_millis(200)); + let key = Key::new(1, 1); + let start = Start::new(key, limits(), timed, ["/bin/program"]); + let given_up = || { + let (mut broker, client) = attached(FIRST, usize::MAX); + let running = run(client, vec![start.clone()], deadline()); + // The start, the credit of the output's pool, then the cancellation. + broker.messages(3); + let (client, executions) = joined(running); + let executions = executions.unwrap(); + let outcome = executions.get(1).unwrap().outcome().unwrap(); + assert_eq!(outcome.status(), None); + (broker, client, executions) + }; + let late = [ + GuestMessage::Output(key, Stream::Stdout, b"late").encode(), + GuestMessage::End(key, Stream::Stdout).encode(), + GuestMessage::Terminal(key, Outcome::new(Some(Status::Exited(0)))).encode(), + ]; + let teardown = || RootFailure::new(RootStage::Teardown, "busy"); + + // The guest's late messages, then `Stopped`, and, in records of 7 bytes, late messages before + // and after a teardown that failed, which the stop returns. + let cases = [ + ([&late[..], &[Message::Stopped]].concat(), None), + ( + [ + &late[..1], + &[Message::RootFailed(teardown().encode())], + &late[1..], + &[Message::Stopped], + ] + .concat(), + Some(teardown()), + ), + ]; + for ((replies, expected), size) in cases.into_iter().zip([OUTER_MAX_PAYLOAD, 7]) { + let (mut broker, client, executions) = given_up(); + let stopping = spawn(client, move |client| { + client.stop_skipping(deadline(), |message| executions.is_about(message)) + }); + assert_eq!(broker.messages(1), [Message::Stop]); + broker.send_data(&encode(&replies), size); + let (_, stopped) = joined(stopping); + assert_eq!(stopped.unwrap(), expected); + } + + // `stop` takes the first late message for its answer, and neither skips a message about an + // execution that the driver did not start, of another ID, or of its ID in another epoch: each + // ends the client. + let others = [ + GuestMessage::Terminal(Key::new(2, 1), Outcome::new(Some(Status::Exited(0)))).encode(), + GuestMessage::Started(Key::new(1, 2)).encode(), + ]; + let cases = [(false, late[0].clone())] + .into_iter() + .chain(others.into_iter().map(|other| (true, other))); + for (skipping, reply) in cases { + let (mut broker, client, executions) = given_up(); + let stopping = spawn(client, move |client| { + if skipping { + client.stop_skipping(deadline(), |message| executions.is_about(message)) + } else { + client.stop(deadline()) + } + }); + assert_eq!(broker.messages(1), [Message::Stop]); + broker.send_data( + &encode(&[reply.clone(), Message::Stopped]), + OUTER_MAX_PAYLOAD, + ); + let (mut client, stopped) = joined(stopping); + let error = stopped.unwrap_err(); + let kinds = (Message::Stop.kind(), reply.kind()); + assert!( + matches!(error, ClientError::Reply { request, reply } if (request, reply) == kinds), + "{skipping}: {error:?}" + ); + assert_ended(&mut client); + } +} diff --git a/tests/runtime-adapter/tests/vm_clone.rs b/tests/runtime-adapter/tests/vm_clone.rs index e97c87daa..f670d9648 100644 --- a/tests/runtime-adapter/tests/vm_clone.rs +++ b/tests/runtime-adapter/tests/vm_clone.rs @@ -1,7 +1,7 @@ //! The host program starts clones of templates on a real hypervisor, through public APIs only: VMs //! that a restore of a generation of a template store starts, in place, which the store leases to //! a sandbox, and which the guest agent's clone mode activates, as `runtime-adapter-host start` -//! asks. +//! and `run` ask. //! //! Each scenario captures a [`Template`] of its own, with 256 MiB of RAM, of a capacity of //! 1 GiB, in a template store of its own, as `vm_template` does, of the platform's copy of the @@ -81,6 +81,16 @@ //! - A selection that names no template leases nothing, with the record of the store's refusal: a //! template store that does not exist, a generation that the store never published, and one that //! `remove` retired. +//! - A run runs one execution in its clone, then tears the clone down, in one command: its +//! preflight stages the clone's images, it starts the clone as a start does, runs the execution +//! in the clone's root, with the environment and the working directory that its start names, as +//! the clone's workload identity, whatever its program's status, and stops the clone, which +//! releases the lease. An execution that the guest refuses leaves the run to tear the clone down +//! as well. +//! - A run whose clone does not start runs nothing: once the preflight staged the clone's images, +//! the store refuses a selection that names no template, or of a generation whose restore window +//! passed, without a lease, and a clone whose activation fails ends the run with the record of +//! its failure, once its VM exited, and the lease released. //! - A start that ended once the store leased the generation, before it recorded the clone, //! leaves a lease whose template store no record names, or only the record of an earlier clone, //! of another template: a stop keeps it, and only its own template store, which `--template` @@ -822,6 +832,30 @@ impl Template { /// Starts a clone in a new sandbox of the scenario `name`, as `start` asks, checks that the /// start exits with `status`, and returns the clone. fn start(&self, name: &str, start: StartOf<'_>, status: i32) -> Cloned { + self.clone_in(name, "start", &start, &[], status) + } + + /// Runs one execution, of `execution`, its options and its program, in a clone in a new + /// sandbox of the scenario `name`, as `run` asks, with the start's options of `start`, checks + /// that the run exits with `status`, and returns the clone, as the run left it. The run's + /// preflight stages the clone's images. + fn run(&self, name: &str, start: StartOf<'_>, execution: &[&str], status: i32) -> Cloned { + assert!(!start.unstaged, "a run stages its clone's images"); + self.clone_in(name, "run", &start, execution, status) + } + + /// Runs the host program's `command`, `start` or `run`, of a clone in a new sandbox of the + /// scenario `name`, as `start` asks, with the arguments `execution` after the start's options, + /// checks that it exits with `status`, and returns the clone. Before a start, the sandbox's + /// store stages the clone's images, unless `start` says otherwise. + fn clone_in( + &self, + name: &str, + command: &str, + start: &StartOf<'_>, + execution: &[&str], + status: i32, + ) -> Cloned { let mut sandbox = Sandbox::new(name); let (distro, identity) = match start.image { None => (erofs(&self.image), self.identity.as_str()), @@ -853,7 +887,8 @@ impl Template { _ => prefixed("scratch:", &scratch), }; let identity = OsStr::new(identity); - if !start.unstaged { + let run = command == "run"; + if !run && !start.unstaged { let mut staging = vec![ OsStr::new("--distro"), &distro, @@ -879,14 +914,22 @@ impl Template { &upper, ]; options.extend(start.options.iter().map(OsStr::new)); - let time = start.time.map(|time| OsString::from(host_time_text(&time))); + options.extend(execution.iter().map(OsStr::new)); + let time = start + .time + .as_ref() + .map(|time| OsString::from(host_time_text(time))); let environment: Vec<(&str, &OsStr)> = time .iter() .map(|time| (HOST_TIME_VARIABLE, time.as_os_str())) .collect(); - let started = sandbox.start("start", &options, &environment); + let started = sandbox.start(command, &options, &environment); let started = sandbox.finish(started, status); - Cloned { sandbox, started } + Cloned { + sandbox, + started, + ran: run, + } } } @@ -969,12 +1012,15 @@ struct StartOf<'a> { unstaged: bool, } -/// A clone of a scenario's template, in a sandbox of its own, as its start left it. +/// A clone of a scenario's template, in a sandbox of its own, as its start, or its run, left it. struct Cloned { /// The clone's sandbox. sandbox: Sandbox, - /// What the start printed. + /// What the start, or the run, printed. started: Printed, + /// Whether a run, whose preflight staged the clone's images, rather than a start, started + /// the clone. + ran: bool, } impl Cloned { @@ -988,25 +1034,23 @@ impl Cloned { /// once its guest applied `mapped` mappings of its export, none for a clone without one. fn assert_mapped(&self, mapped: usize) { let started = &self.started; - assert_eq!(started["generation"], GENERATION.to_string(), "{started:?}"); - let lease = &started["lease"]; - assert!( - lease.len() == 32 && lease.bytes().all(|byte| byte.is_ascii_hexdigit()), - "{started:?}" - ); - started["downtime"].parse::().unwrap(); - let mapped = mapped.to_string(); - let facts = ["root", "build", "table", "mapped", "sandbox"].map(|key| &started[key]); - assert_eq!( - facts, - ["ready", GUEST_BUILD, "applied", mapped.as_str(), "started"], - "{started:?}" - ); + assert_clone_started(started, mapped); let record = self.sandbox.record().expect("the start recorded its VM"); assert_eq!(started["pid"], record.process().pid.to_string()); assert!(!started.contains_key("released"), "{started:?}"); } + /// Checks that the run ran the clone, as the [module documentation](self) describes: its + /// preflight staged the clone's images, it started the clone as a start does, leasing the + /// generation, and it stopped the clone as a stop does, which released the lease. + fn assert_ran(&self) { + let ran = &self.started; + assert_eq!(ran["preflight"], "staged", "{ran:?}"); + assert_clone_started(ran, 0); + ran["pid"].parse::().unwrap(); + self.assert_stopped(ran); + } + /// Runs `probes` in the clone's root, and returns what they found, each fact as /// `PROBE.KEY`. fn probe(&mut self, probes: &[&str]) -> Printed { @@ -1030,6 +1074,13 @@ impl Cloned { /// clone's lease. fn stop(&mut self) { let stopped = self.sandbox.run("stop", &[], 0); + self.assert_stopped(&stopped); + } + + /// Checks that `stopped`, what a stop, or a run, printed, says that the clone stopped + /// gracefully, with a success and a complete teardown, and released its lease, which the + /// sandbox no longer holds, and that the sandbox records no VM. + fn assert_stopped(&self, stopped: &Printed) { let facts = ["stopped", "outcome", "teardown"].map(|key| &stopped[key]); assert_eq!(facts, ["graceful", "success 0", "complete"], "{stopped:?}"); assert_eq!(stopped["released"], self.started["lease"], "{stopped:?}"); @@ -1272,8 +1323,8 @@ impl Cloned { /// Checks that the template store refused the start's selection before the start leased /// anything, as the record that the start printed says: the refusal's `outcome` and /// `evidence`, at the selection, without a VM, the code `code` of NVX's error, and no lease, - /// generation, or quarantine, and as [`assert_refused`](Self::assert_refused) checks it, with - /// a diagnostic that says `problem`. + /// generation, or quarantine, after the preflight of a run, and as + /// [`assert_refused`](Self::assert_refused) checks it, with a diagnostic that says `problem`. fn assert_selection_refused(&self, outcome: &str, evidence: &str, code: &str, problem: &str) { let started = &self.started; assert_eq!( @@ -1282,7 +1333,13 @@ impl Cloned { "{started:?}" ); assert_eq!(started["failure.code"], code, "{started:?}"); - assert_eq!(started.len(), 7, "{started:?}"); + let printed = if self.ran { + assert_eq!(started["preflight"], "staged", "{started:?}"); + 8 + } else { + 7 + }; + assert_eq!(started.len(), printed, "{started:?}"); self.assert_refused(problem); } } @@ -1426,6 +1483,27 @@ fn scratch_image(sandbox: &Sandbox) -> PathBuf { sandbox.root.join("images").join("scratch.ext4") } +/// Checks that `started`, what a start, or a run, printed, says that it started a clone of the +/// template's generation, with a lease of its own, at a downtime, whose guest agent, of the +/// host's build, assembled its root, applied `mapped` mappings of its export, none for a clone +/// without one, and started its sandbox. +fn assert_clone_started(started: &Printed, mapped: usize) { + assert_eq!(started["generation"], GENERATION.to_string(), "{started:?}"); + let lease = &started["lease"]; + assert!( + lease.len() == 32 && lease.bytes().all(|byte| byte.is_ascii_hexdigit()), + "{started:?}" + ); + started["downtime"].parse::().unwrap(); + let mapped = mapped.to_string(); + let facts = ["root", "build", "table", "mapped", "sandbox"].map(|key| &started[key]); + assert_eq!( + facts, + ["ready", GUEST_BUILD, "applied", mapped.as_str(), "started"], + "{started:?}" + ); +} + /// Stages the platform's copy of the fixtures' Ubuntu EROFS image in the directory `platform`, /// beside a copy of its manifest, and returns it. /// @@ -2926,6 +3004,149 @@ fn a_selection_that_names_no_template_leases_nothing() { assert!(store.leases(GENERATION).unwrap().is_empty()); } +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down() { + let template = Template::create("cl-run"); + // The execution's program, its arguments, its environment, and its working directory reach + // the guest in its start, and it runs in the clone's root, the Ubuntu image's, as the clone's + // workload identity. The run ends with success whatever the program's status. + let options = ["--workload-identity", NOBODY]; + let start = StartOf { + upper: Upper::Scratch, + options: &options, + ..StartOf::default() + }; + let script = ". /etc/os-release; echo \"$GREETING $ID $(id -u) $(pwd)\"; echo done >&2; exit 3"; + let execution = [ + "--env", + "GREETING=hello", + "--cwd", + "/tmp", + "--", + "/bin/sh", + "-c", + script, + ]; + let ran = template.run("cl-r", start, &execution, 0); + ran.assert_ran(); + let printed = &ran.started; + let facts = [ + "launch", + "status", + "cause", + "truncated", + "settled", + "stdout", + "stderr", + ] + .map(|key| &printed[&format!("exec.1.{key}")]); + assert_eq!( + facts, + [ + "started", + "exit 3", + "none", + "none", + "reusable", + "hello ubuntu 65534 /tmp\\n", + "done\\n" + ], + "{printed:?}" + ); + assert!(template.leases().is_empty()); + + // An execution that the guest refuses before its program starts, one that names an identity + // in a clone that runs as its own, leaves the run to tear the clone down as well. + let start = StartOf { + options: &options, + ..StartOf::default() + }; + let execution = ["--identity", "1000:1000", "--", "/bin/true"]; + let ran = template.run("cl-r", start, &execution, 0); + ran.assert_ran(); + let printed = &ran.started; + let launch = format!("failed 0 {EINVAL}"); + let facts = ["launch", "status", "settled"].map(|key| &printed[&format!("exec.1.{key}")]); + assert_eq!(facts, [launch.as_str(), "none", "fenced"], "{printed:?}"); + assert!(template.leases().is_empty()); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_run_whose_clone_does_not_start_runs_nothing() { + let template = Template::create("cl-norun"); + let execution = ["--identity", NOBODY, "--", "/bin/true"]; + + // The store refuses a selection that names no template, and one of a generation whose restore + // window passed, once the run's preflight staged the clone's images. + let sandbox = template + .sandbox + .as_ref() + .expect("the template keeps its sandbox"); + let missing = sandbox.root.join("missing"); + let start = StartOf { + templates: Some(&missing), + ..StartOf::default() + }; + let refused = template.run("cl-n", start, &execution, SELECTION_REFUSED); + refused.assert_selection_refused( + "no-template", + "no-store", + "not-found", + "cannot open the template store", + ); + assert!(!missing.exists()); + let contract = template.generation().time().clone(); + let expired = HostTime { + monotonic_ns: contract.capture_monotonic_ns + MAX_DOWNTIME_NS + 1, + ..HostTime::now().unwrap() + }; + let start = StartOf { + time: Some(expired), + ..StartOf::default() + }; + let refused = template.run("cl-n", start, &execution, SELECTION_REFUSED); + let generation = format!("the generation {GENERATION} of the template store"); + refused.assert_selection_refused("expired", "expired", "template-ineligible", &generation); + + // A clone whose activation fails ends the run before its execution, with the record of the + // failure, once the VM exited, and the lease released. + let start = StartOf { + upper: Upper::Zeros, + ..StartOf::default() + }; + let failed = template.run("cl-n", start, &execution, CLONE_FAILED); + let printed = &failed.started; + assert_eq!(printed["preflight"], "staged", "{printed:?}"); + assert!(printed.contains_key("pid"), "{printed:?}"); + let status = status::ACTIVATION_ROOT.to_string(); + assert_eq!( + failure(printed), + [ + "request", + "unusable-root", + "activation", + "activating", + status.as_str(), + "exited" + ], + "{printed:?}" + ); + assert_eq!(printed["quarantined"], "no"); + assert_eq!(printed["released"], printed["lease"], "{printed:?}"); + assert!( + !printed + .keys() + .any(|key| key.starts_with("exec.") || key == "stopped"), + "{printed:?}" + ); + assert!(failed.sandbox.record().is_none()); + assert!(template.leases().is_empty()); + assert_payload(&template, false); +} + /// Leases the generation of `template` to the store of `sandbox`, through NVX's store, as a start /// does before it records its clone, and returns the lease's identity. fn lease_to(template: &Template, sandbox: &Sandbox) -> String { From d7d8d793f66f45dbe4d8a408376be82e9cadbba3 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 26 Oct 2026 11:47:39 -0700 Subject: [PATCH 3/5] runtime-adapter: give a run the network and shares of its start The stage's integration of network templates and exports gives `start` the options `--network` and `--share`, with `--deny`, `--reserved`, and `--share-root`, beside `--first-partition-bound` and `--machine-id`, and `run` takes the options of a start. A run read its network only in its start, though, once its preflight staged the images, and checked the execution's start without the proxy of the network. So `CloneOptions::plan` now reads the clone's network, and plans the clone and the export of its shares, as the start did first, and both commands plan before anything else: a start before it reads the template store, and a run before its preflight stages anything. A network, an image, or shares that the plan refuses then fail a run with nothing created, and the run checks the execution's start with the URL of the network's proxy in its environment, as `exec` gives it to an execution of a clone, before it stages anything. The start's behavior is unchanged. The host program's documentation and the README give `run` the start's options, and say that the execution finds the clone's shares beneath its root. `clone_start` checks the run's new usage errors, and that a run refuses a network or shares that the plan refuses before it stages anything. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7d3db585-441c-495b-901f-ab7203c0f36b --- tests/runtime-adapter/README.md | 40 +++-- .../src/bin/runtime-adapter-host.rs | 150 +++++++++++------- tests/runtime-adapter/tests/clone_start.rs | 84 ++++++++++ 3 files changed, 207 insertions(+), 67 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 7b8b14b42..795884c4d 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -1654,9 +1654,13 @@ before. tears the clone down. Its preflight, which it reports with `preflight=staged`, stages the clone's images in the sandbox's store, as `stage` stages them, verifying the image, which hashes it only when a manifest is beside it, before -the clone's start, which hashes nothing. The command first checks that the -guest takes the execution's start, so a start that it would refuse fails the -command before it stages anything. It then starts the clone as `start` does, +the clone's start, which hashes nothing. The command first reads the clone's +network, and plans the clone and the export of its shares, as `start` does, +then checks that the guest takes the execution's start, with the URL of the +proxy of the clone's network, if any, in its environment, as `exec` gives it to +an execution of a clone, so a network, an image, shares, or a start that it +would refuse fail the command before it stages anything. It then starts the +clone as `start` does, at the host's time, or the one that `RUNTIME_ADAPTER_HOST_TIME` gives, and ends as a start ends once the template store refuses the selection, or the clone fails, with the record of the refusal or of the failure. Once the clone runs, @@ -1665,7 +1669,9 @@ activation assembled, through `host::Executions`, in the session of the adapter's clone order in which the clone's sandbox started, as `exec` runs one. The program, its arguments, and its environment reach the guest in the execution's start, through the guest agent's execution control, never in the -template's boot tokens. Then, whatever the execution did, the command stops the +template's boot tokens. The execution finds the clone's shares beneath the +root, since the restore sequence maps them before the clone's sandbox starts. +Then, whatever the execution did, the command stops the clone as `stop` does, and releases the lease once the store verified that the VM exited: a run leaves no clone running, and no lease held, unless its stop timed out. The stop takes the same session, in which it skips the guest's @@ -1873,18 +1879,20 @@ Each command of `runtime-adapter-host` but `stage`, `list`, `inspect`, and number. The guest refuses a key that it heard of before, and the epochs of a broker instance only advance, while a broker restored from a snapshot has a new instance ID, so no command repeats the key of another's execution. -- `run STORE --template TEMPLATES --generation G --distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] EXECUTION` +- `run STORE --template TEMPLATES --generation G --distro FORMAT:PATH --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] [--machine-id (image | generated)] [--network FILE] [SHARES] EXECUTION` runs one EXECUTION, one of `exec`'s, in a [clone](#clones) of the generation G of the template store TEMPLATES, in the sandbox store STORE, which it creates if needed, then tears the clone down, as [running a clone](#running-a-clone) describes. The execution's options may come among the start's, in any order, and its PROGRAM and ARGs are every - argument after `--`. Its preflight stages the clone's images in STORE, as - `stage` stages them, and prints `preflight=staged`. It then starts the clone - as `start` does, runs the execution in the clone's root as `exec` runs one, - and prints its output and outcome once it finished. Then it stops the clone - as `stop` does, whatever the execution did, which releases the lease once the - VM exited. + argument after `--`. It first plans the clone as `start` does, with its + network and its shares, and checks the execution's start, with the URL of + the network's proxy, if any. Its preflight then stages the clone's images in + STORE, as `stage` stages them, and prints `preflight=staged`. It then starts + the clone as `start` does, runs the execution in the clone's root as `exec` + runs one, and prints its output and outcome once it finished. Then it stops + the clone as `stop` does, whatever the execution did, which releases the + lease once the VM exited. `reattach`, `stop`, `probe`, and `exec` find a VM that booted cold under another runtime identity mismatched, and then signal nothing: the VM keeps @@ -2377,9 +2385,13 @@ Windows, without a hypervisor: shares, which it cannot export, and the template store, which does not exist; - the command lines of `run` that the host program refuses, each with status 2: missing options, a missing `--` or program, the qualifiers of an identity - without one, an execution's start that the guest would refuse, invalid - values, arguments before `--`, and the options of other commands, with a - clock fixture that is not a host's time too; + without one, an execution's start that the guest would refuse, a + first-partition bound for an EROFS image, a share's options without a share, + invalid values, an option given twice, arguments before `--`, and the options + of other commands, with a clock fixture that is not a host's time too; +- a run whose network cannot be read, or is one that the plan refuses, or whose + shares it cannot export, with status 1, before it stages anything or creates + the sandbox's store; - a run whose image the plan refuses, with status 1, before it stages anything or creates the sandbox's store, and runs whose template store does not exist, or does not hold the generation, each with status 10 and the preflight's diff --git a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs index 9569b5c0c..fa09fd5cb 100644 --- a/tests/runtime-adapter/src/bin/runtime-adapter-host.rs +++ b/tests/runtime-adapter/src/bin/runtime-adapter-host.rs @@ -230,24 +230,30 @@ //! snapshot has a new instance ID, so no command repeats the key of another's execution. //! - `runtime-adapter-host run STORE --template TEMPLATES --generation G --distro FORMAT:PATH //! --distro-identity UUID --upper (scratch:PATH | ram) [--memory MIB] [--workload-identity -//! UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] EXECUTION` runs one execution in a -//! clone of the generation G of the template store TEMPLATES, in the sandbox store STORE, which -//! it creates if needed, then tears the clone down. The EXECUTION is one of `exec`'s, whose +//! UID:GID] [--overlay-mode MODE] [--scratch-journal POLICY] [--first-partition-bound BYTES] +//! [--machine-id (image | generated)] [--network FILE] [SHARES] EXECUTION` runs one execution in +//! a clone of the generation G of the template store TEMPLATES, in the sandbox store STORE, +//! which it creates if needed, then tears the clone down. The EXECUTION is one of `exec`'s, whose //! options may come among the start's, in any order, and whose PROGRAM and ARGs are every -//! argument after `--`. The command first checks that the guest takes the execution's start. -//! Its preflight then stages the clone's images in STORE, as `stage` stages them, which verifies -//! the image, and hashes it only when a manifest is beside it, and prints that it did, before -//! the clone's start, which hashes nothing. It starts the clone as `start` does, and ends as a -//! start ends once the template store refuses the selection, or the clone fails. It runs the -//! execution in the clone's root, as `exec` runs one, through the guest agent's execution -//! control, in the session in which the clone's sandbox started, so that the program, its -//! arguments, and its environment reach the guest in the execution's start, never in the -//! template's boot tokens, and prints its output and outcome once it finished. Then it stops -//! the clone, as `stop` does, whatever the execution did, which releases the lease once the VM -//! exited. The stop takes the same session, in which it skips the guest's messages about the -//! execution that may still arrive once the command finished it without them, as when it gave -//! up waiting for its outcome. It holds the sandbox's lock, which the lease took, until then, so -//! no other command reattaches to the clone meanwhile. +//! argument after `--`. The command first reads the clone's network, and plans the clone and the +//! export of its shares, as `start` does, then checks that the guest takes the execution's +//! start, with the URL of the proxy of the clone's network, if any, in its environment, as +//! `exec` gives it to an execution of a clone, so that none of them fails the command once it +//! staged anything. Its preflight then stages the clone's images in STORE, as `stage` stages +//! them, which verifies the image, and hashes it only when a manifest is beside it, and prints +//! that it did, before the clone's start, which hashes nothing. It starts the clone as `start` +//! does, and ends as a start ends once the template store refuses the selection, or the clone +//! fails. It runs the execution in the clone's root, as `exec` runs one, through the guest +//! agent's execution control, in the session in which the clone's sandbox started, so that the +//! program, its arguments, and its environment reach the guest in the execution's start, never +//! in the template's boot tokens, and prints its output and outcome once it finished. The +//! execution finds the clone's shares beneath the root, since the restore sequence maps them +//! before the clone's sandbox starts. Then it stops the clone, as `stop` does, whatever the +//! execution did, which releases the lease once the VM exited. The stop takes the same session, +//! in which it skips the guest's messages about the execution that may still arrive once the +//! command finished it without them, as when it gave up waiting for its outcome. It holds the +//! sandbox's lock, which the lease took, until then, so no other command reattaches to the clone +//! meanwhile. //! //! Each command but `stage`, `list`, `inspect`, and `remove`, which need no artifacts, takes NVX's //! artifacts from the directory that `NVX_ARTIFACTS_DIR` names, a repository checkout or an @@ -841,7 +847,7 @@ impl Serving { } } -/// The clone of `start`, as its options give it. +/// The clone of `start` or `run`, as its options give it. struct CloneOptions { /// The template store. templates: PathBuf, @@ -854,6 +860,44 @@ struct CloneOptions { network: Option, } +impl CloneOptions { + /// Reads the clone's network, if it has one, as a launch reads its network, then plans the + /// clone's layered root, with that network, and the export of the clone's shares, if it has + /// any, as a launch plans its own, or says why it cannot: a network that the plan refuses, + /// then an image, or shares, that it refuses. It needs nothing but the options, and the files + /// that they name, and creates nothing. NVX's plan of the clone then requires the template's + /// network of it. + fn plan(&self) -> Result { + let network = self + .network + .as_deref() + .map(read_network) + .transpose() + .map_err(|error| describe(&error))?; + let request = runtime_adapter::clone::Request { + network, + ..self.request.clone() + }; + let plan = request.plan().map_err(|error| describe(&error))?; + let export = request.export().map_err(|error| describe(&error))?; + Ok(PlannedClone { + request, + plan, + export, + }) + } +} + +/// The clone of `start` or `run`, as [`CloneOptions::plan`] planned it. +struct PlannedClone { + /// The clone's image, upper, memory target, settings, and network. + request: runtime_adapter::clone::Request, + /// The adapter's plan of the clone's image over its upper, with its network. + plan: SandboxPlan, + /// The export of the host paths that the clone shares with its root, if any. + export: Option, +} + /// The options of a command. #[derive(Default)] struct Options { @@ -1358,8 +1402,8 @@ fn parse_option( Ok(()) } -/// Says why the options of `sharing`, of a launch or a start, are not a command's: `--deny`, -/// `--reserved`, or `--share-root` without a `--share`. +/// Says why the options of `sharing`, of a launch, a start, or a run, are not a command's: +/// `--deny`, `--reserved`, or `--share-root` without a `--share`. fn check_shares(sharing: &HostFsSharing) -> Result<(), String> { let options = [ ("--deny", !sharing.denied().is_empty()), @@ -2294,12 +2338,17 @@ fn start( clone: &CloneOptions, time: Option, ) -> Result { + // A network that the plan refuses fails the start before anything else, as it fails a launch, + // then an image, or shares, that it refuses. + let planned = clone.plan()?; // The clone keeps running once the command lets go of its VM, its session, and its sandbox's // lock, as once a launch committed. - Ok(match start_clone(store, guest_build, clone, time)? { - Ok(_running) => ExitCode::SUCCESS, - Err(status) => status, - }) + Ok( + match start_clone(store, guest_build, clone, planned, time)? { + Ok(_running) => ExitCode::SUCCESS, + Err(status) => status, + }, + ) } /// A clone that [`start_clone`] started, whose launch it committed, and which runs. @@ -2316,35 +2365,26 @@ struct Running { client: Client, } -/// Starts a clone of the generation of the template store that `clone` names, as it asks, in the -/// sandbox store `store`, which it creates if needed, with the guest agent's build `guest_build`, -/// at the host's time `time`, or at the time that `HostTime::now` samples without one, and prints -/// what it did. Returns the clone, which runs, once it committed the clone's launch, or the status -/// with which the command ends otherwise: once the template store refused the selection, or the -/// clone failed, as [`clone_failed`] ends it. +/// Starts a clone of the generation of the template store that `clone` names, as `planned`, its +/// plan, asks, in the sandbox store `store`, which it creates if needed, with the guest agent's +/// build `guest_build`, at the host's time `time`, or at the time that `HostTime::now` samples +/// without one, and prints what it did. Returns the clone, which runs, once it committed the +/// clone's launch, or the status with which the command ends otherwise: once the template store +/// refused the selection, or the clone failed, as [`clone_failed`] ends it. fn start_clone( store: &Path, guest_build: &str, clone: &CloneOptions, + planned: PlannedClone, time: Option, ) -> Result, String> { - // A network that the plan refuses fails the start before anything else, as it fails a launch. - // NVX's plan of the clone then requires the template's network of it. - let network = clone - .network - .as_deref() - .map(read_network) - .transpose() - .map_err(|error| describe(&error))?; - let request = runtime_adapter::clone::Request { - network, - ..clone.request.clone() - }; - // A request that the adapter's plan refuses, shares that it cannot export, and a selection - // that the template store refuses as it names no template, fail the start before it needs - // NVX's artifacts or the sandbox's store. - let plan = request.plan().map_err(|error| describe(&error))?; - let export = request.export().map_err(|error| describe(&error))?; + let PlannedClone { + request, + plan, + export, + } = planned; + // A selection that the template store refuses as it names no template fails the start before + // it needs NVX's artifacts or the sandbox's store. let templates = match Store::open(&clone.templates) { Ok(templates) => templates, Err(error) if error.kind() == ErrorKind::NotFound => { @@ -3260,7 +3300,10 @@ fn run_executions( /// guest agent's build `guest_build`, at the host's time `time`, or at the time that /// `HostTime::now` samples without one, then tears the clone down, and prints what it did. /// -/// Its preflight stages the clone's images in the sandbox, as `stage` does, which verifies the +/// It first plans the clone as `start` does, with its network and the export of its shares, and +/// checks that the guest takes the execution's start, with the URL of the proxy of the clone's +/// network, if any, in its environment, as `exec` gives it to an execution of a clone. Its +/// preflight then stages the clone's images in the sandbox, as `stage` does, which verifies the /// image, and prints `preflight=staged`. It then starts the clone, as `start` does, and ends as a /// start ends once the template store refuses the selection, or the clone fails. It runs the /// execution in the clone's root, as `exec` runs one, through the guest agent's execution control, @@ -3283,10 +3326,11 @@ fn run( execution: &Execution, time: Option, ) -> Result { - // A start that the guest would refuse fails the command before it stages anything. - let plan = clone.request.plan().map_err(|error| describe(&error))?; - let starts = starts_of(std::slice::from_ref(execution), Some(&plan))?; - let request = &clone.request; + // A network, an image, or shares that the plan refuses, and a start that the guest would + // refuse, fail the command before it stages anything. + let planned = clone.plan()?; + let starts = starts_of(std::slice::from_ref(execution), Some(&planned.plan))?; + let request = &planned.request; stage( store, &request.distro, @@ -3301,7 +3345,7 @@ fn run( locked, vmm, mut client, - } = match start_clone(store, guest_build, clone, time)? { + } = match start_clone(store, guest_build, clone, planned, time)? { Ok(running) => running, Err(status) => return Ok(status), }; diff --git a/tests/runtime-adapter/tests/clone_start.rs b/tests/runtime-adapter/tests/clone_start.rs index afb4d78dc..8d39b2b16 100644 --- a/tests/runtime-adapter/tests/clone_start.rs +++ b/tests/runtime-adapter/tests/clone_start.rs @@ -400,6 +400,19 @@ fn the_host_program_refuses_options_that_do_not_make_a_run() { with(&["--scratch-journal", "forbidden"]), "--scratch-journal needs --upper scratch:PATH", ), + ( + with(&["--first-partition-bound", "4096"]), + "--first-partition-bound needs --distro gpt:PATH", + ), + ( + with(&["--machine-id", "random"]), + "--machine-id takes image or generated, not \"random\"", + ), + (with(&["--deny", "/srv"]), "--deny needs --share"), + ( + with(&["--network", "a.json", "--network", "b.json"]), + "unexpected option \"--network\" of run", + ), ( with(&["--generation", "2"]), "unexpected option \"--generation\" of run", @@ -548,6 +561,77 @@ fn a_run_refuses_a_selection_that_names_no_template_once_its_preflight_staged_th } } +#[test] +fn a_run_refuses_a_network_or_shares_that_the_plan_refuses_before_it_stages_anything() { + // A run reads its network, then plans the export of its shares, as a start does, before its + // preflight stages anything, or it reads the template store, which does not exist. + let directory = directory("run-unplanned"); + let distro = image(directory.join("distro.erofs"), 4096); + let distro = format!("erofs:{}", distro.display()); + let templates = directory.join("templates"); + let store = directory.join("store"); + let sources = directory.join("sources"); + let outside = directory.join("outside"); + for path in [&sources, &outside] { + fs::create_dir(path).unwrap(); + } + let missing = directory.join("missing.json"); + let refused = directory.join("refused.json"); + let mut form = proxied(); + form["egress"] = json!("allow"); + fs::write(&refused, form.to_string()).unwrap(); + let (sources, outside) = (sources.to_str().unwrap(), outside.to_str().unwrap()); + let unshared = format!("{outside}:/srv/data"); + let cases = [ + ( + vec!["--network", missing.to_str().unwrap()], + format!("cannot read the network {}", missing.display()), + ), + ( + vec!["--network", refused.to_str().unwrap()], + format!( + "the network {} is refused: a network proxy needs egress denied by default", + refused.display() + ), + ), + ( + vec!["--share-root", sources, "--share", &unshared], + format!( + "--share entry {outside} lies outside the host directories that --share-root \ + authorizes" + ), + ), + ]; + for (options, problem) in cases { + let mut arguments = vec![ + "run", + store.to_str().unwrap(), + "--template", + templates.to_str().unwrap(), + "--generation", + "1", + "--distro", + &distro, + "--distro-identity", + IDENTITY, + "--upper", + "ram", + ]; + arguments.extend(options); + arguments.extend(["--identity", "65534:65534", "--", "/bin/true"]); + let output = host(&arguments, None); + assert_eq!(output.status.code(), Some(1), "{arguments:?}: {output:?}"); + assert_eq!(String::from_utf8_lossy(&output.stdout), "", "{output:?}"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains(&problem), "{arguments:?}: {stderr}"); + for other in ["template store", "usage:"] { + assert!(!stderr.contains(other), "{other}: {stderr}"); + } + assert!(!store.exists(), "the run created the sandbox's store"); + assert!(!templates.exists()); + } +} + #[test] fn a_host_s_time_keeps_its_text_form() { let linux = host_time(HostClock::LinuxBoottime); From 66f79e483638eb459b541261ccf2c270df983e18 Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 26 Oct 2026 11:51:03 -0700 Subject: [PATCH 4/5] runtime-adapter: run executions in clones of network templates and with shares, in VMs `run` now takes the start's network and shares, so two rows check that the run's execution gets them in its clone, in one command: - a run of a clone of a proxy-only template gives its execution the URL of the network's proxy in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`, after the execution's own entries, as the executions of a started clone find it, then tears the clone down; a run of a network that lets nothing through finds no template there, and NVX's plan refuses its clone, once the preflight staged the images, before any VMM starts, so the run runs nothing; - a run's execution reads a host file through the run's share, which the restore sequence mapped beneath the clone's root before the clone's sandbox started, and writes another through it, which the host's directory keeps once the run tore the clone down. A third row checks a run whose clone's VM exits during its execution. The execution, as user ID 0, tells the host through the run's share that it runs, then waits, while the test terminates the clone's VMM. The run waits for the VMM's exit, so its stop finds that the VM exited, and the clone fails, as NVX attributes the loss of its status, which quarantines nothing. The run ends with that failure's status 8, after the diagnostic of the executions' failure, once it released the lease, and the template still serves clones. `Cloned::assert_ran_mapped` checks a run whose clone applied mappings, `execution_facts` returns what a run, or an exec, printed of its first execution, and `Template::begin_in` begins a start or a run that the test ends itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7d3db585-441c-495b-901f-ab7203c0f36b --- tests/runtime-adapter/README.md | 3 + tests/runtime-adapter/tests/vm_clone.rs | 229 ++++++++++++++++++++++-- 2 files changed, 221 insertions(+), 11 deletions(-) diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index 795884c4d..e48711eac 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -2835,6 +2835,9 @@ directory that the starts authorize: | `a_selection_that_names_no_template_leases_nothing` | A start that names a template store that does not exist, then, once `remove` retired the generation, which no lease kept, a start of it | Status 10, with the record of a selection that names no template, by the evidence `no-store`, then `retired`, without a lease or a VM; and a store that indexes no generation | | `a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down` | A run over a scratch image, as nobody, whose execution names its environment and its working directory, and ends with status 3, then a run in RAM, as nobody, whose execution names an identity of its own | Each with status 0, in one command: the preflight's `staged`, the clone started as a start starts one, the execution, and a stop that releases the lease, without a VM; the first execution's output, of the Ubuntu image's root, as nobody, in its working directory, with its environment, and its status; the second refused with `EINVAL`; no lease left, and the payload with nothing but the capture's files | | `a_run_whose_clone_does_not_start_runs_nothing` | A run that names a template store that does not exist, one with a clock fixture past the restore window, and one over a scratch image of zeros | Once the preflight staged the images, status 10, with the record of a selection that names no template, by the evidence `no-store`, then of an `expired` generation, without a lease or a VM; then status 8, with the record of a failure of the request at the activation, once the VM exited, the lease released, and no execution or stop | +| `a_run_in_a_template_of_a_network_finds_its_policy_s_proxy` | Of a template of a proxy-only network, a run of its network, whose execution, as user ID 0, runs `env` with an entry of its own, then a run of a network that lets nothing through | The first with status 0, its clone started and stopped as a run's, and the execution's own entry, then the proxy's URL, `http://192.0.2.1:PORT`, at the port of the proxy's listener, in `http_proxy`, `https_proxy`, `HTTP_PROXY`, and `HTTPS_PROXY`; then, once the preflight staged the images, status 8, with NVX's reason, and the record of a failure of the request, at the plan, without a VM, by the evidence of invalid settings, which quarantines nothing, the lease released, and no execution or stop | +| `a_run_s_execution_finds_its_shares_beneath_the_root` | A run over a scratch image, sharing a host directory read-write, whose execution, as user ID 0, reads a file of the directory and writes another | Status 0, its clone started with one mapping and stopped as a run's; the file's contents in the execution's output; and the written file in the host's directory once the run tore the clone down | +| `a_run_whose_clone_s_vm_exits_during_its_execution_ends_with_the_clone_s_failure` | A run over a scratch image, sharing a host directory read-write, whose execution, as user ID 0, writes a file there, then waits, while the test terminates the clone's VMM | Status 8, its clone started with one mapping, the diagnostic of the executions' failure, `stopped=exited`, and the record of a failure of an unknown cause, at the run, by the evidence of the clone's lost status, without a status of the guest's, once the VM exited, which quarantines nothing, and no execution; the lease released, with the records; then another clone that starts and stops | Every network that the scenarios give a template is one that a template takes: a network that lets nothing through, egress allowed, with outbound diff --git a/tests/runtime-adapter/tests/vm_clone.rs b/tests/runtime-adapter/tests/vm_clone.rs index f670d9648..00a810461 100644 --- a/tests/runtime-adapter/tests/vm_clone.rs +++ b/tests/runtime-adapter/tests/vm_clone.rs @@ -131,6 +131,16 @@ //! `/etc/resolv.conf`. //! - A clone of a template with a NIC maps its export once it is ready, below the binds of its //! resolver configuration and of a generated machine ID. +//! - A run of a clone of a proxy-only network gives its execution the URL of the network's proxy, +//! as the executions of a started clone find it, and a run of another network, which NVX's plan +//! refuses before any VMM starts, once the preflight staged the clone's images, runs nothing. +//! - A run's execution finds the run's shares beneath the clone's root, since the restore +//! sequence mapped them before the clone's sandbox started, and the host's directory keeps what +//! it wrote through them once the run tore the clone down. +//! - A run whose clone's VM ends without a word of its guest during the execution ends with the +//! status of the clone's failure, as NVX attributes the loss of its status, after the +//! execution's diagnostic, once its stop, which stops nothing, released the lease: the template +//! still serves clones. //! //! The tests boot VMs, so they are ignored. They read what the `common` module lists, and take //! the template's image and the scratch template from its fixtures. @@ -191,7 +201,7 @@ use runtime_adapter::staged::Staged; use serde_json::{Value, json}; use common::gpt::{self, Contents, Layout, Partition, hexadecimal, small_layer}; -use common::{Fixtures, Printed, Sandbox}; +use common::{Fixtures, Printed, Sandbox, Started}; /// The exit status of the host program's command that failed, after a diagnostic. const FAILED: i32 = 1; @@ -845,9 +855,8 @@ impl Template { } /// Runs the host program's `command`, `start` or `run`, of a clone in a new sandbox of the - /// scenario `name`, as `start` asks, with the arguments `execution` after the start's options, - /// checks that it exits with `status`, and returns the clone. Before a start, the sandbox's - /// store stages the clone's images, unless `start` says otherwise. + /// scenario `name`, as [`begin_in`](Self::begin_in) begins it, checks that it exits with + /// `status`, and returns the clone. fn clone_in( &self, name: &str, @@ -856,6 +865,26 @@ impl Template { execution: &[&str], status: i32, ) -> Cloned { + let (sandbox, started) = self.begin_in(name, command, start, execution); + let started = sandbox.finish(started, status); + Cloned { + sandbox, + started, + ran: command == "run", + } + } + + /// Begins the host program's `command`, `start` or `run`, of a clone in a new sandbox of the + /// scenario `name`, as `start` asks, with the arguments `execution` after the start's options, + /// and returns the sandbox, with the command, which may still run. Before a start, the + /// sandbox's store stages the clone's images, unless `start` says otherwise. + fn begin_in( + &self, + name: &str, + command: &str, + start: &StartOf<'_>, + execution: &[&str], + ) -> (Sandbox, Started) { let mut sandbox = Sandbox::new(name); let (distro, identity) = match start.image { None => (erofs(&self.image), self.identity.as_str()), @@ -924,12 +953,7 @@ impl Template { .map(|time| (HOST_TIME_VARIABLE, time.as_os_str())) .collect(); let started = sandbox.start(command, &options, &environment); - let started = sandbox.finish(started, status); - Cloned { - sandbox, - started, - ran: run, - } + (sandbox, started) } } @@ -1044,9 +1068,15 @@ impl Cloned { /// preflight staged the clone's images, it started the clone as a start does, leasing the /// generation, and it stopped the clone as a stop does, which released the lease. fn assert_ran(&self) { + self.assert_ran_mapped(0); + } + + /// Checks that the run ran the clone, as [`assert_ran`](Self::assert_ran) does, once the + /// clone's guest applied `mapped` mappings of its export, none for a clone without one. + fn assert_ran_mapped(&self, mapped: usize) { let ran = &self.started; assert_eq!(ran["preflight"], "staged", "{ran:?}"); - assert_clone_started(ran, 0); + assert_clone_started(ran, mapped); ran["pid"].parse::().unwrap(); self.assert_stopped(ran); } @@ -1483,6 +1513,12 @@ fn scratch_image(sandbox: &Sandbox) -> PathBuf { sandbox.root.join("images").join("scratch.ext4") } +/// Returns what `printed`, what an `exec` or a run printed, says of its first execution: how it +/// started, how it ended, and its standard output and error, as the host program escapes them. +fn execution_facts(printed: &Printed) -> [&str; 4] { + ["launch", "status", "stdout", "stderr"].map(|key| printed[&format!("exec.1.{key}")].as_str()) +} + /// Checks that `started`, what a start, or a run, printed, says that it started a clone of the /// template's generation, with a lease of its own, at a downtime, whose guest agent, of the /// host's build, assembled its root, applied `mapped` mappings of its export, none for a clone @@ -3866,3 +3902,174 @@ fn a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver() { assert!(template.leases().is_empty()); assert_payload(&template, false); } + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_run_in_a_template_of_a_network_finds_its_policy_s_proxy() { + let template = Template::in_network("cl-runnet", Network::Proxy); + let own = template.network_options(Network::Proxy, GUEST_NETWORK); + let own = own.iter().map(String::as_str).collect::>(); + let url = Network::Proxy.proxy().expect("the network has a proxy"); + let execution = [ + "--identity", + "0:0", + "--env", + "FIRST=1", + "--", + "/usr/bin/env", + ]; + + // The run's execution finds the URL of the proxy of its clone's network in each of the proxy's + // variables, after its own entries, as the executions of a started clone find it, from the + // plan of the run's network, then the run tears the clone down. + let start = StartOf { + options: &own, + ..StartOf::default() + }; + let ran = template.run("cl-rn", start, &execution, 0); + ran.assert_ran(); + let proxies = PROXY_VARIABLES + .map(|variable| format!("{variable}={url}\\n")) + .concat(); + let environment = format!("FIRST=1\\n{proxies}"); + assert_eq!( + execution_facts(&ran.started), + ["started", "exit 0", environment.as_str(), ""], + "{:?}", + ran.started + ); + assert!(template.leases().is_empty()); + + // A run of a network that lets nothing through, which attaches no NIC, finds no template + // there: NVX's plan refuses its clone once the preflight staged the images, and the store + // leased the generation, before any VMM starts, and the run runs nothing. + let deny = template.network_options(Network::Deny, GUEST_NETWORK); + let deny = deny.iter().map(String::as_str).collect::>(); + let start = StartOf { + options: &deny, + ..StartOf::default() + }; + let refused = template.run("cl-rn", start, &execution, CLONE_FAILED); + let printed = &refused.started; + assert_eq!(printed["preflight"], "staged", "{printed:?}"); + refused.assert_unserved( + "the template's generation 1 has a NIC, whose network every clone of it restores, but \ + the adapter's plan of the clone has none", + ); + assert!( + !printed + .keys() + .any(|key| key.starts_with("exec.") || key == "stopped"), + "{printed:?}" + ); + assert!(template.leases().is_empty()); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_run_s_execution_finds_its_shares_beneath_the_root() { + let template = Template::create("cl-runexp"); + let export = Export::write(&template, "data", &[("marker", "data")], &[]); + let shares = export.options(&[(".", "/srv/data")], &[]); + let options = shares.iter().map(String::as_str).collect::>(); + let start = StartOf { + upper: Upper::Scratch, + options: &options, + ..StartOf::default() + }; + + // The execution, as user ID 0, reads the host's file through the share, which the restore + // sequence mapped beneath the clone's root before the clone's sandbox started, and writes + // another one through it. + let script = "cat /srv/data/marker; printf run > /srv/data/written"; + let execution = ["--identity", "0:0", "--", "/bin/sh", "-c", script]; + let ran = template.run("cl-re", start, &execution, 0); + ran.assert_ran_mapped(1); + assert_eq!( + execution_facts(&ran.started), + ["started", "exit 0", "data", ""], + "{:?}", + ran.started + ); + + // Once the run tore the clone down, the host's directory holds what the execution wrote. + let written = [("marker", "data"), ("written", "run")]; + assert_eq!(export.tree(), tree_of(&[], &written)); + assert!(template.leases().is_empty()); + assert_payload(&template, false); +} + +#[test] +#[ignore = "boots VMs; see the module documentation for what they need"] +fn a_run_whose_clone_s_vm_exits_during_its_execution_ends_with_the_clone_s_failure() { + let template = Template::create("cl-runlost"); + let export = Export::write(&template, "data", &[], &[]); + let shares = export.options(&[(".", "/srv/data")], &[]); + let options = shares.iter().map(String::as_str).collect::>(); + let start = StartOf { + upper: Upper::Scratch, + options: &options, + ..StartOf::default() + }; + + // The execution, as user ID 0, tells the host through the share that it runs, then waits, + // while the clone's VMM ends without a word of its guest, as a crash of the VMM would end it. + let script = "printf running > /srv/data/running; exec sleep 600"; + let execution = ["--identity", "0:0", "--", "/bin/sh", "-c", script]; + let (sandbox, mut running) = template.begin_in("cl-rl", "run", &start, &execution); + let marker = export.path("running"); + assert_eq!(sandbox.paused(&mut running, &marker), "running"); + let vmm = sandbox.vmm().expect("the clone's VMM runs"); + vmm.kill().unwrap(); + assert!( + vmm.wait(Instant::now() + Duration::from_secs(10)).unwrap(), + "the clone's VMM still runs" + ); + + // The execution fails to run, and the run's stop stops nothing: the clone's VM ended without + // its holder, with no report of the guest's own end, so the clone fails, as NVX attributes the + // loss of its status, which quarantines nothing, and the run ends with the status of the + // clone's failure, after the execution's diagnostic, once it released the lease. + let ended = sandbox.finish(running, CLONE_FAILED); + assert_eq!(ended["preflight"], "staged", "{ended:?}"); + assert_clone_started(&ended, 1); + assert_eq!(ended["stopped"], "exited", "{ended:?}"); + assert_eq!( + failure(&ended), + [ + "unknown", + "lost-status", + "run", + "committed", + "none", + "exited" + ], + "{ended:?}" + ); + assert_eq!(ended["failure.code"], "exited"); + assert_eq!(ended["quarantined"], "no"); + assert_eq!(ended["released"], ended["lease"], "{ended:?}"); + assert!( + !ended.keys().any(|key| key.starts_with("exec.")), + "{ended:?}" + ); + let stderr = sandbox.stderr(); + for diagnostic in [ + "the executions failed", + "without an outcome report of the guest's own end", + ] { + assert!(stderr.contains(diagnostic), "{stderr}"); + } + assert!(sandbox.record().is_none()); + let store = SandboxStore::open(&sandbox.store).unwrap(); + assert!(CloneRecords::read(&store).unwrap().is_none()); + assert!(template.leases().is_empty()); + assert!(template.store().tombstones().unwrap().is_empty()); + + // The template still serves clones. + let mut another = template.start("cl-rm", StartOf::default(), 0); + another.assert_started(); + another.stop(); + assert_payload(&template, false); +} From 41a4181163435a20e7b6435c57ad51a6971fd58d Mon Sep 17 00:00:00 2001 From: Enrique Saurez Date: Mon, 26 Oct 2026 12:30:32 -0700 Subject: [PATCH 5/5] runtime-adapter: run the stage's scenarios as one suite, by name The adapter's CI step ran its ignored tests, so a VM scenario that was renamed or gone dropped out of it without a failure, and the scenarios without a hypervisor ran only in the nvx-rs job. tests/runtime-adapter/scenarios.txt now names the stage's consolidated suite, each scenario by its test target and test: every operation's outcomes, from create to run, without a hypervisor and in VMs, and every scenario of vm_template, vm_clone, vm_layered, vm_restart and vm_steps, 155 in all. nvx.py test-runtime-adapter runs it. It takes the adapter's test targets from Cargo's metadata and lists the tests of each named one, and fails unless the lines name exactly those tests and neither the library nor another test target has an ignored test. It then runs each target's scenarios by name with --exact, the ignored ones included, and fails unless each of them reported ok and the target passed as many tests as it names. --check-only checks the names alone. The microVM workflow's runtime adapter steps run it on each backend in place of cargo test -- --ignored, and validate-nvx checks its help. The README lists every scenario with what it proves, in the order of the manifest, which a unit test keeps in step. The usage, CI and contribution documents describe the command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7d3db585-441c-495b-901f-ab7203c0f36b --- .github/actions/validate-nvx/action.yml | 3 + .github/workflows/run-nvx-microvm-tests.yml | 50 +-- doc/ci.md | 15 +- doc/contribute.md | 5 +- doc/usage.md | 29 ++ scripts/nvx.py | 9 + scripts/nvx_tools/ci.py | 5 +- scripts/nvx_tools/runtime_adapter_tests.py | 310 +++++++++++++++++ scripts/test_nvx_tools.py | 366 +++++++++++++++++++- tests/runtime-adapter/README.md | 199 ++++++++++- tests/runtime-adapter/scenarios.txt | 173 +++++++++ 11 files changed, 1106 insertions(+), 58 deletions(-) create mode 100644 scripts/nvx_tools/runtime_adapter_tests.py create mode 100644 tests/runtime-adapter/scenarios.txt diff --git a/.github/actions/validate-nvx/action.yml b/.github/actions/validate-nvx/action.yml index e48ab5d28..8892a804a 100644 --- a/.github/actions/validate-nvx/action.yml +++ b/.github/actions/validate-nvx/action.yml @@ -18,6 +18,7 @@ runs: python3 scripts/nvx.py test-openvmm-unit --help python3 scripts/nvx.py test-openvmm --help python3 scripts/nvx.py test-microvm --help + python3 scripts/nvx.py test-runtime-adapter --help python3 scripts/nvx.py test-adversarial --help python3 scripts/nvx.py benchmark --help python3 scripts/nvx.py doctor --help @@ -45,6 +46,8 @@ runs: if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\nvx.py test-microvm --help if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + python scripts\nvx.py test-runtime-adapter --help + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\nvx.py test-adversarial --help if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } python scripts\nvx.py benchmark --help diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 42e15292d..b1b13092f 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -1274,19 +1274,23 @@ jobs: Invoke-Nvx sandbox deprovision --state-dir $state Remove-Item -Recurse -Force $share, $scratch - # The runtime adapter's hypervisor scenarios on this backend: the ignored - # tests of its host-only configuration, which read the backend through - # `nvx::Hypervisor::from_env_or_default`. Its host program launches VMs - # of the artifacts above, with the runtime identity that their build - # provenance gives, whose initramfs runs the adapter's guest agent in - # place of NVX's agents. Later processes of the host program reattach to - # a VM, as after a restart of the orchestrator, and stop it. The layered - # scenarios boot the Ubuntu EROFS image over a copy of the scratch - # template, from the guest artifacts, as the sandbox steps above do, and - # the guest agent's worker assembles the root outside PID 1. The guest - # agent is a static musl executable: Linux builds it, and Windows takes - # the one that distro-artifacts builds on Linux, as it takes the Ubuntu - # and Azure Linux images. + # The runtime adapter's consolidated suite on this backend: each scenario + # that tests/runtime-adapter/scenarios.txt names, by name, in its + # host-only configuration, whose VM scenarios are its ignored tests and + # read the backend through `nvx::Hypervisor::from_env_or_default`. The + # command fails once a scenario names a test that its target does not + # have, a named target has a test that no scenario names, the library or + # another target has an ignored test, or a named test does not run and + # pass. Its host program launches VMs of the artifacts above, with the + # runtime identity that their build provenance gives, whose initramfs runs + # the adapter's guest agent in place of NVX's agents. Later processes of + # the host program reattach to a VM, as after a restart of the + # orchestrator, and stop it. The layered scenarios boot the Ubuntu EROFS + # image over a copy of the scratch template, from the guest artifacts, as + # the sandbox steps above do, and the guest agent's worker assembles the + # root outside PID 1. The guest agent is a static musl executable: Linux + # builds it, and Windows takes the one that distro-artifacts builds on + # Linux, as it takes the Ubuntu and Azure Linux images. - name: Build the runtime adapter's guest agent on Linux if: runner.os != 'Windows' && inputs.guests-and-sandboxes && !inputs.debug-kernel shell: bash @@ -1298,12 +1302,10 @@ jobs: - name: Run runtime adapter scenarios on Linux if: runner.os != 'Windows' && inputs.guests-and-sandboxes && !inputs.debug-kernel shell: bash - working-directory: tests/runtime-adapter - env: - NVX_HYPERVISOR: ${{ inputs.backend }} - NVX_ARTIFACTS_DIR: ${{ github.workspace }} - RUNTIME_ADAPTER_GUEST: ${{ github.workspace }}/tests/runtime-adapter/target/x86_64-unknown-linux-musl/release/runtime-adapter-guest - run: cargo test --locked --features host -- --ignored + run: >- + python3 scripts/nvx.py test-runtime-adapter + --backend "${{ inputs.backend }}" + --guest-agent tests/runtime-adapter/target/x86_64-unknown-linux-musl/release/runtime-adapter-guest - name: Download the runtime adapter's guest agent on Windows if: runner.os == 'Windows' && inputs.guests-and-sandboxes && !inputs.debug-kernel @@ -1315,12 +1317,10 @@ jobs: - name: Run runtime adapter scenarios on Windows if: runner.os == 'Windows' && inputs.guests-and-sandboxes && !inputs.debug-kernel shell: powershell - working-directory: tests/runtime-adapter - env: - NVX_HYPERVISOR: ${{ inputs.backend }} - NVX_ARTIFACTS_DIR: ${{ github.workspace }} - RUNTIME_ADAPTER_GUEST: ${{ github.workspace }}\build\runtime-adapter\runtime-adapter-guest - run: cargo test --locked --features host -- --ignored + run: >- + python scripts\nvx.py test-runtime-adapter + --backend "${{ inputs.backend }}" + --guest-agent build\runtime-adapter\runtime-adapter-guest - name: Upload NVX microVM failure logs if: failure() diff --git a/doc/ci.md b/doc/ci.md index 80eb463cf..976693800 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -1858,9 +1858,14 @@ anything, it checks that each platform's package ships `bin/openvmm`, which `nvx-rs download` installs, and `bin/nvx-rs`, or their `.exe` forms on Windows. Each `nvx-microvm-guests-{kvm,mshv,whp}` job builds the runtime adapter's -host-only configuration on its backend and runs its ignored tests, its VM -scenarios, with `NVX_HYPERVISOR` naming the backend. Through NVX's -supervisor, the adapter's host program +host-only configuration on its backend and runs its consolidated suite, the +[stage scenarios](../tests/runtime-adapter/README.md#stage-scenarios), with +`scripts/nvx.py test-runtime-adapter`, which names the backend in +`NVX_HYPERVISOR`. The command runs each scenario that +`tests/runtime-adapter/scenarios.txt` names, by name, its VM scenarios +included, which are the adapter's ignored tests, and fails once a scenario is +renamed, added, or gone, or does not run and pass. Through NVX's supervisor, +the adapter's host program launches a VM of the job's artifacts, whose initramfs runs the adapter's guest agent in place of NVX's agents, with the runtime identity that the artifacts' build provenance gives. Later processes reattach to the VM, as after a restart @@ -1872,7 +1877,9 @@ where the guest agent's worker, outside PID 1, assembles the root through beneath it, through the root and through `nvx::guest::mappings` with reserved targets of its own, and those that the mapper refuses, and that a failed assembly, an invalid root configuration, and a failed teardown each reach the -host program with the stage that failed, and still end the VM cleanly. The +host program with the stage that failed, and still end the VM cleanly. Its +template and clone scenarios capture templates, and start and run their +clones. The Linux jobs build the guest agent, a static musl executable, first, and the Windows job takes the one that `distro-artifacts` builds on Linux, as it takes the Ubuntu and Azure Linux images. The adapter's other tests, including those diff --git a/doc/contribute.md b/doc/contribute.md index 48d83fe31..44b5f870e 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -119,8 +119,9 @@ mechanics, such as the conformance vectors of the control protocol, belong in `nvx`. Tests of what an independent adapter observes through the public API of `nvx` alone, such as its own framing over the control session between the host and the guest, belong in the -[runtime adapter](../tests/runtime-adapter/README.md), whose ignored tests the -microVM jobs run on each backend. [Continuous integration](ci.md) +[runtime adapter](../tests/runtime-adapter/README.md), whose consolidated +suite the microVM jobs run on each backend with +`scripts/nvx.py test-runtime-adapter --backend BACKEND`. [Continuous integration](ci.md) describes what the OpenVMM VMM tests and the NVX microVM suite cover. ## Test expectations diff --git a/doc/usage.md b/doc/usage.md index 40b0c8c50..ca4d4c01e 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -438,6 +438,7 @@ is authorized for the organization when it enforces single sign-on. | `test-microvm` | Run NVX Linux and device correctness tests through OpenVMM. | | `plan-microvm-shards` | Split the default microVM suite into shards of balanced scenario durations for CI. | | `doctor` | Qualify this host for the NVX time ABI. | +| `test-runtime-adapter` | Run the runtime adapter's consolidated suite by name on a real hypervisor. | | `test-adversarial` | Run a brokered Copilot-driven adversarial campaign. | | `build` | Build the guest artifacts and OpenVMM. | | `benchmark` | Run the OpenVMM-native benchmark coordinator. | @@ -1266,6 +1267,34 @@ check and the subset that CI runs. | `--summary PATH` | none | Append a Markdown summary, for example to `$GITHUB_STEP_SUMMARY`. | | `--timeout SECONDS` | `120` | Set the seconds allowed for each probe or guest. `H4` gets its sampling window on top. | +### `test-runtime-adapter` + +```text +python3 scripts/nvx.py test-runtime-adapter + --backend {kvm,mshv,whp} + [--guest-agent PATH] + [--check-only] + [--cargo CARGO] +``` + +Runs the consolidated suite of the [runtime adapter](../tests/runtime-adapter/README.md), +the [stage scenarios](../tests/runtime-adapter/README.md#stage-scenarios) that +`tests/runtime-adapter/scenarios.txt` names, against a real hypervisor. Each +line of that file names one of the adapter's test targets, then one of its +tests. The command first checks that the lines name exactly the tests of their +targets, and that neither the adapter's library nor another target has an +ignored test, as the adapter's VM scenarios are, so that a scenario that is +renamed, added, or gone fails it rather than drop out of the suite. It then +runs each target's scenarios by +name, in the adapter's host-only configuration, the ignored ones with the +others, with the backend in `NVX_HYPERVISOR`, NVX's artifacts from the +directory that `NVX_ARTIFACTS_DIR` names, from the current directory if it is +relative, or this checkout unless it names one, and the adapter's guest agent +from `--guest-agent` (default +`tests/runtime-adapter/target/x86_64-unknown-linux-musl/release/runtime-adapter-guest`), +and fails unless each of them ran and passed. `--check-only` checks the names +alone, and needs neither a hypervisor nor the guest agent. + ### `test-adversarial` ```text diff --git a/scripts/nvx.py b/scripts/nvx.py index c70f05e7f..f2d7caae3 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -105,6 +105,9 @@ package_release, verify_source_tree, ) +from nvx_tools.runtime_adapter_tests import ( + configure_parser as configure_runtime_adapter_test_parser, +) from nvx_tools.shard_plans import parse_shard from nvx_tools.time_abi import ( benchmark_cpu_profile_refusal, @@ -867,6 +870,12 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) configure_doctor_parser(doctor) + runtime_adapter_tests = subparsers.add_parser( + "test-runtime-adapter", + help="run the runtime adapter's consolidated suite by name on a real hypervisor", + ) + configure_runtime_adapter_test_parser(runtime_adapter_tests) + adversarial_tests = subparsers.add_parser( "test-adversarial", help="run a brokered Copilot-driven adversarial campaign", diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index 16ed82779..a29db5fab 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -262,8 +262,9 @@ r"scripts/nvx_tools/(?!(adversarial|adversarial_broker|adversarial_executor|" r"crate_release|create_linux_source_archive|managed_exec_tests|" r"microvm_durations|microvm_shards|microvm_tests|nightly_ci|" - r"openvmm_process|openvmm_test_shards|sandbox_lifecycle_tests|" - r"sandbox_proxy_tests|sandbox_ram_upper_tests|shard_plans)\.py$|" + r"openvmm_process|openvmm_test_shards|runtime_adapter_tests|" + r"sandbox_lifecycle_tests|sandbox_proxy_tests|sandbox_ram_upper_tests|" + r"shard_plans)\.py$|" r"(microvm|openvmm_test)_shards\.json$|adversarial_cases/|" r"microvm_test_scripts/))" ) diff --git a/scripts/nvx_tools/runtime_adapter_tests.py b/scripts/nvx_tools/runtime_adapter_tests.py new file mode 100644 index 000000000..f8295a674 --- /dev/null +++ b/scripts/nvx_tools/runtime_adapter_tests.py @@ -0,0 +1,310 @@ +"""The runtime adapter's consolidated suite, run by name on a real hypervisor. + +`tests/runtime-adapter/scenarios.txt` names the suite's scenarios, each a test of one of +the adapter's test targets. The command checks that the scenarios name exactly the tests +of their targets, and that neither the library nor another target has an ignored test, +as the scenarios that boot VMs are, so that a scenario that is renamed, added, or gone +fails the command rather than drop out of it. It then runs each target's scenarios by +name, the ignored ones with the others, and checks that each of them ran and passed. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shlex +import subprocess +import sys +from collections.abc import Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any, cast + +from .build_constants import BuildConstants, OpenVMMBuildConstants +from .ci import OPENVMM_TEST_BACKENDS, validate_openvmm_test_backend +from .common import ScriptError, require_file + +ADAPTER_DIRECTORY = BuildConstants.REPO_ROOT / "tests" / "runtime-adapter" +PACKAGE = "runtime-adapter" +SCENARIOS = ADAPTER_DIRECTORY / "scenarios.txt" +GUEST_AGENT = ( + ADAPTER_DIRECTORY + / "target" + / OpenVMMBuildConstants.MUSL_RUST_TARGET + / "release" + / "runtime-adapter-guest" +) +# libtest's line for a test that ended, and the summary of a test target's run. +_TEST_LINE = re.compile(r"^test (\S+) \.\.\. (\S+)") +_RESULT_LINE = re.compile(r"^test result: \S+\. (\d+) passed; ") + + +@dataclass(frozen=True) +class Target: + """A test target of the adapter, with the tests of it that the scenarios name.""" + + name: str + tests: tuple[str, ...] + + +def configure_parser(parser: argparse.ArgumentParser) -> None: + parser.description = ( + "Run the runtime adapter's consolidated suite on a real hypervisor: each " + "scenario that tests/runtime-adapter/scenarios.txt names, by its name." + ) + parser.add_argument("--backend", choices=OPENVMM_TEST_BACKENDS, required=True) + parser.add_argument( + "--guest-agent", + type=Path, + default=GUEST_AGENT, + help="the adapter's guest agent, a static musl executable (default: its " + "release build in tests/runtime-adapter)", + ) + parser.add_argument( + "--check-only", + action="store_true", + help="check that the scenarios name exactly the tests of their targets, and " + "run none", + ) + parser.add_argument( + "--cargo", + default="cargo", + help="cargo executable used to build and run the tests (default: cargo)", + ) + parser.set_defaults(handler=command_test_runtime_adapter) + + +def read_scenarios(path: Path = SCENARIOS) -> list[Target]: + """Returns the targets that the scenarios of `path` name, in the order of their + first lines, each with its tests in the order of their lines. + + A line names a target, then one of its tests; `#` starts a comment. Raises + `ScriptError` for another line, and a scenario that two lines name. + """ + tests: dict[str, list[str]] = {} + for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + fields = line.split("#", 1)[0].split() + if not fields: + continue + where = f"{path}:{number}" + if len(fields) != 2: + raise ScriptError(f"{where}: expected a target and a test, not {line!r}") + target, test = fields + named = tests.setdefault(target, []) + if test in named: + raise ScriptError(f"{where}: {target} {test} is named twice") + named.append(test) + if not tests: + raise ScriptError(f"{path} names no scenario") + return [Target(name, tuple(named)) for name, named in tests.items()] + + +def cargo_test_targets(cargo: str, environment: dict[str, str]) -> list[str]: + """Returns the names of the adapter's test targets as Cargo finds them, those of + the directories of its `tests` directory that hold a `main.rs` included.""" + command = [cargo, "metadata", "--locked", "--no-deps", "--format-version", "1"] + found = listing(command, environment) + if found.returncode != 0: + raise ScriptError(f"cannot read the adapter's targets:\n{found.stderr}") + metadata = cast(dict[str, Any], json.loads(found.stdout)) + for package in cast(list[dict[str, Any]], metadata["packages"]): + if package["name"] == PACKAGE: + return sorted( + cast(str, target["name"]) + for target in cast(list[dict[str, Any]], package["targets"]) + if "test" in target["kind"] + ) + raise ScriptError(f"Cargo's metadata of the adapter has no package {PACKAGE}") + + +def cargo_test(cargo: str, target: str | None, *arguments: str) -> list[str]: + """Returns the command that runs the test target `target` of the adapter's + host-only configuration, or the unit tests of its library for `None`, with + libtest's `arguments`.""" + selection = ["--lib"] if target is None else ["--test", target] + return [ + cargo, + "test", + "--locked", + "--features", + "host", + *selection, + "--", + *arguments, + ] + + +def listed_tests(output: str) -> list[str]: + """Returns the tests that libtest's terse list `output` names.""" + suffix = ": test" + return [ + line.removesuffix(suffix) + for line in output.splitlines() + if line.endswith(suffix) + ] + + +def naming_problems(target: Target, tests: Sequence[str]) -> list[str]: + """Returns why the scenarios of `target` do not name exactly `tests`, the tests + that the target has.""" + present = set(tests) + named = set(target.tests) + problems = [ + f"{target.name} has no test {test}" + for test in target.tests + if test not in present + ] + problems.extend( + f"{target.name} has the test {test}, which no scenario names" + for test in tests + if test not in named + ) + return problems + + +def run_problems(target: Target, output: str, status: int) -> list[str]: + """Returns why the run of the scenarios of `target`, which printed `output` and + exited with `status`, does not show that each of them ran and passed.""" + outcomes: dict[str, str] = {} + passed = 0 + for line in output.splitlines(): + if (ended := _TEST_LINE.match(line)) is not None: + outcomes[ended.group(1)] = ended.group(2) + elif (result := _RESULT_LINE.match(line)) is not None: + passed += int(result.group(1)) + problems: list[str] = [] + for test in target.tests: + outcome = outcomes.get(test) + if outcome is None: + problems.append(f"{target.name} {test} did not run") + elif outcome != "ok": + problems.append(f"{target.name} {test}: {outcome}") + if passed != len(target.tests): + problems.append(f"{target.name} passed {passed} tests, not {len(target.tests)}") + if status != 0 and not problems: + problems.append(f"the tests of {target.name} exited with status {status}") + return problems + + +def listing( + command: list[str], environment: dict[str, str] +) -> subprocess.CompletedProcess[str]: + """Runs `command`, which lists tests, in the adapter's directory, and returns what + it printed.""" + return subprocess.run( + command, + cwd=ADAPTER_DIRECTORY, + env=environment, + check=False, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + + +def streamed(command: list[str], environment: dict[str, str]) -> tuple[str, int]: + """Runs `command` in the adapter's directory, printing its output as it comes, + and returns that output and the command's status.""" + print(f">> {shlex.join(command)}", flush=True) + lines: list[str] = [] + with subprocess.Popen( + command, + cwd=ADAPTER_DIRECTORY, + env=environment, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + encoding="utf-8", + errors="replace", + ) as process: + assert process.stdout is not None + for line in process.stdout: + sys.stdout.write(line) + sys.stdout.flush() + lines.append(line) + status = process.wait() + return "".join(lines), status + + +def suite_problems( + targets: Sequence[Target], cargo: str, environment: dict[str, str] +) -> list[str]: + """Returns why `targets` are not the adapter's suite: each target that is not one + of the adapter's test targets, each scenario that names a test that its target does + not have, each test of a named target that no scenario names, and each ignored test + of the library or of a test target that no scenario names.""" + available = cargo_test_targets(cargo, environment) + problems: list[str] = [] + for target in targets: + if target.name not in available: + problems.append(f"{target.name} is not a test target of the adapter") + continue + listed = listing( + cargo_test(cargo, target.name, "--list", "--format", "terse"), environment + ) + if listed.returncode != 0: + problems.append(f"cannot list the tests of {target.name}:\n{listed.stderr}") + continue + problems.extend(naming_problems(target, listed_tests(listed.stdout))) + named = {target.name for target in targets} + others = [name for name in available if name not in named] + for name in [None, *others]: + command = cargo_test(cargo, name, "--ignored", "--list", "--format", "terse") + listed = listing(command, environment) + what = "the library" if name is None else name + if listed.returncode != 0: + problems.append(f"cannot list the tests of {what}:\n{listed.stderr}") + continue + problems.extend( + f"{what} has the ignored test {test}, which no scenario names" + for test in listed_tests(listed.stdout) + ) + return problems + + +def command_test_runtime_adapter(args: argparse.Namespace) -> int: + backend = cast(str, args.backend) + cargo = cast(str, args.cargo) + check_only = cast(bool, args.check_only) + targets = read_scenarios() + environment = os.environ.copy() + if not check_only: + validate_openvmm_test_backend(backend) + guest = require_file( + cast(Path, args.guest_agent), "runtime adapter guest agent" + ) + environment["NVX_HYPERVISOR"] = backend + # Cargo runs the tests in the adapter's directory, so a relative directory of + # artifacts is the caller's, and NVX takes an empty one for none. + artifacts = environment.get("NVX_ARTIFACTS_DIR") + environment["NVX_ARTIFACTS_DIR"] = os.fspath( + Path(artifacts).resolve() if artifacts else BuildConstants.REPO_ROOT + ) + environment["RUNTIME_ADAPTER_GUEST"] = os.fspath(guest.resolve()) + problems = suite_problems(targets, cargo, environment) + if not problems and not check_only: + for target in targets: + command = cargo_test( + cargo, target.name, "--include-ignored", "--exact", *target.tests + ) + output, status = streamed(command, environment) + problems.extend(run_problems(target, output, status)) + count = sum(len(target.tests) for target in targets) + if problems: + print( + f"the runtime adapter's suite of {count} scenarios failed:", file=sys.stderr + ) + for problem in problems: + print(f" {problem}", file=sys.stderr) + return 1 + if check_only: + print( + f"{count} scenarios name exactly the tests of {len(targets)} targets, and " + "neither the library nor another target has an ignored test" + ) + else: + print(f"the {count} scenarios of {len(targets)} targets passed on {backend}") + return 0 diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 19aeb7f92..92c355bc9 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -59,6 +59,7 @@ openvmm_test_shards, performance, release, + runtime_adapter_tests, shard_plans, ubuntu, ) @@ -6998,6 +6999,8 @@ def load(name: str) -> Any: "openvmm-test-cache-key": set[str](), "check-openvmm-tests": set[str](), "reuse-openvmm-tests": set[str](), + # The runtime adapter's VM scenarios boot what nvx finds. + "test-runtime-adapter": kernel_alpine, # A sandbox boots the Alpine control initramfs under its layers. "sandbox": kernel_alpine, } @@ -7051,9 +7054,6 @@ def reads(step: dict[str, Any], context: Mapping[str, object]) -> set[str]: else: self.assertIn(command, commands) files |= commands[command] - # The runtime adapter's hypervisor scenarios boot what nvx finds. - if "cargo test" in script: - files |= kernel_alpine return files def holds(condition: object, context: Mapping[str, object]) -> bool: @@ -8375,10 +8375,11 @@ def test_ci_change_classification_builds_artifacts_for_the_jobs_that_use_them(se ) def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(self): - # The microVM jobs run the runtime adapter's scenarios on every backend, - # so a change to the adapter alone runs them. Neither OpenVMM suite - # builds or boots the adapter, and neither the benchmarks nor the - # packages use it, so a pull request skips the platform jobs. + # The microVM jobs run the runtime adapter's suite on every backend, so + # a change to the adapter, its suite, or the command that runs it alone + # runs them. Neither OpenVMM suite builds or boots the adapter, and + # neither the benchmarks nor the packages use it, so a pull request + # skips the platform jobs. workflow = ( BuildConstants.REPO_ROOT / ".github" @@ -8391,19 +8392,21 @@ def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(sel workflow, "test", f"Run runtime adapter scenarios on {system}" ) self.assertIn("!inputs.debug-kernel", hook) - self.assertIn("working-directory: tests/runtime-adapter\n", hook) - self.assertIn("NVX_HYPERVISOR: ${{ inputs.backend }}\n", hook) - self.assertIn( - "run: cargo test --locked --features host -- --ignored", hook - ) + self.assertIn("nvx.py test-runtime-adapter\n", hook) + self.assertIn('--backend "${{ inputs.backend }}"\n', hook) adapter = ( "tests/runtime-adapter/Cargo.toml", "tests/runtime-adapter/Cargo.lock", "tests/runtime-adapter/src/lib.rs", "tests/runtime-adapter/src/bin/runtime-adapter-guest.rs", "tests/runtime-adapter/tests/independent_build.rs", + "tests/runtime-adapter/scenarios.txt", ) - for changed_files in (*((path,) for path in adapter), adapter): + for changed_files in ( + *((path,) for path in adapter), + adapter, + ("scripts/nvx_tools/runtime_adapter_tests.py",), + ): with self.subTest(changed_files=changed_files): changes = ci.classify_ci_changes("pull_request", changed_files) self.assertEqual( @@ -8437,7 +8440,7 @@ def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(sel self.assertIn( "### NVX microVM tests: run\nInputs changed: " "`tests/runtime-adapter/Cargo.toml`, `tests/runtime-adapter/Cargo.lock`, " - "`tests/runtime-adapter/src/lib.rs`, and 2 more.\n", + "`tests/runtime-adapter/src/lib.rs`, and 3 more.\n", summary, ) self.assertIn("### Platform benchmarks: skip\nNo input changed.\n", summary) @@ -8486,18 +8489,20 @@ def step(job: str, name: str) -> str: self.assertIn("name: runtime-adapter-guest\n", download) self.assertIn("path: build/runtime-adapter\n", download) for system, setup, guest_agent in ( - ("Linux", linux_build, f"${{{{ github.workspace }}}}/{agent}"), + ("Linux", linux_build, agent), ( "Windows", download, - "${{ github.workspace }}\\build\\runtime-adapter\\runtime-adapter-guest", + "build\\runtime-adapter\\runtime-adapter-guest", ), ): with self.subTest(system=system): name = f"Run runtime adapter scenarios on {system}" hook = step("test", name) - self.assertIn("NVX_ARTIFACTS_DIR: ${{ github.workspace }}\n", hook) - self.assertIn(f"RUNTIME_ADAPTER_GUEST: {guest_agent}\n", hook) + # The command takes NVX's artifacts from the checkout, into which + # the job downloaded them. + self.assertNotIn("NVX_ARTIFACTS_DIR", hook) + self.assertIn(f"--guest-agent {guest_agent}\n", hook) self.assertLess( microvm_workflow.index(setup), microvm_workflow.index(f" - name: {name}\n"), @@ -20705,6 +20710,331 @@ def test_check_lists_every_executable_and_fails_on_dynamic_ones(self): self.assertIn("not static executables", error.getvalue()) +class RuntimeAdapterSuiteTests(unittest.TestCase): + def _scenarios(self, text: str) -> Path: + """Writes `text` as the scenarios of an adapter, and returns their path.""" + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + path = Path(temporary.name) / "scenarios.txt" + path.write_text(text, encoding="utf-8") + return path + + def _run( + self, + path: Path, + listings: Mapping[tuple[str, str], str], + *, + targets: Mapping[str, str] | None = None, + check_only: bool = False, + artifacts: str | None = None, + ) -> tuple[int, list[tuple[list[str], dict[str, str]]], str, str]: + """Runs the handler on the scenarios at `path`, for an adapter whose Cargo + metadata holds the test targets `targets`, by name with their sources, and + whose targets list the tests of `listings`, by target and by the first of + libtest's arguments, and pass each run of named tests, with `artifacts` in + `NVX_ARTIFACTS_DIR`, or without it. Returns the handler's status, the runs, and + what it printed to its standard output and error.""" + if targets is None: + targets = {name: f"tests/{name}.rs" for name in ("vm_a", "unit", "other")} + package = { + "name": "runtime-adapter", + "targets": [ + {"name": "runtime_adapter", "kind": ["lib"], "src_path": "src/lib.rs"}, + {"name": "runtime-adapter-host", "kind": ["bin"], "src_path": "h.rs"}, + *( + {"name": name, "kind": ["test"], "src_path": source} + for name, source in targets.items() + ), + ], + } + metadata = json.dumps({"packages": [package]}) + runs: list[tuple[list[str], dict[str, str]]] = [] + + def fake_listing( + command: list[str], environment: dict[str, str] + ) -> subprocess.CompletedProcess[str]: + if command[1] == "metadata": + return subprocess.CompletedProcess(command, 0, metadata, "") + target = ( + command[command.index("--test") + 1] if "--test" in command else "lib" + ) + first = command[command.index("--") + 1] + return subprocess.CompletedProcess( + command, 0, listings[(target, first)], "" + ) + + def fake_streamed( + command: list[str], environment: dict[str, str] + ) -> tuple[str, int]: + runs.append((command, environment)) + names = command[command.index("--exact") + 1 :] + output = "".join(f"test {name} ... ok\n" for name in names) + return f"{output}\ntest result: ok. {len(names)} passed; 0 failed;\n", 0 + + guest = path.parent / "runtime-adapter-guest" + guest.write_bytes(b"") + args = argparse.Namespace( + backend="kvm", cargo="cargo", check_only=check_only, guest_agent=guest + ) + scenarios = runtime_adapter_tests.read_scenarios(path) + with ( + patch.object( + runtime_adapter_tests, "read_scenarios", return_value=scenarios + ), + patch.object(runtime_adapter_tests, "ADAPTER_DIRECTORY", path.parent), + patch.object(runtime_adapter_tests, "validate_openvmm_test_backend"), + patch.object(runtime_adapter_tests, "listing", fake_listing), + patch.object(runtime_adapter_tests, "streamed", fake_streamed), + patch.dict(os.environ), + contextlib.redirect_stdout(io.StringIO()) as stdout, + contextlib.redirect_stderr(io.StringIO()) as stderr, + ): + os.environ.pop("NVX_ARTIFACTS_DIR", None) + if artifacts is not None: + os.environ["NVX_ARTIFACTS_DIR"] = artifacts + status = runtime_adapter_tests.command_test_runtime_adapter(args) + return status, runs, stdout.getvalue(), stderr.getvalue() + + def test_cli_registers_the_suite(self): + args = nvx.parse_args(["test-runtime-adapter", "--backend", "kvm"]) + + self.assertEqual(args.backend, "kvm") + self.assertEqual(args.cargo, "cargo") + self.assertFalse(args.check_only) + self.assertEqual(args.guest_agent, runtime_adapter_tests.GUEST_AGENT) + self.assertIs(args.handler, runtime_adapter_tests.command_test_runtime_adapter) + + def test_ci_runs_the_suite_by_name_on_each_backend(self): + workflow = ( + BuildConstants.REPO_ROOT + / ".github" + / "workflows" + / "run-nvx-microvm-tests.yml" + ).read_text(encoding="utf-8") + + self.assertEqual(workflow.count("nvx.py test-runtime-adapter"), 2) + self.assertIn("python3 scripts/nvx.py test-runtime-adapter", workflow) + self.assertIn("python scripts\\nvx.py test-runtime-adapter", workflow) + self.assertNotIn("cargo test --locked --features host -- --ignored", workflow) + + def test_scenarios_name_targets_and_tests_in_the_order_of_their_lines(self): + path = self._scenarios("# The suite.\n\nvm_a b # last\nunit x\nvm_a a\n") + + self.assertEqual( + runtime_adapter_tests.read_scenarios(path), + [ + runtime_adapter_tests.Target("vm_a", ("b", "a")), + runtime_adapter_tests.Target("unit", ("x",)), + ], + ) + + def test_scenarios_refuse_other_lines(self): + for text, problem in ( + ("vm_a\n", r":1: expected a target and a test, not 'vm_a'"), + ("unit x\nvm_a a b\n", r":2: expected a target and a test"), + ("vm_a a\nvm_a a\n", r":2: vm_a a is named twice"), + ("# Nothing yet.\n", r"names no scenario"), + ): + with ( + self.subTest(text=text), + self.assertRaisesRegex(common.ScriptError, problem), + ): + runtime_adapter_tests.read_scenarios(self._scenarios(text)) + + def test_the_adapter_s_readme_lists_its_scenarios(self): + targets = runtime_adapter_tests.read_scenarios() + named = [(target.name, test) for target in targets for test in target.tests] + self.assertLessEqual( + {"vm_clone", "vm_template", "vm_layered", "template_operations"}, + {target.name for target in targets}, + ) + readme = (runtime_adapter_tests.ADAPTER_DIRECTORY / "README.md").read_text( + encoding="utf-8" + ) + section = re.split(r"\n#+ ", readme.split("\n### Stage scenarios\n", 1)[1])[0] + rows = re.findall(r"^\| `(\w+)` \| `(\w+)` \|", section, re.MULTILINE) + + self.assertEqual(rows, named) + + def test_libtest_s_terse_list_names_the_tests(self): + self.assertEqual( + runtime_adapter_tests.listed_tests("a: test\nb::c: test\nd: bench\n"), + ["a", "b::c"], + ) + + def test_naming_problems_name_missing_and_unnamed_tests(self): + target = runtime_adapter_tests.Target("vm_a", ("a", "gone")) + + self.assertEqual( + runtime_adapter_tests.naming_problems(target, ["gone", "a"]), [] + ) + self.assertEqual( + runtime_adapter_tests.naming_problems(target, ["a", "new"]), + ["vm_a has no test gone", "vm_a has the test new, which no scenario names"], + ) + + def test_a_run_passes_once_each_scenario_ran_and_passed(self): + target = runtime_adapter_tests.Target("vm_a", ("a", "b")) + output = ( + "running 2 tests\n" + "test b ... ok\n" + "test a has been running for over 60 seconds\n" + "test a ... ok\n\n" + "test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; " + "3 filtered out; finished in 9.00s\n" + ) + + self.assertEqual(runtime_adapter_tests.run_problems(target, output, 0), []) + + def test_a_run_fails_once_a_scenario_failed_or_did_not_run(self): + target = runtime_adapter_tests.Target("vm_a", ("a", "b")) + result = "test result: {}. {} passed; {} failed; 0 ignored; 0 measured; " + for output, status, problems in ( + ( + f"test a ... ok\ntest b ... FAILED\n\n{result.format('FAILED', 1, 1)}", + 101, + ["vm_a b: FAILED", "vm_a passed 1 tests, not 2"], + ), + ( + f"test a ... ok\n\n{result.format('ok', 1, 0)}4 filtered out\n", + 0, + ["vm_a b did not run", "vm_a passed 1 tests, not 2"], + ), + ( + f"running 0 tests\n\n{result.format('ok', 0, 0)}5 filtered out\n", + 0, + [ + "vm_a a did not run", + "vm_a b did not run", + "vm_a passed 0 tests, not 2", + ], + ), + ( + f"test a ... ok\ntest b ... ok\n\n{result.format('ok', 2, 0)}\n", + 1, + ["the tests of vm_a exited with status 1"], + ), + ("error: could not compile `runtime-adapter`\n", 101, None), + ): + with self.subTest(output=output): + found = runtime_adapter_tests.run_problems(target, output, status) + if problems is None: + self.assertIn("vm_a passed 0 tests, not 2", found) + else: + self.assertEqual(found, problems) + + def test_the_handler_checks_the_names_then_runs_each_target_by_name(self): + path = self._scenarios("vm_a a\nvm_a b\nunit x\n") + listings = { + ("vm_a", "--list"): "a: test\nb: test\n", + ("unit", "--list"): "x: test\n", + ("lib", "--ignored"): "", + ("other", "--ignored"): "", + } + + status, runs, stdout, stderr = self._run(path, listings) + + self.assertEqual(status, 0, stderr) + self.assertEqual( + [command for command, _ in runs], + [ + runtime_adapter_tests.cargo_test( + "cargo", "vm_a", "--include-ignored", "--exact", "a", "b" + ), + runtime_adapter_tests.cargo_test( + "cargo", "unit", "--include-ignored", "--exact", "x" + ), + ], + ) + environment = runs[0][1] + self.assertEqual(environment["NVX_HYPERVISOR"], "kvm") + self.assertEqual( + environment["NVX_ARTIFACTS_DIR"], os.fspath(BuildConstants.REPO_ROOT) + ) + self.assertEqual( + Path(environment["RUNTIME_ADAPTER_GUEST"]), + (path.parent / "runtime-adapter-guest").resolve(), + ) + self.assertIn("the 3 scenarios of 2 targets passed on kvm", stdout) + + # A check runs nothing, and needs neither a hypervisor nor the guest agent. + status, runs, stdout, _ = self._run(path, listings, check_only=True) + self.assertEqual((status, runs), (0, [])) + self.assertIn("3 scenarios name exactly the tests of 2 targets", stdout) + + def test_the_handler_takes_a_relative_directory_of_artifacts_from_the_caller(self): + path = self._scenarios("unit x\n") + listings = { + ("unit", "--list"): "x: test\n", + ("lib", "--ignored"): "", + ("vm_a", "--ignored"): "", + ("other", "--ignored"): "", + } + for artifacts, expected in ( + # Cargo runs the tests in the adapter's directory, not the caller's. + ("relative", Path("relative").resolve()), + (os.fspath(path.parent), path.parent.resolve()), + # NVX takes an empty directory for none, and the command this checkout. + ("", BuildConstants.REPO_ROOT), + ): + with self.subTest(artifacts=artifacts): + status, runs, _, stderr = self._run(path, listings, artifacts=artifacts) + self.assertEqual(status, 0, stderr) + self.assertEqual(Path(runs[0][1]["NVX_ARTIFACTS_DIR"]), expected) + + def test_the_handler_runs_nothing_once_the_names_are_not_the_tests(self): + path = self._scenarios("vm_a a\nvm_a gone\ngone x\n") + listings = { + ("vm_a", "--list"): "a: test\nnew: test\n", + ("lib", "--ignored"): "unit::vm: test\n", + ("other", "--ignored"): "vm: test\n", + ("vm_dir", "--ignored"): "vm: test\n", + } + targets = { + "vm_a": "tests/vm_a.rs", + "other": "tests/other.rs", + # Cargo also builds a directory of the tests directory with a main.rs. + "vm_dir": "tests/vm_dir/main.rs", + } + + status, runs, _, stderr = self._run(path, listings, targets=targets) + + self.assertEqual((status, runs), (1, [])) + for problem in ( + "vm_a has no test gone", + "vm_a has the test new, which no scenario names", + "gone is not a test target of the adapter", + "the library has the ignored test unit::vm, which no scenario names", + "other has the ignored test vm, which no scenario names", + "vm_dir has the ignored test vm, which no scenario names", + ): + self.assertIn(problem, stderr) + + def test_the_handler_refuses_an_adapter_that_cargo_does_not_describe(self): + path = self._scenarios("vm_a a\n") + for status, output, problem in ( + (101, "", "cannot read the adapter's targets"), + (0, json.dumps({"packages": []}), "has no package runtime-adapter"), + ): + + def fake_listing( + command: list[str], + environment: dict[str, str], + status: int = status, + output: str = output, + ) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(command, status, output, "error") + + with ( + self.subTest(problem=problem), + patch.object(runtime_adapter_tests, "listing", fake_listing), + self.assertRaisesRegex(common.ScriptError, problem), + ): + runtime_adapter_tests.suite_problems( + runtime_adapter_tests.read_scenarios(path), "cargo", {} + ) + + class BenchmarkTests(unittest.TestCase): def test_kvm_worker_result_decoding(self): completed = subprocess.CompletedProcess( diff --git a/tests/runtime-adapter/README.md b/tests/runtime-adapter/README.md index e48711eac..dcb17dd3f 100644 --- a/tests/runtime-adapter/README.md +++ b/tests/runtime-adapter/README.md @@ -1360,7 +1360,7 @@ The adapter's operations on templates, and on their clones, are commands of | List, inspect | `list`, `inspect` | Reports the generations, their keys, classes, states, eligibility and its reason, and leases, and the metadata of a capture, without loading VM state or hashing; see [Listings](#listings) | | Remove, retire | `remove` | Tombstones a generation, so that it takes no new lease, reports busy, with the status 7, while leases remain, and deletes it once they drained, but never moves or deletes a leased generation | | Start from a template | `start` | Starts a clone of an explicit, eligible generation, with its image, scratch, and settings, which the sandbox stages; reports a selection of no template, of an expired generation, or of an ineligible one, and a clone's failure, attributed to the request, the host, or the template, or whose teardown is unresolved; never captures, boots cold, rehashes, or falls back to another format; see [Clones](#clones) | -| Run from a template | `run` | Stages the clone's images in a preflight that it reports, starts a clone as `start` does, runs one execution in the clone's root, and tears the clone down; see [Running a clone](#running-a-clone) | +| Run from a template | `run` | Plans the clone as `start` does, with its network and shares, stages the clone's images in a preflight that it reports, starts a clone as `start` does, runs one execution in the clone's root, and tears the clone down; see [Running a clone](#running-a-clone) | `runtime-adapter-host create` serves a template of the guest agent from an NVX template store, `nvx::host::template::Store`, with `Store::create_template`: a @@ -2852,6 +2852,183 @@ A test that fails keeps its template's directory, with the template store and the platform's image, as long as it keeps the directory of a clone that leases the template, whose VM might still run. +### Stage scenarios + +The template and clone stage's scenarios run as one suite, which +[`scenarios.txt`](scenarios.txt) names: each line names a test target of +`tests/`, then one of its tests. CI's runtime adapter step runs the suite on +each backend, KVM, MSHV, and WHP, with +`python3 scripts/nvx.py test-runtime-adapter --backend BACKEND`, in the +host-only configuration: it runs each target's scenarios by name, the VM +scenarios, which are ignored, with the others, and checks that each of them ran +and passed. It fails once a line names a test that its target does not have, a +target that a line names has a test that no line names, the library or another +target has an ignored test, or a named test does not run and pass, so that a +scenario that is renamed, added, or gone fails it rather than drop out of the +suite. With `--check-only`, it checks the names alone, and runs nothing. + +The suite holds each operation's outcomes, from `create` to `run`, without a +hypervisor and in VMs, and every scenario of the stage's VMs: launches of +layered roots, captures of templates, and the starts and runs of their clones. + +| Target | Scenario | Proves | +| --- | --- | --- | +| `template_capture` | `a_capture_takes_its_class_and_inputs_from_the_request` | A capture's request: the plan of the placeholder over RAM, with the request's network, the placeholder's binding, the policy revision, the artifacts, and the class, which the network names | +| `template_capture` | `the_host_program_refuses_a_template_before_it_creates_anything` | `create` refuses a network that the plan refuses, or that it cannot read, with status 1 before it needs NVX's artifacts or creates the store | +| `template_operations` | `a_refresh_refuses_a_template_before_it_creates_anything` | `refresh` refuses a network that the plan refuses with status 1 before it needs NVX's artifacts or creates the store, as `create` does | +| `template_operations` | `an_empty_template_store_lists_no_generation` | `list` of a store without generations prints none, and `inspect` and `remove` of a generation that it never published fail with status 1, and tombstone nothing | +| `template_operations` | `the_host_program_refuses_command_lines_of_the_operations` | The command lines of `refresh`, `list`, `inspect`, and `remove` that the host program refuses, each with status 2 | +| `template_operations` | `the_operations_open_a_template_store_and_create_none` | `list`, `inspect`, and `remove` of a directory that does not exist, or that holds no template store, fail with status 1, and create nothing | +| `clone_start` | `a_clone_s_plan_request_and_record_carry_its_network` | A clone's network reaches the adapter's plan, NVX's request of the clone, and the clone's record, from which an execution finds the URL of the network's proxy | +| `clone_start` | `a_clone_s_record_names_its_template_store_and_generation_beside_its_plan` | A clone's record names its template store and its generation, which a stop reads back, beside the plan that the other commands read | +| `clone_start` | `a_clone_s_request_takes_the_staged_bindings_and_the_guest_s_defaults` | NVX's request of a clone takes the staged bindings, the memory target, the guest's defaults or the request's settings, and the export of the request's shares, and refuses images that the store does not stage | +| `clone_start` | `a_host_s_time_keeps_its_text_form` | A host's time keeps the clock fixture's text form, on both clocks, and the texts that are none are refused | +| `clone_start` | `a_run_refuses_a_network_or_shares_that_the_plan_refuses_before_it_stages_anything` | A run refuses a network that it cannot read, or that the plan refuses, and shares that it cannot export, with status 1 before it stages anything or creates the sandbox's store | +| `clone_start` | `a_run_refuses_a_selection_that_names_no_template_once_its_preflight_staged_the_images` | A run refuses an image that the plan refuses with status 1 before it stages anything, and a selection that names no template with status 10 and the refusal's record once its preflight staged the images, without a lease | +| `clone_start` | `a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_anything_else` | A start whose template store does not exist, or does not hold the generation, ends with status 10 and the refusal's record, and one whose image the plan refuses with status 1, before it needs NVX's artifacts or creates the sandbox's store | +| `clone_start` | `a_start_refuses_a_network_that_the_plan_refuses_before_anything_else` | A start refuses a network that it cannot read, or that the plan refuses, with status 1 before its image, its shares, and the template store | +| `clone_start` | `a_start_refuses_shares_that_it_cannot_export_before_anything_else` | A start refuses shares that it cannot export with status 1 before it reads the template store or creates the sandbox's store | +| `clone_start` | `the_host_program_refuses_options_that_do_not_make_a_run` | The command lines of `run` that the host program refuses, its execution's included, each with status 2, and a clock fixture that is not a host's time | +| `clone_start` | `the_host_program_refuses_options_that_do_not_make_a_start` | The command lines of `start` that the host program refuses, each with status 2, and a clock fixture that is not a host's time | +| `host_session` | `a_clone_s_order_ends_its_table_at_a_mapping_that_the_guest_fails_and_stops_the_guest` | A clone's order sends no more of its table, nor `Start`, once the guest fails a mapping, and asks the guest to stop | +| `host_session` | `a_clone_s_order_maps_the_root_of_its_export_with_an_empty_source` | A clone's order sends the mapping of its export's root, whose source is empty, in its table | +| `host_session` | `a_clone_s_order_sends_its_record_first_then_starts_its_sandbox_as_after_a_cold_boot` | A clone's order sends its activation record first, returns the activation that `Ready` reports, then sends the table in parts and `Start`, and fails on a refusal, a deadline, another build, or an answer that does not decode | +| `host_session` | `a_clone_s_session_that_ends_after_its_bootstrap_is_lost_and_the_next_takes_the_same_record` | A clone's session that a reset, the end of its stream, or its transport ended is lost, and the next session sends the same record first | +| `host_session` | `a_mapping_that_the_guest_fails_ends_the_table_and_the_host_stops_the_guest` | A mapping that the guest fails, at the first, a middle, or the last part, ends the table without `Start`, and the host stops the guest and learns how the stop ended | +| `host_session` | `a_receive_that_times_out_keeps_the_partial_message` | A receive that times out inside a message keeps what arrived, and a later receive returns the whole message | +| `host_session` | `a_reset_or_a_disconnect_in_the_middle_of_a_message_drops_it` | A reset or a disconnect in the middle of a message drops the message, in the next epoch or with a new instance | +| `host_session` | `a_send_that_fails_or_times_out_ends_the_client_with_what_it_holds` | A send that times out after part of a record, or whose transport fails, ends the client with the message that it held | +| `host_session` | `a_stop_skips_the_guest_s_late_messages_about_the_executions_that_the_driver_gave_up_on` | A stop skips the guest's late messages about the executions that the driver gave up on, before and after a teardown that failed, and no message about an execution that it did not start | +| `host_session` | `answers_whose_bodies_do_not_decode_end_the_client` | Answers whose bodies do not decode end the client | +| `host_session` | `only_a_session_that_ended_is_lost_to_a_clone_s_order` | A clone's order takes only a session that ended for lost, and its other failures for final | +| `host_session` | `requests_check_their_answers_and_end_the_client_when_they_fail` | Requests check the kind of their answers, and end the client when they fail | +| `host_session` | `the_client_attaches_and_exchanges_messages_that_span_records` | The client attaches with the launch capability, and exchanges messages that span records both ways | +| `host_session` | `the_client_configures_a_layered_root_probes_it_and_learns_how_its_teardown_ended` | The client configures a layered root, probes it, and learns how its teardown ended once it stopped | +| `host_session` | `the_client_fails_on_records_that_the_protocol_does_not_allow` | Sequence, epoch, and instance errors, stale and identity-less resets, record lengths, bootstrap identities, ERROR payloads, and header faults end the client | +| `host_session` | `the_client_reassembles_messages_from_partial_and_shared_records` | Records that end a message and start the next, and records whose bytes arrive in pieces down to one byte, give back whole messages | +| `host_session` | `the_client_refuses_messages_that_its_framing_does_not_allow` | The client refuses malformed headers before their body arrives, messages that only the host program sends, and a message beyond the framing's limit, which it does not send | +| `host_session` | `the_client_returns_the_stage_of_the_root_that_failed` | The client returns the stage of the root that failed, with its problem | +| `host_session` | `the_client_waits_for_the_guest_and_reports_refusals_while_attaching` | The client waits for the guest while it attaches, until its deadline, and reports a refusal, in an ERROR or by the end of the stream | +| `host_session` | `the_driver_ends_its_run_on_a_message_that_breaks_the_rules` | A message about an execution that the driver did not start, about none, that does not decode, or that breaks the engine's rules, and the end of the stream, end the driver's run | +| `host_session` | `the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing` | An execution's timeout fires while the guest says nothing, after which the driver cancels the execution, then fences it | +| `host_session` | `the_driver_refuses_starts_before_it_sends_them` | The driver refuses the starts that the guest would not decode or run, and a start once it ran, before it sends anything, with a client that stays usable | +| `host_session` | `the_driver_runs_executions_at_once_and_collects_their_output_and_outcomes` | The driver runs executions at once, and collects the output of each, byte-exact, and its outcome, a refused launch's included | +| `host_session` | `the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_its_run` | A run whose deadline passes in a receive leaves the client usable, and in a send that waits for a guest that stopped reading, ends it | +| `host_session` | `the_host_orders_a_layered_root_ready_then_its_table_in_parts_then_start` | The host's order: the root configuration that announces the table, the guest's readiness, each part of the table once the guest applied the one before, then `Start` | +| `host_session` | `the_host_sends_nothing_more_to_a_guest_agent_of_another_build_or_whose_root_failed` | The host sends nothing more to a guest agent of another build, or whose root failed | +| `staged_images` | `a_launch_takes_the_bindings_that_the_store_stages_for_its_images` | A launch takes the bindings that the store staged for its images, and not those of other images | +| `staged_images` | `staging_verifies_the_image_and_binds_the_scratch` | Staging verifies an image against the manifest beside it, refuses another image's, stages a GPT disk over RAM, and refuses images that no launch could attach | +| `staged_images` | `the_host_program_stages_images_in_the_store` | `stage` records the staged images, a staging that fails leaves them, and another replaces them | +| `staged_images` | `the_store_records_the_staged_images` | The record of the staged images, as written and read, and the forms that a reader refuses | +| `layered_launch` | `a_launch_claims_its_scratch_image_until_it_releases_its_checked_files` | A launch claims its scratch image, which a second launch waits for, then fails to take, until the first releases its checked files | +| `layered_launch` | `a_launch_reads_only_the_networks_that_the_plan_takes` | A launch refuses the network files that the plan refuses, each for the plan's reason | +| `layered_launch` | `a_launch_records_what_it_launched` | A launch's record holds the bindings that it checked, over a scratch image and over RAM, and a VM without a layered root has one too | +| `layered_launch` | `a_launch_refuses_images_that_are_not_staged_or_changed_since` | A launch refuses images that the store does not stage, of another format or identity, and images that changed, were replaced, or are gone since they were staged | +| `layered_launch` | `a_launch_refuses_shares_that_it_cannot_export` | A launch refuses shares outside the authorized directories, inaccessible ones, reserved targets, a read-only share inside a read-write one, and a hidden path that does not exist | +| `layered_launch` | `a_launch_renders_each_element_of_its_network_through_the_plan` | OpenVMM's arguments of each element of a policy, after the plan's block devices, the proxy's URL, and the publication of listeners, as NVX's lowering renders them | +| `layered_launch` | `a_launch_shares_host_paths_inside_the_authorized_directories` | The export of shares inside an authorized directory maps each at its target, parents first, hides a denied path, and comes with the root configuration and OpenVMM's arguments that attach it | +| `layered_launch` | `a_layered_launch_hands_openvmm_the_checked_images` | OpenVMM's arguments name the checked files, by the descriptors that OpenVMM inherits on Linux and by their final paths on Windows, without a workload identity or a managed lifecycle | +| `layered_launch` | `a_layered_launch_passes_what_the_guest_checks_as_it_is` | A launch passes on the options that the guest checks as they are | +| `layered_launch` | `a_layered_launch_plans_its_blocks_and_boot_tokens_with_the_adapters_plan` | The adapter's plan of a layered launch: its block devices and boot tokens, without the sandbox profile's settings, and the sizes of the images | +| `layered_launch` | `an_execution_finds_the_url_of_its_network_s_proxy_in_its_environment` | An execution's start carries the URL of its network's proxy after its own entries, keeps its own proxy settings, and is unchanged without a proxy | +| `layered_launch` | `shares_read_their_host_path_their_target_and_their_access` | The text of a share, `HOST:TARGET[:ro]` | +| `layered_launch` | `the_host_program_refuses_a_network_before_it_launches` | `launch --network` refuses a network that the plan refuses, or that it cannot read, with status 1 before it needs NVX's artifacts or creates the store | +| `layered_launch` | `the_host_program_refuses_options_that_do_not_make_a_command` | The command lines that the host program refuses, each with status 2, those of `exec`, `--workload-identity`, and `create` included | +| `layered_launch` | `the_host_program_refuses_shares_before_it_touches_the_store` | `launch` refuses shares that it cannot export with status 1 before it creates the store | +| `layered_launch` | `the_record_of_a_launch_gives_back_its_plan` | A launch's record gives back its plan, from which the host finds the proxy's URL, none for a VM without a layered root, and a reader refuses other records | +| `vm_restart` | `a_host_program_of_another_runtime_finds_the_vm_mismatched_and_signals_nothing` | A host program that ships another build of the guest agent finds the VM mismatched, and signals nothing, until the host program of its runtime stops it | +| `vm_restart` | `a_launch_tears_down_a_vm_whose_guest_agent_is_of_another_build` | A launch whose VM's guest agent reports another build sends it neither the table nor `Start`, and fails, which ends the VM | +| `vm_restart` | `a_restarted_host_program_reattaches_to_the_vm_and_stops_it` | Another process's `reattach` holds the VM, which is not its child, in a later epoch, and a third's `stop` stops it gracefully, with no runtime record left | +| `vm_steps` | `a_launch_that_the_hook_ends_after_its_checks_leaves_no_launch` | A launch that the step hook ends at `checked`, with status 5, leaves nothing of a launch in the store, and releases the scratch image's claim | +| `vm_steps` | `images_replaced_after_the_checks_never_reach_the_vm` | Images that files of zeros replace after the checks never reach the VM, whose guest assembles the checked originals | +| `vm_steps` | `the_hook_pauses_a_launch_at_each_of_its_steps` | The step hook pauses a launch at each of its steps, at which the store holds what the step says | +| `vm_layered` | `a_changed_image_is_refused_before_spawn` | A GPT disk written in place after it was staged is refused before the VM spawns, and boots once staged again | +| `vm_layered` | `a_denied_network_lets_no_probe_through` | A network that denies everything attaches no network device, and an execution reaches nothing | +| `vm_layered` | `a_failed_mapping_tears_the_vm_down` | A mapping that the guest refuses ends the table, without `Start`, and the VM ends cleanly once the guest tore the root down | +| `vm_layered` | `a_forward_reaches_its_guest_port` | A forward of a port of host loopback reaches a listener of the guest at its port | +| `vm_layered` | `a_proxy_only_network_lets_only_its_proxy_through` | A proxy-only network lets an execution reach its proxy, through the URL in its environment, and nothing else | +| `vm_layered` | `a_read_only_share_inside_a_writable_one_is_refused_before_launch` | A read-only share inside a writable one is refused before the VM spawns | +| `vm_layered` | `a_replaced_scratch_is_refused_before_spawn` | A scratch image that another copy replaced after it was staged is refused before the VM spawns, and boots once staged again | +| `vm_layered` | `a_share_inside_a_reserved_tree_is_refused_before_launch` | A share inside a tree that the adapter reserves is refused before the VM spawns | +| `vm_layered` | `a_share_over_a_reserved_tree_is_refused_before_launch` | A share that would cover a reserved tree is refused before the VM spawns | +| `vm_layered` | `accountless_workload_identity_fails_its_stage` | A workload identity whose account the image lacks fails its stage once the root is assembled, and the VM ends with 16 | +| `vm_layered` | `allowed_host_loopback_publishes_the_listeners` | Host-loopback access both ways publishes the guest's listeners, which the host reaches, while egress still governs the guest's own connections | +| `vm_layered` | `an_allow_rule_lets_only_its_destination_through` | An allow rule lets an execution reach its destination alone | +| `vm_layered` | `an_unauthorized_host_path_is_refused_before_launch` | A share of a host directory that the adapter does not authorize is refused before the VM spawns | +| `vm_layered` | `busy_root_fails_the_teardown` | A root that a probe keeps open fails its teardown, which the stop reports, and the VM ends with 15 | +| `vm_layered` | `corrupt_layer_fails_the_assembly` | A corrupt layer that asks the kernel to panic on an error fails the assembly without a panic, and the VM ends with 14 | +| `vm_layered` | `cyclic_journal_is_not_loaded` | A layer's cyclic journal, without `needs_recovery`, is never scanned: the root assembles, serves, and tears down | +| `vm_layered` | `erofs_over_ram` | The EROFS image over a tmpfs in RAM, without a scratch, and with no disk but the image's | +| `vm_layered` | `erofs_over_ram_maps_the_table` | The mapping table over a tmpfs, as over a scratch image | +| `vm_layered` | `erofs_over_ram_with_untrusted_lowers` | The image over a tmpfs in the untrusted-lowers mode, which copies up a file's data and refuses the rename of a lower directory | +| `vm_layered` | `erofs_over_scratch` | The EROFS image over a scratch image: the root that the worker assembled outside PID 1, its overlay in the default mode, its cgroups, an empty table, and a clean teardown | +| `vm_layered` | `erofs_over_scratch_maps_the_table` | The export, mounted once below a private `hostfs`, and the table's mappings, in the plan's order, each with its access, `nosuid`, and `nodev` | +| `vm_layered` | `erofs_over_scratch_with_untrusted_lowers` | The image over a scratch image in the untrusted-lowers mode, which copies up a file's data and refuses the rename of a lower directory | +| `vm_layered` | `executions_run_as_the_workload_identity` | Executions that name no identity run as the workload identity, without capabilities, with exactly their start's environment, and one that names an identity is refused with `EINVAL` | +| `vm_layered` | `executions_run_in_the_root` | Executions run at once in the root, as the identities that they name, byte-exact, and end on their own, reusable | +| `vm_layered` | `executions_take_the_settings_of_a_proxy_only_network` | The executions of a proxy-only network find its proxy's URL in their environment, after their own entries, unless they name their own | +| `vm_layered` | `first_partition_over_its_bound_fails_the_assembly` | A GPT disk whose first partition is larger than its bound fails the assembly, and the VM ends with 14 | +| `vm_layered` | `forbidden_journal_fails_the_assembly` | A scratch image whose journal the launch forbids fails the assembly before the guest mounts it, and the VM ends with 14 | +| `vm_layered` | `full_ram_upper_refuses_writes` | A tmpfs upper refuses writes with `ENOSPC` once full, and takes them again once freed | +| `vm_layered` | `full_scratch_refuses_writes` | A scratch upper refuses writes with `ENOSPC` once full, and takes them again once freed | +| `vm_layered` | `gpt_over_ram` | A GPT disk over a tmpfs, with a bound that its first partition meets exactly | +| `vm_layered` | `gpt_over_ram_maps_the_table` | The mapping table over a GPT disk over a tmpfs | +| `vm_layered` | `gpt_over_ram_with_untrusted_lowers` | A GPT disk over a tmpfs in the untrusted-lowers mode | +| `vm_layered` | `gpt_over_scratch` | A GPT disk over a scratch image: its first partition's bytes, and its layers, each mounted read-only once, stacked in order | +| `vm_layered` | `gpt_over_scratch_maps_the_table` | The mapping table over a GPT disk, whose targets the worker creates, with their parents | +| `vm_layered` | `gpt_over_scratch_with_untrusted_lowers` | A GPT disk over a scratch image in the untrusted-lowers mode | +| `vm_layered` | `invalid_configuration_fails_its_stage` | A relative runtime directory fails the configuration's stage, and the VM ends with 11 | +| `vm_layered` | `other_disk_guid_fails_the_assembly` | A GPT disk of another GUID than the launch expects fails the assembly, and the VM ends with 14 | +| `vm_layered` | `other_identity_fails_the_assembly` | An image of another identity than the launch expects fails the assembly, and the VM ends with 14 | +| `vm_layered` | `outbound_host_loopback_publishes_no_listener` | Egress with outbound host-loopback access reaches every destination, and publishes no listener of the guest | +| `vm_layered` | `published_listeners_refuse_binding_and_raw_sockets` | While listeners are published, the guest refuses `CAP_NET_BIND_SERVICE` and `CAP_NET_RAW` with `EPERM`, and keeps the other capabilities of its allowlist | +| `vm_layered` | `recovery_required_layer_fails_the_assembly` | A layer whose journal needs recovery is refused before any mount, whatever `noload` says, and the VM ends with 14 | +| `vm_layered` | `root_profiles_enter_the_root` | Executions enter the root as PID 1 of their namespaces, with their standard streams alone, no inheritable or ambient capability, and no new privileges | +| `vm_layered` | `unstaged_images_are_refused_before_spawn` | Images that the store does not stage are refused before the VM spawns, with nothing of a launch left, and boot once staged | +| `vm_template` | `a_class_that_no_capture_takes_is_refused_before_the_store` | `create` of a class that no capture takes, of 3 processors, fails with status 1 before the store exists | +| `vm_template` | `a_network_that_lets_nothing_through_has_no_nic` | `create` of a network that lets nothing through captures a template without a NIC | +| `vm_template` | `a_proxy_only_network_has_its_nic` | `create` of a proxy-only network captures a template with a NIC | +| `vm_template` | `a_refresh_captures_and_retires_the_generations_that_it_supersedes` | `refresh` captures a new generation although the store holds an eligible one, and its eviction retires the one that it supersedes, or tombstones it while a clone leases it, as `list` and `inspect` report | +| `vm_template` | `a_refresh_refuses_a_class_that_no_capture_takes_before_the_store` | `refresh` of a class that no capture takes fails with status 1 before the store exists | +| `vm_template` | `a_removal_reports_busy_while_leases_keep_the_generation` | `remove` tombstones a leased generation, keeps its payload, and reports busy, with status 7 and its leases, until no lease remains, then deletes it | +| `vm_template` | `a_template_without_a_network_has_no_nic` | `create` captures a template without a NIC, whose key holds what it asked for, and serves it again without a capture | +| `vm_template` | `an_allow_rule_has_its_nic` | `create` of an allow rule captures a template with a NIC | +| `vm_template` | `an_egress_network_has_its_nic` | `create` of a network with egress allowed captures a template with a NIC | +| `vm_template` | `host_loopback_both_ways_is_refused_before_any_vm_starts` | `create` refuses host-loopback access both ways with status 1, before any VM starts, with no generation | +| `vm_template` | `ingress_is_refused_before_any_vm_starts` | `create` refuses ingress with status 1, before any VM starts, with no generation | +| `vm_template` | `outbound_host_loopback_has_its_nic` | `create` with outbound host-loopback access captures a template with a NIC | +| `vm_clone` | `a_clone_binds_the_resolver_of_its_policy_s_dns_server_read_only` | A clone of a network with egress allowed binds a resolver file that names the gateway's DNS server read-only at `/etc/resolv.conf`, and a TCP connection to the server's DNS port opens, which the gateway's DNS proxy accepts; one whose resolver cannot be bound ends with status 57 | +| `vm_clone` | `a_clone_maps_its_export_beneath_its_root_before_its_sandbox_starts` | A clone maps its export's mappings beneath its root before its sandbox starts, each with its access, a denied file hidden, and writes reach the host's directory through the writable ones alone | +| `vm_clone` | `a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver` | A clone of a template with a NIC maps its export below the binds of its machine ID and of its resolver configuration | +| `vm_clone` | `a_clone_reads_no_more_of_its_disks_than_its_root_needs` | A clone's restore reads less than an eighth of each disk, before the guest reads each to its end | +| `vm_clone` | `a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used` | A scratch image whose upper an earlier clone used ends the clone before it is ready, with status 54 | +| `vm_clone` | `a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready` | A clone runs its executions as its record's workload identity once it is ready, and refuses one that names an identity with `EINVAL` | +| `vm_clone` | `a_clone_s_executions_find_the_proxy_of_its_policy` | The executions of a clone of a proxy-only network find its proxy's URL in their environment, and the clone keeps the image's own `/etc/resolv.conf` | +| `vm_clone` | `a_clone_s_time_settles_after_it_is_ready` | A clone's time settles once it is ready, without a violation, after the downtime that the host's monotonic clock measured | +| `vm_clone` | `a_clone_whose_activation_fails_ends_before_it_is_ready` | A clone whose activation fails ends before it is ready, with the status of the failure's class and the failure's record, quarantines nothing, and releases its lease | +| `vm_clone` | `a_clone_whose_mapping_fails_ends_before_its_sandbox_starts` | A clone whose guest fails a mapping ends before its sandbox starts, with the failure's record, and leaves the host's directory unchanged | +| `vm_clone` | `a_clone_whose_view_cannot_be_set_ends_before_it_is_ready` | A generated machine ID that no bind can hold ends the clone before it is ready, with status 55, while the image's own needs no bind | +| `vm_clone` | `a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released` | A clone whose VM exits once a stop found it running fails, with status 8 and the record of a failure by the evidence of its lost status, quarantines nothing, and releases its lease | +| `vm_clone` | `a_generation_is_eligible_by_the_clock_of_its_capture` | The store checks eligibility by the host's monotonic clock on the boot of the capture, and by UTC after a reboot, and refuses an expired or an early generation with status 10, without a lease | +| `vm_clone` | `a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity` | A host program of another runtime ends the clone through its verified identity, without a word to its guest, with status 8, and releases its lease | +| `vm_clone` | `a_run_in_a_template_of_a_network_finds_its_policy_s_proxy` | A run's execution, in a clone of a proxy-only template, finds the proxy's URL in its environment, and a run of another network runs nothing, with status 8 | +| `vm_clone` | `a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down` | A run stages its images, starts a clone, runs one execution in its root, with the execution's environment and working directory, then stops the clone and releases its lease, whatever the execution did | +| `vm_clone` | `a_run_s_execution_finds_its_shares_beneath_the_root` | A run's execution reads and writes through its share, which the clone mapped beneath its root, and the host's directory keeps what it wrote | +| `vm_clone` | `a_run_whose_clone_does_not_start_runs_nothing` | A run whose selection the store refuses ends with status 10, and one whose clone fails with status 8, without an execution or a stop | +| `vm_clone` | `a_run_whose_clone_s_vm_exits_during_its_execution_ends_with_the_clone_s_failure` | A run whose clone's VM exits during its execution ends with the clone's failure, status 8, once its stop found that the VM exited, quarantines nothing, and releases its lease | +| `vm_clone` | `a_selection_that_names_no_template_leases_nothing` | A start of a template store that does not exist, or of a generation that `remove` retired, ends with status 10 and the refusal's record, without a lease or a VM | +| `vm_clone` | `a_start_refuses_an_export_that_a_dormant_filesystem_slot_cannot_bind_before_any_vmm_starts` | An export of two directories, or of a file alone, is refused with status 8 before any VMM starts | +| `vm_clone` | `a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts` | Images, scratch images, and memory targets that the template cannot serve are refused with status 8 before any VMM starts, unstaged images with status 1, and a generation that the store does not hold with status 10 | +| `vm_clone` | `a_template_serves_the_clones_of_its_own_network_policy_alone` | A template with a NIC serves the clones of its own network alone, whose restores attach its NIC under its policy, and NVX's plan refuses the others with status 8 before any VMM starts | +| `vm_clone` | `a_template_without_a_nic_serves_clones_without_a_network` | A template without a NIC serves clones without a network, or with one that lets nothing through, and refuses those of a network with a NIC with status 8 | +| `vm_clone` | `a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease` | A tombstone refuses new clones of its generation with status 10, while the running clone serves on, and releases its lease once it stops | +| `vm_clone` | `clones_at_once_keep_their_own_uppers_and_machine_ids` | Clones at once keep their own uppers, and their own machine IDs, generated, distinct, and bound read-only, or the image's own | +| `vm_clone` | `clones_at_once_see_only_their_own_exports` | Clones of one generation at once see only their own exports, and a clone without one none | +| `vm_clone` | `each_command_reattaches_to_the_clone_from_a_new_process` | `reattach`, `probe`, and `exec` each reattach to the clone from a new process, in a later epoch, without a second activation | +| `vm_clone` | `one_generation_serves_clones_of_each_upper_and_memory_target` | One generation serves clones over a scratch image and in RAM, at each memory target up to its capacity, each with its own lease, and leaves its snapshot unchanged | +| `vm_clone` | `one_generation_serves_clones_of_either_format_over_either_upper` | A template of a GPT placeholder serves EROFS and GPT clones of other layouts over either upper, and a first partition over its bound ends a clone with status 54 | +| `vm_clone` | `the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store` | The lease of a start that ended before it recorded its clone is released only through its own template store, which `--template` names | + ### Running the VM tests The VM tests, `tests/vm_restart.rs`, `tests/vm_steps.rs`, @@ -2875,7 +3052,14 @@ NVX_HYPERVISOR=kvm NVX_ARTIFACTS_DIR="$(cd ../.. && pwd)" \ cargo test --locked --features host -- --ignored ``` -On Windows, take the guest agent from a Linux build. +On Windows, take the guest agent from a Linux build. To run the +[stage scenarios](#stage-scenarios) by name, as CI runs them, run in the +repository's root, after the build of the guest agent above, or with +`--guest-agent PATH` naming another: + +```bash +python3 scripts/nvx.py test-runtime-adapter --backend kvm +``` Run the checks of each configuration in this directory. The guest-only configuration needs Linux and the musl target, which @@ -2906,11 +3090,12 @@ configuration. A change to the adapter is a test input of CI's NVX microVM tests. Their [workflow](../../.github/workflows/run-nvx-microvm-tests.yml) builds the host-only configuration on each backend, in the jobs that run the guest and -sandbox steps, and runs its ignored tests, the adapter's VM scenarios, template -captures, and clones, with `NVX_HYPERVISOR` naming the backend, -`NVX_ARTIFACTS_DIR` the checkout, into which the job downloaded NVX's +sandbox steps, and runs the adapter's consolidated suite, the +[stage scenarios](#stage-scenarios), by name, with +`nvx.py test-runtime-adapter`, which names the backend in `NVX_HYPERVISOR`, the +checkout in `NVX_ARTIFACTS_DIR`, into which the job downloaded NVX's artifacts, their provenance, and the fixtures of the layered scenarios with -the guest artifacts, as NVX's own layered scenarios take them, and -`RUNTIME_ADAPTER_GUEST` the guest agent. The Linux jobs build the guest agent +the guest artifacts, as NVX's own layered scenarios take them, and the guest +agent in `RUNTIME_ADAPTER_GUEST`. The Linux jobs build the guest agent first, and the Windows job takes the one that CI's `distro-artifacts` job builds on Linux. diff --git a/tests/runtime-adapter/scenarios.txt b/tests/runtime-adapter/scenarios.txt new file mode 100644 index 000000000..28ec8f38e --- /dev/null +++ b/tests/runtime-adapter/scenarios.txt @@ -0,0 +1,173 @@ +# The runtime adapter's consolidated suite: every scenario of the template and clone stage, which +# CI's runtime adapter step runs by name on each backend, with +# `python3 scripts/nvx.py test-runtime-adapter`. Each line names a test target of `tests/`, then +# one of its tests, which the step runs whether it is ignored or not. The step fails once a line +# names a test that its target does not have, a target that a line names has a test that no line +# names, the library or a target that no line names has an ignored test, or a named test does not +# run and pass. The README's "Stage scenarios" says what each scenario proves. + +template_capture a_capture_takes_its_class_and_inputs_from_the_request +template_capture the_host_program_refuses_a_template_before_it_creates_anything + +template_operations a_refresh_refuses_a_template_before_it_creates_anything +template_operations an_empty_template_store_lists_no_generation +template_operations the_host_program_refuses_command_lines_of_the_operations +template_operations the_operations_open_a_template_store_and_create_none + +clone_start a_clone_s_plan_request_and_record_carry_its_network +clone_start a_clone_s_record_names_its_template_store_and_generation_beside_its_plan +clone_start a_clone_s_request_takes_the_staged_bindings_and_the_guest_s_defaults +clone_start a_host_s_time_keeps_its_text_form +clone_start a_run_refuses_a_network_or_shares_that_the_plan_refuses_before_it_stages_anything +clone_start a_run_refuses_a_selection_that_names_no_template_once_its_preflight_staged_the_images +clone_start a_start_refuses_a_generation_that_the_template_store_does_not_hold_before_anything_else +clone_start a_start_refuses_a_network_that_the_plan_refuses_before_anything_else +clone_start a_start_refuses_shares_that_it_cannot_export_before_anything_else +clone_start the_host_program_refuses_options_that_do_not_make_a_run +clone_start the_host_program_refuses_options_that_do_not_make_a_start + +host_session a_clone_s_order_ends_its_table_at_a_mapping_that_the_guest_fails_and_stops_the_guest +host_session a_clone_s_order_maps_the_root_of_its_export_with_an_empty_source +host_session a_clone_s_order_sends_its_record_first_then_starts_its_sandbox_as_after_a_cold_boot +host_session a_clone_s_session_that_ends_after_its_bootstrap_is_lost_and_the_next_takes_the_same_record +host_session a_mapping_that_the_guest_fails_ends_the_table_and_the_host_stops_the_guest +host_session a_receive_that_times_out_keeps_the_partial_message +host_session a_reset_or_a_disconnect_in_the_middle_of_a_message_drops_it +host_session a_send_that_fails_or_times_out_ends_the_client_with_what_it_holds +host_session a_stop_skips_the_guest_s_late_messages_about_the_executions_that_the_driver_gave_up_on +host_session answers_whose_bodies_do_not_decode_end_the_client +host_session only_a_session_that_ended_is_lost_to_a_clone_s_order +host_session requests_check_their_answers_and_end_the_client_when_they_fail +host_session the_client_attaches_and_exchanges_messages_that_span_records +host_session the_client_configures_a_layered_root_probes_it_and_learns_how_its_teardown_ended +host_session the_client_fails_on_records_that_the_protocol_does_not_allow +host_session the_client_reassembles_messages_from_partial_and_shared_records +host_session the_client_refuses_messages_that_its_framing_does_not_allow +host_session the_client_returns_the_stage_of_the_root_that_failed +host_session the_client_waits_for_the_guest_and_reports_refusals_while_attaching +host_session the_driver_ends_its_run_on_a_message_that_breaks_the_rules +host_session the_driver_fires_the_timers_of_its_executions_while_the_guest_says_nothing +host_session the_driver_refuses_starts_before_it_sends_them +host_session the_driver_runs_executions_at_once_and_collects_their_output_and_outcomes +host_session the_driver_s_sends_wait_for_a_guest_that_stops_reading_until_the_deadline_of_its_run +host_session the_host_orders_a_layered_root_ready_then_its_table_in_parts_then_start +host_session the_host_sends_nothing_more_to_a_guest_agent_of_another_build_or_whose_root_failed + +staged_images a_launch_takes_the_bindings_that_the_store_stages_for_its_images +staged_images staging_verifies_the_image_and_binds_the_scratch +staged_images the_host_program_stages_images_in_the_store +staged_images the_store_records_the_staged_images + +layered_launch a_launch_claims_its_scratch_image_until_it_releases_its_checked_files +layered_launch a_launch_reads_only_the_networks_that_the_plan_takes +layered_launch a_launch_records_what_it_launched +layered_launch a_launch_refuses_images_that_are_not_staged_or_changed_since +layered_launch a_launch_refuses_shares_that_it_cannot_export +layered_launch a_launch_renders_each_element_of_its_network_through_the_plan +layered_launch a_launch_shares_host_paths_inside_the_authorized_directories +layered_launch a_layered_launch_hands_openvmm_the_checked_images +layered_launch a_layered_launch_passes_what_the_guest_checks_as_it_is +layered_launch a_layered_launch_plans_its_blocks_and_boot_tokens_with_the_adapters_plan +layered_launch an_execution_finds_the_url_of_its_network_s_proxy_in_its_environment +layered_launch shares_read_their_host_path_their_target_and_their_access +layered_launch the_host_program_refuses_a_network_before_it_launches +layered_launch the_host_program_refuses_options_that_do_not_make_a_command +layered_launch the_host_program_refuses_shares_before_it_touches_the_store +layered_launch the_record_of_a_launch_gives_back_its_plan + +vm_restart a_host_program_of_another_runtime_finds_the_vm_mismatched_and_signals_nothing +vm_restart a_launch_tears_down_a_vm_whose_guest_agent_is_of_another_build +vm_restart a_restarted_host_program_reattaches_to_the_vm_and_stops_it + +vm_steps a_launch_that_the_hook_ends_after_its_checks_leaves_no_launch +vm_steps images_replaced_after_the_checks_never_reach_the_vm +vm_steps the_hook_pauses_a_launch_at_each_of_its_steps + +vm_layered a_changed_image_is_refused_before_spawn +vm_layered a_denied_network_lets_no_probe_through +vm_layered a_failed_mapping_tears_the_vm_down +vm_layered a_forward_reaches_its_guest_port +vm_layered a_proxy_only_network_lets_only_its_proxy_through +vm_layered a_read_only_share_inside_a_writable_one_is_refused_before_launch +vm_layered a_replaced_scratch_is_refused_before_spawn +vm_layered a_share_inside_a_reserved_tree_is_refused_before_launch +vm_layered a_share_over_a_reserved_tree_is_refused_before_launch +vm_layered accountless_workload_identity_fails_its_stage +vm_layered allowed_host_loopback_publishes_the_listeners +vm_layered an_allow_rule_lets_only_its_destination_through +vm_layered an_unauthorized_host_path_is_refused_before_launch +vm_layered busy_root_fails_the_teardown +vm_layered corrupt_layer_fails_the_assembly +vm_layered cyclic_journal_is_not_loaded +vm_layered erofs_over_ram +vm_layered erofs_over_ram_maps_the_table +vm_layered erofs_over_ram_with_untrusted_lowers +vm_layered erofs_over_scratch +vm_layered erofs_over_scratch_maps_the_table +vm_layered erofs_over_scratch_with_untrusted_lowers +vm_layered executions_run_as_the_workload_identity +vm_layered executions_run_in_the_root +vm_layered executions_take_the_settings_of_a_proxy_only_network +vm_layered first_partition_over_its_bound_fails_the_assembly +vm_layered forbidden_journal_fails_the_assembly +vm_layered full_ram_upper_refuses_writes +vm_layered full_scratch_refuses_writes +vm_layered gpt_over_ram +vm_layered gpt_over_ram_maps_the_table +vm_layered gpt_over_ram_with_untrusted_lowers +vm_layered gpt_over_scratch +vm_layered gpt_over_scratch_maps_the_table +vm_layered gpt_over_scratch_with_untrusted_lowers +vm_layered invalid_configuration_fails_its_stage +vm_layered other_disk_guid_fails_the_assembly +vm_layered other_identity_fails_the_assembly +vm_layered outbound_host_loopback_publishes_no_listener +vm_layered published_listeners_refuse_binding_and_raw_sockets +vm_layered recovery_required_layer_fails_the_assembly +vm_layered root_profiles_enter_the_root +vm_layered unstaged_images_are_refused_before_spawn + +vm_template a_class_that_no_capture_takes_is_refused_before_the_store +vm_template a_network_that_lets_nothing_through_has_no_nic +vm_template a_proxy_only_network_has_its_nic +vm_template a_refresh_captures_and_retires_the_generations_that_it_supersedes +vm_template a_refresh_refuses_a_class_that_no_capture_takes_before_the_store +vm_template a_removal_reports_busy_while_leases_keep_the_generation +vm_template a_template_without_a_network_has_no_nic +vm_template an_allow_rule_has_its_nic +vm_template an_egress_network_has_its_nic +vm_template host_loopback_both_ways_is_refused_before_any_vm_starts +vm_template ingress_is_refused_before_any_vm_starts +vm_template outbound_host_loopback_has_its_nic + +vm_clone a_clone_binds_the_resolver_of_its_policy_s_dns_server_read_only +vm_clone a_clone_maps_its_export_beneath_its_root_before_its_sandbox_starts +vm_clone a_clone_of_a_template_with_a_nic_maps_its_export_beside_its_resolver +vm_clone a_clone_reads_no_more_of_its_disks_than_its_root_needs +vm_clone a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used +vm_clone a_clone_runs_its_executions_as_its_workload_identity_once_it_is_ready +vm_clone a_clone_s_executions_find_the_proxy_of_its_policy +vm_clone a_clone_s_time_settles_after_it_is_ready +vm_clone a_clone_whose_activation_fails_ends_before_it_is_ready +vm_clone a_clone_whose_mapping_fails_ends_before_its_sandbox_starts +vm_clone a_clone_whose_view_cannot_be_set_ends_before_it_is_ready +vm_clone a_clone_whose_vm_exits_once_a_stop_found_it_fails_before_its_lease_is_released +vm_clone a_generation_is_eligible_by_the_clock_of_its_capture +vm_clone a_host_program_of_another_runtime_ends_the_clone_through_its_verified_identity +vm_clone a_run_in_a_template_of_a_network_finds_its_policy_s_proxy +vm_clone a_run_runs_one_execution_in_its_clone_then_tears_the_clone_down +vm_clone a_run_s_execution_finds_its_shares_beneath_the_root +vm_clone a_run_whose_clone_does_not_start_runs_nothing +vm_clone a_run_whose_clone_s_vm_exits_during_its_execution_ends_with_the_clone_s_failure +vm_clone a_selection_that_names_no_template_leases_nothing +vm_clone a_start_refuses_an_export_that_a_dormant_filesystem_slot_cannot_bind_before_any_vmm_starts +vm_clone a_start_refuses_what_the_template_cannot_serve_before_any_vmm_starts +vm_clone a_template_serves_the_clones_of_its_own_network_policy_alone +vm_clone a_template_without_a_nic_serves_clones_without_a_network +vm_clone a_tombstone_leaves_the_running_clone_of_its_generation_to_its_lease +vm_clone clones_at_once_keep_their_own_uppers_and_machine_ids +vm_clone clones_at_once_see_only_their_own_exports +vm_clone each_command_reattaches_to_the_clone_from_a_new_process +vm_clone one_generation_serves_clones_of_each_upper_and_memory_target +vm_clone one_generation_serves_clones_of_either_format_over_either_upper +vm_clone the_lease_of_a_start_that_ended_before_its_record_is_released_through_its_template_store