From d4f171d5b29a84174b673764e329410cd7659928 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 19:17:40 -0700 Subject: [PATCH 1/6] openvmm: let an untiered restore install a new network policy Pin microsoft/openvmm#4831 at 757a15fa36c973d2cdcd4da8d6e3c8d2e5036abb, two commits on 858b2f93a, the previous pin, which is the head of feature/microvm/dev. A snapshot whose NIC was captured with the complete deny-all policy is a NIC template, and --restore-network-policy lets a restore install a new policy on it, but OpenVMM admitted the option only for a platform snapshot and refused it for a snapshot without a tier (#942). A platform clone also learns its DNS server, and whether to announce its listeners, from its activation after the restore, which the guest of a snapshot without a tier never receives, so under a new policy that guest would keep the deny-all's configuration: no DNS server and no nvx-ingress (#943). The first commit admits the option for a snapshot without a tier, and still refuses it for a workload-start or instance-checkpoint snapshot. Such a snapshot may hold a workload that has already run, but under the deny-all that workload had no connectivity, so nothing in the snapshot depends on the policy. The second hands the guest of such a restore the new policy's DNS server, the gateway that a cold boot under the policy names, and whether to announce its listeners, in version 6 of the restore packet: version 5's fields, with the network byte, which version 5 reserves, giving the server's family and the announcement, and 16 bytes after the entropy giving the server. Such a restore requires the guest's acknowledgement, so the guest applies the settings before host input reaches it. Every other restore, a platform clone's included, keeps version 5, so the pin changes nothing for NVX's guests until a restore asks for version 6, which a later commit teaches them to read. Neither OpenVMM's control-session protocol vectors nor its built-in CPU profiles change, so their copies in nvx-rs stay as they are. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 858b2f93a..757a15fa3 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 858b2f93aa78d35df6fdd217322a7f4cfb57b080 +Subproject commit 757a15fa36c973d2cdcd4da8d6e3c8d2e5036abb From 8120c87d8d15524e8079cc806100d8e03d793890 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 31 Oct 2026 07:33:33 -0700 Subject: [PATCH 2/6] doc: map the untiered network policy restore in the OpenVMM roadmap Record the commits that the previous commit pins as positions 379 and 380 of the microVM chain, and move the NVX pin to 757a15fa3. Position 379 lets --restore-network-policy install a new policy on the NIC template of a snapshot without a tier, and position 380 hands the guest of such a restore the new policy's network settings in version 6 of the restore packet. They change only the tier check of position 347's restore that installs a new policy and restore packet 5 of positions 373-378, so they build on those fork-only commits; say so in the table's remarks. The pin is now 223 commits past the mapped tip. Positions 373-378 are no longer only a proposal: microsoft/openvmm#4830 merged them, so position 378, 858b2f93a, is the head of feature/microvm/dev. Say so in the scope, and name microsoft/openvmm#4831 as the proposal of positions 379 and 380. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 67 ++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 31 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 3e5fd8718..5f01a1c2a 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -9,32 +9,33 @@ - Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR summary, and commit mapping below cover positions 001-157 and the fixups at positions 334-340, 344, and 345. -- NVX's pinned OpenVMM revision: `858b2f93aa78d35df6fdd217322a7f4cfb57b080`, position 378. Positions - 373-378, which microsoft/openvmm#4830 proposes for `feature/microvm/dev`, are six commits on - position 372, `3d7512af6`, the head of `feature/microvm/dev` since microsoft/openvmm#4816 merged - positions 371 and 372. Positions 371 and 372 are two commits on position 370, `b58040ee4`, which - microsoft/openvmm#4823 merged. Position 370 is one commit on position 369, `5e0ab772e`, which - microsoft/openvmm#4820 merged with position 368. Positions 368 and 369 are two commits on position - 367, `bcfc47cff`, which microsoft/openvmm#4818 merged with position 366. Positions 366 and 367 are - two commits on position 365, `e4c741e18`, which microsoft/openvmm#4804 merged with positions - 362-364. Positions 362-365 are four commits on position 361, `c32c57d12`, which - microsoft/openvmm#4807 merged with position 360. Positions 360 and 361 are two commits on position - 359, `bd2b27a2e`, which microsoft/openvmm#4794 merged with positions 355-358. Positions 355-359 - are five commits on position 354, `832d5bb48`, which microsoft/openvmm#4775 merged with position - 353. Positions 353 and 354 are two commits on position 352, `b92daa350`, which - microsoft/openvmm#4773 merged with positions 350 and 351. Positions 350-352 are three commits on - position 349, `d2338ea22`, which microsoft/openvmm#4769 merged with position 348. Positions 348 - and 349 are two commits on position 347, `57d0a18b3`, which microsoft/openvmm#4767 merged with - position 346. Positions 346 and 347 are two commits on position 345, `e68871cc6`, which - microsoft/openvmm#4762 merged with position 344. Positions 344 and 345 are two commits on position - 343, `a12d9167f`, which microsoft/openvmm#4749 merged with positions 341 and 342. Positions - 341-343 are three commits on position 340, `2b681564d`, which microsoft/openvmm#4735 merged with - position 339. Positions 339 and 340 are two commits on position 338, `8a0d945ed`, which - microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head of the - `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 221 commits past the mapped - tip on the same first-parent chain. Positions 158-333, 341-343, 346-370, and 373-378 are not yet - assigned to PRs, and positions 371 and 372 port microsoft/openvmm#4803; see - [Commits after the mapped tip](#commits-after-the-mapped-tip). +- NVX's pinned OpenVMM revision: `757a15fa36c973d2cdcd4da8d6e3c8d2e5036abb`, position 380. Positions + 379 and 380, which microsoft/openvmm#4831 proposes for `feature/microvm/dev`, are two commits on + position 378, `858b2f93a`, the head of `feature/microvm/dev` since microsoft/openvmm#4830 merged + positions 373-378. Positions 373-378 are six commits on position 372, `3d7512af6`, which + microsoft/openvmm#4816 merged with position 371. Positions 371 and 372 are two commits on + position 370, `b58040ee4`, which microsoft/openvmm#4823 merged. Position 370 is one commit on + position 369, `5e0ab772e`, which microsoft/openvmm#4820 merged with position 368. Positions 368 + and 369 are two commits on position 367, `bcfc47cff`, which microsoft/openvmm#4818 merged with + position 366. Positions 366 and 367 are two commits on position 365, `e4c741e18`, which + microsoft/openvmm#4804 merged with positions 362-364. Positions 362-365 are four commits on + position 361, `c32c57d12`, which microsoft/openvmm#4807 merged with position 360. Positions 360 + and 361 are two commits on position 359, `bd2b27a2e`, which microsoft/openvmm#4794 merged with + positions 355-358. Positions 355-359 are five commits on position 354, `832d5bb48`, which + microsoft/openvmm#4775 merged with position 353. Positions 353 and 354 are two commits on + position 352, `b92daa350`, which microsoft/openvmm#4773 merged with positions 350 and 351. + Positions 350-352 are three commits on position 349, `d2338ea22`, which microsoft/openvmm#4769 + merged with position 348. Positions 348 and 349 are two commits on position 347, `57d0a18b3`, + which microsoft/openvmm#4767 merged with position 346. Positions 346 and 347 are two commits on + position 345, `e68871cc6`, which microsoft/openvmm#4762 merged with position 344. Positions 344 + and 345 are two commits on position 343, `a12d9167f`, which microsoft/openvmm#4749 merged with + positions 341 and 342. Positions 341-343 are three commits on position 340, `2b681564d`, which + microsoft/openvmm#4735 merged with position 339. Positions 339 and 340 are two commits on + position 338, `8a0d945ed`, which microsoft/openvmm#4734 merged, and position 338 is one commit on + `79a501b63`, the head of the `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin + is 223 commits past the mapped tip on the same first-parent chain. Positions 158-333, 341-343, + 346-370, and 373-380 are not yet assigned to PRs, and positions 371 and 372 port + microsoft/openvmm#4803; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base. ## Upstream status @@ -326,8 +327,8 @@ The physical branch is one first-parent chain: -> unassigned [341-343] -> PR18 fixups [344-345] -> unassigned [346-370] -> microsoft/openvmm#4803 port [371-372] - -> unassigned [373-378] - -> 858b2f93a (NVX pin) + -> unassigned [373-380] + -> 757a15fa3 (NVX pin) ``` For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main, @@ -349,8 +350,8 @@ PR18's position 111. ## Commits after the mapped tip -The NVX pin extends the chain past position 157 without merges. Of the 221 later commits, -positions 158-333, 341-343, 346-370, and 373-378 are not yet classified or assigned to PRs, +The NVX pin extends the chain past position 157 without merges. Of the 223 later commits, +positions 158-333, 341-343, 346-370, and 373-380 are not yet classified or assigned to PRs, positions 334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02 fixup, positions 339 and 340 are PR20 and PR23 fixups, positions 344 and 345 are PR18 fixups, and positions 371 and 372 port microsoft/openvmm#4803. @@ -410,6 +411,7 @@ This table groups them by topic in chain order; ranges are inclusive, as in the | 370 | `b58040ee4` | 1 | Explicit `--host-loopback allow` requires an IPv4 network of /29 or larger: OpenVMM refuses it on a /30, which leaves no address at which Consomme presents host peers, at a cold boot and at a restore that installs a new policy on a NIC template, instead of presenting IPv4 host peers at the gateway (microsoft/openvmm#4823) | | 371-372 | `2dec9edcb..3d7512af6` | 2 | The KVP IC VMM test boots Windows Server 2025 instead of 2022, whose COM activation race can lose WMI's server-registration notification, so that the guest answers the test's IP information request with `0x80080005` after 120 seconds (microsoft/nvx#1035); and the check-in gates' x64 Linux KVM job downloads the Server 2025 image and keeps running the test, while the x64 Linux MSHV job excludes it because Server 2025's required initial reset fails there (microsoft/openvmm#4816, which ports microsoft/openvmm#4803) | | 373-378 | `a34095ab3..858b2f93a` | 6 | NVX time ABI 2: the VMM pairs each time sample's host UTC with the guest TSC, through the guest TSC's offset from the host TSC that every backend measures at every start, and publishes with it the host's clock correction that the host OS reports (`adjtimex`, `GetSystemTimeAdjustmentPrecise`), the bound of its error, the status and maximum error of host UTC, and a disruption counter, in a 48-byte version-2 sample; restore packet 5 drops host UTC, and the `host-rate-offset-ppm` test hook skews the published host clock (microsoft/openvmm#4830, which rebases microsoft/openvmm#4809) | +| 379-380 | `c40dff64a..757a15fa3` | 2 | `--restore-network-policy` also installs a new policy on the NIC template of a snapshot without a tier, refusing only `workload-start` and `instance-checkpoint` snapshots (microsoft/nvx#942); and such a restore hands its guest the new policy's DNS server and whether to announce its listeners in version 6 of the restore packet, in the network byte, which version 5 reserves, and a 16-byte DNS server after the entropy, and gates host input until the guest acknowledges that it applied them, while every other restore keeps version 5 (microsoft/nvx#943, microsoft/openvmm#4831) | Some of these commits change mapped work rather than add to it. Positions 246-250 remove the legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes @@ -449,7 +451,10 @@ positions added, so they follow them upstream; they don't apply to upstream main Position 377 also adapts petri's microVM configuration, which PR15's position 088 adds, and the ttrpc microVM test, which PR25's position 146 adds, to the portb device's configuration and restore packet 5. Position 378 names restore packet 5 in the CLI reference, apart from the -host-loopback rule that position 370 adds to it. +host-loopback rule that position 370 adds to it. Position 379 changes only the tier check of +position 347's restore that installs a new policy, its tests, and the guides, and position 380 +extends restore packet 5 of positions 373-378, and its encoders, decoders, and tests, so both +build on those fork-only commits. ## PR summary From a4f256772a475f469308451809ec8652628d6924 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 19:40:52 -0700 Subject: [PATCH 3/6] nvx-rs: let nvx-ingress start its announcer for a guest agent A clone's application starts the announcer of the guest's listeners with Announcer::start, which binds the announcer's socket to a privileged source port and runs nvx-ingress inherit on it, so a failure to start it is the start's own, before the clone is ready. A guest agent that cannot hold the announcer, such as a shell script, could only run nvx-ingress in the background, as init does at a cold boot, and could not tell whether the announcer started: whether it found a privileged source port free, or ran at all. nvx-ingress start GATEWAY_IPV4 starts the announcer as Announcer::start does, and exits once the announcer's socket is bound and its program runs, which leaves the announcer to init, or to the nearest subreaper, or exits with status 1, naming why, when the announcer does not start. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- nvx-rs/README.md | 2 +- nvx-rs/src/guest/ingress/mod.rs | 48 ++++++++++++++++++++++++++---- nvx-rs/src/guest/ingress/sender.rs | 2 +- nvx-rs/tests/guest_tools.rs | 35 +++++++++++++++++++++- 4 files changed, 78 insertions(+), 9 deletions(-) diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 7e923ede2..fa6c58119 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2601,7 +2601,7 @@ disk's block device. | `nvx-exit [CODE]` | Stops the VM with an 8-bit exit code through OpenVMM's control port 0x604 (x86_64) | | `nvx-device-io file ...`, `nvx-device-io network ...`, `nvx-device-io map PATH TEXT` | The benchmark helper that measures file and UDP round-trip I/O and prints one JSON line, and reports a file benchmark's progress on stderr; `map` writes TEXT at the start of a file through a shared mapping and holds the dirty mapping until its standard input ends, for the microVM scenarios (x86_64) | | `nvx-managed-agent CONTROL_TTY ROOTFS HOSTNAME UID GID USER HOME` | Serves the managed control session and supervises sequential workloads; `--exec-config-fd FD -- WORKLOAD ...` is the direct agent's internal launch helper | -| `nvx-ingress [inherit] GATEWAY_IPV4` | Announces reachable TCP listeners and settled UDP sockets to the portable-network gateway, from a privileged source port that it binds, or, with `inherit`, from its standard input, a socket that a clone's application bound to one; `test DIRECTORY` runs synthetic procfs snapshots | +| `nvx-ingress [inherit\|start] GATEWAY_IPV4` | Announces reachable TCP listeners and settled UDP sockets to the portable-network gateway, from a privileged source port that it binds, or, with `inherit`, from its standard input, a socket that a clone's application bound to one; `start` starts `inherit` as `Announcer::start` does and exits once it runs, or with status 1 if it does not start, for a guest agent that cannot hold the announcer; `test DIRECTORY` runs synthetic procfs snapshots | | `nvx-time [--report-only] COMMAND` | Enforces NVX time ABI v2: runs its conformance checks, repairs the wall clock after a restore, publishes the time state, and runs the discipline and violation-watcher daemon; `test NAME ...` runs its decoder and process test hooks (x86_64) | | `nvx-time-probe latency\|warp\|clock ...` | Diagnoses clock-read latency, cross-CPU TSC warp and offset, the clocks, CPUID, clock events, kernel messages, and optionally MSRs (x86_64) | diff --git a/nvx-rs/src/guest/ingress/mod.rs b/nvx-rs/src/guest/ingress/mod.rs index 31e3497d2..efb383a3b 100644 --- a/nvx-rs/src/guest/ingress/mod.rs +++ b/nvx-rs/src/guest/ingress/mod.rs @@ -17,6 +17,15 @@ //! publishing the listeners that it last announced, and learns of no later //! change of them. //! +//! A guest agent that cannot hold the announcer, such as the shell script +//! that repairs the restore of a snapshot without a tier whose new policy +//! publishes the guest's listeners, runs `nvx-ingress start GATEWAY_IPV4` +//! instead. It starts the announcer as [`Announcer::start`] does, and exits +//! once the announcer's socket is bound and its program runs, which leaves the +//! announcer to init, or to the nearest subreaper, as at a cold boot; it fails, +//! with status 1, when the announcer does not start, so that the agent learns +//! of the failure before the restore is acknowledged. +//! //! Every 100 ms it reads the kernel's socket tables. A TCP socket in the //! LISTEN state is a listener, and so is an unconnected UDP socket that //! persists for a second, which skips short-lived client sockets, such as @@ -47,7 +56,7 @@ //! which the kernel writes them: a row or number that the kernel would not //! write is rejected, rather than read with the leniency of C's `scanf`. -use std::ffi::CString; +use std::ffi::{CStr, CString}; use std::io::{self, Read, Write}; use std::os::unix::ffi::OsStrExt; use std::path::{Path, PathBuf}; @@ -190,19 +199,23 @@ struct UdpSocket { /// Runs `nvx-ingress` with the process's arguments: `GATEWAY_IPV4`, the announcer, from a socket /// of its own; `inherit GATEWAY_IPV4`, the announcer, from the socket that it inherited as its /// standard input, which must be an IPv4 datagram socket bound to a privileged source port from -/// 512 to 1023; or `test DIRECTORY`. Test mode exits with 0 after the last snapshot, or with 1 -/// when a snapshot fails. The announcer runs until it fails, and exits with 1 when it cannot open -/// or bind its socket, or take the one that it inherited, or with 2 for a usage error. +/// 512 to 1023; `start GATEWAY_IPV4`, which starts `inherit GATEWAY_IPV4` as +/// [`Announcer::start`] does and exits once it runs; or `test DIRECTORY`. Test mode exits with 0 +/// after the last snapshot, or with 1 when a snapshot fails. The announcer runs until it fails, +/// and exits with 1 when it cannot open or bind its socket, or take the one that it inherited, or +/// with 2 for a usage error. `start` exits with 0 once the announcer runs, or with 1 when it does +/// not start. pub fn main() -> ExitCode { run_main(|args| run(args, &mut sys::Stdout::new(), &mut io::stderr())) } -/// Runs test mode or the announcer, as `args` select. +/// Runs test mode, the announcer, or its start, as `args` select. fn run(args: &[CString], out: &mut dyn Write, err: &mut dyn Write) -> Result<(), CommandError> { let announced = match args { [_, mode, directory] if mode.as_bytes() == b"test" => { return test_mode::run(directory, out).map_err(|_| CommandError::Operation); } + [_, mode, gateway] if mode.as_bytes() == b"start" => return start(gateway, err), [_, mode, gateway] if mode.as_bytes() == b"inherit" => sender::run_inherited(gateway), [_, gateway] => sender::run(gateway), _ => return Err(usage(err)), @@ -217,9 +230,32 @@ fn run(args: &[CString], out: &mut dyn Write, err: &mut dyn Write) -> Result<(), } } +/// Starts the announcer of the guest's listeners to `gateway` as [`Announcer::start`] does, and +/// returns once its socket is bound and its program runs, leaving it to run on its own: the +/// process exits then, which passes the announcer to init, or to the nearest subreaper. +fn start(gateway: &CStr, err: &mut dyn Write) -> Result<(), CommandError> { + let Some(gateway) = sender::parse_gateway(gateway) else { + return Err(usage(err)); + }; + match Announcer::start(gateway) { + Ok(announcer) => { + // Dropping the announcer would stop it, which only its owner may do. + std::mem::forget(announcer); + Ok(()) + } + Err(error) => { + sys::report( + err, + &[b"nvx-ingress: ", error.to_string().as_bytes(), b"\n"], + ); + Err(CommandError::Operation) + } + } +} + /// Prints the usage message and returns the usage error. fn usage(err: &mut dyn Write) -> CommandError { - sys::report(err, &[b"usage: nvx-ingress [inherit] GATEWAY_IPV4\n"]); + sys::report(err, &[b"usage: nvx-ingress [inherit|start] GATEWAY_IPV4\n"]); CommandError::Usage } diff --git a/nvx-rs/src/guest/ingress/sender.rs b/nvx-rs/src/guest/ingress/sender.rs index c183a8686..a55dbe89a 100644 --- a/nvx-rs/src/guest/ingress/sender.rs +++ b/nvx-rs/src/guest/ingress/sender.rs @@ -193,7 +193,7 @@ pub(super) fn run_inherited(gateway: &CStr) -> Result<(), Error> { /// Parses `text` as the C program's `inet_pton(AF_INET, ...)` did: as four decimal octets of at /// most three digits each, without leading zeros, which is the form that the standard library /// accepts. -fn parse_gateway(text: &CStr) -> Option { +pub(super) fn parse_gateway(text: &CStr) -> Option { text.to_str().ok()?.parse().ok() } diff --git a/nvx-rs/tests/guest_tools.rs b/nvx-rs/tests/guest_tools.rs index 01d3b6adf..ee99e50cc 100644 --- a/nvx-rs/tests/guest_tools.rs +++ b/nvx-rs/tests/guest_tools.rs @@ -1047,13 +1047,16 @@ fn ingress_test_mode_preserves_udp_settling_between_snapshots() { #[test] fn ingress_rejects_bad_invocations_and_test_snapshots() { - let usage = b"usage: nvx-ingress [inherit] GATEWAY_IPV4\n".to_vec(); + let usage = b"usage: nvx-ingress [inherit|start] GATEWAY_IPV4\n".to_vec(); for arguments in [ &[][..], &["not-an-ipv4-address"], &["inherit"], &["inherit", "not-an-ipv4-address"], &["inherit", "127.0.0.1", "extra"], + &["start"], + &["start", "not-an-ipv4-address"], + &["start", "127.0.0.1", "extra"], ] { assert_eq!( run("nvx-ingress", arguments), @@ -1148,6 +1151,36 @@ fn ingress_inherits_only_an_ipv4_datagram_socket_bound_to_a_privileged_source_po ); } +#[test] +fn ingress_start_fails_when_the_announcer_does_not_start() { + // `start` exits only once the announcer's socket is bound to a privileged source port and + // /sbin/nvx-ingress runs on it, so that a guest agent learns of a failure before it + // acknowledges a restore. Off an NVX guest, the announcer cannot start: without root, the + // socket cannot be bound to such a port, and with it, the program does not exist. + if Path::new("/sbin/nvx-ingress").exists() { + return; + } + let (status, stdout, stderr) = run_within( + "nvx-ingress", + ["start", "127.0.0.1"], + Duration::from_secs(10), + ); + assert_eq!((status, stdout), (Some(1), Vec::new())); + let stderr = String::from_utf8(stderr).unwrap(); + assert!( + [ + "nvx-ingress: the announcer's socket cannot be bound to a privileged source port: ", + "nvx-ingress: no privileged source port, from 1023 down to 512, is free", + "nvx-ingress: /sbin/nvx-ingress did not start: the exec launch stage failed: ", + ] + .iter() + .any(|prefix| stderr.starts_with(prefix)) + && stderr.ends_with('\n') + && stderr.lines().count() == 1, + "{stderr}" + ); +} + #[cfg(target_arch = "x86_64")] /// Returns the first CPU that this process may run on. fn first_allowed_cpu() -> usize { From 8edd29b66d27dade622b54dcedfed9012da58636 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 31 Oct 2026 07:37:51 -0700 Subject: [PATCH 4/6] guest: apply the network settings of an untiered restore When a restore installs a new network policy on the NIC template of a snapshot without a tier, OpenVMM hands the guest the policy's DNS server, and whether to announce its listeners, in version 6 of the restore packet, and gates host input until the guest acknowledges the restore. NVX's guest read version 5 alone, so it failed such a restore, and nothing would have applied the settings: the template's guest booted under the deny-all policy, whose command line names no DNS server and starts no nvx-ingress, and only a platform clone takes them from its activation. nvx-time now reads version 6 as well: version 5's packet, whose reserved byte is the network byte, and whose DNS server follows the entropy. It refuses a network byte with DNS family 3 or another bit set, and a DNS server that does not match its family or is unspecified, and the capture's metadata for nvx-snapshot gains the DNS server, or "-", and the announcement. As for a packet with processors or memory to activate, nvx-time leaves a packet whose settings ask for something to nvx-snapshot, which acknowledges the restore once it applied them. TimeAbi's Restore exposes the settings as a RestoreNetwork, and nvx-time's packet test hook prints them. nvx-snapshot applies them in guest repair step 4, after processors and memory, as init applies those of a cold boot under such a policy: it names the DNS server in /etc/resolv.conf, and starts the announcer of the guest's listeners with nvx-ingress start and the gateway of virtnet_gw=, which returns once the announcer's socket is bound to a privileged source port and its program runs. A failure of either fails the restore, with status 1, before its acknowledgement. A restore whose settings ask for nothing stays quiet, and an untiered restore still leaves entropy and identity to its caller. The restore-protocol, network, and byte-channel design documents describe version 6 and the repair. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/nvx-openvmm-promote/SKILL.md | 2 +- doc/build.md | 2 +- doc/ci.md | 2 +- doc/design/3.1-platform-control-device.md | 3 +- doc/design/3.5-network.md | 30 ++- doc/design/5.5-restore-protocol.md | 52 +++-- guest/common/nvx-snapshot | 78 +++++-- nvx-rs/README.md | 2 +- nvx-rs/src/contract.rs | 12 +- nvx-rs/src/guest/time/api.rs | 69 ++++++- nvx-rs/src/guest/time/commands.rs | 15 +- nvx-rs/src/guest/time/decoders.rs | 204 +++++++++++++++++- nvx-rs/src/guest/time/mod.rs | 2 +- nvx-rs/src/guest/time/test_mode.rs | 10 + nvx-rs/tests/guest_tools.rs | 102 +++++++++ nvx-rs/tests/openvmm_vectors.rs | 7 +- nvx-rs/tests/vectors/README.md | 2 +- scripts/test_microvm_tests.py | 217 +++++++++++++++++++- 18 files changed, 747 insertions(+), 64 deletions(-) diff --git a/.github/skills/nvx-openvmm-promote/SKILL.md b/.github/skills/nvx-openvmm-promote/SKILL.md index f68993841..cee7b5b14 100644 --- a/.github/skills/nvx-openvmm-promote/SKILL.md +++ b/.github/skills/nvx-openvmm-promote/SKILL.md @@ -92,7 +92,7 @@ git diff --check `nvx-rs` reads copies of what it needs from OpenVMM, because the `nvx-rs` CI job cannot check out the submodule: the control-session protocol vectors, excerpts of OpenVMM's sources, and, in `nvx::contract`, the versions of OpenVMM's snapshot format, saved-state -schema, time ABI, and restore packet. Compare them with the revision's, as the NVX CLI +schema, time ABI, and restore packets. Compare them with the revision's, as the NVX CLI jobs do: ```console diff --git a/doc/build.md b/doc/build.md index bbc3dec8a..2cd11dc29 100644 --- a/doc/build.md +++ b/doc/build.md @@ -213,7 +213,7 @@ hypervisor identity, and runs the cross-vCPU warp test that the [time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs skew bound. `nvx-time` implements the time ABI's [guest obligations](design/3.2-clock-and-time.md#3210-guest-obligations): the conformance -checks, the violation watcher, the restore packet v5 and time-sample v2 +checks, the violation watcher, the restore packet v5 and v6 and time-sample v2 parsers, the snapshot agent's time steps, and the wall-clock discipline, which applies the host's clock correction. Before shell-ready it only steps the clock to host UTC and sets its rate; the other boot checks run in diff --git a/doc/ci.md b/doc/ci.md index 095846fca..630438db9 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -695,7 +695,7 @@ compares the submodule's control-session protocol vectors, and the lines of its platform allowlist of kernel command-line tokens, with the copies that the protocol and template tests read in the `nvx-rs` job, which cannot check out OpenVMM, and the versions of OpenVMM's snapshot format, saved-state schema, -time ABI, and restore packet with those of `nvx::contract`; +time ABI, and restore packets with those of `nvx::contract`; see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors). The ignored test `tests/openvmm_cpu_profiles.rs` likewise compares the submodule's built-in CPU profiles with the copy of their identities from which diff --git a/doc/design/3.1-platform-control-device.md b/doc/design/3.1-platform-control-device.md index 297350d0b..d1cd6421c 100644 --- a/doc/design/3.1-platform-control-device.md +++ b/doc/design/3.1-platform-control-device.md @@ -32,7 +32,8 @@ machine omits, and carries: console. - The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), which gives a restored guest what it needs to repair its clocks, - processors, and memory, and the fresh random bytes from which it + processors, and memory, the network settings of a network policy that the + restore installed, and the fresh random bytes from which it [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh). - Host time samples, which pair host UTC with the guest TSC and carry the host's clock correction, from which the guest sets and disciplines its wall diff --git a/doc/design/3.5-network.md b/doc/design/3.5-network.md index 127dc9b66..d96fee653 100644 --- a/doc/design/3.5-network.md +++ b/doc/design/3.5-network.md @@ -248,11 +248,31 @@ failure, or a UDP query beyond that limit, receives `SERVFAIL`, and a TCP query beyond it resets its connection. The snapshot records the network profile, identity, and policy digest. -Restore reconstructs a fresh endpoint and requires the same network profile -and policy; native sockets, DNS requests, and flows are never serialized. -Published guest listeners and explicit host forwards are process-local -attachments, so capture and restore reject private-network ingress and -explicit host-loopback access, which create them. +Restore reconstructs a fresh endpoint and requires the same network profile, +and the same policy unless the NIC is a template: one captured with the +complete deny-all policy, which denies every guest frame, host loopback, and +inbound access, and keeps no proxy endpoint, so that nothing in the snapshot +depends on the policy beyond the NIC's identity. A restore of a template of +the platform tier, or of a snapshot without a tier, may install a policy of +its own with `--restore-network-policy`, which OpenVMM validates as at a cold +boot, binds to the captured identity, and installs, with its forwards, before +any vCPU runs, while the snapshot's contract keeps the template's policy. A +snapshot without a tier may hold a workload that has already run, but under +the deny-all that workload had no connectivity, so the restore grants it +connectivity that it never had. Native sockets, DNS requests, and flows are +never serialized. Published guest listeners and explicit host forwards are +process-local attachments, so capture rejects private-network ingress and +explicit host-loopback access, which create them, and so does every restore +that keeps the captured policy. + +A restored guest keeps its captured command line, which under a template's +deny-all names no DNS server and asks for no listener announcements. A +platform clone takes both from its activation. The guest of a snapshot +without a tier takes them from version 6 of the restore packet, which carries +the new policy's DNS server and whether the guest announces its listeners +([Restore Packet](5.5-restore-protocol.md#555-restore-packet)), and its +repair applies them before it acknowledges the restore, which then gates +host input ([Guest Repair](5.5-restore-protocol.md#557-guest-repair)). Capture quiesces the endpoint, drains completion ownership, and requires the saved queues to contain no unrepresented RX or TX packets. It saves the queue diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index 57a98df15..93a8e8516 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -194,21 +194,24 @@ tier or a processor target requires it. Figure 5.7 shows the layout of the restore packet, and Table 5.7 lists its flags. Every restore exposes one restore packet through the byte channel, for -every tier and for untiered snapshots. +every tier and for untiered snapshots. The packet is version 5, except at a +restore that installs a new network policy on a snapshot without a tier, +whose packet is version 6. ```text offset size field 0 3 magic "OVR" - 3 1 version = 5 + 3 1 version = 5, or 6 with network settings 4 1 flags 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 6 1 memory_range_count 0 unless MEMORY_TARGET - 7 1 reserved = 0 + 7 1 network reserved = 0 in version 5 8 4 generation u32, little-endian, g >= 1 12 4 rate_deviation i32, little-endian, ppm scaled by 2^16 16 8 downtime_ns u64, little-endian, D 24 16*n ranges n x { u64 gpa_start, u64 length }, LE 24+16n 64 entropy + 88+16n 16 dns_server version 6 only ``` **Figure 5.7.** Restore packet layout. @@ -226,13 +229,30 @@ offset size field Every field is sealed before the first restored processor runs. The packet carries no host UTC: the guest takes a [time sample](3.2-clock-and-time.md#329-time-samples) after it, which pairs -host UTC with the guest TSC on the host. A packet without ranges is 88 bytes. -The packet is authoritative, and the status bits mirror it; the -acknowledgement flag spares the guest from inferring gating from the tier and -targets. A VM process exposes at most one packet, and reading its last byte -consumes it, so a guest captured again in the same process never sees an -earlier packet. The guest also ignores a packet whose generation equals its -recorded generation. +host UTC with the guest TSC on the host. A packet without ranges is 88 bytes +in version 5 and 104 bytes in version 6. The packet is authoritative, and the +status bits mirror it; the acknowledgement flag spares the guest from +inferring gating from the tier and targets. A VM process exposes at most one +packet, and reading its last byte consumes it, so a guest captured again in +the same process never sees an earlier packet. The guest also ignores a packet +whose generation equals its recorded generation. + +Version 6 carries the network settings of the policy that the restore +installed in place of a NIC template's deny-all, which the guest's captured +command line does not name +([Bounded State, Snapshots, and Restore](3.5-network.md#356-bounded-state-snapshots-and-restore)). +Bits 0 and 1 of its network byte give the family of the guest's DNS server, 0 +for none, 1 for IPv4, or 2 for IPv6, and bit 2 asks the guest to announce its +listeners; the guest rejects a packet with family 3 or any other bit set. The +DNS server is the gateway address that a cold boot under the policy would +name ([DNS](3.5-network.md#355-dns)): an IPv4 address followed by 12 zero +bytes, an IPv6 address, or 16 zero bytes for none; the guest rejects an +unspecified address and a record that does not match its family. The guest +announces its listeners when the policy publishes them, through private-network +ingress or explicit host-loopback access. Such a restore always sets +`ACK_REQUIRED`, so the guest applies the settings before host input reaches +it, and a guest that reads version 5 alone rejects version 6 and so fails the +restore instead of ignoring its settings. ## 5.5.6 Generations @@ -257,14 +277,20 @@ runs before the acknowledgement: 1. Read the byte-channel status, whose restore-packet bit is always set after a restore. 2. Select the packet, read it four bytes per access, and validate its magic, - version, reserved bits, counts, generation, and generation ID. + version, reserved bits, counts, network settings, generation, and + generation ID. 3. Set the wall clock: in the same port transaction as the packet, take up to three [time samples](3.2-clock-and-time.md#329-time-samples) of the packet's generation, until one pairs host UTC with the guest TSC within 100 µs, step the clock by its offset, and hold the frequency correction that its host clock correction gives, which carries the rate deviation, within ±500 ppm. -4. Activate processors and memory as requested. +4. Activate processors and memory as requested, then apply the network + settings of a version 6 packet as the guest's init applies a cold boot's: + name the DNS server in `/etc/resolv.conf`, and, when the packet asks for + the announcements, start `nvx-ingress`, which announces the guest's + listeners to its IPv4 gateway, and wait until its socket is bound to a + privileged source port and its program runs. 5. Refresh entropy and identity, as [Entropy Refresh](#558-entropy-refresh) describes. 6. Acknowledge through the snapshot port when `ACK_REQUIRED` is set. @@ -321,7 +347,7 @@ As Table 5.8 shows, the refresh depends on the restore path. | `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it | | Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Refreshed, as for a clone | | `instance-checkpoint` tier (resume) | Not reseeded, by design; only the generation ID changes | Preserved | -| Untiered, with nothing to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved | +| Untiered, with no processors or memory to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved | An instance checkpoint is not reseeded because it is single-use: its random state continues in exactly one process, and keeping it preserves the diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index 680a6af1a..e09e6cf3e 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -26,7 +26,7 @@ esac runtime=/run/nvx container_cgroup=/sys/fs/cgroup/container scratch_mount=$runtime/scratch -# Restore packet v5 flags (doc/design/5.5-restore-protocol.md). +# Restore packet v5 and v6 flags (doc/design/5.5-restore-protocol.md). PACKET_MEMORY_TARGET=2 PACKET_ACK_REQUIRED=4 container_frozen=false @@ -397,6 +397,35 @@ EOF done } +# Guest repair step 4 of a restore that replaced the network policy of a +# snapshot without a tier (doc/design/5.5-restore-protocol.md): applies the +# network settings of the new policy, which the restore packet carries, as +# NVX's init applies those of the virtnet_dns= and virtnet_ingress= tokens at +# a cold boot. It names the DNS server DNS_SERVER, unless it is "-", in +# /etc/resolv.conf, and, when ANNOUNCE is 1, starts nvx-ingress, which +# announces the guest's listeners to the IPv4 gateway, which publishes them. +# A failure of either fails the restore. +apply_restore_network() { + dns_server=$1 + announce_listeners=$2 + if [ "$dns_server" != - ]; then + printf 'nameserver %s\n' "$dns_server" >/etc/resolv.conf || + fail_closed "failed to name the DNS server in /etc/resolv.conf" + fi + if [ "$announce_listeners" -eq 1 ]; then + gateway=$(cmdline_value virtnet_gw) || + fail_closed "listener announcements need an IPv4 gateway" + [ -x /sbin/nvx-ingress ] || + fail_closed "listener announcements need /sbin/nvx-ingress" + # start returns once the announcer's socket is bound to a privileged + # source port and its program runs, which then passes to init. Their + # output goes to the console, as at a cold boot, and not to this + # script's caller, which the announcer outlives. + /sbin/nvx-ingress start "$gateway" /dev/null 2>/dev/console || + fail_closed "failed to start nvx-ingress" + fi +} + configured_snapshot_tier=$(cmdline_value nvx_snapshot_tier || true) if [ -n "$configured_snapshot_tier" ]; then case "$configured_snapshot_tier:$scratch_policy" in @@ -432,17 +461,21 @@ export NVX_VM_GENERATION_ID="$generation_id" entropy=$runtime/restore-entropy # Guest repair steps 4 to 7 (doc/design/5.5-restore-protocol.md). nvx-time -# capture has already read restore packet v5, checked its generation, set -# the wall clock from a time sample, and written the entropy. Its metadata is -# FLAGS, ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by -# START and LENGTH for each memory range. +# capture has already read the restore packet, version 5 or 6, checked its +# generation, set the wall clock from a time sample, and written the entropy. +# Its metadata is FLAGS, ONLINE_VP_COUNT, RANGE_COUNT, the new generation ID, +# the DNS server of a version 6 packet's network settings, or "-" for none, +# and 1 if they ask the guest to announce its listeners, or else 0, followed +# by START and LENGTH for each memory range. post_restore() { post_restore_pending=true - [ "$#" -ge 4 ] || fail_closed "restore packet metadata is invalid" + [ "$#" -ge 6 ] || fail_closed "restore packet metadata is invalid" packet_flags=$1 restore_online_count=$2 restore_memory_range_count=$3 - shift 4 + restore_dns_server=$5 + restore_announce_listeners=$6 + shift 6 case "$packet_flags:$restore_online_count:$restore_memory_range_count" in *[!0-9:]* | :* | *::* | *:) fail_closed "restore packet metadata is invalid" @@ -452,13 +485,26 @@ post_restore() { 0 | 1 | 2 | 4 | 8) ;; *) fail_closed "restore packet has an invalid online CPU count" ;; esac - # A plain untiered restore, with no processors or memory to add, prints - # nothing, as before: each console byte is a port exit on its latency - # path. + case "$restore_announce_listeners" in + 0 | 1) ;; + *) fail_closed "restore packet metadata is invalid" ;; + esac + case "$restore_dns_server" in + -) ;; + '' | *[!0-9a-f.:]*) fail_closed "restore packet metadata is invalid" ;; + esac + restore_network=false + if [ "$restore_dns_server" != - ] || [ "$restore_announce_listeners" -eq 1 ]; then + restore_network=true + fi + # A plain untiered restore, with no processors or memory to add and no + # network settings to apply, prints nothing, as before: each console byte + # is a port exit on its latency path. if [ "$snapshot_tier" = legacy ] && [ "$restore_online_count" -eq 0 ] && [ "$restore_memory_range_count" -eq 0 ] && - [ $((packet_flags & PACKET_MEMORY_TARGET)) -eq 0 ]; then + [ $((packet_flags & PACKET_MEMORY_TARGET)) -eq 0 ] && + [ "$restore_network" = false ]; then finish_restore "$packet_flags" quiet return fi @@ -471,6 +517,10 @@ post_restore() { elif [ "$restore_memory_range_count" -ne 0 ]; then fail_closed "restore packet has memory ranges without a memory target" fi + if [ "$restore_network" = true ]; then + console_status "NVX-POST-RESTORE-STAGE: network" + apply_restore_network "$restore_dns_server" "$restore_announce_listeners" + fi # An untiered restore without processors or memory to add leaves the # entropy and identity to its caller, as before. @@ -635,9 +685,9 @@ if [ "$scratch_policy" = fresh ]; then else capture_request=1 fi -# An untiered restore with nothing to activate needs no shell work before -# the acknowledgement, so nvx-time finishes it and reports 2: no second -# process starts on the restore path. +# An untiered restore with nothing to activate and no network settings to +# apply needs no shell work before the acknowledgement, so nvx-time finishes +# it and reports 2: no second process starts on the restore path. finish_option= [ "$snapshot_tier" = legacy ] && finish_option=--finish capture_metadata=$( diff --git a/nvx-rs/README.md b/nvx-rs/README.md index fa6c58119..6210342b3 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2720,7 +2720,7 @@ NVX_BUNDLE=skip cargo test --locked --all-features --lib -- \ `tests/openvmm_vectors.rs` is an ignored test that compares the copies of OpenVMM's vectors in `tests/vectors/openvmm/` with the files of the OpenVMM submodule, which the `nvx-rs` job cannot check out, and the versions of -OpenVMM's snapshot format, saved-state schema, time ABI, and restore packet in +OpenVMM's snapshot format, saved-state schema, time ABI, and restore packets in `nvx::contract` with the submodule's. The NVX CLI jobs run it once they check the submodule out, and so does an OpenVMM pin promotion; see [OpenVMM's vectors](tests/vectors/README.md#openvmms-vectors). diff --git a/nvx-rs/src/contract.rs b/nvx-rs/src/contract.rs index 504be5b99..68cb838bf 100644 --- a/nvx-rs/src/contract.rs +++ b/nvx-rs/src/contract.rs @@ -40,7 +40,10 @@ //! restored guest, and that the guest decodes. It changes with the packet's layout or meaning. //! //! Each of these four must match the pinned OpenVMM's, which the ignored test -//! `tests/openvmm_vectors.rs` checks in the jobs that check out the OpenVMM submodule. The key +//! `tests/openvmm_vectors.rs` checks in the jobs that check out the OpenVMM submodule. So must +//! [`RESTORE_PACKET_NETWORK_VERSION`], the version of the packet that OpenVMM gives instead at a +//! restore that replaces the network policy of a snapshot without a tier, which the guest decodes +//! too. No clone of a template is restored so, so the key does not hold it. The key //! also holds the version of the activation contract, //! [`activation::VERSION`](crate::activation::VERSION), through which each clone receives the //! settings of its own launch. @@ -80,6 +83,13 @@ pub const TIME_ABI_VERSION: u64 = 2; /// /// Version 5 carries no host UTC: the guest takes a time sample to set its clock. pub const RESTORE_PACKET_VERSION: u64 = 5; +/// Version of the restore packet that OpenVMM gives a restored guest in place of +/// [`RESTORE_PACKET_VERSION`]'s at a restore that replaces the network policy of a snapshot +/// without a tier: that version's packet, with the new policy's DNS server and whether the guest +/// announces its listeners under it, which a guest that decodes only the other version refuses. +/// This crate's guest decodes both. It must match `RESTORE_PACKET_NETWORK_VERSION` in OpenVMM's +/// `vm/devices/chipset_resources/src/microvm_time.rs`. +pub const RESTORE_PACKET_NETWORK_VERSION: u64 = 6; /// Returns how the `openvmm` section of an NVX `SOURCE-MANIFEST.json` departs from the contracts /// that this crate speaks: one `NAME: expected VALUE, found VALUE` entry, in JSON, for each field diff --git a/nvx-rs/src/guest/time/api.rs b/nvx-rs/src/guest/time/api.rs index 19c420302..0ad7d146e 100644 --- a/nvx-rs/src/guest/time/api.rs +++ b/nvx-rs/src/guest/time/api.rs @@ -3,6 +3,7 @@ use std::ffi::CString; use std::fmt::{self, Display}; use std::io; +use std::net::IpAddr; use std::os::unix::ffi::OsStrExt; use std::path::PathBuf; @@ -580,6 +581,8 @@ pub struct Restore { generation_id: GenerationId, /// The memory ranges to activate. memory_ranges: Vec, + /// The network settings that the restore packet carries, if any. + network: Option, /// Whether the call finished the restore. finished: bool, /// The restore's fresh entropy, in the in-memory mode, until the caller takes it. @@ -588,8 +591,9 @@ pub struct Restore { impl Restore { /// Returns whether the call finished the restore, as [`CaptureRequest::finish`] asked. - /// Otherwise, the caller activates the restore's processors and memory, refreshes its - /// entropy and identity, and finishes it with [`TimeAbi::restore_finish`]. + /// Otherwise, the caller activates the restore's processors and memory, applies its network + /// settings, refreshes its entropy and identity, and finishes it with + /// [`TimeAbi::restore_finish`]. /// /// A restore whose handoff of step 7 fails after its acknowledgement counts as finished, /// since finishing it again would acknowledge it twice, as `nvx-time capture --finish` @@ -646,6 +650,47 @@ impl Restore { pub(crate) fn online_vp_count(&self) -> u8 { self.online_vp_count } + + /// Returns the network settings that the restore packet carries: those of a restore that + /// replaced the network policy of a snapshot without a tier, which the guest applies before it + /// finishes the restore, or `None` for a restore that carries none. + pub fn network(&self) -> Option { + self.network + } +} + +/// The network settings of a restore that replaced the network policy of a snapshot without a +/// tier, which its restore packet carries: the DNS server of the new policy, and whether the guest +/// announces its listeners under it, as a cold boot under that policy learns them from the +/// `virtnet_dns=` and `virtnet_ingress=` tokens of its command line. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct RestoreNetwork { + /// The DNS server, one of the gateway's addresses, or `None` when the policy lets the guest + /// reach no DNS service of the gateway. + dns_server: Option, + /// Whether the guest announces its listeners to the gateway, which publishes them. + announce_listeners: bool, +} + +impl RestoreNetwork { + /// Returns the network settings with `dns_server` and `announce_listeners`. + pub(crate) const fn new(dns_server: Option, announce_listeners: bool) -> Self { + Self { + dns_server, + announce_listeners, + } + } + + /// Returns the DNS server that the guest's resolver names, or `None` for none. + pub fn dns_server(&self) -> Option { + self.dns_server + } + + /// Returns whether the guest announces its listeners to its IPv4 gateway, which publishes + /// them. + pub fn announces_listeners(&self) -> bool { + self.announce_listeners + } } #[cfg(test)] @@ -658,6 +703,7 @@ impl Restore { online_vp_count: 0, generation_id, memory_ranges: Vec::new(), + network: None, finished: false, seed: None, } @@ -951,6 +997,7 @@ fn repair_restore( header, ranges, entropy, + network, } = packet; let generation_id = entropy.generation_id(); if generation_id == target.previous_id { @@ -998,7 +1045,8 @@ fn repair_restore( && target.daemon_pid.is_some() && header.online_vp_count == 0 && header.range_count == 0 - && !header.memory_target(); + && !header.memory_target() + && !header.network_repair(); let seed = match target.entropy.deliver(entropy) { Ok(seed) => seed, Err(detail) => return Err(repair_failed(runtime, ViolationCode::RepairPacket, detail)), @@ -1031,6 +1079,7 @@ fn repair_restore( .iter() .map(|&(start, length)| MemoryRange { start, length }) .collect(), + network, finished, seed, })) @@ -1718,6 +1767,7 @@ mod tests { online_vp_count: 4, generation_id: GenerationId::from_bytes([7; GenerationId::LENGTH]), memory_ranges: vec![range], + network: None, finished: false, seed: None, }; @@ -1725,17 +1775,27 @@ mod tests { assert!(!restore.finished()); assert_eq!(restore.processor_target(), Some(4)); assert_eq!(restore.memory_ranges(), [range]); + assert_eq!(restore.network(), None); assert_eq!((range.start(), range.length()), (1 << 32, 128 << 20)); assert_eq!( restore.generation_id(), GenerationId::from_bytes([7; GenerationId::LENGTH]) ); + let dns_server = IpAddr::from([10, 0, 0, 1]); + let networked = Restore { + network: Some(RestoreNetwork::new(Some(dns_server), true)), + ..restore + }; + let network = networked.network().unwrap(); + assert_eq!(network.dns_server(), Some(dns_server)); + assert!(network.announces_listeners()); let plain = Restore { flags: 0, online_vp_count: 0, memory_ranges: Vec::new(), + network: None, finished: true, - ..restore + ..networked }; assert!(!plain.acknowledgement_required() && !plain.memory_target()); assert!(plain.finished()); @@ -1798,6 +1858,7 @@ mod tests { online_vp_count: 0, generation_id: seed.generation_id(), memory_ranges: Vec::new(), + network: None, finished: false, seed: Some(seed), }; diff --git a/nvx-rs/src/guest/time/commands.rs b/nvx-rs/src/guest/time/commands.rs index a38fee5d5..f426f0428 100644 --- a/nvx-rs/src/guest/time/commands.rs +++ b/nvx-rs/src/guest/time/commands.rs @@ -221,8 +221,9 @@ fn capture_request(arguments: &[&[u8]]) -> Option { /// Requests a capture, and repairs a restore, which it prints: `0` when the VM continued, or /// `1` or `2`, for a restore that it left unfinished or finished, then the restore packet's -/// flags, online VP count, range count, and new generation ID, and each range's start and -/// length. +/// flags, online VP count, range count, and new generation ID, the DNS server of its network +/// settings, or `-` for none, `1` if they ask the guest to announce its listeners, or else `0`, +/// and each range's start and length. fn cmd_capture( abi: TimeAbi, arguments: &[&[u8]], @@ -241,13 +242,19 @@ fn cmd_capture( Ok(CaptureOutcome::Restored(restore)) => restore, Err(error) => return Err(time_failure(err, error)), }; + let network = restore.network(); + let dns_server = network + .and_then(|network| network.dns_server()) + .map_or_else(|| "-".to_owned(), |server| server.to_string()); let mut line = format!( - "{} {} {} {} {}", + "{} {} {} {} {} {} {}", if restore.finished() { 2 } else { 1 }, restore.flags(), restore.online_vp_count(), restore.memory_ranges().len(), - restore.generation_id() + restore.generation_id(), + dns_server, + u8::from(network.is_some_and(|network| network.announces_listeners())), ); for range in restore.memory_ranges() { let _ = write!(line, " {} {}", range.start(), range.length()); diff --git a/nvx-rs/src/guest/time/decoders.rs b/nvx-rs/src/guest/time/decoders.rs index 6dd894290..0ab8e77e5 100644 --- a/nvx-rs/src/guest/time/decoders.rs +++ b/nvx-rs/src/guest/time/decoders.rs @@ -1,8 +1,10 @@ //! Pure decoders, parsers, and arithmetic of the guest time ABI. use std::fmt::{self, Display, Write as _}; +use std::net::IpAddr; use std::ops::RangeInclusive; +use super::api::RestoreNetwork; use super::cpu_mask::CpuMask; use super::events::ViolationCode; use super::state::HostClockStatus; @@ -39,6 +41,28 @@ const PACKET_VERSION: u8 = { assert!(version <= u8::MAX as u64); version as u8 }; +/// The version of a restore packet that carries network settings, +/// [`RESTORE_PACKET_NETWORK_VERSION`](crate::contract::RESTORE_PACKET_NETWORK_VERSION): +/// [`PACKET_VERSION`]'s packet, whose reserved byte is the network byte, and whose DNS server +/// follows the entropy. +const PACKET_NETWORK_VERSION: u8 = { + let version = crate::contract::RESTORE_PACKET_NETWORK_VERSION; + assert!(version <= u8::MAX as u64); + version as u8 +}; +/// The bits of the network byte that give the DNS server's address family. +const PACKET_DNS_MASK: u8 = 0x03; +/// The network byte's DNS server family of an IPv4 server. +const PACKET_DNS_IPV4: u8 = 0x01; +/// The network byte's DNS server family of an IPv6 server. +const PACKET_DNS_IPV6: u8 = 0x02; +/// The network byte's bit that asks the guest to announce its listeners. +const PACKET_ANNOUNCE_LISTENERS: u8 = 0x04; +/// Mask of the network byte's bits that version six defines. +const PACKET_NETWORK_KNOWN: u8 = PACKET_DNS_MASK | PACKET_ANNOUNCE_LISTENERS; +/// Encoded length of the DNS server of a packet that carries network settings: an IPv6 address, +/// or an IPv4 address and twelve zero bytes. +pub(crate) const PACKET_DNS_SIZE: usize = 16; /// Restore flag indicating that the downtime came from the host's UTC clock. const PACKET_DOWNTIME_UTC: u8 = 0x01; /// Restore flag indicating that memory ranges follow. @@ -151,7 +175,7 @@ pub(crate) enum PacketError { IncompleteHeader, /// The magic number is not `OVR`. Magic([u8; 3]), - /// The version is not the time ABI's. + /// The version is neither [`PACKET_VERSION`] nor [`PACKET_NETWORK_VERSION`]. Version(u8), /// The flags set reserved bits. ReservedFlags(u8), @@ -159,18 +183,25 @@ pub(crate) enum PacketError { OnlineVpCount(u8), /// Memory ranges follow, but the flags select no memory target. RangesWithoutTarget(u8), - /// The reserved byte is not zero. + /// The reserved byte of a packet without network settings is not zero. ReservedByte(u8), + /// The network byte sets reserved bits, or names no DNS server family that version six + /// defines. + NetworkByte(u8), /// The generation is zero. ZeroGeneration, /// The rate deviation exceeds 250 ppm. RateDeviation(i32), /// The downtime exceeds 30 days. Downtime(u64), - /// The packet ends before the memory ranges and the entropy that its header gives. + /// The packet ends before the memory ranges, the entropy, and the DNS server that its header + /// gives. IncompleteBody, /// The memory range at this index is empty, or ends past the address space. Range(usize), + /// The DNS server is unspecified, pads an IPv4 address with a byte that is not zero, or is + /// not zero when the network byte names no DNS server. + DnsServer, } impl Display for PacketError { @@ -195,6 +226,10 @@ impl Display for PacketError { "restore packet has {count} memory ranges without a memory target" ), Self::ReservedByte(byte) => write!(formatter, "restore packet reserved byte is {byte}"), + Self::NetworkByte(byte) => write!( + formatter, + "restore packet network byte 0x{byte:02x} is invalid" + ), Self::ZeroGeneration => formatter.write_str("restore packet generation is 0"), Self::RateDeviation(deviation) => write!( formatter, @@ -206,13 +241,15 @@ impl Display for PacketError { ), Self::IncompleteBody => formatter.write_str("restore packet body is incomplete"), Self::Range(index) => write!(formatter, "restore memory range {index} is invalid"), + Self::DnsServer => formatter.write_str("restore packet DNS server is invalid"), } } } impl std::error::Error for PacketError {} -/// The validated header of a restore packet. +/// The validated header of a restore packet: of [`PACKET_VERSION`], or of +/// [`PACKET_NETWORK_VERSION`], which carries network settings. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct PacketHeader { /// Packet behavior flags. @@ -221,6 +258,9 @@ pub(crate) struct PacketHeader { pub(crate) online_vp_count: u8, /// Number of following memory ranges. pub(crate) range_count: u8, + /// The network byte of a packet that carries network settings, or `None` for one that + /// carries none. + pub(crate) network: Option, /// New VM generation. pub(crate) generation: u32, /// Host rate deviation in signed scaled parts per million. @@ -239,6 +279,7 @@ impl PacketHeader { flags, online_vp_count, range_count, + network: (version == PACKET_NETWORK_VERSION).then_some(reserved), generation: fields.u32()?, rate_deviation: fields.i32()?, downtime_ns: fields.u64()?, @@ -246,7 +287,7 @@ impl PacketHeader { if magic != PACKET_MAGIC { return Err(PacketError::Magic(magic)); } - if version != PACKET_VERSION { + if version != PACKET_VERSION && version != PACKET_NETWORK_VERSION { return Err(PacketError::Version(version)); } if flags & !PACKET_KNOWN_FLAGS != 0 { @@ -258,8 +299,15 @@ impl PacketHeader { if !header.memory_target() && range_count != 0 { return Err(PacketError::RangesWithoutTarget(range_count)); } - if reserved != 0 { - return Err(PacketError::ReservedByte(reserved)); + match header.network { + None if reserved != 0 => return Err(PacketError::ReservedByte(reserved)), + Some(network) + if network & !PACKET_NETWORK_KNOWN != 0 + || network & PACKET_DNS_MASK == PACKET_DNS_MASK => + { + return Err(PacketError::NetworkByte(network)); + } + _ => {} } if header.generation == 0 { return Err(PacketError::ZeroGeneration); @@ -275,7 +323,18 @@ impl PacketHeader { /// Returns the length of the body that follows the header. pub(crate) fn body_size(&self) -> usize { - usize::from(self.range_count) * PACKET_RANGE_SIZE + PACKET_ENTROPY_SIZE + let dns = if self.network.is_some() { + PACKET_DNS_SIZE + } else { + 0 + }; + usize::from(self.range_count) * PACKET_RANGE_SIZE + PACKET_ENTROPY_SIZE + dns + } + + /// Returns whether the packet asks the guest to apply network settings: a DNS server, or the + /// announcement of its listeners. + pub(crate) const fn network_repair(&self) -> bool { + matches!(self.network, Some(network) if network != 0) } /// Returns whether the host measured the downtime with its UTC clock, rather than with its @@ -307,10 +366,13 @@ pub(crate) struct RestorePacket { /// Fresh entropy, which starts with the new VM generation ID, decoded straight into the /// seed's allocation. pub(crate) entropy: Seed, + /// The network settings of a packet that carries them. + pub(crate) network: Option, } impl RestorePacket { - /// Decodes and validates the memory ranges and entropy that follow `header` in `body`. + /// Decodes and validates the memory ranges, the entropy, and the DNS server that follow + /// `header` in `body`. pub(crate) fn parse(header: PacketHeader, body: &[u8]) -> Result { let mut fields = Fields::new(body, PacketError::IncompleteBody); let ranges = (0..usize::from(header.range_count)) @@ -322,10 +384,19 @@ impl RestorePacket { Ok((start, length)) }) .collect::>()?; + let entropy = fields.chunk()?; + let network = match header.network { + Some(network) => Some(RestoreNetwork::new( + parse_dns_server(network, fields.bytes()?)?, + network & PACKET_ANNOUNCE_LISTENERS != 0, + )), + None => None, + }; Ok(Self { header, ranges, - entropy: Seed::copy_from(fields.chunk()?), + entropy: Seed::copy_from(entropy), + network, }) } @@ -335,6 +406,30 @@ impl RestorePacket { } } +/// Decodes the DNS server whose family the `network` byte of a restore packet names, from its +/// encoded `address`: none, which must be zero, an IPv4 address and twelve zero bytes, or an IPv6 +/// address, neither unspecified. +fn parse_dns_server( + network: u8, + address: [u8; PACKET_DNS_SIZE], +) -> Result, PacketError> { + let server = match network & PACKET_DNS_MASK { + PACKET_DNS_IPV4 => match address.split_first_chunk::<4>() { + Some((ipv4, padding)) if padding.iter().all(|&byte| byte == 0) => { + Some(IpAddr::from(*ipv4)) + } + _ => return Err(PacketError::DnsServer), + }, + PACKET_DNS_IPV6 => Some(IpAddr::from(address)), + _ if address == [0; PACKET_DNS_SIZE] => None, + _ => return Err(PacketError::DnsServer), + }; + if server.is_some_and(|server| server.is_unspecified()) { + return Err(PacketError::DnsServer); + } + Ok(server) +} + /// Why a host time sample is invalid. Its `Display` is the sample's diagnostic. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) enum TimeSampleError { @@ -1230,6 +1325,95 @@ mod tests { bytes } + /// Returns a valid packet header of version 6 without memory ranges, of generation 1, with + /// the network byte `network`. + fn network_header(network: u8) -> [u8; PACKET_HEADER_SIZE] { + let mut bytes = valid_header(); + bytes[3] = PACKET_NETWORK_VERSION; + bytes[7] = network; + bytes + } + + #[test] + fn version_6_packets_carry_their_network_settings() { + let entropy: Vec = (0..64).collect(); + let ipv6: std::net::Ipv6Addr = "fd00::a00:1".parse().unwrap(); + for (network, dns_server, expected) in [ + ( + 0x05, + [10, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + RestoreNetwork::new(Some(IpAddr::from([10, 0, 0, 1])), true), + ), + ( + 0x02, + ipv6.octets(), + RestoreNetwork::new(Some(IpAddr::V6(ipv6)), false), + ), + (0x04, [0; 16], RestoreNetwork::new(None, true)), + (0x00, [0; 16], RestoreNetwork::new(None, false)), + ] { + let header = PacketHeader::parse(&network_header(network)).unwrap(); + assert_eq!(header.network, Some(network)); + assert_eq!(header.network_repair(), network != 0); + assert_eq!(header.body_size(), PACKET_ENTROPY_SIZE + PACKET_DNS_SIZE); + let body = [entropy.as_slice(), &dns_server].concat(); + let packet = RestorePacket::parse(header, &body).unwrap(); + assert_eq!(packet.entropy.bytes().as_slice(), entropy); + assert_eq!(packet.network, Some(expected), "{network:#04x}"); + assert_eq!( + RestorePacket::parse(header, &body[1..]).unwrap_err(), + PacketError::IncompleteBody + ); + } + + // Version 5 has no network byte, and no DNS server after its entropy. + let header = PacketHeader::parse(&valid_header()).unwrap(); + assert_eq!(header.network, None); + assert!(!header.network_repair()); + let packet = RestorePacket::parse(header, &entropy).unwrap(); + assert_eq!(packet.network, None); + } + + #[test] + fn version_6_refuses_malformed_network_settings() { + for (network, error) in [ + (0x08, "restore packet network byte 0x08 is invalid"), + (0x03, "restore packet network byte 0x03 is invalid"), + ] { + let error_text = PacketHeader::parse(&network_header(network)) + .unwrap_err() + .to_string(); + assert_eq!(error_text, error); + } + let mut bytes = valid_header(); + bytes[7] = 1; + assert_eq!( + PacketHeader::parse(&bytes), + Err(PacketError::ReservedByte(1)) + ); + bytes[3] = 7; + assert_eq!(PacketHeader::parse(&bytes), Err(PacketError::Version(7))); + + let ipv4 = [10, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]; + let mut padded = ipv4; + padded[15] = 1; + for (network, dns_server) in [ + // A nonzero byte after an IPv4 server. + (0x01, padded), + // A server that the network byte does not name. + (0x00, ipv4), + // Unspecified servers. + (0x01, [0; 16]), + (0x02, [0; 16]), + ] { + let header = PacketHeader::parse(&network_header(network)).unwrap(); + let body = [&[0; PACKET_ENTROPY_SIZE][..], &dns_server].concat(); + let error = RestorePacket::parse(header, &body).unwrap_err(); + assert_eq!(error, PacketError::DnsServer, "{network:#04x}"); + assert_eq!(error.to_string(), "restore packet DNS server is invalid"); + } + } + #[test] fn samples_decode_every_field_of_version_two() { assert_eq!( diff --git a/nvx-rs/src/guest/time/mod.rs b/nvx-rs/src/guest/time/mod.rs index f88a2a8a5..30330f580 100644 --- a/nvx-rs/src/guest/time/mod.rs +++ b/nvx-rs/src/guest/time/mod.rs @@ -77,7 +77,7 @@ use std::process::ExitCode; pub(crate) use api::PreCaptureError; pub use api::{ CaptureOutcome, CaptureRequest, ColdBoot, CpuSet, MemoryRange, Restore, RestoreFinish, - ScratchPolicy, TimeAbi, TimeError, Violation, + RestoreNetwork, ScratchPolicy, TimeAbi, TimeError, Violation, }; use super::status::{CommandError, run_main}; diff --git a/nvx-rs/src/guest/time/test_mode.rs b/nvx-rs/src/guest/time/test_mode.rs index 7fff82f3f..721cd5496 100644 --- a/nvx-rs/src/guest/time/test_mode.rs +++ b/nvx-rs/src/guest/time/test_mode.rs @@ -181,6 +181,16 @@ fn test_packet(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandEr for (start, length) in packet.ranges { let _ = write!(out, " range={start}:{length}"); } + if let Some(network) = packet.network { + let dns_server = network + .dns_server() + .map_or_else(|| "-".to_owned(), |server| server.to_string()); + let _ = write!( + out, + " dns={dns_server} announce={}", + u8::from(network.announces_listeners()) + ); + } let _ = writeln!(out); Ok(()) } diff --git a/nvx-rs/tests/guest_tools.rs b/nvx-rs/tests/guest_tools.rs index ee99e50cc..7b8acc775 100644 --- a/nvx-rs/tests/guest_tools.rs +++ b/nvx-rs/tests/guest_tools.rs @@ -1856,6 +1856,108 @@ fn time_test_mode_checks_packet_and_sample_boundaries() { } } +/// Returns `packet`, a version 5 packet of [`time_packet`], as version 6, with the network byte +/// `network` and, after its entropy, the DNS server record `dns_server`. +#[cfg(target_arch = "x86_64")] +fn time_network_packet(mut packet: Vec, network: u8, dns_server: [u8; 16]) -> Vec { + packet[3] = 6; + packet[7] = network; + packet.extend_from_slice(&dns_server); + packet +} + +#[test] +#[cfg(target_arch = "x86_64")] +fn time_test_mode_decodes_the_network_settings_of_version_6() { + let directory = tempfile::tempdir().unwrap(); + let previous = "ffffffffffffffffffffffffffffffff"; + let base = time_packet(0x04, 0, &[], 1, 0, 0); + let ipv4 = [10, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]; + let ipv6 = "fd00::a00:1" + .parse::() + .unwrap() + .octets(); + for (index, (network, dns_server, expected)) in [ + (0x05, ipv4, " dns=10.0.0.1 announce=1\n"), + (0x02, ipv6, " dns=fd00::a00:1 announce=0\n"), + (0x04, [0; 16], " dns=- announce=1\n"), + (0x00, [0; 16], " dns=- announce=0\n"), + ] + .into_iter() + .enumerate() + { + let packet = time_network_packet(base.clone(), network, dns_server); + let output = run_time_packet( + directory.path(), + &format!("network-{index}"), + &packet, + "0", + previous, + ); + let text = String::from_utf8_lossy(&output.1); + assert_eq!(output.0, Some(0), "case {index}: {text:?}"); + assert!( + text.starts_with("ok flags=4 ") && text.ends_with(expected), + "case {index}: {text:?}" + ); + } + // A version 5 packet carries no network settings, and prints none. + assert!( + run_time_packet(directory.path(), "version-5", &base, "0", previous) + .1 + .ends_with(b" generation_id=000102030405060708090a0b0c0d0e0f\n") + ); + + let mut padded = ipv4; + padded[4] = 1; + let mut version_7 = time_network_packet(base.clone(), 0x05, ipv4); + version_7[3] = 7; + for (index, (packet, error)) in [ + ( + time_network_packet(base.clone(), 0x08, [0; 16]), + "restore packet network byte 0x08 is invalid", + ), + ( + time_network_packet(base.clone(), 0x03, ipv4), + "restore packet network byte 0x03 is invalid", + ), + ( + time_network_packet(base.clone(), 0x01, padded), + "restore packet DNS server is invalid", + ), + ( + time_network_packet(base.clone(), 0x00, ipv4), + "restore packet DNS server is invalid", + ), + ( + time_network_packet(base.clone(), 0x01, [0; 16]), + "restore packet DNS server is invalid", + ), + ( + time_network_packet(base.clone(), 0x02, [0; 16]), + "restore packet DNS server is invalid", + ), + (version_7, "restore packet version is 7"), + ] + .into_iter() + .enumerate() + { + let output = run_time_packet( + directory.path(), + &format!("network-rejected-{index}"), + &packet, + "0", + previous, + ); + assert_eq!(output.0, Some(0)); + assert_eq!( + String::from_utf8_lossy(&output.1), + format!("error G_REPAIR_PACKET {error}\n"), + "case {index}" + ); + } +} + #[test] #[cfg(target_arch = "x86_64")] fn time_test_mode_covers_checks_events_records_and_workers() { diff --git a/nvx-rs/tests/openvmm_vectors.rs b/nvx-rs/tests/openvmm_vectors.rs index 803910b40..d61e30372 100644 --- a/nvx-rs/tests/openvmm_vectors.rs +++ b/nvx-rs/tests/openvmm_vectors.rs @@ -39,7 +39,7 @@ const EXCERPTS: [(&str, &str); 3] = [ /// The versions of `nvx::contract` that are OpenVMM's: for each, the file of the OpenVMM /// submodule that defines OpenVMM's, the start of the line that defines it, up to its value, and /// the version of `nvx::contract`. -const VERSIONS: [(&str, &str, u64); 4] = [ +const VERSIONS: [(&str, &str, u64); 5] = [ ( "openvmm/openvmm_helpers/src/snapshot.rs", "pub const MANIFEST_VERSION: u32 = ", @@ -60,6 +60,11 @@ const VERSIONS: [(&str, &str, u64); 4] = [ "pub const RESTORE_PACKET_VERSION: u8 = ", contract::RESTORE_PACKET_VERSION, ), + ( + "vm/devices/chipset_resources/src/microvm_time.rs", + "pub const RESTORE_PACKET_NETWORK_VERSION: u8 = ", + contract::RESTORE_PACKET_NETWORK_VERSION, + ), ]; #[test] diff --git a/nvx-rs/tests/vectors/README.md b/nvx-rs/tests/vectors/README.md index 7a49fe671..08f8a646d 100644 --- a/nvx-rs/tests/vectors/README.md +++ b/nvx-rs/tests/vectors/README.md @@ -113,7 +113,7 @@ check out the OpenVMM submodule, whose deploy key fork pull requests lack. They read the copies instead, and the ignored test `tests/openvmm_vectors.rs` compares the copies with the submodule's files, and finds each excerpt among the lines of its file. It also checks that the versions of OpenVMM's snapshot -format, saved-state schema, time ABI, and restore packet in `nvx::contract` are +format, saved-state schema, time ABI, and restore packets in `nvx::contract` are the submodule's. The NVX CLI jobs, which check out the submodule, run it, and so does an OpenVMM pin promotion, from the repository's root: diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 5ffc675eb..641294f37 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -5922,7 +5922,14 @@ def test_snapshot_restore_runs_the_time_abi_steps_through_nvx_time(self): snapshot, ) self.assertIn('console_status "NVX-SNAPSHOT-ERROR: $*"', snapshot) - for stage in ("packet", "entropy", "identity", "runtime-hook", "acknowledge"): + for stage in ( + "packet", + "network", + "entropy", + "identity", + "runtime-hook", + "acknowledge", + ): self.assertIn( f'console_status "NVX-POST-RESTORE-STAGE: {stage}"', snapshot, @@ -5957,6 +5964,7 @@ def test_snapshot_console_diagnostics_are_nonfatal_and_ordered(self): markers = [ 'console_status "NVX-POST-RESTORE-STAGE: packet"', + 'console_status "NVX-POST-RESTORE-STAGE: network"', 'console_status "NVX-POST-RESTORE-STAGE: entropy"', 'console_status "NVX-POST-RESTORE-STAGE: identity"', 'console_status "NVX-POST-RESTORE-STAGE: runtime-hook"', @@ -6039,7 +6047,7 @@ def restore(flags: int) -> tuple[str, str, str]: 'nvx_time() { printf "%s\\n" "$*" >>"$calls"; }\n' 'activate_restore_memory() { echo "memory $*"; }\n' f"{functions}\n" - f"post_restore {flags} 0 0 {generation_id}\n" + f"post_restore {flags} 0 0 {generation_id} - 0\n" 'echo "pending=$post_restore_pending"\n' ), text=True, @@ -6108,18 +6116,217 @@ def dispatch_metadata(metadata: str) -> tuple[int, str]: self.assertEqual(dispatch_metadata("0"), (0, "suppressed=true\n")) # nvx-time finished the restore and handed it to the daemon. self.assertEqual( - dispatch_metadata(f"2 4 0 0 {generation_id}"), + dispatch_metadata(f"2 4 0 0 {generation_id} - 0"), (0, "suppressed=false\n"), ) self.assertEqual( - dispatch_metadata(f"1 4 2 0 {generation_id}"), - (0, f"post_restore 4 2 0 {generation_id}\nsuppressed=false\n"), + dispatch_metadata(f"1 4 2 0 {generation_id} 10.0.0.1 1"), + ( + 0, + f"post_restore 4 2 0 {generation_id} 10.0.0.1 1\nsuppressed=false\n", + ), ) self.assertEqual( dispatch_metadata("3"), (1, "fail_closed snapshot capture metadata is invalid\n"), ) + def test_untiered_restore_applies_its_network_settings_before_the_acknowledgement( + self, + ): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + snapshot = ( + Path(__file__).parents[1] / "guest" / "common" / "nvx-snapshot" + ).read_text(encoding="utf-8") + + def function(name: str) -> str: + start = snapshot.index(f"{name}() {{") + return snapshot[start : snapshot.index("\n}\n", start) + 3] + + functions = "".join( + function(name) + for name in ( + "console_status", + "fail_closed", + "cmdline_value", + "apply_restore_network", + "post_restore", + "finish_restore", + ) + ) + for code, stand_in in ( + (">/dev/console", '>>"$console_log"'), + ("/sbin/nvx-exit", "nvx_exit"), + ("/sbin/nvx-time", "nvx_time"), + ("/etc/resolv.conf", '>"$resolver"'), + ("[ -x /sbin/nvx-ingress ]", '[ -x "$ingress" ]'), + ('/sbin/nvx-ingress start "$gateway"', '"$ingress" start "$gateway"'), + ): + self.assertIn(code, functions) + functions = functions.replace(code, stand_in) + generation_id = "0123456789abcdef0123456789abcdef" + + def restore( + dns_server: str, + announce: str, + *, + command_line: str = "virtnet_ip=10.0.0.2 virtnet_gw=10.0.0.1", + ingress: int | None = 0, + resolver: str = "resolv.conf", + ) -> tuple[int, str, str, str, str, str]: + """Run the repair of a restore whose packet names ``dns_server`` + and asks for the announcement of the guest's listeners when + ``announce`` is 1, with a stand-in for nvx-ingress that exits with + ``ingress``, or none, and return the shell's status, stdout, and + stderr, the console, the calls of nvx-ingress and of nvx-time in + order, and the resolver file.""" + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / "cmdline").write_text(command_line + "\n", encoding="ascii") + if ingress is not None: + program = root / "nvx-ingress" + program.write_text( + "#!/bin/sh\n" + f'printf "nvx-ingress %s\\n" "$*" >>"{root}/nvx-time"\n' + f"exit {ingress}\n", + encoding="ascii", + ) + program.chmod(0o755) + result = subprocess.run( + [shell, "-s", "--", temporary, resolver], + input=( + "set -eu\n" + 'console_log="$1/console"\n' + 'calls="$1/nvx-time"\n' + 'cmdline_file="$1/cmdline"\n' + 'resolver="$1/$2"\n' + 'ingress="$1/nvx-ingress"\n' + ': >"$console_log"\n' + ': >"$calls"\n' + "snapshot_tier=legacy\n" + "PACKET_MEMORY_TARGET=2\n" + "PACKET_ACK_REQUIRED=4\n" + "restore_new_cpus=none\n" + "post_restore_pending=false\n" + 'nvx_exit() { echo "exit $1"; }\n' + # The acknowledgement records what the resolver + # names by then. + "nvx_time() {\n" + ' printf "%s resolver=%s\\n" "$*" \\\n' + ' "$(cat "$resolver" 2>/dev/null)" >>"$calls"\n' + "}\n" + f"{functions}\n" + f"post_restore 4 0 0 {generation_id} {dns_server} {announce}\n" + 'echo "pending=$post_restore_pending"\n' + ), + text=True, + capture_output=True, + timeout=5, + check=False, + ) + return ( + result.returncode, + result.stdout, + result.stderr, + (root / "console").read_text(encoding="ascii"), + (root / "nvx-time").read_text(encoding="ascii"), + ( + (root / "resolv.conf").read_text(encoding="ascii") + if (root / "resolv.conf").exists() + else "" + ), + ) + + stages = ( + "NVX-POST-RESTORE-STAGE: packet\n" + "NVX-POST-RESTORE-STAGE: network\n" + "NVX-POST-RESTORE-STAGE: acknowledge\n" + ) + # The resolver names the DNS server, and nvx-ingress starts the + # announcer to the IPv4 gateway, before the acknowledgement. + self.assertEqual( + restore("10.0.0.1", "1"), + ( + 0, + "NVX-POST-RESTORE-OK: tier=legacy\npending=false\n", + "", + stages, + "nvx-ingress start 10.0.0.1\n" + "restore-finish --ack --new-cpus none resolver=nameserver 10.0.0.1\n", + "nameserver 10.0.0.1\n", + ), + ) + # Either setting alone. + self.assertEqual( + restore("fd00::a00:1", "0")[3:], + ( + stages, + "restore-finish --ack --new-cpus none " + "resolver=nameserver fd00::a00:1\n", + "nameserver fd00::a00:1\n", + ), + ) + self.assertEqual( + restore("-", "1")[3:], + ( + stages, + "nvx-ingress start 10.0.0.1\n" + "restore-finish --ack --new-cpus none resolver=\n", + "", + ), + ) + # Settings that ask for nothing leave a plain restore quiet. + self.assertEqual( + restore("-", "0"), + ( + 0, + "pending=false\n", + "", + "", + "restore-finish --ack --new-cpus none resolver=\n", + "", + ), + ) + + # A failure fails the restore, which is never acknowledged. An + # announcer that does not start, as when no privileged source port is + # free for its socket, is one. + failures: tuple[tuple[tuple[str, str, dict[str, Any]], str, str], ...] = ( + ( + ("-", "1", {"command_line": "virtnet_ip=10.0.0.2"}), + "listener announcements need an IPv4 gateway", + "", + ), + ( + ("-", "1", {"ingress": None}), + "listener announcements need /sbin/nvx-ingress", + "", + ), + ( + ("10.0.0.1", "1", {"ingress": 1}), + "failed to start nvx-ingress", + "nvx-ingress start 10.0.0.1\n", + ), + ( + ("10.0.0.1", "0", {"resolver": "missing/resolv.conf"}), + "failed to name the DNS server in /etc/resolv.conf", + "", + ), + (("-", "2", {}), "restore packet metadata is invalid", ""), + (("10.0.0.z", "0", {}), "restore packet metadata is invalid", ""), + ) + for (dns_server, announce, options), error, started in failures: + status, stdout, stderr, _, calls, _ = restore( + dns_server, announce, **options + ) + self.assertEqual(status, 1, (error, stdout, stderr)) + self.assertIn("exit 1\n", stdout) + self.assertIn(f"nvx-snapshot: {error}; terminating the VM\n", stderr) + self.assertEqual(calls, started) + def _run_host_share_write_back( self, mountinfo: str, From 481e6ce6d8b5cfd5d48613d8cb07bd612cdec5e2 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 19:18:47 -0700 Subject: [PATCH 5/6] nvx-rs: let run install a new network policy at a restore OpenVMM lets a restore install a new network policy on a NIC template, a snapshot whose NIC was captured with the complete deny-all policy, with --restore-network-policy, for a platform snapshot and now for one without a tier, whose guest then applies the policy's DNS server and listener announcements before the host releases its input. nvx-rs run could not ask for it: every restore passed the network options to OpenVMM as the policy that the snapshot was captured with, so run could restore a NIC template only under its deny-all. run --restore-network-policy, which needs --restore-snapshot, passes the option to OpenVMM after the restore's other options, so the restore installs the policy of its network options, private-network ingress, explicit host loopback, and forwards included. run leaves the checks of the snapshot and the policy to OpenVMM, as it does for every other restore, rather than read the snapshot itself. The option stays explicit because OpenVMM refuses it for a snapshot that is no NIC template, so run cannot pass it to every restore. nvx-fake-openvmm accepts the option at a restore and refuses it at a cold boot, and, like OpenVMM, refuses a restore without it that asks for private-network ingress or explicit host-loopback access, with OpenVMM's messages. The usage and run guides and the nvx-rs README describe the option. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/run.md | 75 ++++++++++++++++++++++------- doc/usage.md | 10 ++-- nvx-rs/README.md | 14 +++++- nvx-rs/src/host/cli/run.rs | 21 ++++++++- nvx-rs/src/testing/fake_openvmm.rs | 76 +++++++++++++++++++++++++++++- nvx-rs/tests/cli/dry_run.rs | 31 ++++++++++++ nvx-rs/tests/cli/one_shot.rs | 43 +++++++++++++++++ nvx-rs/tests/cli/openvmm.rs | 6 ++- nvx-rs/tests/cli/validation.rs | 13 +++++ nvx-rs/tests/fake_openvmm.rs | 21 +++++++++ 10 files changed, 283 insertions(+), 27 deletions(-) diff --git a/doc/run.md b/doc/run.md index 40ee18a9f..1385f36ff 100644 --- a/doc/run.md +++ b/doc/run.md @@ -442,7 +442,7 @@ IPv6-only, and a UDP socket on `::` unless the guest sets admit them to OpenVMM, for example an inbound rule for `openvmm.exe` on Windows. Snapshot capture rejects ingress `allow`, because the published listeners are live host attachments, and so does a restore, unless it -installs a new policy on a [NIC template](#platform-nic-templates). +installs a new policy on a [NIC template](#networked-snapshots). ### Host-loopback access @@ -493,7 +493,7 @@ OpenVMM presents host peers at addresses inside the guest's network, and a `/30` holds only the gateway and the guest, so OpenVMM rejects explicit `allow` on a `/30` before it starts, as the NVX Rust API and `nvx-rs sandbox` do, and also at a restore that installs a new policy on a -[NIC template](#platform-nic-templates) captured on one. +[NIC template](#networked-snapshots) captured on one. Repeat `--host-loopback-forward tcp:HOST_PORT:GUEST_PORT` or its UDP form, at most 64 times, to publish further localhost ports at guest ports of your @@ -507,7 +507,7 @@ egress policy, but the egress policy filters UDP replies, so forwarded UDP ports need an egress policy that lets them out. Snapshot capture rejects explicit `--host-loopback allow`, because its listeners are live host attachments, and so does a restore, unless it installs a new policy on a -[NIC template](#platform-nic-templates). +[NIC template](#networked-snapshots). ### Networked snapshots @@ -528,14 +528,54 @@ nvx-rs/target/release/nvx-rs run \ The restore fails unless its policy matches the captured one: the default egress, the same rules, whether `--network-egress-allow` and `--network-egress-deny` give them or a policy file compiles to them, -`--host-loopback`, and `--network-proxy`. A restore therefore cannot change -the policy, and, like a capture, it rejects `--network-ingress allow`, explicit -`--host-loopback allow`, and `--host-loopback-forward`, whose published ports -are live host attachments. The restored guest keeps the network configuration -that its boot applied, such as its DNS server; flows from before the capture -do not survive the restore, and new traffic opens fresh ones. A snapshot -captured without `--net` takes no network option at restore. Only a -[platform NIC template](#platform-nic-templates) takes a new policy at restore. +`--host-loopback`, and `--network-proxy`. Such a restore therefore cannot +change the policy, and, like a capture, it rejects `--network-ingress allow`, +explicit `--host-loopback allow`, and `--host-loopback-forward`, whose +published ports are live host attachments. The restored guest keeps the +network configuration that its boot applied, such as its DNS server; flows +from before the capture do not survive the restore, and new traffic opens +fresh ones. A snapshot captured without `--net` takes no network option at +restore. + +A snapshot whose NIC was captured with a complete deny-all policy, +`--network-egress deny` and `--host-loopback deny` without egress rules, +`--network-proxy`, or ingress, is a NIC template, whose restore may install a +policy of its own instead: pass `--restore-network-policy` with the network +options of the new policy, which may include `--network-ingress allow`, +explicit `--host-loopback allow`, and forwards, for example: + +```bash +nvx-rs/target/release/nvx-rs run \ + --restore-snapshot /var/lib/nvx/template \ + --restore-network-policy \ + --network-profile portable \ + --network-egress deny \ + --network-egress-allow 10.0.0.1/32:udp:53 \ + --network-egress-allow 0.0.0.0/0:tcp:443 \ + --host-loopback allow \ + --host-loopback-forward tcp:18080:8080 +``` + +OpenVMM validates the new policy as it validates a cold boot's, binds it to +the NIC's captured addresses, which the restored guest keeps, and installs it, +with its forwards, before any vCPU runs; a forward whose host port is busy +fails the restore. A restore without connectivity passes the template's own +deny-all. The template's guest booted under the deny-all, which names no DNS +server and starts no `nvx-ingress`, so OpenVMM hands the guest the DNS server +of the new policy, and whether to announce its listeners, in version 6 of the +restore packet (see +[Restore Packet](design/5.5-restore-protocol.md#555-restore-packet)). Before +the host releases its input, the guest's repair names that server in +`/etc/resolv.conf` and starts `nvx-ingress`, as the guest's init does at a +cold boot under that policy; a failure of either fails the restore, and so +does a guest that reads restore packet version 5 alone. Such a snapshot may +hold a workload that has already run, but under the deny-all that workload had +no connectivity, so nothing in the snapshot depends on the policy, and the +restore grants it connectivity that it never had. OpenVMM rejects the option +for a cold boot, for a snapshot without a NIC or whose NIC was captured with +any other policy, and for a `workload-start` or `instance-checkpoint` +snapshot, and `nvx-rs run` rejects it without `--restore-snapshot`. A +[platform NIC template](#platform-nic-templates) takes the option too. ### Platform NIC templates @@ -565,12 +605,13 @@ to the NIC's captured addresses, which the clone keeps, and installs it, with its forwards, before any vCPU runs; a forward whose host port is busy fails the restore. A clone without connectivity passes the template's own deny-all. The guest keeps its captured kernel command line, which under a deny-all -policy names no DNS server and starts no `nvx-ingress`, so it learns its DNS -server and whether to report its listeners from its host after the restore. -OpenVMM rejects the option for a cold boot, for a snapshot of another tier or -without a NIC, and for a NIC captured with any other policy, and every other -[networked restore](#networked-snapshots) requires the captured policy. -`nvx-rs run` does not pass the option. +policy names no DNS server and starts no `nvx-ingress`, so a clone learns its +DNS server and whether to report its listeners from its activation after the +restore, and its restore packet stays version 5. OpenVMM rejects the option +for a cold boot, for a `workload-start` or `instance-checkpoint` snapshot or +one without a NIC, and for a NIC captured with any other policy, and every +restore without it requires the captured policy, as +[Networked snapshots](#networked-snapshots) describes. ### Platform filesystem exports diff --git a/doc/usage.md b/doc/usage.md index b31e729c4..d97ea0a1a 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -150,6 +150,7 @@ nvx-rs/target/release/nvx-rs run [--restore-processors {1,2,4,8}] [--restore-memory-mib MIB] [--restore-ready-path PATH] + [--restore-network-policy] [--dry-run] ``` @@ -170,19 +171,20 @@ nvx-rs/target/release/nvx-rs run | `--net IPV4/PREFIX` | none | Enable virtio-net with the static guest IPv4 address and prefix. The guest also gets the IPv6 address that embeds it in `fd00::/96`, such as `fd00::a00:2/120` for `10.0.0.2/24`. A restore takes the addresses from the snapshot and rejects this option. | | `--network-profile {portable}` | none | Select the required cross-platform network behavior contract; must be specified with `--net` on a fresh boot, and without it to restore a snapshot captured with `--net`. | | `--network-egress {allow,deny}` | `allow` | Set the default guest egress policy. | -| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture and restore reject `allow`. | +| `--network-ingress {allow,deny}` | `deny` | Set the default host ingress policy. `allow` publishes every TCP and UDP port on which the guest listens to clients on the host's private networks, and to the host itself only with `--host-loopback allow`; see [Private-network ingress](run.md#private-network-ingress). Snapshot capture rejects `allow`, and so does a restore, unless `--restore-network-policy` installs a new policy. | | `--network-egress-allow CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Allow matching guest egress; repeat to add rules. `CIDR` is an IPv4 or IPv6 address or network, and a rule matches only its own family. `PROTOCOL` is `tcp`, `udp`, or `icmp` and matches every port when `PORT` is omitted; only `tcp` and `udp` take a `PORT` or an inclusive `PORT-PORT` range, such as `8000-8010`. | | `--network-egress-deny CIDR[:PROTOCOL[:PORT[-PORT]]]` | none | Deny matching guest egress; repeat to add rules. Deny rules take precedence. `PROTOCOL` and `PORT` select traffic as for `--network-egress-allow`. | | `--network-egress-policy-file PATH` | none | Load bounded IPv4 and IPv6 ranges and rule-local CIDR exclusions from JSON. Requires explicit `--network-egress`; cannot be mixed with explicit allow/deny rule flags. | -| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture and restore reject `allow`. | +| `--host-loopback {allow,deny}` | existing mapping | Control access between the guest and host loopback services. Explicit `allow` is bidirectional: the guest reaches host loopback through its gateway, and every TCP and UDP port on which the guest listens is published on host loopback for the host alone, or with `--network-ingress allow` admits the host as well; it needs a `--net` prefix of `/29` or shorter, which OpenVMM checks; see [Host-loopback access](run.md#host-loopback-access). Snapshot capture rejects `allow`, and so does a restore, unless `--restore-network-policy` installs a new policy. | | `--network-proxy IPV4:TCP-PORT` | none | Allow one explicit host TCP proxy endpoint. | -| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Snapshot capture and restore reject forwards. | +| `--host-loopback-forward PROTOCOL:HOST_PORT:GUEST_PORT` | none | Publish one TCP or UDP localhost port to the guest; repeat to add forwards. Snapshot capture rejects forwards, and so does a restore, unless `--restore-network-policy` installs a new policy. | | `--outcome-report PATH` | none | Write a bounded local JSON outcome report. | | `--cmdline TEXT` | empty | Append kernel parameters; `nvx_*` and `tsc=` tokens are reserved. | -| `--restore-snapshot PATH` | none | Restore the immutable machine contract and saved state from a snapshot directory. A snapshot captured with `--net` needs `--network-profile portable` and the network policy options of its capture, which the restore cannot change; see [Networked snapshots](run.md#networked-snapshots). | +| `--restore-snapshot PATH` | none | Restore the immutable machine contract and saved state from a snapshot directory. A snapshot captured with `--net` needs `--network-profile portable` and the network policy options of its capture, unless `--restore-network-policy` installs a new policy; see [Networked snapshots](run.md#networked-snapshots). | | `--restore-processors {1,2,4,8}` | none | Bring this contiguous processor prefix online before restore readiness. Requires an opt-in microVM snapshot and cannot exceed `--processors` capacity. | | `--restore-memory-mib MIB` | none | Select the 128 MiB-aligned RAM target for an expansion-capable snapshot restore, in the range of `--memory-mib`. | | `--restore-ready-path PATH` | none | Publish one restore-readiness event to an existing Unix socket or Windows named pipe. | +| `--restore-network-policy` | off | Install the policy of the network options, which may include `--network-ingress allow`, explicit `--host-loopback allow`, and forwards, in place of the complete deny-all policy, `--network-egress deny` and `--host-loopback deny` without egress rules, a proxy, or ingress, that the snapshot's NIC was captured with. The guest applies the new policy's DNS server and listener announcements before the host releases its input; see [Networked snapshots](run.md#networked-snapshots). Requires `--restore-snapshot`. | | `--dry-run` | off | Print the generated OpenVMM command without running it. | `nvx-rs run` checks its options, prints the OpenVMM command after `>> `, and diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 6210342b3..37672922a 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2045,7 +2045,11 @@ status. the addressing of its NIC from the snapshot, as OpenVMM does, so it refuses `--net`, and passes `--network-profile` alone, which OpenVMM requires, with the network policy that the snapshot was captured with, to restore a - snapshot that has a NIC. + snapshot that has a NIC, or, with `--restore-network-policy`, which needs + `--restore-snapshot`, with a policy of its own, which OpenVMM installs on a + NIC template in place of the complete deny-all policy that the template was + captured with, and whose DNS server and listener announcements the guest + then applies; OpenVMM checks the snapshot and the new policy. - One directory `--mount` is OpenVMM's `--mount`. Several, or a shared file, are the children of an aggregate at `/run/nvx/shares`, whose guest targets must be absolute, must not overlap, and must survive the kernel command line, @@ -2321,6 +2325,10 @@ guest agents of NVX advertise bits 0, 4, and 6 in sandbox mode. OpenVMM requires to restore a snapshot that has a NIC. `nvx.py run` took the two only together, so it could not restore such a snapshot: OpenVMM refused its `--net`, or required the `--network-profile` that it omitted. +- `run --restore-network-policy`, which `nvx.py run` lacked, passes OpenVMM's + `--restore-network-policy`, so that a restore of a NIC template installs the + policy of its network options; see + [Networked snapshots](../doc/run.md#networked-snapshots). ### Test hooks @@ -2334,7 +2342,9 @@ boot, or the restore of a snapshot directory, prints that it booted or restored the microVM, writes an outcome report if asked, and exits with status 0, and a restore whose snapshot directory is missing fails with status 1. Like OpenVMM, it refuses with status 2 a restore with `--net`, whose NIC keeps the addressing -that its snapshot saved, and a cold boot with `--network-profile` alone. Kernel +that its snapshot saved, a cold boot with `--network-profile` alone or with +`--restore-network-policy`, and a restore with `--network-ingress allow` or +explicit `--host-loopback allow` without `--restore-network-policy`. Kernel command-line tokens make it fail: `fake_exit_on_start=STATUS`, `fake_exit_status=STATUS`, `fake_guest_features=MASK`, and `fake_legacy_guest=1`, among others; see `nvx::testing::fake_openvmm`. Like diff --git a/nvx-rs/src/host/cli/run.rs b/nvx-rs/src/host/cli/run.rs index 71bbcee8c..a1400a858 100644 --- a/nvx-rs/src/host/cli/run.rs +++ b/nvx-rs/src/host/cli/run.rs @@ -10,7 +10,11 @@ //! A cold boot takes `--net` and `--network-profile` together. A restore takes the addressing of //! its NIC from the snapshot, as OpenVMM does, so it refuses `--net`, and passes //! `--network-profile` alone, which OpenVMM requires, with the network policy that the snapshot -//! was captured with, to restore a snapshot that has a NIC. +//! was captured with, to restore a snapshot that has a NIC. With `--restore-network-policy`, which +//! it passes on, the restore installs the policy of its network options instead, ingress, +//! explicit host loopback, and forwards included, which OpenVMM admits for a NIC template, a +//! snapshot whose NIC was captured with the complete deny-all policy, and the guest applies the +//! new policy's DNS server and listener announcements before the host releases its input. //! //! Its shares differ from those of `sandbox`. `--mount-deny`, `--mount-allow`, and `--mount-write` //! name host paths, which OpenVMM attributes to the share that holds them, and which must be @@ -111,6 +115,10 @@ options: --restore-processors {1,2,4,8} --restore-memory-mib RESTORE_MEMORY_MIB --restore-ready-path RESTORE_READY_PATH + --restore-network-policy + install the network options' policy in place of the + complete deny-all policy of a snapshot's NIC + (doc/run.md) --dry-run", }; @@ -127,7 +135,7 @@ const MACHINES: [&str; 1] = ["microvm"]; const PROCESSORS: [&str; 4] = ["1", "2", "4", "8"]; /// The options of `nvx-rs run`, which parse with the defaults of the former `nvx.py run`. -pub(super) const OPTIONS: [Spec; 29] = [ +pub(super) const OPTIONS: [Spec; 30] = [ Spec::one("--guest"), Spec::one("--hypervisor"), Spec::one("--machine"), @@ -156,6 +164,7 @@ pub(super) const OPTIONS: [Spec; 29] = [ Spec::one("--restore-processors"), Spec::one("--restore-memory-mib"), Spec::one("--restore-ready-path"), + Spec::flag("--restore-network-policy"), Spec::flag("--dry-run"), ]; @@ -196,6 +205,8 @@ pub(crate) struct Options { pub(crate) restore_memory_mib: Option, /// `--restore-ready-path`, as given. pub(crate) restore_ready_path: Option, + /// `--restore-network-policy`. + pub(crate) restore_network_policy: bool, /// `--dry-run`. pub(crate) dry_run: bool, } @@ -220,6 +231,7 @@ impl Default for Options { restore_processors: None, restore_memory_mib: None, restore_ready_path: None, + restore_network_policy: false, dry_run: false, } } @@ -269,6 +281,7 @@ impl Receiver for Options { "--restore-processors" => self.restore_processors = Some(processors(&text())?), "--restore-memory-mib" => self.restore_memory_mib = Some(memory(&text())?), "--restore-ready-path" => self.restore_ready_path = Some(text()), + "--restore-network-policy" => self.restore_network_policy = true, "--dry-run" => self.dry_run = true, _ => return self.network.take(name, &text()), } @@ -311,6 +324,7 @@ fn check(options: &Options) -> Result { (options.restore_ready_path.is_some(), "--restore-ready-path"), (options.restore_processors.is_some(), "--restore-processors"), (options.restore_memory_mib.is_some(), "--restore-memory-mib"), + (options.restore_network_policy, "--restore-network-policy"), ] { if given && !restore { return refused(format!("{option} requires --restore-snapshot")); @@ -383,6 +397,9 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result { if let Some(ready) = &options.restore_ready_path { push(&mut command, "--restore-ready-path", path_text(ready)); } + if options.restore_network_policy { + command.push("--restore-network-policy".to_owned()); + } } else { let kernel = dir.require_kernel()?; let initrd = dir.require_initramfs(options.guest)?; diff --git a/nvx-rs/src/testing/fake_openvmm.rs b/nvx-rs/src/testing/fake_openvmm.rs index 52790eb19..39bdaacc5 100644 --- a/nvx-rs/src/testing/fake_openvmm.rs +++ b/nvx-rs/src/testing/fake_openvmm.rs @@ -1071,7 +1071,10 @@ fn parse(program: &str, arguments: &[String]) -> Result { let mut iterator = arguments.iter(); while let Some(argument) = iterator.next() { let name = argument.as_str(); - if name == "--single-process" || name == "--microvm-control-auth-stdin" { + if name == "--single-process" + || name == "--microvm-control-auth-stdin" + || (run && name == "--restore-network-policy") + { flags.insert(name); } else if VALUED.contains(&name) || ((run || capture) && RUN_VALUED.contains(&name)) @@ -1383,6 +1386,28 @@ fn parse(program: &str, arguments: &[String]) -> Result { if !net && restore.is_none() && values.contains_key("--network-profile") { return Err("--network-profile portable requires --net or --restore-snapshot".to_owned()); } + // A restore installs a policy of its own, which may publish the guest's listeners and forward + // host ports, only with `--restore-network-policy`, which OpenVMM admits for a NIC template; + // every other restore keeps the snapshot's policy, which never has them. A forward requires + // explicit host loopback, which such a restore refuses first, as OpenVMM does. + let replaces_policy = flags.contains("--restore-network-policy"); + if replaces_policy && restore.is_none() { + return Err("--restore-network-policy requires --restore-snapshot".to_owned()); + } + if restore.is_some() && !replaces_policy { + let allow = |name: &str| values.get(name).map(Vec::as_slice) == Some(&["allow"][..]); + for (refused, option) in [ + (allow("--host-loopback"), "--host-loopback allow"), + (allow("--network-ingress"), "--network-ingress allow"), + ] { + if refused { + return Err(format!( + "microVM snapshots do not support {option}, whose published guest listeners \ + are live host attachments, except at a restore with --restore-network-policy" + )); + } + } + } let knob = |name: &str| { tokens .iter() @@ -2814,6 +2839,27 @@ mod tests { assert_eq!(options.network.mode, "rules"); assert_eq!(options.network.allow, 1); assert_eq!(options.network.host_loopback, "deny"); + // A restore that replaces the policy of a NIC template may publish the guest's listeners + // and forward host ports. + let replaced = run_with(&[ + "--restore-snapshot", + "{dir}/snapshot", + "--restore-network-policy", + "--network-profile", + "portable", + "--network-egress", + "allow", + "--network-ingress", + "allow", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:3000:8080", + ]); + let (_temporary, options) = parse_run(&replaced); + let options = options.unwrap(); + assert_eq!(options.network.mode, "allow-all"); + assert_eq!(options.network.host_loopback, "allow"); } #[test] @@ -2862,6 +2908,34 @@ mod tests { boot(&["--network-profile", "portable"]), "--network-profile portable requires --net or --restore-snapshot", ), + ( + boot(&["--restore-network-policy"]), + "--restore-network-policy requires --restore-snapshot", + ), + ( + restore(&[ + "--network-profile", + "portable", + "--network-ingress", + "allow", + ]), + "microVM snapshots do not support --network-ingress allow, whose published guest \ + listeners are live host attachments, except at a restore with \ + --restore-network-policy", + ), + ( + restore(&[ + "--network-profile", + "portable", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:3000:8080", + ]), + "microVM snapshots do not support --host-loopback allow, whose published guest \ + listeners are live host attachments, except at a restore with \ + --restore-network-policy", + ), ( boot(&["--restore-processors", "2"]), "--restore-processors requires --restore-snapshot", diff --git a/nvx-rs/tests/cli/dry_run.rs b/nvx-rs/tests/cli/dry_run.rs index 55755cc5a..d70f5889e 100644 --- a/nvx-rs/tests/cli/dry_run.rs +++ b/nvx-rs/tests/cli/dry_run.rs @@ -997,6 +997,37 @@ fn run_dry_runs() -> Vec { .with("--network-egress-deny", COMPILED_DENY), ) .with_file("policy.json", POLICY), + // With `--restore-network-policy`, a restore installs the policy of its network options + // in place of a NIC template's deny-all, ingress, explicit host loopback, and forwards + // included, which OpenVMM admits only with the option. + dry_run( + "run: restore that replaces the network policy", + run(&[ + "--restore-snapshot", + "{root/snapshot}", + "--restore-network-policy", + "--network-profile", + "portable", + "--network-egress", + "deny", + "--network-egress-allow", + "10.0.0.1/32:udp:53", + "--network-ingress", + "allow", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:8080:80", + ]), + restore() + .flag("--restore-network-policy") + .with("--network-profile", ["portable"]) + .with("--network-egress", ["deny"]) + .with("--network-egress-allow", ["10.0.0.1/32:udp:53"]) + .with("--network-ingress", ["allow"]) + .with("--host-loopback", ["allow"]) + .with("--host-loopback-forward", ["tcp:8080:80"]), + ), dry_run( "run: host loopback forward", run(&["--host-loopback-forward", "tcp:8080:80"]), diff --git a/nvx-rs/tests/cli/one_shot.rs b/nvx-rs/tests/cli/one_shot.rs index 5ef3657b7..e01415258 100644 --- a/nvx-rs/tests/cli/one_shot.rs +++ b/nvx-rs/tests/cli/one_shot.rs @@ -498,6 +498,49 @@ fn run_launches() -> Vec { status: 0, network: ("applied", "rules", 1, 0, "deny"), }), + // A restore that replaces the policy of a NIC template may publish the guest's listeners + // and forward host ports, which OpenVMM refuses at every other restore. + launch( + "run: restore that replaces the network policy", + run(&[ + "--restore-snapshot", + "{root/snapshot}", + "--restore-network-policy", + "--network-profile", + "portable", + "--network-egress", + "deny", + "--network-egress-allow", + "10.0.0.1/32:udp:53", + "--network-ingress", + "allow", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:8080:80", + "--outcome-report", + "{root/reports/openvmm-run.json}", + ]), + restore() + .flag("--restore-network-policy") + .with("--network-profile", ["portable"]) + .with("--network-egress", ["deny"]) + .with("--network-egress-allow", ["10.0.0.1/32:udp:53"]) + .with("--network-ingress", ["allow"]) + .with("--host-loopback", ["allow"]) + .with("--host-loopback-forward", ["tcp:8080:80"]) + .with("--microvm-report", ["{root/reports/openvmm-run.json}"]), + 0, + restored, + ) + .with_file(SNAPSHOT) + .recorded_in(Some("fake-openvmm-reports.json")) + .reporting(Report { + path: "reports/openvmm-run.json", + category: "success", + status: 0, + network: ("applied", "rules", 1, 0, "allow"), + }), launch( "run: network outcome report", run(&[ diff --git a/nvx-rs/tests/cli/openvmm.rs b/nvx-rs/tests/cli/openvmm.rs index cd0ffb9b4..8355cd132 100644 --- a/nvx-rs/tests/cli/openvmm.rs +++ b/nvx-rs/tests/cli/openvmm.rs @@ -11,7 +11,11 @@ use serde_json::{Value, json}; use crate::harness::{HYPERVISOR, Root, is_hex}; /// The options of OpenVMM that take no value. -const FLAGS: [&str; 2] = ["--single-process", "--microvm-control-auth-stdin"]; +const FLAGS: [&str; 3] = [ + "--single-process", + "--microvm-control-auth-stdin", + "--restore-network-policy", +]; /// The kernel command-line tokens with which the sandbox profile selects itself and attaches the /// distro layer of the tests on the first sandbox block device and the scratch disk on the diff --git a/nvx-rs/tests/cli/validation.rs b/nvx-rs/tests/cli/validation.rs index 3cdcd95d6..179890d8d 100644 --- a/nvx-rs/tests/cli/validation.rs +++ b/nvx-rs/tests/cli/validation.rs @@ -786,6 +786,19 @@ fn run_refusals() -> Vec { run_dry(&["--restore-ready-path", "{root/ready.sock}"]), &["--restore-ready-path requires --restore-snapshot"], ), + // A cold boot installs the policy of its options anyway. + refusal( + "run: network policy replacement without snapshot", + "unsupported", + run_dry(&[ + "--restore-network-policy", + "--net", + "10.0.0.2/24", + "--network-profile", + "portable", + ]), + &["--restore-network-policy requires --restore-snapshot"], + ), refusal( "run: restore processors without snapshot", "unsupported", diff --git a/nvx-rs/tests/fake_openvmm.rs b/nvx-rs/tests/fake_openvmm.rs index 0731e409d..86870ece8 100644 --- a/nvx-rs/tests/fake_openvmm.rs +++ b/nvx-rs/tests/fake_openvmm.rs @@ -670,6 +670,27 @@ fn runs_boot_or_restore_a_microvm_on_the_terminal() { text(&output.stderr), "nvx-fake-openvmm: restore takes microVM network addressing from saved state\n" ); + + // Only a restore that replaces the policy of a NIC template may publish the guest's + // listeners. + let ingress = ["--network-ingress", "allow"]; + let output = run_microvm(root, &[&networked[..], &ingress].concat()); + assert_eq!(output.status.code(), Some(2)); + assert_eq!( + text(&output.stderr), + "nvx-fake-openvmm: microVM snapshots do not support --network-ingress allow, whose \ + published guest listeners are live host attachments, except at a restore with \ + --restore-network-policy\n" + ); + let output = run_microvm( + root, + &[&networked[..], &ingress, &["--restore-network-policy"]].concat(), + ); + assert_eq!(output.status.code(), Some(0), "{}", text(&output.stderr)); + assert_eq!( + text(&output.stdout), + "nvx-fake-openvmm: restored the microVM\n" + ); } #[test] From 0ac696f3be76ae033d2bb77f01fe21e87a540c22 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 31 Oct 2026 07:38:46 -0700 Subject: [PATCH 6/6] scripts: test the untiered restore of a new network policy Add the untiered-network-policy microVM scenario, which covers the restore of a NIC template without a tier, a snapshot without sandbox blocks, as the non-sandbox mode captures one, whose NIC was captured with the complete deny-all policy, with --restore-network-policy. It restores the template four times: under the template's own deny-all, under egress rules that admit DNS over IPv4 and one host service, with explicit host loopback and a forward under egress rules that admit DNS over IPv6 alone, and with private-network ingress and allowed egress. The template's guest learned no DNS server and runs no announcer of its listeners, so each restore's repair applies those of the restore's policy, which restore packet version 6 carries, before it acknowledges the restore. The guest's resolver must name exactly the policy's DNS server, and nvx-ingress must run exactly when the policy publishes the guest's listeners. Each restore must reach through its gateway exactly the host services that its policy admits, which receive no other request. The host must reach the guest's server through the forward, and through the port that the policy publishes: on host loopback alone, which no other host address reaches, or, for private-network ingress, through a host address other than loopback, since a probe that binds every host address also fails over a loopback-only listener. It must find the server on no host loopback port without a publication. OpenVMM must refuse, before the guest runs, a restore that keeps the template's policy but passes another, and one that passes private-network ingress without the option, and no restore may change the template. The template's NIC is on the directional network, 192.0.2.0/24, a documentation range that no host address shares: an ingress listener refuses peers inside the guest's subnets, so a host address there, as on hosts whose network is 10.0.0.0/24, would not reach the port that private-network ingress publishes. The scenario joins the default suite, so plan the shards again, with its time in local runs on virtual-machine hosts of each backend, 14.0 s on KVM, 14.1 s on MSHV, and 18.8 s on WHP, until CI's reports measure it. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 22 + scripts/nvx_tools/microvm_shards.json | 32 +- .../untiered-network-policy.sh.in | 88 ++ scripts/nvx_tools/microvm_tests.py | 496 ++++++++++++ scripts/test_microvm_tests.py | 765 ++++++++++++++++++ 5 files changed, 1390 insertions(+), 13 deletions(-) create mode 100644 scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in diff --git a/doc/ci.md b/doc/ci.md index 630438db9..c0f0d3f34 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -243,6 +243,28 @@ forward whose host port another socket holds, the option on a platform snapshot captured with plain deny-all egress, which leaves host loopback allowed, and the option on a cold boot, and no restore may change the template. +The `untiered-network-policy` scenario covers a NIC template without a tier, a +snapshot without sandbox blocks, as the non-sandbox mode captures one, whose +NIC was captured with the complete deny-all policy. It restores the template +with `--restore-network-policy` four times: under the template's own deny-all, +under egress rules that admit DNS over IPv4 and one host service, with +explicit host loopback and a forward under egress rules that admit DNS over +IPv6 alone, and with private-network ingress and allowed egress. The +template's guest learned no DNS server and runs no announcer of its listeners, +so each restore's repair applies those of the restore's policy, which restore +packet version 6 carries, before it acknowledges the restore: the guest's +resolver must name exactly the policy's DNS server, and no gateway without +one, and the announcer must run exactly when the policy publishes the guest's +listeners. Each restore must reach through its gateway exactly the +host-loopback services that its policy admits, which must receive no other +request, and the host must reach the guest's server through the forward and +through the port that the policy publishes: on host loopback alone, which no +other host address reaches, or to the host's private networks, through a host +address other than loopback. It must find the server on no host loopback port +without a publication. Before the guest runs, OpenVMM must refuse a restore +that keeps the template's policy but passes another, and one that passes +private-network ingress without the option, and no restore may change the +template. The `platform-filesystem` scenario covers a share that a platform restore attaches to the filesystem slot that the capture left dormant, as a clone's export does. It captures a platform snapshot without `--mount` and restores it diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index 23c0968d6..4a82c1875 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -64,21 +64,22 @@ "structured-outcome": 0.6, "time-abi-conformance": 0.7, "time-host-correction": 14.0, + "untiered-network-policy": 14.0, "virtio-net": 3.8, "workload-identity": 0.7 }, "shards": [ { - "seconds": 258.5, + "seconds": 265.4, "scenarios": [ "command-line-limit", "console-snapshot", "directional-network-policy", + "endpoint-policy-snapshot", "filesystem-comma-paths", "filesystem-snapshot", "managed-lifecycle", "network-snapshot", - "platform-aggregate-export", "platform-restore-processors", "restore-downtime", "restore-memory", @@ -91,12 +92,11 @@ ] }, { - "seconds": 258.3, + "seconds": 265.4, "scenarios": [ "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", - "endpoint-policy-snapshot", "filesystem-owner", "filesystem-policy", "filesystem-shares", @@ -108,6 +108,7 @@ "l3-l4-egress-port-ranges", "l3-l4-egress-protocols", "lifecycle", + "platform-aggregate-export", "platform-filesystem", "platform-network-policy", "platform-zero-fill", @@ -116,6 +117,7 @@ "scratch-snapshot", "smp", "structured-outcome", + "untiered-network-policy", "virtio-net", "workload-identity" ] @@ -164,19 +166,20 @@ "structured-outcome": 0.5, "time-abi-conformance": 0.8, "time-host-correction": 14.0, + "untiered-network-policy": 14.1, "virtio-net": 3.7, "workload-identity": 0.6 }, "shards": [ { - "seconds": 239.3, + "seconds": 246.7, "scenarios": [ "console-snapshot", "directional-network-policy", + "endpoint-policy-snapshot", "filesystem-comma-paths", "filesystem-shares", "filesystem-snapshot", - "network-snapshot", "platform-filesystem", "platform-restore-processors", "platform-zero-fill", @@ -187,17 +190,17 @@ "smp-snapshot", "snapshot-core", "snapshot-tiers", - "time-host-correction" + "time-host-correction", + "untiered-network-policy" ] }, { - "seconds": 239.7, + "seconds": 246.4, "scenarios": [ "command-line-limit", "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", - "endpoint-policy-snapshot", "filesystem-owner", "filesystem-policy", "guest-boot", @@ -209,6 +212,7 @@ "l3-l4-egress-protocols", "lifecycle", "managed-lifecycle", + "network-snapshot", "platform-aggregate-export", "platform-network-policy", "private-network-ingress", @@ -264,12 +268,13 @@ "structured-outcome": 0.8, "time-abi-conformance": 1.6, "time-host-correction": 14.0, + "untiered-network-policy": 18.8, "virtio-net": 4.5, "workload-identity": 1.1 }, "shards": [ { - "seconds": 338.7, + "seconds": 348.4, "scenarios": [ "console-snapshot", "denied-filesystem-paths", @@ -279,7 +284,6 @@ "filesystem-snapshot", "host-loopback-policy", "network-snapshot", - "platform-aggregate-export", "platform-filesystem", "platform-restore-processors", "restore-downtime", @@ -288,11 +292,12 @@ "scratch-snapshot", "smp", "smp-snapshot", - "snapshot-tiers" + "snapshot-tiers", + "untiered-network-policy" ] }, { - "seconds": 338.7, + "seconds": 347.8, "scenarios": [ "command-line-limit", "console-exit", @@ -308,6 +313,7 @@ "l3-l4-egress-protocols", "lifecycle", "managed-lifecycle", + "platform-aggregate-export", "platform-network-policy", "platform-zero-fill", "private-network-ingress", diff --git a/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in b/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in new file mode 100644 index 000000000..e1a72ce5b --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in @@ -0,0 +1,88 @@ +set -eu +on_exit() { + status=$? + if [ "$status" -ne 0 ]; then + nvx-exit "$status" + fi +} +trap on_exit EXIT +fail() { + echo "NVX-UNTIERED-NETWORK-FAIL $1" + exit "$2" +} +# The nameservers of the guest's resolver, which the restore's repair names +# when the restore's policy has a DNS server. +resolver=$(awk '$1 == "nameserver" { list = list sep $2; sep = "," } + END { print (list == "" ? "none" : list) }' /etc/resolv.conf 2>/dev/null) || + resolver=none +# The repair starts the announcer of the guest's listeners when the restore's +# policy publishes them. +if pidof nvx-ingress >/dev/null; then + announcer=running +else + announcer=none +fi +# Sets result to whether the guest reached the host service that answers $1 +# at port $2 of the gateway, which the endpoint maps to host loopback. +probe() { + rm -f /tmp/nvx-untiered-network-probe + if wget -T @PROBE_TIMEOUT@ -qO /tmp/nvx-untiered-network-probe \ + "http://@GATEWAY_IPV4@:$2/$1" 2>/dev/null; then + [ "$(cat /tmp/nvx-untiered-network-probe)" = "NVX-PLATFORM-NETWORK-$1" ] || + fail "unexpected $1 answer" 81 + result=reached + else + result=denied + fi +} +probe allowed @ALLOWED_PORT@ +allowed=$result +probe other @OTHER_PORT@ +other=$result +echo "NVX-UNTIERED-NETWORK-REPORT resolver=$resolver announcer=$announcer allowed=$allowed other=$other" +# The guest's TCP server answers the host, one client at a time, through the +# ports that the restore's policy publishes or forwards. +rm -f /tmp/nvx-untiered-network-inbound-* +( + client=0 + while [ "$client" -lt @MAX_CLIENTS@ ]; do + printf 'NVX-HOST-LOOPBACK-TCP-ACK\n' | + nc -l -p @GUEST_PORT@ -w 20 \ + >"/tmp/nvx-untiered-network-inbound-$client" 2>/dev/null || true + client=$((client + 1)) + done +) & +server=$! +port_hex=$(printf '%04X' @GUEST_PORT@) +tries=0 +while ! awk -v pattern=":$port_hex\$" \ + '$2 ~ pattern && $4 == "0A" { found = 1 } END { exit !found }' \ + /proc/net/tcp /proc/net/tcp6; do + [ "$tries" -lt 100 ] || fail "the TCP server did not listen" 82 + tries=$((tries + 1)) + sleep 0.05 +done +# The terminal would echo the host's release into the lines after it. +stty -echo || fail "the terminal kept echoing" 85 +echo NVX-UNTIERED-NETWORK-LISTENING +# The host's release says how many lines it exchanged with the server. The +# last one may still be on the way when the release arrives, so the guest +# waits for each before it stops the server. +read -r release exchanges || fail "the host did not release the guest" 83 +[ "$release" = NVX-UNTIERED-NETWORK-CHECKED ] || fail "release $release" 84 +case "$exchanges" in +'' | *[!0-9]*) fail "release $release $exchanges" 84 ;; +esac +client=0 +while [ "$client" -lt "$exchanges" ]; do + tries=0 + while ! grep -qx NVX-HOST-LOOPBACK-INBOUND \ + "/tmp/nvx-untiered-network-inbound-$client" 2>/dev/null; do + [ "$tries" -lt 100 ] || fail "client $client sent no line" 86 + tries=$((tries + 1)) + sleep 0.05 + done + client=$((client + 1)) +done +kill "$server" 2>/dev/null || true +echo "NVX-UNTIERED-NETWORK-INBOUND $exchanges" diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 6e8cb3781..badd296a3 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -141,6 +141,7 @@ "structured-outcome", "time-abi-conformance", "time-host-correction", + "untiered-network-policy", "virtio-net", "workload-identity", ) @@ -401,6 +402,41 @@ PLATFORM_NETWORK_LISTENING_MARKER, RESTORE_MARKER, ) +# A NIC template without a tier is a snapshot without sandbox blocks whose NIC +# was captured with the complete deny-all policy. Each restore installs a +# policy of its own, and its guest's repair applies the DNS server and the +# listener announcement of that policy, which the restore packet carries. +UNTIERED_POST_RESTORE_MARKER = b"NVX-POST-RESTORE-OK: tier=legacy" +UNTIERED_NETWORK_FAILURE_MARKER = b"NVX-UNTIERED-NETWORK-FAIL" +UNTIERED_NETWORK_LISTENING_MARKER = b"NVX-UNTIERED-NETWORK-LISTENING" +UNTIERED_NETWORK_CHECKED = b"NVX-UNTIERED-NETWORK-CHECKED" +UNTIERED_NETWORK_REPORT_PREFIX = b"NVX-UNTIERED-NETWORK-REPORT " +UNTIERED_NETWORK_INBOUND_PREFIX = b"NVX-UNTIERED-NETWORK-INBOUND " +# The template's NIC is on the directional network, whose documentation range +# no host address shares: an ingress listener refuses peers inside the guest's +# own subnets, so a host address there would not reach the port that +# private-network ingress publishes. +UNTIERED_NETWORK_CIDR = DIRECTIONAL_NETWORK_CIDR +UNTIERED_NETWORK_GATEWAY_IPV4 = DIRECTIONAL_NETWORK_GATEWAY_IPV4 +# The gateway's IPv6 address, which a policy that admits DNS over IPv6 alone +# names as the guest's DNS server. +UNTIERED_NETWORK_GATEWAY_IPV6 = DIRECTIONAL_NETWORK_GATEWAY_IPV6 +# The most clients that the guest's server answers, as many as a restore's +# host reaches it through: its forward and its published port. +UNTIERED_NETWORK_MAX_CLIENTS = 2 +# The restores of the NIC template that OpenVMM refuses before the guest runs, +# and why. +UNTIERED_NETWORK_REJECTED = ( + ("kept-policy", b"restore-time egress policy does not match the snapshot"), + ("ingress", b"microVM snapshots do not support --network-ingress allow"), +) +# Lines that only a restored guest prints, so that a refused restore prints +# none of them. +UNTIERED_NETWORK_GUEST_MARKERS = ( + UNTIERED_POST_RESTORE_MARKER, + UNTIERED_NETWORK_LISTENING_MARKER, + RESTORE_MARKER, +) # A platform restore attaches a host directory to the filesystem slot that the # capture left dormant. The clone mounts it after the restore's input gate # releases, reads the host's seed from it, and writes its result to it. @@ -6593,6 +6629,451 @@ def check_template(context: str) -> None: raise RuntimeError("a refused restore reached a host service") +def untiered_network_policy_script( + *, allowed_port: int, other_port: int, guest_port: int +) -> str: + """Return the checks that each restore of the NIC template without a tier + runs: it reports the nameservers of its resolver, whether the announcer of + its listeners runs, and whether it reaches the host services at + ``allowed_port`` and ``other_port`` of the gateway, then serves the host at + ``guest_port`` until the host releases it.""" + return _render_script( + "untiered-network-policy.sh.in", + GATEWAY_IPV4=UNTIERED_NETWORK_GATEWAY_IPV4, + ALLOWED_PORT=str(allowed_port), + OTHER_PORT=str(other_port), + GUEST_PORT=str(guest_port), + MAX_CLIENTS=str(UNTIERED_NETWORK_MAX_CLIENTS), + PROBE_TIMEOUT=str(PLATFORM_NETWORK_PROBE_TIMEOUT_SECONDS), + ) + + +@dataclass(frozen=True) +class UntieredNetworkRestore: + """A restore of the NIC template without a tier: the policy that it + installs; the DNS server that its guest learns from the restore packet; + whether its guest reaches the allowed and the other host service; and how + the host reaches the guest's server: through a forward, and through the + port that the policy publishes on host loopback, ``host-loopback``, or to + the host's private networks, ``private-networks``, for which the guest + announces its listeners.""" + + name: str + policy: tuple[str, ...] + dns_server: str | None + allowed: bool + other: bool + forward: bool + publication: str | None + + @property + def announces(self) -> bool: + """Return whether the guest announces its listeners: whether the + policy publishes them.""" + return self.publication is not None + + @property + def network_settings(self) -> bool: + """Return whether the guest's repair applies network settings, after + which it reports the restore's completion.""" + return self.dns_server is not None or self.announces + + @property + def exchanges(self) -> int: + """Return how many lines the host exchanges with the guest's server: + one through the forward, and one through the published port.""" + return int(self.forward) + int(self.announces) + + +def untiered_network_restores( + *, allowed_port: int, host_port: int, guest_port: int +) -> tuple[UntieredNetworkRestore, ...]: + """Return the restores of the NIC template without a tier: one under the + template's own deny-all, which names no DNS server and publishes nothing; + egress rules that admit DNS over IPv4 and the allowed service; explicit + host loopback with a forward from ``host_port`` to ``guest_port``, under + egress rules that admit DNS over IPv6 alone; and private-network ingress + with allowed egress.""" + gateway = UNTIERED_NETWORK_GATEWAY_IPV4 + return ( + UntieredNetworkRestore( + "deny-all", + PLATFORM_NETWORK_TEMPLATE_POLICY, + dns_server=None, + allowed=False, + other=False, + forward=False, + publication=None, + ), + UntieredNetworkRestore( + "egress-dns", + ( + "--network-egress", + "deny", + "--network-egress-allow", + f"{gateway}/32:udp:53", + "--network-egress-allow", + f"{gateway}/32:tcp:{allowed_port}", + ), + dns_server=gateway, + allowed=True, + other=False, + forward=False, + publication=None, + ), + UntieredNetworkRestore( + "host-loopback", + ( + "--network-egress", + "deny", + "--network-egress-allow", + f"{UNTIERED_NETWORK_GATEWAY_IPV6}/128:udp:53", + "--host-loopback", + "allow", + "--host-loopback-forward", + f"tcp:{host_port}:{guest_port}", + ), + dns_server=UNTIERED_NETWORK_GATEWAY_IPV6, + allowed=False, + other=False, + forward=True, + publication="host-loopback", + ), + UntieredNetworkRestore( + "private-ingress", + ( + "--network-egress", + "allow", + "--network-ingress", + "allow", + "--host-loopback", + "allow", + ), + dns_server=gateway, + allowed=True, + other=True, + forward=False, + publication="private-networks", + ), + ) + + +def _untiered_network_report(output: bytes, context: str) -> dict[str, str]: + """Return the fields of the one line in ``output`` in which a restored + guest reported its resolver, its announcer, and its probes.""" + reports = [ + line + for line in _output_lines(output) + if line.startswith(UNTIERED_NETWORK_REPORT_PREFIX) + ] + if len(reports) != 1: + raise RuntimeError(f"{context} printed {len(reports)} network reports, not 1") + fields: dict[str, str] = {} + text = reports[0].removeprefix(UNTIERED_NETWORK_REPORT_PREFIX) + for field in text.decode("ascii", "replace").split(): + key, separator, value = field.partition("=") + if not separator or key in fields: + raise RuntimeError(f"{context} printed a malformed report: {reports[0]!r}") + fields[key] = value + if set(fields) != {"resolver", "announcer", "allowed", "other"}: + raise RuntimeError(f"{context} printed a malformed report: {reports[0]!r}") + return fields + + +def _check_untiered_network_report( + restore: UntieredNetworkRestore, report: dict[str, str], context: str +) -> None: + """Check that ``report``, the guest's, has the resolver, the announcer, + and the probes that ``restore``'s policy gives it. Without a DNS server, + the resolver keeps what the guest's image names, which is no gateway.""" + resolver = report["resolver"] + nameservers = [] if resolver == "none" else resolver.split(",") + if restore.dns_server is not None: + if nameservers != [restore.dns_server]: + raise RuntimeError( + f"{context}: the guest's resolver names {resolver}, not " + f"{restore.dns_server}" + ) + elif {UNTIERED_NETWORK_GATEWAY_IPV4, UNTIERED_NETWORK_GATEWAY_IPV6}.intersection( + nameservers + ): + raise RuntimeError( + f"{context}: the guest's resolver names {resolver}, although the " + "policy names no DNS server" + ) + expected = { + "announcer": "running" if restore.announces else "none", + "allowed": "reached" if restore.allowed else "denied", + "other": "reached" if restore.other else "denied", + } + for key, value in expected.items(): + if report[key] != value: + raise RuntimeError(f"{context}: the guest reported {key}={report[key]}") + + +def _accepts_tcp(address: str, port: int) -> bool: + """Return whether a TCP connection to ``address:port`` completes.""" + try: + connection = socket.create_connection((address, port), timeout=1) + except OSError: + return False + connection.close() + return True + + +def _untiered_network_exchanges( + restore: UntieredNetworkRestore, + *, + host_port: int, + guest_port: int, + timeout: float, +) -> int: + """Exchange a line with the guest's server through each way that + ``restore``'s policy gives the host, and return how many it exchanged. + + The forward binds ``host_port`` on host loopback before the guest runs. A + published port is ``guest_port``, which the gateway publishes once the + guest's announcer has announced the server: on host loopback alone, which + no other host address reaches, or on every host address, through one of + which, other than loopback, the host reaches it as a peer on its private + networks would. Without an announcer, nothing publishes the guest's port. + """ + exchanges = 0 + if restore.forward: + _host_loopback_tcp_exchange("127.0.0.1", host_port, timeout) + exchanges += 1 + if restore.publication == "host-loopback": + _wait_until( + lambda: _holds_loopback(socket.AF_INET, socket.SOCK_STREAM, guest_port), + timeout, + f"guest port {guest_port} was not published on host loopback", + ) + for address in _host_ipv4_addresses(): + if _accepts_tcp(address, guest_port): + raise RuntimeError( + f"host-loopback access published guest port {guest_port} on " + f"{address}" + ) + _host_loopback_tcp_exchange("127.0.0.1", guest_port, timeout) + exchanges += 1 + elif restore.publication == "private-networks": + addresses = _host_ipv4_addresses() + if not addresses: + raise RuntimeError( + "the host has no known IPv4 address other than loopback, through " + "which to reach the port that private-network ingress publishes" + ) + # A wildcard bind fails over a loopback-only listener too, so only the + # exchange through another address shows the port's publication to + # the host's private networks. + _wait_until( + lambda: _publishes(guest_port, socket.SOCK_STREAM), + timeout, + f"guest port {guest_port} was not published to the host", + ) + _host_loopback_tcp_exchange(addresses[0], guest_port, timeout) + exchanges += 1 + elif _holds_loopback(socket.AF_INET, socket.SOCK_STREAM, guest_port): + raise RuntimeError(f"guest port {guest_port} was published without ingress") + return exchanges + + +def _run_untiered_network_restore( + command: list[str], + restore: UntieredNetworkRestore, + *, + services: tuple[_HostLoopbackService, _HostLoopbackService], + host_port: int, + guest_port: int, + timeout: float, + log_path: Path, +) -> None: + """Restore the NIC template with ``command``, which installs + ``restore``'s policy, exchange lines with its guest's server while the + guest serves the host, and check what the guest reports, whether its + repair applied network settings, and the requests that the allowed and the + other host service in ``services`` received.""" + context = f"untiered network policy restore {restore.name}" + try: + with OpenvmmProcess(command, log_path) as process: + process.wait_for_line(UNTIERED_NETWORK_LISTENING_MARKER, timeout) + exchanges = _untiered_network_exchanges( + restore, host_port=host_port, guest_port=guest_port, timeout=timeout + ) + process.send_line(f"{UNTIERED_NETWORK_CHECKED.decode()} {exchanges}") + result = process.wait(timeout) + except Exception as error: + failures = [ + line.decode("utf-8", "replace") + for line in _output_lines( + log_path.read_bytes() if log_path.is_file() else b"" + ) + if line.startswith(UNTIERED_NETWORK_FAILURE_MARKER) + ] + raise RuntimeError( + f"{context} failed{f': {failures[0]}' if failures else ''}; see {log_path}" + ) from error + if result.returncode != 0: + raise RuntimeError(f"{context} exited with {result.returncode}") + _check_untiered_network_report( + restore, _untiered_network_report(result.output, context), context + ) + for marker in ( + UNTIERED_NETWORK_INBOUND_PREFIX + str(restore.exchanges).encode(), + RESTORE_MARKER, + ): + if not contains_output_line(result.output, marker): + raise RuntimeError(f"{context} did not report {marker.decode()!r}") + # A restore without network settings to apply finishes quietly, as a plain + # restore without a tier does. + if ( + contains_output_line(result.output, UNTIERED_POST_RESTORE_MARKER) + != restore.network_settings + ): + raise RuntimeError( + f"{context}: the guest's repair " + f"{'did not report' if restore.network_settings else 'reported'} " + f"{UNTIERED_POST_RESTORE_MARKER.decode()!r}" + ) + requests = tuple(service.take_requests() for service in services) + expected = (int(restore.allowed), int(restore.other)) + if requests != expected: + raise RuntimeError( + f"{context}: the allowed and other host services received " + f"{requests} requests, not {expected}" + ) + + +def run_untiered_network_policy( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Restore one NIC template without a tier with four network policies. + + The template is a snapshot without sandbox blocks, as the non-sandbox + mode captures one, whose NIC was captured with a complete deny-all + policy, so its guest learned no DNS server and announces no listeners. + Each restore installs another policy with ``--restore-network-policy``: + the template's deny-all; egress rules that admit DNS over IPv4 and one + host service; explicit host loopback with a forward, under egress rules + that admit DNS over IPv6 alone; and private-network ingress with allowed + egress. Each restore's guest must name the DNS server of its policy in its + resolver, and no gateway without one, run the announcer of its listeners + exactly when its policy publishes them, and reach exactly the host + services that its policy admits, which must receive no other request. The + host must reach the guest's server through the forward, and through the + port that the policy publishes: on host loopback alone, which no other + host address reaches, or to the host's private networks, through a host + address other than loopback. It must find the server on no host loopback + port without a publication. OpenVMM must refuse, before the guest runs, a + restore that keeps the template's policy but passes another, and one that + passes private-network ingress without the option. No restore may change + the template. + """ + with ExitStack() as stack: + services = ( + _HostLoopbackService("allowed", timeout), + _HostLoopbackService("other", timeout), + ) + for service in services: + stack.callback(service.close) + allowed, other = services + guest_port = _host_loopback_publication_port() + _, host_port = _available_tcp_address() + while host_port == guest_port: + _, host_port = _available_tcp_address() + restores = untiered_network_restores( + allowed_port=allowed.port, host_port=host_port, guest_port=guest_port + ) + root = Path( + stack.enter_context( + tempfile.TemporaryDirectory(prefix="nvx-untiered-network-policy-") + ) + ) + template = root / "template" + capture_snapshot( + [ + *workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "", + network=UNTIERED_NETWORK_CIDR, + ), + *PLATFORM_NETWORK_TEMPLATE_POLICY, + "--snapshot-destination", + str(template), + ], + template, + timeout=timeout, + processors=1, + post_restore_script=untiered_network_policy_script( + allowed_port=allowed.port, + other_port=other.port, + guest_port=guest_port, + ), + log_path=output_dir / "untiered-network-policy-capture.log", + ) + fingerprint = _snapshot_fingerprint(template) + + def restore_command(*arguments: str) -> list[str]: + return [ + *snapshot_restore_command( + executable, backend, template, network_profile="portable" + ), + *arguments, + ] + + def check_template(context: str) -> None: + if _snapshot_fingerprint(template) != fingerprint: + raise RuntimeError(f"{context} modified the NIC template") + errors = [error for service in services for error in service.errors] + if errors: + raise RuntimeError(f"a host service failed by {context}") from errors[0] + + check_template("the capture") + # The template's guest probed nothing before its capture. + if any(service.take_requests() for service in services): + raise RuntimeError("the NIC template's capture reached a host service") + for restore in restores: + _run_untiered_network_restore( + restore_command("--restore-network-policy", *restore.policy), + restore, + services=services, + host_port=host_port, + guest_port=guest_port, + timeout=timeout, + log_path=output_dir / f"untiered-network-policy-{restore.name}.log", + ) + check_template(f"restore {restore.name}") + + rejected_commands = { + "kept-policy": restore_command("--network-egress", "allow"), + "ingress": restore_command( + *PLATFORM_NETWORK_TEMPLATE_POLICY, "--network-ingress", "allow" + ), + } + for name, reason in UNTIERED_NETWORK_REJECTED: + _expect_process_failure( + rejected_commands[name], + output_dir / f"untiered-network-policy-rejected-{name}.log", + timeout, + reason, + forbidden_markers=UNTIERED_NETWORK_GUEST_MARKERS, + ) + check_template("a refused restore") + if any(service.take_requests() for service in services): + raise RuntimeError("a refused restore reached a host service") + + def run_platform_filesystem( executable: Path, kernel: Path, @@ -10065,6 +10546,21 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) + if "untiered-network-policy" in scenarios: + report.start( + "untiered-network-policy", + "Running microVM untiered-snapshot network policy replacement on " + f"OpenVMM/{args.backend}", + ) + run_untiered_network_policy( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "platform-filesystem" in scenarios: report.start( "platform-filesystem", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 641294f37..52f6fd7a4 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -2606,6 +2606,11 @@ def test_scripts_awaiting_a_release_turn_the_echo_off_first(self): f"echo {microvm_tests.PLATFORM_NETWORK_LISTENING_MARKER.decode()}", "read -r release exchange", ), + ( + "untiered-network-policy.sh.in", + f"echo {microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER.decode()}", + "read -r release exchanges", + ), ): with self.subTest(name=name): script = microvm_tests._read_script(name) @@ -11882,6 +11887,766 @@ def test_platform_network_policy_keeps_the_template_intact(self): Path(temporary), fingerprints=fingerprints ) + def test_untiered_network_policy_script_reports_the_repair_and_serves_the_host( + self, + ): + script = microvm_tests.untiered_network_policy_script( + allowed_port=40001, other_port=40002, guest_port=50001 + ) + self.assertNotRegex(script, r"@[A-Z0-9_]+@") + # The guest reports the resolver and the announcer that its repair set + # up, and probes both host services. + self.assertIn("' /etc/resolv.conf 2>/dev/null) ||\n resolver=none", script) + self.assertIn("if pidof nvx-ingress >/dev/null; then", script) + self.assertIn('"http://192.0.2.1:$2/$1"', script) + self.assertIn("wget -T 2 -qO /tmp/nvx-untiered-network-probe", script) + self.assertIn("\nprobe allowed 40001\n", script) + self.assertIn("\nprobe other 40002\n", script) + self.assertIn( + 'echo "' + + microvm_tests.UNTIERED_NETWORK_REPORT_PREFIX.decode() + + 'resolver=$resolver announcer=$announcer allowed=$allowed other=$other"', + script, + ) + # The guest's server answers the host, one client at a time, as many + # as a restore's host reaches it through. + ack = microvm_tests.HOST_LOOPBACK_TCP_ACK.decode().removesuffix("\n") + self.assertIn('while [ "$client" -lt 2 ]; do', script) + self.assertIn(f"printf '{ack}\\n' |\n nc -l -p 50001 -w 20", script) + self.assertIn("printf '%04X' 50001", script) + # Once released, the guest waits for each line that the host sent, + # which the release may overtake, before it stops its server. + request = microvm_tests.HOST_LOOPBACK_TCP_REQUEST.decode().strip() + self.assertIn("read -r release exchanges ||", script) + self.assertIn( + f'grep -qx {request} \\\n "/tmp/nvx-untiered-network-inbound-$client"', + script, + ) + self.assertLess( + script.index("read -r release exchanges"), script.index('kill "$server"') + ) + for marker in ( + microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER, + microvm_tests.UNTIERED_NETWORK_CHECKED, + ): + self.assertIn(marker.decode(), script) + self.assertTrue( + script.endswith( + 'echo "' + + microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX.decode() + + '$exchanges"\n' + ) + ) + self.assertIn( + 'echo "' + microvm_tests.UNTIERED_NETWORK_FAILURE_MARKER.decode() + ' $1"', + script, + ) + + def test_untiered_network_restores_name_what_their_policies_give_the_guest(self): + restores = microvm_tests.untiered_network_restores( + allowed_port=40001, host_port=40003, guest_port=50001 + ) + self.assertEqual( + [ + ( + restore.name, + restore.dns_server, + restore.announces, + restore.allowed, + restore.other, + restore.forward, + restore.publication, + restore.exchanges, + restore.network_settings, + ) + for restore in restores + ], + [ + ("deny-all", None, False, False, False, False, None, 0, False), + ("egress-dns", "192.0.2.1", False, True, False, False, None, 0, True), + ( + "host-loopback", + "fd00::c000:201", + True, + False, + False, + True, + "host-loopback", + 2, + True, + ), + ( + "private-ingress", + "192.0.2.1", + True, + True, + True, + False, + "private-networks", + 1, + True, + ), + ], + ) + deny_all, egress_dns, host_loopback, private_ingress = restores + # A restore without connectivity keeps the template's own policy. + self.assertEqual( + deny_all.policy, microvm_tests.PLATFORM_NETWORK_TEMPLATE_POLICY + ) + self.assertEqual( + egress_dns.policy, + ( + "--network-egress", + "deny", + "--network-egress-allow", + "192.0.2.1/32:udp:53", + "--network-egress-allow", + "192.0.2.1/32:tcp:40001", + ), + ) + self.assertEqual( + host_loopback.policy, + ( + "--network-egress", + "deny", + "--network-egress-allow", + "fd00::c000:201/128:udp:53", + "--host-loopback", + "allow", + "--host-loopback-forward", + "tcp:40003:50001", + ), + ) + self.assertEqual( + private_ingress.policy, + ( + "--network-egress", + "allow", + "--network-ingress", + "allow", + "--host-loopback", + "allow", + ), + ) + # Each restore's expectations are what OpenVMM gives its policy. + for restore in restores: + with self.subTest(restore=restore.name): + self.assertEqual( + self._untiered_network_settings(restore.policy, 40001), + ( + restore.dns_server, + restore.announces, + restore.allowed, + restore.other, + ), + ) + + def test_untiered_network_reports_hold_what_each_policy_gives_the_guest(self): + restores = { + restore.name: restore + for restore in microvm_tests.untiered_network_restores( + allowed_port=40001, host_port=40003, guest_port=50001 + ) + } + + def line(fields: str) -> bytes: + return ( + microvm_tests.UNTIERED_NETWORK_REPORT_PREFIX + fields.encode() + b"\r\n" + ) + + report = microvm_tests._untiered_network_report( + b"NVX-POST-RESTORE-OK: tier=legacy\r\n" + + line("resolver=192.0.2.1 announcer=none allowed=reached other=denied"), + "restore egress-dns", + ) + self.assertEqual( + report, + { + "resolver": "192.0.2.1", + "announcer": "none", + "allowed": "reached", + "other": "denied", + }, + ) + microvm_tests._check_untiered_network_report( + restores["egress-dns"], report, "restore egress-dns" + ) + # Without a DNS server, the resolver keeps the image's own, which may + # name a local resolver but never a gateway. + for resolver in ("none", "127.0.0.53"): + microvm_tests._check_untiered_network_report( + restores["deny-all"], + { + "resolver": resolver, + "announcer": "none", + "allowed": "denied", + "other": "denied", + }, + "restore deny-all", + ) + + def fields( + resolver: str, announcer: str, allowed: str, other: str + ) -> dict[str, str]: + return { + "resolver": resolver, + "announcer": announcer, + "allowed": allowed, + "other": other, + } + + for name, report, message in ( + ( + "egress-dns", + fields("none", "none", "reached", "denied"), + r"names none, not 192\.0\.2\.1$", + ), + ( + "egress-dns", + fields("192.0.2.1,127.0.0.53", "none", "reached", "denied"), + r"names 192\.0\.2\.1,127\.0\.0\.53, not 192\.0\.2\.1$", + ), + ( + "deny-all", + fields("192.0.2.1", "none", "denied", "denied"), + "although the policy names no DNS server$", + ), + ( + "deny-all", + fields("127.0.0.53,fd00::c000:201", "none", "denied", "denied"), + "although the policy names no DNS server$", + ), + ( + "host-loopback", + fields("fd00::c000:201", "none", "denied", "denied"), + "reported announcer=none$", + ), + ( + "egress-dns", + fields("192.0.2.1", "running", "reached", "denied"), + "reported announcer=running$", + ), + ( + "private-ingress", + fields("192.0.2.1", "running", "reached", "denied"), + "reported other=denied$", + ), + ): + with ( + self.subTest(name=name, report=report), + self.assertRaisesRegex(RuntimeError, f"^restore {name}: .*{message}"), + ): + microvm_tests._check_untiered_network_report( + restores[name], report, f"restore {name}" + ) + + valid = line("resolver=none announcer=none allowed=denied other=denied") + for output, message in ( + (b"", "printed 0 network reports, not 1$"), + (valid + valid, "printed 2 network reports, not 1$"), + (line("resolver=none announcer=none allowed=denied"), "malformed report"), + ( + line("resolver=none announcer=none allowed=denied other=a other=b"), + "malformed report", + ), + ( + line("resolver announcer=none allowed=denied other=denied"), + "malformed report", + ), + ): + with ( + self.subTest(output=output), + self.assertRaisesRegex(RuntimeError, f"^restore deny-all .*{message}"), + ): + microvm_tests._untiered_network_report(output, "restore deny-all") + + @staticmethod + def _untiered_network_settings( + policy: Sequence[str], allowed_port: int + ) -> tuple[str | None, bool, bool, bool]: + """Return what OpenVMM gives the guest of a restore under ``policy``: + the DNS server, as its gateway_dns_server derives it, whether the guest + announces its listeners, and whether the guest reaches the host service + at ``allowed_port`` of the gateway and the other host service.""" + + def values(option: str) -> list[str]: + return [ + policy[index + 1] + for index, value in enumerate(policy) + if value == option + ] + + allow_all = values("--network-egress") == ["allow"] + rules = values("--network-egress-allow") + if allow_all or "192.0.2.1/32:udp:53" in rules: + dns_server: str | None = "192.0.2.1" + elif "fd00::c000:201/128:udp:53" in rules: + dns_server = "fd00::c000:201" + else: + dns_server = None + announces = values("--network-ingress") == ["allow"] or values( + "--host-loopback" + ) == ["allow"] + allowed = allow_all or f"192.0.2.1/32:tcp:{allowed_port}" in rules + return dns_server, announces, allowed, allow_all + + def _simulate_untiered_network_guest( + self, command: list[str], ports: dict[str, int] + ) -> list[bytes]: + """Return the lines of the guest that ``command`` restores before its + release, after it requests each host service in ``ports`` that its + policy admits.""" + policy = command[command.index("--restore-network-policy") + 1 :] + dns_server, announces, allowed, other = self._untiered_network_settings( + policy, ports["allowed"] + ) + for name, reached in (("allowed", allowed), ("other", other)): + if not reached: + continue + with socket.create_connection( + ("127.0.0.1", ports[name]), timeout=5 + ) as client: + client.sendall( + f"GET /{name} HTTP/1.1\r\nHost: 192.0.2.1\r\n\r\n".encode() + ) + answer = b"" + while chunk := client.recv(4096): + answer += chunk + self.assertTrue( + answer.endswith(f"\r\n\r\nNVX-PLATFORM-NETWORK-{name}".encode()) + ) + lines: list[bytes] = [] + if dns_server is not None or announces: + lines.append(microvm_tests.UNTIERED_POST_RESTORE_MARKER) + lines.append( + microvm_tests.UNTIERED_NETWORK_REPORT_PREFIX + + ( + f"resolver={dns_server or 'none'} " + f"announcer={'running' if announces else 'none'} " + f"allowed={'reached' if allowed else 'denied'} " + f"other={'reached' if other else 'denied'}" + ).encode() + ) + return lines + + def _run_untiered_network_policy( + self, + output_dir: Path, + *, + guest: Callable[[list[str], dict[str, int]], list[bytes]] | None = None, + fingerprints: Sequence[tuple[str, str, str]] | None = None, + host_addresses: Sequence[str] = ("192.0.2.10",), + accepts_tcp: bool = False, + exchange: Callable[[str, int, float], None] | None = None, + ) -> dict[str, Any]: + """Run the scenario with a simulated OpenVMM, whose restores request + the scenario's real host services as their policies admit, on a host + whose addresses other than loopback are ``host_addresses``, through + which a TCP connection completes if ``accepts_tcp``, and with + ``exchange`` standing for each exchange with the guest's server, and + return its calls.""" + captures: list[tuple[list[str], dict[str, Any]]] = [] + restores: list[list[str]] = [] + releases: list[str] = [] + ports: dict[str, int] = {} + + def capture(command: list[str], *_args: object, **kwargs: Any) -> None: + captures.append((command, kwargs)) + script = kwargs["post_restore_script"] + for name in ("allowed", "other"): + match = re.search(rf"\nprobe {name} (\d+)\n", script) + assert match is not None + ports[name] = int(match[1]) + match = re.search(r"nc -l -p (\d+) ", script) + assert match is not None + ports["guest"] = int(match[1]) + + simulated_guest = guest or self._simulate_untiered_network_guest + + class FakeProcess: + def __init__(self, command: list[str], log_path: Path) -> None: + restores.append(command) + self.command = command + self.log_path = log_path + self.lines: list[bytes] = [] + self.exchanges: bytes | None = None + + def __enter__(self) -> "FakeProcess": + return self + + def __exit__(self, *_exception: object) -> None: + return None + + def wait_for_line(self, marker: bytes, _timeout: float) -> None: + assert marker == microvm_tests.UNTIERED_NETWORK_LISTENING_MARKER + self.lines = simulated_guest(self.command, ports) + + def send_line(self, line: str) -> None: + releases.append(line) + self.exchanges = line.split()[1].encode() + + def wait(self, _timeout: float) -> openvmm_process.OpenvmmProcessResult: + assert self.exchanges is not None + lines = [ + *self.lines, + microvm_tests.UNTIERED_NETWORK_INBOUND_PREFIX + self.exchanges, + benchmark.RESTORE_MARKER, + ] + output = b"".join(line + b"\r\n" for line in lines) + self.log_path.write_bytes(output) + return openvmm_process.OpenvmmProcessResult(0, output) + + fingerprint = ("manifest", "state", "memory") + output_dir.mkdir(parents=True, exist_ok=True) + with ( + patch.object( + microvm_tests, + "workload_boot_command", + return_value=["openvmm", "boot"], + ) as workload_boot_command, + patch.object(microvm_tests, "capture_snapshot", side_effect=capture), + patch.object(microvm_tests, "OpenvmmProcess", FakeProcess), + patch.object( + microvm_tests, "_host_loopback_tcp_exchange", side_effect=exchange + ) as exchange_mock, + patch.object(microvm_tests, "_wait_until") as wait_until, + patch.object( + microvm_tests, "_holds_loopback", return_value=False + ) as holds_loopback, + patch.object( + microvm_tests, "_host_ipv4_addresses", return_value=list(host_addresses) + ), + patch.object( + microvm_tests, "_accepts_tcp", return_value=accepts_tcp + ) as accepts_tcp_mock, + patch.object( + microvm_tests, + "_snapshot_fingerprint", + side_effect=fingerprints or itertools.repeat(fingerprint), + ), + patch.object(microvm_tests, "_expect_process_failure") as expect_failure, + ): + microvm_tests.run_untiered_network_policy( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + memory_mib=128, + timeout=60, + output_dir=output_dir, + ) + return { + "workload_boot_command": workload_boot_command, + "captures": captures, + "restores": restores, + "releases": releases, + "exchange": exchange_mock, + "wait_until": wait_until, + "holds_loopback": holds_loopback, + "accepts_tcp": accepts_tcp_mock, + "expect_failure": expect_failure, + "ports": ports, + } + + def test_untiered_network_policy_restores_one_template_with_four_policies(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) / "logs" + calls = self._run_untiered_network_policy(output_dir) + logs = sorted(path.name for path in output_dir.iterdir()) + + self.assertIn("untiered-network-policy", microvm_tests.MICROVM_TEST_SCENARIOS) + # Every guest runs it: the non-sandbox mode captures the template. + self.assertNotIn( + "untiered-network-policy", microvm_tests.SANDBOX_CONTROL_SCENARIOS + ) + self.assertNotIn( + "untiered-network-policy", microvm_tests.UBUNTU_UNSUPPORTED_SCENARIOS + ) + boot_call = calls["workload_boot_command"].call_args + self.assertEqual(boot_call.args[5], "") + self.assertEqual( + boot_call.kwargs["network"], microvm_tests.UNTIERED_NETWORK_CIDR + ) + + # The template is a snapshot without a tier or sandbox blocks, whose + # NIC has the complete deny-all policy. + ((template, template_kwargs),) = calls["captures"] + self.assertEqual( + template[:-2], + [ + "openvmm", + "boot", + *microvm_tests.PLATFORM_NETWORK_TEMPLATE_POLICY, + ], + ) + self.assertEqual(template[-2], "--snapshot-destination") + self.assertNotIn("--snapshot-tier", template) + self.assertNotIn("--microvm-sandbox-block", template) + ports = calls["ports"] + self.assertEqual( + template_kwargs["post_restore_script"], + microvm_tests.untiered_network_policy_script( + allowed_port=ports["allowed"], + other_port=ports["other"], + guest_port=ports["guest"], + ), + ) + self.assertEqual(template_kwargs["processors"], 1) + + # Each restore installs its own policy on the template. + restores = calls["restores"] + names = ("deny-all", "egress-dns", "host-loopback", "private-ingress") + forward_port = None + for command, name in zip(restores, names, strict=True): + with self.subTest(restore=name): + self.assertEqual( + command[command.index("--restore-snapshot") + 1], template[-1] + ) + self.assertEqual( + command[command.index("--network-profile") + 1], "portable" + ) + self.assertNotIn("--microvm-sandbox-block", command) + self.assertNotIn("--restore-absent-scratch", command) + self.assertIn("--restore-network-policy", command) + if "--host-loopback-forward" in command: + forward = command[command.index("--host-loopback-forward") + 1] + forward_port = int(forward.split(":")[1]) + self.assertEqual(forward, f"tcp:{forward_port}:{ports['guest']}") + self.assertIsNotNone(forward_port) + self.assertNotEqual(forward_port, ports["guest"]) + # The host reaches the guest's server through the forward and the port + # that a policy publishes, on host loopback alone, which no other host + # address reaches, or through another host address for private-network + # ingress, and finds it on no host loopback port when the policy + # publishes nothing. + self.assertEqual( + [entry.args for entry in calls["exchange"].call_args_list], + [ + ("127.0.0.1", forward_port, 60), + ("127.0.0.1", ports["guest"], 60), + ("192.0.2.10", ports["guest"], 60), + ], + ) + self.assertEqual( + [entry.args for entry in calls["accepts_tcp"].call_args_list], + [("192.0.2.10", ports["guest"])], + ) + self.assertEqual(calls["wait_until"].call_count, 2) + self.assertEqual( + [entry.args for entry in calls["holds_loopback"].call_args_list], + [(socket.AF_INET, socket.SOCK_STREAM, ports["guest"])] * 2, + ) + self.assertEqual( + calls["releases"], + [ + "NVX-UNTIERED-NETWORK-CHECKED 0", + "NVX-UNTIERED-NETWORK-CHECKED 0", + "NVX-UNTIERED-NETWORK-CHECKED 2", + "NVX-UNTIERED-NETWORK-CHECKED 1", + ], + ) + self.assertEqual( + logs, + sorted(f"untiered-network-policy-{name}.log" for name in names), + ) + + # OpenVMM refuses the other restores before the guest runs. + failures = calls["expect_failure"].call_args_list + self.assertEqual( + [entry.args[3] for entry in failures], + [reason for _, reason in microvm_tests.UNTIERED_NETWORK_REJECTED], + ) + for entry in failures: + self.assertEqual( + entry.kwargs["forbidden_markers"], + microvm_tests.UNTIERED_NETWORK_GUEST_MARKERS, + ) + self.assertNotIn("--restore-network-policy", entry.args[0]) + kept_policy, ingress = (entry.args[0] for entry in failures) + self.assertEqual(kept_policy[-2:], ["--network-egress", "allow"]) + self.assertEqual( + ingress[-6:], + [ + *microvm_tests.PLATFORM_NETWORK_TEMPLATE_POLICY, + "--network-ingress", + "allow", + ], + ) + + def test_untiered_network_policy_requires_what_each_policy_gives_the_guest(self): + def guest_reporting( + replace: bytes, by: bytes + ) -> Callable[[list[str], dict[str, int]], list[bytes]]: + def guest(command: list[str], ports: dict[str, int]) -> list[bytes]: + return [ + line.replace(replace, by) + for line in self._simulate_untiered_network_guest(command, ports) + ] + + return guest + + def guest_dropping( + dropped: bytes, + ) -> Callable[[list[str], dict[str, int]], list[bytes]]: + def guest(command: list[str], ports: dict[str, int]) -> list[bytes]: + return [ + line + for line in self._simulate_untiered_network_guest(command, ports) + if line != dropped + ] + + return guest + + def guest_reporting_completion( + command: list[str], ports: dict[str, int] + ) -> list[bytes]: + lines = self._simulate_untiered_network_guest(command, ports) + if microvm_tests.UNTIERED_POST_RESTORE_MARKER in lines: + return lines + return [microvm_tests.UNTIERED_POST_RESTORE_MARKER, *lines] + + for guest, message in ( + # A guest whose repair did not name the policy's DNS server. + ( + guest_reporting(b"resolver=192.0.2.1", b"resolver=none"), + r"^untiered network policy restore egress-dns: the guest's " + r"resolver names none, not 192\.0\.2\.1$", + ), + # A guest whose repair started no announcer. + ( + guest_reporting(b"announcer=running", b"announcer=none"), + "^untiered network policy restore host-loopback: the guest " + "reported announcer=none$", + ), + # A guest whose repair applied settings without reporting it. + ( + guest_dropping(microvm_tests.UNTIERED_POST_RESTORE_MARKER), + "^untiered network policy restore egress-dns: the guest's repair " + "did not report 'NVX-POST-RESTORE-OK: tier=legacy'$", + ), + # A guest without settings whose repair did not finish quietly. + ( + guest_reporting_completion, + "^untiered network policy restore deny-all: the guest's repair " + "reported 'NVX-POST-RESTORE-OK: tier=legacy'$", + ), + ): + with ( + self.subTest(message=message), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex(RuntimeError, message): + self._run_untiered_network_policy(Path(temporary), guest=guest) + + # The host services must receive exactly the requests that a policy + # admits, whatever the guest reports. + def guest_reaching_more( + command: list[str], ports: dict[str, int] + ) -> list[bytes]: + lines = self._simulate_untiered_network_guest(command, ports) + with socket.create_connection( + ("127.0.0.1", ports["other"]), timeout=5 + ) as client: + client.sendall(b"GET /other HTTP/1.1\r\n\r\n") + while client.recv(4096): + pass + return lines + + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaisesRegex( + RuntimeError, + r"^untiered network policy restore deny-all: the allowed and " + r"other host services received \(0, 1\) requests, not \(0, 0\)$", + ): + self._run_untiered_network_policy( + Path(temporary), guest=guest_reaching_more + ) + + def test_untiered_network_policy_reports_guest_failures(self): + def failing_guest(_command: list[str], _ports: dict[str, int]) -> list[bytes]: + log_path = output_dir / "untiered-network-policy-deny-all.log" + log_path.write_bytes( + b"NVX-UNTIERED-NETWORK-FAIL the TCP server did not listen\r\n" + ) + raise TimeoutError("line marker was not observed") + + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + with self.assertRaises(RuntimeError) as raised: + self._run_untiered_network_policy(output_dir, guest=failing_guest) + self.assertEqual( + str(raised.exception), + "untiered network policy restore deny-all failed: " + "NVX-UNTIERED-NETWORK-FAIL the TCP server did not listen; see " + f"{output_dir / 'untiered-network-policy-deny-all.log'}", + ) + self.assertIsInstance(raised.exception.__cause__, TimeoutError) + + def test_untiered_network_policy_checks_where_ingress_publishes_the_guest_port( + self, + ): + def loopback_only(address: str, port: int, _timeout: float) -> None: + # A listener on host loopback alone, which the wildcard bind probe + # cannot tell from one on every host address. + if address != "127.0.0.1": + raise TimeoutError(f"{address}:{port} did not answer") + + cases: tuple[tuple[dict[str, Any], str, str], ...] = ( + # Private-network ingress that published the guest's port on host + # loopback alone. + ( + {"exchange": loopback_only}, + "private-ingress", + r"^192\.0\.2\.10:\d+ did not answer$", + ), + # Host-loopback access that published the guest's port on another + # host address too. + ( + {"accepts_tcp": True}, + "host-loopback", + r"^host-loopback access published guest port \d+ on 192\.0\.2\.10$", + ), + # A host without an address through which to reach the port that + # private-network ingress publishes. + ( + {"host_addresses": ()}, + "private-ingress", + "^the host has no known IPv4 address other than loopback, ", + ), + ) + for options, restore, cause in cases: + with ( + self.subTest(restore=restore, cause=cause), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex( + RuntimeError, + f"^untiered network policy restore {restore} failed; see ", + ) as raised: + self._run_untiered_network_policy(Path(temporary), **options) + self.assertRegex(str(raised.exception.__cause__), cause) + + def test_untiered_network_policy_keeps_the_template_intact(self): + original = ("manifest", "state", "memory") + changed = ("manifest", "state", "changed") + for fingerprints, message in ( + # The capture, then the first restore. + ([original, original, changed], "^restore deny-all modified the NIC"), + # The capture and the four restores, then the refused restores. + ( + [original] * 6 + [changed], + "^a refused restore modified the NIC template$", + ), + ): + with ( + self.subTest(message=message), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex(RuntimeError, message): + self._run_untiered_network_policy( + Path(temporary), fingerprints=fingerprints + ) + def test_platform_filesystem_script_uses_the_share_that_the_restore_attaches( self, ):