Repository navigation
guest: apply the DNS server and listener announcements of a policy replaced at an untiered restore - #1072
guest: apply the DNS server and listener announcements of a policy replaced at an untiered restore#1072Pedro Henrique Penna (ppenna) wants to merge 6 commits into
Conversation
Pin microsoft/openvmm#4831 at f1eb78302a296f455a99f8ac622a46f4c0862908, two commits on 3d7512af6, the previous pin, which is the head of feature/microvm/dev. A snapshot whose NIC was captured with the complete deny-all policy is a NIC template, and --restore-network-policy lets a restore install a new policy on it, but OpenVMM admitted the option only for a platform snapshot and refused it for a snapshot without a tier (#942). A platform clone also learns its DNS server, and whether to announce its listeners, from its activation after the restore, which the guest of a snapshot without a tier never receives, so under a new policy that guest would keep the deny-all's configuration: no DNS server and no nvx-ingress (#943). The first commit admits the option for a snapshot without a tier, and still refuses it for a workload-start or instance-checkpoint snapshot. Such a snapshot may hold a workload that has already run, but under the deny-all that workload had no connectivity, so nothing in the snapshot depends on the policy. The second hands the guest of such a restore the new policy's DNS server, the gateway that a cold boot under the policy names, and whether to announce its listeners, in version 5 of the restore packet: its network byte, which version 4 reserves, gives the server's family and the announcement, and 16 bytes after the entropy give the server. Such a restore requires the guest's acknowledgement, so the guest applies the settings before host input reaches it. Every other restore, a platform clone's included, keeps version 4, so the pin changes nothing for NVX's guests until a restore asks for version 5, which a later commit teaches them to read. Neither OpenVMM's control-session protocol vectors nor its built-in CPU profiles change, so their copies in nvx-rs stay as they are. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the commits that the previous commit pins as positions 373 and 374 of the microVM chain, and move the NVX pin to f1eb78302. Position 373 lets --restore-network-policy install a new policy on the NIC template of a snapshot without a tier, and position 374 hands the guest of such a restore the new policy's network settings in version 5 of the restore packet. They change only the tier check of position 347's restore that installs a new policy and the time ABI's restore packet, which positions 159-269 add, so they build on those fork-only commits; say so in the table's remarks. The pin is now 217 commits past the mapped tip. Positions 371 and 372 are no longer only a proposal: microsoft/openvmm#4816 merged them, so position 372, 3d7512af6, is the head of feature/microvm/dev. Say so in the scope, and name microsoft/openvmm#4831 as the proposal of positions 373 and 374. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The immediate process-existence check does not confirm announcer socket initialization, allowing restore acknowledgment and host input release after startup failure; a bounded startup handshake, fail-closed handling, and regression coverage are needed.
1 open issue
What changed in this PR
This PR enables untiered NIC template restores to apply a replacement network policy’s DNS server and listener announcements before acknowledging the restore, addressing #942 and #943.
Changes:
- Adds restore packet v5 decoding and guest-side network repair while preserving v4 support.
- Exposes
nvx-rs run --restore-network-policy, with CLI validation and fake OpenVMM coverage. - Adds a four-policy microVM scenario, updates test shards, and documents the restore behavior and OpenVMM integration.
| File | Description |
|---|---|
scripts/test_microvm_tests.py |
Tests network repair, failures, and scenario behavior. |
scripts/nvx_tools/microvm_tests.py |
Adds the four-policy untiered restore scenario. |
scripts/nvx_tools/microvm_test_scripts/untiered-network-policy.sh.in |
Checks guest settings and network traffic. |
scripts/nvx_tools/microvm_shards.json |
Adds the scenario and rebalances shards. |
nvx-rs/tests/vectors/README.md |
Documents checks for both restore packet versions. |
nvx-rs/tests/openvmm_vectors.rs |
Checks the v5 constant against OpenVMM. |
nvx-rs/tests/guest_tools.rs |
Tests v5 decoding and malformed settings. |
nvx-rs/tests/fake_openvmm.rs |
Tests restore-policy ingress acceptance and refusal. |
nvx-rs/tests/cli/validation.rs |
Tests rejection without a restore snapshot. |
nvx-rs/tests/cli/openvmm.rs |
Recognizes the new valueless option. |
nvx-rs/tests/cli/one_shot.rs |
Tests launching a policy-replacing restore. |
nvx-rs/tests/cli/dry_run.rs |
Tests generated restore-policy arguments. |
nvx-rs/src/testing/fake_openvmm.rs |
Models restore-policy option validation. |
nvx-rs/src/host/cli/run.rs |
Parses, validates, and forwards the new option. |
nvx-rs/src/guest/time/test_mode.rs |
Prints decoded network settings in test mode. |
nvx-rs/src/guest/time/mod.rs |
Exports RestoreNetwork. |
nvx-rs/src/guest/time/decoders.rs |
Decodes and validates v5 network records. |
nvx-rs/src/guest/time/commands.rs |
Includes network settings in capture metadata. |
nvx-rs/src/guest/time/api.rs |
Exposes settings and defers completion for network repair. |
nvx-rs/src/contract.rs |
Defines the restore packet v5 constant. |
nvx-rs/README.md |
Documents CLI behavior and compatibility checks. |
guest/common/nvx-snapshot |
Applies DNS and starts listener announcements before acknowledgement. |
doc/usage.md |
Updates restore and network option references. |
doc/run.md |
Explains NIC template policy replacement. |
doc/openvmm-upstream-roadmap.md |
Records the new pin and its two commits. |
doc/design/5.5-restore-protocol.md |
Specifies v5 layout and network repair ordering. |
doc/design/3.5-network.md |
Describes replacement policies and guest configuration. |
doc/design/3.1-platform-control-device.md |
Documents restore packet network settings. |
doc/ci.md |
Describes the new scenario and version checks. |
doc/build.md |
Documents v4 and v5 parser support. |
.github/skills/nvx-openvmm-promote/SKILL.md |
Updates guidance to reference multiple restore packet versions. |
Review effort: Balanced
A clone's application starts the announcer of the guest's listeners with Announcer::start, which binds the announcer's socket to a privileged source port and runs nvx-ingress inherit on it, so a failure to start it is the start's own, before the clone is ready. A guest agent that cannot hold the announcer, such as a shell script, could only run nvx-ingress in the background, as init does at a cold boot, and could not tell whether the announcer started: whether it found a privileged source port free, or ran at all. nvx-ingress start GATEWAY_IPV4 starts the announcer as Announcer::start does, and exits once the announcer's socket is bound and its program runs, which leaves the announcer to init, or to the nearest subreaper, or exits with status 1, naming why, when the announcer does not start. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When a restore installs a new network policy on the NIC template of a snapshot without a tier, OpenVMM hands the guest the policy's DNS server, and whether to announce its listeners, in version 5 of the restore packet, and gates host input until the guest acknowledges the restore. NVX's guest read version 4 alone, so it failed such a restore, and nothing would have applied the settings: the template's guest booted under the deny-all policy, whose command line names no DNS server and starts no nvx-ingress, and only a platform clone takes them from its activation. nvx-time now reads version 5 as well. It refuses a network byte with DNS family 3 or another bit set, and a DNS server that does not match its family or is unspecified, and the capture's metadata for nvx-snapshot gains the DNS server, or "-", and the announcement. As for a packet with processors or memory to activate, nvx-time leaves a packet whose settings ask for something to nvx-snapshot, which acknowledges the restore once it applied them. TimeAbi's Restore exposes the settings as a RestoreNetwork, and nvx-time's packet test hook prints them. nvx-snapshot applies them in guest repair step 4, after processors and memory, as init applies those of a cold boot under such a policy: it names the DNS server in /etc/resolv.conf, and starts the announcer of the guest's listeners with nvx-ingress start and the gateway of virtnet_gw=, which returns once the announcer's socket is bound to a privileged source port and its program runs. A failure of either fails the restore, with status 1, before its acknowledgement. A restore whose settings ask for nothing stays quiet, and an untiered restore still leaves entropy and identity to its caller. The restore-protocol, network, and byte-channel design documents describe version 5 and the repair. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OpenVMM lets a restore install a new network policy on a NIC template, a snapshot whose NIC was captured with the complete deny-all policy, with --restore-network-policy, for a platform snapshot and now for one without a tier, whose guest then applies the policy's DNS server and listener announcements before the host releases its input. nvx-rs run could not ask for it: every restore passed the network options to OpenVMM as the policy that the snapshot was captured with, so run could restore a NIC template only under its deny-all. run --restore-network-policy, which needs --restore-snapshot, passes the option to OpenVMM after the restore's other options, so the restore installs the policy of its network options, private-network ingress, explicit host loopback, and forwards included. run leaves the checks of the snapshot and the policy to OpenVMM, as it does for every other restore, rather than read the snapshot itself. The option stays explicit because OpenVMM refuses it for a snapshot that is no NIC template, so run cannot pass it to every restore. nvx-fake-openvmm accepts the option at a restore and refuses it at a cold boot, and, like OpenVMM, refuses a restore without it that asks for private-network ingress or explicit host-loopback access, with OpenVMM's messages. The usage and run guides and the nvx-rs README describe the option. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
86819b6 to
816d6bf
Compare
816d6bf to
329d6b0
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-repository restore ABI and input-release ordering require human confirmation of exact-pin OpenVMM integration and pending Windows/WHP end-to-end validation.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
329d6b0 to
50d841d
Compare
Add the untiered-network-policy microVM scenario, which covers the restore of a NIC template without a tier, a snapshot without sandbox blocks, as the non-sandbox mode captures one, whose NIC was captured with the complete deny-all policy, with --restore-network-policy. It restores the template four times: under the template's own deny-all, under egress rules that admit DNS over IPv4 and one host service, with explicit host loopback and a forward under egress rules that admit DNS over IPv6 alone, and with private-network ingress and allowed egress. The template's guest learned no DNS server and runs no announcer of its listeners, so each restore's repair applies those of the restore's policy, which restore packet version 5 carries, before it acknowledges the restore. The guest's resolver must name exactly the policy's DNS server, and nvx-ingress must run exactly when the policy publishes the guest's listeners. Each restore must reach through its gateway exactly the host services that its policy admits, which receive no other request. The host must reach the guest's server through the forward, and through the port that the policy publishes: on host loopback alone, which no other host address reaches, or, for private-network ingress, through a host address other than loopback, since a probe that binds every host address also fails over a loopback-only listener. It must find the server on no host loopback port without a publication. OpenVMM must refuse, before the guest runs, a restore that keeps the template's policy but passes another, and one that passes private-network ingress without the option, and no restore may change the template. The template's NIC is on the directional network, 192.0.2.0/24, a documentation range that no host address shares: an ingress listener refuses peers inside the guest's subnets, so a host address there, as on hosts whose network is 10.0.0.0/24, would not reach the port that private-network ingress publishes. The scenario joins the default suite, so plan the shards again, with its time in local runs on virtual-machine hosts of each backend, 14.0 s on KVM, 14.1 s on MSHV, and 18.8 s on WHP, until CI's reports measure it. Part of #943 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
50d841d to
82a9f9c
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The cross-repository restore ABI and host-input gating changes need maintainer confirmation of Windows/WHP end-to-end validation and that the exact pinned OpenVMM revision lands before integration.
No open issues 🎉
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
The pinned dependency microsoft/openvmm#4831 is still open, and its unchanged landing on feature/microvm/dev must be confirmed before merging the NVX integration.
No open issues 🎉
Review effort: Balanced
|
Removed from the merge queue so that #1018 could land first; requeue this once it's rebased. microsoft/openvmm#4830 (time ABI 2) is now merged: The two OpenVMM changes also collide on restore packet version 5:
To land this pull request:
|

Fixes #943. Fixes #942. Pins microsoft/openvmm#4831 at
f1eb78302a296f455a99f8ac622a46f4c0862908. That pin lets the restore of a NIC template without a tier install a network policy of its own, and hands that restore's guest the new policy's DNS server and listener announcements in version 5 of the restore packet. On the NVX side:nvx-rs run --restore-network-policyasks for such a restore;Based on
devat2d1edb3fd. microsoft/openvmm#4831 is two commits directly on3d7512af6, the head offeature/microvm/devand the revision thatdevpins. Its CI is green.Change
fb4aef9a1openvmmopenvmm3d7512af6→f1eb78302: the two commits of microsoft/openvmm#4831 (gitlink only)77845c67bdocdoc/openvmm-upstream-roadmap.md9cc72095fnvx-rsnvx-rs/src/guest/ingress/*,nvx-rs/tests/guest_tools.rs,nvx-rs/README.mdnvx-ingress start GATEWAY_IPV4starts the announcer asAnnouncer::startdoes: it binds the privileged source port, runsnvx-ingress inheriton it, and exits once that runs. It exits 1 if the announcer does not startcd4e2512eguestnvx-rs/src/guest/time/*,nvx-rs/src/contract.rs,guest/common/nvx-snapshot, guest and vector tests, design docs 3.1, 3.5, 5.5nvx-timedecodes restore packet versions 4 and 5.nvx-snapshotapplies the DNS server and listener announcements in guest repair step 4, before the acknowledgementa2cc8211bnvx-rsnvx-rs/src/host/cli/run.rs,nvx-rs/src/testing/fake_openvmm.rs, CLI tests,doc/run.md,doc/usage.md,nvx-rs/README.mdnvx-rs run --restore-network-policypasses OpenVMM's option. The fake OpenVMM mirrors OpenVMM's refusals82a9f9cabscriptsscripts/nvx_tools/microvm_tests.py,untiered-network-policy.sh.in,scripts/test_microvm_tests.py,microvm_shards.json,doc/ci.mduntiered-network-policy; shards replanned with its measured durationsOpenVMM changes (microsoft/openvmm#4831)
e4fd1655a, OpenVMM prerequisite: a new network policy at the restore of a NIC template without a tier #942).--restore-network-policynow admits a snapshot without a tier as well as a platform snapshot, and still refusesworkload-startandinstance-checkpointsnapshots.f1eb78302, guest: apply the DNS server and listener announcements of a replaced policy after a non-sandbox restore #943). A platform clone learns its DNS server and listener announcements from its activation. The guest of an untiered snapshot gets no activation, and its captured command line names no DNS server and starts nonvx-ingress.--restore-network-policy, OpenVMM writes packet version 5. Byte 7, reserved in version 4, becomes the network byte: bits 0-1 give the DNS family (none, IPv4, or IPv6), bit 2 asks for listener announcements, and the other bits are zero. A 16-byte DNS server follows the entropy: IPv4 plus 12 zero bytes, IPv6, or zeros.gateway_dns_server(), what a cold boot would pass asvirtnet_dns=. The announcement is set exactly when the policy publishes listeners.NVX changes
Guest decoding (
nvx-rs/src/guest/time).nvx-timeaccepts versions 4 and 5. It refuses a network byte with family 3 or unknown bits, and a DNS record that does not match its family or is unspecified. It exposesRestoreNetworkonTimeAbi'sRestore.nvx-timedoes not finish a packet whose settings ask for something itself, just as for processors or memory, sonvx-snapshotcan apply them before the acknowledgement.nvx::contract::RESTORE_PACKET_NETWORK_VERSION(5) is checked against OpenVMM bytests/openvmm_vectors.rs.RESTORE_PACKET_VERSIONand the template key stay at 4, since clones never see version 5.Guest repair (
guest/common/nvx-snapshot). Step 4 now also:nameserver ADDRto/etc/resolv.conf;/sbin/nvx-ingress startwithvirtnet_gw=, which returns only once the announcer's socket is bound to a privileged source port and its program runs.A failure of either fails the restore, with status 1, before the acknowledgement. A restore whose settings ask for nothing stays quiet, and an untiered restore still leaves entropy and identity to its caller.
nvx-rs run --restore-network-policy. It requires--restore-snapshotand passes OpenVMM's option, so the restore installs the policy of the network options, ingress, explicit host loopback, and forwards included. The option stays explicit because OpenVMM refuses it for a snapshot that is not a NIC template.Docs:
doc/run.md(Networked snapshots),doc/usage.md(run table), design 3.1, 3.5.6, 5.5.5, and 5.5.7,nvx-rs/README.md,doc/ci.md.Tests
untiered-network-policycaptures an untiered NIC template in non-sandbox mode, on192.0.2.0/24, and restores it four times with--restore-network-policy: the template's own deny-all; egress rules admitting DNS over IPv4 and one host service; explicit host loopback and a forward, with egress rules admitting DNS over IPv6 alone; and private-network ingress with allowed egress. For each restore:nvx-ingressmust run exactly when the policy publishes listeners.OpenVMM must also refuse a restore that keeps the template's policy but passes another, and one that passes ingress without the option. No restore may change the template.
Unit tests:
nvx-timepacket hook;nvx-ingress start's usage errors and startup failure;nvx-snapshot's network stage, ordering against the acknowledgement, and failures, including an announcer that does not start;[DO NOT REVIEW] microVM: let an untiered restore install a new network policy on a NIC template openvmm#4831 adds unit and golden tests for the tier check, packet v5 encoding and decoding, and the network settings of ten policies.
Evidence
Hosts:
nvx-kvm-test(Linux/KVM, Azure Linux 3, kernel 6.6),nvx-mshv-test(Linux/MSHV, Azure Linux 3, kernel 6.18.34.mshv3), andnvx-whp-test(Windows/WHP, build 26100), all virtual machines. Each ran in a separate worktree, with OpenVMM built there from a cleanf1eb78302and the other artifacts of development releasev0.1.0-dev.bad440220c67. Each Linux host rebuilt the Alpine initramfs from this branch, and the WHP host used the KVM host's.untiered-network-policyat50d841d6c(= head but the shard plan)816d6bf47¹¹
816d6bf47differs from the head only in the new scenario's host-side checks and template subnet, their tests and docs, and the shard plan. Guest, OpenVMM, and every other scenario are identical.Every restore reported the same outcomes on every backend. On KVM:
deny-all--network-egress deny --host-loopback denyresolver=none announcer=none allowed=denied other=deniedegress-dns192.0.2.1/32:udp:53,192.0.2.1/32:tcp:ALLOWEDresolver=192.0.2.1 announcer=none allowed=reached other=deniedNVX-POST-RESTORE-OK: tier=legacyhost-loopbackfd00::c000:201/128:udp:53,--host-loopback allow, forwardresolver=fd00::c000:201 announcer=running allowed=denied other=deniedNVX-POST-RESTORE-OK: tier=legacyprivate-ingress--network-egress allow --network-ingress allow --host-loopback allowresolver=192.0.2.1 announcer=running allowed=reached other=reachedNVX-POST-RESTORE-OK: tier=legacy10.0.0.14)The refusals failed before the guest ran:
restore-time egress policy does not match the snapshot contract(another policy without the option);microVM snapshots do not support --network-ingress allow, ..., except at a restore with --restore-network-policy.Negative checks on KVM:
b92daa350(before OpenVMM prerequisite: a new network policy at the restore of a NIC template without a tier #942): the first restore fails with--restore-network-policy requires a platform snapshot.NVX-TIME-ABI-VIOLATION: ... code=G_REPAIR_PACKET ... detail="restore packet version is 5", so the guest never ignores the new settings.Validation
nvx-kvm-test:guest_tools(38 passed), andguest::ingressplusguest::timelib tests (161 passed);nvx-snapshotandinit.nvx.py verify, ruff check and format, pyright (Linux and Windows), andcheck_markdown_links.py;test_nvx_tools.py,test_markdown_links.py,test_time_abi.py,test_module_diagrams.py, andtest_guest_binaries.py;test_microvm_tests.pypasses, except 7 subtests of the existingtest_host_loopback_allow_script_checks_publication_and_trafficandtest_private_network_ingress_script_checks_policy_and_traffic. Those time out under Gitsh.exeon this machine, identically at the unchanged base.openvmm_vectorsandopenvmm_cpu_profiles; docs; MSRV; the runtime adapter.Review
Copilot's review raised two points, both fixed in this branch:
nvx-snapshotchecked only thatnvx-ingresswas spawned, not that it started.nvx-ingress startnow confirms that the announcer's socket is bound and its program runs before the acknowledgement.192.0.2.0/24, since Consomme refuses ingress peers inside the guest's subnets, and the hosts sit on10.0.0.0/24.