Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/skills/nvx-openvmm-promote/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ git diff --check
`nvx-rs` reads copies of what it needs from OpenVMM, because the `nvx-rs` CI job cannot
check out the submodule: the control-session protocol vectors, excerpts of OpenVMM's
sources, and, in `nvx::contract`, the versions of OpenVMM's snapshot format, saved-state
schema, time ABI, and restore packet. Compare them with the revision's, as the NVX CLI
schema, time ABI, and restore packets. Compare them with the revision's, as the NVX CLI
jobs do:

```console
Expand Down
2 changes: 1 addition & 1 deletion doc/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ hypervisor identity, and runs the cross-vCPU warp test that the
[time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs
skew bound. `nvx-time` implements the time ABI's
[guest obligations](design/3.2-clock-and-time.md#3210-guest-obligations): the conformance
checks, the violation watcher, the restore packet v5 and time-sample v2
checks, the violation watcher, the restore packet v5 and v6 and time-sample v2
parsers, the snapshot agent's time steps, and the wall-clock discipline, which
applies the host's clock correction. Before shell-ready it only steps the
clock to host UTC and sets its rate; the other boot checks run in
Expand Down
24 changes: 23 additions & 1 deletion doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,28 @@ forward whose host port another socket holds, the option on a platform
snapshot captured with plain deny-all egress, which leaves host loopback
allowed, and the option on a cold boot, and no restore may change the
template.
The `untiered-network-policy` scenario covers a NIC template without a tier, a
snapshot without sandbox blocks, as the non-sandbox mode captures one, whose
NIC was captured with the complete deny-all policy. It restores the template
with `--restore-network-policy` four times: under the template's own deny-all,
under egress rules that admit DNS over IPv4 and one host service, with
explicit host loopback and a forward under egress rules that admit DNS over
IPv6 alone, and with private-network ingress and allowed egress. The
template's guest learned no DNS server and runs no announcer of its listeners,
so each restore's repair applies those of the restore's policy, which restore
packet version 6 carries, before it acknowledges the restore: the guest's
resolver must name exactly the policy's DNS server, and no gateway without
one, and the announcer must run exactly when the policy publishes the guest's
listeners. Each restore must reach through its gateway exactly the
host-loopback services that its policy admits, which must receive no other
request, and the host must reach the guest's server through the forward and
through the port that the policy publishes: on host loopback alone, which no
other host address reaches, or to the host's private networks, through a host
address other than loopback. It must find the server on no host loopback port
without a publication. Before the guest runs, OpenVMM must refuse a restore
that keeps the template's policy but passes another, and one that passes
private-network ingress without the option, and no restore may change the
template.
The `platform-filesystem` scenario covers a share that a platform restore
attaches to the filesystem slot that the capture left dormant, as a clone's
export does. It captures a platform snapshot without `--mount` and restores it
Expand Down Expand Up @@ -695,7 +717,7 @@ compares the submodule's control-session protocol vectors, and the lines of its
platform allowlist of kernel command-line tokens, with the copies that the
protocol and template tests read in the `nvx-rs` job, which cannot check out
OpenVMM, and the versions of OpenVMM's snapshot format, saved-state schema,
time ABI, and restore packet with those of `nvx::contract`;
time ABI, and restore packets with those of `nvx::contract`;
see [OpenVMM's vectors](../nvx-rs/tests/vectors/README.md#openvmms-vectors).
The ignored test `tests/openvmm_cpu_profiles.rs` likewise compares the
submodule's built-in CPU profiles with the copy of their identities from which
Expand Down
3 changes: 2 additions & 1 deletion doc/design/3.1-platform-control-device.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ machine omits, and carries:
console.
- The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet),
which gives a restored guest what it needs to repair its clocks,
processors, and memory, and the fresh random bytes from which it
processors, and memory, the network settings of a network policy that the
restore installed, and the fresh random bytes from which it
[refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh).
- Host time samples, which pair host UTC with the guest TSC and carry the
host's clock correction, from which the guest sets and disciplines its wall
Expand Down
30 changes: 25 additions & 5 deletions doc/design/3.5-network.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,11 +248,31 @@ failure, or a UDP query beyond that limit, receives `SERVFAIL`, and a TCP query
beyond it resets its connection.

The snapshot records the network profile, identity, and policy digest.
Restore reconstructs a fresh endpoint and requires the same network profile
and policy; native sockets, DNS requests, and flows are never serialized.
Published guest listeners and explicit host forwards are process-local
attachments, so capture and restore reject private-network ingress and
explicit host-loopback access, which create them.
Restore reconstructs a fresh endpoint and requires the same network profile,
and the same policy unless the NIC is a template: one captured with the
complete deny-all policy, which denies every guest frame, host loopback, and
inbound access, and keeps no proxy endpoint, so that nothing in the snapshot
depends on the policy beyond the NIC's identity. A restore of a template of
the platform tier, or of a snapshot without a tier, may install a policy of
its own with `--restore-network-policy`, which OpenVMM validates as at a cold
boot, binds to the captured identity, and installs, with its forwards, before
any vCPU runs, while the snapshot's contract keeps the template's policy. A
snapshot without a tier may hold a workload that has already run, but under
the deny-all that workload had no connectivity, so the restore grants it
connectivity that it never had. Native sockets, DNS requests, and flows are
never serialized. Published guest listeners and explicit host forwards are
process-local attachments, so capture rejects private-network ingress and
explicit host-loopback access, which create them, and so does every restore
that keeps the captured policy.

A restored guest keeps its captured command line, which under a template's
deny-all names no DNS server and asks for no listener announcements. A
platform clone takes both from its activation. The guest of a snapshot
without a tier takes them from version 6 of the restore packet, which carries
the new policy's DNS server and whether the guest announces its listeners
([Restore Packet](5.5-restore-protocol.md#555-restore-packet)), and its
repair applies them before it acknowledges the restore, which then gates
host input ([Guest Repair](5.5-restore-protocol.md#557-guest-repair)).

Capture quiesces the endpoint, drains completion ownership, and requires the
saved queues to contain no unrepresented RX or TX packets. It saves the queue
Expand Down
52 changes: 39 additions & 13 deletions doc/design/5.5-restore-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,21 +194,24 @@ tier or a processor target requires it.

Figure 5.7 shows the layout of the restore packet, and Table 5.7 lists its
flags. Every restore exposes one restore packet through the byte channel, for
every tier and for untiered snapshots.
every tier and for untiered snapshots. The packet is version 5, except at a
restore that installs a new network policy on a snapshot without a tier,
whose packet is version 6.

```text
offset size field
0 3 magic "OVR"
3 1 version = 5
3 1 version = 5, or 6 with network settings
4 1 flags
5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8
6 1 memory_range_count 0 unless MEMORY_TARGET
7 1 reserved = 0
7 1 network reserved = 0 in version 5
8 4 generation u32, little-endian, g >= 1
12 4 rate_deviation i32, little-endian, ppm scaled by 2^16
16 8 downtime_ns u64, little-endian, D
24 16*n ranges n x { u64 gpa_start, u64 length }, LE
24+16n 64 entropy
88+16n 16 dns_server version 6 only
```

**Figure 5.7.** Restore packet layout.
Expand All @@ -226,13 +229,30 @@ offset size field
Every field is sealed before the first restored processor runs. The packet
carries no host UTC: the guest takes a
[time sample](3.2-clock-and-time.md#329-time-samples) after it, which pairs
host UTC with the guest TSC on the host. A packet without ranges is 88 bytes.
The packet is authoritative, and the status bits mirror it; the
acknowledgement flag spares the guest from inferring gating from the tier and
targets. A VM process exposes at most one packet, and reading its last byte
consumes it, so a guest captured again in the same process never sees an
earlier packet. The guest also ignores a packet whose generation equals its
recorded generation.
host UTC with the guest TSC on the host. A packet without ranges is 88 bytes
in version 5 and 104 bytes in version 6. The packet is authoritative, and the
status bits mirror it; the acknowledgement flag spares the guest from
inferring gating from the tier and targets. A VM process exposes at most one
packet, and reading its last byte consumes it, so a guest captured again in
the same process never sees an earlier packet. The guest also ignores a packet
whose generation equals its recorded generation.

Version 6 carries the network settings of the policy that the restore
installed in place of a NIC template's deny-all, which the guest's captured
command line does not name
([Bounded State, Snapshots, and Restore](3.5-network.md#356-bounded-state-snapshots-and-restore)).
Bits 0 and 1 of its network byte give the family of the guest's DNS server, 0
for none, 1 for IPv4, or 2 for IPv6, and bit 2 asks the guest to announce its
listeners; the guest rejects a packet with family 3 or any other bit set. The
DNS server is the gateway address that a cold boot under the policy would
name ([DNS](3.5-network.md#355-dns)): an IPv4 address followed by 12 zero
bytes, an IPv6 address, or 16 zero bytes for none; the guest rejects an
unspecified address and a record that does not match its family. The guest
announces its listeners when the policy publishes them, through private-network
ingress or explicit host-loopback access. Such a restore always sets
`ACK_REQUIRED`, so the guest applies the settings before host input reaches
it, and a guest that reads version 5 alone rejects version 6 and so fails the
restore instead of ignoring its settings.

## 5.5.6 Generations

Expand All @@ -257,14 +277,20 @@ runs before the acknowledgement:
1. Read the byte-channel status, whose restore-packet bit is always set after
a restore.
2. Select the packet, read it four bytes per access, and validate its magic,
version, reserved bits, counts, generation, and generation ID.
version, reserved bits, counts, network settings, generation, and
generation ID.
3. Set the wall clock: in the same port transaction as the packet, take up to
three [time samples](3.2-clock-and-time.md#329-time-samples) of the
packet's generation, until one pairs host UTC with the guest TSC within
100 µs, step the clock by its offset, and hold the frequency correction
that its host clock correction gives, which carries the rate deviation,
within ±500 ppm.
4. Activate processors and memory as requested.
4. Activate processors and memory as requested, then apply the network
settings of a version 6 packet as the guest's init applies a cold boot's:
name the DNS server in `/etc/resolv.conf`, and, when the packet asks for
the announcements, start `nvx-ingress`, which announces the guest's
listeners to its IPv4 gateway, and wait until its socket is bound to a
privileged source port and its program runs.
5. Refresh entropy and identity, as
[Entropy Refresh](#558-entropy-refresh) describes.
6. Acknowledge through the snapshot port when `ACK_REQUIRED` is set.
Expand Down Expand Up @@ -321,7 +347,7 @@ As Table 5.8 shows, the refresh depends on the restore path.
| `platform` or `workload-start` tier (clone) | Reseeded before the acknowledgement with the packet's 64 fresh bytes | Machine ID and hostname refreshed; the runtime hook runs if present, and `workload-start` requires it |
| Untiered, with processors or memory to activate | Reseeded before the acknowledgement, as for a clone | Refreshed, as for a clone |
| `instance-checkpoint` tier (resume) | Not reseeded, by design; only the generation ID changes | Preserved |
| Untiered, with nothing to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved |
| Untiered, with no processors or memory to activate | Not reseeded; the fresh entropy is handed to the snapshot's caller | Preserved |

An instance checkpoint is not reseeded because it is single-use: its random
state continues in exactly one process, and keeping it preserves the
Expand Down
Loading
Loading