Skip to content

openvmm: let a platform restore attach an aggregate to a dormant filesystem slot #948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pedro Henrique Penna (ppenna) merged 4 commits into from
Oct 29, 2026
Merged

openvmm: let a platform restore attach an aggregate to a dormant filesystem slot #948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2052335
openvmm: let a platform restore attach an aggregate to a dormant slot
ppenna Oct 28, 2026
464b891
doc: map the aggregate export commits in the OpenVMM roadmap
ppenna Oct 28, 2026
3bdcee7
doc: describe the aggregate export of a platform restore
ppenna Oct 28, 2026
c241bbb
scripts: test aggregate exports at platform restores
ppenna Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions doc/ci.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,26 @@ mount the share, read the file that the host wrote, and write one that the
host then reads, and the restore may not change the snapshot. Until OpenVMM's
virtio-fs queue workers waited for the gate to release (#904), the clone's
mount succeeded but its first read never returned.
The `platform-aggregate-export` scenario covers an aggregate of host
directories and files that a platform restore attaches to the filesystem slot
that the capture left dormant, as a clone's export does. It captures a
platform snapshot without a filesystem and restores it with the aggregate of
the `filesystem-shares` scenario: a read-write workspace and a read-only tool
cache, each with a denied path, a read-write data directory whose root exposes
only its allowed paths and whose notes alone are writable, and a read-only and
a read-write file. The clone keeps its captured command line, which names no
filesystem, so its guest must mount the aggregate itself, find the five
children in its read-only root, which only root can enter, check each child's
kind, mode, and policy, and bind a directory and a file child, and the host
must then find exactly the writes that the policy admits. Before the guest
runs, OpenVMM must refuse the restores whose aggregates a cold boot refuses,
with overlapping directories, a file inside a directory child, a child of the
wrong kind, a repeated name, a policy path of a file child, one outside every
child, or a relative one, and on a Windows host caller ownership, and no
restore may change the snapshot. `filesystem-snapshot` also requires OpenVMM
to refuse an aggregate for the dormant slot of its untiered snapshot, which
binds only a `--mount`, because only a platform snapshot's dormant slot binds
an aggregate.
The `filesystem-snapshot` scenario's blockless capture of a live share covers
the write-back of shared mappings (#555). A guest process dirties a page of a
file in the share through a shared mapping with `nvx-device-io map` and holds
Expand Down
39 changes: 22 additions & 17 deletions doc/openvmm-upstream-roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,20 @@
- Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR
summary, and commit mapping below cover positions 001-157 and the fixups at positions
334-340, 344, and 345.
- NVX's pinned OpenVMM revision: `b92daa350c93914b2eb556acf47886cc313d396b`, position 352.
Positions 350-352, which microsoft/openvmm#4773 proposes for `feature/microvm/dev`, are
three commits on position 349, `d2338ea22`, the head of `feature/microvm/dev` since
microsoft/openvmm#4769 merged positions 348 and 349. Those are two commits on position 347,
`57d0a18b3`, which microsoft/openvmm#4767 merged with position 346. Positions 346 and 347 are
two commits on position 345, `e68871cc6`, which microsoft/openvmm#4762 merged with position
344. Positions 344 and 345 are two commits on position 343, `a12d9167f`, which
microsoft/openvmm#4749 merged with positions 341 and 342. Positions 341-343 are three commits
on position 340, `2b681564d`, which microsoft/openvmm#4735 merged with position 339.
Positions 339 and 340 are two commits on position 338, `8a0d945ed`, which
microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head of the
`feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 195 commits past the
mapped tip on the same first-parent chain. Positions 158-333, 341-343, and 346-352 are not
- NVX's pinned OpenVMM revision: `832d5bb48dadd2cc5221f2c8b8905e6b80d13d53`, position 354.
Positions 353 and 354, which microsoft/openvmm#4775 proposes for `feature/microvm/dev`, are
two commits on position 352, `b92daa350`, the head of `feature/microvm/dev` since
microsoft/openvmm#4773 merged positions 350-352. Those are three commits on position 349,
`d2338ea22`, which microsoft/openvmm#4769 merged with position 348. Positions 348 and 349 are
two commits on position 347, `57d0a18b3`, which microsoft/openvmm#4767 merged with position
346. Positions 346 and 347 are two commits on position 345, `e68871cc6`, which
microsoft/openvmm#4762 merged with position 344. Positions 344 and 345 are two commits on
position 343, `a12d9167f`, which microsoft/openvmm#4749 merged with positions 341 and 342.
Positions 341-343 are three commits on position 340, `2b681564d`, which microsoft/openvmm#4735
merged with position 339. Positions 339 and 340 are two commits on position 338, `8a0d945ed`,
which microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head
of the `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 197 commits past
the mapped tip on the same first-parent chain. Positions 158-333, 341-343, and 346-354 are not
yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip).
- Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base.

Expand Down Expand Up @@ -312,8 +313,8 @@ The physical branch is one first-parent chain:
-> PR14 fixup [336] -> PR09 fixup [337]
-> PR02 fixup [338] -> PR20 fixup [339] -> PR23 fixup [340]
-> unassigned [341-343] -> PR18 fixups [344-345]
-> unassigned [346-352]
-> b92daa350 (NVX pin)
-> unassigned [346-354]
-> 832d5bb48 (NVX pin)
```

For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main,
Expand All @@ -335,8 +336,8 @@ PR18's position 111.

## Commits after the mapped tip

The NVX pin extends the chain past position 157 without merges. Of the 195 later commits,
positions 158-333, 341-343, and 346-352 are not yet classified or assigned to PRs, positions
The NVX pin extends the chain past position 157 without merges. Of the 197 later commits,
positions 158-333, 341-343, and 346-354 are not yet classified or assigned to PRs, positions
334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02
fixup, positions 339 and 340 are PR20 and PR23 fixups, and positions 344 and 345 are PR18
fixups.
Expand Down Expand Up @@ -386,13 +387,17 @@ This table groups them by topic in chain order; ranges are inclusive, as in the
| 347 | `57d0a18b3` | 1 | `--restore-network-policy` lets a platform restore install the policy of its own options on a NIC template, including private-network ingress, explicit host loopback, and forwards, before any vCPU runs, while the NIC keeps its identity and the contract keeps the template's deny-all (microsoft/openvmm#4767) |
| 348-349 | `689577604..d2338ea22` | 2 | A second revision of the Genoa CPU profile, `amd.genoa.v2`, which `auto` selects: `amd.genoa.v1` without the basic leaves above `0xd` and without `LFENCE` serialization, NoSmmCtlMsr, and the TSA immunities in `0x80000021`, none of which the Microsoft hypervisor of the Azure Genoa VMs fingerprinted so far presents, with the nested-WHP Genoa host of microsoft/nvx#928 as a test fixture (microsoft/openvmm#4769); revises position 282's Genoa profile |
| 350-352 | `8c1772d83..b92daa350` | 3 | HTTP blob disks bound each request in time, retry the ones that fail transiently on a new connection, and show them in inspect, so that a network path that silently stops delivering a connection's packets no longer hangs a guest's disk read (microsoft/openvmm#4773) |
| 353 | `057a2eaa6` | 1 | Test that the microVM virtio-fs device takes a dormant slot's state into the device of an aggregate of a directory and a file, which then saves the aggregate |
| 354 | `832d5bb48` | 1 | A platform restore may bind an aggregate, `--mount-aggregate` with its children, policy paths, and owner mode, to the filesystem slot that the snapshot left dormant, validated as at a cold boot, while the slot's device and the captured command line stay as captured; other tiers keep binding only a `--mount` (microsoft/openvmm#4775) |

Some of these commits change mapped work rather than add to it. Positions 246-250 remove the
legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes
manifest version 6 the only snapshot format and changes files that PR11 and PR17 also change.
Positions 342 and 343 change the sandbox block attachment of PR19's position 115 and the
restore of positions 285 and 286. Positions 346 and 347 change the snapshot network policy and
the forward refusal of PR21's position 129 and the snapshot refusals of positions 306 and 307.
Position 354 changes the dormant-slot restore of PR18's position 113, as the aggregates of
position 293 restricted it.
Assigning the later commits therefore means revisiting those PRs, not only appending new ones.
Positions 350-352, by contrast, change only `disk_blob`, which upstream main and the fork base
have alike and no other commit of the chain changes, so they can go upstream on their own.
Expand Down
38 changes: 37 additions & 1 deletion doc/run.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,39 @@ without a NIC, and for a NIC captured with any other policy, and every other
[networked restore](#networked-snapshots) requires the captured policy.
`nvx-rs run` does not pass the option.

### Platform filesystem exports

A platform snapshot captured without `--mount` or `--mount-aggregate` leaves
the virtio-fs slot dormant. Each restore may bind one directory to it with
`--mount`, or an aggregate of directories and files with `--mount-aggregate`
and its `--mount-child` options, for example:

```bash
./bin/openvmm \
--single-process \
--machine microvm \
--hypervisor kvm \
--restore-snapshot /var/lib/nvx/template \
--microvm-sandbox-block distro:file:/var/lib/nvx/image.erofs,ro \
--restore-absent-scratch \
--mount-aggregate /run/nvx/shares \
--mount-child 0,/srv/checkout,rw \
--mount-child 1,/opt/hostedtoolcache,ro \
--mount-child 2,/srv/app/config/settings.json,ro,file \
--mount-deny /srv/checkout/.git/credentials
```

OpenVMM validates the aggregate's children, their `--mount-deny`,
`--mount-allow`, and `--mount-write` paths, and the `--mount-owner` mode as it
validates a cold boot's, and attaches the aggregate before any vCPU runs. The
slot's device and the guest's kernel command line stay as captured, without a
`virtfs_*` or `nvx_share=` token, so the guest must learn of the export from
its host, mount tag `microvm` itself, and bind each child where it needs it.
The dormant slot of a snapshot of another tier, or without a tier, still
binds only a `--mount`, and a snapshot captured with an export requires that
export again. `nvx-rs run`, which passes no sandbox blocks, cannot restore a
platform snapshot.

## Migration from the retired profile

`microvm` is now the only selector and launches the contract previously named
Expand Down Expand Up @@ -747,7 +780,10 @@ Capture and restore inspect and reopen the guest's open files as OpenVMM's
user, so unless OpenVMM runs as root, they fail while the guest holds a file
that only its caller can reach or reopen, such as one open for writing.
A snapshot captured without a mapping may restore with one new `--mount` of a
directory, but not with several or with a file; after resume, mount it
directory, but not with several or with a file, because `nvx-rs run` restores
only snapshots without sandbox blocks, whose dormant slot takes no aggregate;
OpenVMM binds an aggregate to the dormant slot of a
[platform snapshot](#platform-filesystem-exports). After resume, mount it
explicitly inside the guest because the initramfs hook has already completed:

```sh
Expand Down
6 changes: 3 additions & 3 deletions doc/usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1104,9 +1104,9 @@ omit. Alpine remains the
default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject
the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and
the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`,
`platform-zero-fill`, `platform-filesystem`, `platform-network-policy`,
`managed-exec-config`, `sandbox-lifecycle`, and `sandbox-proxy-only`
scenarios. Ubuntu also rejects
`platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`,
`platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, and
`sandbox-proxy-only` scenarios. Ubuntu also rejects
the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots
the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task
detectors the guest's time ABI watcher reports; without `--scenario`, it runs
Expand Down
2 changes: 1 addition & 1 deletion openvmm
Open in desktop
Submodule openvmm updated 7 files
+12 −4 Guide/src/reference/devices/virtio/virtio-fs.md
+11 −5 Guide/src/reference/openvmm/management/cli.md
+3 −1 Guide/src/user_guide/openvmm/snapshots.md
+7 −2 openvmm/openvmm_entry/src/cli_args/microvm.rs
+213 −20 openvmm/openvmm_entry/src/microvm/filesystem.rs
+61 −1 openvmm/openvmm_entry/src/microvm/restore.rs
+72 −0 vm/devices/virtio/virtiofs/src/microvm/device.rs
28 changes: 17 additions & 11 deletions scripts/nvx_tools/microvm_shards.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
"lifecycle": 1.7,
"managed-lifecycle": 4.2,
"network-snapshot": 17.0,
"platform-aggregate-export": 3.4,
"platform-filesystem": 6.0,
"platform-network-policy": 13.7,
"platform-restore-processors": 16.0,
Expand All @@ -67,8 +68,9 @@
},
"shards": [
{
"seconds": 249.9,
"seconds": 251.4,
"scenarios": [
"command-line-limit",
"console-snapshot",
"directional-network-policy",
"endpoint-policy-snapshot",
Expand All @@ -87,9 +89,8 @@
]
},
{
"seconds": 249.5,
"seconds": 251.4,
"scenarios": [
"command-line-limit",
"console-exit",
"cpu-profile-fallback",
"denied-filesystem-paths",
Expand All @@ -104,6 +105,7 @@
"l3-l4-egress-port-ranges",
"l3-l4-egress-protocols",
"lifecycle",
"platform-aggregate-export",
"platform-filesystem",
"platform-network-policy",
"platform-zero-fill",
Expand Down Expand Up @@ -142,6 +144,7 @@
"lifecycle": 1.6,
"managed-lifecycle": 4.3,
"network-snapshot": 16.8,
"platform-aggregate-export": 4.5,
"platform-filesystem": 6.0,
"platform-network-policy": 16.5,
"platform-restore-processors": 16.0,
Expand All @@ -163,7 +166,7 @@
},
"shards": [
{
"seconds": 230.5,
"seconds": 232.5,
"scenarios": [
"console-snapshot",
"directional-network-policy",
Expand All @@ -173,8 +176,8 @@
"network-snapshot",
"platform-filesystem",
"platform-restore-processors",
"platform-zero-fill",
"restore-downtime",
"restore-memory",
"restore-processors",
"scratch-snapshot",
"smp",
Expand All @@ -184,7 +187,7 @@
]
},
{
"seconds": 230.0,
"seconds": 232.5,
"scenarios": [
"command-line-limit",
"console-exit",
Expand All @@ -202,9 +205,10 @@
"l3-l4-egress-protocols",
"lifecycle",
"managed-lifecycle",
"platform-aggregate-export",
"platform-network-policy",
"platform-zero-fill",
"private-network-ingress",
"restore-memory",
"sandbox-blocks",
"structured-outcome",
"time-abi-conformance",
Expand Down Expand Up @@ -238,6 +242,7 @@
"lifecycle": 2.1,
"managed-lifecycle": 17.8,
"network-snapshot": 17.6,
"platform-aggregate-export": 9.1,
"platform-filesystem": 6.0,
"platform-network-policy": 24.8,
"platform-restore-processors": 16.0,
Expand All @@ -259,7 +264,7 @@
},
"shards": [
{
"seconds": 327.1,
"seconds": 331.7,
"scenarios": [
"console-snapshot",
"denied-filesystem-paths",
Expand All @@ -268,6 +273,7 @@
"filesystem-snapshot",
"host-loopback-policy",
"network-snapshot",
"platform-aggregate-export",
"platform-filesystem",
"platform-restore-processors",
"restore-downtime",
Expand All @@ -276,12 +282,11 @@
"scratch-snapshot",
"smp",
"smp-snapshot",
"snapshot-tiers",
"virtio-net"
"snapshot-tiers"
]
},
{
"seconds": 327.2,
"seconds": 331.7,
"scenarios": [
"command-line-limit",
"console-exit",
Expand All @@ -305,6 +310,7 @@
"snapshot-core",
"structured-outcome",
"time-abi-conformance",
"virtio-net",
"workload-identity"
]
}
Expand Down
85 changes: 85 additions & 0 deletions scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
set -eu
on_exit() {
status=$?
if [ "$status" -ne 0 ]; then
nvx-exit "$status"
fi
}
trap on_exit EXIT
fail() {
echo "NVX-PLATFORM-AGGREGATE-FAIL $1"
exit "$2"
}
expect_error() {
case "$1" in
*"$2"*) ;;
*) fail "$3: $1" "$4" ;;
esac
}
# The restore attached an aggregate to the slot that the platform capture left
# dormant. The guest keeps its captured command line, which discovers the slot
# but names no filesystem, so the guest mounts the aggregate itself.
grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline || fail "discovery" 80
if grep -Eq '(^| )(virtfs_|nvx_share=)' /proc/cmdline; then
fail "command line" 81
fi
shares=/run/nvx/shares
mkdir -p "$shares"
mount -t virtiofs microvm "$shares" || fail "mount" 82
[ "$(ls "$shares" | tr '\n' ' ')" = "0 1 2 3 4 " ] || fail "children" 83
[ "$(stat -c %a "$shares")" = 500 ] || fail "root mode" 84

# The read-write workspace hides its denied path and accepts writes.
[ "$(cat "$shares/0/seed")" = NVX-WORKSPACE ] || fail "workspace read" 85
[ ! -e "$shares/0/secrets" ] || fail "workspace denied path" 86
printf 'NVX-GUEST-WRITE\n' >"$shares/0/from-guest" || fail "workspace write" 87
mkdir "$shares/0/guest-directory" || fail "workspace directory" 88

# The read-only tool cache hides its denied path and rejects writes.
[ "$(cat "$shares/1/tools/node")" = NVX-TOOL ] || fail "tool cache read" 89
[ ! -e "$shares/1/credentials" ] || fail "tool cache denied path" 90
if error=$(touch "$shares/1/mutation" 2>&1); then
fail "tool cache write" 91
fi
expect_error "$error" 'Read-only file system' "tool cache write" 92

# The data directory's root exposes only its allowed paths, and only the notes
# are writable.
[ "$(ls "$shares/2" | tr '\n' '|')" = "notes one.txt|public|" ] || fail "data listing" 93
[ ! -e "$shares/2/private.txt" ] || fail "data hidden path" 94
[ "$(cat "$shares/2/public/readme")" = NVX-PUBLIC ] || fail "data allowed path" 95
printf 'NVX-GUEST-NOTE\n' >>"$shares/2/notes one.txt" || fail "data writable path" 96
if error=$(touch "$shares/2/public/mutation" 2>&1); then
fail "data read-only path" 97
fi
expect_error "$error" 'Read-only file system' "data read-only path" 98
if error=$(touch "$shares/2/added" 2>&1); then
fail "data hidden root" 99
fi
expect_error "$error" 'Permission denied' "data hidden root" 100

# Each file child is the file alone, with its own mode.
[ -f "$shares/3" ] && [ -f "$shares/4" ] || fail "file children" 101
[ "$(cat "$shares/3")" = NVX-SETTINGS ] || fail "settings read" 102
if error=$(ls "$shares/3/secret.json" 2>&1); then
fail "settings directory" 103
fi
expect_error "$error" 'Not a directory' "settings directory" 104
if error=$( (printf 'append\n' >>"$shares/3") 2>&1); then
fail "settings write" 105
fi
expect_error "$error" 'Read-only file system' "settings write" 106
printf 'NVX-GUEST-OUTPUT\n' >"$shares/4" || fail "output write" 107
if error=$(rm -f "$shares/4" 2>&1); then
fail "output removal" 108
fi
expect_error "$error" 'Read-only file system' "output removal" 109

# The guest binds the children where it needs them, a file child at a file.
mkdir -p /workspace /config
mount --bind "$shares/0" /workspace || fail "workspace bind" 110
[ "$(cat /workspace/from-guest)" = NVX-GUEST-WRITE ] || fail "workspace bind read" 111
: >/config/settings.json
mount --bind "$shares/3" /config/settings.json || fail "settings bind" 112
[ "$(cat /config/settings.json)" = NVX-SETTINGS ] || fail "settings bind read" 113
echo NVX-PLATFORM-AGGREGATE-OK
Loading
Loading