From 5d43e0a99f8d515b3c89e43a1eca6edd96ee10ae Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 14:03:08 -0700 Subject: [PATCH 1/5] guest: assemble the layered root over a tmpfs upper nvx::protocol::root defines two uppers for a layered root: the ext4 scratch disk, which nvx_scratch=0xd0006000,ext4 names, and RAM, which nvx_overlay_upper=ramfs names, with no scratch block device. OpenVMM admits the RAM layout with exactly one read-only distro block, and the guest library assembles it, but NVX's shell init, nvx-init-agent, refused the token, so no NVX sandbox could keep its upper in RAM (#558). The init now reads either upper with the parser's rules: each token once and with its one value, and exactly one upper. It refuses both tokens together, an image without an upper, and any other value, such as nvx_overlay_upper=tmpfs, with the parser's messages. The token keeps its name: it names a RAM-backed layout, and the guest mounts a tmpfs for it. For a RAM upper, the init mounts a tmpfs at /run/nvx/tmpfs, where nvx::guest::root mounts one too: nosuid, nodev, with mode 0755, the mode of an ext4 scratch's root, rather than tmpfs's world-writable default, and at the kernel's default size, half of the guest's RAM. The overlay keeps its options, metacopy=on and xino=on; the pinned kernel enables the tmpfs xattrs that metacopy needs. The teardown unmounts the upper that the init mounted, last, as it unmounted scratch. The parser also refuses a RAM upper beside a scratch block device, which it finds in the virtio_mmio.device= parameters as the kernel reads them, with their quoting and number formats. The init asks the kernel instead: before it mounts anything, it refuses a RAM upper if /proc/iomem lists a virtio-mmio device at 0xd0006000, where resolve_block_device finds the scratch disk. A scratch upper still fails when its device does not appear. The shared vectors of nvx-rs/tests/fixtures/root-tokens.json no longer expect the shell to refuse a RAM upper. test_nvx_tools.py now requires it to accept every EROFS layout over either upper that the parser accepts, and to read the tokens of the cases that the parser refuses for a scratch device, whose /proc/iomem check a new test covers. Other tests check the init's messages for conflicting uppers, the mounts of each upper, that the RAM check precedes every mount, and the teardown's order and failure message for the tmpfs upper. The protocol's documentation and doc/project-structure.md describe what the init assembles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/project-structure.md | 3 +- guest/common/nvx-init-agent | 107 ++++++++--- nvx-rs/src/protocol/root.rs | 5 +- nvx-rs/tests/fixtures/root-tokens.json | 48 ++--- scripts/test_nvx_tools.py | 244 ++++++++++++++++++++++--- 5 files changed, 334 insertions(+), 73 deletions(-) diff --git a/doc/project-structure.md b/doc/project-structure.md index 8215c22a9..ebdbed8ce 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -222,7 +222,8 @@ the entry script stages Alpine's musl loader and `setpriv` inside the workload's root. The sandbox scripts resolve fixed virtio-blk roles through -sysfs, assemble EROFS lower layers over ext4 scratch, place the workload in its +sysfs, assemble EROFS lower layers over the upper that the host declares, an +ext4 scratch disk or a tmpfs, place the workload in its cgroup before release, construct its mount/PID/UTS namespaces, enter its filesystem root after dropping capabilities, and retain the agent as the outer PID 1. In a managed sandbox, the agent supervises `nvx-managed-agent`, which diff --git a/guest/common/nvx-init-agent b/guest/common/nvx-init-agent index a5b225f99..dcad77a21 100644 --- a/guest/common/nvx-init-agent +++ b/guest/common/nvx-init-agent @@ -6,6 +6,8 @@ cmdline=$(cat /proc/cmdline) runtime=/run/nvx layers=$runtime/layers scratch=$runtime/scratch +# The mount point of a RAM upper, which nvx::guest::root names alike. +tmpfs=$runtime/tmpfs rootfs=$runtime/rootfs # The host export of a managed sandbox goes at $hostfs/root, where only root # can enter, as in the guest's plain path. @@ -194,7 +196,7 @@ read_layer_token() { return 0 ;; distro,0xd0003000,gpt | distro,0xd0003000,gpt:*) - fatal "nvx_layer=$1 names a GPT image, which this init cannot assemble: it assembles only an EROFS image over a scratch disk" + fatal "nvx_layer=$1 names a GPT image, which this init cannot assemble: it assembles only an EROFS image" ;; distro,0xd0003000,*) layer_format=${layer_image%%:*} @@ -211,20 +213,23 @@ read_layer_token() { # Reads the boot tokens of the layered root, with the rules of the host's # renderer, nvx::protocol::root: nvx_layer names the image of the distro block # device, and nvx_scratch=0xd0006000,ext4 or nvx_overlay_upper=ramfs the upper, -# each once. This init assembles only an EROFS image over the ext4 scratch -# disk, so it also refuses a GPT image and a RAM upper. Sets layer_uuid to the -# filesystem UUID that the image must have, or to nothing if the token names -# none. The tests replay nvx-rs/tests/fixtures/root-tokens.json through it. +# each once. This init assembles only an EROFS image, so it also refuses a GPT +# image. Sets layer_uuid to the filesystem UUID that the image must have, or to +# nothing if the token names none, and upper to scratch or ram. The parser also +# refuses a RAM upper beside a scratch block device, which this init checks +# against the kernel's devices instead, with refuse_scratch_device. The tests +# replay nvx-rs/tests/fixtures/root-tokens.json through it. read_root_tokens() { layer_seen=false scratch_seen=false + ram_seen=false layer_uuid= # Kernel command-line tokens are deliberately split on spaces. # shellcheck disable=SC2086 set -f for token in $cmdline; do case "$token" in - nvx_layer | nvx_scratch) + nvx_layer | nvx_scratch | nvx_overlay_upper) fatal "the $token kernel parameter has no value" ;; nvx_layer=*) @@ -240,16 +245,76 @@ read_root_tokens() { [ "${token#*=}" = 0xd0006000,ext4 ] || fatal "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not '${token#*=}'" ;; - nvx_overlay_upper | nvx_overlay_upper=*) - fatal "$token asks for a RAM upper, which this init cannot assemble: it assembles only an EROFS image over a scratch disk" + nvx_overlay_upper=*) + [ "$ram_seen" = false ] || + fatal "the nvx_overlay_upper kernel parameter is repeated" + ram_seen=true + [ "${token#*=}" = ramfs ] || + fatal "the nvx_overlay_upper kernel parameter must be ramfs, not '${token#*=}'" ;; esac done set +f + [ "$scratch_seen" = false ] || [ "$ram_seen" = false ] || + fatal "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper" [ "$layer_seen" = true ] || fatal "the command line names no distro layer: this init needs nvx_layer=distro,0xd0003000,erofs[:UUID]" - [ "$scratch_seen" = true ] || - fatal "the command line names no scratch disk: this init needs nvx_scratch=0xd0006000,ext4" + if [ "$scratch_seen" = true ]; then + upper=scratch + elif [ "$ram_seen" = true ]; then + upper=ram + else + fatal "the nvx_layer kernel parameter needs an upper, nvx_scratch or nvx_overlay_upper" + fi +} + +# Fails if the kernel has a virtio-mmio device at the scratch block device's +# MMIO base, 0xd0006000, which a RAM upper must not have. nvx::protocol::root +# reads the devices from the virtio_mmio.device= parameters of the command +# line, with the kernel's rules; this init asks the kernel instead, which lists +# the window of each device that it added in /proc/iomem, where +# resolve_block_device finds them too. +refuse_scratch_device() { + scratch_address=$(printf '%d' 0xd0006000) + while IFS=' ' read -r range separator platform _; do + [ "$separator" = : ] || continue + case "$platform" in virtio-mmio.*) ;; *) continue ;; esac + start=$(printf '%d' "0x${range%%-*}") || continue + [ "$start" -ne "$scratch_address" ] || + fatal "a RAM upper takes no scratch block device, but the kernel has a virtio-mmio device at 0xd0006000" + done &2 status=1 fi - if ! umount "$scratch"; then - echo "NVX-SANDBOX-ERROR: failed to unmount scratch" >&2 + if ! umount "$upper_point"; then + echo "NVX-SANDBOX-ERROR: failed to unmount $upper_name" >&2 status=1 fi echo "NVX-SANDBOX-EXIT: status=$status" @@ -589,30 +654,26 @@ teardown() { done } -mkdir -p "$layers" "$scratch" "$rootfs" +mkdir -p "$layers" "$rootfs" # init mounts /run before the time ABI boot check, whose state lives there. mountpoint -q /run || mount -t tmpfs -o mode=755,nosuid,nodev tmpfs /run || fatal "failed to mount runtime tmpfs" -mkdir -p "$layers" "$scratch" "$rootfs" +mkdir -p "$layers" "$rootfs" configure_cgroups layer_capacity=$(cmdline_value nvx_layer_capacity_sectors || true) read_root_tokens +# A RAM upper is refused beside a scratch block device before anything mounts. +[ "$upper" = scratch ] || refuse_scratch_device mount_layer distro 0xd0003000 "$layer_uuid" "$layer_capacity" layer_count=1 lowerdirs=$layers/distro -scratch_device=$(resolve_block_device 0xd0006000) || - fatal "scratch device at 0xd0006000 did not appear" -blockdev --flushbufs "$scratch_device" || - fatal "failed to drop cached blocks for scratch" -mount -t ext4 -o rw,nosuid,nodev "$scratch_device" "$scratch" || - fatal "failed to mount scratch from $scratch_device" -mkdir -p "$scratch/upper" "$scratch/work" +mount_upper mount -t overlay overlay \ - -o "lowerdir=$lowerdirs,upperdir=$scratch/upper,workdir=$scratch/work,metacopy=on,xino=on" \ + -o "lowerdir=$lowerdirs,upperdir=$upper_point/upper,workdir=$upper_point/work,metacopy=on,xino=on" \ "$rootfs" || fatal "failed to assemble the container overlay" entrypoint=$(cmdline_value nvx_entrypoint || true) diff --git a/nvx-rs/src/protocol/root.rs b/nvx-rs/src/protocol/root.rs index a2aea358d..a5fa91cd8 100644 --- a/nvx-rs/src/protocol/root.rs +++ b/nvx-rs/src/protocol/root.rs @@ -17,8 +17,9 @@ //! //! Guests read the tokens with [`RootLayout::parse`], which accepts exactly these forms and //! refuses every other, with no fallback that probes the image or the devices. NVX's shell init -//! reads them with the same rules, and refuses the GPT image and the RAM upper, which it cannot -//! assemble. +//! reads them with the same rules, and refuses the GPT image, which it cannot assemble. It finds a +//! scratch block device beside a RAM upper among the devices that the kernel added, rather than +//! on the command line. //! //! The tokens belong to the boot contract that //! [`BOOT_CONTRACT_VERSION`](crate::contract::BOOT_CONTRACT_VERSION) versions, so a change to diff --git a/nvx-rs/tests/fixtures/root-tokens.json b/nvx-rs/tests/fixtures/root-tokens.json index cb2afa8bd..3f717e019 100644 --- a/nvx-rs/tests/fixtures/root-tokens.json +++ b/nvx-rs/tests/fixtures/root-tokens.json @@ -1,18 +1,18 @@ { - "description": "Kernel command lines and the layout of a layered root that nvx::protocol::root parses from each, null for none, or the error with which it refuses the command line. The tests of nvx-rs replay them, and scripts/test_nvx_tools.py replays them through the shell init, guest/common/nvx-init-agent, which reads the same tokens: it accepts what the parser accepts of an EROFS image over scratch, and refuses everything else, with shell_error in its message where a case names one.", + "description": "Kernel command lines and the layout of a layered root that nvx::protocol::root parses from each, null for none, or the error with which it refuses the command line. The tests of nvx-rs replay them, and scripts/test_nvx_tools.py replays them through the shell init, guest/common/nvx-init-agent, which reads the same tokens: it accepts what the parser accepts of an EROFS image over either upper, and refuses everything else, with shell_error in its message where a case names one. It reads the tokens of a RAM upper beside a scratch block device too, and refuses that device when it finds it in /proc/iomem, where the kernel lists the virtio-mmio devices that it added.", "cases": [ {"name": "an EROFS image with its UUID over scratch", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, {"name": "an EROFS image without an identity over scratch", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}}, {"name": "a GPT image with its disk GUID over scratch", "command_line": "nvx_layer=distro,0xd0003000,gpt:fedcba98-7654-3210-fedc-ba9876543210 nvx_scratch=0xd0006000,ext4", "layout": {"format": "gpt", "identity": "fedcba98-7654-3210-fedc-ba9876543210", "upper": "scratch"}, "shell_error": "GPT image"}, {"name": "a GPT image without an identity over scratch", "command_line": "nvx_layer=distro,0xd0003000,gpt nvx_scratch=0xd0006000,ext4", "layout": {"format": "gpt", "identity": null, "upper": "scratch"}, "shell_error": "GPT image"}, - {"name": "an EROFS image with its UUID in RAM", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}, "shell_error": "RAM upper"}, - {"name": "an EROFS image without an identity in RAM", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, + {"name": "an EROFS image with its UUID in RAM", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, + {"name": "an EROFS image without an identity in RAM", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, {"name": "a GPT image with its disk GUID in RAM", "command_line": "nvx_layer=distro,0xd0003000,gpt:fedcba98-7654-3210-fedc-ba9876543210 nvx_overlay_upper=ramfs", "layout": {"format": "gpt", "identity": "fedcba98-7654-3210-fedc-ba9876543210", "upper": "ram"}, "shell_error": "GPT image"}, {"name": "a GPT image without an identity in RAM", "command_line": "nvx_layer=distro,0xd0003000,gpt nvx_overlay_upper=ramfs", "layout": {"format": "gpt", "identity": null, "upper": "ram"}, "shell_error": "GPT image"}, {"name": "the upper before the image", "command_line": "nvx_scratch=0xd0006000,ext4 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, - {"name": "the RAM upper before the image", "command_line": "nvx_overlay_upper=ramfs nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}, "shell_error": "RAM upper"}, + {"name": "the RAM upper before the image", "command_line": "nvx_overlay_upper=ramfs nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, {"name": "the tokens of a managed sandbox and of the VMM", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_entrypoint=/bin/sh nvx_hostname=nvx-sandbox nvx_arg=-c nvx_memory_max=268435456 nvx_pids_max=65 nvx_share=0-4f9cd4a1a4e2b5c3d6e7f8091a2b3c4d,/workspace,rw nr_cpus=1 nvx_workload_uid=65534 nvx_workload_gid=65534 nvx_lifecycle=managed virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11 virtio_mmio.device=0x1000@0xd0007000:3 nvx_control_tty=hvc2 virtfs_dir=/run/nvx/shares virtfs_tag=microvm virtfs_mode=rw virtfs_aggregate=1", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, - {"name": "a RAM upper beside the distro and control devices", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_entrypoint=/bin/sh virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0007000:3", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}, "shell_error": "RAM upper"}, + {"name": "a RAM upper beside the distro and control devices", "command_line": "quiet nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_entrypoint=/bin/sh virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0007000:3", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "ram"}}, {"name": "tabs and newlines between tokens", "command_line": "quiet\tnvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef\nnvx_scratch=0xd0006000,ext4\tnvx_sandbox=1", "layout": {"format": "erofs", "identity": "0123abcd-4567-89ef-0123-456789abcdef", "upper": "scratch"}}, {"name": "parameters whose names only begin like the boot tokens", "command_line": "nvx_layer_capacity_sectors=2048 nvx_layers=x nvx_scratchpad=1 nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 nvx_overlay_upper_size=1 xnvx_layer=1", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}}, {"name": "no boot tokens, as for a blockless root", "command_line": "quiet nvx_sandbox=1 nvx_entrypoint=/bin/sh", "layout": null}, @@ -51,34 +51,34 @@ {"name": "two images of different formats", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_layer=distro,0xd0003000,gpt nvx_scratch=0xd0006000,ext4", "error": "the nvx_layer kernel parameter is repeated"}, {"name": "an image without an upper", "command_line": "quiet nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef", "error": "the nvx_layer kernel parameter needs an upper, nvx_scratch or nvx_overlay_upper"}, {"name": "a scratch upper without an image", "command_line": "quiet nvx_scratch=0xd0006000,ext4", "error": "the nvx_scratch kernel parameter needs an image, which nvx_layer names"}, - {"name": "a RAM upper without an image", "command_line": "quiet nvx_overlay_upper=ramfs", "error": "the nvx_overlay_upper kernel parameter needs an image, which nvx_layer names", "shell_error": "RAM upper"}, + {"name": "a RAM upper without an image", "command_line": "quiet nvx_overlay_upper=ramfs", "error": "the nvx_overlay_upper kernel parameter needs an image, which nvx_layer names"}, {"name": "the scratch form without a filesystem", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000", "error": "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not '0xd0006000'"}, {"name": "another scratch filesystem", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,xfs", "error": "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not '0xd0006000,xfs'"}, {"name": "the image's address for the scratch block device", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0003000,ext4", "error": "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not '0xd0003000,ext4'"}, {"name": "a scratch upper with an empty value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=", "error": "the nvx_scratch kernel parameter must be 0xd0006000,ext4, not ''"}, {"name": "a scratch upper without a value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch", "error": "the nvx_scratch kernel parameter has no value"}, {"name": "two scratch uppers", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 nvx_scratch=0xd0006000,ext4", "error": "the nvx_scratch kernel parameter is repeated"}, - {"name": "two RAM uppers", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs nvx_overlay_upper=ramfs", "error": "the nvx_overlay_upper kernel parameter is repeated", "shell_error": "RAM upper"}, - {"name": "both uppers", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_overlay_upper=ramfs", "error": "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper", "shell_error": "RAM upper"}, - {"name": "both uppers, the RAM upper first", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_scratch=0xd0006000,ext4", "error": "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper", "shell_error": "RAM upper"}, + {"name": "two RAM uppers", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs nvx_overlay_upper=ramfs", "error": "the nvx_overlay_upper kernel parameter is repeated"}, + {"name": "both uppers", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_scratch=0xd0006000,ext4 nvx_overlay_upper=ramfs", "error": "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper"}, + {"name": "both uppers, the RAM upper first", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs nvx_scratch=0xd0006000,ext4", "error": "a layered root has one upper, but the command line names both nvx_scratch and nvx_overlay_upper"}, {"name": "another RAM upper", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=tmpfs", "error": "the nvx_overlay_upper kernel parameter must be ramfs, not 'tmpfs'"}, {"name": "a RAM upper with an empty value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=", "error": "the nvx_overlay_upper kernel parameter must be ramfs, not ''"}, {"name": "a RAM upper without a value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper", "error": "the nvx_overlay_upper kernel parameter has no value"}, - {"name": "a RAM upper with a scratch block device", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device that a dashed parameter adds", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio-mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device at an address in capitals", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4K@0XD0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device at a decimal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@3489685504:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device at an octal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@032000060000:11:5", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device at a negative address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@-0xffffffff2fffa000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device at an address past 64 bits", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0x100000000d0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device in a quoted parameter", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs \"virtio_mmio.device=0x1000@0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device whose quoted value spaces its address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=\"0x1000@ 0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper with a scratch block device after a vertical tab", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs quiet\u000bvirtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000", "shell_error": "RAM upper"}, - {"name": "a RAM upper beside a device token without an address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000 virtio_mmio.device=0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, - {"name": "a RAM upper beside a scratch device token whose IRQ is 0", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:0", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, - {"name": "a RAM upper beside a scratch device token with text after its IRQ", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:11:", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, - {"name": "a RAM upper beside a scratch device token in a quoted value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs x=\"y virtio_mmio.device=0x1000@0xd0006000:11\"", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, - {"name": "a RAM upper beside a scratch device token after --", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs -- virtio_mmio.device=0x1000@0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}, "shell_error": "RAM upper"}, + {"name": "a RAM upper with a scratch block device", "command_line": "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device that a dashed parameter adds", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio-mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device at an address in capitals", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4K@0XD0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device at a decimal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@3489685504:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device at an octal address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=4096@032000060000:11:5", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device at a negative address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@-0xffffffff2fffa000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device at an address past 64 bits", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0x100000000d0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device in a quoted parameter", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs \"virtio_mmio.device=0x1000@0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device whose quoted value spaces its address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=\"0x1000@ 0xd0006000:11\"", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper with a scratch block device after a vertical tab", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs quiet\u000bvirtio_mmio.device=0x1000@0xd0006000:11", "error": "a RAM upper takes no scratch block device, but the command line adds a virtio-mmio device at 0xd0006000"}, + {"name": "a RAM upper beside a device token without an address", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000 virtio_mmio.device=0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token whose IRQ is 0", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:0", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token with text after its IRQ", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs virtio_mmio.device=0x1000@0xd0006000:11:", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token in a quoted value", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs x=\"y virtio_mmio.device=0x1000@0xd0006000:11\"", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, + {"name": "a RAM upper beside a scratch device token after --", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_overlay_upper=ramfs -- virtio_mmio.device=0x1000@0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "ram"}}, {"name": "a scratch upper beside its block device", "command_line": "nvx_layer=distro,0xd0003000,erofs nvx_scratch=0xd0006000,ext4 virtio_mmio.device=0x1000@0xd0003000:4 virtio_mmio.device=0x1000@0xd0006000:11", "layout": {"format": "erofs", "identity": null, "upper": "scratch"}} ] } diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index c118e7ebe..7cc1df1e4 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -18720,7 +18720,12 @@ def _run( return result, log def _run_teardown( - self, script: str, *, shares: tuple[str, ...], failing: str = "" + self, + script: str, + *, + shares: tuple[str, ...], + failing: str = "", + upper: str = "scratch", ) -> tuple[subprocess.CompletedProcess[str], list[str]]: runtime = self.root / "run" runtime.mkdir(exist_ok=True) @@ -18737,6 +18742,11 @@ def _run_teardown( mountpoints = "|".join( f"{self.rootfs.as_posix()}{target}" for target in reversed(shares) ) + # mount_upper sets the upper's mount point and name. + upper_point, upper_name = { + "scratch": ("$runtime/scratch", "scratch"), + "ram": ("$runtime/tmpfs", "the tmpfs upper"), + }[upper] result = subprocess.run( [ self.shell, @@ -18752,7 +18762,7 @@ def _run_teardown( "runtime=$1\nlog=$2\nfailing=$4\n" "share_mountpoints=$(printf '%s' \"$3\" | tr '|' '\\n')\n" "rootfs=$runtime/rootfs\nlayers=$runtime/layers\n" - "scratch=$runtime/scratch\n" + f'upper_point={upper_point}\nupper_name="{upper_name}"\n' 'umount() { printf "umount %s\\n" "$1" >>"$log"; ' '[ "$1" != "$failing" ]; }\n' 'mountpoint() { [ "$2" = "$layers/distro" ]; }\n' @@ -18848,6 +18858,42 @@ def test_agent_teardown_unmounts_share_before_overlay_layers_and_scratch(self): self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(log[0], f"umount {runtime}/rootfs") + def test_agent_teardown_unmounts_the_tmpfs_upper_last(self): + runtime = (self.root / "run").as_posix() + result, log = self._run_teardown( + "teardown 0", shares=("/workspace",), upper="ram" + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + log, + [ + f"umount {self.rootfs.as_posix()}/workspace", + f"umount {runtime}/rootfs", + f"umount {runtime}/layers/distro", + f"umount {runtime}/tmpfs", + "exit 0", + ], + ) + + for upper, point, name in ( + ("scratch", "scratch", "scratch"), + ("ram", "tmpfs", "the tmpfs upper"), + ): + with self.subTest(upper=upper): + result, log = self._run_teardown( + "teardown 0", + shares=(), + failing=f"{runtime}/{point}", + upper=upper, + ) + self.assertEqual(result.returncode, 1, result.stdout) + self.assertEqual( + result.stderr, + f"NVX-SANDBOX-ERROR: failed to unmount {name}\n", + ) + self.assertEqual(log[-2:], [f"umount {runtime}/{point}", "exit 1"]) + self.assertIn("NVX-SANDBOX-EXIT: status=1", result.stdout) + def test_agent_teardown_unmounts_shares_in_reverse_mount_order(self): rootfs = self.rootfs.as_posix() runtime = (self.root / "run").as_posix() @@ -19398,6 +19444,14 @@ class SandboxRootTokenAgentTests(unittest.TestCase): Path(__file__).parents[1] / "nvx-rs" / "tests" / "fixtures" / "root-tokens.json" ) UUID = "0123abcd-4567-89ef-0123-456789abcdef" + # nvx::protocol::root refuses a RAM upper beside a scratch block device, + # which it finds in the virtio_mmio.device= parameters of the command line + # as the kernel reads them. The shell init reads the tokens of those cases, + # and refuses the device that the kernel lists in /proc/iomem instead. + SCRATCH_DEVICE_ERROR = ( + "a RAM upper takes no scratch block device, but the command line adds a " + "virtio-mmio device at 0xd0006000" + ) def setUp(self): shell = _posix_shell() @@ -19411,7 +19465,7 @@ def setUp(self): def _read(self, command_line: str) -> subprocess.CompletedProcess[str]: """Runs the agent's reader of the root's boot tokens on `command_line`, - and prints the UUID that it expects of the distro layer.""" + and prints the UUID that it expects of the distro layer and the upper.""" functions = "".join( _shell_function(self.source, name) for name in ("is_canonical_uuid", "read_layer_token", "read_root_tokens") @@ -19428,7 +19482,7 @@ def _read(self, command_line: str) -> subprocess.CompletedProcess[str]: 'fatal() { echo "FATAL: $*" >&2; exit 125; }\n' f"{functions}" "read_root_tokens\n" - 'echo "layer_uuid=$layer_uuid"\n' + 'echo "layer_uuid=$layer_uuid upper=$upper"\n' ), text=True, capture_output=True, @@ -19437,31 +19491,35 @@ def _read(self, command_line: str) -> subprocess.CompletedProcess[str]: def test_agent_reads_the_boot_tokens_as_nvx_rs_reads_them(self): cases = json.loads(self.FIXTURE.read_text(encoding="utf-8"))["cases"] - accepted = 0 + accepted: set[str] = set() + devices = 0 for case in cases: with self.subTest(case=case["name"]): result = self._read(case["command_line"]) layout = case.get("layout") - if ( - layout is not None - and layout["format"] == "erofs" - and layout["upper"] == "scratch" - ): + if case.get("error") == self.SCRATCH_DEVICE_ERROR: + self.assertNotIn("shell_error", case) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertRegex(result.stdout, r"\Alayer_uuid=\S* upper=ram\n\Z") + devices += 1 + continue + if layout is not None and layout["format"] == "erofs": + self.assertNotIn("shell_error", case) self.assertEqual(result.returncode, 0, result.stderr) identity = layout["identity"] or "" - self.assertEqual(result.stdout, f"layer_uuid={identity}\n") - accepted += 1 + self.assertEqual( + result.stdout, + f"layer_uuid={identity} upper={layout['upper']}\n", + ) + accepted.add(layout["upper"]) continue self.assertEqual(result.returncode, 125, result.stdout) self.assertRegex(result.stderr, r"\AFATAL: [^\n]+\n\Z") self.assertIn(case.get("shell_error", ""), result.stderr) - self.assertGreater(accepted, 0) + self.assertEqual(accepted, {"scratch", "ram"}) + self.assertGreater(devices, 0) def test_agent_explains_the_roots_that_it_cannot_assemble(self): - unsupported = ( - "which this init cannot assemble: it assembles only an EROFS image " - "over a scratch disk" - ) for command_line, message in ( ( f"nvx_layer=distro,0xd0003000,{self.UUID} nvx_scratch=0xd0006000,ext4", @@ -19469,15 +19527,40 @@ def test_agent_explains_the_roots_that_it_cannot_assemble(self): f"gpt: 'distro,0xd0003000,{self.UUID}'", ), ( - f"nvx_layer=distro,0xd0003000,gpt:{self.UUID} " - "nvx_scratch=0xd0006000,ext4", + f"nvx_layer=distro,0xd0003000,gpt:{self.UUID} nvx_overlay_upper=ramfs", f"nvx_layer=distro,0xd0003000,gpt:{self.UUID} names a GPT image, " - f"{unsupported}", + "which this init cannot assemble: it assembles only an EROFS image", + ), + ): + with self.subTest(command_line=command_line): + result = self._read(command_line) + self.assertEqual(result.returncode, 125, result.stdout) + self.assertEqual(result.stderr, f"FATAL: {message}\n") + + def test_agent_requires_one_upper(self): + layer = f"nvx_layer=distro,0xd0003000,erofs:{self.UUID}" + for command_line, message in ( + ( + f"{layer} nvx_overlay_upper=ramfs nvx_scratch=0xd0006000,ext4", + "a layered root has one upper, but the command line names both " + "nvx_scratch and nvx_overlay_upper", + ), + ( + f"quiet {layer}", + "the nvx_layer kernel parameter needs an upper, nvx_scratch or " + "nvx_overlay_upper", + ), + ( + f"{layer} nvx_overlay_upper", + "the nvx_overlay_upper kernel parameter has no value", + ), + ( + f"{layer} nvx_overlay_upper=tmpfs", + "the nvx_overlay_upper kernel parameter must be ramfs, not 'tmpfs'", ), ( - f"nvx_layer=distro,0xd0003000,erofs:{self.UUID} " - "nvx_overlay_upper=ramfs", - f"nvx_overlay_upper=ramfs asks for a RAM upper, {unsupported}", + f"{layer} nvx_overlay_upper=ramfs nvx_overlay_upper=ramfs", + "the nvx_overlay_upper kernel parameter is repeated", ), ): with self.subTest(command_line=command_line): @@ -19485,6 +19568,121 @@ def test_agent_explains_the_roots_that_it_cannot_assemble(self): self.assertEqual(result.returncode, 125, result.stdout) self.assertEqual(result.stderr, f"FATAL: {message}\n") + def test_agent_refuses_a_scratch_device_beside_a_ram_upper(self): + iomem = self.root / "iomem" + function = _shell_function(self.source, "refuse_scratch_device").replace( + "/proc/iomem", iomem.as_posix() + ) + for content, refused in ( + ("", False), + ( + "d0003000-d0003fff : virtio-mmio.0\n" + " d0003000-d0003fff : virtio-mmio.0\n" + "d0007000-d0007fff : virtio-mmio.1\n", + False, + ), + # Only a virtio-mmio device whose window starts at the base counts. + ("d0006000-d0006fff : reserved\n", False), + ("d0005000-d0006fff : virtio-mmio.2\n", False), + ( + "d0003000-d0003fff : virtio-mmio.0\n" + "d0006000-d0006fff : virtio-mmio.1\n" + " d0006000-d0006fff : virtio-mmio.1\n", + True, + ), + ): + with self.subTest(content=content): + iomem.write_text(content, encoding="utf-8") + result = subprocess.run( + [self.shell, "-s"], + input=( + "set -eu\n" + 'fatal() { echo "FATAL: $*" >&2; exit 125; }\n' + f"{function}refuse_scratch_device\necho accepted\n" + ), + text=True, + capture_output=True, + check=False, + ) + if refused: + self.assertEqual(result.returncode, 125, result.stdout) + self.assertEqual( + result.stderr, + "FATAL: a RAM upper takes no scratch block device, but the " + "kernel has a virtio-mmio device at 0xd0006000\n", + ) + else: + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout, "accepted\n") + + def test_agent_checks_a_ram_upper_before_it_mounts_anything(self): + check = self.source.index('\n[ "$upper" = scratch ] || refuse_scratch_device\n') + self.assertLess(self.source.index("\nread_root_tokens\n"), check) + self.assertLess(check, self.source.index("\nmount_layer distro ")) + self.assertLess( + self.source.index("\nmount_layer distro "), + self.source.index("\nmount_upper\n"), + ) + self.assertIn( + "upperdir=$upper_point/upper,workdir=$upper_point/work,metacopy=on,xino=on", + self.source, + ) + + def test_agent_mounts_the_upper_that_the_tokens_name(self): + runtime = (self.root / "run").as_posix() + log = self.root / "commands.log" + for upper, commands, point, name in ( + ( + "scratch", + [ + f"mkdir -p {runtime}/scratch", + "resolve 0xd0006000", + "blockdev --flushbufs /dev/vdb", + f"mount -t ext4 -o rw,nosuid,nodev /dev/vdb {runtime}/scratch", + f"mkdir -p {runtime}/scratch/upper {runtime}/scratch/work", + ], + f"{runtime}/scratch", + "scratch", + ), + ( + "ram", + [ + f"mkdir -p {runtime}/tmpfs", + f"mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs {runtime}/tmpfs", + f"mkdir -p {runtime}/tmpfs/upper {runtime}/tmpfs/work", + ], + f"{runtime}/tmpfs", + "the tmpfs upper", + ), + ): + with self.subTest(upper=upper): + log.unlink(missing_ok=True) + result = subprocess.run( + [self.shell, "-s", "--", runtime, log.as_posix(), upper], + input=( + "set -eu\n" + "runtime=$1\nlog=$2\nupper=$3\n" + "scratch=$runtime/scratch\ntmpfs=$runtime/tmpfs\n" + 'fatal() { echo "FATAL: $*" >&2; exit 125; }\n' + 'resolve_block_device() { echo "resolve $*" >>"$log"; ' + "echo /dev/vdb; }\n" + 'blockdev() { echo "blockdev $*" >>"$log"; }\n' + 'mount() { echo "mount $*" >>"$log"; }\n' + 'mkdir() { echo "mkdir $*" >>"$log"; }\n' + f"{_shell_function(self.source, 'mount_upper')}" + "mount_upper\n" + 'echo "upper_point=$upper_point upper_name=$upper_name"\n' + ), + text=True, + capture_output=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + result.stdout, f"upper_point={point} upper_name={name}\n" + ) + self.assertEqual(log.read_text(encoding="utf-8").splitlines(), commands) + def test_agent_checks_the_layer_uuid_only_when_the_token_names_one(self): log = self.root / "commands.log" for expected, outcome in ( From d3f784794872819ebaee267e1aab93339aa36fbe Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 14:10:53 -0700 Subject: [PATCH 2/5] guest: check the upper that the sandbox declares in the smoke probe nvx-sandbox-smoke, the probe that the sandbox scenarios run as the workload, failed unless the root overlay's upper directory was /run/nvx/scratch/upper. A sandbox whose upper is in RAM, which the init agent now assembles, keeps it at /run/nvx/tmpfs/upper instead, so the probe refused every such sandbox, and accepted a scratch upper whatever the host had asked for. The probe now reads the upper that the kernel command line declares, as it already reads nvx_hostname there: a tmpfs for nvx_overlay_upper=ramfs, and scratch otherwise. It fails unless the overlay's upper directory is on that upper, so a sandbox that declares one upper and mounts the other fails either way. It also fails unless the overlay keeps metacopy=on and xino=on, the options that the init agent sets: overlayfs drops metacopy without a word on an upper that refuses its xattrs, so the probe shows that the pinned kernel applies both options on either upper. The image check calls its write a write to the upper rather than a scratch-backed one. Both readers of the command line now read all of /proc/cmdline, as the init agent does, rather than its first line: a caller's --cmdline keeps its newlines, and the root's tokens and nvx_hostname follow it, so a sandbox launched with a multi-line --cmdline failed the hostname check and, with a RAM upper, the mount check. test_nvx_tools.py gives the mount check a command line, and covers a tmpfs upper that a RAM sandbox declares, the mismatch in both directions, tokens that merely resemble the RAM upper's, an overlay without either option, and RAM and hostname tokens that follow a line break. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/run.md | 8 +-- guest/common/nvx-sandbox-smoke | 51 +++++++++++++------ scripts/test_nvx_tools.py | 89 +++++++++++++++++++++++++++++----- 3 files changed, 120 insertions(+), 28 deletions(-) diff --git a/doc/run.md b/doc/run.md index 86f27fd13..f4ca2998d 100644 --- a/doc/run.md +++ b/doc/run.md @@ -830,7 +830,9 @@ check, it requires: PID namespace and the `--hostname` value as its host name; - exactly one mount at each of `/`, `/proc`, `/sys`, `/dev`, `/dev/pts`, and `/dev/shm`, none of which propagates to another namespace: an overlay root - over the EROFS layers whose upper directory is on scratch, `nosuid,nodev,noexec` + over the EROFS layers, with `metacopy=on` and `xino=on`, whose upper + directory is on the upper that the kernel command line declares, scratch, or + a tmpfs for `nvx_overlay_upper=ramfs`, `nosuid,nodev,noexec` procfs, read-only sysfs, and a private `/dev` tmpfs that holds only `fd`, `full`, `null`, `ptmx`, `pts`, `random`, `shm`, `stderr`, `stdin`, `stdout`, `tty`, `urandom`, and `zero`, with its own devpts instance; @@ -840,8 +842,8 @@ check, it requires: - no port I/O: a port read without the permission ends the reader with `SIGSEGV`, and a request for the permission fails; - the agent-owned `/container` cgroup; and -- Ubuntu image files that the workload cannot modify, and a scratch-backed - `/tmp` write that lands in the overlay. +- Ubuntu image files that the workload cannot modify, and a `/tmp` write that + lands in the overlay, on its upper. A managed workload that mounted into the agent's namespace would stack its runtime mounts on those of earlier requests, so the single-mount check also diff --git a/guest/common/nvx-sandbox-smoke b/guest/common/nvx-sandbox-smoke index f1e4ea458..a19f65568 100644 --- a/guest/common/nvx-sandbox-smoke +++ b/guest/common/nvx-sandbox-smoke @@ -86,9 +86,11 @@ check_namespaces() { [ "$init_uids" = "$workload_ids" ] || fail "PID 1 of the workload PID namespace runs as $init_uids" expected_hostname=nvx-sandbox - read -r kernel_command_line /usr/nvx-sandbox-image-write) 2>/dev/null; then fail "the workload can create files in image directory /usr" fi - printf 'scratch-backed-write\n' >/tmp/nvx-ubuntu-sandbox-smoke - [ "$(cat /tmp/nvx-ubuntu-sandbox-smoke)" = scratch-backed-write ] || - fail "the scratch-backed write did not persist" + printf 'upper-write\n' >/tmp/nvx-ubuntu-sandbox-smoke + [ "$(cat /tmp/nvx-ubuntu-sandbox-smoke)" = upper-write ] || + fail "the write to the upper did not persist" [ "$(stat -f -c %T /tmp/nvx-ubuntu-sandbox-smoke)" = overlayfs ] || - fail "the scratch-backed write did not land in the overlay" + fail "the write to the upper did not land in the overlay" } # Starts sleepers until a fork fails or $2 have started, then stops and reaps diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 7cc1df1e4..3ff2251ac 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -19971,6 +19971,13 @@ def test_eperm_share_requires_reads_to_fail_with_eperm_and_writes_to_fail(self): "25 24 0:25 / /dev/pts rw,relatime - devpts devpts rw,mode=620,ptmxmode=666\n" "26 24 0:26 / /dev/shm rw,nosuid,nodev,relatime - tmpfs tmpfs rw\n" ) +# The kernel command line of a sandbox whose upper is on scratch, which the +# smoke check reads from /proc/cmdline. +_SCRATCH_COMMAND_LINE = ( + "console=hvc0 nvx_sandbox=1 " + "nvx_layer=distro,0xd0003000,erofs:0123abcd-4567-89ef-0123-456789abcdef " + "nvx_scratch=0xd0006000,ext4" +) _SANDBOX_STATUS = ( "Name:\tsh\nUid:\t65534\t65534\t65534\t65534\n" "Gid:\t65534\t65534\t65534\t65534\nGroups:\t\nNoNewPrivs:\t1\n" @@ -20101,6 +20108,13 @@ def run( self.assertEqual(result.stdout, "checked\n") result = run(private, command_line="console=hvc0", hostname="nvx-sandbox") self.assertEqual(result.returncode, 0, result.stderr) + # A caller's --cmdline keeps its newlines, so a token may follow one. + multiline = "console=hvc0 quiet\nloglevel=0\nnvx_hostname=nvx-lifecycle" + result = run(private, command_line=multiline) + self.assertEqual(result.returncode, 0, result.stderr) + result = run(private, command_line=multiline, hostname="nvx-sandbox") + self.assertEqual(result.returncode, 1, result.stdout) + self.assertIn("expected nvx-lifecycle", result.stderr) for name, initial in ( ("mnt", "mnt:[4026531832]"), @@ -20128,14 +20142,20 @@ def run( self.assertEqual(result.returncode, 1, result.stdout) self.assertIn("expected nvx-lifecycle", result.stderr) - def test_mount_check_requires_private_runtime_mounts_once(self): - functions = ("mount_records", "require_mount", "check_mounts") - mountinfo = self._fixture("mountinfo", _SANDBOX_MOUNTINFO) - result = self._run( - functions, + def _check_mounts( + self, mountinfo: str, command_line: str + ) -> subprocess.CompletedProcess[str]: + return self._run( + ("mount_records", "require_mount", "check_mounts"), "check_mounts\necho checked", - {"/proc/self/mountinfo": mountinfo}, + { + "/proc/self/mountinfo": self._fixture("mountinfo", mountinfo), + "/proc/cmdline": self._fixture("cmdline", command_line + "\n"), + }, ) + + def test_mount_check_requires_private_runtime_mounts_once(self): + result = self._check_mounts(_SANDBOX_MOUNTINFO, _SCRATCH_COMMAND_LINE) self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(result.stdout, "checked\n") @@ -20166,6 +20186,16 @@ def test_mount_check_requires_private_runtime_mounts_once(self): "lowerdir=/srv/distro", "does not use the image layers", ), + "metacopy": ( + ",metacopy=on,", + ",", + "the root overlay is mounted without metacopy=on", + ), + "xino": ( + ",xino=on\n", + "\n", + "the root overlay is mounted without xino=on", + ), "missing-shm": ( "26 24 0:26 / /dev/shm rw,nosuid,nodev,relatime - tmpfs tmpfs rw\n", "", @@ -20180,15 +20210,52 @@ def test_mount_check_requires_private_runtime_mounts_once(self): for mutation, (old, new, message) in mutations.items(): with self.subTest(mutation=mutation): self.assertIn(old, _SANDBOX_MOUNTINFO) - mountinfo = self._fixture( - "mountinfo", _SANDBOX_MOUNTINFO.replace(old, new, 1) - ) - result = self._run( - functions, "check_mounts", {"/proc/self/mountinfo": mountinfo} + result = self._check_mounts( + _SANDBOX_MOUNTINFO.replace(old, new, 1), _SCRATCH_COMMAND_LINE ) self.assertEqual(result.returncode, 1, result.stdout) self.assertIn(message, result.stderr) + def test_mount_check_requires_the_upper_that_the_command_line_declares(self): + tmpfs = _SANDBOX_MOUNTINFO.replace("/run/nvx/scratch/", "/run/nvx/tmpfs/") + ram = _SCRATCH_COMMAND_LINE.replace( + "nvx_scratch=0xd0006000,ext4", "nvx_overlay_upper=ramfs" + ) + # A caller's --cmdline keeps its newlines, and the root's tokens follow + # it. + multiline = ram.replace("console=hvc0 ", "console=hvc0 quiet\nloglevel=0\n") + for mountinfo, command_line, message in ( + (tmpfs, ram, None), + (tmpfs, multiline, None), + (_SANDBOX_MOUNTINFO, ram, "upper layer is not on tmpfs"), + (_SANDBOX_MOUNTINFO, multiline, "upper layer is not on tmpfs"), + (tmpfs, _SCRATCH_COMMAND_LINE, "upper layer is not on scratch"), + # Only the RAM upper's token declares it. + ( + tmpfs, + _SCRATCH_COMMAND_LINE + " xnvx_overlay_upper=ramfs", + "upper layer is not on scratch", + ), + ( + tmpfs, + _SCRATCH_COMMAND_LINE + " nvx_overlay_upper=tmpfs", + "upper layer is not on scratch", + ), + ( + tmpfs.replace(",metacopy=on,", ","), + ram, + "the root overlay is mounted without metacopy=on", + ), + ): + with self.subTest(mountinfo=mountinfo, command_line=command_line): + result = self._check_mounts(mountinfo, command_line) + if message is None: + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout, "checked\n") + else: + self.assertEqual(result.returncode, 1, result.stdout) + self.assertIn(message, result.stderr) + def test_cgroup_check_requires_the_container_cgroup(self): for content, returncode in ( ("0::/container\n", 0), From f12224505f1b6937bc4de172824660d261ba23e9 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 14:43:40 -0700 Subject: [PATCH 3/5] nvx-rs: select the sandbox's upper with --upper nvx-rs sandbox always planned its distro layer over the ext4 image of --scratch, so no NVX launcher could request the RAM upper that the guest now assembles, although the typed plan, the prepared launch, and OpenVMM already admit it (#558). run and provision now take --upper scratch, the default, or --upper ram. A RAM upper attaches the distro layer alone, renders nvx_overlay_upper=ramfs in place of nvx_scratch=0xd0006000,ext4, and needs, binds, and claims no scratch image. --upper ram refuses --scratch and still requires --layer. The option names the RAM upper `ram`, as RootUpper and the plan's serde form do, whatever its token or the guest's mount call it. start, exec, stop, and deprovision refuse it: a provisioned sandbox keeps the upper that it recorded. SandboxLaunch::new now takes a SandboxUpper, as SandboxPlan::new does, and SandboxLaunch::scratch returns None for an upper in RAM. provision now records config.json in format 8: format 7 with the upper under `upper`, `scratch` or `ram`, and, for a RAM upper, `null` under `scratch` and `scratch_binding`. start still reads formats 7 and 1, whose upper is on scratch. It refuses as malformed a format 8 configuration without `upper` or with another value, a scratch upper without its image, a RAM upper with an image or a binding, and an earlier format that names an upper. Each start of a RAM sandbox begins with an empty upper. The fake OpenVMM admits a RAM-backed overlay as OpenVMM does: one nvx_overlay_upper=ramfs and the distro layer alone. The CLI tests cover a dry run, a one-shot run, and a managed lifecycle with a RAM upper and no scratch image in the root, and the configuration that provision records. They also cover the refusals of --upper ram beside --scratch or without --layer, of --upper outside run and provision, of another value, and of the malformed configurations above. doc/usage.md, doc/run.md, and the nvx-rs README describe the option, the record, and the trade-offs of each upper. The Ubuntu sandbox steps of run-nvx-microvm-tests.yml that check the configuration that provision records for live shares now require format 8, on Linux and Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/run-nvx-microvm-tests.yml | 10 +- doc/run.md | 49 +++++- doc/usage.md | 6 +- nvx-rs/README.md | 29 ++-- nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs | 19 ++- .../src/host/cli/sandbox/lifecycle/start.rs | 20 ++- nvx-rs/src/host/cli/sandbox/mod.rs | 45 +++-- nvx-rs/src/host/cli/sandbox/options.rs | 19 ++- nvx-rs/src/host/cli/sandbox/state.rs | 158 +++++++++++------- nvx-rs/src/host/cli/sandbox/tests.rs | 16 +- nvx-rs/src/host/sandbox/mod.rs | 53 +++--- nvx-rs/src/host/sandbox/options.rs | 36 +++- nvx-rs/src/host/sandbox/options/tests.rs | 70 +++++++- nvx-rs/src/host/sandbox/plan/tests.rs | 40 ++++- nvx-rs/src/host/sandbox/prepared.rs | 11 +- nvx-rs/src/host/sandbox/prepared/tests.rs | 8 +- nvx-rs/src/host/sandbox/specification.rs | 30 ++-- nvx-rs/src/host/sandbox/tests.rs | 42 +++-- nvx-rs/src/testing/fake_openvmm.rs | 40 ++++- nvx-rs/tests/checked_handoff.rs | 48 ++++-- nvx-rs/tests/cli/dry_run.rs | 19 +++ nvx-rs/tests/cli/formats.rs | 7 +- nvx-rs/tests/cli/lifecycle.rs | 128 ++++++++++++-- nvx-rs/tests/cli/one_shot.rs | 43 ++++- nvx-rs/tests/cli/openvmm.rs | 78 +++++++-- nvx-rs/tests/cli/usage.rs | 8 + nvx-rs/tests/cli/validation.rs | 25 +++ 27 files changed, 816 insertions(+), 241 deletions(-) diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index 964fc2315..b8920a8df 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -526,7 +526,7 @@ jobs: with open(sys.argv[1], encoding="utf-8") as stream: config = json.load(stream) - if config["format"] != 7 or config["export"]["owner"] != "caller": + if config["format"] != 8 or config["export"]["owner"] != "caller": sys.exit(f"managed sandbox did not persist caller ownership: {config}") EOF if nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}"; then @@ -664,7 +664,7 @@ jobs: ("/opt/hostedtoolcache", "", "ro", ["nvx-denied"], "directory"), ("/results/output.txt", "", "rw", [], "file"), ] - if config["format"] != 7 or "mounts" in config or shares != expected: + if config["format"] != 8 or "mounts" in config or shares != expected: sys.exit(f"managed sandbox did not persist every share: {config}") EOF nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}" @@ -800,7 +800,7 @@ jobs: ["nvx-writable"], ), ] - if config["format"] != 7 or "mounts" in config or shares != expected: + if config["format"] != 8 or "mounts" in config or shares != expected: sys.exit(f"managed sandbox did not persist the access policy: {config}") EOF nvx-rs/target/release/nvx-rs sandbox start --state-dir "${state}" @@ -1125,7 +1125,7 @@ jobs: $denied = ($child.denied | ForEach-Object { $_.Substring($child.root.Length + 1).Replace("\", "/") }) -join ";" "$($bind.target),$($bind.source),$mode,$denied,$kind" }) -join " " - if ($config.format -ne 7 -or $shares -ne "/workspace,,rw,nvx-denied,directory /config/settings.json,,ro,,file /opt/hostedtoolcache,,ro,nvx-denied,directory /results/output.txt,,rw,,file") { + if ($config.format -ne 8 -or $shares -ne "/workspace,,rw,nvx-denied,directory /config/settings.json,,ro,,file /opt/hostedtoolcache,,ro,nvx-denied,directory /results/output.txt,,rw,,file") { throw "managed sandbox did not persist every share: $($config | ConvertTo-Json -Compress -Depth 5)" } try { @@ -1236,7 +1236,7 @@ jobs: function Get-Relative($paths) { ($paths | ForEach-Object { $_.Substring($child.root.Length + 1).Replace("\", "/") }) -join "," } - if ($config.format -ne 7 -or @($config.export.children).Count -ne 1 -or @($config.export.binds).Count -ne 1 -or $bind.target -ne "/workspace" -or $bind.readOnly -or (Get-Relative $child.denied) -ne "nvx-denied" -or (Get-Relative $child.allowed) -ne "nvx-denied/nvx-allowed" -or (Get-Relative $child.write) -ne "nvx-writable") { + if ($config.format -ne 8 -or @($config.export.children).Count -ne 1 -or @($config.export.binds).Count -ne 1 -or $bind.target -ne "/workspace" -or $bind.readOnly -or (Get-Relative $child.denied) -ne "nvx-denied" -or (Get-Relative $child.allowed) -ne "nvx-denied/nvx-allowed" -or (Get-Relative $child.write) -ne "nvx-writable") { throw "managed sandbox did not persist the access policy: $($config | ConvertTo-Json -Compress -Depth 5)" } try { diff --git a/doc/run.md b/doc/run.md index f4ca2998d..709c270b8 100644 --- a/doc/run.md +++ b/doc/run.md @@ -794,8 +794,10 @@ mount -t virtiofs microvm /mnt/host ## Experimental single-workload sandbox The `sandbox` command of [`nvx-rs`](usage.md#nvx-rs) launches the microVM -with one compressed EROFS distro layer and one preformatted ext4 scratch -image. Build the distro layer with `nvx.py`: +with one compressed EROFS distro layer under a writable upper: by default one +preformatted ext4 scratch image, or a tmpfs in guest memory with +`--upper ram`, as [Overlay upper](#overlay-upper) describes. Build the distro +layer with `nvx.py`: ```bash python3 scripts/nvx.py build-distro-layer \ @@ -888,7 +890,8 @@ digits. Each UID/GID is in `1..4294967294`, memory in `1..18446744073709551615` bytes, and the process limit in `1..4194303` (the guest adds init's process). The command validates the files before launch, orders roles independently of option order, -attaches layers read-only, and reserves the writable slot for scratch. +attaches layers read-only, and reserves the writable slot for scratch, which a +RAM upper leaves empty. Conversion and scratch formatting stay off the start path. The Ubuntu converter verifies immutable inputs, applies the deny-by-default metadata policy, creates `/nonexistent` for UID/GID 65534, and invokes `mkfs.erofs`. @@ -927,6 +930,41 @@ enters only the assembled root with `chroot`, because Linux cannot [managed](#managed-lifecycle) execution instead moves the assembled root over `/` in its own mount namespace. +### Overlay upper + +The overlay keeps the workload's writes in its upper, which `--upper` selects +for `sandbox run` and `sandbox provision`: + +- `scratch`, the default, keeps the upper on the ext4 image of `--scratch`, + which OpenVMM attaches as the `scratch` block device. What the workload + writes uses host storage, fails with `ENOSPC` when the image is full, and + survives a managed sandbox's `stop` and next `start`. +- `ram` keeps the upper in guest memory, with no scratch image or block device. + The kernel command line declares `nvx_overlay_upper=ramfs`, and the guest + mounts a tmpfs for the upper, `nosuid` and `nodev`, at the kernel's default + size, half of the guest's RAM, beyond which a write fails with `ENOSPC`. Its + pages are charged to the workload's memory cgroup, so they count against + `--memory-max`, and the kernel cannot reclaim them while the files exist, + since the guest has no swap. Nothing that the workload writes outlives its + VM: each `start` of a managed sandbox begins with an empty upper. + +```bash +nvx-rs/target/release/nvx-rs sandbox \ + --layer distro,build/ubuntu-distro.erofs,11111111-1111-1111-1111-111111111111 \ + --upper ram \ + --entrypoint /bin/sh \ + --memory-mib 512 \ + --memory-max 268435456 +``` + +`--upper ram` refuses `--scratch`, and only `run` and `provision` take +`--upper`: a provisioned sandbox keeps the upper that its configuration +records. OpenVMM admits a RAM upper only with exactly one read-only `distro` +block device, and refuses it for snapshot capture and restore, and the guest +refuses a RAM upper beside a scratch block device. With either upper, the +overlay keeps `metacopy=on` and `xino=on`, which `/sbin/nvx-sandbox-smoke` +checks. + ### Live host shares `sandbox run` and `sandbox provision` accept repeatable @@ -1036,7 +1074,7 @@ inside a writable part of a share. On Windows, it refuses a path that only a verbatim `\\?\` path names, such as that of a directory whose name ends in a period or a space, which the path without the prefix would not name. The shares' children follow their resolved host paths. A managed sandbox stores -the plan in its configuration, in format 7, and each `start` checks it again, +the plan in its configuration, in format 8, and each `start` checks it again, as `run` does once it has verified the distro layer: it refuses a host path that no longer holds a directory or regular file as recorded, a recorded path whose file identity changed, and shares whose host paths came to overlap, such @@ -1323,6 +1361,7 @@ it runs, a second sandbox, or a `run`, on the same image fails with `another sandbox claims the ext4 scratch image PATH`. On Linux, the identity is the device and inode number, which a file that is created once the image is removed can take again: such a replacement, of the same length, is not refused. +A sandbox whose upper is in RAM has no scratch image to bind or claim. `run` and `start` hand OpenVMM the files that their checks opened, rather than paths that something could replace meanwhile. On Linux, OpenVMM inherits their descriptors, and its command line names them `/proc/self/fd/N`, as the command @@ -1430,7 +1469,7 @@ PID namespace, through the guest agent's execution entry, which also makes every mount but `/dev` and its devpts instance `nodev`. The workload sees one machine ID for the life of the VM. On `stop`, the guest agent unmounts the mappings of the live shares, and the guest then unmounts the host export, -overlay, layers, and scratch in dependency order before the VM powers off, as +overlay, layers, and upper in dependency order before the VM powers off, as it unmounts the shares when a one-shot workload exits. The legacy operation-less `sandbox` form is `sandbox run`; it remains one-shot and rejects `--state-dir` or any diff --git a/doc/usage.md b/doc/usage.md index 6c27549df..22f16b179 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -4,7 +4,7 @@ NVX has two command-line interfaces, which run from the repository root: - [`nvx-rs`](#nvx-rs) runs NVX microVMs and sandboxes: [`run`](#run) boots or restores a microVM on the terminal, [`sandbox`](#sandbox) runs or manages - workloads over EROFS layers and private ext4 scratch, and + workloads over EROFS layers and a private ext4 scratch or tmpfs upper, and [`download`](#download) installs the release of the checkout, which `run` and `sandbox` also download into a checkout that holds no artifacts. - `scripts/nvx.py`, with Python 3.10 or newer, builds, tests, benchmarks, and @@ -228,6 +228,7 @@ nvx-rs/target/release/nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [--layer ROLE,PATH,EROFS_UUID]... [--scratch PATH] + [--upper {scratch,ram}] [--state-dir PATH] [--entrypoint PATH] [--arg VALUE]... @@ -280,7 +281,8 @@ do, and their names cannot be abbreviated either. | --- | --- | --- | | `{run,provision,start,exec,stop,deprovision}` | `run` | Select a one-shot run or a managed lifecycle operation. | | `--layer ROLE,PATH,EROFS_UUID` | required for `run` and `provision` | Attach the `distro` EROFS layer. Only `distro` is accepted. The UUID must be lowercase hexadecimal in canonical hyphenated form; braces, prefixes, uppercase, and unhyphenated forms are refused. | -| `--scratch PATH` | required for `run` and `provision` | Attach a preformatted ext4 scratch image as the writable overlay. | +| `--scratch PATH` | required for `run` and `provision` with a scratch upper | Attach a preformatted ext4 scratch image as the overlay's writable upper. `--upper ram` refuses it. | +| `--upper {scratch,ram}` | `scratch` | Select the overlay's writable upper for `run` or `provision`: `scratch`, the image of `--scratch`, or `ram`, a tmpfs in guest memory with no scratch image, which loses the workload's writes when the VM stops. A RAM upper holds at most half of the guest's RAM, and its pages count against `--memory-max`; see [Overlay upper](run.md#overlay-upper). | | `--state-dir PATH` | required for managed operations | Select persistent sandbox state. One-shot `run` rejects this option. | | `--entrypoint PATH` | `/bin/sh` | Select an absolute workload entrypoint without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. | | `--arg VALUE` | none | Append one nonempty entrypoint argument without whitespace, control characters, double quotes, or any UTF-8 byte that Linux treats as whitespace. Repeat to pass multiple arguments. | diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 4a3320678..b896cee9f 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -778,10 +778,10 @@ and OpenVMM's outcome report. The launch profile of `nvx-rs sandbox`, with native input validation: `SandboxLaunch`, with the distro `SandboxLayer`, an EROFS layer or a GPT disk, -the one layer that a sandbox takes, the ext4 scratch disk, the entrypoint, -arguments, hostname, workload identity, and `SandboxSettings` resource limits -(`Option` for memory and `Option` for processes), and the -`SandboxPlan` that it makes of them. +the one layer that a sandbox takes, its `SandboxUpper`, the ext4 scratch disk +or RAM, the entrypoint, arguments, hostname, workload identity, and +`SandboxSettings` resource limits (`Option` for memory and `Option` +for processes), and the `SandboxPlan` that it makes of them. `plan`: `SandboxPlan`, the typed plan of a cold layered launch, which is persistable in a versioned serde form: one read-only `distro` block device, an @@ -1471,7 +1471,9 @@ deprovision the sandbox and provision it again`, and one that another sandbox claims with `another sandbox claims the ext4 scratch image IMAGE`. `run` binds and claims the scratch image itself, and runs OpenVMM through a prepared launch until it exits; its command, which it prints, names the checked files, -except in a dry run, which launches nothing. +except in a dry run, which launches nothing. A sandbox whose upper is in RAM, +`--upper ram`, has no scratch image: `provision` records `null` for the image +and its binding, and neither `start` nor `run` binds or claims one. ## Sandbox input contract @@ -1790,9 +1792,9 @@ before them. elsewhere, as `nvx.py` quoted it with `subprocess.list2cmdline` and `shlex.join`. Unless `--dry-run` is given, it then runs OpenVMM with the standard streams and exits with its status; the command then names the - checked distro layer and scratch image as `/proc/self/fd/N` on Linux, which - OpenVMM inherits, and by their final paths on Windows, as - [Checked handoff](#checked-handoff) describes. + checked distro layer, and the scratch image of an upper on scratch, as + `/proc/self/fd/N` on Linux, which OpenVMM inherits, and by their final paths + on Windows, as [Checked handoff](#checked-handoff) describes. `--outcome-report` asks OpenVMM for its outcome report. - `provision` records the sandbox in its state directory. `start` launches OpenVMM detached with the managed lifecycle, passes it a fresh capability @@ -1816,7 +1818,7 @@ earlier `nvx-rs` provisioned, which has no bindings or no typed network: | File | Contents | | --- | --- | -| `config.json` | The provisioned sandbox, in format 7, with the export plan of its shares under `export`, `null` for none, the binding of its distro layer under `distro_binding`, that of its scratch image under `scratch_binding`, and its network under `network`, `null` for none, written by `serde_json`'s pretty printer with the members of every object sorted by name, with mode 0600, through a temporary file. `start` reads it as strict JSON, as egress policy files are read, into the typed fields of format 7, and also reads the format 1 that earlier builds wrote for a sandbox without shares, with `mount` `null` and no `export` | +| `config.json` | The provisioned sandbox, in format 8, with the export plan of its shares under `export`, `null` for none, its upper under `upper`, `scratch` or `ram`, the binding of its distro layer under `distro_binding`, its scratch image under `scratch` and that image's binding under `scratch_binding`, both `null` for an upper in RAM, and its network under `network`, `null` for none, written by `serde_json`'s pretty printer with the members of every object sorted by name, with mode 0600, through a temporary file. `start` reads it as strict JSON, as egress policy files are read, into the typed fields of format 8, and also reads the format 7 that earlier builds wrote, which records no `upper` and keeps the upper on scratch, and the format 1 that they wrote for a sandbox without shares, with `mount` `null` and no `export` | | `sandbox.lock` | The store's lock, which serializes the lifecycle transitions | | `launch.json` | The store's journal of a start in progress, or of one that was interrupted | | `runtime.json` | The store's runtime record of the running VM, in its format 4: the OpenVMM process, by its ID, its start time, and the host's boot, its control endpoint, `STATE/control.sock` or `//./pipe/openvmm-microvm-HEX` on Windows, and the [runtime identity](#runtime-identity) of its start | @@ -2037,10 +2039,11 @@ command-line tokens make it fail: `fake_exit_on_start=STATUS`, OpenVMM, it opens the sandbox's block devices at the paths that its options name and holds them until it exits, after `fake_block_delay_ms=MS` if asked, and records what each held in `fake-openvmm-TOKEN-blocks.json` next to the -kernel. It also runs the VMs of templates' captures, whose block devices it -opens as OpenVMM does, without following a link and, on Windows, sharing only -reads and deletion, recording in that file whether a handle could write them; -it then publishes the snapshot of the manifest that +kernel; it takes a scratch disk, or, for `nvx_overlay_upper=ramfs`, the +`distro` layer alone. It also runs the VMs of templates' captures, whose block +devices it opens as OpenVMM does, without following a link and, on Windows, +sharing only reads and deletion, recording in that file whether a handle could +write them; it then publishes the snapshot of the manifest that `fake-openvmm-capture.json` next to the kernel names, or fails, never ends, or crashes, as that scenario asks. A managed `start` also reads the root's [runtime identity](#runtime-identity), so a test root holds build provenance, diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs index 82bc3f3c9..10a08c8ad 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/mod.rs @@ -7,9 +7,9 @@ //! //! ```text //! / -//! config.json the launch that provision records, in format 7, with the export plan of -//! its shares or `null` without shares, the bindings of its distro layer -//! and of its scratch image, and its network +//! config.json the launch that provision records, in format 8, with the export plan of +//! its shares or `null` without shares, its upper, the bindings of its +//! distro layer and of its scratch image, if any, and its network //! sandbox.lock the store's lock, which serializes the lifecycle transitions //! launch.json the store's journal of a start in progress, or of an interrupted one //! runtime.json the store's runtime record of the running VM, in its format 4 @@ -26,7 +26,7 @@ //! [`provision`], [`fn@crate::host::cli::sandbox::lifecycle::start`], [`fn@crate::host::cli::sandbox::lifecycle::stop`], and [`deprovision`] hold the [`LifecycleLock`] for their //! whole transition, and [`exec_workload`] only while it finds the running VM, so a long workload //! cannot delay a stop. `config.json` also records the binding of the distro layer that -//! `provision` verified, and that of the scratch image, which [`fn@crate::host::cli::sandbox::lifecycle::start`] checks from metadata +//! `provision` verified, and that of the scratch image of an upper on scratch, which [`fn@crate::host::cli::sandbox::lifecycle::start`] checks from metadata //! alone, and the network as its typed policy, which [`fn@crate::host::cli::sandbox::lifecycle::start`] checks again, in place of the //! network options that `sandbox_lifecycle.py` recorded: it refuses to start a sandbox that //! `nvx.py` or an earlier `nvx-rs` provisioned, which has no bindings or no typed network. It @@ -49,9 +49,10 @@ //! //! [`fn@crate::host::cli::sandbox::lifecycle::start`] hands OpenVMM, through a [`crate::host::sandbox::PreparedLaunch`] of the checked files, the objects that //! the checks found, rather than paths that something could replace meanwhile, and claims the -//! scratch image for this sandbox while OpenVMM runs. It holds the checked files until the guest -//! is ready, when it releases them, and on Windows OpenVMM's copies of them, or until it has ended -//! an OpenVMM whose start failed. +//! scratch image, if any, for this sandbox while OpenVMM runs. It holds the checked files until +//! the guest is ready, when it releases them, and on Windows OpenVMM's copies of them, or until it +//! has ended an OpenVMM whose start failed. A sandbox whose upper is in RAM starts each time with +//! an empty upper, which its stop discards. mod directory; mod outcome; @@ -104,7 +105,7 @@ pub(crate) const REQUIRED_FEATURES: GuestFeatures = GuestFeatures::LAYERED_SANDB /// Records a launch and its distro layer, at the path at which it was verified, with the layer's /// binding, in a new state directory, as `provision` does, with the binding of its scratch image, -/// at the path that it records for it. +/// if its upper is on scratch, at the path that it records for it. pub(crate) fn provision( state_path: &str, launch: &SandboxLaunch, @@ -126,7 +127,7 @@ pub(crate) fn provision( let scratch = RecordedScratch::new(launch)?; write_json( &config_path, - &serialize_launch(launch, distro, &scratch, config)?, + &serialize_launch(launch, distro, scratch.as_ref(), config)?, ) } diff --git a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs index daf76cf4e..656e72dff 100644 --- a/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs +++ b/nvx-rs/src/host/cli/sandbox/lifecycle/start.rs @@ -71,7 +71,8 @@ fn start_locked( // OpenVMM those objects: on Linux, OpenVMM inherits their descriptors; on Windows, it opens // them by their final paths, which the files, and OpenVMM's copies of them, keep anything from // replacing before OpenVMM, whose opens share them, has opened them, even if the start ends - // first. The scratch image's check also claims it for this sandbox, until OpenVMM exits. + // first. The scratch image's check also claims it for this sandbox, until OpenVMM exits. An + // upper in RAM has no scratch image, and its configuration no binding of one. let binding = config.distro_binding()?; let scratch_binding = config.scratch_binding()?; let distro = binding @@ -79,9 +80,14 @@ fn start_locked( .map_err(binding_error)?; // Reading the network checks its policy again, as provision did, before anything launches. let launch = launch.with_network(config.network()?); - let scratch = scratch_binding - .check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT) - .map_err(binding_error)?; + let scratch = match (scratch_binding, launch.scratch()) { + (Some(scratch_binding), Some(scratch)) => Some( + scratch_binding + .check(scratch, Instant::now() + CLAIM_RELEASE_WAIT) + .map_err(binding_error)?, + ), + _ => None, + }; // The plan takes the format and identity of the distro layer from its binding. let plan = launch.bound_plan(&binding)?; let [executable, kernel, initrd] = artifacts.require()?; @@ -94,7 +100,7 @@ fn start_locked( )); } let memory_mib = config.memory_mib(); - let prepared = PreparedLaunch::new(plan, distro, Some(scratch), config.cmdline())?; + let prepared = PreparedLaunch::new(plan, distro, scratch, config.cmdline())?; // The control endpoint, which the launch chooses, goes between these arguments. let mut head = vec![executable]; head.extend(prepared.openvmm_arguments().iter().cloned()); @@ -146,8 +152,8 @@ fn start_locked( } session.map_host_paths(&maps, timeout).inspect_err(|error| { // The agent then refuses workloads, so the guest shuts down as a stop has it do: - // it unmounts what it mapped, the export, the overlay, and the scratch disk before - // the VM powers off. A failed transfer closed the session, so a guest that stopped + // it unmounts what it mapped, the export, the overlay, and the upper before the + // VM powers off. A failed transfer closed the session, so a guest that stopped // answering is terminated at once. if !matches!(error, Exit::Interrupted) { stopping = session.stop(timeout).is_ok(); diff --git a/nvx-rs/src/host/cli/sandbox/mod.rs b/nvx-rs/src/host/cli/sandbox/mod.rs index 90ac61058..3477a4125 100644 --- a/nvx-rs/src/host/cli/sandbox/mod.rs +++ b/nvx-rs/src/host/cli/sandbox/mod.rs @@ -4,10 +4,11 @@ //! //! - `run`, the default, boots a one-shot sandbox: it prints the OpenVMM command as `>> COMMAND`, //! then runs it on the terminal, unless `--dry-run`, and exits with OpenVMM's status. It runs -//! OpenVMM through a [`PreparedLaunch`] of the distro layer that it verified and of the scratch -//! image, which it claims, so the command names their checked files: on Linux as -//! `/proc/self/fd/N`, and on Windows by their final paths. A dry run, which launches nothing, -//! names their paths. +//! OpenVMM through a [`PreparedLaunch`] of the distro layer that it verified and, for an upper +//! on scratch, of the scratch image, which it claims, so the command names their checked files: +//! on Linux as `/proc/self/fd/N`, and on Windows by their final paths. A dry run, which launches +//! nothing, names their paths. `--upper ram` keeps the upper in RAM, on a tmpfs that the guest +//! mounts, and attaches no scratch image. //! - `provision`, `start`, `exec`, `stop`, and `deprovision` manage a persistent sandbox in its //! `--state-dir`; see [`lifecycle`]. //! @@ -35,8 +36,11 @@ use crate::host::sandbox::options::{ SandboxOperation, SandboxOptions, sandbox_mounts, validate_sandbox_systemd_policy, }; use crate::host::sandbox::scratch::{CLAIM_RELEASE_WAIT, ScratchBinding}; -use crate::host::sandbox::{PreparedLaunch, SandboxLaunch, SandboxPlan, SandboxSettings}; +use crate::host::sandbox::{ + PreparedLaunch, SandboxLaunch, SandboxPlan, SandboxSettings, SandboxUpper, +}; use crate::host::store::RuntimeIdentity; +use crate::protocol::root::RootUpper; pub(crate) use self::lifecycle::ProvisionConfig; pub(super) use self::one_shot::run_openvmm; @@ -151,6 +155,7 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result { mount_owner: options.mount_owner, layers: !options.layers.is_empty(), scratch: options.scratch.is_some(), + upper: options.upper, }; let environment = checks.check()?; let artifacts = Artifacts { root }; @@ -182,11 +187,15 @@ pub(super) fn run(root: Option<&str>, options: Options) -> Result { memory_max: options.memory_max, pids_max: options.pids_max, }; - let launch = SandboxLaunch::new( - options.layers.clone(), - options.scratch.as_deref().unwrap_or_default(), - settings, - )?; + // The checks above require --scratch for an upper on scratch, and refuse it for one in + // RAM. + let upper = match options.upper.unwrap_or(RootUpper::Scratch) { + RootUpper::Scratch => { + SandboxUpper::scratch(options.scratch.as_deref().unwrap_or_default()) + } + RootUpper::Ram => SandboxUpper::RAM, + }; + let launch = SandboxLaunch::new(options.layers.clone(), upper, settings)?; validate_mounts(&mounts)?; launch.validated()?; // The shares are planned once, from their resolved host paths, and `provision` records @@ -360,9 +369,9 @@ fn run_plan(launch: &SandboxLaunch, binding: &DistroBinding) -> Result Result { let distro = binding.check(launch.distro(), DISTRO_POLICY_REVISION)?; - let scratch = ScratchBinding::bind(launch.scratch())? - .check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT)?; - PreparedLaunch::one_shot(plan, distro, Some(scratch), cmdline).map_err(Exit::from) + let scratch = match launch.scratch() { + Some(scratch) => Some( + ScratchBinding::bind(scratch)?.check(scratch, Instant::now() + CLAIM_RELEASE_WAIT)?, + ), + None => None, + }; + PreparedLaunch::one_shot(plan, distro, scratch, cmdline).map_err(Exit::from) } #[cfg(test)] diff --git a/nvx-rs/src/host/cli/sandbox/options.rs b/nvx-rs/src/host/cli/sandbox/options.rs index 4258a1c19..85ab995b6 100644 --- a/nvx-rs/src/host/cli/sandbox/options.rs +++ b/nvx-rs/src/host/cli/sandbox/options.rs @@ -18,6 +18,7 @@ use crate::host::sandbox::{ DEFAULT_ENTRYPOINT, DEFAULT_HOSTNAME, DEFAULT_WORKLOAD_IDENTITY, PIDS_MAX_LIMIT, SandboxLayer, SandboxNetwork, parse_workload_identity, }; +use crate::protocol::root::RootUpper; /// The usage and help of the parser. pub(in crate::host::cli) const SYNTAX: Syntax = Syntax { @@ -31,6 +32,10 @@ options: -h, --help show this help message and exit --layer ROLE,PATH,EROFS_UUID --scratch SCRATCH + --upper {scratch,ram} + where the overlay keeps the sandbox's writes: scratch + (default), on the ext4 image of --scratch, or ram, on + a tmpfs in guest memory, without a scratch image --state-dir STATE_DIR --entrypoint ENTRYPOINT --arg SANDBOX_ARG @@ -112,6 +117,8 @@ options: /// The operations, in the order in which usage messages list them. const OPERATIONS: [&str; 6] = ["run", "provision", "start", "exec", "stop", "deprovision"]; +/// The choices of `--upper`, the names of the uppers. +const UPPERS: [&str; 2] = [RootUpper::Scratch.as_str(), RootUpper::Ram.as_str()]; /// The choices of `--hypervisor`. pub(in crate::host::cli) const HYPERVISORS: [&str; 4] = ["auto", "whp", "kvm", "mshv"]; /// The choices of `--mount-owner`. @@ -132,9 +139,10 @@ fn seconds(value: &str) -> std::result::Result { /// The options of `nvx-rs sandbox`, which parse with the defaults of the former /// `nvx.py sandbox`. -pub(in crate::host::cli) const OPTIONS: [Spec; 35] = [ +pub(in crate::host::cli) const OPTIONS: [Spec; 36] = [ Spec::one("--layer"), Spec::one("--scratch"), + Spec::one("--upper"), Spec::one("--state-dir"), Spec::one("--entrypoint"), Spec::one("--arg"), @@ -179,6 +187,8 @@ pub(crate) struct Options { pub(crate) layers: Vec, /// `--scratch`, as given. pub(crate) scratch: Option, + /// `--upper`, if given; a launch keeps its upper on scratch without it. + pub(crate) upper: Option, /// `--state-dir`, as given. pub(crate) state_dir: Option, /// `--entrypoint`. @@ -232,6 +242,7 @@ impl Default for Options { operation: SandboxOperation::Run, layers: Vec::new(), scratch: None, + upper: None, state_dir: None, entrypoint: DEFAULT_ENTRYPOINT.to_owned(), args: Vec::new(), @@ -277,6 +288,12 @@ impl Receiver for Options { .layers .push(SandboxLayer::parse(&text()).map_err(|error| error.message().to_owned())?), "--scratch" => self.scratch = Some(text()), + "--upper" => { + self.upper = Some(match choice(&text(), &UPPERS)? { + "ram" => RootUpper::Ram, + _ => RootUpper::Scratch, + }); + } "--state-dir" => self.state_dir = Some(text()), "--entrypoint" => self.entrypoint = text(), "--arg" => self.args.push(text()), diff --git a/nvx-rs/src/host/cli/sandbox/state.rs b/nvx-rs/src/host/cli/sandbox/state.rs index b6531f74f..4e627aa29 100644 --- a/nvx-rs/src/host/cli/sandbox/state.rs +++ b/nvx-rs/src/host/cli/sandbox/state.rs @@ -1,28 +1,30 @@ -//! The CLI's own record in a state directory, `config.json`, which `provision` writes in format 7 -//! and `start` also reads in format 1. +//! The CLI's own record in a state directory, `config.json`, which `provision` writes in format 8 +//! and `start` also reads in formats 1 and 7. //! //! - [`write_json`] writes JSON text and a line end, with each `\n` turned into the line separator //! of the host, into a temporary file of mode 0600 that then replaces the file. //! - [`read_config`] reads a configuration as [`read_object`] reads a JSON file, whose `format` is -//! 1 or 7, and [`Config::new`] holds the fields that `start` reads, as their types require them: -//! a configuration without one, or with a value of another type, is malformed. Guest RAM +//! 1, 7, or 8, and [`Config::new`] holds the fields that `start` reads, as their types require +//! them: a configuration without one, or with a value of another type, is malformed. Guest RAM //! outside the range of `--memory-mib`, which `nvx.py` and earlier releases of `nvx-rs` //! recorded, is refused with the advice to provision the sandbox again. -//! - [`serialize_launch`] writes a provisioned launch in format 7, with the export plan of its -//! shares under `export`, `null` without shares, with its members sorted by name, as -//! [`pretty_serde`] writes them, and [`Config::launch`] reads it back. It also reads format 1, which -//! earlier builds wrote for a sandbox without shares, with `mount` `null` and no `export`. -//! Formats 2 to 6 recorded the shares as they were requested; `start` no longer reads them, and -//! such a sandbox is provisioned again. +//! - [`serialize_launch`] writes a provisioned launch in format 8, with the export plan of its +//! shares under `export`, `null` without shares, and its upper under `upper`, `scratch` or +//! `ram`, with its members sorted by name, as [`pretty_serde`] writes them, and +//! [`Config::launch`] reads it back. It also reads format 7, which earlier builds wrote, the +//! same without `upper`, whose upper is on scratch, and format 1, which they wrote for a sandbox +//! without shares, with `mount` `null` and no `export`. Formats 2 to 6 recorded the shares as +//! they were requested; `start` no longer reads them, and such a sandbox is provisioned again. //! - Every configuration also records the [`DistroBinding`] of the sandbox's distro layer, under //! `distro_binding` in its serde form, which [`Config::distro_binding`] reads back for `start` //! to check. `nvx.py` never wrote one, and `start` refuses a configuration without one. The //! binding is of the layer at the path that the configuration records, which a //! [`RecordedDistro`] holds. -//! - It records the [`ScratchBinding`] of the sandbox's scratch image too, under -//! `scratch_binding`, which [`Config::scratch_binding`] reads back for `start` to check and -//! claim, of the image at the path that the configuration records, which a [`RecordedScratch`] -//! holds. `start` refuses a configuration without one as well. +//! - The configuration of an upper on scratch records the [`ScratchBinding`] of the sandbox's +//! scratch image too, under `scratch_binding`, which [`Config::scratch_binding`] reads back for +//! `start` to check and claim, of the image at the path that the configuration records under +//! `scratch`, which a [`RecordedScratch`] holds. `start` refuses such a configuration without +//! one as well. That of an upper in RAM has no scratch image, and records `null` under both. //! - Every configuration also records the sandbox's [`SandboxNetwork`], under `network` in the //! serde form of the plan's network, or `null` for no network device, which //! [`Config::network`] reads back for `start` to render. It replaces the network options that @@ -48,16 +50,22 @@ use crate::host::paths; use crate::host::sandbox::distro::{DISTRO_POLICY_REVISION, DistroBinding}; use crate::host::sandbox::layer::{LayerRole, SandboxLayer}; use crate::host::sandbox::scratch::ScratchBinding; -use crate::host::sandbox::{SandboxLaunch, SandboxNetwork, SandboxSettings}; +use crate::host::sandbox::{SandboxLaunch, SandboxNetwork, SandboxSettings, SandboxUpper}; +use crate::protocol::root::RootUpper; /// The configuration that earlier builds wrote for a sandbox without shares, whose `mount` is /// `null`, which `start` still reads. pub(crate) const CONFIG_FORMAT: u64 = 1; -/// The configuration that `provision` writes, which records the export plan of the sandbox's -/// shares under `export`, in its serde form, or `null` for a sandbox without shares. +/// The configuration that earlier builds wrote, which records the export plan of the sandbox's +/// shares under `export`, in its serde form, or `null` for a sandbox without shares, and whose +/// upper is on scratch, which `start` still reads. pub(crate) const EXPORT_CONFIG_FORMAT: u64 = 7; +/// The configuration that `provision` writes: that of [`EXPORT_CONFIG_FORMAT`], with the upper +/// under `upper`, `scratch` or `ram`. +pub(crate) const UPPER_CONFIG_FORMAT: u64 = 8; /// Every configuration format that `start` reads. -pub(crate) const CONFIG_FORMATS: [u64; 2] = [CONFIG_FORMAT, EXPORT_CONFIG_FORMAT]; +pub(crate) const CONFIG_FORMATS: [u64; 3] = + [CONFIG_FORMAT, EXPORT_CONFIG_FORMAT, UPPER_CONFIG_FORMAT]; /// What messages call a configuration. const DESCRIPTION: &str = "sandbox configuration"; @@ -207,11 +215,15 @@ pub(crate) struct RecordedScratch { impl RecordedScratch { /// Returns the scratch image of `launch` at the path that `provision` records, the path given, - /// resolved, with a binding of the image there, which claims nothing. - pub(crate) fn new(launch: &SandboxLaunch) -> Result { - let path = resolved(launch.scratch())?; + /// resolved, with a binding of the image there, which claims nothing, or `None` for a launch + /// whose upper is in RAM, which has no scratch image. + pub(crate) fn new(launch: &SandboxLaunch) -> Result> { + let Some(scratch) = launch.scratch() else { + return Ok(None); + }; + let path = resolved(scratch)?; let binding = ScratchBinding::bind(Path::new(&path))?; - Ok(Self { path, binding }) + Ok(Some(Self { path, binding })) } } @@ -237,15 +249,17 @@ struct ConfigLayer { uuid: String, } -/// The configuration that `provision` records, in format 7. +/// The configuration that `provision` records, in format 8. #[derive(Serialize)] struct Record<'a> { - /// [`EXPORT_CONFIG_FORMAT`]. + /// [`UPPER_CONFIG_FORMAT`]. format: u64, /// The distro layer, the one layer that a launch holds. layers: [RecordedLayer<'a>; 1], - /// The scratch image, at the path that `provision` bound. - scratch: &'a str, + /// Where the overlay keeps its upper: `scratch` or `ram`. + upper: &'a str, + /// The scratch image, at the path that `provision` bound, or `None` for an upper in RAM. + scratch: Option<&'a str>, /// The workload's hostname. hostname: &'a str, /// The workload's UID. @@ -266,8 +280,8 @@ struct Record<'a> { cmdline: &'a str, /// The binding of the distro layer. distro_binding: &'a DistroBinding, - /// The binding of the scratch image. - scratch_binding: &'a ScratchBinding, + /// The binding of the scratch image, or `None` for an upper in RAM. + scratch_binding: Option<&'a ScratchBinding>, /// The export plan of the shares, or `None` without shares. export: Option<&'a ExportPlan>, } @@ -275,24 +289,25 @@ struct Record<'a> { /// Returns the JSON text of the configuration that `provision` records for `launch` and the other /// options, with `distro`, the launch's distro layer at the path that `provision` verified and /// records, and its binding, and `scratch`, the launch's scratch image at the path that -/// `provision` records, and its binding. +/// `provision` records, and its binding, or `None` for an upper in RAM. pub(crate) fn serialize_launch( launch: &SandboxLaunch, distro: &RecordedDistro, - scratch: &RecordedScratch, + scratch: Option<&RecordedScratch>, config: &ProvisionConfig<'_>, ) -> Result { let layer = &distro.layer; let settings = launch.settings(); let (workload_uid, workload_gid) = settings.workload_identity; let record = Record { - format: EXPORT_CONFIG_FORMAT, + format: UPPER_CONFIG_FORMAT, layers: [RecordedLayer { role: layer.role().as_str(), path: &distro.path, uuid: layer.uuid(), }], - scratch: &scratch.path, + upper: launch.upper().kind().as_str(), + scratch: scratch.map(|scratch| scratch.path.as_str()), hostname: &settings.hostname, workload_uid, workload_gid, @@ -303,7 +318,7 @@ pub(crate) fn serialize_launch( network: launch.network(), cmdline: config.cmdline, distro_binding: &distro.binding, - scratch_binding: &scratch.binding, + scratch_binding: scratch.map(|scratch| &scratch.binding), export: launch.export(), }; pretty_serde(&record, DESCRIPTION) @@ -323,19 +338,22 @@ fn present<'de, D: Deserializer<'de>>( Value::deserialize(deserializer).map(Some) } -/// The fields of a configuration of format 1 or 7 that `start` reads. +/// The fields of a configuration of format 1, 7, or 8 that `start` reads. /// -/// Every field is required, with its type, but the bindings, the network, and the shares, whose -/// absence [`Config`]'s methods report: `nvx.py` wrote no bindings and no typed network, and only -/// format 7 records `export`. Other fields, such as the network options of `nvx.py`, are ignored. +/// Every field is required, with its type, but the bindings, the network, the shares, and the +/// upper, whose absence [`Config`]'s methods report: `nvx.py` wrote no bindings and no typed +/// network, only formats 7 and 8 record `export`, and only format 8 `upper`, without which the +/// upper is on scratch. Other fields, such as the network options of `nvx.py`, are ignored. #[derive(Deserialize)] pub(crate) struct Config { /// The format, which [`read_config`] checked. format: u64, /// The read-only layers. layers: Vec, - /// The scratch image. - scratch: String, + /// Where the overlay keeps its upper, `scratch` or `ram`, as format 8 records it. + upper: Option, + /// The scratch image, or `None` for an upper in RAM. + scratch: Option, /// The workload's hostname. hostname: String, /// The workload's UID. @@ -352,7 +370,7 @@ pub(crate) struct Config { memory_mib: u64, /// The user's kernel command line. cmdline: String, - /// The export plan of the shares, `null` for none, as format 7 records it, if recorded. + /// The export plan of the shares, `null` for none, as formats 7 and 8 record it, if recorded. #[serde(default, deserialize_with = "present")] export: Option, /// The shares as formats 1 to 6 recorded them, `null` for none, if recorded. @@ -361,7 +379,7 @@ pub(crate) struct Config { /// The binding of the distro layer, if recorded. #[serde(default, deserialize_with = "present")] distro_binding: Option, - /// The binding of the scratch image, if recorded. + /// The binding of the scratch image, `null` for an upper in RAM, if recorded. #[serde(default, deserialize_with = "present")] scratch_binding: Option, /// The sandbox's network, `null` for none, if recorded. @@ -415,14 +433,15 @@ impl Config { /// Reads the launch that the configuration records and checks it against the host. /// - /// A layer whose role is not `distro`, or an export plan that is not of its serde form, fails - /// as `sandbox configuration is malformed`. A configuration whose format is not the one that - /// its shares need, format 7 for one that records `export` and format 1 for one that does not, - /// fails as `sandbox configuration format does not match its mounts`, and so does one with a - /// share under `mount`, which only formats that `start` no longer reads recorded. The launch - /// must pass the checks of [`SandboxLaunch::new`], then the export plan those of - /// [`SandboxLaunch::with_export`], and the launch those of [`SandboxLaunch::validated`], which - /// checks again the host objects that the plan recorded. + /// A layer whose role is not `distro`, an export plan that is not of its serde form, or an + /// upper that [`upper`](Self::upper) refuses fails as `sandbox configuration is malformed`. A + /// configuration whose format is not the one that its shares need, format 7 or 8 for one that + /// records `export` and format 1 for one that does not, fails as `sandbox configuration format + /// does not match its mounts`, and so does one with a share under `mount`, which only formats + /// that `start` no longer reads recorded. The launch must pass the checks of + /// [`SandboxLaunch::new`], then the export plan those of [`SandboxLaunch::with_export`], and + /// the launch those of [`SandboxLaunch::validated`], which checks again the host objects that + /// the plan recorded. pub(crate) fn launch(&self) -> Result { let layers = self .layers @@ -440,7 +459,7 @@ impl Config { }; let launch = SandboxLaunch::new( layers, - &self.scratch, + self.upper()?, SandboxSettings { hostname: self.hostname.clone(), workload_identity: (self.workload_uid, self.workload_gid), @@ -450,6 +469,7 @@ impl Config { }, )?; let (expected, mounted) = match &self.export { + Some(_) if self.upper.is_some() => (UPPER_CONFIG_FORMAT, false), Some(_) => (EXPORT_CONFIG_FORMAT, false), None => ( CONFIG_FORMAT, @@ -467,6 +487,24 @@ impl Config { Ok(launch) } + /// Returns the upper that the configuration records. Format 8 names it under `upper`: for + /// `scratch`, the scratch image under `scratch`, and for `ram`, RAM, with `scratch` `null`. + /// Formats 1 and 7 name none, and keep the upper on the scratch image under `scratch`. Any + /// other combination fails as `sandbox configuration is malformed`. + fn upper(&self) -> Result { + let (scratch, ram) = (RootUpper::Scratch.as_str(), RootUpper::Ram.as_str()); + match (self.format, self.upper.as_deref(), self.scratch.as_deref()) { + (UPPER_CONFIG_FORMAT, Some(upper), Some(path)) if upper == scratch => { + Ok(SandboxUpper::scratch(path)) + } + (UPPER_CONFIG_FORMAT, Some(upper), None) if upper == ram => Ok(SandboxUpper::RAM), + (CONFIG_FORMAT | EXPORT_CONFIG_FORMAT, None, Some(path)) => { + Ok(SandboxUpper::scratch(path)) + } + _ => Err(malformed()), + } + } + /// Reads the binding of the distro layer that the configuration records, which `start` checks. /// /// A configuration without one, as `nvx.py` and earlier releases of `nvx-rs` wrote, fails as @@ -478,13 +516,21 @@ impl Config { } /// Reads the binding of the scratch image that the configuration records, which `start` - /// checks and claims. + /// checks and claims, or `None` for an upper in RAM, whose configuration records `null`. /// - /// A configuration without one, as earlier releases of `nvx-rs` wrote, fails as `sandbox - /// configuration has no scratch binding`, and one that is not of the binding's serde form as - /// `sandbox configuration is malformed`, as [`distro_binding`](Self::distro_binding) fails. - pub(crate) fn scratch_binding(&self) -> Result { - recorded(self.scratch_binding.as_ref(), "scratch binding") + /// A configuration of an upper on scratch without one, as earlier releases of `nvx-rs` wrote, + /// fails as `sandbox configuration has no scratch binding`, and one that is not of the + /// binding's serde form as `sandbox configuration is malformed`, as + /// [`distro_binding`](Self::distro_binding) fails, and so does a configuration of an upper in + /// RAM that records anything but `null`. + pub(crate) fn scratch_binding(&self) -> Result> { + if self.upper.as_deref() == Some(RootUpper::Ram.as_str()) { + return match self.scratch_binding { + Some(Value::Null) => Ok(None), + _ => Err(malformed()), + }; + } + recorded(self.scratch_binding.as_ref(), "scratch binding").map(Some) } /// Reads the network that the configuration records, in its typed form, which `start` diff --git a/nvx-rs/src/host/cli/sandbox/tests.rs b/nvx-rs/src/host/cli/sandbox/tests.rs index 98547eccf..298a658e3 100644 --- a/nvx-rs/src/host/cli/sandbox/tests.rs +++ b/nvx-rs/src/host/cli/sandbox/tests.rs @@ -47,6 +47,20 @@ fn resource_limits_are_bounded_ascii_decimal() { assert!(parse(&["--pids-max".to_owned(), "4194304".to_owned()]).is_err()); } +#[test] +fn the_upper_is_scratch_or_ram() { + let upper = |value: &str| { + parse(&["--upper".to_owned(), value.to_owned()]).map(|(options, _)| options.upper) + }; + assert_eq!(parse(&[]).unwrap().0.upper, None); + assert_eq!(upper("scratch").unwrap(), Some(RootUpper::Scratch)); + assert_eq!(upper("ram").unwrap(), Some(RootUpper::Ram)); + // The option names the RAM upper `ram`, whatever its token or the guest's mount call it. + for value in ["ramfs", "tmpfs", "RAM", "", "scratch,ram"] { + assert!(upper(value).is_err(), "{value:?}"); + } +} + proptest! { #[test] fn policy_option_sequences_record_the_preceding_mount( @@ -109,7 +123,7 @@ fn run_checks_again_that_its_shares_do_not_overlap() { &path("distro.erofs"), "11111111-1111-1111-1111-111111111111", )], - &path("scratch.ext4"), + SandboxUpper::scratch(&path("scratch.ext4")), SandboxSettings::default(), ) .unwrap() diff --git a/nvx-rs/src/host/sandbox/mod.rs b/nvx-rs/src/host/sandbox/mod.rs index 8aac7468b..80ca77327 100644 --- a/nvx-rs/src/host/sandbox/mod.rs +++ b/nvx-rs/src/host/sandbox/mod.rs @@ -20,9 +20,9 @@ //! guest targets and the checks of host paths, plans the shares of a sandbox through the planner //! of every NVX caller, [`host::filesystem`](crate::host::filesystem), and renders the plan as //! the guest's shell reads it at boot. -//! - [`SandboxLaunch`]: the launch of `nvx-rs sandbox`, its layers over its scratch image with -//! its settings and the export plan of its shares, which plans its distro layer, in the layer's -//! format, over the scratch image. +//! - [`SandboxLaunch`]: the launch of `nvx-rs sandbox`, its layers over its upper, an ext4 +//! scratch image or RAM, with its settings and the export plan of its shares, which plans its +//! distro layer, in the layer's format, over that upper. //! - [`options`]: the checks that `nvx-rs sandbox` and `nvx-rs run` apply to their options: the //! attribution of access-policy paths to shares, the systemd entrypoint check and the preflight //! that verifies the distro layer, the targets of `nvx-rs run` shares, and the managed exec @@ -353,9 +353,9 @@ fn validate_settings(settings: &SandboxSettings) -> Result<()> { Ok(()) } -/// The launch of a sandbox of `nvx-rs sandbox`: its read-only layers, its ext4 scratch image, its -/// settings, the export plan of its shares, and its network, which it plans as its distro layer, -/// in the layer's format, over the scratch image. +/// The launch of a sandbox of `nvx-rs sandbox`: its read-only layers, its upper, an ext4 scratch +/// image or RAM, its settings, the export plan of its shares, and its network, which it plans as +/// its distro layer, in the layer's format, over its upper. /// /// Its layers are keyed by the role of each `--layer` value, of which a launch takes only the /// distro layer. [`plan`](Self::plan) returns its [`SandboxPlan`], which renders its OpenVMM @@ -366,15 +366,14 @@ fn validate_settings(settings: &SandboxSettings) -> Result<()> { pub struct SandboxLaunch { /// The read-only layers, in the order given: the distro layer alone. layers: Vec, - /// The plan: the distro layer, in its format and with its identity, over the scratch image, - /// with the settings and the export plan. + /// The plan: the distro layer, in its format and with its identity, over the upper, with the + /// settings and the export plan. plan: SandboxPlan, } impl SandboxLaunch { - /// Returns the launch of `layers` over the ext4 image at `scratch`, which is spelled with its - /// components alone, as [`Path::components`] reads them, without shares, which - /// [`with_export`](Self::with_export) adds. + /// Returns the launch of `layers` over `upper`, an ext4 scratch image or RAM, without shares, + /// which [`with_export`](Self::with_export) adds. /// /// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported), checking in this /// order: no duplicate roles, exactly one layer, the distro layer, a UUID of the distro layer @@ -382,7 +381,7 @@ impl SandboxLaunch { /// checks them. pub fn new( layers: Vec, - scratch: &str, + upper: SandboxUpper, settings: SandboxSettings, ) -> Result { let mut duplicates: Vec<&str> = LAYER_ROLES @@ -418,7 +417,7 @@ impl SandboxLaunch { }; let plan = SandboxPlan::new( SandboxDistro::at(distro.path().to_path_buf(), image), - SandboxUpper::scratch(scratch), + upper, settings, )?; Ok(Self { layers, plan }) @@ -434,15 +433,17 @@ impl SandboxLaunch { &self.layers[0] } - /// Returns the scratch image. - pub fn scratch(&self) -> &Path { - self.plan - .upper() - .scratch_path() - .expect("a launch keeps its upper on its scratch image") + /// Returns the upper: an ext4 scratch image, or RAM. + pub fn upper(&self) -> &SandboxUpper { + self.plan.upper() + } + + /// Returns the scratch image, or `None` for an upper in RAM. + pub fn scratch(&self) -> Option<&Path> { + self.plan.upper().scratch_path() } - /// Returns the settings besides the layers and the scratch image. + /// Returns the settings besides the layers and the upper. pub fn settings(&self) -> &SandboxSettings { self.plan .settings() @@ -481,8 +482,8 @@ impl SandboxLaunch { } /// Returns the plan of the launch: its distro layer, in the format and with the identity of - /// its layer, as a `--layer` value names an EROFS layer and its UUID, over its scratch image, - /// with its settings, the export plan of its shares, and its network. + /// its layer, as a `--layer` value names an EROFS layer and its UUID, over its upper, with its + /// settings, the export plan of its shares, and its network. pub fn plan(&self) -> &SandboxPlan { &self.plan } @@ -500,10 +501,10 @@ impl SandboxLaunch { } /// Checks the launch against the host, as [`SandboxPlan::validated`] checks its plan: the - /// distro layer and the scratch image must be regular files whose paths have no comma or - /// semicolon, and whose names, and those of the files to which their paths resolve, do not end - /// in an extension with which OpenVMM would not attach them as raw images, and the export - /// plan's directories and files must still not overlap, and it must pass + /// distro layer and the scratch image of an upper on scratch must be regular files whose paths + /// have no comma or semicolon, and whose names, and those of the files to which their paths + /// resolve, do not end in an extension with which OpenVMM would not attach them as raw images, + /// and the export plan's directories and files must still not overlap, and it must pass /// [`ExportPlan::verify`]. /// /// Fails with [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) for a missing image, diff --git a/nvx-rs/src/host/sandbox/options.rs b/nvx-rs/src/host/sandbox/options.rs index 3c68e8d87..b26ae643f 100644 --- a/nvx-rs/src/host/sandbox/options.rs +++ b/nvx-rs/src/host/sandbox/options.rs @@ -25,6 +25,7 @@ use super::mount::{ }; use super::{SandboxLaunch, refused}; use crate::error::Result; +use crate::protocol::root::RootUpper; /// The entrypoints of systemd, which the sandbox security profile does not run. pub const SYSTEMD_ENTRYPOINTS: [&str; 2] = ["/usr/lib/systemd/systemd", "/lib/systemd/systemd"]; @@ -127,6 +128,8 @@ pub struct SandboxOptions { pub layers: bool, /// Whether `--scratch` was given. pub scratch: bool, + /// `--upper`, if given: a launch keeps its upper on scratch without it. + pub upper: Option, } impl SandboxOptions { @@ -149,6 +152,7 @@ impl SandboxOptions { mount_owner: None, layers: false, scratch: false, + upper: None, } } @@ -160,9 +164,10 @@ impl SandboxOptions { /// [`exec_environment_from_file`] or [`validate_exec_environment`] refuses, network policy /// options except for `run` and `provision`, a systemd entrypoint for `run`, `provision`, /// and `exec`, `--outcome-report` except for `run` and `exec`, share options without a - /// `--mount`, `--mount` except for `run` and `provision`, and for `run` and `provision`, - /// `--net` without `--network-profile` or the reverse, and a missing `--layer` or - /// `--scratch`. + /// `--mount`, `--mount` and `--upper` except for `run` and `provision`, and for `run` and + /// `provision`, `--net` without `--network-profile` or the reverse, a missing `--layer`, and + /// a missing `--scratch` for an upper on scratch, or one given for `--upper ram`, whose upper + /// is in RAM. /// /// Fails with [`ErrorKind::Unsupported`](crate::ErrorKind::Unsupported). pub fn check(&self) -> Result>> { @@ -225,16 +230,33 @@ impl SandboxOptions { "--mount is only valid for sandbox run or provision", )); } + if self.upper.is_some() && !operation.launches() { + return Err(refused( + "--upper is only valid for sandbox run or provision", + )); + } if operation.launches() { if self.net != self.network_profile { return Err(refused( "--net and --network-profile must be specified together", )); } - if !self.layers || !self.scratch { - return Err(refused(format!( - "sandbox {operation} requires --layer and --scratch" - ))); + match self.upper.unwrap_or(RootUpper::Scratch) { + RootUpper::Scratch if !self.layers || !self.scratch => { + return Err(refused(format!( + "sandbox {operation} requires --layer and --scratch" + ))); + } + RootUpper::Ram if self.scratch => { + return Err(refused( + "--upper ram and --scratch are mutually exclusive: an upper in RAM has \ + no scratch image", + )); + } + RootUpper::Ram if !self.layers => { + return Err(refused(format!("sandbox {operation} requires --layer"))); + } + RootUpper::Scratch | RootUpper::Ram => {} } } Ok(environment) diff --git a/nvx-rs/src/host/sandbox/options/tests.rs b/nvx-rs/src/host/sandbox/options/tests.rs index 7c029c8ee..1e7589014 100644 --- a/nvx-rs/src/host/sandbox/options/tests.rs +++ b/nvx-rs/src/host/sandbox/options/tests.rs @@ -12,7 +12,7 @@ use crate::ErrorKind; use crate::host::paths::{self, resolve}; use crate::host::sandbox::layer::LayerRole; use crate::host::sandbox::mount::{aggregate_child_name, plan_mounts, validate_mounts}; -use crate::host::sandbox::{SandboxLayer, SandboxSettings}; +use crate::host::sandbox::{SandboxLayer, SandboxSettings, SandboxUpper}; fn message(result: Result) -> String { let error = result.unwrap_err(); @@ -83,7 +83,7 @@ impl Layer { &self.path("distro.erofs"), "11111111-1111-1111-1111-111111111111", )], - &self.path("scratch.ext4"), + SandboxUpper::scratch(&self.path("scratch.ext4")), SandboxSettings { entrypoint: entrypoint.to_owned(), ..SandboxSettings::default() @@ -788,6 +788,70 @@ fn managed_non_launch_operations_reject_network_policy_options() { ); } +#[test] +fn sandbox_launches_take_the_scratch_image_that_their_upper_needs() { + for operation in [SandboxOperation::Run, SandboxOperation::Provision] { + // Without --upper, as with --upper scratch, the upper is on the image of --scratch. + for upper in [None, Some(RootUpper::Scratch)] { + let options = SandboxOptions { + upper, + ..launching(operation) + }; + assert_eq!(options.check().unwrap(), None); + let missing = SandboxOptions { + scratch: false, + ..options + }; + assert_eq!( + message(missing.check()), + format!("sandbox {operation} requires --layer and --scratch") + ); + } + // An upper in RAM has no scratch image, and needs the distro layer alone. + let ram = SandboxOptions { + upper: Some(RootUpper::Ram), + scratch: false, + ..launching(operation) + }; + assert_eq!(ram.check().unwrap(), None); + let with_scratch = SandboxOptions { + scratch: true, + ..ram.clone() + }; + assert_eq!( + message(with_scratch.check()), + "--upper ram and --scratch are mutually exclusive: an upper in RAM has no scratch \ + image" + ); + let without_layer = SandboxOptions { + layers: false, + ..ram + }; + assert_eq!( + message(without_layer.check()), + format!("sandbox {operation} requires --layer") + ); + } + // A provisioned sandbox keeps the upper that its provision recorded. + for operation in [ + SandboxOperation::Start, + SandboxOperation::Exec, + SandboxOperation::Stop, + SandboxOperation::Deprovision, + ] { + for upper in [RootUpper::Scratch, RootUpper::Ram] { + let options = SandboxOptions { + upper: Some(upper), + ..SandboxOptions::new(operation) + }; + assert_eq!( + message(options.check()), + "--upper is only valid for sandbox run or provision" + ); + } + } +} + #[test] fn sandbox_command_rejects_misplaced_mount_options() { for kind in POLICY_KINDS { @@ -887,7 +951,7 @@ impl Shares { &self.path("distro.erofs"), "11111111-1111-1111-1111-111111111111", )], - &self.path("scratch.ext4"), + SandboxUpper::scratch(&self.path("scratch.ext4")), SandboxSettings::default(), )?; let mounts = sandbox_mounts(&mounts, &policy_paths, owner)?; diff --git a/nvx-rs/src/host/sandbox/plan/tests.rs b/nvx-rs/src/host/sandbox/plan/tests.rs index f3da9d27e..2bac136b8 100644 --- a/nvx-rs/src/host/sandbox/plan/tests.rs +++ b/nvx-rs/src/host/sandbox/plan/tests.rs @@ -1044,7 +1044,7 @@ fn launches_plan_their_distro_layer_over_their_scratch_image() { "distro.erofs", EROFS_UUID, )], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), ) .unwrap(); @@ -1056,7 +1056,7 @@ fn launches_plan_their_distro_layer_over_their_scratch_image() { ); assert_eq!(plan.upper().kind(), RootUpper::Scratch); assert_eq!(plan.upper().scratch_path(), Some(Path::new("scratch.ext4"))); - assert_eq!(launch.scratch(), Path::new("scratch.ext4")); + assert_eq!(launch.scratch(), Some(Path::new("scratch.ext4"))); assert_eq!(plan.settings(), Some(&SandboxSettings::default())); // The guest takes the UUID in its canonical text, which `--layer` writes, and nothing else. for invalid in ["", "not-a-uuid", "11111111-1111-1111-1111-11111111111A"] { @@ -1066,7 +1066,7 @@ fn launches_plan_their_distro_layer_over_their_scratch_image() { "distro.erofs", invalid, )], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { entrypoint: "relative".to_owned(), ..SandboxSettings::default() @@ -1081,6 +1081,36 @@ fn launches_plan_their_distro_layer_over_their_scratch_image() { } } +#[test] +fn launches_plan_their_distro_layer_over_ram() { + let launch = SandboxLaunch::new( + vec![SandboxLayer::new( + LayerRole::Distro, + "distro.erofs", + EROFS_UUID, + )], + SandboxUpper::RAM, + SandboxSettings::default(), + ) + .unwrap(); + assert_eq!(launch.upper(), &SandboxUpper::RAM); + assert_eq!(launch.scratch(), None); + let plan = launch.plan(); + assert_eq!(plan.upper().kind(), RootUpper::Ram); + // The distro layer is the one block device, and the guest keeps the upper in RAM. + assert_eq!( + plan.blocks(), + [(BlockRole::Distro, Path::new("distro.erofs"))] + ); + let command_line = plan.kernel_command_line("").unwrap(); + let tokens: Vec<&str> = command_line.split(' ').collect(); + assert!(tokens.contains(&RAM_UPPER_TOKEN), "{command_line}"); + assert!( + !tokens.iter().any(|token| token.starts_with(SCRATCH_TOKEN)), + "{command_line}" + ); +} + #[test] fn launches_plan_the_image_that_the_binding_of_their_distro_layer_records() { let directory = tempfile::tempdir().unwrap(); @@ -1092,7 +1122,7 @@ fn launches_plan_the_image_that_the_binding_of_their_distro_layer_records() { layer.to_str().unwrap(), EROFS_UUID, )], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), ) .unwrap(); @@ -1899,7 +1929,7 @@ fn launches_carry_their_network_into_their_plans() { layer.to_str().unwrap(), EROFS_UUID, )], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), ) .unwrap(); diff --git a/nvx-rs/src/host/sandbox/prepared.rs b/nvx-rs/src/host/sandbox/prepared.rs index 495769b04..c30a75cfc 100644 --- a/nvx-rs/src/host/sandbox/prepared.rs +++ b/nvx-rs/src/host/sandbox/prepared.rs @@ -98,14 +98,17 @@ use crate::host::sys; /// # fn launch( /// # launch: &SandboxLaunch, /// # distro_binding: &DistroBinding, -/// # scratch_binding: &ScratchBinding, +/// # scratch_binding: Option<&ScratchBinding>, /// # capability: &[u8; nvx::protocol::CAPABILITY_LEN], /// # ) -> Result<(), Box> { /// // Check the bindings that the caller recorded when it staged the images: from their metadata -/// // alone, and claiming the scratch image for this sandbox. +/// // alone, and claiming the scratch image of an upper on scratch for this sandbox. /// let distro = distro_binding.check(launch.distro(), DISTRO_POLICY_REVISION)?; -/// let scratch = scratch_binding.check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT)?; -/// let prepared = PreparedLaunch::new(launch.bound_plan(distro_binding)?, distro, Some(scratch), "")?; +/// let scratch = match (launch.scratch(), scratch_binding) { +/// (Some(path), Some(binding)) => Some(binding.check(path, Instant::now() + CLAIM_RELEASE_WAIT)?), +/// _ => None, +/// }; +/// let prepared = PreparedLaunch::new(launch.bound_plan(distro_binding)?, distro, scratch, "")?; /// /// let mut arguments: Vec = /// prepared.openvmm_arguments().iter().map(OsString::from).collect(); diff --git a/nvx-rs/src/host/sandbox/prepared/tests.rs b/nvx-rs/src/host/sandbox/prepared/tests.rs index 0d6ac5dc2..4e65ef0ff 100644 --- a/nvx-rs/src/host/sandbox/prepared/tests.rs +++ b/nvx-rs/src/host/sandbox/prepared/tests.rs @@ -108,7 +108,7 @@ impl Staged { fn launch(&self) -> SandboxLaunch { SandboxLaunch::new( vec![self.layer("distro.erofs")], - &self.text("scratch.ext4"), + SandboxUpper::scratch(&self.text("scratch.ext4")), SandboxSettings::default(), ) .unwrap() @@ -530,7 +530,7 @@ fn prepared_launches_refuse_final_paths_that_openvmm_cannot_take() { let checked = binding.check(&layer, DISTRO_POLICY_REVISION).unwrap(); let launch = SandboxLaunch::new( vec![layer], - &staged.text("scratch.ext4"), + SandboxUpper::scratch(&staged.text("scratch.ext4")), SandboxSettings::default(), ) .unwrap(); @@ -549,7 +549,7 @@ fn prepared_launches_refuse_final_paths_that_openvmm_cannot_take() { let (binding, checked) = staged.checked_distro("distro.erofs"); let launch = SandboxLaunch::new( vec![staged.layer("distro.erofs")], - &staged.text(scratch), + SandboxUpper::scratch(&staged.text(scratch)), SandboxSettings::default(), ) .unwrap(); @@ -613,7 +613,7 @@ fn prepared_launches_accept_balanced_brackets_in_final_paths() { let (binding, checked) = staged.checked_distro("images\\distro.erofs"); let launch = SandboxLaunch::new( vec![staged.layer("images\\distro.erofs")], - &staged.text("images\\scratch.ext4"), + SandboxUpper::scratch(&staged.text("images\\scratch.ext4")), SandboxSettings::default(), ) .unwrap(); diff --git a/nvx-rs/src/host/sandbox/specification.rs b/nvx-rs/src/host/sandbox/specification.rs index bd5cb2ba2..a7d7998c0 100644 --- a/nvx-rs/src/host/sandbox/specification.rs +++ b/nvx-rs/src/host/sandbox/specification.rs @@ -18,7 +18,7 @@ const UUID: &str = "11111111-2222-3333-4444-555555555555"; fn launch(settings: SandboxSettings) -> SandboxLaunch { SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), settings, ) .unwrap() @@ -67,8 +67,12 @@ fn layer_values_are_role_path_uuid() { assert_eq!(layer.path(), Path::new("distro.erofs")); assert_eq!(layer.format(), DistroFormat::Erofs); assert_eq!(layer.uuid(), UUID); - let launch = - SandboxLaunch::new(vec![layer], "scratch.ext4", SandboxSettings::default()).unwrap(); + let launch = SandboxLaunch::new( + vec![layer], + SandboxUpper::scratch("scratch.ext4"), + SandboxSettings::default(), + ) + .unwrap(); assert!( launch .plan() @@ -92,12 +96,12 @@ fn layer_values_are_role_path_uuid() { } refused(SandboxLaunch::new( Vec::new(), - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), )); refused(SandboxLaunch::new( vec![launch.distro().clone(); 2], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), )); } @@ -213,7 +217,7 @@ fn launch_arguments_follow_the_openvmm_contract() { }; let launch = SandboxLaunch::new( vec![layer], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { workload_identity: identity, ..SandboxSettings::default() @@ -320,7 +324,7 @@ fn accepted_launch_shapes_keep_their_arguments_and_kernel_tokens() { "layer.erofs", "11111111-1111-1111-1111-111111111111", )], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), ) .unwrap() @@ -428,7 +432,7 @@ fn launch_settings_follow_the_guest_contract() { for (memory, pids) in [(0_u32, 1_u32), (1, 0), (1, 4_194_304)] { refused(SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { memory_max: Some(u64::from(memory)), pids_max: Some(pids), @@ -439,7 +443,7 @@ fn launch_settings_follow_the_guest_contract() { for entrypoint in ["bin/sh", "/bin/a b", "/bin/a\"b"] { refused(SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { entrypoint: entrypoint.to_owned(), ..SandboxSettings::default() @@ -548,7 +552,7 @@ fn kernel_token_values_refuse_separators() { ] { refused(SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), settings, )); } @@ -626,7 +630,7 @@ fn the_reproduction_of_microsoft_nvx_723_is_refused() { } refused(SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { args: vec!["x\u{e0}tsc=unstable".to_owned()], ..SandboxSettings::default() @@ -676,7 +680,7 @@ proptest! { let uuid = Uuid::from_bytes(bytes).to_string(); let layer = SandboxLayer::parse(&format!("distro,distro.erofs,{uuid}")).unwrap(); prop_assert_eq!(layer.uuid(), &uuid); - let launch = SandboxLaunch::new(vec![layer], "scratch.ext4", SandboxSettings::default()).unwrap(); + let launch = SandboxLaunch::new(vec![layer], SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default()).unwrap(); let command_line = launch.plan().kernel_command_line("").unwrap(); let expected = format!("nvx_layer=distro,0xd0003000,erofs:{uuid}"); prop_assert!(command_line.split_ascii_whitespace().any(|token| token == expected)); @@ -851,7 +855,7 @@ proptest! { let targets = [format!("/share-0/{first}"), format!("/share-1/{second}")]; let launch = SandboxLaunch::new( vec![SandboxLayer::new(LayerRole::Distro, "distro.erofs", UUID)], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { entrypoint: entrypoint.clone(), args: vec![argument.clone()], diff --git a/nvx-rs/src/host/sandbox/tests.rs b/nvx-rs/src/host/sandbox/tests.rs index cff2007d6..9ea8c13e5 100644 --- a/nvx-rs/src/host/sandbox/tests.rs +++ b/nvx-rs/src/host/sandbox/tests.rs @@ -37,8 +37,12 @@ fn mount(value: &str, denied: &[&str]) -> SandboxMount { } fn with_export(export: Option) -> Result { - SandboxLaunch::new(vec![distro()], "scratch.ext4", SandboxSettings::default())? - .with_export(export) + SandboxLaunch::new( + vec![distro()], + SandboxUpper::scratch("scratch.ext4"), + SandboxSettings::default(), + )? + .with_export(export) } /// Returns `path`, an absolute POSIX path, as an absolute host path: on Windows, on drive `C:`. @@ -136,7 +140,7 @@ pub(super) fn native(path: &str) -> String { fn launch_contract_builds_agent_command_line() { let launch = SandboxLaunch::new( vec![distro()], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings { entrypoint: "/bin/workload".to_owned(), args: vec!["--serve".to_owned()], @@ -189,7 +193,7 @@ fn launch_contract_builds_agent_command_line() { fn launch_contract_rejects_duplicates_and_reserved_tokens() { let error = SandboxLaunch::new( vec![distro(), distro()], - "scratch.ext4", + SandboxUpper::scratch("scratch.ext4"), SandboxSettings::default(), ) .unwrap_err(); @@ -238,12 +242,16 @@ fn launch_settings_are_checked_in_order() { let settings = |change: fn(&mut SandboxSettings)| { let mut settings = SandboxSettings::default(); change(&mut settings); - message(SandboxLaunch::new(vec![distro()], "s.ext4", settings)) + message(SandboxLaunch::new( + vec![distro()], + SandboxUpper::scratch("s.ext4"), + settings, + )) }; assert_eq!( message(SandboxLaunch::new( Vec::new(), - "s.ext4", + SandboxUpper::scratch("s.ext4"), SandboxSettings::default() )), "a sandbox requires exactly one read-only distro layer" @@ -251,7 +259,7 @@ fn launch_settings_are_checked_in_order() { assert_eq!( message(SandboxLaunch::new( vec![distro(); 4], - "s.ext4", + SandboxUpper::scratch("s.ext4"), SandboxSettings::default() )), "duplicate sandbox layer role(s): distro" @@ -264,7 +272,11 @@ fn launch_settings_are_checked_in_order() { ..SandboxSettings::default() }; assert_eq!( - message(SandboxLaunch::new(vec![distro()], "s.ext4", settings)), + message(SandboxLaunch::new( + vec![distro()], + SandboxUpper::scratch("s.ext4"), + settings + )), "sandbox entrypoint must be an absolute path without whitespace, control characters, or quotes" ); } @@ -279,7 +291,7 @@ fn launch_settings_are_checked_in_order() { for hostname in ["", "-a", "a-", "A", "a.b", "a\n"] { let launch = SandboxLaunch::new( vec![distro()], - "s.ext4", + SandboxUpper::scratch("s.ext4"), SandboxSettings { hostname: hostname.to_owned(), ..SandboxSettings::default() @@ -310,7 +322,11 @@ fn launch_rejects_limits_the_guest_cgroup_cannot_store() { let settings = |change: fn(&mut SandboxSettings)| { let mut settings = SandboxSettings::default(); change(&mut settings); - message(SandboxLaunch::new(vec![distro()], "s.ext4", settings)) + message(SandboxLaunch::new( + vec![distro()], + SandboxUpper::scratch("s.ext4"), + settings, + )) }; assert_eq!(parse_decimal::("18446744073709551616"), None); assert_eq!( @@ -806,7 +822,7 @@ impl Fixture { &self.path("distro.erofs"), "11111111-1111-1111-1111-111111111111", )], - &self.path("scratch.ext4"), + SandboxUpper::scratch(&self.path("scratch.ext4")), SandboxSettings::default(), ) .unwrap() @@ -1181,7 +1197,7 @@ fn launch_contract_rejects_disk_option_delimiters_and_unbalanced_brackets() { &layer, "11111111-1111-1111-1111-111111111111", )], - &fixture.path("scratch.ext4"), + SandboxUpper::scratch(&fixture.path("scratch.ext4")), SandboxSettings::default(), ) .unwrap(); @@ -1206,7 +1222,7 @@ fn launch_validation_requires_the_layers_and_the_scratch_image() { &fixture.path("missing.erofs"), "11111111-1111-1111-1111-111111111111", )], - &fixture.path("scratch.ext4"), + SandboxUpper::scratch(&fixture.path("scratch.ext4")), SandboxSettings::default(), ) .unwrap(); diff --git a/nvx-rs/src/testing/fake_openvmm.rs b/nvx-rs/src/testing/fake_openvmm.rs index 9340d56a2..32b03682e 100644 --- a/nvx-rs/src/testing/fake_openvmm.rs +++ b/nvx-rs/src/testing/fake_openvmm.rs @@ -60,7 +60,9 @@ //! The fake also serves the sandbox profile of `nvx-rs sandbox`, which a //! `--microvm-sandbox-block` option selects: read-only `distro`, `runtime`, and `custom` layers //! and a `scratch` disk, each an existing file, with `nvx_sandbox=1`, `nvx_entrypoint=`, and -//! `nvx_hostname=` on the kernel command line. Like OpenVMM, it opens each block device at the +//! `nvx_hostname=` on the kernel command line, or, for a RAM-backed overlay, which one +//! `nvx_overlay_upper=ramfs` there requests, as OpenVMM reads it, the `distro` layer alone. Like +//! OpenVMM, it opens each block device at the //! path that its option names, a layer read-only and the scratch disk for reading and writing, //! and holds them until it exits; it reads them whole, and records what each held, with its role //! and path, in `fake-openvmm--blocks.json` next to the kernel. The @@ -1212,8 +1214,10 @@ fn parse(program: &str, arguments: &[String]) -> Result { if !tokens.contains(&"nvx_sandbox=1") { return Err("a sandbox block device requires nvx_sandbox=1".to_owned()); } - let blocks = - check_sandbox_blocks(values.get("--microvm-sandbox-block").into_iter().flatten())?; + let blocks = check_sandbox_blocks( + values.get("--microvm-sandbox-block").into_iter().flatten(), + ram_overlay_requested(&tokens)?, + )?; if capture { // A template's VM boots the invariant configuration alone, without a tenant. if tokens != ["nvx_sandbox=1"] { @@ -1646,9 +1650,11 @@ fn check_control_console(endpoint: &str, snapshot: Option<&Path>) -> Result<(), /// Checks the sandbox block devices of a sandbox: each layer, `ROLE:file:PATH,ro` with a role of /// `distro`, `runtime`, or `custom`, at most once each, and one scratch disk, `scratch:file:PATH`, -/// each an existing regular file, and returns them in order. +/// each an existing regular file, and returns them in order. A RAM-backed overlay, `ram_overlay`, +/// takes the `distro` layer alone instead, as OpenVMM admits it. fn check_sandbox_blocks<'a>( blocks: impl Iterator, + ram_overlay: bool, ) -> Result, String> { let mut roles = BTreeSet::new(); let mut checked = Vec::new(); @@ -1672,12 +1678,36 @@ fn check_sandbox_blocks<'a>( read_only, }); } - if !roles.contains("scratch") || roles.len() < 2 { + if ram_overlay { + if roles != BTreeSet::from(["distro"]) { + return Err( + "microVM RAM-backed overlay requires exactly one read-only distro block".to_owned(), + ); + } + } else if !roles.contains("scratch") || roles.len() < 2 { return Err("a sandbox needs a layer and a scratch disk".to_owned()); } Ok(checked) } +/// Returns whether the kernel command line of `tokens` requests a RAM-backed overlay upper, with +/// one `nvx_overlay_upper=ramfs`, as OpenVMM reads it: a token of that name with another value, +/// none, or a second one fails. +fn ram_overlay_requested(tokens: &[&str]) -> Result { + let values: Vec<&str> = tokens + .iter() + .filter_map(|token| { + let (name, value) = token.split_once('=').unwrap_or((token, "")); + (name == "nvx_overlay_upper").then_some(value) + }) + .collect(); + match values[..] { + [] => Ok(false), + ["ramfs"] => Ok(true), + _ => Err("microVM nvx_overlay_upper must appear once and equal ramfs".to_owned()), + } +} + /// Returns the number of host mappings that the `nvx_maps=` tokens of a kernel command line, /// whose values are `counts`, announce: none without a token, and otherwise from 1 through /// [`MAX_HOST_MAPPINGS`], given once. diff --git a/nvx-rs/tests/checked_handoff.rs b/nvx-rs/tests/checked_handoff.rs index 67558cc84..440df0467 100644 --- a/nvx-rs/tests/checked_handoff.rs +++ b/nvx-rs/tests/checked_handoff.rs @@ -25,7 +25,9 @@ use std::time::{Duration, Instant}; use nvx::host::sandbox::distro::{DISTRO_POLICY_REVISION, DistroBinding}; use nvx::host::sandbox::scratch::{CLAIM_RELEASE_WAIT, ScratchBinding}; -use nvx::host::sandbox::{LayerRole, PreparedLaunch, SandboxLaunch, SandboxLayer, SandboxSettings}; +use nvx::host::sandbox::{ + LayerRole, PreparedLaunch, SandboxLaunch, SandboxLayer, SandboxSettings, SandboxUpper, +}; use nvx::host::supervisor::{self, Launched, VmmCommand}; use nvx::protocol::CAPABILITY_LEN; @@ -315,7 +317,7 @@ fn launch_of(distro: &Path, scratch: &Path) -> SandboxLaunch { distro.to_str().unwrap(), UUID, )], - scratch.to_str().unwrap(), + SandboxUpper::scratch(scratch.to_str().unwrap()), SandboxSettings::default(), ) .unwrap() @@ -474,12 +476,15 @@ fn a_prepared_launch_hands_the_vmm_its_checked_images_alone() { // Staging binds the images, where hashing belongs; the launch checks them from their metadata // alone, and claims the scratch image. let binding = DistroBinding::verify(launch.distro()).unwrap(); - let scratch_binding = ScratchBinding::bind(launch.scratch()).unwrap(); + let scratch_binding = ScratchBinding::bind(launch.scratch().unwrap()).unwrap(); let distro = binding .check(launch.distro(), DISTRO_POLICY_REVISION) .unwrap(); let scratch = scratch_binding - .check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT) + .check( + launch.scratch().unwrap(), + Instant::now() + CLAIM_RELEASE_WAIT, + ) .unwrap(); let prepared = PreparedLaunch::new( launch.bound_plan(&binding).unwrap(), @@ -693,9 +698,13 @@ fn a_vmm_keeps_the_scratch_images_claim_once_its_launcher_is_gone() { let staged = Staged::new(); let launch = staged.launch(); let binding = DistroBinding::verify(launch.distro()).unwrap(); - let scratch_binding = ScratchBinding::bind(launch.scratch()).unwrap(); - let check_scratch = - || scratch_binding.check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT); + let scratch_binding = ScratchBinding::bind(launch.scratch().unwrap()).unwrap(); + let check_scratch = || { + scratch_binding.check( + launch.scratch().unwrap(), + Instant::now() + CLAIM_RELEASE_WAIT, + ) + }; let prepared = PreparedLaunch::new( launch.bound_plan(&binding).unwrap(), binding @@ -739,9 +748,14 @@ fn a_vmm_keeps_the_scratch_images_claim_once_its_launcher_is_gone() { // The claim lasts while the VMM writes the image, which changes its length here, so that a // binding of the written image tells the claim apart, and it ends with the VMM. - let written = ScratchBinding::bind(launch.scratch()).unwrap(); + let written = ScratchBinding::bind(launch.scratch().unwrap()).unwrap(); assert_ne!(written, scratch_binding); - let check_written = || written.check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT); + let check_written = || { + written.check( + launch.scratch().unwrap(), + Instant::now() + CLAIM_RELEASE_WAIT, + ) + }; let claimed = check_written().unwrap_err(); assert_eq!(claimed.kind(), ErrorKind::TimedOut, "{claimed}"); File::create(vmm.path().join(EXIT_MARKER)).unwrap(); @@ -828,12 +842,15 @@ fn a_vmm_finds_the_checked_images_past_replaced_links_from_another_directory() { .map_or_else(|_| link.clone(), Path::to_path_buf); let launch = launch_of(&named.join("distro.erofs"), &named.join("scratch.ext4")); let binding = DistroBinding::verify(launch.distro()).unwrap(); - let scratch_binding = ScratchBinding::bind(launch.scratch()).unwrap(); + let scratch_binding = ScratchBinding::bind(launch.scratch().unwrap()).unwrap(); let distro = binding .check(launch.distro(), DISTRO_POLICY_REVISION) .unwrap(); let scratch = scratch_binding - .check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT) + .check( + launch.scratch().unwrap(), + Instant::now() + CLAIM_RELEASE_WAIT, + ) .unwrap(); let prepared = PreparedLaunch::new( launch.bound_plan(&binding).unwrap(), @@ -894,14 +911,17 @@ fn launcher_fixture() { let staged = Staged::new(); let launch = staged.launch(); let binding = DistroBinding::verify(launch.distro()).unwrap(); - let scratch_binding = ScratchBinding::bind(launch.scratch()).unwrap(); + let scratch_binding = ScratchBinding::bind(launch.scratch().unwrap()).unwrap(); let distro = binding .check(launch.distro(), DISTRO_POLICY_REVISION) .unwrap(); // SAFETY: descriptor 0, standard input, is open, and nothing in this process reads it. nix::unistd::close(unsafe { OwnedFd::from_raw_fd(0) }).unwrap(); let scratch = scratch_binding - .check(launch.scratch(), Instant::now() + CLAIM_RELEASE_WAIT) + .check( + launch.scratch().unwrap(), + Instant::now() + CLAIM_RELEASE_WAIT, + ) .unwrap(); assert_eq!(scratch.file().as_raw_fd(), 0); let prepared = PreparedLaunch::new( @@ -919,7 +939,7 @@ fn launcher_fixture() { assert!(fs::read_link("/proc/self/fd/0").is_err()); assert_eq!( scratch_binding - .check(launch.scratch(), Instant::now()) + .check(launch.scratch().unwrap(), Instant::now()) .unwrap_err() .kind(), ErrorKind::TimedOut diff --git a/nvx-rs/tests/cli/dry_run.rs b/nvx-rs/tests/cli/dry_run.rs index 72b57a5ff..55755cc5a 100644 --- a/nvx-rs/tests/cli/dry_run.rs +++ b/nvx-rs/tests/cli/dry_run.rs @@ -145,6 +145,25 @@ fn sandbox_dry_runs() -> Vec { ) .with_file("distro[[1][2]].erofs", Content::Text("distro\n")) .with_file("scratch[][1].ext4", Content::Text("scratch\n")), + dry_run( + "explicit scratch upper", + sandbox_run(&["--upper", "scratch"]), + one_shot(), + ), + // An upper in RAM attaches the distro layer alone, and asks the guest for a tmpfs upper. + dry_run( + "RAM upper", + [ + "sandbox", + "run", + "--layer", + LAYER, + "--upper", + "ram", + "--dry-run", + ], + one_shot().in_ram(), + ), dry_run( "every layer and option", sandbox_run(&[ diff --git a/nvx-rs/tests/cli/formats.rs b/nvx-rs/tests/cli/formats.rs index 3354c5b0f..d3f8c4514 100644 --- a/nvx-rs/tests/cli/formats.rs +++ b/nvx-rs/tests/cli/formats.rs @@ -1,8 +1,9 @@ //! The configurations of every format that a build of `nvx-rs` or `nvx.py` wrote, from the -//! hand-written `config.json` files of `tests/fixtures/state/`: `start` reads format 7, which this -//! build writes, with shares or without, and format 1, which earlier builds wrote for a sandbox +//! hand-written `config.json` files of `tests/fixtures/state/`: `start` reads format 7, which +//! earlier builds wrote, with shares or without, and format 1, which they wrote for a sandbox //! without shares, and refuses the others, and the configurations that `nvx.py` wrote, without -//! bindings, before it launches OpenVMM. +//! bindings, before it launches OpenVMM. The lifecycle tests cover format 8, which this build +//! writes. //! //! A fixture names the paths of a root as `{root/RELATIVE}`, the hypervisor that a provision //! records on this host as `{hypervisor}`, and the bindings of the root's distro layer and scratch diff --git a/nvx-rs/tests/cli/lifecycle.rs b/nvx-rs/tests/cli/lifecycle.rs index a19f1f0a4..d060f93f7 100644 --- a/nvx-rs/tests/cli/lifecycle.rs +++ b/nvx-rs/tests/cli/lifecycle.rs @@ -24,8 +24,8 @@ use crate::harness::{ in_parallel, is_hex, json, }; use crate::openvmm::{ - NO_NETWORK, NetworkPolicy, Options, WORKLOAD, assert_checked_blocks, assert_endpoint, - assert_opened_images, assert_outcome_report, managed, + NO_NETWORK, NetworkPolicy, Options, WORKLOAD, assert_checked_blocks, assert_checked_distro, + assert_endpoint, assert_opened_distro, assert_opened_images, assert_outcome_report, managed, }; /// The entries of the state directory of a sandbox that never started: its configuration and @@ -65,13 +65,13 @@ pub(crate) fn provision(state: &str, options: &[&str]) -> Vec { sandbox("provision", state, &all) } -/// Returns the configuration that provision records for the minimal sandbox, in format 7, with +/// Returns the configuration that provision records for the minimal sandbox, in format 8, with /// placeholders, and without the bindings of its images, which [`assert_bindings`] checks. pub(crate) fn minimal_config() -> Value { json!({ "cmdline": "", "export": null, - "format": 7, + "format": 8, "hostname": "nvx-sandbox", "hypervisor": HYPERVISOR, "layers": [{"path": "{root/distro.erofs}", "role": "distro", "uuid": UUID}], @@ -80,6 +80,7 @@ pub(crate) fn minimal_config() -> Value { "network": null, "pids_max": null, "scratch": "{root/scratch.ext4}", + "upper": "scratch", "workload_gid": 65534, "workload_uid": 65534, }) @@ -640,6 +641,86 @@ fn lifecycles_of_every_configuration() { in_parallel(&cases, check_lifecycle); } +/// A sandbox whose upper is in RAM needs no scratch image: provision records `ram` and neither a +/// scratch image nor its binding, and each start attaches the distro layer alone, which OpenVMM +/// opens, and asks the guest for the RAM upper. +#[test] +fn lifecycle_of_a_sandbox_whose_upper_is_in_ram() { + let root = Root::new(); + root.remove("scratch.ext4"); + let state = root.below("state"); + operate( + &root, + "provision", + "state", + &["--layer", LAYER, "--upper", "ram"], + ) + .assert_silent_success(); + assert_eq!(entries(&state), PROVISIONED); + let mut config = json(&state.join("config.json")); + let distro = config + .as_object_mut() + .unwrap() + .remove("distro_binding") + .expect("a distro binding"); + let layer = SandboxLayer::new( + LayerRole::Distro, + root.below("distro.erofs").to_str().unwrap(), + UUID, + ); + assert_eq!( + distro, + serde_json::to_value(DistroBinding::verify(&layer).unwrap()).unwrap() + ); + let mut expected = minimal_config(); + for (name, value) in [ + ("upper", json!("ram")), + ("scratch", Value::Null), + ("scratch_binding", Value::Null), + ] { + expected[name] = value; + } + assert_eq!(config, root.expand_json(&expected)); + // The upper of each start is a new one. + for round in 1..=2 { + let context = format!("start {round}"); + operate(&root, "start", "state", &[]).assert_silent_success(); + assert_eq!(entries(&state), running(), "{context}"); + let (endpoint, process) = + assert_runtime_record(&root, "state", &json(&state.join("runtime.json"))); + let mut options = Options::parse(&root.openvmm_arguments("build/fake-openvmm-state.json")); + assert_checked_distro(&root, &options.take("--microvm-sandbox-block")); + assert_opened_distro(&root, "state"); + let mut expected = managed("state", &endpoint).in_ram(); + expected.take("--microvm-sandbox-block"); + expected.assert_matches(&root, &options, &context); + let outcome = operate( + &root, + "exec", + "state", + &["--entrypoint", "/bin/echo", "--arg", "hello"], + ); + outcome.assert_success(); + assert_eq!(outcome.stdout(), "hello\n", "{context}"); + operate(&root, "stop", "state", &[]).assert_silent_success(); + assert!( + process + .wait(Instant::now() + Duration::from_secs(10)) + .unwrap(), + "{context}: OpenVMM runs on" + ); + assert_eq!(entries(&state), STOPPED, "{context}"); + assert_outcome_report( + &json(&state.join("outcome.json")), + HYPERVISOR, + ("managed", "success", 0), + NO_NETWORK, + ); + } + operate(&root, "deprovision", "state", &[]).assert_silent_success(); + assert!(!state.exists()); +} + /// A workload that `exec` runs in a running sandbox, and how it ends. struct Execution { /// The name of the case. @@ -1077,12 +1158,13 @@ fn start_without_openvmm(root: &Root) { /// A start refuses a configuration of a format that it does not read, one that is not an object, /// one that is not strict JSON, one without the fields of its format or with fields of other -/// types or ranges, and one whose format is not that of its shares, before it launches OpenVMM. +/// types or ranges, one whose format is not that of its shares, and one whose upper is not as its +/// format records it, before it launches OpenVMM. fn malformed_configuration(root: &Root) { root.nvx_rs(&provision("{root/state}", &[])) .assert_silent_success(); - // Format 7 records the shares under `export`, `null` for none, and format 1 records no - // `export`. + // Format 8 records the shares under `export`, `null` for none, and the upper under `upper`; + // format 7 records no `upper`, and format 1 records no `export` either. let config = json(&root.below("state/config.json")); let mut without_export = config.clone(); without_export.as_object_mut().unwrap().remove("export"); @@ -1090,22 +1172,35 @@ fn malformed_configuration(root: &Root) { memory_text["memory_mib"] = json!("256"); let mut uid_beyond_32_bits = config.clone(); uid_beyond_32_bits["workload_uid"] = json!(4_294_967_296_u64); - let mut format_1_with_export = config; + let mut without_upper = config.clone(); + without_upper.as_object_mut().unwrap().remove("upper"); + let mut format_7_with_upper = config.clone(); + format_7_with_upper["format"] = json!(7); + let mut other_upper = config.clone(); + other_upper["upper"] = json!("tmpfs"); + let mut scratch_without_image = config.clone(); + scratch_without_image["scratch"] = Value::Null; + // An upper in RAM has no scratch image, nor its binding. + let mut ram_with_scratch = config.clone(); + ram_with_scratch["upper"] = json!("ram"); + let mut ram_with_binding = ram_with_scratch.clone(); + ram_with_binding["scratch"] = Value::Null; + let mut format_1_with_export = without_upper.clone(); format_1_with_export["format"] = json!(1); let unsupported = &["unsupported format", "{root/state/config.json}"][..]; let mismatched = &["format does not match its mounts"][..]; let malformed = &["sandbox configuration is malformed"][..]; for (text, code, fragments) in [ ( - r#"{"format": 8}"#.to_owned(), + r#"{"format": 9}"#.to_owned(), "format-unsupported", unsupported, ), ("[]".to_owned(), "format-unsupported", unsupported), - // Only the integer 7 or 1 is a format, and the configuration is strict JSON, in which no - // object repeats a field. + // Only the integers 8, 7, and 1 are formats, and the configuration is strict JSON, in + // which no object repeats a field. ( - r#"{"format": 7.0}"#.to_owned(), + r#"{"format": 8.0}"#.to_owned(), "format-unsupported", unsupported, ), @@ -1118,7 +1213,7 @@ fn malformed_configuration(root: &Root) { ][..], ), ( - r#"{"format": 7, "format": 7}"#.to_owned(), + r#"{"format": 8, "format": 8}"#.to_owned(), "io", &[ "failed to read sandbox configuration", @@ -1127,8 +1222,15 @@ fn malformed_configuration(root: &Root) { ][..], ), (r#"{"format": 1}"#.to_owned(), "io", malformed), + (r#"{"format": 8}"#.to_owned(), "io", malformed), (memory_text.to_string(), "io", malformed), (uid_beyond_32_bits.to_string(), "io", malformed), + (without_upper.to_string(), "io", malformed), + (format_7_with_upper.to_string(), "io", malformed), + (other_upper.to_string(), "io", malformed), + (scratch_without_image.to_string(), "io", malformed), + (ram_with_scratch.to_string(), "io", malformed), + (ram_with_binding.to_string(), "io", malformed), (without_export.to_string(), "format-unsupported", mismatched), ( format_1_with_export.to_string(), diff --git a/nvx-rs/tests/cli/one_shot.rs b/nvx-rs/tests/cli/one_shot.rs index 22694bf1b..f81df4a6b 100644 --- a/nvx-rs/tests/cli/one_shot.rs +++ b/nvx-rs/tests/cli/one_shot.rs @@ -5,8 +5,9 @@ use crate::harness::{Content, LAYER, OPENVMM, Root, SCRATCH, in_parallel, native}; use crate::openvmm::{ - NO_NETWORK, NetworkPolicy, Options, WORKLOAD, assert_checked_blocks, assert_opened_images, - assert_outcome_report, child, cold_boot, one_shot, restore, share_token, + NO_NETWORK, NetworkPolicy, Options, WORKLOAD, assert_checked_blocks, assert_checked_distro, + assert_opened_distro, assert_opened_images, assert_outcome_report, child, cold_boot, one_shot, + restore, share_token, }; /// The snapshot that a restore restores. @@ -212,6 +213,28 @@ fn sandbox_launches() -> Vec { 1, "", ), + // An upper in RAM attaches the distro layer alone, and needs no scratch image. + launch( + "RAM upper", + [ + "sandbox", + "run", + "--layer", + LAYER, + "--upper", + "ram", + "--entrypoint", + "/bin/echo", + "--arg", + "hello", + "--arg", + "world", + ], + echo().in_ram(), + 0, + "hello world\n", + ) + .without("scratch.ext4"), launch( "missing program", sandbox_run("/bin/missing", &[]), @@ -637,15 +660,23 @@ fn one_shot_runs_exit_with_the_workload_or_openvmm_status() { } let mut options = Options::parse(arguments); let mut expected = case.options.clone(); - // A sandbox's OpenVMM opens the images through the files that the run checked. - if !expected.take("--microvm-sandbox-block").is_empty() { - assert_checked_blocks(&root, &options.take("--microvm-sandbox-block")); + // A sandbox's OpenVMM opens the images through the files that the run checked: the distro + // layer alone for an upper in RAM, and the scratch image too for one on scratch. + let blocks = expected.take("--microvm-sandbox-block"); + if !blocks.is_empty() { + let checked = options.take("--microvm-sandbox-block"); let token = case .record .and_then(|record| record.strip_prefix("build/fake-openvmm-")) .and_then(|name| name.strip_suffix(".json")) .unwrap(); - assert_opened_images(&root, token); + if blocks.len() == 1 { + assert_checked_distro(&root, &checked); + assert_opened_distro(&root, token); + } else { + assert_checked_blocks(&root, &checked); + assert_opened_images(&root, token); + } } expected.assert_matches(&root, &options, case.name); if let Some(report) = &case.report { diff --git a/nvx-rs/tests/cli/openvmm.rs b/nvx-rs/tests/cli/openvmm.rs index fa341b4ab..d2c810df6 100644 --- a/nvx-rs/tests/cli/openvmm.rs +++ b/nvx-rs/tests/cli/openvmm.rs @@ -22,6 +22,10 @@ pub(crate) const SANDBOX: [&str; 3] = [ "nvx_scratch=0xd0006000,ext4", ]; +/// The kernel command-line token with which a sandbox keeps its upper in RAM, in place of the +/// scratch disk's, [`SANDBOX`]'s last. +pub(crate) const RAM_UPPER: &str = "nvx_overlay_upper=ramfs"; + /// The kernel command-line tokens of the default workload of a sandbox: `/bin/sh` as the /// entrypoint, and `nvx-sandbox` as the hostname. pub(crate) const WORKLOAD: [&str; 2] = ["nvx_entrypoint=/bin/sh", "nvx_hostname=nvx-sandbox"]; @@ -96,6 +100,18 @@ impl Options { self.with("--cmdline", [line]) } + /// Returns these options of a sandbox with its upper in RAM: the distro layer of the root + /// alone on a sandbox block device, and [`RAM_UPPER`] in place of the scratch disk's token on + /// the kernel command line. + pub(crate) fn in_ram(self) -> Self { + let line = self.value("--cmdline").replace(SANDBOX[2], RAM_UPPER); + self.with( + "--microvm-sandbox-block", + ["distro:file:{root/distro.erofs},ro"], + ) + .with("--cmdline", [line]) + } + /// Returns the values of `name`, none if the command line does not give it. pub(crate) fn values(&self, name: &str) -> &[String] { self.0.get(name).map_or(&[], Vec::as_slice) @@ -240,10 +256,7 @@ pub(crate) fn assert_checked_blocks(root: &Root, blocks: &[String]) { let [distro, scratch] = blocks else { panic!("not two sandbox block devices: {blocks:?}"); }; - let distro = distro - .strip_prefix("distro:file:") - .and_then(|rest| rest.strip_suffix(",ro")) - .unwrap_or_else(|| panic!("not a read-only distro layer: {distro}")); + let distro = checked_distro(distro); let scratch = scratch .strip_prefix("scratch:file:") .unwrap_or_else(|| panic!("not a scratch disk: {scratch}")); @@ -257,18 +270,63 @@ pub(crate) fn assert_checked_blocks(root: &Root, blocks: &[String]) { format!(r"\\?\{}", root.expand("{root/scratch.ext4}")) ); } else { - let descriptor = |path: &str| { - path.strip_prefix("/proc/self/fd/") - .and_then(|number| number.parse::().ok()) - .unwrap_or_else(|| panic!("not an inherited descriptor: {path}")) - }; assert_ne!(descriptor(distro), descriptor(scratch)); } } +/// Asserts that `blocks`, the `--microvm-sandbox-block` values of a launch whose upper is in RAM, +/// attach the distro layer of the root alone, read-only, through a file that the launch checked, +/// as [`assert_checked_blocks`] says. +pub(crate) fn assert_checked_distro(root: &Root, blocks: &[String]) { + let [distro] = blocks else { + panic!("not one sandbox block device: {blocks:?}"); + }; + let distro = checked_distro(distro); + if cfg!(windows) { + assert_eq!( + distro, + format!(r"\\?\{}", root.expand("{root/distro.erofs}")) + ); + } else { + descriptor(distro); + } +} + +/// Returns the checked file of `block`, a `--microvm-sandbox-block` value that attaches a +/// read-only distro layer. +fn checked_distro(block: &str) -> &str { + block + .strip_prefix("distro:file:") + .and_then(|rest| rest.strip_suffix(",ro")) + .unwrap_or_else(|| panic!("not a read-only distro layer: {block}")) +} + +/// Returns the number of the descriptor that `path`, `/proc/self/fd/N`, names. +fn descriptor(path: &str) -> u32 { + path.strip_prefix("/proc/self/fd/") + .and_then(|number| number.parse::().ok()) + .unwrap_or_else(|| panic!("not an inherited descriptor: {path}")) +} + /// Asserts that the fake OpenVMM of the launch whose records are named for `token` read the /// distro layer and then the scratch image of the root, whole, through its sandbox block devices. pub(crate) fn assert_opened_images(root: &Root, token: &str) { + assert_opened( + root, + token, + &[("distro", "distro\n"), ("scratch", "scratch\n")], + ); +} + +/// Asserts that the fake OpenVMM of the launch whose records are named for `token`, whose upper is +/// in RAM, read the distro layer of the root alone, whole, through its sandbox block device. +pub(crate) fn assert_opened_distro(root: &Root, token: &str) { + assert_opened(root, token, &[("distro", "distro\n")]); +} + +/// Asserts that the fake OpenVMM of the launch whose records are named for `token` read the block +/// devices of `expected`, each with its role and content, whole, in order. +fn assert_opened(root: &Root, token: &str, expected: &[(&str, &str)]) { let opened = root.json(&format!("build/fake-openvmm-{token}-blocks.json")); let opened: Vec<(&str, &str)> = opened .as_array() @@ -281,7 +339,7 @@ pub(crate) fn assert_opened_images(root: &Root, token: &str) { ) }) .collect(); - assert_eq!(opened, [("distro", "distro\n"), ("scratch", "scratch\n")]); + assert_eq!(opened, expected); } /// What an outcome report of OpenVMM's says of a VM's network policy: its status, such as diff --git a/nvx-rs/tests/cli/usage.rs b/nvx-rs/tests/cli/usage.rs index d12896290..0cdf2a201 100644 --- a/nvx-rs/tests/cli/usage.rs +++ b/nvx-rs/tests/cli/usage.rs @@ -182,6 +182,13 @@ const USAGE_ERRORS: &[UsageError] = &[ code: "argument-invalid", fragments: &["--hypervisor", "\"hyperv\""], }, + // The RAM upper's option names it `ram`, whatever the guest mounts for it. + UsageError { + name: "invalid upper", + arguments: &["sandbox", "--upper", "tmpfs"], + code: "argument-invalid", + fragments: &["--upper", "\"tmpfs\"", "\"scratch\"", "\"ram\""], + }, UsageError { name: "invalid operation", arguments: &["sandbox", "restart"], @@ -674,6 +681,7 @@ const HELP: [Help; 6] = [ usage: "usage: nvx-rs sandbox [{run,provision,start,exec,stop,deprovision}] [OPTIONS]", options: &[ "--layer ROLE,PATH,EROFS_UUID", + "--upper {scratch,ram}", "--state-dir STATE_DIR", "--dry-run", ], diff --git a/nvx-rs/tests/cli/validation.rs b/nvx-rs/tests/cli/validation.rs index 7880be35e..3cdcd95d6 100644 --- a/nvx-rs/tests/cli/validation.rs +++ b/nvx-rs/tests/cli/validation.rs @@ -393,6 +393,31 @@ fn sandbox_refusals() -> Vec { ["sandbox", "run", "--scratch", SCRATCH], &["sandbox run", "--layer", "--scratch"], ), + refusal( + "explicit scratch upper without scratch", + "unsupported", + ["sandbox", "run", "--layer", LAYER, "--upper", "scratch"], + &["sandbox run", "--layer", "--scratch"], + ), + // An upper in RAM has no scratch image, and needs the distro layer alone. + refusal( + "RAM upper with scratch", + "unsupported", + sandbox_run(&["--upper", "ram"]), + &["--upper ram", "--scratch", "mutually exclusive"], + ), + refusal( + "RAM upper without layer", + "unsupported", + ["sandbox", "provision", "--upper", "ram"], + &["sandbox provision requires --layer"], + ), + refusal( + "upper outside run", + "unsupported", + ["sandbox", "start", "--upper", "ram"], + &["--upper", "run or provision"], + ), refusal( "policy file with rules", "unsupported", From fef707fbb6beb7f21fafbcf7d534bb7402ad884d Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 15:06:07 -0700 Subject: [PATCH 4/5] tests: accept sandboxes whose overlay upper is in RAM The init agent can now assemble the layered root over a tmpfs upper, and nvx-rs sandbox --upper ram plans that launch, but no scenario booted such a sandbox: the unit tests stop at the shell init's commands and the CLI's fake OpenVMM, and the public sandbox scenarios all attach a scratch image. Nothing showed that the pinned OpenVMM admits the launch, that the kernel mounts the overlay with its options over a tmpfs, or that a RAM upper holds a workload's writes, forgets them at the next start, and charges them to the workload. The new sandbox-ram-upper scenario of test-microvm drives the public nvx-rs sandbox commands as subprocesses, as the other public sandbox scenarios do, with the guest-artifact build's Ubuntu layer and no scratch image: - provision refuses --upper ram beside --scratch, and start refuses --upper, both before they create any state; - a one-shot run of /sbin/nvx-sandbox-smoke prints an OpenVMM command with one sandbox block device, the read-only distro layer, and the RAM upper's token in place of the scratch token, and the probe passes its profile and resource-limit checks, which now require the overlay's upper on a tmpfs. Its outcome report reports success and a complete teardown. The report lies under the scenario's temporary directory, so its path names the run's OpenVMM to the process checks and cleanup, which find VMs by that directory: no other argument of a run without a scratch image does, so a hung run would otherwise outlive cleanup; - a managed sandbox records format 8 with the RAM upper and no scratch image or binding, passes the probe's managed and resource-limit checks, shows its guest one virtio block device and the RAM upper's token, and reads back a file that an earlier request wrote; - under a 32 MiB memory limit, a 20 MiB file in the upper leaves no room for a 20 MiB allocation, which the memory controller kills, and the same allocation succeeds once the file is gone, since the guest has no swap to which the upper's pages could go; - after a stop and a new start, the file is gone, and deprovision removes the state. Like managed-exec-config, sandbox-lifecycle, and sandbox-proxy-only, it needs the Ubuntu layer that packages omit, so it runs only when named, and only with Alpine control. The public managed sandbox acceptance of run-nvx-microvm-tests.yml names it on Linux and Windows, so every backend runs it. A change to the scenario's module, like one to the other test drivers, leaves the performance inputs alone. test_microvm_tests.py runs the scenario against a fake nvx-rs that emulates a RAM upper's planner and guest, and shows that it fails for each regression that it must detect: a scratch image accepted beside a RAM upper or recorded for it, an upper accepted by start, a scratch block device or token, a failed probe, a run whose OpenVMM command does not name the fixture or that reports a failure or an incomplete teardown, an uncharged upper, and a file that survives a restart. It also covers the dispatch of the scenario, and that it runs only when named. The documentation lists the scenario with the other public sandbox scenarios, and Run describes what it accepts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/run-nvx-microvm-tests.yml | 2 + doc/ci.md | 11 +- doc/contribute.md | 5 +- doc/project-structure.md | 1 + doc/run.md | 24 ++ doc/usage.md | 10 +- scripts/nvx_tools/ci.py | 2 +- scripts/nvx_tools/microvm_tests.py | 21 +- scripts/nvx_tools/sandbox_ram_upper_tests.py | 321 +++++++++++++++++ scripts/test_microvm_tests.py | 351 +++++++++++++++++++ scripts/test_nvx_tools.py | 1 + 11 files changed, 733 insertions(+), 16 deletions(-) create mode 100644 scripts/nvx_tools/sandbox_ram_upper_tests.py diff --git a/.github/workflows/run-nvx-microvm-tests.yml b/.github/workflows/run-nvx-microvm-tests.yml index b8920a8df..42e15292d 100644 --- a/.github/workflows/run-nvx-microvm-tests.yml +++ b/.github/workflows/run-nvx-microvm-tests.yml @@ -211,6 +211,7 @@ jobs: --scenario managed-exec-config --scenario sandbox-lifecycle --scenario sandbox-proxy-only + --scenario sandbox-ram-upper --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run public managed sandbox acceptance on Windows @@ -222,6 +223,7 @@ jobs: --scenario managed-exec-config --scenario sandbox-lifecycle --scenario sandbox-proxy-only + --scenario sandbox-ram-upper --output-dir "build/test-results/public-exec-${{ inputs.backend }}" - name: Run Ubuntu guest smoke tests on Linux diff --git a/doc/ci.md b/doc/ci.md index c2664a973..de66eb81b 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -409,9 +409,9 @@ container cgroup, the overlay root over read-only image layers with its upper directory on scratch, and that the memory controller kills an allocation of twice the limit and the pids controller stops the workload at eight processes. The live-share runs below repeat the profile checks. Each -backend also runs the public `managed-exec-config`, `sandbox-lifecycle`, and -`sandbox-proxy-only` scenarios of `test-microvm`, which drive -`nvx-rs sandbox provision`, `start`, +backend also runs the public `managed-exec-config`, `sandbox-lifecycle`, +`sandbox-proxy-only`, and `sandbox-ram-upper` scenarios of `test-microvm`, +which drive `nvx-rs sandbox run`, `provision`, `start`, `exec`, `stop`, and `deprovision` as subprocesses (see [Run](run.md#managed-lifecycle)): request-scoped working directories and environments, timeouts, nonzero exits, stdout and stderr, bounded output and @@ -419,8 +419,9 @@ outcome reports, warm-guest state across requests and restarts, identity refusal, fail-closed lifecycle transitions, a crashed OpenVMM's stale runtime state, teardown that leaves no OpenVMM process, control endpoint, capability, or runtime record while the layer and scratch artifacts survive, -and a proxy-only network, whose guest reaches a test proxy on host loopback at -its gateway and no other destination. +a proxy-only network, whose guest reaches a test proxy on host loopback at +its gateway and no other destination, and sandboxes whose overlay upper is a +tmpfs in guest memory, with no scratch image. Their bounded evidence is part of the backend's failure logs. The same entrypoint then verifies a live virtio-fs share inside the container: a read-write `/workspace` share with a denied diff --git a/doc/contribute.md b/doc/contribute.md index b9c13bb81..469c11fd2 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -100,8 +100,9 @@ devices. Add those scenarios to `scripts/nvx_tools/microvm_test_scripts` and retain complete failure logs. The scenarios that drive the public `nvx-rs sandbox` commands live in `scripts/nvx_tools/managed_exec_tests.py`, -`scripts/nvx_tools/sandbox_lifecycle_tests.py`, and -`scripts/nvx_tools/sandbox_proxy_tests.py`, and `microvm_tests.py` +`scripts/nvx_tools/sandbox_lifecycle_tests.py`, +`scripts/nvx_tools/sandbox_proxy_tests.py`, and +`scripts/nvx_tools/sandbox_ram_upper_tests.py`, and `microvm_tests.py` registers them. `sandbox` and the control protocol belong to `nvx-rs` ([microsoft/nvx#359](https://github.com/microsoft/nvx/issues/359)), so the scenarios drive them as commands, compile egress policies with diff --git a/doc/project-structure.md b/doc/project-structure.md index ebdbed8ce..b1ecabff4 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -144,6 +144,7 @@ nvx/ | | |-- release.py Release source collection and packaging | | |-- sandbox_lifecycle_tests.py Managed sandbox lifecycle and security tests | | |-- sandbox_proxy_tests.py Proxy-only sandbox networking tests +| | |-- sandbox_ram_upper_tests.py Sandboxes whose overlay upper is in RAM | | |-- time_abi.py Time ABI harness checks and CPU profile guidance | | `-- ubuntu.py Verified Ubuntu rootfs and EROFS preparation | |-- setup/ Host and Actions runner bootstrap scripts diff --git a/doc/run.md b/doc/run.md index 709c270b8..237f35baa 100644 --- a/doc/run.md +++ b/doc/run.md @@ -965,6 +965,30 @@ refuses a RAM upper beside a scratch block device. With either upper, the overlay keeps `metacopy=on` and `xino=on`, which `/sbin/nvx-sandbox-smoke` checks. +The explicit `test-microvm --scenario sandbox-ram-upper --backend BACKEND` +scenario is the acceptance for the RAM upper. Like the +[managed lifecycle](#managed-lifecycle) scenarios, it runs `nvx-rs sandbox` as +subprocesses with the guest-artifact build's Ubuntu layer, so it also runs only +when named, but it attaches no scratch image. It checks that `provision` +refuses `--upper ram` beside `--scratch`, and `start` refuses `--upper`, before +either creates any state. A one-shot `run` of `/sbin/nvx-sandbox-smoke` must +print an OpenVMM command with one sandbox block device, the read-only distro +layer, and `nvx_overlay_upper=ramfs` in place of the scratch token, and the +probe must pass its profile and resource-limit checks over a tmpfs upper. The +run's `--outcome-report`, under the scenario's temporary directory, must report +success and a complete teardown, and also names its OpenVMM to the scenario's +process checks and cleanup, which find VMs by that directory. A managed sandbox +must record `ram` without a scratch image, pass the probe's managed and +resource-limit checks, show its guest one virtio block device and the RAM +upper's token, and read back a file that an earlier request wrote. +Under a 32 MiB `--memory-max`, a 20 MiB file in its upper must leave no room +for a 20 MiB allocation, which the memory controller kills, and which succeeds +once the file is removed. After a `stop` and a new `start`, the file is gone. +The scenario's evidence, with a `sandbox-ram-upper-` prefix, holds its bounded +command observations, the runtime record, the output of the one-shot run, of +the probe, of the guest's view, and of the memory check, and the outcome +reports of the one-shot run and of both stops. + ### Live host shares `sandbox run` and `sandbox provision` accept repeatable diff --git a/doc/usage.md b/doc/usage.md index 22f16b179..16391ef55 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1104,19 +1104,19 @@ python3 scripts/nvx.py test-microvm Runs NVX-owned Linux, SMP, virtio, sandbox, and snapshot correctness scenarios against the public OpenVMM CLI. Repeat `--scenario` to select a subset; without -it, every scenario supported by the selected guest runs, except four that run +it, every scenario supported by the selected guest runs, except five that run only when named. `smp-lapic` repeats `smp` and also asserts that every CPU uses the one-shot counting LAPIC. `managed-exec-config`, `sandbox-lifecycle`, -and `sandbox-proxy-only` drive the public `nvx-rs sandbox` lifecycle -commands (see [Run](run.md#managed-lifecycle)) and also require the +`sandbox-proxy-only`, and `sandbox-ram-upper` drive the public `nvx-rs sandbox` +lifecycle commands (see [Run](run.md#managed-lifecycle)) and also require the guest-artifact build's Ubuntu layer and scratch template, which packages omit. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, `platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, -`platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, and -`sandbox-proxy-only` scenarios. Ubuntu also rejects +`platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, +`sandbox-proxy-only`, and `sandbox-ram-upper` scenarios. Ubuntu also rejects the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index a42cc0b66..16ed82779 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -263,7 +263,7 @@ r"crate_release|create_linux_source_archive|managed_exec_tests|" r"microvm_durations|microvm_shards|microvm_tests|nightly_ci|" r"openvmm_process|openvmm_test_shards|sandbox_lifecycle_tests|" - r"sandbox_proxy_tests|shard_plans)\.py$|" + r"sandbox_proxy_tests|sandbox_ram_upper_tests|shard_plans)\.py$|" r"(microvm|openvmm_test)_shards\.json$|adversarial_cases/|" r"microvm_test_scripts/))" ) diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index cd643451b..ea8a2b357 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -73,6 +73,7 @@ from .openvmm_process import OpenvmmProcess, TcpConsole from .sandbox_lifecycle_tests import run_sandbox_lifecycle from .sandbox_proxy_tests import run_sandbox_proxy_only +from .sandbox_ram_upper_tests import run_sandbox_ram_upper from .shard_plans import parse_shard from .time_abi import ( ABI_VERSION, @@ -131,6 +132,7 @@ "sandbox-blocks", "sandbox-lifecycle", "sandbox-proxy-only", + "sandbox-ram-upper", "scratch-snapshot", "smp", "smp-snapshot", @@ -153,6 +155,7 @@ "sandbox-blocks", "sandbox-lifecycle", "sandbox-proxy-only", + "sandbox-ram-upper", "scratch-snapshot", "snapshot-tiers", ) @@ -161,7 +164,12 @@ # the guest-artifact build produces and packages omit, so they run only when # named. PUBLIC_SANDBOX_SCENARIOS = frozenset( - ("managed-exec-config", "sandbox-lifecycle", "sandbox-proxy-only") + ( + "managed-exec-config", + "sandbox-lifecycle", + "sandbox-proxy-only", + "sandbox-ram-upper", + ) ) # The same-host restore cases also run on the CI debug kernel (doc/ci.md), # whose soft-lockup and hung-task detectors the guest's time ABI watcher @@ -521,8 +529,9 @@ def configure_parser(parser: argparse.ArgumentParser) -> None: choices=(*MICROVM_TEST_SCENARIOS, *MICROVM_EXPLICIT_SCENARIOS), help=( "scenario to run; repeat to select multiple (default: all except " - "smp-lapic, managed-exec-config, sandbox-lifecycle, and " - "sandbox-proxy-only, which run only when named)" + "smp-lapic, managed-exec-config, sandbox-lifecycle, " + "sandbox-proxy-only, and sandbox-ram-upper, which run only when " + "named)" ), ) parser.add_argument( @@ -9127,6 +9136,12 @@ def _run_scenarios( run_sandbox_proxy_only( args.backend, timeout=args.timeout, output_dir=output_dir ) + if "sandbox-ram-upper" in scenarios: + report.start( + "sandbox-ram-upper", + f"Running public RAM-upper sandboxes on OpenVMM/{args.backend}", + ) + run_sandbox_ram_upper(args.backend, timeout=args.timeout, output_dir=output_dir) if "managed-lifecycle" in scenarios: report.start( "managed-lifecycle", diff --git a/scripts/nvx_tools/sandbox_ram_upper_tests.py b/scripts/nvx_tools/sandbox_ram_upper_tests.py new file mode 100644 index 000000000..6a535ea4a --- /dev/null +++ b/scripts/nvx_tools/sandbox_ram_upper_tests.py @@ -0,0 +1,321 @@ +"""Public CLI acceptance for a sandbox whose overlay upper is in RAM. + +The scenario runs `nvx-rs sandbox --upper ram` as subprocesses against the +real kernel, the Alpine control initramfs, and the Ubuntu EROFS layer, with no +scratch image. It checks that the planner refuses `--upper ram` beside +`--scratch`, and `--upper` outside `run` and `provision`, before it creates +any state. A one-shot run attaches the distro layer alone, declares the RAM +upper on the kernel command line, passes the smoke probe, which requires the +overlay's upper on a tmpfs, and reports its success. A managed sandbox records +the RAM upper without a scratch image, and passes the probe's managed and +resource-limit checks. Its guest has one virtio block device and declares the +RAM upper, a later request reads a file that an earlier one wrote, and a file +in the upper counts against the workload's memory limit. Once the sandbox +stops and starts again, the file is gone. +""" + +from __future__ import annotations + +import json +import tempfile +from pathlib import Path +from typing import cast + +from .managed_exec_tests import load_workload_artifacts +from .sandbox_lifecycle_tests import ( + CONFIG, + OUTCOME, + PROBE, + PROVISIONED_ENTRIES, + STOPPED_ENTRIES, + LifecycleAcceptance, + Refusal, + check_managed_report, + managed_report, + run_acceptance, +) + +MEMORY_MIB = 256 +MEMORY_MAX = 32 * 1024 * 1024 +PIDS_MAX = 8 +RAM_UPPER_TOKEN = "nvx_overlay_upper=ramfs" +SCRATCH_TOKEN = "nvx_scratch=" +# The outcome that OpenVMM reports for a one-shot run whose workload succeeded. +RUN_SUCCESS = {"operation": "run", "category": "success", "status_code": 0} +STATE_PATH = "/tmp/nvx-ram-upper-state" +STATE_CONTENT = b"kept in RAM\n" +# A file of FILL_MIB MiB in the RAM upper leaves too little of MEMORY_MAX for +# an allocation of ALLOCATION_MIB MiB, which the memory controller then kills, +# and the same allocation fits once the file is gone, since the guest has no +# swap to which the upper's pages could go. +FILL_MIB = 20 +ALLOCATION_MIB = 20 +CHARGE_MARKER = b"NVX-RAM-UPPER-CHARGED" +CHARGE_SCRIPT = f"""\ +head -c {FILL_MIB}M /dev/zero >/tmp/nvx-ram-upper-fill || exit 10 +status=0 +{{ dd if=/dev/zero of=/dev/null bs={ALLOCATION_MIB}M count=1; }} 2>/dev/null || + status=$? +rm -f /tmp/nvx-ram-upper-fill +[ "$status" -eq 137 ] || exit 11 +{{ dd if=/dev/zero of=/dev/null bs={ALLOCATION_MIB}M count=1; }} 2>/dev/null || + exit 12 +echo {CHARGE_MARKER.decode()} +""" +# The guest's block devices, one per line, then its kernel command line. +VIEW_SCRIPT = "ls /sys/block && cat /proc/cmdline" + + +def probe_markers(*, entry: bool) -> tuple[bytes, ...]: + """Returns what the smoke probe reports when its profile, limit, and, for + a managed execution, `entry` checks pass.""" + markers = [b"NVX-UBUNTU-SANDBOX-PROFILE-OK uid=65534 gid=65534"] + if entry: + markers.append(b"NVX-UBUNTU-SANDBOX-ENTRY-OK") + markers.extend( + ( + f"NVX-UBUNTU-SANDBOX-LIMITS-OK memory_max={MEMORY_MAX} " + f"pids_max={PIDS_MAX}".encode(), + b"NVX-UBUNTU-SANDBOX-OK uid=65534 gid=65534", + ) + ) + return tuple(markers) + + +def check_ram_tokens(command_line: str, context: str) -> None: + """Checks that `command_line`, or a quoted command that holds it, declares + the RAM upper once, and no scratch upper.""" + tokens = [token.strip("'\"") for token in command_line.split()] + if tokens.count(RAM_UPPER_TOKEN) != 1 or any( + token.startswith(SCRATCH_TOKEN) for token in tokens + ): + raise RuntimeError( + f"{context} does not declare the RAM upper alone: {command_line!r}" + ) + + +def check_run_report(path: Path, backend: str) -> None: + """Checks the outcome report of a one-shot run whose workload succeeded and + whose VM was torn down.""" + try: + report: object = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError) as error: + raise RuntimeError("the one-shot run published no outcome report") from error + if not isinstance(report, dict): + raise RuntimeError(f"the one-shot run's outcome report is {report!r}") + fields = cast(dict[str, object], report) + outcome = fields.get("outcome") + if outcome != RUN_SUCCESS or fields.get("backend") != backend: + raise RuntimeError( + f"the one-shot run reported {outcome!r} on {fields.get('backend')!r}" + ) + # microvm_tests, which runs this scenario, imports this module. + from .microvm_tests import OUTCOME_TEARDOWN_FIELDS + + teardown = fields.get("teardown") + if ( + not isinstance(teardown, dict) + or set(cast(dict[str, object], teardown)) != set(OUTCOME_TEARDOWN_FIELDS) + or not all( + value is True for value in cast(dict[str, object], teardown).values() + ) + ): + raise RuntimeError(f"the one-shot run's teardown is {teardown!r}") + + +class RamUpperAcceptance(LifecycleAcceptance): + evidence_prefix = "sandbox-ram-upper-" + + def ram_options(self) -> list[str]: + return [ + "--hypervisor", + self.backend, + "--layer", + f"distro,{self.artifacts.distro},{self.artifacts.layer_uuid}", + "--upper", + "ram", + "--memory-mib", + str(MEMORY_MIB), + "--memory-max", + str(MEMORY_MAX), + "--pids-max", + str(PIDS_MAX), + ] + + def run(self) -> None: + self.check_refusals() + self.check_one_shot() + self.check_managed() + self.require_processes((), "after the RAM-upper sandboxes") + + def check_refusals(self) -> None: + """The planner refuses a scratch image for a RAM upper, and an upper + for an operation other than `run` and `provision`, before it creates + any state.""" + self.invoke( + "provision", + *self.ram_options(), + "--scratch", + str(self.artifacts.scratch_template), + state=self.state, + expected=1, + refusal=Refusal("unsupported", ("--upper ram", "--scratch")), + ) + self.invoke( + "start", + "--upper", + "ram", + state=self.state, + expected=1, + refusal=Refusal("unsupported", ("--upper", "run or provision")), + ) + if self.state.exists(): + raise RuntimeError("a refused RAM-upper request created sandbox state") + + def check_one_shot(self) -> None: + """A one-shot run attaches the distro layer alone, declares the RAM + upper, passes the smoke probe, and reports its success.""" + # The report's path, under the fixture, also names the run's OpenVMM to + # the process checks and to cleanup, as no other argument does: the + # distro layer is the build's, and the run has no scratch image. + report = self.root / "run-outcome.json" + result = self.invoke( + "run", + *self.ram_options(), + "--outcome-report", + str(report), + "--entrypoint", + PROBE, + "--arg=limits", + f"--arg={MEMORY_MAX}", + f"--arg={PIDS_MAX}", + state=None, + ) + console = result.stdout + result.stderr + self.save_evidence("run.log", console) + lines = result.stdout.splitlines() + command = lines[0].decode(errors="replace") if lines else "" + if ( + not command.startswith(">> ") + or command.count("--microvm-sandbox-block") != 1 + or "distro:file:" not in command + or "scratch:file:" in command + ): + raise RuntimeError( + f"the one-shot run did not attach its distro layer alone: {command!r}" + ) + if self.root.name not in command: + raise RuntimeError( + f"the one-shot run's OpenVMM command does not name the fixture: " + f"{command!r}" + ) + check_ram_tokens(command, "the one-shot run's OpenVMM command") + for marker in probe_markers(entry=False): + if marker not in result.stdout: + raise RuntimeError( + f"the one-shot RAM-upper probe did not report {marker.decode()}" + ) + self.copy_evidence(report, "run-outcome.json") + check_run_report(report, self.backend) + self.require_processes((), "after the one-shot run") + + def check_managed(self) -> None: + """A managed sandbox records the RAM upper without a scratch image, and + its upper holds the workload's writes until the sandbox stops.""" + self.invoke("provision", *self.ram_options(), state=self.state) + self.require_entries(self.state, PROVISIONED_ENTRIES, "after provision") + config = json.loads((self.state / CONFIG).read_text(encoding="utf-8")) + recorded = { + name: config.get(name) + for name in ("format", "upper", "scratch", "scratch_binding") + } + if recorded != { + "format": 8, + "upper": "ram", + "scratch": None, + "scratch_binding": None, + }: + raise RuntimeError(f"provision recorded the RAM upper as {recorded!r}") + self.start(description="first start") + result = self.exec( + PROBE, + "--arg=entry", + "--arg=limits", + f"--arg={MEMORY_MAX}", + f"--arg={PIDS_MAX}", + ) + self.save_evidence("probe.log", result.stdout + result.stderr) + for marker in probe_markers(entry=True): + if marker not in result.stdout: + raise RuntimeError( + f"the managed RAM-upper probe did not report {marker.decode()}" + ) + self.check_guest_view() + result = self.exec( + "/bin/sh", + "--arg=-c", + f"--arg=printf 'kept in RAM\\n' >{STATE_PATH}", + expected=None, + ) + self.require_output(result, b"", "the RAM upper's state-writing request") + result = self.exec("/bin/cat", f"--arg={STATE_PATH}", expected=None) + self.require_output(result, STATE_CONTENT, "the RAM upper's state") + result = self.exec( + "/bin/sh", "--arg=-c", f"--arg={CHARGE_SCRIPT}", expected=None + ) + self.save_evidence("charge.log", result.stdout + result.stderr) + self.require_output( + result, CHARGE_MARKER + b"\n", "the memory charge of the RAM upper" + ) + self.stop_sandbox("first stop") + # A new start begins with an empty upper. + self.start(description="start after the stop") + result = self.exec( + "/bin/sh", "--arg=-c", f"--arg=test ! -e {STATE_PATH}", expected=None + ) + self.require_output(result, b"", "the restarted sandbox's empty upper") + self.stop_sandbox("second stop") + self.invoke("deprovision", state=self.state) + if self.state.exists(): + raise RuntimeError("deprovision kept the RAM-upper sandbox's state") + + def check_guest_view(self) -> None: + """The guest of a RAM upper has one virtio block device, the distro + layer's, and declares the RAM upper on its kernel command line.""" + result = self.exec("/bin/sh", "--arg=-c", f"--arg={VIEW_SCRIPT}", expected=None) + self.save_evidence("view.log", result.stdout + result.stderr) + lines = result.stdout.decode(errors="replace").splitlines() + if result.returncode != 0 or not lines: + raise RuntimeError( + f"the guest view of the RAM upper returned {result.returncode}" + ) + devices = [name for name in lines[:-1] if name.startswith("vd")] + if devices != ["vda"]: + raise RuntimeError( + f"the guest of a RAM upper has the virtio block devices {devices}" + ) + check_ram_tokens(lines[-1], "the guest's kernel command line") + + def stop_sandbox(self, description: str) -> None: + self.invoke("stop", state=self.state) + self.require_entries(self.state, STOPPED_ENTRIES, f"after the {description}") + self.copy_evidence( + self.state / OUTCOME, f"{description.replace(' ', '-')}-outcome.json" + ) + check_managed_report(managed_report(self.state), self.backend) + self.require_processes((), f"after the {description}") + + +def run_sandbox_ram_upper(backend: str, *, timeout: float, output_dir: Path) -> None: + artifacts = load_workload_artifacts() + output_dir.mkdir(parents=True, exist_ok=True) + root = Path(tempfile.mkdtemp(prefix="nvx-public-ram-upper-")) + run_acceptance( + RamUpperAcceptance( + backend, + timeout=timeout, + output_dir=output_dir, + artifacts=artifacts, + root=root, + ) + ) diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index c0d8479f2..11e3dd354 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -44,6 +44,7 @@ openvmm_process, sandbox_lifecycle_tests, sandbox_proxy_tests, + sandbox_ram_upper_tests, time_abi, ) from nvx_tools.build_constants import ( # noqa: E402 @@ -2603,6 +2604,321 @@ def test_scripts_awaiting_a_release_turn_the_echo_off_first(self): self.assertLess(script.index(ready), script.index(release)) +class _FakeRamUpperSandboxCli(_FakeSandboxCli): + """Emulates `nvx-rs sandbox --upper ram` and its guest: no scratch image, + a distro layer alone, the RAM upper's token, and an upper that each start + begins empty. + + Each fault names a regression of the planner, the guest, or the record + that the acceptance must detect. + """ + + SMOKE_RUN = ( + b"NVX-UBUNTU-SANDBOX-PROFILE-OK uid=65534 gid=65534\n" + + f"NVX-UBUNTU-SANDBOX-LIMITS-OK " + f"memory_max={sandbox_ram_upper_tests.MEMORY_MAX} " + f"pids_max={sandbox_ram_upper_tests.PIDS_MAX}\n".encode() + + b"NVX-UBUNTU-SANDBOX-OK uid=65534 gid=65534\n" + ) + + def _command_line(self) -> str: + upper = ( + "nvx_scratch=0xd0006000,ext4" + if "declares-scratch" in self.faults + else sandbox_ram_upper_tests.RAM_UPPER_TOKEN + ) + return f"nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs {upper}" + + def _run( + self, state: Path | None, values: dict[str, str], _: list[str] + ) -> _CliResult: + assert state is None + assert values["--upper"] == "ram" + assert values["--entrypoint"] == sandbox_lifecycle_tests.PROBE + assert _ == [ + "limits", + str(sandbox_ram_upper_tests.MEMORY_MAX), + str(sandbox_ram_upper_tests.PIDS_MAX), + ] + blocks = f"--microvm-sandbox-block distro:file:{values['--layer']},ro" + if "attaches-scratch" in self.faults: + blocks += " --microvm-sandbox-block scratch:file:scratch.ext4" + command = f">> openvmm {blocks}" + if "drops-report" not in self.faults: + report = Path(values["--outcome-report"]) + command += f" --microvm-report {report}" + outcome = ( + {"operation": "run", "category": "guest-exit", "status_code": 1} + if "run-reports-failure" in self.faults + else sandbox_ram_upper_tests.RUN_SUCCESS + ) + teardown = dict.fromkeys(microvm_tests.OUTCOME_TEARDOWN_FIELDS, True) + if "partial-teardown" in self.faults: + del teardown["virtiofs_released"] + if "incomplete-teardown" in self.faults: + teardown["openvmm_process_terminated"] = False + report.write_text( + json.dumps( + { + "backend": values["--hypervisor"], + "outcome": outcome, + "teardown": teardown, + } + ), + encoding="utf-8", + ) + command += f" --cmdline '{self._command_line()}'\n" + output = b"" if "probe-fails" in self.faults else self.SMOKE_RUN + return 0, command.encode() + output, b"" + + def _provision( + self, state: Path | None, values: dict[str, str], _: list[str] + ) -> _CliResult: + if "--scratch" in values and "accepts-scratch" not in self.faults: + return self._error( + "unsupported", + "--upper ram and --scratch are mutually exclusive: an upper in RAM " + "has no scratch image", + ) + result = super()._provision(state, values, _) + if result[0] == 0: + assert state is not None + path = state / "config.json" + config = json.loads(path.read_text(encoding="utf-8")) + config.update(format=8, upper="ram", scratch=None, scratch_binding=None) + if "records-scratch" in self.faults: + config.update(upper="scratch", scratch="scratch.ext4") + path.write_text(json.dumps(config), encoding="utf-8") + return result + + def _start( + self, state: Path | None, values: dict[str, str], _: list[str] + ) -> _CliResult: + if "--upper" in values and "start-takes-upper" not in self.faults: + return self._error( + "unsupported", "--upper is only valid for sandbox run or provision" + ) + result = super()._start(state, values, _) + # A RAM upper begins empty at each start. + if result[0] == 0 and state is not None and "keeps-state" not in self.faults: + self.guest_files.pop(state, None) + return result + + def _exec( + self, state: Path | None, values: dict[str, str], arguments: list[str] + ) -> _CliResult: + assert state is not None + state_path = sandbox_ram_upper_tests.STATE_PATH + if values["--entrypoint"] != "/bin/sh" or arguments[1].startswith("test ! -e "): + return super()._exec(state, values, arguments) + _, refusal = self._running(state) + if refusal is not None: + return refusal + script = arguments[1] + if script == sandbox_ram_upper_tests.VIEW_SCRIPT: + devices = "vda\nvdb\n" if "has-scratch-device" in self.faults else "vda\n" + return 0, f"loop0\n{devices}{self._command_line()}\n".encode(), b"" + if script == f"printf 'kept in RAM\\n' >{state_path}": + self.guest_files.setdefault(state, {})[state_path] = ( + sandbox_ram_upper_tests.STATE_CONTENT + ) + return 0, b"", b"" + if script == sandbox_ram_upper_tests.CHARGE_SCRIPT: + if "uncharged" in self.faults: + return 11, b"", b"" + return 0, sandbox_ram_upper_tests.CHARGE_MARKER + b"\n", b"" + raise AssertionError(f"unexpected script {script!r}") + + +class PublicSandboxRamUpperTests(unittest.TestCase): + EXPECTED_OPERATIONS = [ + # The planner refuses a scratch image beside a RAM upper, and an upper + # outside run and provision. + "provision", + "start", + # A one-shot run. + "run", + # A managed sandbox: the probe, the guest's view, a written and a read + # file, and the memory charge of a file in the upper. + "provision", + "start", + "exec", + "exec", + "exec", + "exec", + "exec", + "stop", + # Its next start begins with an empty upper. + "start", + "exec", + "stop", + "deprovision", + ] + + def _run(self, root: Path, fake: _FakeRamUpperSandboxCli) -> Path: + distro = root / "ubuntu-distro.erofs" + distro.write_bytes(b"distro") + template = root / "ubuntu-smoke-scratch.ext4" + template.write_bytes(bytes(4096)) + artifacts = managed_exec_tests.WorkloadArtifacts( + distro, "12345678-1234-1234-1234-123456789abc", template + ) + output_dir = root / "results" + mkdtemp = tempfile.mkdtemp + + def fixture(prefix: str) -> str: + return mkdtemp(prefix=prefix, dir=root) + + with ( + patch.object( + sandbox_ram_upper_tests, + "load_workload_artifacts", + return_value=artifacts, + ), + patch.object( + sandbox_ram_upper_tests.tempfile, "mkdtemp", side_effect=fixture + ), + patch.object(sandbox_lifecycle_tests.subprocess, "run", side_effect=fake), + patch.object( + sandbox_lifecycle_tests.nvx_rs, + "require_binary", + return_value=Path("nvx-rs"), + ), + patch.object( + sandbox_lifecycle_tests, + "openvmm_processes", + side_effect=fake.processes, + ), + patch.object( + sandbox_lifecycle_tests, + "control_endpoint_present", + side_effect=fake.endpoint_present, + ), + patch.object( + sandbox_lifecycle_tests, + "terminate_process", + side_effect=fake.terminate, + ), + patch.object( + sandbox_lifecycle_tests, "openvmm_threads_remain", return_value=False + ), + ): + sandbox_ram_upper_tests.run_sandbox_ram_upper( + "mshv", timeout=2, output_dir=output_dir + ) + return output_dir + + def test_acceptance_runs_sandboxes_whose_upper_is_in_ram(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + fake = _FakeRamUpperSandboxCli() + output_dir = self._run(root, fake) + checks = json.loads( + (output_dir / "sandbox-ram-upper-checks.json").read_text( + encoding="utf-8" + ) + ) + evidence = {path.name for path in output_dir.iterdir()} + fixtures = list(root.glob("nvx-public-ram-upper-*")) + + self.assertEqual(fake.operations, self.EXPECTED_OPERATIONS) + self.assertEqual(fixtures, []) + self.assertFalse(any(fake.running.values())) + self.assertEqual( + evidence, + { + "sandbox-ram-upper-checks.json", + "sandbox-ram-upper-runtime.json", + "sandbox-ram-upper-run.log", + "sandbox-ram-upper-run-outcome.json", + "sandbox-ram-upper-probe.log", + "sandbox-ram-upper-view.log", + "sandbox-ram-upper-charge.log", + "sandbox-ram-upper-first-stop-outcome.json", + "sandbox-ram-upper-second-stop-outcome.json", + }, + ) + self.assertEqual( + [record["operation"] for record in checks], self.EXPECTED_OPERATIONS + ) + # Every launch keeps its upper in RAM, and only the refused one names a + # scratch image. + launches = [ + record["argv"] + for record in checks + if record["operation"] in {"run", "provision"} + ] + for argv in launches: + self.assertEqual(argv[argv.index("--upper") + 1], "ram") + self.assertEqual( + ["--scratch" in argv for argv in launches], [True, False, False] + ) + + def test_acceptance_detects_ram_upper_regressions(self): + regressions = { + "accepts-scratch": "provision returned 0, expected 1", + "start-takes-upper": "with error 'unsupported'", + "attaches-scratch": "did not attach its distro layer alone", + "declares-scratch": "does not declare the RAM upper alone", + "probe-fails": "the one-shot RAM-upper probe did not report", + "drops-report": "the one-shot run's OpenVMM command does not name", + "run-reports-failure": "the one-shot run reported", + "partial-teardown": "the one-shot run's teardown is", + "incomplete-teardown": "the one-shot run's teardown is", + "records-scratch": "provision recorded the RAM upper as", + "has-scratch-device": "has the virtio block devices ['vda', 'vdb']", + "uncharged": "the memory charge of the RAM upper returned status 11", + "keeps-state": "the restarted sandbox's empty upper returned status 1", + } + for fault, message in regressions.items(): + with ( + self.subTest(fault=fault), + tempfile.TemporaryDirectory() as temporary, + ): + root = Path(temporary) + fake = _FakeRamUpperSandboxCli(fault) + with self.assertRaisesRegex(RuntimeError, re.escape(message)): + self._run(root, fake) + # Cleanup ends every OpenVMM process and removes the fixture. + self.assertFalse(any(fake.running.values())) + self.assertEqual(list(root.glob("nvx-public-ram-upper-*")), []) + self.assertTrue( + (root / "results" / "sandbox-ram-upper-checks.json").is_file() + ) + + def test_charge_script_needs_a_kill_that_only_the_upper_file_causes(self): + shell = _posix_shell() + if shell is None: + self.skipTest("POSIX shell is unavailable") + # The stand-in dd is killed as the memory controller kills the real one: + # while the file holds the upper's pages, or always, or never. + for dd, expected in ( + ('[ -e "$fill" ] && return 137; return 0', (0, "NVX-RAM-UPPER-CHARGED\n")), + ("return 0", (11, "")), + ("return 137", (12, "")), + ): + with self.subTest(dd=dd), tempfile.TemporaryDirectory() as temporary: + fill = f"{Path(temporary).as_posix()}/fill" + script = sandbox_ram_upper_tests.CHARGE_SCRIPT.replace( + "/tmp/nvx-ram-upper-fill", fill + ) + result = subprocess.run( + [ + shell, + "-c", + f"fill='{fill}'\nhead() {{ :; }}\ndd() {{ {dd}; }}\n{script}", + ], + capture_output=True, + text=True, + timeout=5, + check=False, + ) + self.assertEqual( + (result.returncode, result.stdout), expected, result.stderr + ) + self.assertFalse(Path(fill).exists()) + + class GuestIdentityScriptTests(unittest.TestCase): def test_guest_identity_checks_fail_before_success_markers(self): descriptor = microvm_tests.guest_descriptor("ubuntu") @@ -12740,6 +13056,36 @@ def require(path: Path, _description: str) -> Path: ) lifecycle.assert_not_called() + def test_runner_dispatches_public_sandbox_ram_upper(self): + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "whp", + "--scenario", + "sandbox-ram-upper", + "--output-dir", + str(output_dir), + ] + ) + + def require(path: Path, _description: str) -> Path: + return path + + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file", side_effect=require), + patch.object(microvm_tests, "run_sandbox_ram_upper") as run, + patch.object(microvm_tests, "run_sandbox_lifecycle") as lifecycle, + ): + self.assertEqual(microvm_tests.run(args), 0) + run.assert_called_once_with( + "whp", timeout=args.timeout, output_dir=output_dir + ) + lifecycle.assert_not_called() + def test_runner_runs_public_sandbox_scenarios_only_when_named(self): def require(path: Path, _description: str) -> Path: return path @@ -12757,6 +13103,7 @@ def require(path: Path, _description: str) -> Path: "managed-exec-config", "sandbox-lifecycle", "sandbox-proxy-only", + "sandbox-ram-upper", "guest-boot", ), ), @@ -12764,6 +13111,7 @@ def require(path: Path, _description: str) -> Path: patch.object(microvm_tests, "run_guest_boot") as guest_boot, patch.object(microvm_tests, "run_sandbox_lifecycle") as lifecycle, patch.object(microvm_tests, "run_sandbox_proxy_only") as proxy_only, + patch.object(microvm_tests, "run_sandbox_ram_upper") as ram_upper, patch.object( microvm_tests, "run_managed_exec_configuration" ) as managed_exec, @@ -12773,9 +13121,12 @@ def require(path: Path, _description: str) -> Path: guest_boot.assert_called_once() lifecycle.assert_not_called() proxy_only.assert_not_called() + ram_upper.assert_not_called() managed_exec.assert_not_called() self.assertIn("sandbox-lifecycle", microvm_tests.SANDBOX_CONTROL_SCENARIOS) self.assertIn("sandbox-proxy-only", microvm_tests.SANDBOX_CONTROL_SCENARIOS) + self.assertIn("sandbox-ram-upper", microvm_tests.SANDBOX_CONTROL_SCENARIOS) + self.assertIn("sandbox-ram-upper", microvm_tests.PUBLIC_SANDBOX_SCENARIOS) def test_managed_executions_enter_through_the_agent_not_the_container_helpers(self): root = Path(__file__).resolve().parent.parent diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 3ff2251ac..928c5f384 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -13573,6 +13573,7 @@ def test_ci_runs_the_public_sandbox_acceptance_on_every_backend(self): self.assertIn("--scenario managed-exec-config", acceptance) self.assertIn("--scenario sandbox-lifecycle", acceptance) self.assertIn("--scenario sandbox-proxy-only", acceptance) + self.assertIn("--scenario sandbox-ram-upper", acceptance) self.assertIn( '--output-dir "build/test-results/public-exec-${{ inputs.backend }}"', acceptance, From 27efb148bdcbb5cdcf58413e6d2bd36cfd0ba1a7 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 28 Oct 2026 15:09:04 -0700 Subject: [PATCH 5/5] doc: describe the RAM upper in the architecture guide The architecture guide still described the layered sandbox root as an image layer over private ext4 scratch: its goals, Table 4.1, Figure 4.1, and the assembly steps named only scratch, and 4.1 said that the guest agents always keep the overlay's upper on scratch and refuse a RAM upper, so that the machine's RAM-backed overlay layout served only other guests. The guest agents now assemble the root over either upper, and nvx-rs sandbox --upper ram launches it. - 1.1 makes the workload filesystem an image layer, a private writable upper on scratch or in guest memory, and live host shares, and the chapter 4 contents call 4.1's subject the writable upper. - 4.1 lists tmpfs with extended attributes among the kernel facilities, which overlayfs needs for metadata-only copy-up on a tmpfs upper. Its table, its figure, and its token list describe both uppers, and its steps refuse a scratch device beside a RAM upper while they resolve the roles, mount scratch or a tmpfs at the kernel's default size, half of the guest's RAM, and unmount whichever upper they mounted. It now refuses only a GPT image, and it compares the two uppers' resources: scratch spends host storage and keeps a managed sandbox's changes from one start to the next, while a RAM upper spends guest memory that the writer's memory cgroup is charged for, cannot be reclaimed without swap, fails with ENOSPC beyond the tmpfs size, and dies with the virtual machine. A RAM upper uses the RAM-backed overlay layout, which has one read-only distro block, and supports neither capture nor restore. - 4.3 notes that the workload's memory limit also bounds the files that it writes into a RAM upper. - 5.2 notes that a managed sandbox's writes persist across a stop and start on scratch, but not in a RAM upper, which each start begins empty. - 2.1, 3.4, and 5.1, which described the RAM-backed overlay layout only as what the machine admits, now say that the guest agents keep its upper on a tmpfs, and Table 2.7 no longer presents the scratch block as always present: it is absent in that layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/1.1-goals.md | 4 +- .../2.1-machine-model-and-configuration.md | 4 +- ...device-topology-transport-and-discovery.md | 2 +- doc/design/3.4-block-storage.md | 3 +- doc/design/4-guest-runtime-architecture.md | 2 +- .../4.1-guest-filesystem-and-image-model.md | 55 ++++++++++++------- .../4.3-isolation-and-resource-model.md | 5 +- doc/design/5.1-boot-protocol.md | 5 +- .../5.2-readiness-and-runtime-control.md | 5 +- 9 files changed, 56 insertions(+), 29 deletions(-) diff --git a/doc/design/1.1-goals.md b/doc/design/1.1-goals.md index ad6550f1c..6e02344d3 100644 --- a/doc/design/1.1-goals.md +++ b/doc/design/1.1-goals.md @@ -11,6 +11,6 @@ The architecture has the following goals: - Capture a running VM into immutable artifacts that can be restored in a new process without serializing native host handles. - Run exactly one workload per microVM. -- Assemble the workload filesystem from an immutable image layer, private - writable scratch, and optional live host shares. +- Assemble the workload filesystem from an immutable image layer, a private + writable upper on scratch or in guest memory, and optional live host shares. - Keep workload policy and host-resource attachment under host control. diff --git a/doc/design/2.1-machine-model-and-configuration.md b/doc/design/2.1-machine-model-and-configuration.md index d3701baa2..469fca199 100644 --- a/doc/design/2.1-machine-model-and-configuration.md +++ b/doc/design/2.1-machine-model-and-configuration.md @@ -79,7 +79,9 @@ disks. The roles are the layers of the sandbox root filesystem: a read-only A disk without a role is rejected, and a disk-backed topology ends with writable scratch. The only exception is the RAM-backed overlay layout, which requires exactly one read-only `distro` block and no scratch, keeps the -guest's overlay upper layer in RAM, and supports neither capture nor restore. +guest's overlay upper layer in RAM, on a tmpfs that the guest agents mount +(see [Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root)), +and supports neither capture nor restore. Snapshot capture with blocks requires a read-only `distro` layer followed by writable scratch, both backed by cached, regular raw files with nonzero, 512-byte-aligned geometry. Blockless machines are also snapshot-capable and diff --git a/doc/design/2.5-device-topology-transport-and-discovery.md b/doc/design/2.5-device-topology-transport-and-discovery.md index 73d351f6e..0f3ba1707 100644 --- a/doc/design/2.5-device-topology-transport-and-discovery.md +++ b/doc/design/2.5-device-topology-transport-and-discovery.md @@ -16,7 +16,7 @@ are resolved. | Host filesystem | `fs:microvm0` | `0xd0001000..0xd0001fff` | 6 | Reserved slot; attachment optional | | Boot console | `console:microvm-virtio0` | `0xd0002000..0xd0002fff` | 7 | Optional | | `distro` block | `blk:sandbox:distro` | `0xd0003000..0xd0003fff` | 4 | Optional, read-only | -| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Writable final role | +| `scratch` block | `blk:sandbox:scratch` | `0xd0006000..0xd0006fff` | 11 | Optional, writable final role; absent in the RAM-backed overlay layout | | Control console | `console:microvm-control0` | `0xd0007000..0xd0007fff` | 3 | Optional | Placement is explicit rather than allocated sequentially, so an omitted device diff --git a/doc/design/3.4-block-storage.md b/doc/design/3.4-block-storage.md index f7f96e2c5..4df589d8c 100644 --- a/doc/design/3.4-block-storage.md +++ b/doc/design/3.4-block-storage.md @@ -15,7 +15,8 @@ arbitrary disks. Roles are unique and appear in fixed order; an omitted `distro` role leaves its slot empty. A disk-backed topology ends with scratch, except in the -RAM-backed overlay layout, which uses exactly one read-only `distro` block. +RAM-backed overlay layout, which uses exactly one read-only `distro` block and +whose guest keeps the overlay's upper on a tmpfs in guest memory. The role, not Linux enumeration order, defines identity. ## 3.4.1 Construction and Discovery diff --git a/doc/design/4-guest-runtime-architecture.md b/doc/design/4-guest-runtime-architecture.md index 2acbe7e88..9e741c1b1 100644 --- a/doc/design/4-guest-runtime-architecture.md +++ b/doc/design/4-guest-runtime-architecture.md @@ -17,7 +17,7 @@ isolated workload sandbox. ## Table of Contents - [4.1 Guest Filesystem and Image Model](4.1-guest-filesystem-and-image-model.md): - Explains image layers, writable scratch, overlay assembly, and live shares. + Explains image layers, the writable upper, overlay assembly, and live shares. - [4.2 Guest Agents and Workload Model](4.2-guest-agents-and-workload-model.md): Defines the relationship between the guest agents and the workload. - [4.3 Isolation and Resource Model](4.3-isolation-and-resource-model.md): diff --git a/doc/design/4.1-guest-filesystem-and-image-model.md b/doc/design/4.1-guest-filesystem-and-image-model.md index 56578c9de..4d64c77b5 100644 --- a/doc/design/4.1-guest-filesystem-and-image-model.md +++ b/doc/design/4.1-guest-filesystem-and-image-model.md @@ -9,23 +9,26 @@ As Table 4.1 shows, the guest agents run workloads in one of two roots. | Root | Construction | Host content | | --- | --- | --- | -| Layered sandbox root | An overlay of one read-only image layer and private writable scratch | Live host shares bound at declared targets | +| Layered sandbox root | An overlay of one read-only image layer and a private writable upper, on scratch or in guest memory | Live host shares bound at declared targets | | Blockless managed root | The guest's own initramfs root, without block devices | Host directories bound by a host mapping table | The guest kernel provides the facilities both roots need: virtio-blk, -virtio-fs, EROFS with compression and extended attributes, overlayfs, cgroup -v2 with the memory and process controllers, namespaces, BPF system calls, and -cgroup BPF programs. +virtio-fs, EROFS with compression and extended attributes, tmpfs with +extended attributes, overlayfs, cgroup v2 with the memory and process +controllers, namespaces, BPF system calls, and cgroup BPF programs. ## 4.1.1 Layered Sandbox Root As Figure 4.1 shows, the layered root combines one immutable EROFS `distro` -layer with one private ext4 scratch device. +layer with one private writable upper: either an ext4 scratch device or a +tmpfs in guest memory. ```text /run/nvx/layers/distro (EROFS) ----------> lower directory /run/nvx/scratch/upper (ext4) -----------> upper directory + or /run/nvx/tmpfs/upper (tmpfs) /run/nvx/scratch/work (same ext4) ------> work directory + or /run/nvx/tmpfs/work (same tmpfs) overlay --> /run/nvx/rootfs one-shot: virtio-fs tag microvm (optional share) ---> /run/nvx/rootfs/TARGET or an aggregate ---> /run/nvx/shares, each child bound at @@ -51,15 +54,15 @@ one contract module defines, with the host's renderer and a parser for guests: filesystem UUID or the disk GUID in canonical lowercase text. No token names a partition GUID. - `nvx_scratch=0xd0006000,ext4` keeps the overlay's upper on the whole-device - ext4 `scratch` disk, and `nvx_overlay_upper=ramfs` keeps it in RAM, with no - `scratch` block device. + ext4 `scratch` disk, and `nvx_overlay_upper=ramfs` keeps it in guest + memory, on a tmpfs, with no `scratch` block device. Guests refuse a token without a value, a repeated token, both uppers, an image without an upper or an upper without an image, a RAM upper beside a `scratch` block device, and every other form, including the earlier `nvx_layer` form without a format, rather than probe the devices. The guest agents described -here assemble only an EROFS image over scratch: they refuse a GPT image and a -RAM upper with a message that says so. +here assemble only an EROFS image, over either upper: they refuse a GPT image +with a message that says so. The guest agents assemble the root in this order: @@ -67,12 +70,15 @@ The guest agents assemble the root in this order: container cgroups, and move the guest agents into the guest-agent cgroup. 2. Read the root's boot tokens, and resolve each role's MMIO address through the kernel's platform-device resources rather than assuming a device name - or enumeration order. + or enumeration order. For a RAM upper, refuse any device at the `scratch` + role's address. 3. Check any expected sector count, flush cached block-device buffers, mount the `distro` layer read-only as EROFS, and check its filesystem UUID when the token names one. -4. Resolve and flush scratch, mount it as ext4 with `rw,nosuid,nodev`, and - create its upper and work directories. +4. Mount the upper's file system, and create the overlay's upper and work + directories on it: resolve and flush scratch and mount it as ext4 with + `rw,nosuid,nodev`, or mount a tmpfs with `mode=0755,nosuid,nodev` at the + kernel's default size, half of the guest's RAM. 5. Mount the overlay with the `distro` layer as its lower directory, metadata-only copy-up, and extended inode numbers. 6. Verify the workload identity in the assembled root, mount each live share @@ -83,21 +89,32 @@ The guest agents assemble the root in this order: before releasing it; a managed sandbox's agent first maps the shares from the export, as their mapping table orders. 8. Wait for the workload, unmount the shares in reverse order and then the - overlay, the `distro` layer, and scratch, and report the workload's status. + overlay, the `distro` layer, and the upper, and report the workload's + status. The UUID check detects attachment mistakes; it is not a content-integrity proof, and neither the guest nor the VMM verifies layer contents. Metadata-only copy-up interprets overlay metadata stored in the `distro` layer, so that layer is a trusted input that must not carry untrusted overlay attributes. A metadata-only change avoids copying file contents, but a data -write still copies the whole lower file into scratch, so scratch capacity must -account for copy-up. Scratch consumes host storage rather than guest RAM and -can run out of space independently of guest memory pressure. +write still copies the whole lower file into the upper, so the upper's +capacity must account for copy-up. The two uppers spend different resources: + +- Scratch consumes host storage rather than guest RAM, can run out of space + independently of guest memory pressure, and keeps a managed sandbox's + changes from one start to the next. +- A RAM upper consumes guest memory. The memory cgroup of the process that + writes a page, normally the workload's, is charged for it, so the + workload's memory limit also bounds its files, and since the guest has no + swap, those pages stay resident until their files are removed. A write + beyond the tmpfs size fails with `ENOSPC`, and nothing the workload writes + outlives the virtual machine. The initramfs remains the guest agents' root; it is not another workload -layer. The guest agents always keep the overlay upper layer on scratch, so the -machine's RAM-backed overlay layout, which has no scratch, serves only guests -that declare it. +layer. A RAM upper uses the machine's RAM-backed overlay layout, which has +exactly one read-only `distro` block and no scratch, and which supports +neither capture nor restore; see +[Storage Configuration](2.1-machine-model-and-configuration.md#213-storage-configuration). ## 4.1.2 Live Host Shares diff --git a/doc/design/4.3-isolation-and-resource-model.md b/doc/design/4.3-isolation-and-resource-model.md index 3b4f96816..28a0d27a9 100644 --- a/doc/design/4.3-isolation-and-resource-model.md +++ b/doc/design/4.3-isolation-and-resource-model.md @@ -67,7 +67,10 @@ released. The guest agents' cgroup carries a default memory protection of 16 MiB, a soft reservation that shields the supervisor from reclaim under workload memory pressure but does not guarantee its survival. The workload cgroup carries optional memory and process-count limits and is the unit of -capture freeze, termination, and descendant cleanup. +capture freeze, termination, and descendant cleanup. When the overlay's upper +is in guest memory, the files that the workload writes into its root are +charged to the workload cgroup, so its memory limit also bounds them; see +[Layered Sandbox Root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root). In the blockless managed root, each execution joins a dedicated cgroup before the workload runs, for termination and descendant accounting. This path does diff --git a/doc/design/5.1-boot-protocol.md b/doc/design/5.1-boot-protocol.md index 29d9b5146..5000e8698 100644 --- a/doc/design/5.1-boot-protocol.md +++ b/doc/design/5.1-boot-protocol.md @@ -83,8 +83,9 @@ Launchers add runtime tokens of their own: the [boot tokens of a layered root](4.1-guest-filesystem-and-image-model.md#411-layered-sandbox-root), one share token per host-share child, the number of host mappings for a blockless managed root, and a request to create the workload account. A caller -may also declare the RAM-backed overlay layout with exactly one -`nvx_overlay_upper=ramfs` token; any other form of that token is rejected. +may also declare the RAM-backed overlay layout, whose overlay upper the guest +agents keep on a tmpfs, with exactly one `nvx_overlay_upper=ramfs` token; any +other form of that token is rejected. Embedded NULs are rejected, and the complete NUL-terminated command line must fit in 64 KiB. The effective string and its SHA-256 digest become part of the diff --git a/doc/design/5.2-readiness-and-runtime-control.md b/doc/design/5.2-readiness-and-runtime-control.md index ee6c9c00f..da35dd8b5 100644 --- a/doc/design/5.2-readiness-and-runtime-control.md +++ b/doc/design/5.2-readiness-and-runtime-control.md @@ -57,7 +57,10 @@ Lifecycle transitions fail closed and take effect one at a time per sandbox, while executions run beside them. The host identifies the running VMM process of each sandbox, reconciles VMM processes with recorded state, and recovers interrupted launches. Teardown leaves no VMM process, control endpoint, -capability, or runtime record, while supplied layers and scratch survive. +capability, or runtime record, while supplied layers and scratch survive. A +sandbox's writes to its root therefore persist from one start to the next on +scratch, but not when its overlay's upper is in guest memory, which each start +begins empty. ## 5.2.5 Authority