Skip to content

sandbox: keep the overlay upper on scratch or in RAM - #977

Open
Pedro Henrique Penna (ppenna) wants to merge 5 commits into
devfrom
agents/issue-558-ram-overlay-upper
Open

Pedro Henrique Penna (ppenna) wants to merge 5 commits into
devfrom
agents/issue-558-ram-overlay-upper

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #558.

NVX documented a RAM upper for the layered sandbox root, but only scratch worked end to end. OpenVMM at the pin already admits the RAM-backed overlay layout: exactly one nvx_overlay_upper=ramfs token, exactly one read-only distro block, no scratch, and no capture or restore. The typed plan, PreparedLaunch, and nvx::guest::root support it too. What blocked it:

  • the shell init, nvx-init-agent, refused the token;
  • the smoke probe required /run/nvx/scratch/upper;
  • nvx-rs sandbox required --scratch;
  • no scenario booted the layout;
  • the architecture guide said that the guest agents always keep the upper on scratch.

This PR makes the RAM upper work end to end:

  • The init mounts a tmpfs for the upper when the command line declares the RAM layout, and refuses a scratch device beside it.
  • The probe checks the upper that the command line declares.
  • nvx-rs sandbox run and provision take --upper scratch|ram.
  • A new sandbox-ram-upper scenario runs on every backend in CI.
  • The guides describe both uppers.

Based on dev at 2d14547b6. There is no OpenVMM change: the pinned 832d5bb48, whose two commits since b92daa350 concern platform restores, boots these sandboxes as they are.

Decisions on the issue's open questions

  • Name and token. The token keeps its value, nvx_overlay_upper=ramfs, so OpenVMM needs no change or pin promotion, and the docs say that the guest mounts a tmpfs. The CLI value is ram, the serde form of RootUpper::Ram in the shared plan.
  • Where the guest support lands. In the shell nvx-init-agent, which boots layered sandboxes today. Its contract is the one that nvx::guest::root implements, so the Rust PID 1 of nvx-rs: run NVX's layered sandbox root from a Rust PID 1 on the guest library #817 can replace it unchanged.
  • Size. The kernel's default, half of the guest's RAM, as nvx::guest::root mounts it. A write beyond it fails with ENOSPC. There is no host token.
  • Memory limit. The upper shares the workload's limit: the writer's memory cgroup is charged for tmpfs pages, and without swap they stay resident until their files are removed. The new scenario shows a 20 MiB file under a 32 MiB --memory-max getting a 20 MiB allocation killed, and the same allocation succeeding once the file is gone.

Of #818, this delivers the RAM upper: --upper ram plans no scratch block or token, binds and claims no scratch, and the configuration names the upper. Its GPT images and protected paths remain. #591's draft docs may need a rebase on the doc/design/ changes here.

Change

Commit Files Change
4691e61f1 guest guest/common/nvx-init-agent, scripts/test_nvx_tools.py, nvx-rs/tests/fixtures/root-tokens.json, nvx-rs/src/protocol/root.rs, doc/project-structure.md The init accepts nvx_overlay_upper=ramfs under the shared parser's rules, refuses a scratch device at 0xd0006000 beside it before it mounts anything, mounts tmpfs with mode=0755,nosuid,nodev under /run/nvx/tmpfs, builds the overlay with metacopy=on,xino=on over either upper, and unmounts whichever one it mounted. The shared token vectors no longer mark the RAM upper as a shell refusal.
c31fc862a guest guest/common/nvx-sandbox-smoke, scripts/test_nvx_tools.py, doc/run.md The smoke probe requires the overlay's upper on the upper that /proc/cmdline declares, and requires metacopy=on and xino=on, which overlayfs would drop without a word on an upper that refuses its xattrs. It reads all of /proc/cmdline, as the init agent does, because a caller's --cmdline keeps its newlines.
36bfa904c nvx-rs nvx-rs/src/host/cli/sandbox/*, nvx-rs/src/host/sandbox/*, nvx-rs/src/testing/fake_openvmm.rs, nvx-rs/tests/*, nvx-rs/README.md, doc/run.md, doc/usage.md, .github/workflows/run-nvx-microvm-tests.yml `--upper scratch
c280e5e64 tests scripts/nvx_tools/sandbox_ram_upper_tests.py, scripts/nvx_tools/microvm_tests.py, scripts/nvx_tools/ci.py, scripts/test_microvm_tests.py, scripts/test_nvx_tools.py, .github/workflows/run-nvx-microvm-tests.yml, doc/*.md The sandbox-ram-upper scenario runs only when named, and each backend's public sandbox acceptance names it (described below).
cdc1246b5 doc doc/design/1.1, 2.1, 2.5, 3.4, 4, 4.1, 4.3, 5.1, 5.2 The guide describes both uppers: Table 4.1, Figure 4.1, the assembly and teardown steps, the resource trade-offs, the RAM layout's restrictions, the memory accounting, and the loss of a RAM upper's writes at stop. It drops the "always on scratch" statements and gives the scratch block's absence in Table 2.7.

The scenario

test-microvm --scenario sandbox-ram-upper drives the public commands as subprocesses with the Ubuntu layer and no scratch image:

  • provision --upper ram --scratch … and start --upper ram are refused before any state exists.
  • A one-shot run of /sbin/nvx-sandbox-smoke prints an OpenVMM command with one block, --microvm-sandbox-block distro:file:…,ro, and nvx_overlay_upper=ramfs without nvx_scratch=. The probe passes its profile and limit checks. The run's --outcome-report reports success and a complete teardown. Its path, under the scenario's directory, also lets the process checks and cleanup find the run's VM, which no other argument would.
  • A managed sandbox records {format: 8, upper: "ram", scratch: null, scratch_binding: null}, passes the probe's entry and limit checks, and has one vd* device and the RAM token in its view of the guest. A later request reads a file that an earlier one wrote.
  • The memory-charge check described above.
  • After stop and a new start, the file is gone. deprovision removes the state.

Its unit tests run it against a fake nvx-rs and show that it fails for each regression it must catch:

  • scratch accepted beside a RAM upper, or recorded for one;
  • start accepting --upper;
  • a scratch block or token;
  • a failed probe;
  • a one-shot command that drops the report, or a run that reports a failure or an incomplete teardown;
  • an uncharged upper;
  • a file that survives a restart.

Validation

The branch was first validated on 1853252ca, at 77ba6c624; since then, the last commit has gained the 2.1, 2.5, 3.4, and 5.1 wording, and the branch has been rebased onto 50210fe97, ba30ff904, and 2d14547b6. Guest artifacts came from this branch: the Alpine initramfs and the Ubuntu layer, built from the Dockerfile's initramfs-artifacts and ubuntu-distro-layer-artifacts targets. Their provenance records the new nvx-init-agent and nvx-sandbox-smoke. The kernel and OpenVMM b92daa350 were reused; the local runs predate dev's move to 832d5bb48, which CI covers.

After the rebase, which resolved the nvx-rs README conflict with #968 by moving this PR's change to nvx::host::sandbox into the module's section:

  • On Windows:
    • nvx-rs: fmt, the three Clippy runs, Clippy for musl, the three test runs (980, 841, and 260 library tests), doc -D warnings, and the 1.89 check.
    • The runtime adapter's configurations.
    • ruff and pyright.
    • The Markdown links.
    • The validate-nvx suite, apart from test_setup_azure.py: 1,324 tests.
  • On Linux/KVM, at e17f060dc: the four public sandbox scenarios.

After the second rebase, which kept #971's bracket cases in three test files and gave their launches this PR's SandboxUpper, on Windows:

  • The same nvx-rs gates: 1,149, 922, and 260 tests.
  • The runtime adapter, ruff, pyright, and the Markdown links.
  • 1,335 tests of the suite. One of them, test_host_loopback_allow_script_checks_publication_and_traffic, intermittently exceeds its 10-second timeout under Git's sh on this machine: it passed and failed on reruns of this branch, which doesn't touch its script.

After the third rebase, which listed platform-aggregate-export from #948 beside sandbox-ram-upper in doc/usage.md, on Windows:

  • ruff, pyright, and the Markdown links.
  • 1,343 tests of the suite. Under heavy load, test_host_loopback_allow_script_checks_publication_and_traffic and test_private_network_ingress_script_checks_policy_and_traffic timed out under Git's sh. That rebase brought no change to nvx-rs, the runtime adapter, or the guest.

For the review fixes:

  • On Linux/KVM:
    • The probe's test classes, shellcheck, and shfmt.
    • With the Ubuntu layer rebuilt, nvx-rs sandbox --upper ram and --upper scratch boot and pass the probe with --cmdline "$(printf 'quiet\nloglevel=0')" --hostname nvx-multiline, whose root tokens reach the guest after a line break.
  • On Linux/KVM and Windows/WHP: the sandbox-ram-upper scenario with the one-shot outcome report.
  • On Windows: ruff, pyright, and the scenario's unit tests.

Before the rebase:

  • Linux/KVM (nvx-kvm-test, Azure Linux 3 VM):
    • test-microvm:
      • the public managed-exec-config, sandbox-lifecycle, sandbox-proxy-only, and sandbox-ram-upper;
      • the layered-root sandbox-blocks, scratch-snapshot, snapshot-tiers, and platform-filesystem, -zero-fill, -network-policy, and -restore-processors.
    • The workflow's six Linux Ubuntu sandbox steps, from layer smoke to share access policy, with the format-8 checks.
    • The runtime adapter's VM scenarios: 43 layered, 3 restart, 3 step, and 9 template.
    • verify, shellcheck, and shfmt.
    • The full validate-nvx unit suite: 1,373 tests, 27 skipped.
    • nvx-rs test and test --lib --no-default-features, and doc -D warnings.
    • test --all-features passes except three guest tests in modules this PR does not touch, which fail the same way in the guest-binary tests. guest::entry gets EOPNOTSUPP from a mount call, and two guest::exec tests get EINVAL where they expect ESRCH. The host runs kernel 6.6, and these tests assume newer kernel behavior.
  • Linux/MSHV (nvx-mshv-test): the four public sandbox scenarios, sandbox-ram-upper included.
  • Windows/WHP (local): the four public sandbox scenarios, with the OpenVMM b92daa350 of release v0.1.0-dev.3beaf1646086 and the guest artifacts above.
  • Windows (local):
    • ruff, ruff format, and pyright for Linux and Windows.
    • The Markdown links.
    • The validate-nvx suite, apart from 18 test_setup_azure.py shell cases that fail under Git's sh with Windows paths. This PR leaves those files alone, and they pass on Linux.
    • nvx-rs: fmt, the three Clippy runs, Clippy for x86_64-unknown-linux-musl (all features, --no-default-features, and the guest binaries' --features guest), the three test runs, doc -D warnings, and the 1.89 MSRV check.
    • The runtime adapter: protocol and host configurations, guest Clippy, and the host MSRV check.

Each sandbox ran with these options:

nvx-rs sandbox --hypervisor kvm --layer distro,build/ubuntu-distro.erofs,UUID --upper ram \
    --entrypoint /sbin/nvx-sandbox-smoke --arg=limits --arg=33554432 --arg=8 \
    --memory-mib 256 --memory-max 33554432 --pids-max 8
>> openvmm --machine microvm --microvm-sandbox-block distro:file:/proc/self/fd/4,ro … \
    --cmdline 'nvx_sandbox=1 nvx_layer=distro,0xd0003000,erofs:UUID nvx_overlay_upper=ramfs …'
NVX-SANDBOX-READY: pid=89 layers=1
NVX-UBUNTU-SANDBOX-OK uid=65534 gid=65534
NVX-SANDBOX-EXIT: status=0

Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The smoke probe reads only the first line of /proc/cmdline, missing valid newline-separated root tokens and incorrectly failing RAM-upper launches.

1 open issue
What changed in this PR

This PR enables RAM-backed writable uppers for NVX layered sandboxes, addressing #558 without changing the OpenVMM pin. It retains scratch-backed storage as the default and aligns the CLI, guest behavior, tests, and documentation.

Changes:

  • Adds --upper scratch|ram, optional scratch handling, and format-8 configuration records while preserving reads of formats 1 and 7.
  • Enables tmpfs upper assembly and teardown, with smoke checks for the declared upper and overlay options.
  • Adds cross-backend acceptance coverage for RAM-upper lifecycle, memory accounting, and data loss after restart.
File Description
scripts/​test_nvx_tools.py Tests guest parsing, mounts, teardown, and CI registration.
scripts/​test_microvm_tests.py Tests scenario dispatch and regression detection.
scripts/​nvx_tools/​sandbox_ram_upper_tests.py Adds public RAM-upper acceptance checks.
scripts/​nvx_tools/​microvm_tests.py Registers and dispatches the explicit scenario.
scripts/​nvx_tools/​ci.py Includes the new scenario module in CI change detection.
nvx-rs/​tests/​fixtures/​root-tokens.json Updates shared vectors for shell RAM-upper support.
nvx-rs/​tests/​cli/​validation.rs Covers invalid upper-option combinations.
nvx-rs/​tests/​cli/​usage.rs Checks upper-option help and errors.
nvx-rs/​tests/​cli/​openvmm.rs Adds RAM-only launch assertion helpers.
nvx-rs/​tests/​cli/​one_shot.rs Covers one-shot launches without scratch.
nvx-rs/​tests/​cli/​lifecycle.rs Tests RAM lifecycle and format-8 validation.
nvx-rs/​tests/​cli/​formats.rs Clarifies legacy-format test coverage.
nvx-rs/​tests/​cli/​dry_run.rs Covers explicit scratch and RAM dry runs.
nvx-rs/​tests/​checked_handoff.rs Adapts handoff tests to optional scratch paths.
nvx-rs/​src/​testing/​fake_openvmm.rs Admits the distro-only RAM layout.
nvx-rs/​src/​protocol/​root.rs Updates shell-parser documentation.
nvx-rs/​src/​host/​sandbox/​tests.rs Updates tests to use typed uppers.
nvx-rs/​src/​host/​sandbox/​specification.rs Adapts contract tests to typed uppers.
nvx-rs/​src/​host/​sandbox/​prepared/​tests.rs Updates prepared-launch test construction.
nvx-rs/​src/​host/​sandbox/​prepared.rs Documents optional scratch binding in the example.
nvx-rs/​src/​host/​sandbox/​plan/​tests.rs Verifies RAM plans omit scratch blocks and tokens.
nvx-rs/​src/​host/​sandbox/​options/​tests.rs Tests upper selection and operation restrictions.
nvx-rs/​src/​host/​sandbox/​options.rs Validates upper and scratch combinations.
nvx-rs/​src/​host/​sandbox/​mod.rs Makes launch construction accept SandboxUpper.
nvx-rs/​src/​host/​cli/​sandbox/​tests.rs Tests upper parsing and updates launch fixtures.
nvx-rs/​src/​host/​cli/​sandbox/​state.rs Writes format 8 and reads upper-aware configurations.
nvx-rs/​src/​host/​cli/​sandbox/​options.rs Adds CLI parsing and help for --upper.
nvx-rs/​src/​host/​cli/​sandbox/​mod.rs Plans and prepares launches with either upper.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start.rs Skips scratch checking and claiming for RAM.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​mod.rs Provisions optional scratch records.
nvx-rs/​README.md Documents typed uppers and format-8 records.
guest/​common/​nvx-sandbox-smoke Checks the declared upper and overlay options.
guest/​common/​nvx-init-agent Validates, mounts, and tears down tmpfs uppers.
doc/​usage.md Documents upper options and scenario selection.
doc/​run.md Explains upper behavior, limits, and acceptance checks.
doc/​project-structure.md Lists the scenario and updated guest behavior.
doc/​design/​5.2-readiness-and-runtime-control.md Clarifies persistence across sandbox restarts.
doc/​design/​5.1-boot-protocol.md Explains the RAM token’s tmpfs implementation.
doc/​design/​4.3-isolation-and-resource-model.md Documents workload memory charges for upper files.
doc/​design/​4.1-guest-filesystem-and-image-model.md Describes both uppers, assembly, and resource trade-offs.
doc/​design/​4-guest-runtime-architecture.md Updates the filesystem chapter description.
doc/​design/​3.4-block-storage.md Describes tmpfs in the RAM-backed layout.
doc/​design/​2.5-device-topology-transport-and-discovery.md Marks scratch absent in RAM layouts.
doc/​design/​2.1-machine-model-and-configuration.md Aligns storage-layout documentation with tmpfs support.
doc/​design/​1.1-goals.md Includes both writable-upper choices.
doc/​contribute.md Lists the new public acceptance module.
doc/​ci.md Documents RAM-upper backend coverage.
.github/​workflows/​run-nvx-microvm-tests.yml Runs the scenario and expects format-8 records.

Review effort: Balanced

Comment thread guest/common/nvx-sandbox-smoke Outdated
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from 7d15e41 to e17f060 Compare October 28, 2026 22:50
Copilot AI balanced review requested due to automatic review settings October 28, 2026 22:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The one-shot launch lacks a fixture-specific OpenVMM argument, so lifecycle tracking and cleanup can miss the VM and leave its child process consuming CI resources after a timeout.

2 open issues

Review effort: Balanced

Comment thread scripts/nvx_tools/sandbox_ram_upper_tests.py
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from e17f060 to 3e246d8 Compare October 28, 2026 23:03
Copilot AI balanced review requested due to automatic review settings October 28, 2026 23:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The one-shot launch lacks a fixture-local argument, so a timeout can leave an OpenVMM child that cleanup cannot identify or terminate.

1 open issue
1 resolved since last review

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from 3e246d8 to 435eaa9 Compare October 28, 2026 23:09
Copilot AI balanced review requested due to automatic review settings October 28, 2026 23:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The teardown check accepts incomplete reports, allowing one-shot acceptance to succeed without evidence that all required resources were released.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread scripts/nvx_tools/sandbox_ram_upper_tests.py Outdated
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from 435eaa9 to 09215e2 Compare October 28, 2026 23:17
Copilot AI balanced review requested due to automatic review settings October 28, 2026 23:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the RAM-backed writable-upper changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from 09215e2 to 9ff6d10 Compare October 29, 2026 00:02
Copilot AI balanced review requested due to automatic review settings October 29, 2026 00:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the RAM-backed overlay changes.

No open issues 🎉

Review effort: Balanced

nvx::protocol::root defines two uppers for a layered root: the ext4
scratch disk, which nvx_scratch=0xd0006000,ext4 names, and RAM, which
nvx_overlay_upper=ramfs names, with no scratch block device. OpenVMM
admits the RAM layout with exactly one read-only distro block, and the
guest library assembles it, but NVX's shell init, nvx-init-agent,
refused the token, so no NVX sandbox could keep its upper in RAM (#558).

The init now reads either upper with the parser's rules: each token
once and with its one value, and exactly one upper. It refuses both
tokens together, an image without an upper, and any other value, such
as nvx_overlay_upper=tmpfs, with the parser's messages. The token keeps
its name: it names a RAM-backed layout, and the guest mounts a tmpfs
for it. For a RAM upper, the init mounts a tmpfs at /run/nvx/tmpfs,
where nvx::guest::root mounts one too: nosuid, nodev, with mode 0755,
the mode of an ext4 scratch's root, rather than tmpfs's world-writable
default, and at the kernel's default size, half of the guest's RAM. The
overlay keeps its options, metacopy=on and xino=on; the pinned kernel
enables the tmpfs xattrs that metacopy needs. The teardown unmounts the
upper that the init mounted, last, as it unmounted scratch.

The parser also refuses a RAM upper beside a scratch block device,
which it finds in the virtio_mmio.device= parameters as the kernel
reads them, with their quoting and number formats. The init asks the
kernel instead: before it mounts anything, it refuses a RAM upper if
/proc/iomem lists a virtio-mmio device at 0xd0006000, where
resolve_block_device finds the scratch disk. A scratch upper still
fails when its device does not appear.

The shared vectors of nvx-rs/tests/fixtures/root-tokens.json no longer
expect the shell to refuse a RAM upper. test_nvx_tools.py now requires
it to accept every EROFS layout over either upper that the parser
accepts, and to read the tokens of the cases that the parser refuses
for a scratch device, whose /proc/iomem check a new test covers. Other
tests check the init's messages for conflicting uppers, the mounts of
each upper, that the RAM check precedes every mount, and the teardown's
order and failure message for the tmpfs upper. The protocol's
documentation and doc/project-structure.md describe what the init
assembles.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nvx-sandbox-smoke, the probe that the sandbox scenarios run as the
workload, failed unless the root overlay's upper directory was
/run/nvx/scratch/upper. A sandbox whose upper is in RAM, which the init
agent now assembles, keeps it at /run/nvx/tmpfs/upper instead, so the
probe refused every such sandbox, and accepted a scratch upper whatever
the host had asked for.

The probe now reads the upper that the kernel command line declares, as
it already reads nvx_hostname there: a tmpfs for nvx_overlay_upper=ramfs,
and scratch otherwise. It fails unless the overlay's upper directory is
on that upper, so a sandbox that declares one upper and mounts the other
fails either way. It also fails unless the overlay keeps metacopy=on and
xino=on, the options that the init agent sets: overlayfs drops metacopy
without a word on an upper that refuses its xattrs, so the probe shows
that the pinned kernel applies both options on either upper. The image
check calls its write a write to the upper rather than a scratch-backed
one.

Both readers of the command line now read all of /proc/cmdline, as the
init agent does, rather than its first line: a caller's --cmdline keeps
its newlines, and the root's tokens and nvx_hostname follow it, so a
sandbox launched with a multi-line --cmdline failed the hostname check
and, with a RAM upper, the mount check.

test_nvx_tools.py gives the mount check a command line, and covers a
tmpfs upper that a RAM sandbox declares, the mismatch in both
directions, tokens that merely resemble the RAM upper's, an overlay
without either option, and RAM and hostname tokens that follow a line
break.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nvx-rs sandbox always planned its distro layer over the ext4 image of
--scratch, so no NVX launcher could request the RAM upper that the guest
now assembles, although the typed plan, the prepared launch, and
OpenVMM already admit it (#558).

run and provision now take --upper scratch, the default, or --upper ram.
A RAM upper attaches the distro layer alone, renders
nvx_overlay_upper=ramfs in place of nvx_scratch=0xd0006000,ext4, and
needs, binds, and claims no scratch image. --upper ram refuses --scratch
and still requires --layer. The option names the RAM upper `ram`, as
RootUpper and the plan's serde form do, whatever its token or the
guest's mount call it. start, exec, stop, and deprovision refuse it: a
provisioned sandbox keeps the upper that it recorded.
SandboxLaunch::new now takes a SandboxUpper, as SandboxPlan::new does,
and SandboxLaunch::scratch returns None for an upper in RAM.

provision now records config.json in format 8: format 7 with the upper
under `upper`, `scratch` or `ram`, and, for a RAM upper, `null` under
`scratch` and `scratch_binding`. start still reads formats 7 and 1,
whose upper is on scratch. It refuses as malformed a format 8
configuration without `upper` or with another value, a scratch upper
without its image, a RAM upper with an image or a binding, and an
earlier format that names an upper. Each start of a RAM sandbox begins
with an empty upper.

The fake OpenVMM admits a RAM-backed overlay as OpenVMM does: one
nvx_overlay_upper=ramfs and the distro layer alone. The CLI tests cover
a dry run, a one-shot run, and a managed lifecycle with a RAM upper and
no scratch image in the root, and the configuration that provision
records. They also cover the refusals of --upper ram beside --scratch
or without --layer, of --upper outside run and provision, of another
value, and of the malformed configurations above. doc/usage.md,
doc/run.md, and the nvx-rs README describe the option, the record, and
the trade-offs of each upper. The Ubuntu sandbox steps of
run-nvx-microvm-tests.yml that check the configuration that provision
records for live shares now require format 8, on Linux and Windows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The init agent can now assemble the layered root over a tmpfs upper, and
nvx-rs sandbox --upper ram plans that launch, but no scenario booted
such a sandbox: the unit tests stop at the shell init's commands and the
CLI's fake OpenVMM, and the public sandbox scenarios all attach a scratch
image. Nothing showed that the pinned OpenVMM admits the launch, that
the kernel mounts the overlay with its options over a tmpfs, or that a
RAM upper holds a workload's writes, forgets them at the next start, and
charges them to the workload.

The new sandbox-ram-upper scenario of test-microvm drives the public
nvx-rs sandbox commands as subprocesses, as the other public sandbox
scenarios do, with the guest-artifact build's Ubuntu layer and no
scratch image:

- provision refuses --upper ram beside --scratch, and start refuses
  --upper, both before they create any state;
- a one-shot run of /sbin/nvx-sandbox-smoke prints an OpenVMM command
  with one sandbox block device, the read-only distro layer, and the RAM
  upper's token in place of the scratch token, and the probe passes its
  profile and resource-limit checks, which now require the overlay's
  upper on a tmpfs. Its outcome report reports success and a complete
  teardown. The report lies under the scenario's temporary directory, so
  its path names the run's OpenVMM to the process checks and cleanup,
  which find VMs by that directory: no other argument of a run without
  a scratch image does, so a hung run would otherwise outlive cleanup;
- a managed sandbox records format 8 with the RAM upper and no scratch
  image or binding, passes the probe's managed and resource-limit
  checks, shows its guest one virtio block device and the RAM upper's
  token, and reads back a file that an earlier request wrote;
- under a 32 MiB memory limit, a 20 MiB file in the upper leaves no room
  for a 20 MiB allocation, which the memory controller kills, and the
  same allocation succeeds once the file is gone, since the guest has no
  swap to which the upper's pages could go;
- after a stop and a new start, the file is gone, and deprovision
  removes the state.

Like managed-exec-config, sandbox-lifecycle, and sandbox-proxy-only, it
needs the Ubuntu layer that packages omit, so it runs only when named,
and only with Alpine control. The public managed sandbox acceptance of
run-nvx-microvm-tests.yml names it on Linux and Windows, so every
backend runs it. A change to the scenario's module, like one to the
other test drivers, leaves the performance inputs alone.

test_microvm_tests.py runs the scenario against a fake nvx-rs that
emulates a RAM upper's planner and guest, and shows that it fails for
each regression that it must detect: a scratch image accepted beside a
RAM upper or recorded for it, an upper accepted by start, a scratch
block device or token, a failed probe, a run whose OpenVMM command does
not name the fixture or that reports a failure or an incomplete
teardown, an uncharged upper, and a file that survives a restart. It
also covers the dispatch of the scenario, and that it runs only when
named. The documentation lists the scenario with the other public
sandbox scenarios, and Run describes what it accepts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The architecture guide still described the layered sandbox root as an
image layer over private ext4 scratch: its goals, Table 4.1, Figure 4.1,
and the assembly steps named only scratch, and 4.1 said that the guest
agents always keep the overlay's upper on scratch and refuse a RAM
upper, so that the machine's RAM-backed overlay layout served only other
guests. The guest agents now assemble the root over either upper, and
nvx-rs sandbox --upper ram launches it.

- 1.1 makes the workload filesystem an image layer, a private writable
  upper on scratch or in guest memory, and live host shares, and the
  chapter 4 contents call 4.1's subject the writable upper.
- 4.1 lists tmpfs with extended attributes among the kernel facilities,
  which overlayfs needs for metadata-only copy-up on a tmpfs upper. Its
  table, its figure, and its token list describe both uppers, and its
  steps refuse a scratch device beside a RAM upper while they resolve
  the roles, mount scratch or a tmpfs at the kernel's default size, half
  of the guest's RAM, and unmount whichever upper they mounted. It now
  refuses only a GPT image, and it compares the two uppers' resources:
  scratch spends host storage and keeps a managed sandbox's changes from
  one start to the next, while a RAM upper spends guest memory that the
  writer's memory cgroup is charged for, cannot be reclaimed without
  swap, fails with ENOSPC beyond the tmpfs size, and dies with the
  virtual machine. A RAM upper uses the RAM-backed overlay layout, which
  has one read-only distro block, and supports neither capture nor
  restore.
- 4.3 notes that the workload's memory limit also bounds the files that
  it writes into a RAM upper.
- 5.2 notes that a managed sandbox's writes persist across a stop and
  start on scratch, but not in a RAM upper, which each start begins
  empty.
- 2.1, 3.4, and 5.1, which described the RAM-backed overlay layout only
  as what the machine admits, now say that the guest agents keep its
  upper on a tmpfs, and Table 2.7 no longer presents the scratch block
  as always present: it is absent in that layout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-558-ram-overlay-upper branch from 9ff6d10 to cdc1246 Compare October 29, 2026 01:09
Copilot AI balanced review requested due to automatic review settings October 29, 2026 01:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the RAM-backed writable-upper changes.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overlay upper: support virtio-blk scratch or tmpfs end to end, and align docs

2 participants