Skip to content

openvmm: verify pinned CPU profiles at partition creation - #1016

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-567-validate-pinned-cpu-profiles
Oct 29, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-567-validate-pinned-cpu-profiles

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #567. Pins microsoft/openvmm#4807 at c32c57d129a36fc8c82308d6659f8d7ab5955474, which verifies each pinned CPU profile before a partition uses it, as step 1 of section 2.2.1.3 of the architecture guide requires at partition creation, for cold boot and restore: once per process, the profile that OpenVMM builds from its built-in data must be valid, with its density, ownership masks, and CPU time bits, and its built-in encoding, which snapshots embed, must be its canonical encoding. Profile validation also gains the CPU time bits of section 3.2.3, so the host profile document that a snapshot carries must fix them too.

Based on dev at 67f395e54. microsoft/openvmm#4807 is two commits directly on bd2b27a2e, the head of feature/microvm/dev and the revision that dev pins since #974 merged, and the pin is that pull request's head commit itself, so this branch's two commits are its whole diff.

Change

Commit Files Change
7f15ed6f1 openvmm openvmm bd2b27a2e → c32c57d12: the two commits of microsoft/openvmm#4807
5be4850e8 doc doc/openvmm-upstream-roadmap.md Records them as positions 360 and 361, which build on the fork's cpu_profile crate; moves the NVX pin; and records that microsoft/openvmm#4794 merged positions 355-359 into feature/microvm/dev

No other NVX change is needed: OpenVMM's control-session protocol vectors and built-in CPU profiles are unchanged at the pin, and section 2.2.1.3 already says what OpenVMM now verifies.

OpenVMM changes

  • CPU time bits (c9ac93a63). CpuProfile::validate, which every way to build or decode a profile runs, checks every bit of Table 3.8: the hypervisor, TSC, and RDTSCP bits set; ARAT alone in leaf 0x6 and the invariant TSC alone in leaf 0x80000007; no TSC-deadline timer, PDCM, or IA32_TSC_ADJUST; and leaves 0xa, 0x15, and 0x16 zero. It names every violating bit by leaf, subleaf, register, and bit. Before, only WHP's preflight checked the time bits, on the CPUID that the backend presents, so on KVM and MSHV a snapshot's host profile document did not have to fix them; now decoding such a document, and so its restore, fails with E_PROFILE_DIGEST. Every pinned and derived profile complies.
  • Pinned profile verification (c32c57d12). The catalog built a pinned profile from pinned_data.rs on first use with "no parsing, no validation, and no hashing, which the tests do for every pinned profile instead". PinnedProfile::verify now checks that the built profile is valid, time bits included, and that the built-in encoding is its canonical encoding, and the catalog runs it once per process, on the profile's first use, and hands out only verified profiles: select (cold boot, and the VM worker's restore), pinned_for_restore (the restore preflight), pinned, and PartitionProfile::pinned fail with E_PROFILE_DIGEST for a defective profile, naming it and what is wrong, before the partition exists. select_auto verifies only the revision that it selects.

Design choice and cost

The issue offered to verify each pinned profile when it is first loaded, measuring the cold-boot cost, or to state that the build and tests verify pinned profiles. This verifies at first load exactly what step 1 lists: validity and the canonical encoding. The digest, which step 4 compares on restore, stays a built-in constant that OpenVMM's tests check against the golden digests, as they check the static data against the reviewed JSON files, so a pinned profile's cold boot and restore still parse no JSON and hash nothing. Hashing the 9 KB encoding at every start would have nearly doubled the cost on CPUs without the SHA extensions, such as the Skylake-SP of the benchmark hosts.

Verification adds to the first profile use of each OpenVMM process, which NVX's --single-process OpenVMM makes once per cold boot or restore. Medians of the first select or pinned_for_restore in fresh processes of a release build, for the Skylake-SP, Ice Lake-SP, and Genoa profiles; the "warmed" runs busy-wait 100 ms first, so that the clock is up, as after the work that precedes selection in a starting OpenVMM:

Host CPU Before After Warmed, before Warmed, after
Windows laptop Core i9-12900H 8-19 us 50-69 us 10-27 us 54-86 us
nvx-whp-test, Azure VM, Windows Xeon Platinum 8370C 14-31 us 68-84 us 18-37 us 75-91 us
prometheus32, bare metal, Linux Xeon Silver 4114 (Skylake-SP) 8-27 us 158-170 us 2-8 us 45-49 us
prometheus28, bare metal, Windows Xeon Silver 4114 (Skylake-SP) 24-66 us 129-224 us 17-69 us 138-229 us

Validation and the encoding take about a third and two thirds of it. NVX's restores take 36-200 ms and its cold starts 275-720 ms in the current baselines, so the added cost is a fraction of a percent of either.

Validation

  • NVX's default microVM suite with this pin, run by test-microvm after build-openvmm, with the guest artifacts of v0.1.0-dev.299d0b4d9851. Every scenario cold boots with --cpu-profile auto, which selects the host's pinned profile, many capture and restore, and cpu-profile-fallback captures and restores a host profile. Each run executed 47 scenarios, 41 distinct ones, and all passed, as did the teardown:

    • Linux/KVM on prometheus32, bare metal with a Xeon Silver 4114 (intel.skylake-sp.v1), at this pull request's head 5be4850e8 and so at the pin itself.
    • Linux/KVM on nvx-kvm-test, an Azure VM with a Xeon Platinum 8573C (intel.emeraldrapids.v1).
    • Linux/MSHV on prometheus30, bare metal with a Xeon Silver 4114 (intel.skylake-sp.v1).
    • Windows/WHP on nvx-whp-test, an Azure VM with a Xeon Platinum 8370C (intel.icelake-sp.v1).

    The last three ran OpenVMM 0565b55bc, which differs from the pin only in its commit message: both have tree 96769697845a.

  • Defective built-in data, on nvx-kvm-test with OpenVMM 0565b55bc. Each defect was built into this pin and into the previous one. A good build captured a snapshot, which the defective build then restored, and the defective build cold booted:

    • Setting the TSC-deadline bit, CPUID.1:ECX[24], in the values of intel.emeraldrapids.v1 but not in its encoding, as a hand edit that forgets the encoding would. With this pin, the restore and the cold boot each fail within 0.01 s with [E_PROFILE_DIGEST] pinned CPU profile intel.emeraldrapids.v1 is invalid: it violates the CPU time bits: CPUID 0x1 ECX bit 24 is set. The previous pin only fails later, at the backend support check, with E_PROFILE_UNSUPPORTED, and suggests --cpu-profile host.
    • Clearing ERMS, CPUID.(7,0):EBX[9], in the values but not in the encoding, as a stale pinned_data.rs would. The values remain a valid profile that every backend supports. With this pin, the restore and the cold boot each fail with [E_PROFILE_DIGEST] pinned CPU profile intel.emeraldrapids.v1 differs from its constant canonical encoding. The previous pin passes guest-boot and snapshot-core: its guest has no ERMS, but its snapshots record a profile that has it.
  • OpenVMM at the pinned tree, on Windows with Rust 1.95.0: cargo clippy --all-targets -- -D warnings for cpu_profile, virt, virt_whp, openvmm_helpers, openvmm_core, hypervisor_resources, openvmm_hypervisors, openvmm_entry, and petri; cargo doc --no-deps -p cpu_profile; cargo nextest run --profile agent for cpu_profile (119 passed) and, on nvx-whp-test, for cpu_profile, virt, virt_whp, openvmm_helpers, and openvmm_core (460 passed, 18 skipped); and cargo xtask fmt --fix (clean). On Linux: the same clippy set plus virt_kvm and virt_mshv (clean), and nextest for cpu_profile, virt, virt_kvm, virt_mshv, openvmm_helpers, and openvmm_core (541 passed, 26 skipped).

  • NVX at 5be4850e8: python scripts\nvx.py verify (consistent); the openvmm_vectors and openvmm_cpu_profiles comparisons with --ignored (3 and 1 passed), against the pinned revision; python scripts\check_markdown_links.py; and git diff --check.

  • Every commit is signed and verified, both these and [DO NOT REVIEW] cpu_profile: verify pinned CPU profiles at partition creation openvmm#4807's.

  • CI: this pull request's run at 5be4850e8 passed. [DO NOT REVIEW] cpu_profile: verify pinned CPU profiles at partition creation openvmm#4807's checks at c32c57d12 all passed, its VMM tests on Linux (KVM, MSHV, and SNP on MSHV) and Windows (Intel, AMD, SNP, TDX, and Arm64 hosts) included.
    Merge [DO NOT REVIEW] cpu_profile: verify pinned CPU profiles at partition creation openvmm#4807 into feature/microvm/dev, by fast-forwarding that branch to c32c57d12, before merging this pull request.

Pin microsoft/openvmm#4807 at c32c57d129a36fc8c82308d6659f8d7ab5955474,
two commits on bd2b27a2e, the previous pin, which is the head of
feature/microvm/dev since microsoft/openvmm#4794 merged.

At partition creation, for cold boot and restore, step 1 of the CPU
profile verification in section 2.2.1.3 of the architecture guide
requires the VMM to verify that the CPU profile is valid, including its
canonical encoding, density, ownership masks, and CPU time bits.
OpenVMM did so for host profiles, but built each pinned profile from the
constants of pinned_data.rs on first use without validating or encoding
it, so a stale or malformed pinned_data.rs reached partition creation
unchecked; only the catalog's tests would have caught it. Its profile
validation did not check the CPU time bits either, so the host profile
document that a snapshot carries did not have to fix them on KVM and
MSHV, whose preflights, unlike WHP's, do not check the time bits that
the backend presents.

The first commit adds the CPU time bits of section 3.2.3 to profile
validation, naming every bit that violates them, so that decoding a
host profile document that violates them, and so restoring its
snapshot, fails with E_PROFILE_DIGEST. The second verifies each pinned
profile once per process, on its first use: its built-in values must
form a valid profile, CPU time bits included, whose canonical encoding
is the built-in encoding that snapshots embed. A defective profile
fails the cold boot or restore that selects it with E_PROFILE_DIGEST,
before the partition exists.

The verification encodes the profile but does not hash it: its digest
stays a constant that OpenVMM's tests check against the golden digests.
It adds about 40 to 55 us to the first profile use of each OpenVMM
process, which NVX's single-process OpenVMM makes once per cold boot or
restore, on current hosts, and 40 to 180 us on Skylake-SP hosts.

The protocol vectors and built-in CPU profiles that nvx-rs copies are
unchanged.

Refs #567

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the two commits that the previous commit pins as positions 360
and 361 of the microVM chain, and move the NVX pin to c32c57d12.

Positions 360 and 361 verify CPU profiles as the architecture guide's
partition-creation checks require: profile validation checks the CPU
time bits, so a snapshot's host profile document must fix them too,
and the catalog verifies each pinned profile once per process, before a
partition uses it, while the tests keep checking the golden digests.
They change only cpu_profile, the fork's CPU profile crate that
positions 159-269 add, a test helper of virt_mshv, and the snapshot
guide, so they build on that fork-only code; say so in the table's
remarks. The pin is now 204 commits past the mapped tip.

Positions 355-359 are no longer only a proposal:
microsoft/openvmm#4794 merged them, so position 359, bd2b27a2e, is the
head of feature/microvm/dev. Say so in the scope, and name
microsoft/openvmm#4807 as the proposal of positions 360 and 361.

Refs #567

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 18:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

CPU-profile validation changes span cold boot and restore across three backends, with exact-head CI, WHP and corrupted-profile validation, and prerequisite upstream landing still pending.

No open issues 🎉

What changed in this PR

This PR promotes an OpenVMM revision that validates pinned CPU profiles before partition creation for cold boot and restore, addressing #567. The roadmap documents the new pin and its upstream dependencies.

Changes:

  • Updates the documented NVX pin to c32c57d12 at position 361.
  • Records CPU time-bit validation and once-per-process pinned-profile verification at positions 360–361.
  • Updates chain counts and records the merge of OpenVMM #4794.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin, commit-chain diagram, and roadmap entries for CPU-profile validation.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

microVM: validate pinned CPU profiles at partition creation

2 participants