[Pedro Henrique Penna (ppenna)]

Names every difference in the E_CPU_SURFACE errors of the microVM's CPU checks, and reports the failures of every CPU check of a partition together, for microsoft/nvx#568. The VM worker counted every CPUID difference but named at most 16 and dropped the rest, both when a restore compares the effective CPUID that it recomputes with the snapshot's record and when the worker compares VP 0's CPUID with the effective CPUID before any VP runs. It also stopped at the first of its CPU checks that failed. It now names each difference, each recorded entry on one line with the registers in which it differs, and reports the non-zero CPUID entries outside the profile's tables that VP 0 reads together with the results that VP 0 does not observe. It runs every check even after one fails, and reports their failures in one error that leads with the first.

Based on feature/microvm/dev. This branch is four commits directly on c32c57d12, the head of feature/microvm/dev since #4807 merged, which microsoft/nvx's dev pins. It was rebased there from bd2b27a2e when #4807 merged, and git range-diff shows its four commits unchanged. microsoft/nvx#1014 pins this branch's head.

Why

Section 2.2.1.3 of the NVX architecture guide states that CPU-profile verification reports every violation together, naming the leaf, subleaf, register, and bit. check_recorded_cpuid and check_presented applied MAX_REPORTED_CPUID_DIFFERENCES (16), so a build or host that differed in more entries showed the first 16, and a fix of those could fail again on entries that the error had not named. The restore's comparison also printed each differing entry's four results and masks on each side with {:#x?}, which pretty-prints every array over six lines: 24 lines per entry, without saying which register differs, which listing every entry would have multiplied. And the worker ran its three checks in turn, each with ?, so a restore whose record differed did not say whether VP 0 also read other values or the backend also lacked a feature of the profile, which surfaced only once the first failure was fixed.

Registers of a recorded entry (69ddc715b)

For an entry that both the recomputed effective CPUID and the snapshot's record hold, recorded_cpuid_differences names each register in which they differ, with its value and mask on each side, on one line, as cpuid_differences names the registers of the results that VP 0 does not observe:

CPUID 0x0 EAX is 0x0000001b under mask 0xffffffff, and 0x0000001a under mask 0xffffffff in the snapshot

An entry that only one side has is named as before.

Every difference (c55db5d9e)

Both errors name every difference and keep counting them. The lists stay bounded: the effective CPUID of each pinned profile has between 64 entries (Skylake-SP) and 94 (Turin), its table and the 22 topology and identity entries that OpenVMM adds, so VP 0's comparison names at most that many results, and a snapshot's record holds at most MAX_EFFECTIVE_CPUID_ENTRIES (1024), which the manifest decoder enforces, so the restore's comparison names at most that many more entries that only the snapshot has.

Entries outside the profile with VP 0's differences (e1fa0f028)

check_presented checked VP 0's view at the host's entries outside the profile's tables (E_CPU_UNLISTED) only once VP 0 observed every result, so a report that failed both named the results but none of those entries. It now names them together, as cpu_profile::verify_support names the entries outside the profile after what a backend lacks: an E_CPU_SURFACE failure names every result that VP 0 does not observe, then every non-zero entry outside the profile's tables, and a report whose only violations are such entries still fails with E_CPU_UNLISTED. No failure changes its code.

Every check's failures together (e4c741e18)

check_cpu_profile runs the worker's three CPU checks in order, each even after an earlier one fails: on restore, the comparison of the recomputed effective CPUID with the snapshot's record; then the check of VP 0's CPUID; then the check that the backend supports the profile. report_together returns a failure that occurs alone unchanged, so that its type and code stay what they were, and combines several into one error that gives each failure's chain in turn, joined with ; , as a restore on Linux/KVM shows it:

fatal error: [E_CPU_SURFACE] failed to launch vm worker

Caused by:
    0: failed to launch worker
    1: [E_CPU_SURFACE] the effective CPUID of CPU profile intel.host.v1 differs from the snapshot's in 1 entries: CPUID 0x80000001 ECX is 0x00000161 under mask 0xffffffff, and 0x00000121 under mask 0xffffffff in the snapshot; [E_PROFILE_UNSUPPORTED] the backend does not support CPU profile intel.host.v1: CPUID 0x80000001 ECX bit 6 is not supported

The error leads with the first failure, and so with the code that the worker reported before: the exit line, the hints of --cpu-profile auto (hint_host_profile), and petri's host-profile fallback all take the first bracketed code of the chain. A partition without a time ABI backend fails both of the backend's checks with that failure, after any failure of the snapshot's record. The time ABI CPU checks passed log line, with the checks' timings, moves into check_cpu_profile, so its target is now openvmm_core::worker::dispatch::time_abi. The Guide's snapshot page says that the checks report together.

The error stays bounded. For Turin, the largest pinned profile, an error in which every register of all 94 entries differs in both comparisons and the backend supports nothing takes 61,717 bytes, on one line: 37,223 for the record, 22,815 for VP 0's view, and 1,675 for the support check. Skylake-SP's takes 41,854. These figures come from a scratch test over every pinned profile, which is not part of this branch.

The other checks

The checks themselves name every violation: cpu_profile::verify_support every feature, limit, XSAVE layout, address width, and MSR value that a backend lacks (E_PROFILE_UNSUPPORTED), and, for a fingerprint's probe partition, every non-zero entry outside the profile; hv_banks::profile_features every feature that MSHV's and WHP's processor feature banks lack; and cpu_profile::check_unlisted_cpuid every non-zero entry outside the profile (E_CPU_UNLISTED). The generation check, and a restore's checks of the profile's ID, digest, and document, each have a single violation, and run before the partition exists, which the other checks need. MSHV and WHP refuse a profile feature that their processor feature banks lack when they create the partition, also before these checks run. The backends' time ABI preflight, which checks the identity leaves, the CPU time bits, and, on WHP, that the processor feature banks hide the timer features, enforces the backend obligations of section 3.2.6 of the guide rather than these steps, and still fails at its first violation.

Tests

• recorded_cpuid_must_match: a flipped bit is named with its register, value, and mask, and the entries of another topology are named on one line.
• recorded_cpuid_names_every_difference and presented_cpuid_names_every_difference: every entry of a 2-VP Ice Lake-SP partition's record, and every one of its 69 results, differs, and the errors count and name each one, in order. Both fail with the limit restored.
• presented_cpuid_rejects_non_zero_unlisted_entries: a report that drops two governed results and reads a non-zero entry past Ice Lake-SP's maximum basic leaf fails with E_CPU_SURFACE, naming both results and then the entry. It fails if the entry is left out.
• cpu_checks_report_every_failure_together: with a fake time ABI backend, a 2-VP Ice Lake-SP partition whose snapshot record differs in one entry, whose VP 0 does not observe the brand string's first leaf, and whose backend lacks AVX2, alone and in every combination, and a partition without a time ABI backend. It checks the combined errors and their leading code, and that a failure alone is the check's own error, down to the ProfileError of the support check. It fails if the checks stop at the first failure.

Validation

• On this branch, on Windows: for openvmm_core, cargo clippy --all-targets -- -D warnings, cargo doc --no-deps, and cargo nextest run --profile agent at e4c741e18 (121 tests passed), with cpu_profile's 119, and cargo xtask fmt, all clean. Before the rebase, the first three commits passed 118, 120, and 120 tests on bd2b27a2e.

• End to end on NVX's Azure test VMs, with openvmm: name every CPUID difference in CPU-surface errors nvx#1014's head, which pins this branch, and the guest artifacts of NVX release v0.1.0-dev.299d0b4d9851, whose OpenVMM, bd2b27a2e, served as the baseline:

  • Several failures. On Linux/KVM (Xeon Platinum 8573C), a snapshot that this branch captured on the host profile intel.host.v1 (--cpu-profile host), whose profile document in the manifest then had SSE4A (CPUID 0x80000001 ECX bit 6), which Intel CPUs lack, set and its digest updated, fails to restore on each binary with the same exit line, fatal error: [E_CPU_SURFACE] failed to launch vm worker. This branch names both failures, as shown above; 235da5cb8, this branch's third commit before the rebase, names the record's difference alone, and the baseline names it over 24 lines. MSHV and WHP refuse such a profile when they create the partition, before these checks run ([E_PROFILE_UNSUPPORTED] the hypervisor's processor features cannot present CPU profile intel.host.v1: they lack sse4a_support), on this branch as before.

  • Every difference. A snapshot that this branch captured, whose manifest's effective CPUID record then had bit 0 of EAX flipped in every entry, fails to restore on both binaries with the same exit line, fatal error: [E_CPU_SURFACE] failed to launch vm worker. The baseline counts every entry but names 16, over 321 lines; this branch names each one, on one line, and no other failure, as the other checks, which now run too, pass:

    Backend	Host CPU	CPU profile	Entries	Baseline names	This branch names
    Linux/KVM	Xeon Platinum 8573C	intel.emeraldrapids.v1	68	16 (6.9 KB, 321 lines)	68 (7.4 KB, 1 line)
    Linux/MSHV	Xeon Platinum 8573C	intel.emeraldrapids.v1	68	16 (6.9 KB, 321 lines)	68 (7.4 KB, 1 line)
    Windows/WHP	Xeon Platinum 8370C	intel.icelake-sp.v1	69	16 (6.9 KB, 321 lines)	69 (7.5 KB, 1 line)

• NVX's microVM scenarios smp-snapshot (1, 2, 4, and 8 vCPUs), snapshot-core, and time-abi-conformance (8 vCPUs), whose cold boots and restores pass every CPU check, pass at e4c741e18, as they did at 93cf08c4d before the rebase, on Linux/KVM, Linux/MSHV, and Windows/WHP on those VMs, and a --x-time-abi-verify cold boot on each logs time ABI CPU checks passed with the checks' timings.

• CI: every check passed at e4c741e18, OpenVMM's VMM tests included, which ran only there. The first attempt of run 40958505253 failed its two verify openhcl binary size jobs: they compare with the feature/microvm/dev CI run of c32c57d12, which was still running, and their rerun passed once it completed.

Notes

• Every check passes and fails as before, and a failure keeps the code that it had: an error leads with the failure of the first check that fails. A restore whose record differs now also runs the check of VP 0's CPUID and the support check, as every boot does. Both only read, before any VP runs: VP 0's CPUID as the backend presents it, and the CPU surface that the backend supports.
• Only the microVM profile's time ABI checks change.

[github-actions]

27 Petri tests failed (28 unstable)

[github-actions]

1 Petri tests failed (2 unstable)