[Pedro Henrique Penna (ppenna)]

Moves every microVM to NVX time ABI 2, for microsoft/nvx#1009: OpenVMM pairs each time sample's host UTC with the guest TSC on the host and publishes with it the host's clock correction, which the guest applies instead of estimating over minutes how fast host UTC runs.

Based on feature/microvm/dev. This branch is six commits directly on bcfc47cff, the head of feature/microvm/dev since #4818 merged. It was rebased there from c32c57d12: git range-diff shows commits 1-3 and 5-6 unchanged; commit 4 also makes the CPU-check test backend that the newer base added implement the new guest_tsc_offset obligation. The second commit retains the earlier fix that reads Windows' precise time adjustment in the right direction (see Validation). microsoft/nvx#1018 pins this head.

Why

Under time ABI 1 the guest pairs host UTC with its own clock: it reads that clock around the port write that latches a time sample and accepts the pairing only within 1 ms at boot and restore, or 50 µs for its discipline. A port write that a loaded host serves late costs a sample, and three cost a boot or a restore. The guest then estimates how fast host UTC runs against its TSC by steering the kernel's phase-locked loop over minutes, although the host's time service already knows the correction that it applies to host UTC. Time ABI 2 has the VMM pair host UTC with the guest TSC, which does not depend on how fast the host serves the port write, and publish the host's correction.

Shared host TSC bracket (fc442fda6)

HostTscBracket, which reads the host TSC around an operation with LFENCE on both sides, cycles_to_ns_ceil, and pair_host_sample move from the KVM backend to virt::time_abi::tsc, since every backend now needs them. The KVM backend uses them unchanged.

Host UTC's discipline (60b80ecc2)

host_clock_discipline reads the host's correction from the host OS, whatever time service runs there, rather than from a hypervisor feature:

• Linux: adjtimex with no modes: the frequency correction plus the tick's deviation from its nominal 10 ms, the status (STA_CLOCKERR is unreliable, STA_UNSYNC free-running, anything else synchronized), and the maximum error. The current clocksource names the clock's basis: the TSC at the kernel's kHz calibration, or at the hypervisor's rate through the Hyper-V TSC page.
• Windows: GetSystemTimeAdjustmentPrecise. Its increment is the performance counter's frequency, and Windows counts a second of UTC for every adjustment's worth of the counter's ticks. UTC therefore runs at the increment over the adjustment of the counter's rate, and a larger adjustment slows it. An adjustment that Windows does not apply is a free-running clock, and a zero adjustment an unknown one.

host_suspended_ns reveals a host suspension: CLOCK_BOOTTIME minus CLOCK_MONOTONIC, and the interrupt time minus the unbiased interrupt time.

Publishing (732f827f0)

virt::time_abi::publish computes the host's fields of a time sample:

• Host UTC paired with the guest TSC: the tightest of up to 16 pairings of a host time sample with the host TSC, plus the guest TSC's offset from the host TSC, and the pairing's bound, which adds the offset's uncertainty.
• The correction against the guest TSC counted at the declared rate, in ppm scaled by 2^16: (1 + host correction) * F / F_d - 1. On a restore it includes the native rate's deviation from the declared one.
• A bound on the correction's error: the rounding when the host's clock and the backend's native rate come from one calibration (the kernel's kHz rate for KVM, the hypervisor's rate for MSHV and WHP), one kHz of the native rate when they do not, and unknown when the host's clocksource may not follow the TSC.
• The status and maximum error of host UTC, as its time service reports them.
• A disruption counter, which changes on a host suspension, on a step of host UTC of 1 ms or more beyond what the correction explains, and on a change of the guest TSC's offset.

It adds the host-rate-offset-ppm test hook, which publishes host UTC as running that many ppm faster than it does, with the correction to match, so that tests can check that a guest follows a host clock whose rate differs from the TSC's. The publisher also applies utc-offset-ms, which the portb device applied until now.

The guest TSC's offset on every backend (dcd96dee4)

TimeAbiBackend::guest_tsc_offset runs at every start of the VM, with every VP bound and stopped, after the TSC writes of a cold boot, a restore, or a reset, and before any VP runs:

• KVM returns VP 0's KVM_VCPU_TSC_OFFSET, which preflight verifies is the unscaled offset.
• MSHV and WHP read VP 0's TSC between two host TSC reads, up to eight times, and keep the tightest reading. Both freeze partition time until a partition starts, so the measurement resumes it first, as the partition's start would right after.

Preflight also reports what each backend's native rate measures, which bounds the published correction's error.

Time ABI 2 (61374b068)

The time sample that writing 0xa7 to the portb status port latches becomes version 2, 48 bytes, little-endian:

Offset	Size	Field
0	1	Version, 2
1	1	Flags; bit 3: a test hook is active
2	1	Status of host UTC: 0 unknown, 1 synchronized, 2 free-running, 3 unreliable
3	1	Reserved, 0
4	4	Generation
8	8	Host UTC, ns
16	8	Guest TSC at the same instant
24	4	Pairing bound, ns
28	4	Disruption counter
32	8	Correction, signed, ppm × 2^16
40	4	Bound of the correction's error, ppm × 2^16; 0xffffffff unknown
44	4	Maximum error of host UTC, µs; 0xffffffff unknown

The portb device takes the host's fields from a platform resource, TimeSampleSourceKind, which the VM worker resolves to the time ABI's clock; the device keeps the generation, the test-hook flag, and the sample-delay-us hook. The partition unit now depends on the time-abi unit, so the offset's measurement follows the TSC writes of every start and precedes the VPs' release, including a resume after a failed capture. Until the first measurement, or when the host cannot pair a sample, the window stays empty and the guest rejects the sample.

The restore packet becomes version 5, which drops host UTC: the guest takes a time sample for its repair, so the packet no longer latches UTC at its first selection, and its header shrinks to 24 bytes.

CPUID leaf 0x40000002 reports time ABI 2.0 in EBX (0x00020000), and snapshots record time ABI version 2. A snapshot of time ABI 1 fails restore with E_MANIFEST_TIME and must be recaptured. The CPU time bits do not change, so every pinned CPU profile serves time ABI 2 as it is.

Petri's microVM configuration no longer sets the portb device's utc_offset_ms, which the publisher applies now, and the guest of the ttrpc microVM VMM test reads restore packet 5: version 5, and 17 header bytes after the range count instead of 25.

Guide (090fbab88)

The snapshots page describes restore packet 5 and the version-2 sample field by field, how OpenVMM pairs the guest TSC and where it reads the correction, that the samples discipline only the guest's wall clock and are no paravirtual clock, and that a snapshot of time ABI 1 must be recaptured. The CLI and gRPC references name restore packet version 5.

Tests

• virt::time_abi::tsc: the cycle conversion, and the bracket's midpoint and its refusal of a TSC that went backwards.
• virt::time_abi::host: the Windows adjustment's arithmetic, its direction with two Windows hosts' readings, and its rounding, that each host reports a status and a plausible correction, and the clock kinds.
• virt::time_abi::publish: the correction's arithmetic against exact values, its bounds, the pairing of a sample, the disruptions of Linux and Windows clocks, the hooks, and a real host sample's pairing.
• chipset_resources::microvm_time: the version-2 sample and the version-5 packet, encoded and decoded byte for byte.
• chipset::microvm: the portb device latches the source's fields into its window, and leaves it empty without a sample.
• openvmm_core: the time-abi unit's start measures the offset that samples use, and a changed offset is a disruption. A restore rejects a version-1 time contract.
• Hardware tests: the MSHV test checks that the measurement resumes partition time and agrees with a direct read of VP 0's TSC, and a new WHP test checks that the offset holds while time runs (on a Windows 11 host, two measurements 500 ms apart differed by 87 cycles, within the 3,700 cycles of their uncertainty).

Validation

• Windows, WHP: cargo clippy --all-targets -- -D warnings, cargo doc --no-deps, and cargo nextest run for the changed packages, the WHP time-ABI hardware tests, and cargo xtask fmt: clean.
• Linux, KVM (an Azure VM): cargo nextest run for the changed packages (796 passed) and the KVM time-ABI hardware tests (5 passed).
• Linux, MSHV (an Azure VM): the MSHV time-ABI hardware tests (5 passed) and cargo doc --no-deps. cargo clippy --all-targets -- -D warnings for every changed package found two useless_conversion errors in host.rs, which the second commit now fixes; clippy and virt's time_abi tests (66) then passed.
• This pull request's first CI run failed CI's quick check on petri, which still set the portb device's removed utc_offset_ms. The fifth commit now drops it, and the quick check's clippy, cargo clippy --locked --keep-going --all-targets --workspace --all-features --target x86_64-unknown-linux-gnu --exclude guest_test_uefi --exclude opentmk_executor with -Dwarnings, passes on the Azure MSHV VM. Looking for other users of the changed formats, the ttrpc microVM VMM test's guest still parsed restore packet 4, so the fifth commit updates it too. cargo xflowey vmm-tests-run passed both microVM VMM tests, ttrpc::test_ttrpc_microvm_linux_direct_lifecycle_and_snapshot and x86_64::microvm::openvmm_linux_x64_phase_1_lifecycle, on the Azure KVM VM, and the ttrpc one failed without the update (RESTORE-PACKET-INVALID).
• After the rebase onto bcfc47cff, at 090fbab88: git range-diff keeps commits 1-3 and 5-6 patch-equivalent and adds only the newer base's CpuBackend::guest_tsc_offset mock to commit 4. On Windows, package-scoped check, clippy with -D warnings, rustdoc, and 697 unit tests pass for every generic and WHP crate; 18 ignored WHP hardware tests and cargo xtask fmt --fix pass. On Linux/KVM, the generic and KVM crates pass check, clippy, rustdoc, 731 unit tests, 12 ignored hardware tests, and both affected VMM tests. On Linux/MSHV, virt_mshv passes check, clippy, rustdoc, 79 unit tests, and 13 ignored hardware tests.
• The host readers on real hosts: adjtimex reports synchronized clocks on the Azure Linux test VMs and all ten Linux CI runners, at corrections from -76 to +7 ppm, and GetSystemTimeAdjustmentPrecise reports applied adjustments on an Azure Windows VM, on all ten Windows CI runners, from -72.5 to +6.9 ppm, and at +17.2 ppm on a bare-metal Windows host (read in the right direction; see below).
• NVX: time ABI: publish the host's clock correction with each time sample nvx#1018 pins this head, moves NVX's guest to time ABI 2, and adds a microVM scenario, time-host-correction, which checks with the host-rate-offset-ppm hook that the guest follows a host clock that runs at another rate than the TSC. With this branch's OpenVMM, NVX's default microVM suite and the scenario passed on Linux/KVM, Linux/MSHV, and Windows/WHP Azure test VMs.
• The Windows correction's direction. NVX's CI ran that scenario on a Windows runner whose time service ran host UTC 48.6 ppm slow against the guest TSC, while this OpenVMM published +48.7 ppm.
  • Cause: the second commit read GetSystemTimeAdjustmentPrecise as the legacy GetSystemTimeAdjustment, which adds its adjustment once per increment.
  • Evidence: a 20 s comparison of UTC with QueryPerformanceCounter showed that a larger adjustment slows UTC. On that runner, whose adjustment moved from 10,000,501 to 10,000,588 for an increment of 10,000,000, UTC ran 53.1 ppm slow. On the Windows test VM, at an adjustment of 10,000,005, UTC ran 0.447 ppm slow.
  • Fix: the amended second commit takes the increment over the adjustment, minus one, and its tests check that direction with these readings.
  • Validation at the head: virt's formatting, clippy with -Dwarnings, and its time ABI tests (66) pass on Windows and Linux.
  • End to end on the Windows/WHP VM: NVX's scenario published corrections within 0.04 to 0.12 ppm of host UTC's measured rate against the guest TSC; the old reading left gaps of 0.24 to 0.93 ppm. NVX's H4 read -0.600 ppm, where UTC measured -0.612 ppm against the counter.
• CI at the previous head 75765b660 passed. Exact-head CI for 090fbab88 is https://github.com/microsoft/openvmm/actions/runs/41015566000. The earlier binary-size check needed its base commit's CI run as a baseline and passed on rerun; the size comparison found nothing. The Petri tests that the bot reports are OpenHCL UEFI Windows tests marked unstable, which timed out.

Notes

• This is a breaking change of the time ABI: a guest of time ABI 1 fails its conformance check against this OpenVMM, and snapshots of time ABI 1 must be recaptured. NVX moves its guest to time ABI 2 in the commit that pins this head.
• The samples discipline only the guest's wall clock. They are no paravirtual clock: the guest's clocksource stays its TSC.

[github-actions]

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

[github-actions]

2 Petri tests failed (3 unstable)