[Pedro Henrique Penna (ppenna)]

Fixes #1009. Pins microsoft/openvmm#4809 at 090fbab88f2cc2e069147c8dca122e343c9363ed, which moves every microVM to time ABI 2: OpenVMM pairs each time sample's host UTC with the guest TSC on the host and publishes with it the host's clock correction, which the guest now applies instead of estimating it over minutes. The guest, nvx.py doctor's H4, the architecture guide, and a new microVM scenario move with the pin.

Based on dev at 78c626dac. microsoft/openvmm#4809 is six commits directly on bcfc47cff, the head of feature/microvm/dev since microsoft/openvmm#4818 merged, and the pin is that pull request's rebased head commit itself.

Change

Commit	Files	Change
edcce2828 openvmm	openvmm, nvx-rs/, guest/common/nvx-snapshot, wording in scripts/	bcfc47cff -> 090fbab88, and the guest of time ABI 2 and restore packet 5, which must move with it: the version-2 sample's decoder, the pairing against the guest's own TSC bracket, the frequency correction from the host's correction, the discipline that holds it, the restore repair's sample, the time state's new fields, nvx-time sample, and the template fixtures' manifests, regenerated at the pin
d0491ef80 doc	doc/design/{1.2,3.1,3.2,5.5}, doc/build.md	Specifies the version-2 sample, the guest's pairing and frequency, the discipline, and restore packet 5
7a8d13b75 doctor	scripts/nvx_tools/doctor.py, scripts/test_time_abi.py, doc/ci.md, validate-runner	H4 keeps the 250 ppm rate budget and requires a synchronized host clock correction, read as OpenVMM reads it; the direct correction is evidence
2e3f894dc test-microvm	scripts/nvx_tools/microvm_tests.py, its guest script and tests, microvm_shards.json, doc/ci.md, doc/usage.md	The time-host-correction scenario
4aeff73a8 doc	doc/openvmm-upstream-roadmap.md	Records the pin's commits as positions 368-373 and moves the NVX pin

OpenVMM's control-session protocol vectors and built-in CPU profiles are unchanged at the pin.

Time ABI 2

OpenVMM (microsoft/openvmm#4809). Writing 0xa7 to the portb status port latches a 48-byte version-2 sample: host UTC and the guest TSC at the same instant, paired on the host through the guest TSC's offset, which every backend measures before any VP runs (KVM's KVM_VCPU_TSC_OFFSET; VP 0's TSC between two host TSC reads on MSHV and WHP), with the pairing's bound; the status of host UTC; a disruption counter, which changes on a host suspension, a step of host UTC, or a change of the offset; and the host's correction, how fast host UTC runs against the guest TSC counted at the declared rate, with the bounds of its error and of host UTC's. OpenVMM reads the correction from the host OS, whatever time service runs there: adjtimex on Linux, and on Windows GetSystemTimeAdjustmentPrecise, whose increment is the performance counter's frequency and whose adjustment is the counter's ticks in each second of UTC, so that UTC runs at the increment over the adjustment of the counter's rate. After a restore the correction includes the restore's rate deviation. Restore packet 5 drops host UTC; CPUID leaf 0x40000002 reports time ABI 2.0, and snapshots record time ABI version 2.

Guest (nvx-rs). nvx-time reads its TSC around the selector write and its realtime clock between two more TSC reads, accepts a sample only if its TSC lies in that bracket, widened by the pairing bound and the 1 µs skew bound, and takes the offset from host UTC at the sample's TSC. It sets the kernel's frequency correction so that its realtime clock, which Linux counts at the TSC rate rounded down to the kHz, runs at host UTC's published rate, within Linux's ±500 ppm, and reports frequency_limited beyond it. Boot steps to host UTC and holds that frequency with STA_FREQHOLD, so that the kernel's PLL corrects only the phase; every poll holds the latest correction, and a changed disruption counter returns the discipline to its fast cadence. Boot, restore, and polls take up to three samples within 100 µs, where time ABI 1 accepted 1 ms at boot and restore. A restore's repair takes its sample in the packet's port transaction, so a clone runs at the destination host's correction from its repair on. The time state reports the host's status, correction and its bound, host UTC's maximum error, and the disruption counter.

The issue's next steps

• Specification. 3.2 specifies the published values, their transport, and the discipline's use of them, and says that they are no paravirtual clock; 1.2 names VMClock among the excluded ones.
• OpenVMM. [DO NOT REVIEW] microVM: publish the host's clock correction with each time sample openvmm#4809 reads the correction on Linux and Windows and publishes it with each sample.
• nvx-rs. The discipline and the restore repair apply it.
• doctor. H4 keeps dev's 250 ppm guest-budget gate, chronyd re-reads, warning, kernel-PLL fallback, and average-since-boot evidence, and additionally requires the host to publish a synchronized correction. The directly read status, correction, and maximum error are evidence; the independent rate gate still protects the guest's correction budget.
• Tests. time-host-correction reads the host's own correction from an unhooked cold boot. OpenVMM's host-rate-offset-ppm hook then selects ordinary corrections below Linux's limit and a 900 ppm limit phase. The ordinary hooks solve OpenVMM's multiplicative rate composition and stay within its +/-10,000 ppm parser range and fail explicitly if no accepted hook can avoid saturation; the frequency allowance is derived from the reported TSC rate down to the ABI's 500 MHz minimum. It runs on KVM, MSHV, and WHP.

The open questions:

• Transport. Version 2 of the 0xa7 sample, 48 bytes, rather than a page that the guest maps.
• Refresh. OpenVMM reads the host's correction for every sample, and the guest holds the correction of each sample that it accepts.
• Time ABI 1 snapshots. A time ABI 2 OpenVMM refuses them (E_MANIFEST_TIME); snapshots and templates must be recaptured.
• Host sleep. A host suspension changes the disruption counter.

Compatibility

• The guest and OpenVMM must both implement time ABI 2: a guest of either version fails the other's conformance check, so the guest artifacts, the release, and nvx-rs move together with the pin.
• Snapshots and templates of time ABI 1 must be recaptured.
• H4 fails a host whose clock no time service synchronizes, or that reports it unreliable. On the test VMs, chronyd synchronizes the Linux hosts' clocks and the Windows Time service the Windows host's; each CI job's Validate runner step reports its runner's.

Validation

This branch was rebuilt on dev four times while dev moved, each time carrying the same patches:

• At 565d1ebe5, on 67f395e54, pinning OpenVMM b27be7e6a.
• At cf7453276, on f8469b80a, pinning 93b2c4d91.
• At 013c960c8, on d89d37090, pinning a417f946a.
• At the rebased head 4aeff73a8, on 78c626dac, pinning 090fbab88. This rebuild preserves dev's newer H4 budget, chronyd, PLL, warning, and average-correction work; folds in the current review fixes; and carries the Windows correction fix below.

The Windows correction's sign. CI's run of 032774707 failed the second WHP shard on azure-windows-3, in time-host-correction's hooked cold boot. There, host UTC ran at +151.4 ppm against the guest TSC while OpenVMM published +248.7 ppm. Without the hook, OpenVMM published +48.7 ppm as the host's own correction while host UTC ran at -48.6 ppm.

• Cause. OpenVMM read GetSystemTimeAdjustmentPrecise as the legacy GetSystemTimeAdjustment, which adds its adjustment once per increment. The precise API instead reports frequencies: the increment is the performance counter's, and Windows counts a second of UTC for every adjustment's worth of the counter's ticks. A larger adjustment therefore slows UTC.
• Evidence. A 20 s comparison of UTC with QueryPerformanceCounter showed this on both Windows hosts measured:
  • On azure-windows-3, whose time service moved its adjustment from 10,000,501 to 10,000,588 for an increment of 10,000,000, UTC ran 53.1 ppm slow.
  • On the Windows test VM, at an adjustment of 10,000,005, UTC ran 0.447 ppm slow.
• Fix. Commit 2 of [DO NOT REVIEW] microVM: publish the host's clock correction with each time sample openvmm#4809 and the doctor now take the increment over the adjustment, minus one, and their unit tests check that direction with these readings.
• Consequence. The Windows runners' corrections in the H4 probe below have the opposite sign of those that this pull request first reported.

The validation:

• OpenVMM (see [DO NOT REVIEW] microVM: publish the host's clock correction with each time sample openvmm#4809):
  • Before the Windows fix: clippy, rustdoc, nextest, and the time ABI hardware tests on Windows/WHP, Linux/KVM, and Linux/MSHV.
  • At 090fbab88: formatting; package-scoped check, clippy with -D warnings, rustdoc, and unit tests for every changed crate on Windows and Linux; 18 WHP, 12 KVM, and 13 MSHV ignored hardware tests; and the two affected KVM VMM tests all pass.
  • Its CI found petri still setting the portb device's removed utc_offset_ms, and a VMM test showed the ttrpc microVM test's guest reading restore packet 4. The pin fixes both, and both microVM VMM tests passed on Linux/KVM.
• NVX on Windows:
  • At 4aeff73a8: nvx.py verify; the time ABI (75), microVM (305), and nvx_tools (566) suites; ruff, Linux and Windows pyright, Markdown links; and the OpenVMM vectors and CPU profiles (3 and 1).
  • At 990e4b425, before the final base-only refresh: shell formatting and linting; the nvx-rs feature matrix, rustdoc, MSRV, and runtime-adapter protocol, host, and Linux-musl guest clippy checks.
  • At the earlier d17c209d5, before a review corrected design doc 3.2's pairing equations: nvx.py verify; the time ABI, scenario, shard, and Markdown tests (185) and test_nvx_tools.py (566); ruff, pyright, and scripts/check_markdown_links.py; and the openvmm_vectors and openvmm_cpu_profiles comparisons, ignored tests included (3 and 1).
  • At cf7453276: nvx-rs and runtime adapter fmt and clippy, for the host and for x86_64-unknown-linux-musl.
• NVX on Linux, on an Azure MSHV VM:
  • At 990e4b425: the full validate-nvx suite (1,503 tests), adversarial (84), host-connect (4), Specula (25), and Markdown links; every check-nvx-rs step, including guest binaries and the runtime adapter's protocol, host, and guest configurations.
  • At 990e4b425 with OpenVMM 090fbab88 and freshly built guest artifacts: time-host-correction and doctor --checks H3 H4 --ci-schedule pass on Linux/KVM and Linux/MSHV.
  • At 032774707 with the scenario's probe: the full validate-nvx Python suite (1,467 tests), plus the adversarial (84) and host-connect (4) tests.
  • At 032774707: the runtime adapter's VM scenarios (154), including dev's new clone and template scenarios.
  • At cf7453276, every step of check-nvx-rs:
    • fmt and clippy;
    • the tests with all, default, and no features;
    • the bundling check and rustdoc;
    • the MSRV (1.89) and macOS checks;
    • the guest binaries' checks;
    • the runtime adapter's protocol, host, and guest configurations.
• Template fixtures: the fixtures regenerated at 75765b660 remain byte-identical at 090fbab88; the exact-pin tests pass.
• End to end on three Azure test VMs, with OpenVMM, the guest artifacts (built with Docker), and nvx-rs built from the branch:
  • At d17c209d5 on Windows/WHP, with OpenVMM 75765b660:
    • time-host-correction ran twice. Its published corrections matched host UTC's measured rate against the guest TSC within 0.04 to 0.12 ppm; the old reading left gaps of 0.24 to 0.93 ppm, twice the host's correction.
    • nvx.py doctor --checks H3 H4 --ci-schedule passed. H4 read -0.600 ppm for an adjustment of 10,000,006, and host UTC measured -0.612 ppm against the counter.
  • At 032774707 with the scenario's probe: time-host-correction passed 3 times on each backend.
    • Duration: 13.4 to 13.6 s on Linux/KVM and Linux/MSHV, and 17.5 to 20 s on Windows/WHP.
    • The probe read the host's own correction as -17 ppm on KVM, -44 to -69 ppm on MSHV, and about 0 ppm on WHP.
    • The hooks then published about 200 and 150 ppm against that sign, negative on WHP.
    • The limit phase held 500000 ppb in the host correction's direction with frequency_limited=1.
  • At 565d1ebe5:
    • nvx.py doctor --checks H1 H2 H3 H4 H6 on the qualification schedule passed on Linux/KVM, Linux/MSHV, and Windows/WHP.
    • The default microVM suite passed on all three backends.
    • The runtime adapter's VM scenarios passed on Linux/KVM and Linux/MSHV.
  • At cf7453276 and ea808fa91 on Linux/MSHV:
    • time-host-correction, time-abi-conformance, snapshot-core, restore-downtime, smp-snapshot, and cpu-profile-fallback passed.
    • nvx.py doctor --checks H1 H2 H3 H4 --ci-schedule passed.
  • Throughout: the guest held the host's correction to the kernel's kHz rounding (0, 207, and 70 ppb on KVM, MSHV, and WHP), and its clock ran within 0.85 ppm of it.
• Host UTC slews. While validating, host UTC ran up to 24 ppm away from its own correction over a 5 s window whenever the host's time service updated it in between; otherwise it follows adjtimex to 0.03 ppm.
  • A check of the guest's clock directly against host UTC failed 1 of 12 runs on such a slew.
  • The scenario therefore checks the guest's clock against the correction that it holds, through the samples' pairing of that clock with the guest TSC.
  • It checks host UTC against its published correction within 75 ppm.
• All 20 CI runners report synchronized host clocks, so the new H4 gate passes on each of them. The Linux runners report neither STA_UNSYNC nor STA_CLOCKERR, the Windows ones an applied time adjustment, and their corrections range from -76 to +6.9 ppm.
• CI:
  • The full run at ea808fa91, before the review's fixes, passed.
  • The run at 032774707 failed only where it found the Windows sign above.
  • The run at the previous head a91e5e5f6 (40963908778) passed:
    • On azure-windows-3, time-host-correction read the host's own correction as -52.8 ppm and published corrections that host UTC followed.
    • Its first attempt failed only the runtime adapter's vm_clone::a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used on KVM. That test fails intermittently on dev as well (runtime-adapter: a_clone_refuses_a_scratch_image_whose_upper_an_earlier_clone_used fails intermittently #1020); dev's own push run at 4d9e76d90 failed it the same hour. A rerun passed.
  • [DO NOT REVIEW] microVM: publish the host's clock correction with each time sample openvmm#4809's exact-head CI for 090fbab88 passed on attempt 10: https://github.com/microsoft/openvmm/actions/runs/41015566000.
  • Exact-head NVX CI for 4aeff73a8 is https://github.com/microsoft/nvx/actions/runs/41077274125; its WHP jobs are waiting for offline runners.

Merge microsoft/openvmm#4809 into feature/microvm/dev, by fast-forwarding that branch to 090fbab88, before merging this pull request.

[Copilot]

🟡 Changes recommended

The checker can accept stale clock corrections, and the restore-limit phase can pass without exercising Linux’s frequency limit, leaving both scenarios insufficiently validated.

2 open issues

• Validate held correction against independent host samples · New
• Require frequency-limit phase to verify clamping · New

What changed in this PR

Moves NVX to time ABI 2 so guests apply the host’s published clock correction rather than estimating it. This coordinates guest boot and restore behavior with the OpenVMM pin and addresses the host-clock qualification problems described in #1009 and #1008.

Changes:

• Updates time samples, restore packets, guest frequency control, and reported clock state.
• Changes doctor H4 to check TSC stability and synchronized host correction without gating on correction size.
• Adds a host-correction microVM scenario, regression coverage, and updated architecture documentation.

File	Description
scripts/​test_time_abi.py	Updates ABI diagnostics and host-clock qualification tests.
scripts/​test_nvx_tools.py	Updates comments describing qualification and time ABI behavior.
scripts/​test_microvm_tests.py	Adds host-correction scenario and report-checker tests.
scripts/​nvx_tools/​time_abi.py	Updates harness descriptions and errors to ABI v2.
scripts/​nvx_tools/​microvm_tests.py	Adds host-correction boot and restore validation.
scripts/​nvx_tools/​microvm_test_scripts/​time-host-correction.sh.in	Reports guest state and paired time samples.
scripts/​nvx_tools/​microvm_test_scripts/​snapshot-core.sh	Removes an outdated packet-version comment.
scripts/​nvx_tools/​microvm_shards.json	Adds the scenario and rebalances test shards.
scripts/​nvx_tools/​doctor.py	Reads host clock corrections and revises H4 qualification.
nvx-rs/​tests/​guest_tools.rs	Updates packet, sample, and command-output tests.
nvx-rs/​tests/​fixtures/​template/​README.md	Documents regenerated ABI v2 fixture manifests.
nvx-rs/​src/​host/​template/​test_support.rs	Uses the current time ABI in manifest fixtures.
nvx-rs/​src/​host/​template/​manifest/​tests.rs	Tests rejection of incompatible time ABI versions.
nvx-rs/​src/​host/​launch.rs	Removes version-specific wording from a comment.
nvx-rs/​src/​guest/​time/​test_mode.rs	Updates decoder and clock-arithmetic test hooks.
nvx-rs/​src/​guest/​time/​system.rs	Pairs samples with guest clocks and applies host corrections.
nvx-rs/​src/​guest/​time/​state.rs	Records host status, correction, bounds, and disruptions.
nvx-rs/​src/​guest/​time/​mod.rs	Identifies the guest implementation as ABI v2.
nvx-rs/​src/​guest/​time/​daemon.rs	Holds published frequency corrections and handles disruptions.
nvx-rs/​src/​guest/​time/​cpu_mask.rs	Removes version-specific wording from a comment.
nvx-rs/​src/​guest/​time/​commands.rs	Expands nvx-time sample output and pairing validation.
nvx-rs/​src/​guest/​time/​checks.rs	Checks the ABI v2 CPUID identity.
nvx-rs/​src/​guest/​time/​api.rs	Applies sampled host corrections during boot and restore.
nvx-rs/​src/​guest/​sys/​mod.rs	Exposes the frequency-hold type internally.
nvx-rs/​src/​guest/​sys/​discipline.rs	Holds frequency while allowing kernel phase correction.
nvx-rs/​src/​contract.rs	Sets time ABI 2 and restore packet 5.
nvx-rs/​README.md	Updates guest API and tool descriptions to ABI v2.
guest/​common/​nvx-snapshot	Updates comments for packet v5 and sampled restore time.
doc/​usage.md	Describes the host-correction scenario.
doc/​openvmm-upstream-roadmap.md	Records the new pin and its six commits.
doc/​design/​5.5-restore-protocol.md	Specifies packet v5 and sample-based clock repair.
doc/​design/​3.2-clock-and-time.md	Specifies ABI v2 samples, corrections, and discipline.
doc/​design/​3.1-platform-control-device.md	Describes host-paired time samples.
doc/​design/​1.2-non-goals.md	Clarifies the distinction from paravirtual clocks.
doc/​ci.md	Documents revised H4 checks and scenario expectations.
doc/​build.md	Updates guest-tool and boot-time behavior descriptions.
.github/​actions/​validate-runner/​action.yml	Updates comments describing host qualification.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Fixed boot and restore offsets can push host corrections beyond Linux’s ±500 ppm limit, causing the non-limit checks to reject guests that correctly clamp their corrections.

1 open issue

• Adjust H4 hooks to keep host corrections within Linux limits · New

2 resolved since last review

• Require frequency-limit phase to verify clamping
• Validate held correction against independent host samples

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The ABI equations mix TSC-derived seconds with nanosecond timestamps, requiring a seconds-to-nanoseconds conversion to avoid incorrect extrapolation and uncertainty calculations.

2 open issues

• Convert TSC-derived seconds to nanoseconds · New
• Adjust H4 hooks to keep host corrections within Linux limits

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The coordinated guest time-ABI and snapshot-format changes require maintainer confirmation of exact-pin compatibility, artifact deployment, and pending cross-platform validation across KVM, MSHV, and WHP.

1 open issue

• Convert TSC-derived seconds to nanoseconds

1 resolved since last review

• Adjust H4 hooks to keep host corrections within Linux limits

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The breaking time-ABI and restore-protocol migration needs maintainer review and exact-pin KVM/MSHV/WHP validation, with the coupled OpenVMM PR still open and its full CI unfinished.

No open issues 🎉

1 resolved since last review

• Convert TSC-derived seconds to nanoseconds

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The default time-correction scenario generates offsets outside OpenVMM’s accepted range, and the rounding allowance can reject valid sub-GHz guests.

2 open issues

• Low TSC rate correction allowance rejects valid guests · New
• Generated host correction exceeds OpenVMM offset range · New

_{Review effort: Max}

[Copilot]

🟡 Changes recommended

H4 still rejects stable, synchronized hosts when H3 supplies native_tsc_hz and correction-derived rate deviation exceeds 250 ppm, preserving the qualification failure the change intends to remove.

1 open issue

• Make correction-derived rate deviations evidence-only with H3 · New

2 resolved since last review

• Generated host correction exceeds OpenVMM offset range
• Low TSC rate correction allowance rejects valid guests

_{Review effort: Max}

[Copilot]

🟡 Changes recommended

The hook-rate calculation adds rates instead of composing them multiplicatively, allowing saturating phases and rejecting valid ones, with the same error reflected in the tests.

1 open issue

• Solve and validate the composed host-rate and hook rate · New

1 resolved since last review

• Make correction-derived rate deviations evidence-only with H3

_{Review effort: Max}