From bf3f14217d259576254126f2f1c300bc178b5d08 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 11:30:51 -0700 Subject: [PATCH 1/5] openvmm: publish the host's clock correction with each time sample Pin microsoft/openvmm#4830 at 858b2f93aa78d35df6fdd217322a7f4cfb57b080, six commits on 3d7512af653717aa9123595e993a56cb8587106a, the previous pin and the head of feature/microvm/dev. They are the commits of microsoft/openvmm#4809, rebased there unchanged. Under time ABI 1 the guest paired host UTC with its own clock: nvx-time read CLOCK_REALTIME around the port write that latches a time sample and accepted the pairing only within 1 ms at boot and restore, or 50 us for its discipline, so a port write that a loaded host served late cost a sample, and three cost a boot or a restore. It then estimated how fast host UTC runs against its TSC by steering the kernel's PLL over minutes, although the host's time service already knows that rate. Time ABI 2 has OpenVMM pair each sample's host UTC with the guest TSC on the host and publish with it the host's clock correction, which the host OS reports (adjtimex on Linux, GetSystemTimeAdjustmentPrecise on Windows), the bound of the correction's error, the status and maximum error of host UTC, and a disruption counter, which changes on a host suspension, a step of host UTC, or a change of the guest TSC's offset. The version-2 sample is 48 bytes. CPUID leaf 0x40000002 reports time ABI 2.0 (EBX 0x00020000). Restore packet 5 drops host UTC, so its header shrinks to 24 bytes, and the guest's repair takes a time sample instead. The guest moves with the pin, since a guest of one time ABI fails the other's conformance check: - nvx::contract: TIME_ABI_VERSION 2 and RESTORE_PACKET_VERSION 5. - Pairing: the guest reads its TSC before and after the selector write, and CLOCK_REALTIME between those reads, and accepts a sample only if its TSC lies within that bracket, widened by the published pairing bound and 1 us of skew. It compares host UTC with CLOCK_REALTIME at the sample's TSC, moving its CLOCK_REALTIME read there at the rate at which Linux counts it; the offset's uncertainty is the pairing bound, half the bracket, and 2,000 ppm of that move. - Rate: from the host's correction and the TSC rate, which the guest reads once from the Hyper-V TSC frequency MSR (0x40000022) and keeps in its time state, the guest computes the adjtimex frequency that makes CLOCK_REALTIME, which Linux counts at its kHz rounding of the TSC rate, run at host UTC's rate. It clamps that frequency to Linux's 500 ppm limit and reports whether it did. - Discipline: boot steps the clock to host UTC and holds that frequency with STA_FREQHOLD, so that the kernel's PLL corrects only the phase. Every poll holds the latest correction and slews the measured offset, or steps it from 128 ms on. A changed disruption counter returns the daemon to its fast cadence. Boot, restore, and polls take up to three samples within 100 us; a boot that gets none fails its check, and a restore fails its repair. The maximum and estimated errors that the guest sets include host UTC's maximum error only when the host reports a synchronized clock; the guest never marks its clock unsynchronized on the host's behalf. - Restore: the repair reads packet 5 and takes its time sample in the same port transaction, then steps the clock and holds the frequency. - State: the time state records the host's status, correction, and correction bound, host UTC's maximum error, the disruption counter, and whether the frequency was limited. nvx-time sample prints a sample's fields and the guest's pairing, and the test hooks packet, sample, pair, frequency, and correction take the new fields. Templates record time ABI version 2, so the manifest reader refuses a template of time ABI 1, which must be recaptured. The fixtures' manifests, which OpenVMM's generator regenerated at the pin, differ from the previous ones only in that version. The template-key vectors keep the contract versions that their test fixes. Comments, the nvx-rs README, and nvx-snapshot name the new versions. OpenVMM's protocol vectors and CPU profiles are unchanged at the pin. Part of #1009 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- SOURCE-MANIFEST.json | 2 + doc/distribution.md | 5 +- guest/common/nvx-snapshot | 10 +- nvx-rs/README.md | 11 +- nvx-rs/build.rs | 2 +- nvx-rs/src/contract.rs | 48 +- nvx-rs/src/guest/sys/discipline.rs | 100 ++- nvx-rs/src/guest/sys/mod.rs | 2 +- nvx-rs/src/guest/time/api.rs | 256 ++++--- nvx-rs/src/guest/time/checks.rs | 11 +- nvx-rs/src/guest/time/commands.rs | 50 +- nvx-rs/src/guest/time/cpu_mask.rs | 2 +- nvx-rs/src/guest/time/daemon.rs | 236 +++++-- nvx-rs/src/guest/time/decoders.rs | 636 +++++++++++++++--- nvx-rs/src/guest/time/mod.rs | 2 +- nvx-rs/src/guest/time/state.rs | 229 ++++++- nvx-rs/src/guest/time/system.rs | 493 ++++++++++++-- nvx-rs/src/guest/time/test_mode.rs | 224 ++++-- nvx-rs/src/host/artifacts/mod.rs | 2 +- nvx-rs/src/host/artifacts/tests.rs | 4 +- nvx-rs/src/host/cli/download/install/tests.rs | 3 + nvx-rs/src/host/launch.rs | 2 +- nvx-rs/src/host/template/manifest/tests.rs | 14 +- nvx-rs/src/host/template/test_support.rs | 2 +- nvx-rs/tests/fixtures/template/README.md | 13 +- .../template/platform-kvm/manifest.bin | Bin 13554 -> 13554 bytes .../platform-mshv-network/manifest.bin | Bin 14035 -> 14035 bytes nvx-rs/tests/guest_tools.rs | 112 +-- openvmm | 2 +- .../microvm_test_scripts/snapshot-core.sh | 2 +- scripts/nvx_tools/time_abi.py | 4 +- scripts/test_microvm_tests.py | 2 +- scripts/test_nvx_tools.py | 8 +- scripts/test_time_abi.py | 2 +- 34 files changed, 2019 insertions(+), 472 deletions(-) diff --git a/SOURCE-MANIFEST.json b/SOURCE-MANIFEST.json index 86cb8a7d7..505cc3370 100644 --- a/SOURCE-MANIFEST.json +++ b/SOURCE-MANIFEST.json @@ -6,6 +6,8 @@ }, "openvmm": { "microvm_abi_version": 2, + "time_abi_version": 2, + "restore_packet_version": 5, "control_session_protocol_version": 1, "control_contract_revision": "nvx-microvm-v2-control-v2" }, diff --git a/doc/distribution.md b/doc/distribution.md index 485e44e33..52fffe2ba 100644 --- a/doc/distribution.md +++ b/doc/distribution.md @@ -162,8 +162,9 @@ publishes. A pull request changes it to start another series, such as `tests/runtime-adapter/Cargo.lock`. The `bundled` feature of the published crate downloads the release that [`nvx-rs/artifacts.json`](../nvx-rs/artifacts.json) pins and fails unless that -release declares the crate's control contract, which CI checks before it -publishes. When the contract has changed, a pull request repins the release; +release declares the crate's microVM, time, restore-packet, control-session, +and control contracts, which CI checks before it publishes. When a contract has +changed, publication stays blocked until a pull request repins a compatible release; see [Bundled artifacts](../nvx-rs/README.md#bundled-artifacts). When CI cannot publish, as when the newest version on crates.io records no diff --git a/guest/common/nvx-snapshot b/guest/common/nvx-snapshot index fb531014e..680a6af1a 100644 --- a/guest/common/nvx-snapshot +++ b/guest/common/nvx-snapshot @@ -26,7 +26,7 @@ esac runtime=/run/nvx container_cgroup=/sys/fs/cgroup/container scratch_mount=$runtime/scratch -# Restore packet v4 flags (doc/design/5.5-restore-protocol.md). +# Restore packet v5 flags (doc/design/5.5-restore-protocol.md). PACKET_MEMORY_TARGET=2 PACKET_ACK_REQUIRED=4 container_frozen=false @@ -432,10 +432,10 @@ export NVX_VM_GENERATION_ID="$generation_id" entropy=$runtime/restore-entropy # Guest repair steps 4 to 7 (doc/design/5.5-restore-protocol.md). nvx-time -# capture has already read restore packet v4, checked its generation, set -# the wall clock, and written the entropy. Its metadata is FLAGS, -# ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by START -# and LENGTH for each memory range. +# capture has already read restore packet v5, checked its generation, set +# the wall clock from a time sample, and written the entropy. Its metadata is +# FLAGS, ONLINE_VP_COUNT, RANGE_COUNT, and the new generation ID, followed by +# START and LENGTH for each memory range. post_restore() { post_restore_pending=true [ "$#" -ge 4 ] || fail_closed "restore packet metadata is invalid" diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 6aed10f2f..0ae36fccc 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -134,7 +134,7 @@ changes only when the module's purpose does. | [`nvx::guest::cgroup`](#nvxguestcgroup) | Cgroup v2 handles through which a guest init or agent places, accounts, kills, and settles its executions | | [`nvx::guest::exec`](#nvxguestexec) | `Execution`: drives one program as an execution, for any guest init or agent | | [`nvx::guest::exec::binding`](#nvxguestexecbinding) | `Binding`: an `nvx::exec::Runner` bound to an `Execution`, the glue that a guest adapter would otherwise write | -| [`nvx::guest::time`](#nvxguesttime) | x86_64: the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent | +| [`nvx::guest::time`](#nvxguesttime) | x86_64: the steps of NVX time ABI v2 as typed calls in the caller's process, for any guest init or agent | | [`nvx::guest::clone`](#nvxguestclone) | x86_64: the preparation of a template's guest for a capture, and the repair and activation of each clone | | [`nvx::guest::reseed`](#nvxguestreseed) | A restore's fresh entropy, and the reseed of the kernel's random number generator with it | | [`nvx::guest::root`](#nvxguestroot) | The operations of a layered root: its bootstrap, assembly, shares, export, mappings, and teardown | @@ -1188,7 +1188,7 @@ binding could not read the launch report. ### `nvx::guest::time` -x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's +x86_64: `TimeAbi`, the steps of NVX time ABI v2 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a @@ -2467,8 +2467,9 @@ Without `NVX_BUNDLE_DIR`, the build script downloads the release package that [`artifacts.json`](artifacts.json) pins with `curl` and extracts it with `tar`; Windows packages are ZIP archives, which need bsdtar (Windows' own `tar`) or `unzip`. Every staged file must match the source's `SHA256SUMS`, a download -must match its pinned digest, and the release must declare the control contract -of `nvx::contract`. The build script extracts and verifies only the files that +must match its pinned digest, and the release must declare the microVM, time, +restore-packet, control-session, and control contracts of `nvx::contract`. The +build script extracts and verifies only the files that the bundle holds, so a release that ships `bin/nvx-rs[.exe]` stages as the older release that `artifacts.json` pins, which doesn't. The build scripts of dependents receive the staged @@ -2515,7 +2516,7 @@ disk's block device. | `nvx-device-io file ...`, `nvx-device-io network ...`, `nvx-device-io map PATH TEXT` | The benchmark helper that measures file and UDP round-trip I/O and prints one JSON line, and reports a file benchmark's progress on stderr; `map` writes TEXT at the start of a file through a shared mapping and holds the dirty mapping until its standard input ends, for the microVM scenarios (x86_64) | | `nvx-managed-agent CONTROL_TTY ROOTFS HOSTNAME UID GID USER HOME` | Serves the managed control session and supervises sequential workloads; `--exec-config-fd FD -- WORKLOAD ...` is the direct agent's internal launch helper | | `nvx-ingress [inherit] GATEWAY_IPV4` | Announces reachable TCP listeners and settled UDP sockets to the portable-network gateway, from a privileged source port that it binds, or, with `inherit`, from its standard input, a socket that a clone's application bound to one; `test DIRECTORY` runs synthetic procfs snapshots | -| `nvx-time [--report-only] COMMAND` | Enforces NVX time ABI v1: runs its conformance checks, repairs the wall clock after a restore, publishes the time state, and runs the discipline and violation-watcher daemon; `test NAME ...` runs its decoder and process test hooks (x86_64) | +| `nvx-time [--report-only] COMMAND` | Enforces NVX time ABI v2: runs its conformance checks, repairs the wall clock after a restore, publishes the time state, and runs the discipline and violation-watcher daemon; `test NAME ...` runs its decoder and process test hooks (x86_64) | | `nvx-time-probe latency\|warp\|clock ...` | Diagnoses clock-read latency, cross-CPU TSC warp and offset, the clocks, CPUID, clock events, kernel messages, and optionally MSRs (x86_64) | Each binary's command line, exit statuses, and byte output are compatibility diff --git a/nvx-rs/build.rs b/nvx-rs/build.rs index 15036cf03..761294168 100644 --- a/nvx-rs/build.rs +++ b/nvx-rs/build.rs @@ -99,7 +99,7 @@ mod bundle { .unwrap_or_else(|error| fail(format!("{MANIFEST} is not valid JSON: {error}"))); if !crate::contract::manifest_is_compatible(&manifest) { fail(format!( - "{} does not declare the {} control contract", + "{} does not declare the compatible OpenVMM contracts ({})", source.join(MANIFEST).display(), crate::contract::CONTROL_CONTRACT_REVISION )); diff --git a/nvx-rs/src/contract.rs b/nvx-rs/src/contract.rs index fa52b03bd..5c12bdcab 100644 --- a/nvx-rs/src/contract.rs +++ b/nvx-rs/src/contract.rs @@ -22,8 +22,8 @@ //! - [`CONTROL_CONTRACT_REVISION`] is the revision of the control contract that the session //! carries: the requests, responses, and features of the guest agent. It changes with them. //! -//! A release declares the microVM ABI version, the control session version, and the control -//! contract revision of its artifacts in its `SOURCE-MANIFEST.json`. +//! A release declares the microVM ABI, time ABI, restore-packet, control-session, and control +//! contract versions of its artifacts in its `SOURCE-MANIFEST.json`. //! //! The compatibility key of a template, `nvx::host::template::TemplateKey`, holds the boot and //! control versions, and the versions of the contracts through which a template's clones are @@ -70,11 +70,16 @@ pub const DEVICE_STATE_SCHEMA_VERSION: u64 = 1; /// Version of the time ABI that this crate speaks: the guest's clocks and timers, and their repair /// after a restore. It must match `TIME_ABI_VERSION` in OpenVMM's /// `vmm_core/virt/src/time_abi/mod.rs`. -pub const TIME_ABI_VERSION: u64 = 1; +/// +/// Version 2 pairs each time sample's host UTC with the guest TSC on the host, and publishes the +/// host's clock correction with it, which the guest applies instead of estimating it. +pub const TIME_ABI_VERSION: u64 = 2; /// Version of the restore packet that OpenVMM gives a restored guest, and that this crate's guest /// decodes. It must match `RESTORE_PACKET_VERSION` in OpenVMM's /// `vm/devices/chipset_resources/src/microvm_time.rs`. -pub const RESTORE_PACKET_VERSION: u64 = 4; +/// +/// Version 5 carries no host UTC: the guest takes a time sample to set its clock. +pub const RESTORE_PACKET_VERSION: u64 = 5; /// Returns whether the `openvmm` section of an NVX `SOURCE-MANIFEST.json` declares this contract. #[cfg(feature = "host")] @@ -85,8 +90,43 @@ pub fn manifest_is_compatible(manifest: &serde_json::Value) -> bool { .and_then(|section| section.get(name)) }; field("microvm_abi_version").and_then(serde_json::Value::as_u64) == Some(MICROVM_ABI_VERSION) + && field("time_abi_version").and_then(serde_json::Value::as_u64) == Some(TIME_ABI_VERSION) + && field("restore_packet_version").and_then(serde_json::Value::as_u64) + == Some(RESTORE_PACKET_VERSION) && field("control_session_protocol_version").and_then(serde_json::Value::as_u64) == Some(CONTROL_SESSION_PROTOCOL_VERSION) && field("control_contract_revision").and_then(serde_json::Value::as_str) == Some(CONTROL_CONTRACT_REVISION) } + +#[cfg(all(test, feature = "host"))] +mod tests { + use serde_json::json; + + use super::*; + + #[test] + fn release_manifests_must_match_every_bundled_contract() { + let manifest = json!({ + "openvmm": { + "microvm_abi_version": MICROVM_ABI_VERSION, + "time_abi_version": TIME_ABI_VERSION, + "restore_packet_version": RESTORE_PACKET_VERSION, + "control_session_protocol_version": CONTROL_SESSION_PROTOCOL_VERSION, + "control_contract_revision": CONTROL_CONTRACT_REVISION, + } + }); + assert!(manifest_is_compatible(&manifest)); + for field in [ + "microvm_abi_version", + "time_abi_version", + "restore_packet_version", + "control_session_protocol_version", + "control_contract_revision", + ] { + let mut incompatible = manifest.clone(); + incompatible["openvmm"][field] = serde_json::Value::Null; + assert!(!manifest_is_compatible(&incompatible), "{field}"); + } + } +} diff --git a/nvx-rs/src/guest/sys/discipline.rs b/nvx-rs/src/guest/sys/discipline.rs index e8109373c..a0f71cc0f 100644 --- a/nvx-rs/src/guest/sys/discipline.rs +++ b/nvx-rs/src/guest/sys/discipline.rs @@ -9,8 +9,9 @@ use nix::libc; use super::Errno; /// The discipline status of the frequency and slew adjustments: the phase-locked loop, with -/// offsets in nanoseconds rather than microseconds. -const PLL_NANOSECONDS: i32 = libc::STA_PLL | libc::STA_NANO; +/// offsets in nanoseconds rather than microseconds, and with its frequency held, so that it slews +/// only the phase and keeps the frequency correction that the discipline sets from the host's. +const PLL_NANOSECONDS: i32 = libc::STA_PLL | libc::STA_NANO | libc::STA_FREQHOLD; /// A change to the wall clock's discipline, or a query of it. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -24,40 +25,45 @@ pub(crate) enum ClockAdjustment { /// Nonnegative nanoseconds that complete the step. nanoseconds: i64, }, - /// Sets the frequency correction and selects the phase-locked loop in nanosecond mode. - Frequency { - /// Frequency correction in parts per million, scaled by 2^16. - scaled_ppm: i64, - }, - /// Slews the wall clock by `offset_ns` through the phase-locked loop in nanosecond mode, and - /// sets its time constant and error estimates. + /// Sets the frequency correction and the error estimates, and selects the phase-locked loop + /// in nanosecond mode with its frequency held. + Frequency(FrequencyHold), + /// Slews the wall clock by `offset_ns` through the phase-locked loop in nanosecond mode, with + /// its frequency held, and sets its time constant, the frequency correction, and the error + /// estimates. Slew { /// Phase offset in nanoseconds. offset_ns: i64, /// Time constant of the phase-locked loop. time_constant: i64, - /// Maximum error in microseconds. - maximum_error_us: i64, - /// Estimated error in microseconds. - estimated_error_us: i64, + /// The frequency correction and the error estimates. + hold: FrequencyHold, }, } +/// The frequency correction that the discipline holds, and the error estimates that it sets with +/// it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct FrequencyHold { + /// Frequency correction in parts per million, scaled by 2^16. + pub(crate) scaled_ppm: i64, + /// Maximum error in microseconds. + pub(crate) maximum_error_us: i64, + /// Estimated error in microseconds. + pub(crate) estimated_error_us: i64, +} + impl ClockAdjustment { /// Returns the `modes` bits with which `adjtimex` applies the adjustment. pub(crate) fn modes(self) -> u32 { + /// The modes that set the held frequency and the error estimates. + const HOLD: u32 = + libc::ADJ_FREQUENCY | libc::ADJ_MAXERROR | libc::ADJ_ESTERROR | libc::ADJ_STATUS; match self { Self::Query => 0, Self::Step { .. } => libc::ADJ_SETOFFSET | libc::ADJ_NANO, - Self::Frequency { .. } => libc::ADJ_FREQUENCY | libc::ADJ_STATUS, - Self::Slew { .. } => { - libc::ADJ_OFFSET - | libc::ADJ_MAXERROR - | libc::ADJ_ESTERROR - | libc::ADJ_STATUS - | libc::ADJ_TIMECONST - | libc::ADJ_NANO - } + Self::Frequency(_) => HOLD, + Self::Slew { .. } => HOLD | libc::ADJ_OFFSET | libc::ADJ_TIMECONST | libc::ADJ_NANO, } } @@ -65,7 +71,7 @@ impl ClockAdjustment { pub(crate) fn status(self) -> i32 { match self { Self::Query | Self::Step { .. } => 0, - Self::Frequency { .. } | Self::Slew { .. } => PLL_NANOSECONDS, + Self::Frequency(_) | Self::Slew { .. } => PLL_NANOSECONDS, } } } @@ -77,6 +83,15 @@ pub(crate) struct ClockDiscipline { pub(crate) scaled_ppm: i64, } +impl FrequencyHold { + /// Writes the frequency correction and the error estimates to `value`. + fn apply(self, value: &mut libc::timex) { + value.freq = self.scaled_ppm; + value.maxerror = self.maximum_error_us; + value.esterror = self.estimated_error_us; + } +} + /// Applies `adjustment` to the wall clock's discipline, which needs `CAP_SYS_TIME` for any /// adjustment but a query, and returns the discipline that results. pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result { @@ -96,17 +111,15 @@ pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result value.freq = scaled_ppm, + ClockAdjustment::Frequency(hold) => hold.apply(&mut value), ClockAdjustment::Slew { offset_ns, time_constant, - maximum_error_us, - estimated_error_us, + hold, } => { value.offset = offset_ns; value.constant = time_constant; - value.maxerror = maximum_error_us; - value.esterror = estimated_error_us; + hold.apply(&mut value); } } // SAFETY: `value` is initialized, and writable for the kernel's reply, for the call. @@ -128,16 +141,21 @@ mod tests { seconds: -2, nanoseconds: 872_000_000, }; - let frequency = ClockAdjustment::Frequency { scaled_ppm: 65_536 }; + let hold = FrequencyHold { + scaled_ppm: 65_536, + maximum_error_us: 2, + estimated_error_us: 1, + }; + let frequency = ClockAdjustment::Frequency(hold); let slew = ClockAdjustment::Slew { offset_ns: 5, time_constant: 4, - maximum_error_us: 1, - estimated_error_us: 1, + hold, }; assert_eq!((step.modes(), step.status()), (0x2100, 0)); - assert_eq!((frequency.modes(), frequency.status()), (0x0012, 0x2001)); - assert_eq!((slew.modes(), slew.status()), (0x203d, 0x2001)); + // The status holds the frequency, `STA_FREQHOLD`, with the loop and nanoseconds. + assert_eq!((frequency.modes(), frequency.status()), (0x001e, 0x2081)); + assert_eq!((slew.modes(), slew.status()), (0x203f, 0x2081)); assert_eq!( ( ClockAdjustment::Query.modes(), @@ -147,6 +165,22 @@ mod tests { ); } + #[test] + fn a_hold_writes_the_frequency_and_the_error_estimates() { + // SAFETY: all zeros is the kernel's baseline for every field of `timex`. + let mut value: libc::timex = unsafe { std::mem::zeroed() }; + FrequencyHold { + scaled_ppm: -13_107_200, + maximum_error_us: 70, + estimated_error_us: 3, + } + .apply(&mut value); + assert_eq!( + (value.freq, value.maxerror, value.esterror), + (-13_107_200, 70, 3) + ); + } + #[test] fn a_query_reports_the_frequency_without_privileges() { assert!(adjust_clock(ClockAdjustment::Query).is_ok()); diff --git a/nvx-rs/src/guest/sys/mod.rs b/nvx-rs/src/guest/sys/mod.rs index aa1ad7164..802c4a640 100644 --- a/nvx-rs/src/guest/sys/mod.rs +++ b/nvx-rs/src/guest/sys/mod.rs @@ -88,7 +88,7 @@ pub(crate) use nix::sys::time::{TimeSpec, TimeValLike}; pub(crate) use clock::{Clock, clock_time, monotonic_ms, sleep_once, sleep_until_monotonic}; #[cfg(target_arch = "x86_64")] pub(crate) use clock::{monotonic_ns, syscall_monotonic_nanoseconds}; -pub(crate) use discipline::{ClockAdjustment, ClockDiscipline, adjust_clock}; +pub(crate) use discipline::{ClockAdjustment, ClockDiscipline, FrequencyHold, adjust_clock}; #[cfg(test)] pub(crate) use entry::Simulation; pub(crate) use entry::{EntryChild, file_size}; diff --git a/nvx-rs/src/guest/time/api.rs b/nvx-rs/src/guest/time/api.rs index 8535289fa..19c420302 100644 --- a/nvx-rs/src/guest/time/api.rs +++ b/nvx-rs/src/guest/time/api.rs @@ -11,14 +11,13 @@ use super::cpu_mask::CpuMask; use super::daemon; use super::decoders::{self, PACKET_HEADER_SIZE, PacketError, PacketHeader, RestorePacket}; use super::events::{CheckId, ViolationCode}; -use super::state::{ - CheckState, CheckStatus, Discontinuity, DowntimeSource, SampleRejection, TimeState, -}; +use super::state::{CheckState, CheckStatus, Discontinuity, DowntimeSource, TimeState}; use super::system::{ - self, PORTB_GENERATION_ID_AVAILABLE, PORTB_PACKET_AVAILABLE, PRIVATE_FILE_MODE, PowerOffStatus, - RESTORE_PATH, RESTORE_TEMP_PATH, RealOps, Runtime, SELECT_GENERATION_ID, SELECT_PACKET, - SampleError, SaveError, SpawnError, clock_ns, cpu_time_ns, current_frequency, fork_process, - initial_sample, portb_read, set_frequency, state_apply, state_load, state_store, step_clock, + self, BestSample, PORTB_GENERATION_ID_AVAILABLE, PORTB_PACKET_AVAILABLE, PRIVATE_FILE_MODE, + PowerOffStatus, RESTORE_PATH, RESTORE_TEMP_PATH, RealOps, Runtime, SAMPLE_ATTEMPTS, + SAMPLE_BOUND_NS, SELECT_GENERATION_ID, SELECT_PACKET, SampleError, SampleResult, SaveError, + SpawnError, clock_ns, cpu_time_ns, current_frequency, fork_process, initial_sample, portb_read, + state_apply, state_load, state_store, }; use super::{NSEC_PER_USEC, Phase}; use crate::guest::GenerationId; @@ -28,17 +27,13 @@ use crate::guest::sys::{ }; use crate::protocol::cmdline::{self, Token}; -/// Maximum accepted restore pairing uncertainty. -const REPAIR_BOUND_NS: i64 = 1_000_000; -/// Samples that repair takes when the packet's pairing is too uncertain. -const REPAIR_SAMPLE_ATTEMPTS: usize = 2; /// The kernel command-line token that makes the time ABI report-only, `nvx_time_abi=report-only`. const REPORT_ONLY_TOKEN: Token<'static> = Token { name: b"nvx_time_abi", value: Some(b"report-only"), }; -/// The guest side of NVX time ABI version one: its cold-boot time step and the start of its time +/// The guest side of NVX time ABI version two: its cold-boot time step and the start of its time /// daemon, its VM generation ID, the capture request with the repair of a restore, and the /// restore's finish, which run in the calling process. /// @@ -127,13 +122,15 @@ impl TimeAbi { } /// Runs the cold-boot time step synchronously: takes the initial host time sample, steps - /// the wall clock to it, and publishes the time state, which starts at generation zero. + /// the wall clock to it, holds the frequency correction that the host's clock correction + /// asks for, and publishes the time state, which starts at generation zero. /// - /// A sample that cannot be taken is a violation of check C12, which does not fail the call - /// in report-only mode: the step then leaves the wall clock unsynchronized, which - /// [`ColdBoot::synchronized`] tells. The step starts no process: - /// [`start_daemon`](Self::start_daemon) starts the time daemon from its outcome, - /// separately, which `nvx-time boot` does next. + /// The sample pairs the frequency MSR's declared TSC rate, which it reads through + /// `/dev/cpu/0/msr`, so devtmpfs must be mounted first. A sample that cannot be taken within + /// its bound is a violation of check C12, which does not fail the call in report-only mode: + /// the step then leaves the wall clock unsynchronized, which [`ColdBoot::synchronized`] + /// tells. The step starts no process: [`start_daemon`](Self::start_daemon) starts the time + /// daemon from its outcome, separately, which `nvx-time boot` does next. pub fn cold_boot(&self) -> Result { let runtime = self.runtime(); settled(&runtime, cold_boot(&runtime)) @@ -341,12 +338,10 @@ fn cold_boot(runtime: &Runtime<'_>) -> Result { }; let mut state = TimeState::default(); state.synchronized = initial.is_some(); - state.samples = u64::from(initial.is_some()); - if let Some((sample, _)) = initial { - state.offset_ns = sample.theta; - state.uncertainty_ns = sample.epsilon; + match &initial { + Some((sample, _)) => state.record_sample(sample), + None => state.frequency_ppb = decoders::frequency_to_ppb(current_frequency(runtime)), } - state.frequency_ppb = decoders::frequency_to_ppb(current_frequency(runtime)); if let Err(error) = state_store(&state) { return Err(check_failed( runtime, @@ -789,7 +784,7 @@ fn capture(runtime: &Runtime<'_>, request: CaptureRequest) -> Result, request: CaptureRequest) -> Result) { if let Ok(state) = state_load() { runtime.set_generation(state.generation as u32); + runtime.set_tsc_hz(state.tsc_hz); } } @@ -855,24 +852,21 @@ enum PacketRead { Stale, /// The packet's generation, which does not follow the recorded generation. Generation(u32), - /// The packet, or why its body is invalid, and the realtime clock around its selection. - Packet { - /// Realtime before the selection. - t0: i64, - /// Realtime after the selection. - t1: i64, - /// The packet, or why its body is invalid. - packet: Result, - }, + /// The packet, with the time sample of its generation that repair took right after it, in + /// the same transaction, or why it could take none within the bound; or why the packet's + /// body is invalid. + Packet(Result<(RestorePacket, SampleRead), PacketError>), } +/// The time sample that repair selected, or why it could select none. +type SampleRead = Result<(SampleResult, BestSample), SampleError>; + /// Repair step 2's port reads, after the capture read the status, step 1: selects the restore /// packet, and reads its header and, when the header is valid and follows the `recorded` -/// generation, its body, with four-byte reads. -fn read_packet(ports: &PortAccess<'_>, recorded: u32) -> PacketRead { - let t0 = clock_ns(sys::Clock::Realtime); +/// generation, its body, with four-byte reads, and then, for a valid packet, takes the time +/// sample of the packet's generation, to which repair steps the wall clock. +fn read_packet(runtime: &Runtime<'_>, ports: &PortAccess<'_>, recorded: u32) -> PacketRead { ports.write_u8(Port::PortbStatus, SELECT_PACKET); - let t1 = clock_ns(sys::Clock::Realtime); let mut header = [0_u8; PACKET_HEADER_SIZE]; portb_read(ports, Port::PortbData, &mut header); let header = match PacketHeader::parse(&header) { @@ -890,7 +884,19 @@ fn read_packet(ports: &PortAccess<'_>, recorded: u32) -> PacketRead { let packet = RestorePacket::parse(header, &body); // The packet owns its entropy now, in the seed's allocation. sys::zeroize(&mut body); - PacketRead::Packet { t0, t1, packet } + PacketRead::Packet(packet.map(|packet| { + let sample = system::clock_rates(runtime).and_then(|rates| { + system::select_sample( + ports, + runtime.ops, + SAMPLE_ATTEMPTS, + SAMPLE_BOUND_NS, + header.generation, + rates, + ) + }); + (packet, sample) + })) } /// What a restore transaction writes, and where. @@ -905,15 +911,16 @@ struct RepairTarget { daemon_pid: Option, } -/// Repair steps 2 and 3, once the port transaction that read the packet ended: validates it, -/// steps and slews the clock, hands over its entropy, and records the restore. +/// Repair steps 2 and 3, once the port transaction that read the packet and its time sample +/// ended: validates them, steps the clock to the sample and holds the frequency correction that +/// its host clock correction asks for, hands over the packet's entropy, and records the restore. fn repair_restore( runtime: &Runtime<'_>, read: PacketRead, target: RepairTarget, start_ns: i64, ) -> Result { - let (t0, t1, packet) = match read { + let packet = match read { PacketRead::InvalidHeader(error) => { return Err(repair_failed( runtime, @@ -936,9 +943,9 @@ fn repair_restore( ), )); } - PacketRead::Packet { t0, t1, packet } => (t0, t1, packet), + PacketRead::Packet(packet) => packet, }; - let packet = packet + let (packet, sample) = packet .map_err(|error| repair_failed(runtime, ViolationCode::RepairPacket, error.to_string()))?; let RestorePacket { header, @@ -954,38 +961,31 @@ fn repair_restore( )); } runtime.set_generation(header.generation); - let (mut theta, mut epsilon) = decoders::pair_clocks(t0, t1, header.utc_ns); - if epsilon > REPAIR_BOUND_NS { - match system::best_sample( - runtime, - REPAIR_SAMPLE_ATTEMPTS, - REPAIR_BOUND_NS, - header.generation, - ) { - Err(error @ SampleError::Generation { .. }) => { - return Err(repair_failed( - runtime, - ViolationCode::RepairGeneration, - error.to_string(), - )); - } - Ok((_, sample)) if sample.epsilon < epsilon => { - theta = sample.theta; - epsilon = sample.epsilon; - } - _ => {} + let sample = match sample { + Ok((SampleResult::Accepted, sample)) => sample, + Ok((SampleResult::Wide, sample)) => { + return Err(repair_failed( + runtime, + ViolationCode::RepairSample, + system::wide_sample(sample.epsilon), + )); } - if epsilon > REPAIR_BOUND_NS { + Err(error @ SampleError::Generation { .. }) => { + return Err(repair_failed( + runtime, + ViolationCode::RepairGeneration, + error.to_string(), + )); + } + Err(error) => { return Err(repair_failed( runtime, ViolationCode::RepairSample, - format!("UTC pairing uncertainty {epsilon} ns exceeds 1 ms"), + error.to_string(), )); } - } - let frequency = decoders::restore_frequency(header.rate_deviation); - if let Err(error) = step_clock(runtime, theta).and_then(|()| set_frequency(runtime, frequency)) - { + }; + if let Err(error) = system::synchronize(runtime, &sample) { return Err(repair_failed( runtime, ViolationCode::RepairClock, @@ -1004,15 +1004,7 @@ fn repair_restore( Err(detail) => return Err(repair_failed(runtime, ViolationCode::RepairPacket, detail)), }; if let Err(error) = state_apply(|state| { - apply_repair( - state, - &header, - theta, - epsilon, - step_realtime_ns, - frequency, - elapsed_us, - ); + apply_repair(state, &header, &sample, step_realtime_ns, elapsed_us); }) { return Err(repair_failed( runtime, @@ -1044,20 +1036,19 @@ fn repair_restore( })) } -/// Applies one successful restore packet to persistent state. +/// Applies one successful restore packet, and the time sample to which repair stepped the wall +/// clock, to persistent state. fn apply_repair( state: &mut TimeState, packet: &PacketHeader, - theta: i64, - epsilon: i64, + sample: &BestSample, step_realtime_ns: i64, - frequency: i64, elapsed_us: i64, ) { state.generation = u64::from(packet.generation); state.discontinuities += 1; state.last_discontinuity = Discontinuity::Restore; - state.last_step_ns = theta; + state.last_step_ns = sample.theta; state.last_step_realtime_ns = step_realtime_ns; state.last_downtime_ns = packet.downtime_ns; state.last_downtime_source = if packet.downtime_from_utc() { @@ -1066,11 +1057,7 @@ fn apply_repair( DowntimeSource::Monotonic }; state.synchronized = true; - state.offset_ns = theta; - state.uncertainty_ns = epsilon; - state.frequency_ppb = decoders::frequency_to_ppb(frequency); - state.samples += 1; - state.last_sample_error = SampleRejection::None; + state.record_sample(sample); // Repair step 3 marks this restore's check pending, so `nvx-time status` waits for the // daemon's step 7. Until then, `restore_elapsed_us` is the readiness path up to the clock // step, at `last_step_realtime_ns`; `restore_cpu_us` counts only step 7, which records it. @@ -1085,7 +1072,7 @@ fn apply_repair( } } -/// A set of the CPUs that time ABI version one covers: CPUs 0 to 63. +/// A set of the CPUs that the time ABI covers: CPUs 0 to 63. /// /// Its `Display` form is a CPU list in the kernel's syntax, such as `1-3,6`, or `none` for the /// empty set. @@ -1428,7 +1415,11 @@ mod tests { use std::sync::atomic::Ordering; use super::*; - use crate::guest::time::system::tests::FakeOps; + use crate::guest::sys::{ClockAdjustment, FrequencyHold}; + use crate::guest::time::decoders::HostClock; + use crate::guest::time::state::HostClockStatus; + use crate::guest::time::system::PORTB_TIME_WINDOW; + use crate::guest::time::system::tests::{FakeOps, sample_input, sample_tsc}; use crate::protocol::status; /// Returns whether `error` holds the error number `errno`. @@ -1847,19 +1838,27 @@ mod tests { /// Returns the port input with which a restore answers a capture: port B's status, then a /// packet of `generation`, which carries `entropy` and nothing to activate, in four-byte - /// reads. + /// reads, and then port B's status, with the time window, and a time sample of the + /// generation, within the bound. fn restore_input(generation: u32, entropy: &[u8; Seed::LENGTH]) -> Vec { let mut packet = [0_u8; PACKET_HEADER_SIZE + Seed::LENGTH]; - packet[..4].copy_from_slice(b"OVR\x04"); + packet[..4].copy_from_slice(b"OVR\x05"); packet[8..12].copy_from_slice(&generation.to_le_bytes()); - let utc_ns = clock_ns(sys::Clock::Realtime).unsigned_abs(); - packet[24..32].copy_from_slice(&utc_ns.to_le_bytes()); packet[PACKET_HEADER_SIZE..].copy_from_slice(entropy); let words = packet .chunks(size_of::()) .map(|word| u32::from_ne_bytes(word.try_into().unwrap())); + let sample = sample_input( + generation, + FakeOps::REALTIME_NS as u64, + sample_tsc(0), + 400, + HostClock::default(), + ); std::iter::once(u32::from(PORTB_PACKET_AVAILABLE)) .chain(words) + .chain(std::iter::once(u32::from(PORTB_TIME_WINDOW))) + .chain(sample) .collect() } @@ -1995,6 +1994,73 @@ mod tests { assert!(!boot.synchronized()); assert_eq!(boot.failures, 1); assert!(!state_load().unwrap().synchronized); + // A sample beyond the bound fails check C12 too. + let host = HostClock { + status: HostClockStatus::FreeRunning, + disruption: 0, + correction: 13_107_200, + correction_bound: None, + max_error_us: Some(16_000_000), + }; + let utc_ns = FakeOps::REALTIME_NS as u64 + 1_000; + let window = |pairing_ns| { + std::iter::once(u32::from(PORTB_TIME_WINDOW)) + .chain(sample_input(0, utc_ns, sample_tsc(0), pairing_ns, host)) + .collect::>() + }; + let ops = FakeOps::new(window(150_000)); + let boot = super::cold_boot(&Runtime::new(&ops, true)).unwrap(); + assert_eq!((boot.synchronized(), boot.failures), (false, 1)); + let console = String::from_utf8(ops.console.lock().unwrap().clone()).unwrap(); + assert!( + console.contains("time sample uncertainty 151026 ns exceeds 100000 ns"), + "{console}" + ); + // A sample within the bound steps the wall clock to host UTC, holds the frequency that + // the host's correction asks for, and publishes the host's discipline. A free-running + // host's error does not count. + let ops = FakeOps::new(window(400)); + let boot = super::cold_boot(&Runtime::new(&ops, false).strict()).unwrap(); + assert_eq!((boot.synchronized(), boot.failures), (true, 0)); + assert_eq!( + ops.adjustments.lock().unwrap().as_slice(), + [ + ClockAdjustment::Query, + ClockAdjustment::Step { + seconds: 0, + nanoseconds: 1_050, + }, + ClockAdjustment::Frequency(FrequencyHold { + scaled_ppm: 13_107_200, + maximum_error_us: 2, + estimated_error_us: 2, + }), + ] + ); + let state = state_load().unwrap(); + assert!(state.synchronized); + assert_eq!( + (state.samples, state.offset_ns, state.uncertainty_ns), + (1, 1_050, 1_426) + ); + assert_eq!( + (state.frequency_ppb, state.frequency_limited), + (200_000, false) + ); + assert_eq!( + ( + state.host_status, + state.host_correction_ppb, + state.host_correction_bound_ppb, + state.host_max_error_us + ), + ( + HostClockStatus::FreeRunning, + 200_000, + None, + Some(16_000_000) + ) + ); } #[test] diff --git a/nvx-rs/src/guest/time/checks.rs b/nvx-rs/src/guest/time/checks.rs index 718b141a8..e8fa912e0 100644 --- a/nvx-rs/src/guest/time/checks.rs +++ b/nvx-rs/src/guest/time/checks.rs @@ -42,9 +42,12 @@ const HYPERV_VENDOR: [u32; 3] = [ const HYPERV_MAXIMUM_LEAF: u32 = cpuid::LEAF_HYPERVISOR + 5; /// The interface signature of Hyper-V's CPUID leaf 0x40000001, "Hv#1". const HYPERV_INTERFACE: u32 = u32::from_le_bytes(*b"Hv#1"); -/// The build of leaf 0x40000002, OpenVMM's NVX build: "NVX", then version 1.0 as the major -/// version in the upper half of EBX. -const HYPERV_BUILD: [u32; 2] = [u32::from_le_bytes(*b"NVX\0"), 1 << 16]; +/// The build of leaf 0x40000002, OpenVMM's NVX build: "NVX", then the time ABI version as the +/// major version in the upper half of EBX, version 2.0. +const HYPERV_BUILD: [u32; 2] = [ + u32::from_le_bytes(*b"NVX\0"), + (crate::contract::TIME_ABI_VERSION as u32) << 16, +]; /// The partition privileges of leaf 0x40000003's EAX: the hypercall, VP-index, frequency, and /// TSC invariant-control MSRs. const HYPERV_PRIVILEGES: u32 = (1 << cpuid::HYPERV_EAX_HYPERCALL) @@ -1291,7 +1294,7 @@ mod tests { assert_eq!(HYPERV_MAXIMUM_LEAF, 0x4000_0005); assert_eq!(HYPERV_VENDOR, [0x7263_694d, 0x666f_736f, 0x7648_2074]); assert_eq!(HYPERV_INTERFACE, 0x3123_7648); - assert_eq!(HYPERV_BUILD, [0x0058_564e, 0x0001_0000]); + assert_eq!(HYPERV_BUILD, [0x0058_564e, 0x0002_0000]); assert_eq!(HYPERV_PRIVILEGES, 0x8860); assert_eq!(HYPERV_FEATURES, 0x100); } diff --git a/nvx-rs/src/guest/time/commands.rs b/nvx-rs/src/guest/time/commands.rs index 44e9e0057..a38fee5d5 100644 --- a/nvx-rs/src/guest/time/commands.rs +++ b/nvx-rs/src/guest/time/commands.rs @@ -292,13 +292,18 @@ fn cmd_status(runtime: &Runtime<'_>, out: &mut dyn Write) -> Result<(), CommandE } } -/// Prints one raw host time sample and its derived pairing. +/// Prints one raw host time sample, its pairing with the guest's clocks, and the frequency +/// correction that its host clock correction asks for. fn cmd_sample( runtime: &Runtime<'_>, out: &mut dyn Write, err: &mut dyn Write, ) -> Result<(), CommandError> { - let sample = { + let rates = system::clock_rates(runtime).map_err(|error| { + report(err, &[format!("nvx-time: {error}\n").as_bytes()]); + CommandError::Operation + })?; + let taken = { let ports = runtime .open_ports() .map_err(|error| port_failure(err, error))?; @@ -320,19 +325,46 @@ fn cmd_sample( ); CommandError::Operation })?; - system::take_sample(&ports) + system::take_sample(&ports, runtime.ops) }; - let sample = sample.map_err(|error| { + let (sample, reads) = taken.map_err(|error| { report(err, &[format!("nvx-time: {error}\n").as_bytes()]); CommandError::Operation })?; - let (theta, epsilon) = decoders::pair_clocks(sample.t0, sample.t1, sample.utc_ns); - let line = format!( - "flags=0x{:02x} generation={} utc_ns={} t0={} t1={} theta_ns={} epsilon_ns={}\n", - sample.flags, sample.generation, sample.utc_ns, sample.t0, sample.t1, theta, epsilon + let host = &sample.host; + let bound = |bound: Option| bound.map_or_else(|| "unknown".to_owned(), |b| b.to_string()); + let mut line = format!( + "flags=0x{:02x} generation={} status={} utc_ns={} tsc={} pairing_ns={} disruption={} \ + correction={} correction_bound={} max_error_us={} before={} after={} realtime_ns={}", + sample.flags, + sample.generation, + host.status, + sample.utc_ns, + sample.tsc, + sample.pairing_ns, + host.disruption, + host.correction, + bound(host.correction_bound), + bound(host.max_error_us), + reads.before, + reads.after, + reads.realtime_ns ); + let pairing = decoders::pair_sample(&sample, &reads, rates); + if let Ok((theta, epsilon)) = pairing { + let (frequency, limited) = decoders::guest_frequency(host.correction, rates.tsc_hz); + let _ = write!( + line, + " theta_ns={theta} epsilon_ns={epsilon} frequency={frequency} frequency_limited={}", + u8::from(limited) + ); + } + line.push('\n'); let _ = out.write_all(line.as_bytes()); - Ok(()) + pairing.map(drop).map_err(|error| { + report(err, &[format!("nvx-time: {error}\n").as_bytes()]); + CommandError::Operation + }) } /// Reads and prints the VM generation ID. diff --git a/nvx-rs/src/guest/time/cpu_mask.rs b/nvx-rs/src/guest/time/cpu_mask.rs index 7539c83c2..f41264e2c 100644 --- a/nvx-rs/src/guest/time/cpu_mask.rs +++ b/nvx-rs/src/guest/time/cpu_mask.rs @@ -8,7 +8,7 @@ use crate::guest::kernel_text; // Each CPU that the time ABI covers is one bit of the mask. const _: () = assert!(MAX_CPUS == u64::BITS as usize); -/// A set of CPUs below `MAX_CPUS`, the CPUs that time ABI version one covers. +/// A set of CPUs below `MAX_CPUS`, the CPUs that the time ABI covers. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] pub(crate) struct CpuMask(u64); diff --git a/nvx-rs/src/guest/time/daemon.rs b/nvx-rs/src/guest/time/daemon.rs index aa82f1e7a..ae85f204a 100644 --- a/nvx-rs/src/guest/time/daemon.rs +++ b/nvx-rs/src/guest/time/daemon.rs @@ -13,10 +13,10 @@ use super::decoders; use super::events::{CheckId, EventSource, ViolationCode}; use super::state::{Discontinuity, SampleRejection, TimeState}; use super::system::{ - self, DAEMON_NAME, DAEMON_PID_PATH, DAEMON_PID_TEMP_PATH, DEFAULT_STALL_TIMEOUT_S, - PowerOffStatus, RCU_SYNC_PATH, RESTORE_PATH, Runtime, SNAPSHOT_ACKNOWLEDGE, SampleError, - SampleResult, SpawnError, clock_ns, cpu_time_ns, current_frequency, fork_process, - set_frequency, state_apply, state_load, suppression_restore, + self, BestSample, DAEMON_NAME, DAEMON_PID_PATH, DAEMON_PID_TEMP_PATH, DEFAULT_STALL_TIMEOUT_S, + PowerOffStatus, RCU_SYNC_PATH, RESTORE_PATH, Runtime, SAMPLE_ATTEMPTS, SAMPLE_BOUND_NS, + SNAPSHOT_ACKNOWLEDGE, SampleError, SampleResult, SpawnError, clock_ns, cpu_time_ns, + fork_process, state_apply, state_load, suppression_restore, }; use super::{DEFERRED_START_NS, IDLE_BOUND_NS, NSEC_PER_MSEC, NSEC_PER_SEC, NSEC_PER_USEC, Phase}; use crate::guest::kernel_text::KmsgRecord; @@ -25,10 +25,6 @@ use crate::guest::sys::{ PortAccess, PortError, ProcessId, Signal, SignalDescriptor, Timer, kernel, }; -/// Maximum accepted polling bracket width. -const POLL_BOUND_NS: i64 = 50_000; -/// Samples attempted by each discipline poll. -const SAMPLE_ATTEMPTS: usize = 3; /// Maximum age of the latest accepted sample while synchronized. const SYNCHRONIZED_AGE_NS: i64 = 128 * NSEC_PER_SEC; /// Poll interval during initial convergence. @@ -134,48 +130,45 @@ fn kmsg_drain(log: &FileDescriptor, mut consume: impl FnMut(&[u8])) -> Result<() /// What one discipline poll did with its time sample. enum PollOutcome { - /// It rejected the sample, or could not take or apply one. - Rejected, + /// It rejected the sample, or could not take or apply one, for this reason. + Rejected(SampleRejection), /// It accepted the sample and corrected the wall clock. Accepted { - /// Accepted clock offset in nanoseconds. - theta: i64, - /// Accepted uncertainty in nanoseconds. - epsilon: i64, - /// The frequency correction afterwards, in scaled parts per million. - frequency: i64, + /// The accepted sample, with the frequency correction that the discipline holds. + sample: BestSample, /// The wall clock right after a step, or `None` for a slew. step_realtime_ns: Option, }, } -/// Applies one poll's outcome, and whether the discipline is synchronized, to the time state. -fn apply_poll(state: &mut TimeState, outcome: &PollOutcome, synchronized: bool) { +/// Applies one poll's outcome, and whether the discipline is synchronized, to the time state, +/// and returns whether the accepted sample's disruption counter differs from the last accepted +/// sample's: host UTC stepped, the host resumed from a suspension, or the guest TSC's offset +/// from the host's changed. +fn apply_poll(state: &mut TimeState, outcome: &PollOutcome, synchronized: bool) -> bool { + let mut disrupted = false; match *outcome { PollOutcome::Accepted { - theta, - epsilon, - frequency, + sample, step_realtime_ns, } => { - state.samples += 1; - state.offset_ns = theta; - state.uncertainty_ns = epsilon; - state.frequency_ppb = decoders::frequency_to_ppb(frequency); - state.last_sample_error = SampleRejection::None; + disrupted = + state.samples != 0 && state.host_disruption != u64::from(sample.host.disruption); + state.record_sample(&sample); if let Some(realtime_ns) = step_realtime_ns { state.discontinuities += 1; state.last_discontinuity = Discontinuity::Step; - state.last_step_ns = theta; + state.last_step_ns = sample.theta; state.last_step_realtime_ns = realtime_ns; } } - PollOutcome::Rejected => { + PollOutcome::Rejected(reason) => { state.rejected_samples += 1; - state.last_sample_error = SampleRejection::Uncertain; + state.last_sample_error = reason; } } state.synchronized = synchronized; + disrupted } /// Repair step 7 work that a queued signal scheduled, which starts once it is due. @@ -266,50 +259,67 @@ fn discipline_poll(runtime: &Runtime<'_>, daemon: &mut Daemon) { // The port-B lock serializes the sample with the other port transactions, and the state // update with the sample. let lock = system::lock_portb().ok(); - let mut outcome = PollOutcome::Rejected; + let mut outcome = PollOutcome::Rejected(SampleRejection::Invalid); if lock.is_some() { let sample = system::best_sample( runtime, SAMPLE_ATTEMPTS, - POLL_BOUND_NS, + SAMPLE_BOUND_NS, runtime.generation(), ); - match sample { - Ok((SampleResult::Accepted, sample)) => { - let frequency = current_frequency(runtime); - let correction = decoders::plan_correction(sample.theta, sample.epsilon, fast); - let stepped = matches!(correction, sys::ClockAdjustment::Step { .. }); - let applied = runtime.ops.adjust_clock(correction).is_ok() - && (!stepped || set_frequency(runtime, frequency).is_ok()); - let step_realtime_ns = clock_ns(sys::Clock::Realtime); - let frequency = current_frequency(runtime); - if applied { - daemon.last_accepted_ns = Some(clock_ns(sys::Clock::Monotonic)); - outcome = PollOutcome::Accepted { - theta: sample.theta, - epsilon: sample.epsilon, - frequency, - step_realtime_ns: stepped.then_some(step_realtime_ns), - }; - } - } + outcome = match sample { + Ok((SampleResult::Accepted, sample)) => correct_clock(runtime, sample, fast), + Ok((SampleResult::Wide, _)) => PollOutcome::Rejected(SampleRejection::Uncertain), Err(SampleError::Generation { .. }) => return, - // Nothing else would report it, and the daemon goes on without the sample. - Err(SampleError::Ports(error @ PortError::Revoke(_))) => { - runtime.log_release_failure(error); + Err(error) => { + // Nothing else would report it, and the daemon goes on without the sample. + if let SampleError::Ports(error @ PortError::Revoke(_)) = error { + runtime.log_release_failure(error); + } + PollOutcome::Rejected(error.rejection()) } - Ok((SampleResult::Wide, _)) - | Err(SampleError::Ports(_) | SampleError::NoTimeWindow(_) | SampleError::Invalid(_)) => - {} - } + }; + } + if matches!(outcome, PollOutcome::Accepted { .. }) { + daemon.last_accepted_ns = Some(clock_ns(sys::Clock::Monotonic)); } let synchronized = daemon.last_accepted_ns.is_some_and(|accepted_ns| { clock_ns(sys::Clock::Monotonic).wrapping_sub(accepted_ns) <= SYNCHRONIZED_AGE_NS }); - let _ = state_apply(|state| apply_poll(state, &outcome, synchronized)); + let mut disrupted = false; + let _ = state_apply(|state| disrupted = apply_poll(state, &outcome, synchronized)); + // A disruption restarts the discipline at the fast cadence, as a restore does. + if disrupted { + daemon.polls = 0; + } drop(lock); } +/// Corrects the wall clock to the accepted `sample`, which the discipline took at the fast or +/// slow cadence: steps it to host UTC and then holds the frequency correction that the host's +/// clock correction asks for, or slews it while holding that frequency. +fn correct_clock(runtime: &Runtime<'_>, sample: BestSample, fast: bool) -> PollOutcome { + let host_error_us = sample.host.error_us(); + let correction = decoders::plan_correction( + sample.theta, + sample.epsilon, + fast, + sample.frequency, + host_error_us, + ); + let stepped = matches!(correction, sys::ClockAdjustment::Step { .. }); + let hold = decoders::frequency_hold(0, sample.epsilon, sample.frequency, host_error_us); + let applied = runtime.ops.adjust_clock(correction).is_ok() + && (!stepped || system::hold_frequency(runtime, hold).is_ok()); + if !applied { + return PollOutcome::Rejected(SampleRejection::Invalid); + } + PollOutcome::Accepted { + sample, + step_realtime_ns: stepped.then(|| clock_ns(sys::Clock::Realtime)), + } +} + /// A kernel setting forced to a value while the owner lives. Dropping the owner restores the /// setting's saved value, best effort, unless [`restore`](Self::restore) did. struct ForcedSetting { @@ -676,7 +686,7 @@ fn handle_signals(runtime: &Runtime<'_>, daemon: &mut Daemon) { match event.signal { Some(Signal::SIGTERM) => sys::exit_immediately(0), // `sigqueue` carries the readiness path's wall time. Step 7 is due - // `DEFERRED_START_NS` later; the repair paired the clock with the packet's UTC now. + // `DEFERRED_START_NS` later; the repair stepped the clock to a time sample now. Some(Signal::SIGUSR1) => { let now = clock_ns(sys::Clock::Monotonic); daemon.last_accepted_ns = Some(now); @@ -1005,11 +1015,12 @@ pub(crate) fn empty_restore_timing() -> RestoreTiming { #[cfg(test)] mod tests { use super::*; + use crate::guest::sys::{ClockAdjustment, FrequencyHold}; use crate::guest::time::system::tests::FakeOps; #[test] fn daemon_timing_constants_match_the_abi() { - assert_eq!(POLL_BOUND_NS, 50_000); + assert_eq!((SAMPLE_BOUND_NS, SAMPLE_ATTEMPTS), (100_000, 3)); assert_eq!(SYNCHRONIZED_AGE_NS, 128_000_000_000); assert_eq!( (FAST_PERIOD.as_secs(), SLOW_PERIOD.as_secs(), FAST_POLLS), @@ -1017,6 +1028,111 @@ mod tests { ); } + /// A sample of the host clock whose disruption counter is `disruption`, 1.5 µs ahead of the + /// wall clock, and whose host correction asks for 100 ppm. + fn polled(disruption: u32) -> BestSample { + BestSample { + theta: 1_500, + epsilon: 900, + host: decoders::HostClock { + disruption, + correction: 6_553_600, + ..decoders::HostClock::default() + }, + frequency: 6_553_600, + limited: false, + } + } + + #[test] + fn a_poll_records_its_sample_and_reports_a_disruption() { + let mut state = TimeState::default(); + let accepted = |disruption, step_realtime_ns| PollOutcome::Accepted { + sample: polled(disruption), + step_realtime_ns, + }; + // The first accepted sample sets the disruption counter, and is no disruption. + assert!(!apply_poll(&mut state, &accepted(5, None), true)); + assert_eq!((state.samples, state.host_disruption), (1, 5)); + assert_eq!((state.offset_ns, state.frequency_ppb), (1_500, 100_000)); + assert!(!apply_poll(&mut state, &accepted(5, None), true)); + assert_eq!(state.discontinuities, 0); + // A changed counter is a disruption, and a step a discontinuity. + assert!(apply_poll(&mut state, &accepted(6, Some(42)), true)); + assert_eq!((state.host_disruption, state.discontinuities), (6, 1)); + assert_eq!(state.last_discontinuity, Discontinuity::Step); + assert_eq!( + (state.last_step_ns, state.last_step_realtime_ns), + (1_500, 42) + ); + // A rejection keeps the counter, and records its reason. + for reason in [ + SampleRejection::Uncertain, + SampleRejection::Tsc, + SampleRejection::Invalid, + ] { + assert!(!apply_poll( + &mut state, + &PollOutcome::Rejected(reason), + false + )); + assert_eq!(state.last_sample_error, reason); + } + assert_eq!( + (state.samples, state.rejected_samples, state.synchronized), + (3, 3, false) + ); + assert!(!apply_poll(&mut state, &accepted(6, None), true)); + assert_eq!(state.last_sample_error, SampleRejection::None); + } + + #[test] + fn a_correction_slews_or_steps_while_holding_the_host_frequency() { + let ops = FakeOps::new([]); + let runtime = Runtime::new(&ops, false); + let PollOutcome::Accepted { + step_realtime_ns: None, + .. + } = correct_clock(&runtime, polled(0), true) + else { + panic!("the slew was not accepted"); + }; + let far = BestSample { + theta: -1_128_000_000, + ..polled(0) + }; + let PollOutcome::Accepted { + step_realtime_ns: Some(_), + .. + } = correct_clock(&runtime, far, false) + else { + panic!("the step was not accepted"); + }; + let hold = FrequencyHold { + scaled_ppm: 6_553_600, + maximum_error_us: 1, + estimated_error_us: 1, + }; + assert_eq!( + ops.adjustments.lock().unwrap().as_slice(), + [ + ClockAdjustment::Slew { + offset_ns: 1_500, + time_constant: 4, + hold: FrequencyHold { + maximum_error_us: 3, + ..hold + }, + }, + ClockAdjustment::Step { + seconds: -2, + nanoseconds: 872_000_000, + }, + ClockAdjustment::Frequency(hold), + ] + ); + } + #[test] fn poll_deadlines_round_up_to_milliseconds() { let now = clock_ns(sys::Clock::Monotonic); diff --git a/nvx-rs/src/guest/time/decoders.rs b/nvx-rs/src/guest/time/decoders.rs index 3a0a94fea..6dd894290 100644 --- a/nvx-rs/src/guest/time/decoders.rs +++ b/nvx-rs/src/guest/time/decoders.rs @@ -5,18 +5,19 @@ use std::ops::RangeInclusive; use super::cpu_mask::CpuMask; use super::events::ViolationCode; +use super::state::HostClockStatus; use super::{MAX_CPUS, NSEC_PER_SEC, NSEC_PER_USEC}; use crate::guest::kernel_text::{ self, ClockEventState, SyslogLine, TickDevice, TickMode, TimerList, }; use crate::guest::parse::parse_decimal; use crate::guest::reseed::Seed; -use crate::guest::sys::ClockAdjustment; use crate::guest::sys::x86::{CpuidResult, cpuid}; +use crate::guest::sys::{ClockAdjustment, FrequencyHold}; use crate::protocol::cmdline::{self, Parameter}; /// Fixed restore-packet header length. -pub(crate) const PACKET_HEADER_SIZE: usize = 32; +pub(crate) const PACKET_HEADER_SIZE: usize = 24; /// Encoded length of one memory range. pub(crate) const PACKET_RANGE_SIZE: usize = 16; /// Entropy payload length appended to every restore packet: a restore's [`Seed`]. @@ -24,13 +25,13 @@ pub(crate) const PACKET_ENTROPY_SIZE: usize = 64; // The packet's entropy is decoded straight into a seed. const _: () = assert!(PACKET_ENTROPY_SIZE == Seed::LENGTH); /// Fixed host time-sample record length. -pub(crate) const SAMPLE_SIZE: usize = 16; +pub(crate) const SAMPLE_SIZE: usize = 48; /// VM generation identifier length. pub(crate) const GENERATION_ID_SIZE: usize = 16; /// The magic number that starts every restore packet. const PACKET_MAGIC: [u8; 3] = *b"OVR"; -/// The restore-packet version of time ABI version one, +/// The restore-packet version of time ABI version two, /// [`RESTORE_PACKET_VERSION`](crate::contract::RESTORE_PACKET_VERSION), which a template's /// compatibility key records. const PACKET_VERSION: u8 = { @@ -44,22 +45,35 @@ const PACKET_DOWNTIME_UTC: u8 = 0x01; pub(crate) const PACKET_MEMORY_TARGET: u8 = 0x02; /// Restore flag requiring a snapshot acknowledgement. pub(crate) const PACKET_ACK_REQUIRED: u8 = 0x04; -/// Mask of restore flags defined by ABI version one. +/// Mask of restore flags defined by the ABI. const PACKET_KNOWN_FLAGS: u8 = 0x0f; /// The online VP counts that a restore packet may give. const PACKET_ONLINE_VP_COUNTS: [u8; 5] = [0, 1, 2, 4, 8]; -/// The time-sample version of time ABI version one. -const SAMPLE_VERSION: u8 = 1; +/// The time-sample version of time ABI version two. +const SAMPLE_VERSION: u8 = 2; /// Time-sample flag reserved for host test hooks. const SAMPLE_TEST_HOOKS: u8 = 0x08; +/// The encoding of an unknown bound in a time sample. +const SAMPLE_UNKNOWN: u32 = u32::MAX; /// Maximum signed restore rate deviation, equivalent to 250 ppm. const RATE_DEVIATION_LIMIT: i32 = 16_384_000; /// One part per million in Linux's frequency correction, which `adjtimex` scales by 2^16. const SCALED_PPM: i64 = 1 << 16; +/// One, in Linux's frequency unit: a million parts per million, scaled by 2^16. +const SCALED_ONE: i128 = SCALED_PPM as i128 * 1_000_000; /// Parts per billion in a part per million. const PPB_PER_PPM: i64 = 1000; /// Linux's limit of the frequency correction, 500 ppm. const FREQUENCY_LIMIT: i64 = 500 * SCALED_PPM; +/// Hz in a kHz, the unit to which Linux reads the TSC rate from the frequency MSR. +const HZ_PER_KHZ: u64 = 1000; +/// The time ABI's bound on the skew between any two processors' TSCs, in nanoseconds, by which a +/// sample's guest TSC, of the first processor, may lie outside the reading processor's reads. +const SKEW_BOUND_NS: u64 = 1000; +/// The most, in parts per million, by which the realtime clock's rate may differ from its +/// frequency correction's while the kernel slews an offset below the step threshold at the fast +/// time constant: 128 ms per 2^6 seconds. +const SLEW_RATE_PPM: i128 = 2000; /// Seconds in a day. const SECONDS_PER_DAY: u64 = 24 * 60 * 60; /// Maximum accepted downtime of thirty days. @@ -109,11 +123,6 @@ impl<'a, E: Copy> Fields<'a, E> { Ok(field) } - /// Reads the next 16-bit field. - fn u16(&mut self) -> Result { - self.bytes().map(u16::from_le_bytes) - } - /// Reads the next 32-bit field. fn u32(&mut self) -> Result { self.bytes().map(u32::from_le_bytes) @@ -128,6 +137,11 @@ impl<'a, E: Copy> Fields<'a, E> { fn u64(&mut self) -> Result { self.bytes().map(u64::from_le_bytes) } + + /// Reads the next signed 64-bit field. + fn i64(&mut self) -> Result { + self.bytes().map(i64::from_le_bytes) + } } /// Why a restore packet is invalid. Its `Display` is the diagnostic that repair reports. @@ -137,7 +151,7 @@ pub(crate) enum PacketError { IncompleteHeader, /// The magic number is not `OVR`. Magic([u8; 3]), - /// The version is not version one's. + /// The version is not the time ABI's. Version(u8), /// The flags set reserved bits. ReservedFlags(u8), @@ -153,8 +167,6 @@ pub(crate) enum PacketError { RateDeviation(i32), /// The downtime exceeds 30 days. Downtime(u64), - /// The UTC time is zero or does not fit the clocks' signed nanoseconds. - Utc(u64), /// The packet ends before the memory ranges and the entropy that its header gives. IncompleteBody, /// The memory range at this index is empty, or ends past the address space. @@ -192,7 +204,6 @@ impl Display for PacketError { formatter, "restore packet downtime {downtime} ns exceeds 30 days" ), - Self::Utc(utc) => write!(formatter, "restore packet UTC {utc} is invalid"), Self::IncompleteBody => formatter.write_str("restore packet body is incomplete"), Self::Range(index) => write!(formatter, "restore memory range {index} is invalid"), } @@ -201,7 +212,7 @@ impl Display for PacketError { impl std::error::Error for PacketError {} -/// The validated header of a version-one restore packet. +/// The validated header of a restore packet. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct PacketHeader { /// Packet behavior flags. @@ -216,8 +227,6 @@ pub(crate) struct PacketHeader { pub(crate) rate_deviation: i32, /// Restore downtime in nanoseconds. pub(crate) downtime_ns: u64, - /// Host UTC at packet creation in nanoseconds. - pub(crate) utc_ns: u64, } impl PacketHeader { @@ -233,7 +242,6 @@ impl PacketHeader { generation: fields.u32()?, rate_deviation: fields.i32()?, downtime_ns: fields.u64()?, - utc_ns: fields.u64()?, }; if magic != PACKET_MAGIC { return Err(PacketError::Magic(magic)); @@ -262,9 +270,6 @@ impl PacketHeader { if header.downtime_ns > DOWNTIME_LIMIT_NS { return Err(PacketError::Downtime(header.downtime_ns)); } - if header.utc_ns == 0 || i64::try_from(header.utc_ns).is_err() { - return Err(PacketError::Utc(header.utc_ns)); - } Ok(header) } @@ -290,7 +295,7 @@ impl PacketHeader { } } -/// A validated version-one restore packet. +/// A validated restore packet. /// /// It is neither `Clone` nor comparable, since its entropy is a [`Seed`], which has one owner. #[derive(Debug)] @@ -335,66 +340,122 @@ impl RestorePacket { pub(crate) enum TimeSampleError { /// The sample is shorter than its record. Incomplete, - /// The version is not version one's. + /// The version is not version two's. Version(u8), - /// The flags set bits other than the test-hook flag, or the reserved field is not zero. + /// The flags set bits other than the test-hook flag, or the reserved byte is not zero. Flags { /// The sample's flags. flags: u8, - /// The sample's reserved field. - reserved: u16, + /// The sample's reserved byte. + reserved: u8, }, + /// The host clock status is not one that version two defines. + Status(u8), /// The UTC time is zero or does not fit the clocks' signed nanoseconds. Utc(u64), + /// The sample's guest TSC lies outside the guest's reads of its TSC around the sample's + /// latch, by more than the sample's pairing uncertainty and the skew bound allow. + Tsc { + /// The sample's guest TSC. + tsc: u64, + /// The guest's TSC before the latch. + before: u64, + /// The guest's TSC after the latch. + after: u64, + }, } impl Display for TimeSampleError { - /// Writes the diagnostic, such as `time sample version is 2`. + /// Writes the diagnostic, such as `time sample version is 1`. fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match *self { Self::Incomplete => formatter.write_str("time sample is incomplete"), Self::Version(version) => write!(formatter, "time sample version is {version}"), Self::Flags { flags, reserved } => write!( formatter, - "time sample flags 0x{flags:02x} or reserved 0x{reserved:04x}" + "time sample flags 0x{flags:02x} or reserved 0x{reserved:02x}" ), + Self::Status(status) => write!(formatter, "time sample host status is {status}"), Self::Utc(utc) => write!(formatter, "time sample UTC {utc} is invalid"), + Self::Tsc { tsc, before, after } => write!( + formatter, + "time sample TSC {tsc} is outside the guest's reads {before} to {after}" + ), } } } impl std::error::Error for TimeSampleError {} +/// The discipline of host UTC that a time sample publishes. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct HostClock { + /// The status of host UTC. + pub(crate) status: HostClockStatus, + /// A counter that changes when host UTC steps, the host resumes from a suspension, or the + /// guest TSC's offset from the host's changes. + pub(crate) disruption: u32, + /// How fast host UTC runs against the guest TSC counted at the declared rate, minus one, in + /// parts per million scaled by 2^16. + pub(crate) correction: i64, + /// A bound on the error of `correction`, in parts per million scaled by 2^16, if the host + /// knows one. + pub(crate) correction_bound: Option, + /// The host time service's bound on the error of host UTC, in microseconds, if it reports + /// one. + pub(crate) max_error_us: Option, +} + +impl HostClock { + /// Returns the host's bound on the error of host UTC that the guest adds to its own maximum + /// error, in microseconds: the time service's while it synchronizes host UTC, and otherwise + /// none. + pub(crate) fn error_us(&self) -> i64 { + match (self.status, self.max_error_us) { + (HostClockStatus::Synchronized, Some(error)) => i64::from(error), + _ => 0, + } + } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] -/// One host clock sample paired with guest clock brackets. +/// One host time sample: host UTC paired with the guest TSC, and the discipline of host UTC. pub(crate) struct TimeSample { /// Sample feature/test flags. pub(crate) flags: u8, /// VM generation associated with the sample. pub(crate) generation: u32, - /// Host UTC sample in nanoseconds. + /// Host UTC at the latch, in nanoseconds since the Unix epoch. pub(crate) utc_ns: u64, - /// Guest realtime immediately before requesting the sample. - pub(crate) t0: i64, - /// Guest realtime immediately after requesting the sample. - pub(crate) t1: i64, + /// The guest TSC, of the first processor, at the instant of `utc_ns`. + pub(crate) tsc: u64, + /// The most the instants of `tsc` and `utc_ns` can differ, in nanoseconds. + pub(crate) pairing_ns: u32, + /// The discipline of host UTC. + pub(crate) host: HostClock, } impl TimeSample { - /// Decodes and validates the sample record at the start of `bytes`, without the guest's - /// brackets. + /// Decodes and validates the sample record at the start of `bytes`. pub(crate) fn parse(bytes: &[u8]) -> Result { let mut fields = Fields::new(bytes, TimeSampleError::Incomplete); - let [version, flags] = fields.bytes()?; - let reserved = fields.u16()?; + let [version, flags, status, reserved] = fields.bytes()?; let generation = fields.u32()?; let utc_ns = fields.u64()?; + let tsc = fields.u64()?; + let pairing_ns = fields.u32()?; + let disruption = fields.u32()?; + let correction = fields.i64()?; + let bound = |value: u32| (value != SAMPLE_UNKNOWN).then_some(value); + let correction_bound = bound(fields.u32()?); + let max_error_us = bound(fields.u32()?); if version != SAMPLE_VERSION { return Err(TimeSampleError::Version(version)); } if flags & !SAMPLE_TEST_HOOKS != 0 || reserved != 0 { return Err(TimeSampleError::Flags { flags, reserved }); } + let status = HostClockStatus::from_byte(status).ok_or(TimeSampleError::Status(status))?; if utc_ns == 0 || i64::try_from(utc_ns).is_err() { return Err(TimeSampleError::Utc(utc_ns)); } @@ -402,12 +463,131 @@ impl TimeSample { flags, generation, utc_ns, - t0: 0, - t1: 0, + tsc, + pairing_ns, + host: HostClock { + status, + disruption, + correction, + correction_bound, + max_error_us, + }, }) } } +/// The guest's reads around the latch of one time sample, with which it pairs the sample's +/// guest TSC with its realtime clock. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct GuestReads { + /// The TSC right before the selector write that latched the sample. + pub(crate) before: u64, + /// The TSC right after the selector write. + pub(crate) after: u64, + /// The realtime clock, in nanoseconds, read right after the selector write. + pub(crate) realtime_ns: i64, + /// The TSC at the realtime read: the midpoint of the TSC reads around it. + pub(crate) realtime_tsc: u64, + /// Half the TSC cycles between the TSC reads around the realtime read. + pub(crate) realtime_half_cycles: u64, +} + +/// The rates with which the guest's realtime clock counts its TSC. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct ClockRates { + /// The declared TSC rate `F`, in Hz, which the frequency MSR reports. + pub(crate) tsc_hz: u64, + /// The kernel's frequency correction, in parts per million scaled by 2^16. + pub(crate) frequency: i64, +} + +/// Divides `numerator` by the positive `denominator`, rounding to the nearest, half away from +/// zero. +fn divide_rounded(numerator: i128, denominator: i128) -> i128 { + let half = denominator / 2; + if numerator >= 0 { + (numerator + half) / denominator + } else { + (numerator - half) / denominator + } +} + +/// Divides the nonnegative `numerator` by the positive `denominator`, rounding up. +fn divide_up(numerator: i128, denominator: i128) -> i128 { + (numerator + denominator - 1) / denominator +} + +/// Saturates `value` to an `i64`. +fn saturate(value: i128) -> i64 { + value.clamp(i64::MIN.into(), i64::MAX.into()) as i64 +} + +/// Returns the rate at which the kernel counts the TSC for the declared rate `tsc_hz`: Linux +/// reads the frequency MSR to the kHz. +pub(crate) const fn kernel_tsc_hz(tsc_hz: u64) -> u64 { + tsc_hz / HZ_PER_KHZ * HZ_PER_KHZ +} + +/// Pairs `sample` with the guest's realtime clock through `reads`, taken at `rates`, and returns +/// the offset of host UTC from the guest's realtime clock at the sample's guest TSC, `theta`, +/// and its uncertainty, `epsilon`, both in nanoseconds. +/// +/// The sample's guest TSC must lie between the reads around the latch, give or take the sample's +/// pairing uncertainty and the skew bound, since the host reads its UTC while it handles the +/// selector write. The guest's realtime clock at that TSC extrapolates its read after the latch +/// at the kernel's rate, and `epsilon` adds to the host's pairing uncertainty the cross-CPU skew +/// bound, half the reads around the realtime read, and what a slew can change the rate by over +/// the extrapolation. +pub(crate) fn pair_sample( + sample: &TimeSample, + reads: &GuestReads, + rates: ClockRates, +) -> Result<(i64, i64), TimeSampleError> { + const NSEC: i128 = NSEC_PER_SEC as i128; + let tsc_hz = i128::from(rates.tsc_hz.max(1)); + let kernel_hz = i128::from(kernel_tsc_hz(rates.tsc_hz).max(HZ_PER_KHZ)); + let pairing = i128::from(sample.pairing_ns); + let tolerance = divide_up((pairing + i128::from(SKEW_BOUND_NS)) * tsc_hz, NSEC); + let tsc = i128::from(sample.tsc); + if tsc < i128::from(reads.before) - tolerance || tsc > i128::from(reads.after) + tolerance { + return Err(TimeSampleError::Tsc { + tsc: sample.tsc, + before: reads.before, + after: reads.after, + }); + } + // The kernel's correction never exceeds its limit, which bounds the arithmetic. + let frequency = i128::from(rates.frequency.clamp(-FREQUENCY_LIMIT, FREQUENCY_LIMIT)); + let elapsed = divide_rounded((i128::from(reads.realtime_tsc) - tsc) * NSEC, kernel_hz); + let extrapolated = elapsed + divide_rounded(elapsed * frequency, SCALED_ONE); + let realtime = i128::from(reads.realtime_ns) - extrapolated; + let theta = i128::from(sample.utc_ns) - realtime; + let local = divide_up(i128::from(reads.realtime_half_cycles) * NSEC, kernel_hz); + let slew = divide_up(extrapolated.abs() * SLEW_RATE_PPM, 1_000_000); + Ok(( + saturate(theta), + saturate(pairing + i128::from(SKEW_BOUND_NS) + local + slew), + )) +} + +/// Returns the frequency correction, in parts per million scaled by 2^16, that runs the wall +/// clock at host UTC's rate when host UTC runs `correction` faster than the TSC counted at the +/// declared rate `tsc_hz`, and whether Linux's ±500 ppm limit clamped it. +/// +/// The kernel counts the TSC at `F_k`, the declared rate `F` to the kHz, so the correction is +/// `(1 + correction) * F_k / F - 1`, rounded to the nearest. +pub(crate) fn guest_frequency(correction: i64, tsc_hz: u64) -> (i64, bool) { + let tsc_hz_wide = i128::from(tsc_hz.max(1)); + // A correction beyond 100% is beyond the limit too, which bounds the arithmetic. + let correction = i128::from(correction).clamp(-SCALED_ONE, SCALED_ONE); + let numerator = + (SCALED_ONE + correction) * i128::from(kernel_tsc_hz(tsc_hz)) - SCALED_ONE * tsc_hz_wide; + let frequency = divide_rounded(numerator, tsc_hz_wide); + let limit = i128::from(FREQUENCY_LIMIT); + let held = frequency.clamp(-limit, limit); + (held as i64, held != frequency) +} + /// Encodes bytes as lowercase hexadecimal without separators. pub(crate) fn format_hex(bytes: &[u8]) -> String { let mut text = String::with_capacity(bytes.len() * 2); @@ -430,25 +610,17 @@ pub(crate) fn parse_hex(text: &[u8]) -> Option<[u8; N]> { Some(bytes) } -/// Computes clock offset and half-round-trip uncertainty with wrapping C arithmetic. -pub(crate) fn pair_clocks(t0: i64, t1: i64, utc_ns: u64) -> (i64, i64) { - let epsilon = t1.wrapping_sub(t0) / 2; - ( - (utc_ns as i64).wrapping_sub(t0.wrapping_add(epsilon)), - epsilon, - ) -} - -/// Converts host rate deviation into a clamped Linux frequency correction. -pub(crate) fn restore_frequency(rate_deviation: i32) -> i64 { - (-i64::from(rate_deviation)).clamp(-FREQUENCY_LIMIT, FREQUENCY_LIMIT) -} - /// Converts Linux scaled-ppm frequency into rounded parts per billion. pub(crate) fn frequency_to_ppb(frequency: i64) -> i64 { - let scaled = frequency * PPB_PER_PPM; - let half = SCALED_PPM / 2; - (scaled + if scaled >= 0 { half } else { -half }) / SCALED_PPM + saturate(divide_rounded( + i128::from(frequency) * i128::from(PPB_PER_PPM), + i128::from(SCALED_PPM), + )) +} + +/// Converts a bound in Linux's scaled-ppm unit into parts per billion, rounded up. +pub(crate) fn bound_to_ppb(bound: u32) -> u64 { + (u64::from(bound) * PPB_PER_PPM as u64).div_ceil(SCALED_PPM as u64) } /// Splits a signed nanosecond offset into whole seconds, rounded toward negative infinity, and @@ -460,10 +632,41 @@ pub(crate) fn split_offset(offset_ns: i64) -> (i64, i64) { ) } +/// Rounds nonnegative nanoseconds up to microseconds, saturating. +fn ceil_us(nanoseconds: i64) -> i64 { + nanoseconds.saturating_add(NSEC_PER_USEC - 1) / NSEC_PER_USEC +} + +/// Returns the frequency hold of a wall clock that is `offset` nanoseconds off host UTC, give or +/// take the uncertainty `epsilon`, with the frequency correction `frequency`, in scaled ppm, and +/// a host whose UTC is within `host_error_us` microseconds of the reference: the maximum error +/// adds the host's to the offset and its uncertainty, and the estimated error is the uncertainty. +pub(crate) fn frequency_hold( + offset: u64, + epsilon: i64, + frequency: i64, + host_error_us: i64, +) -> FrequencyHold { + let offset = i64::try_from(offset).unwrap_or(i64::MAX); + FrequencyHold { + scaled_ppm: frequency, + maximum_error_us: ceil_us(offset.saturating_add(epsilon)).saturating_add(host_error_us), + estimated_error_us: ceil_us(epsilon), + } +} + /// Selects and parameterizes a step or slew of the wall clock for the offset `theta` and the /// uncertainty `epsilon`, both in nanoseconds: a step at or above the step threshold, and -/// otherwise a slew with the fast or slow time constant. -pub(crate) fn plan_correction(theta: i64, epsilon: i64, fast: bool) -> ClockAdjustment { +/// otherwise a slew with the fast or slow time constant, which also holds the frequency +/// correction `frequency` with error estimates that add `host_error_us`, as [`frequency_hold`] +/// derives them. A step leaves its caller to hold the frequency after it. +pub(crate) fn plan_correction( + theta: i64, + epsilon: i64, + fast: bool, + frequency: i64, + host_error_us: i64, +) -> ClockAdjustment { let magnitude = theta.unsigned_abs(); if magnitude >= STEP_THRESHOLD_NS { let (seconds, nanoseconds) = split_offset(theta); @@ -472,7 +675,6 @@ pub(crate) fn plan_correction(theta: i64, epsilon: i64, fast: bool) -> ClockAdju nanoseconds, }; } - let ceil_us = |nanoseconds: i64| nanoseconds.saturating_add(NSEC_PER_USEC - 1) / NSEC_PER_USEC; ClockAdjustment::Slew { offset_ns: theta, time_constant: if fast { @@ -480,9 +682,7 @@ pub(crate) fn plan_correction(theta: i64, epsilon: i64, fast: bool) -> ClockAdju } else { SLOW_TIME_CONSTANT }, - // Below the step threshold, the magnitude fits an `i64`. - maximum_error_us: ceil_us((magnitude as i64).saturating_add(epsilon)), - estimated_error_us: ceil_us(epsilon), + hold: frequency_hold(magnitude, epsilon, frequency, host_error_us), } } @@ -869,7 +1069,7 @@ pub(crate) fn check_cpuinfo( } } -/// Validates the kernel command line required by time ABI version one: the kernel must apply none +/// Validates the kernel command line required by the time ABI: the kernel must apply none /// of the clock parameters that the ABI forbids among the parameters that it reads from the /// command line, as [`cmdline::parameters`] reads them, in quotes, after any of its whitespace, /// and up to a `--`. @@ -970,16 +1170,15 @@ mod tests { /// A valid packet header without memory ranges, of generation 1. fn valid_header() -> [u8; PACKET_HEADER_SIZE] { let mut bytes = [0; PACKET_HEADER_SIZE]; - bytes[..4].copy_from_slice(b"OVR\x04"); + bytes[..4].copy_from_slice(b"OVR\x05"); bytes[8..12].copy_from_slice(&1_u32.to_le_bytes()); - bytes[24..32].copy_from_slice(&1_u64.to_le_bytes()); bytes } #[test] fn packets_decode_their_header_then_their_body() { let header = PacketHeader::parse(&valid_header()).unwrap(); - assert_eq!((header.generation, header.utc_ns), (1, 1)); + assert_eq!((header.generation, header.downtime_ns), (1, 0)); assert_eq!(header.body_size(), PACKET_ENTROPY_SIZE); let entropy: Vec = (0..64).collect(); let packet = RestorePacket::parse(header, &entropy).unwrap(); @@ -990,9 +1189,13 @@ mod tests { PacketError::IncompleteBody ); assert_eq!( - PacketHeader::parse(&valid_header()[..31]), + PacketHeader::parse(&valid_header()[..23]), Err(PacketError::IncompleteHeader) ); + // A packet of version 4, which carried host UTC, is refused. + let mut bytes = valid_header(); + bytes[3] = 4; + assert_eq!(PacketHeader::parse(&bytes), Err(PacketError::Version(4))); } #[test] @@ -1012,23 +1215,217 @@ mod tests { assert_eq!(error.to_string(), "restore memory range 0 is invalid"); } + /// A valid time sample of generation 4, as OpenVMM's `TimeSample::encode` writes it. + fn valid_sample() -> [u8; SAMPLE_SIZE] { + let mut bytes = [0; SAMPLE_SIZE]; + bytes[..4].copy_from_slice(&[2, 8, 1, 0]); + bytes[4..8].copy_from_slice(&4_u32.to_le_bytes()); + bytes[8..16].copy_from_slice(&1_790_841_600_123_456_789_u64.to_le_bytes()); + bytes[16..24].copy_from_slice(&123_456_789_u64.to_le_bytes()); + bytes[24..28].copy_from_slice(&640_u32.to_le_bytes()); + bytes[28..32].copy_from_slice(&3_u32.to_le_bytes()); + bytes[32..40].copy_from_slice(&(-6_552_945_i64).to_le_bytes()); + bytes[40..44].copy_from_slice(&66_u32.to_le_bytes()); + bytes[44..48].copy_from_slice(&u32::MAX.to_le_bytes()); + bytes + } + #[test] - fn samples_check_their_version_flags_and_time() { - let mut sample = [0_u8; SAMPLE_SIZE]; - sample[0] = 1; - sample[8] = 1; - assert_eq!(TimeSample::parse(&sample).unwrap().utc_ns, 1); + fn samples_decode_every_field_of_version_two() { assert_eq!( - TimeSample::parse(&sample[..15]), + TimeSample::parse(&valid_sample()), + Ok(TimeSample { + flags: 8, + generation: 4, + utc_ns: 1_790_841_600_123_456_789, + tsc: 123_456_789, + pairing_ns: 640, + host: HostClock { + status: HostClockStatus::Synchronized, + disruption: 3, + correction: -6_552_945, + correction_bound: Some(66), + max_error_us: None, + }, + }) + ); + assert_eq!( + TimeSample::parse(&valid_sample()[..47]), Err(TimeSampleError::Incomplete) ); - sample[2] = 1; + } + + #[test] + fn samples_check_their_version_flags_status_and_time() { + let mutated = |offset: usize, bytes: &[u8]| { + let mut sample = valid_sample(); + sample[offset..offset + bytes.len()].copy_from_slice(bytes); + TimeSample::parse(&sample).unwrap_err().to_string() + }; + assert_eq!(mutated(0, &[1]), "time sample version is 1"); + assert_eq!(mutated(1, &[1]), "time sample flags 0x01 or reserved 0x00"); + assert_eq!(mutated(3, &[1]), "time sample flags 0x08 or reserved 0x01"); + assert_eq!(mutated(2, &[4]), "time sample host status is 4"); + assert_eq!(mutated(8, &[0; 8]), "time sample UTC 0 is invalid"); assert_eq!( - TimeSample::parse(&sample).unwrap_err().to_string(), - "time sample flags 0x00 or reserved 0x0001" + mutated(8, &u64::MAX.to_le_bytes()), + format!("time sample UTC {} is invalid", u64::MAX) ); } + #[test] + fn the_host_error_counts_only_while_a_time_service_synchronizes_the_host() { + let host = |status, max_error_us| HostClock { + status, + max_error_us, + ..HostClock::default() + }; + assert_eq!( + host(HostClockStatus::Synchronized, Some(12_500)).error_us(), + 12_500 + ); + assert_eq!(host(HostClockStatus::Synchronized, None).error_us(), 0); + assert_eq!( + host(HostClockStatus::FreeRunning, Some(16_000_000)).error_us(), + 0 + ); + assert_eq!(host(HostClockStatus::Unknown, Some(5)).error_us(), 0); + } + + /// A sample of guest TSC 1,000,000 at host UTC 10^18, paired within 500 ns. + fn paired_sample(tsc: u64) -> TimeSample { + TimeSample { + utc_ns: 1_000_000_000_000_000_000, + tsc, + pairing_ns: 500, + ..TimeSample::parse(&valid_sample()).unwrap() + } + } + + /// The guest's reads around the latch of [`paired_sample`]: its TSC 1,000 cycles before and + /// 2,000 after the sample's, and its realtime clock 4,000 cycles after, within 40 cycles. + const READS: GuestReads = GuestReads { + before: 999_000, + after: 1_002_000, + realtime_ns: 999_999_999_999_000_000, + realtime_tsc: 1_004_000, + realtime_half_cycles: 40, + }; + + /// A 2 GHz TSC, without a frequency correction. + const RATES: ClockRates = ClockRates { + tsc_hz: 2_000_000_000, + frequency: 0, + }; + + #[test] + fn a_pairing_extrapolates_the_realtime_clock_back_to_the_sample() { + // The realtime read is 4,000 cycles, 2 µs, after the sample's TSC, so the guest's clock + // was 999_999_999_998_998_000 then, 1_002_000 ns behind host UTC. The uncertainty adds + // 1 µs of cross-CPU skew, 20 ns of the realtime read, and 4 ns of a slew over the + // 2 µs, to the host's 500 ns. + assert_eq!( + pair_sample(&paired_sample(1_000_000), &READS, RATES), + Ok((1_002_000, 1_524)) + ); + // The kernel counts the TSC at the declared rate to the kHz, with its frequency + // correction. + let late = GuestReads { + realtime_tsc: 1_000_000 + 2_000_000_000, + ..READS + }; + let (theta, _) = pair_sample( + &paired_sample(1_000_000), + &late, + ClockRates { + tsc_hz: 2_000_000_999, + frequency: 0, + }, + ) + .unwrap(); + assert_eq!(theta, 1_000_000 + 1_000_000_000); + let (theta, epsilon) = pair_sample( + &paired_sample(1_000_000), + &GuestReads { + realtime_tsc: 1_000_000 + 20_000_000, + ..READS + }, + ClockRates { + frequency: 6_553_600, + ..RATES + }, + ) + .unwrap(); + assert_eq!(theta, 1_000_000 + 10_001_000); + assert_eq!(epsilon, 500 + 1_000 + 20 + 20_002); + } + + #[test] + fn a_sample_outside_the_guest_reads_is_refused() { + // The tolerance is 1.5 µs, the pairing and the skew bound: 3,000 cycles at 2 GHz. + for tsc in [996_000, 1_005_000] { + assert!( + pair_sample(&paired_sample(tsc), &READS, RATES).is_ok(), + "{tsc}" + ); + } + for tsc in [995_999, 1_005_001, 0, u64::MAX] { + let error = pair_sample(&paired_sample(tsc), &READS, RATES).unwrap_err(); + assert_eq!( + error, + TimeSampleError::Tsc { + tsc, + before: 999_000, + after: 1_002_000, + } + ); + assert_eq!( + error.to_string(), + format!("time sample TSC {tsc} is outside the guest's reads 999000 to 1002000") + ); + } + } + + #[test] + fn the_guest_frequency_applies_the_host_correction_to_the_kernel_rate() { + // A declared rate of whole kHz: the correction itself. + assert_eq!( + guest_frequency(13_107_200, 2_100_000_000), + (13_107_200, false) + ); + assert_eq!( + guest_frequency(-6_552_945, 2_000_000_000), + (-6_552_945, false) + ); + // The kernel's rate, to the kHz, is 123 Hz slow: about 0.056 ppm. + assert_eq!(guest_frequency(0, 2_194_843_123), (-3_673, false)); + assert_eq!( + guest_frequency(13_107_200, 2_194_843_123), + (13_103_527, false) + ); + // Linux's limit of 500 ppm holds the frequency, and the call reports it. + assert_eq!( + guest_frequency(39_321_600, 2_100_000_000), + (32_768_000, true) + ); + assert_eq!( + guest_frequency(-39_321_600, 2_100_000_000), + (-32_768_000, true) + ); + assert_eq!(guest_frequency(i64::MAX, 2_100_000_000), (32_768_000, true)); + } + + #[test] + fn rates_convert_to_parts_per_billion() { + assert_eq!(frequency_to_ppb(13_107_200), 200_000); + assert_eq!(frequency_to_ppb(-6_552_945), -99_990); + assert_eq!(frequency_to_ppb(-3_673), -56); + assert_eq!(frequency_to_ppb(i64::MIN), -(1_i64 << 47) * 1000); + assert_eq!(bound_to_ppb(66), 2); + assert_eq!(bound_to_ppb(13_107_200), 200_000); + assert_eq!(bound_to_ppb(0), 0); + } + #[test] fn generation_ids_are_exact_hexadecimal() { assert_eq!(parse_hex::<2>(b"0aFf"), Some([0x0a, 0xff])); @@ -1041,15 +1438,15 @@ mod tests { #[test] fn correction_boundary_matches_the_abi() { assert!(matches!( - plan_correction(128_000_000, 10, true), + plan_correction(128_000_000, 10, true, 0, 0), ClockAdjustment::Step { .. } )); assert!(matches!( - plan_correction(127_999_999, 10, true), + plan_correction(127_999_999, 10, true, 0, 0), ClockAdjustment::Slew { .. } )); assert_eq!( - plan_correction(-1_128_000_000, 0, false), + plan_correction(-1_128_000_000, 0, false, 0, 0), ClockAdjustment::Step { seconds: -2, nanoseconds: 872_000_000, @@ -1057,6 +1454,30 @@ mod tests { ); } + #[test] + fn a_slew_holds_the_frequency_with_the_host_error() { + assert_eq!( + plan_correction(-1_500, 2_100, false, 13_107_200, 12_500), + ClockAdjustment::Slew { + offset_ns: -1_500, + time_constant: 6, + hold: FrequencyHold { + scaled_ppm: 13_107_200, + maximum_error_us: 4 + 12_500, + estimated_error_us: 3, + }, + } + ); + assert_eq!( + frequency_hold(0, 524, -3_673, 0), + FrequencyHold { + scaled_ppm: -3_673, + maximum_error_us: 1, + estimated_error_us: 1, + } + ); + } + #[test] fn offsets_split_toward_negative_infinity() { assert_eq!(split_offset(-1), (-1, 999_999_999)); @@ -1066,25 +1487,62 @@ mod tests { #[test] fn extreme_clock_arithmetic_neither_wraps_nor_panics() { - assert_eq!(pair_clocks(i64::MIN, i64::MAX, 0), (i64::MIN, 0)); assert_eq!( - plan_correction(i64::MIN, 0, true), + plan_correction(i64::MIN, 0, true, 0, 0), ClockAdjustment::Step { seconds: -9_223_372_037, nanoseconds: 145_224_192, } ); - let slew = plan_correction(-5, i64::MAX, true); + let slew = plan_correction(-5, i64::MAX, true, 7, i64::MAX); assert_eq!( slew, ClockAdjustment::Slew { offset_ns: -5, time_constant: 4, - maximum_error_us: i64::MAX / 1000, - estimated_error_us: i64::MAX / 1000, + hold: FrequencyHold { + scaled_ppm: 7, + maximum_error_us: i64::MAX, + estimated_error_us: i64::MAX / 1000, + }, } ); - assert_eq!((slew.modes(), slew.status()), (0x203d, 0x2001)); + assert_eq!((slew.modes(), slew.status()), (0x203f, 0x2081)); + let extreme = TimeSample { + utc_ns: i64::MAX as u64, + pairing_ns: u32::MAX, + ..paired_sample(u64::MAX) + }; + let reads = GuestReads { + before: u64::MAX, + after: u64::MAX, + realtime_ns: i64::MIN, + realtime_tsc: 0, + realtime_half_cycles: u64::MAX, + }; + let (theta, epsilon) = pair_sample( + &extreme, + &reads, + ClockRates { + tsc_hz: u64::MAX, + frequency: i64::MAX, + }, + ) + .unwrap(); + assert_eq!(theta, i64::MAX); + assert!(epsilon > i64::from(u32::MAX), "{epsilon}"); + let (theta, epsilon) = pair_sample( + &extreme, + &reads, + ClockRates { + tsc_hz: 0, + frequency: i64::MIN, + }, + ) + .unwrap(); + assert_eq!((theta, epsilon), (i64::MIN, i64::MAX)); + assert_eq!(guest_frequency(i64::MIN, u64::MAX), (-32_768_000, true)); + assert_eq!(guest_frequency(0, 0), (-32_768_000, true)); } #[test] @@ -1615,7 +2073,7 @@ mod tests { } /// Returns whether the kernel applies, in the parameter `name` with `value`, a clock - /// parameter that time ABI v1 forbids, as `init/main.c` matches parameters with their + /// parameter that the time ABI forbids, as `init/main.c` matches parameters with their /// handlers: an `early_param` by `parameq` of its name, which NVX applies to /// `lapic_timer_hz` too, and a `__setup` string by `parameqn` of the parameter's text over the /// string's length, which passes the rest of the text to the handler. diff --git a/nvx-rs/src/guest/time/mod.rs b/nvx-rs/src/guest/time/mod.rs index 3a3e00910..f88a2a8a5 100644 --- a/nvx-rs/src/guest/time/mod.rs +++ b/nvx-rs/src/guest/time/mod.rs @@ -1,4 +1,4 @@ -//! `nvx-time`, the guest side of NVX time ABI v1, and the typed calls through which any guest +//! `nvx-time`, the guest side of NVX time ABI v2, and the typed calls through which any guest //! init or agent runs the ABI's steps in its own process. //! //! [`TimeAbi`] runs the cold-boot time step and, separately, starts the time daemon, a process diff --git a/nvx-rs/src/guest/time/state.rs b/nvx-rs/src/guest/time/state.rs index f71f9a9f2..b80fb5b4b 100644 --- a/nvx-rs/src/guest/time/state.rs +++ b/nvx-rs/src/guest/time/state.rs @@ -4,6 +4,8 @@ use std::fmt::{self, Write as _}; use std::str::FromStr; use super::Phase; +use super::decoders; +use super::system::BestSample; /// Defines an enumeration of the tokens of one state-file field, with each token's spelling. macro_rules! tokens { @@ -89,6 +91,53 @@ tokens! { None => "none", /// No sample was narrow enough. Uncertain => "G_SAMPLE_UNCERTAIN", + /// A sample's guest TSC lay outside the guest's reads of its TSC around the sample's + /// latch, by more than the sample's pairing uncertainty allows. + Tsc => "G_SAMPLE_TSC", + /// No attempt took a valid sample, or the discipline could not apply one. + Invalid => "G_SAMPLE_INVALID", + } +} + +tokens! { + /// The status of host UTC that the latest accepted time sample reported. + #[derive(Default)] + HostClockStatus { + /// The host does not report its clock's status. + #[default] + Unknown => "unknown", + /// A time service synchronizes host UTC with a reference. + Synchronized => "synchronized", + /// No time service synchronizes host UTC, which runs on its own. + FreeRunning => "free-running", + /// The host reports a fault of its clock. + Unreliable => "unreliable", + } +} + +impl HostClockStatus { + /// Returns the status that a time sample's status byte encodes, if any. + pub(crate) const fn from_byte(byte: u8) -> Option { + match byte { + 0 => Some(Self::Unknown), + 1 => Some(Self::Synchronized), + 2 => Some(Self::FreeRunning), + 3 => Some(Self::Unreliable), + _ => None, + } + } +} + +/// Writes an optional bound of the state file: its value, or `unknown`. +struct Bound(Option); + +impl fmt::Display for Bound { + /// Writes the value, or `unknown` for none. + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.0 { + Some(value) => write!(formatter, "{value}"), + None => formatter.write_str("unknown"), + } } } @@ -109,7 +158,7 @@ pub(crate) struct CheckState { pub(crate) failures: Option, } -/// Complete version-one state published under `/run/nvx/time/state`. +/// Complete state published under `/run/nvx/time/state`, of version one of its format. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct TimeState { /// Current VM generation. @@ -140,6 +189,22 @@ pub(crate) struct TimeState { pub(crate) uncertainty_ns: i64, /// Current frequency correction in parts per billion. pub(crate) frequency_ppb: i64, + /// Whether Linux's limit of the frequency correction, ±500 ppm, clamped the correction that + /// the host's clock correction asked for. + pub(crate) frequency_limited: bool, + /// The status of host UTC that the latest accepted sample reported. + pub(crate) host_status: HostClockStatus, + /// The host's clock correction that the latest accepted sample published: how fast host UTC + /// runs against the TSC counted at the declared rate, minus one, in parts per billion. + pub(crate) host_correction_ppb: i64, + /// The host's bound on the error of its correction, in parts per billion, if it knows one. + pub(crate) host_correction_bound_ppb: Option, + /// The host time service's bound on the error of host UTC, in microseconds, if it reports + /// one. + pub(crate) host_max_error_us: Option, + /// The disruption counter of the latest accepted sample, which changes when host UTC steps, + /// the host resumes from a suspension, or the guest TSC's offset from the host's changes. + pub(crate) host_disruption: u64, /// Number of accepted time samples. pub(crate) samples: u64, /// Number of rejected time samples. @@ -172,6 +237,12 @@ impl Default for TimeState { offset_ns: 0, uncertainty_ns: 0, frequency_ppb: 0, + frequency_limited: false, + host_status: HostClockStatus::Unknown, + host_correction_ppb: 0, + host_correction_bound_ppb: None, + host_max_error_us: None, + host_disruption: 0, samples: 0, rejected_samples: 0, last_sample_error: SampleRejection::None, @@ -181,6 +252,24 @@ impl Default for TimeState { } impl TimeState { + /// Records the accepted time sample `sample`: the wall clock's offset from host UTC and its + /// uncertainty, the frequency correction that the discipline holds, whether Linux's limit + /// clamped it, and the discipline of host UTC that the sample published. + pub(crate) fn record_sample(&mut self, sample: &BestSample) { + let host = &sample.host; + self.samples += 1; + self.offset_ns = sample.theta; + self.uncertainty_ns = sample.epsilon; + self.frequency_ppb = decoders::frequency_to_ppb(sample.frequency); + self.frequency_limited = sample.limited; + self.host_status = host.status; + self.host_correction_ppb = decoders::frequency_to_ppb(host.correction); + self.host_correction_bound_ppb = host.correction_bound.map(decoders::bound_to_ppb); + self.host_max_error_us = host.max_error_us.map(u64::from); + self.host_disruption = u64::from(host.disruption); + self.last_sample_error = SampleRejection::None; + } + /// Parses the state file's `key=value` lines, ignoring unknown keys, or returns `None` for a /// known key whose value is malformed. pub(crate) fn parse(text: &[u8]) -> Option { @@ -199,6 +288,13 @@ impl TimeState { fn number(value: &str) -> Option { value.parse().ok() } + /// Parses an optional bound: a decimal number, or `unknown` for none. + fn bound(value: &str) -> Option> { + match value { + "unknown" => Some(None), + value => number(value).map(Some), + } + } let check = Phase::CHECKS .into_iter() .zip(&mut self.checks) @@ -233,6 +329,12 @@ impl TimeState { "offset_ns" => self.offset_ns = number(value)?, "uncertainty_ns" => self.uncertainty_ns = number(value)?, "frequency_ppb" => self.frequency_ppb = number(value)?, + "frequency_limited" => self.frequency_limited = number::(value)? != 0, + "host_status" => self.host_status = HostClockStatus::parse(value)?, + "host_correction_ppb" => self.host_correction_ppb = number(value)?, + "host_correction_bound_ppb" => self.host_correction_bound_ppb = bound(value)?, + "host_max_error_us" => self.host_max_error_us = bound(value)?, + "host_disruption" => self.host_disruption = number(value)?, "samples" => self.samples = number(value)?, "rejected_samples" => self.rejected_samples = number(value)?, "last_sample_error" => self.last_sample_error = SampleRejection::parse(value)?, @@ -380,6 +482,28 @@ impl fmt::Display for TimeState { writeln!(formatter, "offset_ns={}", self.offset_ns)?; writeln!(formatter, "uncertainty_ns={}", self.uncertainty_ns)?; writeln!(formatter, "frequency_ppb={}", self.frequency_ppb)?; + writeln!( + formatter, + "frequency_limited={}", + u8::from(self.frequency_limited) + )?; + writeln!(formatter, "host_status={}", self.host_status)?; + writeln!( + formatter, + "host_correction_ppb={}", + self.host_correction_ppb + )?; + writeln!( + formatter, + "host_correction_bound_ppb={}", + Bound(self.host_correction_bound_ppb) + )?; + writeln!( + formatter, + "host_max_error_us={}", + Bound(self.host_max_error_us) + )?; + writeln!(formatter, "host_disruption={}", self.host_disruption)?; writeln!(formatter, "samples={}", self.samples)?; writeln!(formatter, "rejected_samples={}", self.rejected_samples)?; writeln!(formatter, "last_sample_error={}", self.last_sample_error)?; @@ -420,6 +544,12 @@ mod tests { "offset_ns=-42\n", "uncertainty_ns=1800\n", "frequency_ppb=-1200\n", + "frequency_limited=0\n", + "host_status=synchronized\n", + "host_correction_ppb=-1185\n", + "host_correction_bound_ppb=2\n", + "host_max_error_us=12500\n", + "host_disruption=1\n", "samples=17\n", "rejected_samples=1\n", "last_sample_error=none\n", @@ -485,12 +615,107 @@ mod tests { b"generation=-1\n", b"boot_failures=-1\n", b"samples=\xff\n", + b"host_status=synced\n", + b"host_max_error_us=-1\n", + b"host_correction_bound_ppb=none\n", + b"last_sample_error=G_SAMPLE_LATE\n", ] { assert_eq!(TimeState::parse(text), None, "{}", text.escape_ascii()); } assert!(TimeState::parse(b"unknown=bad\n").is_some()); } + #[test] + fn host_fields_round_trip_with_unknown_bounds() { + let state = TimeState::parse( + b"host_status=free-running\nhost_correction_ppb=200000\n\ + host_correction_bound_ppb=unknown\nhost_max_error_us=unknown\n\ + frequency_limited=1\nlast_sample_error=G_SAMPLE_TSC\n", + ) + .unwrap(); + assert_eq!(state.host_status, HostClockStatus::FreeRunning); + assert_eq!(state.host_correction_ppb, 200_000); + assert_eq!( + (state.host_correction_bound_ppb, state.host_max_error_us), + (None, None) + ); + assert!(state.frequency_limited); + assert_eq!(state.last_sample_error, SampleRejection::Tsc); + let text = state.to_string(); + for line in [ + "\nfrequency_limited=1\nhost_status=free-running\nhost_correction_ppb=200000\n\ + host_correction_bound_ppb=unknown\nhost_max_error_us=unknown\nhost_disruption=0\n", + "\nlast_sample_error=G_SAMPLE_TSC\n", + ] { + assert!(text.contains(line), "{text}"); + } + assert_eq!(TimeState::parse(text.as_bytes()).unwrap(), state); + } + + #[test] + fn an_accepted_sample_records_the_host_discipline() { + let mut state = TimeState { + samples: 2, + last_sample_error: SampleRejection::Uncertain, + ..TimeState::default() + }; + state.record_sample(&BestSample { + theta: -42, + epsilon: 1_800, + host: decoders::HostClock { + status: HostClockStatus::Synchronized, + disruption: 3, + correction: 39_321_600, + correction_bound: Some(66), + max_error_us: Some(12_500), + }, + frequency: 32_768_000, + limited: true, + }); + assert_eq!( + ( + state.samples, + state.offset_ns, + state.uncertainty_ns, + state.frequency_ppb, + state.frequency_limited + ), + (3, -42, 1_800, 500_000, true) + ); + assert_eq!( + ( + state.host_status, + state.host_correction_ppb, + state.host_correction_bound_ppb, + state.host_max_error_us, + state.host_disruption + ), + ( + HostClockStatus::Synchronized, + 600_000, + Some(2), + Some(12_500), + 3 + ) + ); + assert_eq!(state.last_sample_error, SampleRejection::None); + } + + #[test] + fn host_statuses_decode_from_their_sample_bytes() { + assert_eq!( + (0..=4).map(HostClockStatus::from_byte).collect::>(), + [ + Some(HostClockStatus::Unknown), + Some(HostClockStatus::Synchronized), + Some(HostClockStatus::FreeRunning), + Some(HostClockStatus::Unreliable), + None + ] + ); + assert_eq!(HostClockStatus::default(), HostClockStatus::Unknown); + } + #[test] fn records_round_trip_with_their_tokens() { let text = "version=1\ngeneration=2\nboot_status=ok\nrestore_status=fail\nboot_cpus=8\n\ @@ -499,6 +724,8 @@ mod tests { lapic_hz=0\ndiscontinuities=1\nlast_discontinuity=restore\nlast_step_ns=-5\n\ last_step_realtime_ns=6\nlast_downtime_ns=7\nlast_downtime_source=monotonic\n\ synchronized=1\noffset_ns=-8\nuncertainty_ns=9\nfrequency_ppb=-10\n\ + frequency_limited=0\nhost_status=unknown\nhost_correction_ppb=-11\n\ + host_correction_bound_ppb=unknown\nhost_max_error_us=14\nhost_disruption=15\n\ samples=11\nrejected_samples=12\nlast_sample_error=G_SAMPLE_UNCERTAIN\n\ violations=13\n"; let state = TimeState::parse(text.as_bytes()).unwrap(); diff --git a/nvx-rs/src/guest/time/system.rs b/nvx-rs/src/guest/time/system.rs index 0a25a0180..d5a67d8b8 100644 --- a/nvx-rs/src/guest/time/system.rs +++ b/nvx-rs/src/guest/time/system.rs @@ -3,18 +3,21 @@ use std::ffi::{CStr, CString}; use std::fmt::{self, Display}; -use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering}; use std::sync::{Mutex, PoisonError}; +use super::Phase; use super::api::Violation; -use super::decoders::{self, SAMPLE_SIZE, TimeSample, TimeSampleError}; +use super::decoders::{ + self, ClockRates, GuestReads, HostClock, SAMPLE_SIZE, TimeSample, TimeSampleError, +}; use super::events::{EventSource, ViolationCode, format_event}; -use super::state::TimeState; -use super::{NSEC_PER_MSEC, Phase}; +use super::state::{SampleRejection, TimeState}; +use crate::guest::sys::x86::{self, msr}; use crate::guest::sys::{ self, ClockAdjustment, ClockDiscipline, CreatedFile, Errno, FileDescriptor, FileLock, - ForkResult, Mode, MsrAccess, MsrFile, OpenFlags, Port, PortAccess, PortError, TimeValLike, - kernel, + ForkResult, FrequencyHold, Mode, MsrAccess, MsrFile, OpenFlags, Port, PortAccess, PortError, + TimeValLike, kernel, }; use crate::protocol::status; @@ -116,16 +119,27 @@ const KERNEL_LOG_PRIORITY: &[u8] = b"<2>"; /// The priority prefix of a diagnostic that the program writes to the kernel log: `KERN_ERR`. const DIAGNOSTIC_PRIORITY: &[u8] = b"<3>"; -/// Number of host samples considered when selecting the narrowest. -const SAMPLE_ATTEMPTS: usize = 3; +/// Number of host samples that boot, restore repair, and each discipline poll take at most. +pub(crate) const SAMPLE_ATTEMPTS: usize = 3; +/// The largest uncertainty of a time sample that the guest accepts, in nanoseconds: the host +/// pairs its UTC with the guest TSC within 100 µs. +pub(crate) const SAMPLE_BOUND_NS: i64 = 100_000; +/// The smallest declared TSC rate, in Hz, that pairs a time sample: any lower rate is a misread +/// of the frequency MSR. +const MIN_TSC_HZ: u64 = 1_000_000; /// Seconds after which RCU reports a CPU stall in NVX's guest kernels, which the checks require /// and the daemon assumes when it cannot read the setting. pub(crate) const DEFAULT_STALL_TIMEOUT_S: i64 = 21; -/// The privileged operations of the time tools, which a fake replaces in tests, so that their -/// retry and failure paths are testable. +/// The privileged operations of the time tools, and the clock reads with which the guest pairs a +/// time sample, which a fake replaces in tests, so that their retry and failure paths are +/// testable. pub(crate) trait PrivilegedOps: Sync { + /// Reads the time-stamp counter after the instructions before it. + fn read_tsc(&self) -> u64; + /// Reads the realtime clock, in nanoseconds since the Unix epoch. + fn realtime_ns(&self) -> i64; /// Takes the calling thread's permission for the time ABI's ports, for one transaction, /// which [`PortAccess::release`] ends. fn open_ports(&self) -> Result, PortError>; @@ -163,6 +177,14 @@ pub(crate) enum MsrDescriptor { pub(crate) struct RealOps; impl PrivilegedOps for RealOps { + fn read_tsc(&self) -> u64 { + x86::rdtsc_lfence() + } + + fn realtime_ns(&self) -> i64 { + clock_ns(sys::Clock::Realtime) + } + fn open_ports(&self) -> Result, PortError> { PortAccess::request() } @@ -236,6 +258,8 @@ pub(crate) struct Runtime<'a> { violation: Mutex>, /// Current VM generation. generation: AtomicU32, + /// The declared TSC rate, `F`, in Hz, once known, or zero. + tsc_hz: AtomicU64, /// Whether an asynchronous child may have interrupted console output. async_output: AtomicBool, /// Serializes multi-destination violation records. @@ -251,6 +275,7 @@ impl<'a> Runtime<'a> { strict: false, violation: Mutex::new(None), generation: AtomicU32::new(0), + tsc_hz: AtomicU64::new(0), async_output: AtomicBool::new(false), report_lock: Mutex::new(()), } @@ -294,6 +319,34 @@ impl<'a> Runtime<'a> { self.generation.store(value, Ordering::Relaxed); } + /// Records the declared TSC rate, `F`, in Hz, that the boot checks validated, unless it is + /// zero, for none. + pub(crate) fn set_tsc_hz(&self, value: u64) { + if value != 0 { + self.tsc_hz.store(value, Ordering::Relaxed); + } + } + + /// Returns the declared TSC rate, `F`, in Hz: the one that the runtime recorded, or else the + /// frequency MSR's, of CPU 0, which it then records. + pub(crate) fn tsc_hz(&self) -> Result { + let recorded = self.tsc_hz.load(Ordering::Relaxed); + if recorded != 0 { + return Ok(recorded); + } + let read = self + .ops + .open_msr(0, MsrAccess::Read) + .and_then(|descriptor| self.ops.read_msr(&descriptor, msr::HV_TSC_FREQUENCY)); + match read { + Ok(rate) if rate >= MIN_TSC_HZ => { + self.tsc_hz.store(rate, Ordering::Relaxed); + Ok(rate) + } + read => Err(SampleError::TscRate(read)), + } + } + /// Marks whether console output may need a leading newline. pub(crate) fn set_async_output(&self, value: bool) { self.async_output.store(value, Ordering::Relaxed); @@ -670,27 +723,45 @@ pub(crate) fn portb_lock_failure(error: Errno) -> Vec { describe(b"portb lock: ", error) } -/// Requests and decodes one host time sample, bracketed by realtime reads. -pub(crate) fn take_sample(ports: &PortAccess<'_>) -> Result { +/// Latches and decodes one host time sample: brackets the selector write with TSC reads, then +/// reads the realtime clock between two more, with which the guest pairs the sample's guest TSC, +/// before it reads the time window. +pub(crate) fn take_sample( + ports: &PortAccess<'_>, + ops: &dyn PrivilegedOps, +) -> Result<(TimeSample, GuestReads), TimeSampleError> { let mut bytes = [0_u8; SAMPLE_SIZE]; - let t0 = clock_ns(sys::Clock::Realtime); + let before = ops.read_tsc(); ports.write_u8(Port::PortbStatus, SELECT_TIME_SAMPLE); - let t1 = clock_ns(sys::Clock::Realtime); + let after = ops.read_tsc(); + let realtime_ns = ops.realtime_ns(); + let end = ops.read_tsc(); portb_read(ports, Port::PortbWindow, &mut bytes); - Ok(TimeSample { - t0, - t1, - ..TimeSample::parse(&bytes)? - }) + let half_cycles = end.saturating_sub(after).div_ceil(2); + let reads = GuestReads { + before, + after, + realtime_ns, + realtime_tsc: after.saturating_add(half_cycles), + realtime_half_cycles: half_cycles, + }; + Ok((TimeSample::parse(&bytes)?, reads)) } -/// Narrowest valid sample observed during one attempt group. -#[derive(Clone, Copy, Debug)] +/// Narrowest valid sample observed during one attempt group, and the frequency correction that +/// its host clock correction asks for. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(crate) struct BestSample { /// Guest-to-host offset in nanoseconds. pub(crate) theta: i64, - /// Half-round-trip uncertainty in nanoseconds. + /// Pairing uncertainty in nanoseconds. pub(crate) epsilon: i64, + /// The discipline of host UTC that the sample published. + pub(crate) host: HostClock, + /// The frequency correction, in scaled ppm, that runs the wall clock at host UTC's rate. + pub(crate) frequency: i64, + /// Whether Linux's ±500 ppm limit clamped `frequency`. + pub(crate) limited: bool, } /// Whether the selected sample satisfied the requested uncertainty bound. @@ -707,6 +778,9 @@ pub(crate) enum SampleResult { pub(crate) enum SampleError { /// The thread could not take, or give up, the permission for the ports. Ports(PortError), + /// The guest could not read the declared TSC rate, with which it pairs a sample, from the + /// frequency MSR: the error, or the implausible rate that it read. + TscRate(Result), /// The port window, of this status, offers no time sample. NoTimeWindow(u8), /// A sample came from another VM generation, which a restore must handle first. @@ -720,11 +794,32 @@ pub(crate) enum SampleError { Invalid(Option), } +impl SampleError { + /// Returns how the discipline records the rejection of a poll's sample that failed so. + pub(crate) fn rejection(&self) -> SampleRejection { + match self { + Self::Invalid(Some(TimeSampleError::Tsc { .. })) => SampleRejection::Tsc, + _ => SampleRejection::Invalid, + } + } +} + impl Display for SampleError { /// Writes the diagnostic, such as `time sample generation 7, expected 6`. fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match *self { Self::Ports(error) => error.fmt(formatter), + Self::TscRate(Err(error)) => write!( + formatter, + "MSR {:#010x} read: {}", + msr::HV_TSC_FREQUENCY, + error.desc() + ), + Self::TscRate(Ok(rate)) => write!( + formatter, + "MSR {:#010x} (TSC rate) is {rate} Hz", + msr::HV_TSC_FREQUENCY + ), Self::NoTimeWindow(status) => write!( formatter, "portb status 0x{status:02x} has no time-sample window" @@ -741,6 +836,15 @@ impl Display for SampleError { impl std::error::Error for SampleError {} +/// Returns the rates with which the guest's realtime clock counts its TSC: the declared TSC rate, +/// and the kernel's frequency correction. +pub(crate) fn clock_rates(runtime: &Runtime<'_>) -> Result { + Ok(ClockRates { + tsc_hz: runtime.tsc_hz()?, + frequency: current_frequency(runtime), + }) +} + /// Selects the lowest-uncertainty valid sample from a bounded attempt count, in one port /// transaction, which ends before this returns. pub(crate) fn best_sample( @@ -749,18 +853,22 @@ pub(crate) fn best_sample( bound_ns: i64, generation: u32, ) -> Result<(SampleResult, BestSample), SampleError> { + let rates = clock_rates(runtime)?; let ports = runtime.open_ports().map_err(SampleError::Ports)?; - let selected = select_sample(&ports, attempts, bound_ns, generation); + let selected = select_sample(&ports, runtime.ops, attempts, bound_ns, generation, rates); ports.release().map_err(SampleError::Ports)?; selected } -/// Selects the lowest-uncertainty valid sample through `ports`, as [`best_sample`] does. -fn select_sample( +/// Selects the lowest-uncertainty valid sample through `ports`, pairing each with the guest's +/// clocks at `rates`, as [`best_sample`] does. +pub(crate) fn select_sample( ports: &PortAccess<'_>, + ops: &dyn PrivilegedOps, attempts: usize, bound_ns: i64, generation: u32, + rates: ClockRates, ) -> Result<(SampleResult, BestSample), SampleError> { let status = ports.read_u8(Port::PortbStatus); if status & PORTB_TIME_WINDOW == 0 { @@ -769,8 +877,8 @@ fn select_sample( let mut best: Option = None; let mut invalid = None; for _ in 0..attempts { - let sample = match take_sample(ports) { - Ok(sample) => sample, + let (sample, reads) = match take_sample(ports, ops) { + Ok(taken) => taken, Err(error) => { invalid = Some(error); continue; @@ -782,8 +890,21 @@ fn select_sample( expected: generation, }); } - let (theta, epsilon) = decoders::pair_clocks(sample.t0, sample.t1, sample.utc_ns); - let candidate = BestSample { theta, epsilon }; + let (theta, epsilon) = match decoders::pair_sample(&sample, &reads, rates) { + Ok(pairing) => pairing, + Err(error) => { + invalid = Some(error); + continue; + } + }; + let (frequency, limited) = decoders::guest_frequency(sample.host.correction, rates.tsc_hz); + let candidate = BestSample { + theta, + epsilon, + host: sample.host, + frequency, + limited, + }; if best.is_none_or(|best| epsilon < best.epsilon) { best = Some(candidate); } @@ -807,11 +928,11 @@ pub(crate) fn step_clock(runtime: &Runtime<'_>, offset_ns: i64) -> Result<(), Er .map(drop) } -/// Programs Linux's scaled-ppm frequency correction. -pub(crate) fn set_frequency(runtime: &Runtime<'_>, scaled_ppm: i64) -> Result<(), Errno> { +/// Holds Linux's scaled-ppm frequency correction, with its error estimates. +pub(crate) fn hold_frequency(runtime: &Runtime<'_>, hold: FrequencyHold) -> Result<(), Errno> { runtime .ops - .adjust_clock(ClockAdjustment::Frequency { scaled_ppm }) + .adjust_clock(ClockAdjustment::Frequency(hold)) .map(drop) } @@ -823,22 +944,43 @@ pub(crate) fn current_frequency(runtime: &Runtime<'_>) -> i64 { .map_or(0, |discipline| discipline.scaled_ppm) } -/// Takes the initial bounded sample and steps realtime to it. +/// Steps the wall clock to host UTC by `sample`'s offset, and then holds the frequency +/// correction that its host clock correction asks for, with error estimates of its uncertainty +/// and the host's error. +pub(crate) fn synchronize(runtime: &Runtime<'_>, sample: &BestSample) -> Result<(), Errno> { + step_clock(runtime, sample.theta)?; + hold_frequency( + runtime, + decoders::frequency_hold(0, sample.epsilon, sample.frequency, sample.host.error_us()), + ) +} + +/// Takes the initial sample within the bound, steps realtime to it, and holds the frequency +/// correction that the host's clock correction asks for. pub(crate) fn initial_sample(runtime: &Runtime<'_>) -> Result> { - let (_, sample) = { + let (result, sample) = { let _lock = lock_portb().map_err(portb_lock_failure)?; best_sample( runtime, SAMPLE_ATTEMPTS, - NSEC_PER_MSEC, + SAMPLE_BOUND_NS, runtime.generation(), ) .map_err(|error| error.to_string().into_bytes())? }; - step_clock(runtime, sample.theta).map_err(|error| describe(b"adjtimex step: ", error))?; + if result == SampleResult::Wide { + return Err(wide_sample(sample.epsilon)); + } + synchronize(runtime, &sample).map_err(|error| describe(b"adjtimex: ", error))?; Ok(sample) } +/// Formats the diagnostic of a selected sample of the uncertainty `epsilon`, wider than the +/// bound. +pub(crate) fn wide_sample(epsilon: i64) -> Vec { + format!("time sample uncertainty {epsilon} ns exceeds {SAMPLE_BOUND_NS} ns").into_bytes() +} + /// Writes `entropy` at the start of the restored entropy file, and reports a failure to write /// or close it with the file's path. pub(crate) fn write_entropy( @@ -880,8 +1022,9 @@ pub(crate) mod tests { use super::*; use crate::guest::sys::PortBackend; - use crate::guest::sys::x86::msr; + use crate::guest::time::NSEC_PER_MSEC; use crate::guest::time::events::CheckId; + use crate::guest::time::state::HostClockStatus; /// Privileged operations that record their requests and replay canned port input. pub(crate) struct FakeOps { @@ -903,9 +1046,24 @@ pub(crate) mod tests { pub(crate) kernel_log: Mutex>, /// Console output, in order. pub(crate) console: Mutex>, + /// What the frequency MSR reads: the declared TSC rate, or an error. + pub(crate) tsc_hz: Result, + /// The next TSC read, which each read advances by [`FakeOps::TSC_STEP`]. + pub(crate) tsc: AtomicU64, + /// What every read of the realtime clock returns. + pub(crate) realtime_ns: i64, } impl FakeOps { + /// The fake's declared TSC rate: 2 GHz, two cycles a nanosecond. + pub(crate) const TSC_HZ: u64 = 2_000_000_000; + /// The fake's first TSC read. + pub(crate) const TSC_START: u64 = 1_000_000; + /// The cycles by which each TSC read advances the next. + pub(crate) const TSC_STEP: u64 = 100; + /// The fake's realtime clock. + pub(crate) const REALTIME_NS: i64 = 1_790_841_600_000_000_000; + /// Returns a fake whose port reads return `input`. pub(crate) fn new(input: impl IntoIterator) -> Self { Self { @@ -918,6 +1076,9 @@ pub(crate) mod tests { power: AtomicUsize::new(0), kernel_log: Mutex::new(Vec::new()), console: Mutex::new(Vec::new()), + tsc_hz: Ok(Self::TSC_HZ), + tsc: AtomicU64::new(Self::TSC_START), + realtime_ns: Self::REALTIME_NS, } } @@ -929,6 +1090,50 @@ pub(crate) mod tests { } } + /// Returns the port input of a time sample of `generation` whose guest TSC is `tsc` and whose + /// host UTC is `utc_ns`, paired within `pairing_ns`, and which publishes `host`: the window's + /// four-byte reads. + pub(crate) fn sample_input( + generation: u32, + utc_ns: u64, + tsc: u64, + pairing_ns: u32, + host: HostClock, + ) -> Vec { + let unknown = |bound: Option| bound.unwrap_or(u32::MAX); + let mut sample = [0_u8; SAMPLE_SIZE]; + sample[..4].copy_from_slice(&[2, 0, host_status_byte(host.status), 0]); + sample[4..8].copy_from_slice(&generation.to_le_bytes()); + sample[8..16].copy_from_slice(&utc_ns.to_le_bytes()); + sample[16..24].copy_from_slice(&tsc.to_le_bytes()); + sample[24..28].copy_from_slice(&pairing_ns.to_le_bytes()); + sample[28..32].copy_from_slice(&host.disruption.to_le_bytes()); + sample[32..40].copy_from_slice(&host.correction.to_le_bytes()); + sample[40..44].copy_from_slice(&unknown(host.correction_bound).to_le_bytes()); + sample[44..48].copy_from_slice(&unknown(host.max_error_us).to_le_bytes()); + sample + .chunks(size_of::()) + .map(|word| u32::from_ne_bytes(word.try_into().unwrap())) + .collect() + } + + /// Returns the byte that encodes `status` in a time sample. + fn host_status_byte(status: HostClockStatus) -> u8 { + match status { + HostClockStatus::Unknown => 0, + HostClockStatus::Synchronized => 1, + HostClockStatus::FreeRunning => 2, + HostClockStatus::Unreliable => 3, + } + } + + /// Returns a guest TSC midway between the fake's reads around the selector write of the + /// `index`th time sample that it latches, counting from zero: each sample reads the TSC three + /// times. + pub(crate) fn sample_tsc(index: u64) -> u64 { + FakeOps::TSC_START + 3 * index * FakeOps::TSC_STEP + FakeOps::TSC_STEP / 2 + } + impl PortBackend for FakeOps { fn set_permission(&self, first: u16, count: u16, enable: bool) -> Result<(), Errno> { self.permissions @@ -957,6 +1162,14 @@ pub(crate) mod tests { } impl PrivilegedOps for FakeOps { + fn read_tsc(&self) -> u64 { + self.tsc.fetch_add(Self::TSC_STEP, Ordering::Relaxed) + } + + fn realtime_ns(&self) -> i64 { + self.realtime_ns + } + fn open_ports(&self) -> Result, PortError> { PortAccess::with_backend(self) } @@ -965,8 +1178,11 @@ pub(crate) mod tests { Ok(MsrDescriptor::Fake) } - fn read_msr(&self, _: &MsrDescriptor, _: u32) -> Result { - Err(Errno::EIO) + fn read_msr(&self, _: &MsrDescriptor, register: u32) -> Result { + match register { + msr::HV_TSC_FREQUENCY => self.tsc_hz, + _ => Err(Errno::EIO), + } } fn write_msr(&self, _: &MsrDescriptor, _: u32, _: u64) -> Result<(), Errno> { @@ -1006,8 +1222,8 @@ pub(crate) mod tests { } #[test] - fn packet_without_ranges_costs_exactly_twenty_four_data_reads() { - let ops = FakeOps::new(0..24); + fn packet_without_ranges_costs_exactly_twenty_two_data_reads() { + let ops = FakeOps::new(0..22); let ports = ops.open_ports().unwrap(); let mut header = [0; decoders::PACKET_HEADER_SIZE]; let mut entropy = [0; decoders::PACKET_ENTROPY_SIZE]; @@ -1015,6 +1231,13 @@ pub(crate) mod tests { portb_read(&ports, Port::PortbData, &mut entropy); ports.release().unwrap(); assert!(ops.input.lock().unwrap().is_empty()); + // A time sample costs twelve. + let ops = FakeOps::new(0..12); + let ports = ops.open_ports().unwrap(); + let mut sample = [0; SAMPLE_SIZE]; + portb_read(&ports, Port::PortbWindow, &mut sample); + ports.release().unwrap(); + assert!(ops.input.lock().unwrap().is_empty()); } #[test] @@ -1131,16 +1354,25 @@ pub(crate) mod tests { assert_eq!(sys::check_released(), Err(PortError::Revoke(Errno::ENOMEM))); } + /// The host clock of the fake's samples: synchronized, 100 ppm fast, to 1 ppb, within 2 ms. + const HOST: HostClock = HostClock { + status: HostClockStatus::Synchronized, + disruption: 1, + correction: 6_553_600, + correction_bound: Some(66), + max_error_us: Some(2_000), + }; + + /// Returns the port input of port B's status, with the time window, then of `samples`. + fn window_input(samples: impl IntoIterator>) -> Vec { + std::iter::once(u32::from(PORTB_TIME_WINDOW)) + .chain(samples.into_iter().flatten()) + .collect() + } + #[test] fn a_sample_from_another_generation_is_a_generation_error() { - let mut sample = [0_u8; SAMPLE_SIZE]; - sample[0] = 1; - sample[4..8].copy_from_slice(&7_u32.to_le_bytes()); - sample[8..16].copy_from_slice(&1_u64.to_le_bytes()); - let words = sample - .chunks(size_of::()) - .map(|word| u32::from_ne_bytes(word.try_into().unwrap())); - let ops = FakeOps::new(std::iter::once(u32::from(PORTB_TIME_WINDOW)).chain(words)); + let ops = FakeOps::new(window_input([sample_input(7, 1, sample_tsc(0), 0, HOST)])); let runtime = Runtime::new(&ops, false); let error = best_sample(&runtime, 1, NSEC_PER_MSEC, 6).unwrap_err(); assert_eq!( @@ -1157,12 +1389,138 @@ pub(crate) mod tests { ); } + #[test] + fn a_sample_pairs_its_guest_tsc_with_the_realtime_clock() { + // The sample's TSC is midway between the reads around the selector write, 100 cycles, + // 50 ns, before the realtime read, whose own reads are 100 cycles apart. + let utc_ns = FakeOps::REALTIME_NS as u64 + 1_000; + let ops = FakeOps::new(window_input([sample_input( + 0, + utc_ns, + sample_tsc(0), + 400, + HOST, + )])); + let runtime = Runtime::new(&ops, false); + let (result, sample) = best_sample(&runtime, 3, SAMPLE_BOUND_NS, 0).unwrap(); + assert_eq!(result, SampleResult::Accepted); + assert_eq!( + sample, + BestSample { + theta: 1_050, + epsilon: 400 + 1_000 + 25 + 1, + host: HOST, + frequency: 6_553_600, + limited: false, + } + ); + // One transaction latched one sample, after the rate's MSR read and a query of the + // frequency. + assert_eq!(ops.transactions(), 1); + assert_eq!( + ops.output.lock().unwrap().as_slice(), + [(Port::PortbStatus, SELECT_TIME_SAMPLE)] + ); + assert_eq!( + ops.adjustments.lock().unwrap().as_slice(), + [ClockAdjustment::Query] + ); + } + + #[test] + fn a_sample_outside_the_guest_tsc_reads_is_rejected() { + // The tolerance is the pairing and the skew bound, 1.4 µs: 2,800 cycles. + let ops = FakeOps::new(window_input([sample_input( + 0, + 1, + FakeOps::TSC_START - 2_801, + 400, + HOST, + )])); + let runtime = Runtime::new(&ops, false); + let error = best_sample(&runtime, 1, SAMPLE_BOUND_NS, 0).unwrap_err(); + assert_eq!( + error, + SampleError::Invalid(Some(TimeSampleError::Tsc { + tsc: FakeOps::TSC_START - 2_801, + before: FakeOps::TSC_START, + after: FakeOps::TSC_START + FakeOps::TSC_STEP, + })) + ); + assert_eq!(error.rejection(), SampleRejection::Tsc); + assert_eq!( + SampleError::Invalid(Some(TimeSampleError::Version(1))).rejection(), + SampleRejection::Invalid + ); + assert_eq!( + SampleError::NoTimeWindow(0).rejection(), + SampleRejection::Invalid + ); + } + + #[test] + fn a_wide_sample_gives_way_to_a_narrow_one() { + let utc_ns = FakeOps::REALTIME_NS as u64; + let wide = sample_input(0, utc_ns, sample_tsc(0), 150_000, HOST); + let ops = FakeOps::new(window_input([wide.clone()])); + let runtime = Runtime::new(&ops, false); + let (result, sample) = best_sample(&runtime, 1, SAMPLE_BOUND_NS, 0).unwrap(); + assert_eq!((result, sample.epsilon), (SampleResult::Wide, 151_026)); + let narrow = sample_input(0, utc_ns, sample_tsc(1), 600, HOST); + let ops = FakeOps::new(window_input([wide, narrow])); + let runtime = Runtime::new(&ops, false); + let (result, sample) = best_sample(&runtime, 3, SAMPLE_BOUND_NS, 0).unwrap(); + assert_eq!((result, sample.epsilon), (SampleResult::Accepted, 1_626)); + assert_eq!(ops.output.lock().unwrap().len(), 2); + } + + #[test] + fn a_sample_needs_the_declared_tsc_rate() { + for (rate, message) in [ + ( + Err(Errno::EACCES), + format!("MSR 0x40000022 read: {}", Errno::EACCES.desc()), + ), + ( + Ok(999_999), + "MSR 0x40000022 (TSC rate) is 999999 Hz".to_owned(), + ), + ] { + let mut ops = FakeOps::new(window_input([])); + ops.tsc_hz = rate; + let runtime = Runtime::new(&ops, false); + let error = best_sample(&runtime, 1, SAMPLE_BOUND_NS, 0).unwrap_err(); + assert_eq!(error, SampleError::TscRate(rate)); + assert_eq!(error.to_string(), message); + // The sample took no port transaction. + assert!(ops.permissions.lock().unwrap().is_empty()); + } + // The rate that the boot checks recorded needs no MSR. + let mut ops = FakeOps::new([]); + ops.tsc_hz = Err(Errno::EIO); + let runtime = Runtime::new(&ops, false); + runtime.set_tsc_hz(0); + assert!(runtime.tsc_hz().is_err()); + runtime.set_tsc_hz(3_000_000_000); + assert_eq!(runtime.tsc_hz(), Ok(3_000_000_000)); + // The MSR's rate is read once. + let ops = FakeOps::new([]); + let runtime = Runtime::new(&ops, false); + assert_eq!(runtime.tsc_hz(), Ok(FakeOps::TSC_HZ)); + assert_eq!(runtime.tsc_hz.load(Ordering::Relaxed), FakeOps::TSC_HZ); + } + #[test] fn privileged_adjustment_and_power_paths_are_injectable() { let ops = FakeOps::new([]); let runtime = Runtime::new(&ops, false); step_clock(&runtime, -1_128_000_000).unwrap(); - set_frequency(&runtime, 65_536).unwrap(); + let hold = FrequencyHold { + scaled_ppm: 65_536, + maximum_error_us: 3, + estimated_error_us: 1, + }; + hold_frequency(&runtime, hold).unwrap(); assert_eq!(current_frequency(&runtime), 0); assert_eq!( ops.adjustments.lock().unwrap().as_slice(), @@ -1171,7 +1529,7 @@ pub(crate) mod tests { seconds: -2, nanoseconds: 872_000_000, }, - ClockAdjustment::Frequency { scaled_ppm: 65_536 }, + ClockAdjustment::Frequency(hold), ClockAdjustment::Query, ] ); @@ -1179,6 +1537,39 @@ pub(crate) mod tests { assert_eq!(ops.power.load(Ordering::Relaxed), 195); } + #[test] + fn a_synchronization_steps_then_holds_the_host_frequency() { + let ops = FakeOps::new([]); + let runtime = Runtime::new(&ops, false); + let sample = BestSample { + theta: -1_128_000_000, + epsilon: 1_500, + host: HOST, + frequency: 6_553_600, + limited: false, + }; + synchronize(&runtime, &sample).unwrap(); + assert_eq!( + ops.adjustments.lock().unwrap().as_slice(), + [ + ClockAdjustment::Step { + seconds: -2, + nanoseconds: 872_000_000, + }, + // The maximum error adds the synchronized host's 2 ms. + ClockAdjustment::Frequency(FrequencyHold { + scaled_ppm: 6_553_600, + maximum_error_us: 2 + 2_000, + estimated_error_us: 2, + }), + ] + ); + assert_eq!( + String::from_utf8(wide_sample(100_001)).unwrap(), + "time sample uncertainty 100001 ns exceeds 100000 ns" + ); + } + #[test] fn repair_failures_report_restore_events_and_power_off_only_when_enforcing() { for (report_only, strict, prefix, power) in [ diff --git a/nvx-rs/src/guest/time/test_mode.rs b/nvx-rs/src/guest/time/test_mode.rs index f55c2b5b9..41a15a1c4 100644 --- a/nvx-rs/src/guest/time/test_mode.rs +++ b/nvx-rs/src/guest/time/test_mode.rs @@ -21,8 +21,8 @@ use std::sync::mpsc; use super::cpu_mask::CpuMask; use super::daemon::{self, Promoter}; use super::decoders::{ - self, GENERATION_ID_SIZE, PACKET_HEADER_SIZE, PacketHeader, RestorePacket, SAMPLE_SIZE, - TimeSample, + self, ClockRates, GENERATION_ID_SIZE, GuestReads, PACKET_HEADER_SIZE, PacketHeader, + RestorePacket, SAMPLE_SIZE, TimeSample, }; use super::events::{CheckId, DETAIL_MAX, EventSource, ViolationCode, escape_text, format_event}; use super::state::TimeState; @@ -166,16 +166,14 @@ fn test_packet(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandEr } let _ = write!( out, - "ok flags={} online={} ranges={} generation={} rate_deviation={} \ - frequency={} downtime_ns={} utc_ns={} generation_id={}", + "ok flags={} online={} ranges={} generation={} rate_deviation={} downtime_ns={} \ + generation_id={}", header.flags, header.online_vp_count, header.range_count, header.generation, header.rate_deviation, - decoders::restore_frequency(header.rate_deviation), header.downtime_ns, - header.utc_ns, decoders::format_hex(packet.generation_id()) ); for (start, length) in packet.ranges { @@ -190,6 +188,7 @@ fn test_sample(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandEr let [path, generation] = exactly(arguments)?; let bytes = record(&read_fixture(path)?, 0, SAMPLE_SIZE); let generation: u64 = number(generation)?; + let bound = |bound: Option| bound.map_or_else(|| "unknown".to_owned(), |b| b.to_string()); match TimeSample::parse(&bytes) { Err(error) => { let _ = writeln!(out, "error {error}"); @@ -198,21 +197,66 @@ fn test_sample(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandEr let _ = writeln!(out, "error generation {}", sample.generation); } Ok(sample) => { + let host = &sample.host; let _ = writeln!( out, - "ok flags={} generation={} utc_ns={}", - sample.flags, sample.generation, sample.utc_ns + "ok flags={} generation={} status={} utc_ns={} tsc={} pairing_ns={} \ + disruption={} correction={} correction_bound={} max_error_us={}", + sample.flags, + sample.generation, + host.status, + sample.utc_ns, + sample.tsc, + sample.pairing_ns, + host.disruption, + host.correction, + bound(host.correction_bound), + bound(host.max_error_us) ); } } Ok(()) } -/// Computes and prints a synthetic clock pairing. +/// Pairs a synthetic time sample's guest TSC and host UTC with synthetic reads of the guest's +/// clocks, and prints the offset and its uncertainty, or why the pairing failed. fn test_pair(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandError> { - let [t0, t1, utc_ns] = exactly(arguments)?; - let (theta, epsilon) = decoders::pair_clocks(number(t0)?, number(t1)?, number(utc_ns)?); - let _ = writeln!(out, "theta_ns={theta} epsilon_ns={epsilon}"); + let [ + utc_ns, + tsc, + pairing_ns, + before, + after, + realtime_ns, + realtime_tsc, + half_cycles, + tsc_hz, + frequency, + ] = exactly(arguments)?; + let sample = TimeSample { + flags: 0, + generation: 0, + utc_ns: number(utc_ns)?, + tsc: number(tsc)?, + pairing_ns: number(pairing_ns)?, + host: decoders::HostClock::default(), + }; + let reads = GuestReads { + before: number(before)?, + after: number(after)?, + realtime_ns: number(realtime_ns)?, + realtime_tsc: number(realtime_tsc)?, + realtime_half_cycles: number(half_cycles)?, + }; + let rates = ClockRates { + tsc_hz: number(tsc_hz)?, + frequency: number(frequency)?, + }; + match decoders::pair_sample(&sample, &reads, rates) { + Ok((theta, epsilon)) => writeln!(out, "theta_ns={theta} epsilon_ns={epsilon}"), + Err(error) => writeln!(out, "error {error}"), + } + .ok(); Ok(()) } @@ -365,10 +409,17 @@ fn test_timer_list(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), Comma } /// Plans and prints one step or slew correction, with the `adjtimex` modes and status that the -/// system layer would apply. +/// system layer would apply, for a discipline that holds a frequency correction and adds a host's +/// error. fn test_correction(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandError> { - let [theta, epsilon, speed] = exactly(arguments)?; - let correction = decoders::plan_correction(number(theta)?, number(epsilon)?, speed == b"fast"); + let [theta, epsilon, speed, frequency, host_error_us] = exactly(arguments)?; + let correction = decoders::plan_correction( + number(theta)?, + number(epsilon)?, + speed == b"fast", + number(frequency)?, + number(host_error_us)?, + ); let modes = correction.modes(); match correction { ClockAdjustment::Step { @@ -383,30 +434,34 @@ fn test_correction(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), Comma ClockAdjustment::Slew { offset_ns, time_constant, - maximum_error_us, - estimated_error_us, + hold, } => { let _ = writeln!( out, "slew modes=0x{modes:x} offset={offset_ns} status=0x{:x} \ - constant={time_constant} maxerror={maximum_error_us} \ - esterror={estimated_error_us}", - correction.status() + constant={time_constant} freq={} maxerror={} esterror={}", + correction.status(), + hold.scaled_ppm, + hold.maximum_error_us, + hold.estimated_error_us ); } - ClockAdjustment::Query | ClockAdjustment::Frequency { .. } => {} + ClockAdjustment::Query | ClockAdjustment::Frequency(_) => {} } Ok(()) } -/// Converts one restore rate deviation to the frequency correction and its parts per billion. +/// Converts one host clock correction, against the TSC counted at a declared rate, to the +/// frequency correction that the discipline holds, its parts per billion, and whether Linux's +/// limit clamped it. fn test_frequency(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), CommandError> { - let [rate_deviation] = exactly(arguments)?; - let frequency = decoders::restore_frequency(number(rate_deviation)?); + let [correction, tsc_hz] = exactly(arguments)?; + let (frequency, limited) = decoders::guest_frequency(number(correction)?, number(tsc_hz)?); let _ = writeln!( out, - "frequency={frequency} ppb={}", - decoders::frequency_to_ppb(frequency) + "frequency={frequency} ppb={} limited={}", + decoders::frequency_to_ppb(frequency), + u8::from(limited) ); Ok(()) } @@ -804,16 +859,52 @@ mod tests { #[test] fn malformed_hooks_fail_with_the_usage_status() { + /// The arguments of a well-formed `pair` hook. + const PAIR: [&[u8]; 11] = [ + b"pair", + b"1000000000000000000", + b"1000000", + b"500", + b"999000", + b"1002000", + b"999999999999000000", + b"1004000", + b"40", + b"2000000000", + b"0", + ]; let mut out = Vec::new(); assert_eq!(run(&[], &mut out), Err(CommandError::Usage)); assert_eq!(run(&[b"unknown"], &mut out), Err(CommandError::Usage)); assert_eq!(run(&[b"pair", b"1"], &mut out), Err(CommandError::Usage)); + // Time ABI 1's pairing of a realtime bracket is no longer a hook. + assert_eq!( + run(&[b"pair", b"1000", b"3000", b"5000"], &mut out), + Err(CommandError::Usage) + ); + let mut plus = PAIR; + plus[2] = b"+1000000"; + assert_eq!(run(&plus, &mut out), Err(CommandError::Usage)); + let mut negative = PAIR; + negative[3] = b"-500"; + assert_eq!(run(&negative, &mut out), Err(CommandError::Usage)); assert_eq!( - run(&[b"pair", b"1000", b"+3000", b"5000"], &mut out), + run(&[b"frequency", b"-40000000"], &mut out), Err(CommandError::Usage) ); assert_eq!( - run(&[b"frequency", b"2147483648"], &mut out), + run( + &[b"frequency", b"9223372036854775808", b"2100000000"], + &mut out + ), + Err(CommandError::Usage) + ); + assert_eq!( + run(&[b"frequency", b"0", b"-1"], &mut out), + Err(CommandError::Usage) + ); + assert_eq!( + run(&[b"correction", b"-5", b"10", b"slow"], &mut out), Err(CommandError::Usage) ); assert_eq!( @@ -847,8 +938,8 @@ mod tests { Err(CommandError::Usage) ); assert!(out.is_empty()); - assert_eq!(run(&[b"pair", b"1000", b"3000", b"5000"], &mut out), Ok(())); - assert_eq!(out, b"theta_ns=3000 epsilon_ns=1000\n"); + assert_eq!(run(&PAIR, &mut out), Ok(())); + assert_eq!(out, b"theta_ns=1002000 epsilon_ns=1524\n"); } #[test] @@ -970,27 +1061,42 @@ mod tests { #[test] fn corrections_print_the_kernel_modes() { let mut out = Vec::new(); - run(&[b"correction", b"128000000", b"10", b"fast"], &mut out).unwrap(); - run(&[b"correction", b"-5", b"10", b"slow"], &mut out).unwrap(); + run( + &[b"correction", b"128000000", b"10", b"fast", b"0", b"0"], + &mut out, + ) + .unwrap(); + run( + &[b"correction", b"-5", b"10", b"slow", b"65536", b"7"], + &mut out, + ) + .unwrap(); assert_eq!( String::from_utf8(out).unwrap(), "step modes=0x2100 sec=0 nsec=128000000\n\ - slew modes=0x203d offset=-5 status=0x2001 constant=6 maxerror=1 esterror=1\n" + slew modes=0x203f offset=-5 status=0x2081 constant=6 freq=65536 maxerror=8 \ + esterror=1\n" ); } #[test] fn ported_corrections_print_exact_results() { let mut out = Vec::new(); - run(&[b"correction", b"127999999", b"10002", b"fast"], &mut out).unwrap(); - run(&[b"correction", b"-500", b"20", b"slow"], &mut out).unwrap(); - run(&[b"correction", b"128000000", b"10", b"fast"], &mut out).unwrap(); - run(&[b"correction", b"-1128000000", b"10", b"slow"], &mut out).unwrap(); + for arguments in [ + [&b"127999999"[..], b"10002", b"fast"], + [b"-500", b"20", b"slow"], + [b"128000000", b"10", b"fast"], + [b"-1128000000", b"10", b"slow"], + ] { + let hook = [&[&b"correction"[..]][..], &arguments, &[b"0", b"0"]].concat(); + run(&hook, &mut out).unwrap(); + } assert_eq!( String::from_utf8(out).unwrap(), - "slew modes=0x203d offset=127999999 status=0x2001 constant=4 \ + "slew modes=0x203f offset=127999999 status=0x2081 constant=4 freq=0 \ maxerror=128011 esterror=11\n\ - slew modes=0x203d offset=-500 status=0x2001 constant=6 maxerror=1 esterror=1\n\ + slew modes=0x203f offset=-500 status=0x2081 constant=6 freq=0 maxerror=1 \ + esterror=1\n\ step modes=0x2100 sec=0 nsec=128000000\n\ step modes=0x2100 sec=-2 nsec=872000000\n" ); @@ -998,19 +1104,45 @@ mod tests { #[test] fn pairing_and_frequency_hooks_preserve_signed_arithmetic() { + let pair = |tsc: &[u8]| { + hook(&[ + b"pair", + b"1000000000000000000", + tsc, + b"500", + b"999000", + b"1002000", + b"999999999999000000", + b"1004000", + b"40", + b"2000000000", + b"0", + ]) + }; + assert_eq!(pair(b"1000000"), b"theta_ns=1002000 epsilon_ns=1524\n"); assert_eq!( - hook(&[b"pair", b"10000", b"10400", b"9000"]), - b"theta_ns=-1200 epsilon_ns=200\n" + pair(b"995999"), + b"error time sample TSC 995999 is outside the guest's reads 999000 to 1002000\n" ); - for (deviation, expected) in [ + for (correction, tsc_hz, expected) in [ ( b"13107200".as_slice(), - b"frequency=-13107200 ppb=-200000\n".as_slice(), + b"2100000000".as_slice(), + b"frequency=13107200 ppb=200000 limited=0\n".as_slice(), + ), + ( + b"39321600", + b"2100000000", + b"frequency=32768000 ppb=500000 limited=1\n", + ), + (b"0", b"2194843123", b"frequency=-3673 ppb=-56 limited=0\n"), + ( + b"-65536", + b"2000000000", + b"frequency=-65536 ppb=-1000 limited=0\n", ), - (b"-40000000", b"frequency=32768000 ppb=500000\n"), - (b"-65536", b"frequency=65536 ppb=1000\n"), ] { - assert_eq!(hook(&[b"frequency", deviation]), expected); + assert_eq!(hook(&[b"frequency", correction, tsc_hz]), expected); } } } diff --git a/nvx-rs/src/host/artifacts/mod.rs b/nvx-rs/src/host/artifacts/mod.rs index f177aaba5..962ca67e7 100644 --- a/nvx-rs/src/host/artifacts/mod.rs +++ b/nvx-rs/src/host/artifacts/mod.rs @@ -359,7 +359,7 @@ pub fn check_manifest(path: &Path) -> Result<()> { Ok(()) } else { Err(Error::unavailable(format!( - "NVX manifest {} does not declare the {} control contract", + "NVX manifest {} does not declare the compatible OpenVMM contracts ({})", path.display(), contract::CONTROL_CONTRACT_REVISION ))) diff --git a/nvx-rs/src/host/artifacts/tests.rs b/nvx-rs/src/host/artifacts/tests.rs index 62e304a43..c9a3867c8 100644 --- a/nvx-rs/src/host/artifacts/tests.rs +++ b/nvx-rs/src/host/artifacts/tests.rs @@ -8,6 +8,7 @@ use crate::ErrorKind; use crate::host::store::ArtifactIdentity; const COMPATIBLE_MANIFEST: &str = r#"{"openvmm":{"microvm_abi_version":2, + "time_abi_version":2,"restore_packet_version":5, "control_session_protocol_version":1, "control_contract_revision":"nvx-microvm-v2-control-v2"}}"#; @@ -200,7 +201,8 @@ fn incompatible_manifests_are_unavailable() { let manifest = directory.path().join(MANIFEST_NAME); fs::write( &manifest, - r#"{"openvmm":{"microvm_abi_version":2,"control_session_protocol_version":2, + r#"{"openvmm":{"microvm_abi_version":2,"time_abi_version":2, + "restore_packet_version":5,"control_session_protocol_version":2, "control_contract_revision":"nvx-microvm-v2-control-v2"}}"#, ) .unwrap(); diff --git a/nvx-rs/src/host/cli/download/install/tests.rs b/nvx-rs/src/host/cli/download/install/tests.rs index 1c308882d..ad7092c11 100644 --- a/nvx-rs/src/host/cli/download/install/tests.rs +++ b/nvx-rs/src/host/cli/download/install/tests.rs @@ -7,6 +7,7 @@ use serde_json::json; use super::*; use crate::contract::{ CONTROL_CONTRACT_REVISION, CONTROL_SESSION_PROTOCOL_VERSION, MICROVM_ABI_VERSION, + RESTORE_PACKET_VERSION, TIME_ABI_VERSION, }; /// Returns the manifest of a package that declares the guests of `sections` and, if @@ -15,6 +16,8 @@ fn manifest(sections: &[&str], compatible: bool) -> Value { let mut manifest = json!({ "openvmm": { "microvm_abi_version": MICROVM_ABI_VERSION, + "time_abi_version": TIME_ABI_VERSION, + "restore_packet_version": RESTORE_PACKET_VERSION, "control_session_protocol_version": CONTROL_SESSION_PROTOCOL_VERSION, "control_contract_revision": if compatible { CONTROL_CONTRACT_REVISION } else { "other" }, }, diff --git a/nvx-rs/src/host/launch.rs b/nvx-rs/src/host/launch.rs index 7db8b76dc..690446eee 100644 --- a/nvx-rs/src/host/launch.rs +++ b/nvx-rs/src/host/launch.rs @@ -362,7 +362,7 @@ pub fn kernel_parameters_problem(extra: &str, hostname: &str) -> Option } /// Returns whether extra kernel parameters may not hold `parameter`: the workload hostname's, -/// which the launch sets; `tsc`, which time ABI v1 governs; and those that configure the guest, +/// which the launch sets; `tsc`, which the time ABI governs; and those that configure the guest, /// whose names start with `nvx_` or, since the kernel matches them alike, `nvx-`. fn is_reserved(parameter: Parameter<'_>) -> bool { parameter.name.starts_with(b"nvx_") diff --git a/nvx-rs/src/host/template/manifest/tests.rs b/nvx-rs/src/host/template/manifest/tests.rs index d18ed2e57..6a6dd41ad 100644 --- a/nvx-rs/src/host/template/manifest/tests.rs +++ b/nvx-rs/src/host/template/manifest/tests.rs @@ -301,11 +301,15 @@ fn other_versions_of_the_format_are_unsupported() { unsupported, "boot layout version 1", ); - assert_refused( - |spec| spec.time().abi_version = 2, - unsupported, - "has a time contract of time ABI version 2", - ); + // A time contract of another time ABI version, such as a snapshot of time ABI version 1, + // whose guest estimated the host's clock correction, is refused. + for version in [1, 3] { + assert_refused( + |spec| spec.time().abi_version = version, + unsupported, + &format!("has a time contract of time ABI version {version}"), + ); + } // A manifest of another version is reported as such, even with fields that this build does // not know, such as a field that version 6 retired. let mut spec = ManifestSpec::template(); diff --git a/nvx-rs/src/host/template/test_support.rs b/nvx-rs/src/host/template/test_support.rs index 4587466b1..6f83f7af5 100644 --- a/nvx-rs/src/host/template/test_support.rs +++ b/nvx-rs/src/host/template/test_support.rs @@ -773,7 +773,7 @@ impl ManifestSpec { memory_block_size: 128 * MIB, expansion_ranges: vec![(512 * MIB, 1536 * MIB)], time: Some(TimeSpec { - abi_version: 1, + abi_version: crate::contract::TIME_ABI_VERSION as u32, tsc_hz: 2_100_000_000, tolerance_ppm: 250, apic_hz: 200_000_000, diff --git a/nvx-rs/tests/fixtures/template/README.md b/nvx-rs/tests/fixtures/template/README.md index 9d407578b..965cc20c1 100644 --- a/nvx-rs/tests/fixtures/template/README.md +++ b/nvx-rs/tests/fixtures/template/README.md @@ -13,19 +13,20 @@ against its own idea of them. | [`platform-mshv-network`](platform-mshv-network) | The same on MSHV, with a fixed NIC at `10.0.0.2/24`, its IPv6 identity, and an allow-list policy | Both are platform-tier captures for clones, after the invariant configuration -alone, with OpenVMM's test records of the time contract and of the CPU profile -`intel.icelake-sp.v1`. +alone, with OpenVMM's test records of the time contract, of time ABI version 2, +and of the CPU profile `intel.icelake-sp.v1`. ## How they were made [`generate.rs`](generate.rs) builds each manifest as a capture does, with `microvm_machine_contract` of OpenVMM's `openvmm_helpers`, checks it with OpenVMM's own validation, and encodes it with `mesh`. It wrote these files at -OpenVMM commit `79a501b63fcd2f150e94928d0ceba79cd2fe96c4`. +OpenVMM commit `75765b6608c3dd1f0be49fc4d5c1fc84a7b6ce96`. -To write them again, as when a new version of the snapshot format retires these, -append `generate.rs` to `openvmm/openvmm_helpers/src/snapshot/microvm.rs` in a -scratch checkout of the pinned OpenVMM, and run, from that checkout's root: +To write them again, as when a new version of the snapshot format or of the +time ABI retires these, append `generate.rs` to +`openvmm/openvmm_helpers/src/snapshot/microvm.rs` in a scratch checkout of the +pinned OpenVMM, and run, from that checkout's root: ```bash cargo xflowey restore-packages --no-compat-igvm diff --git a/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin b/nvx-rs/tests/fixtures/template/platform-kvm/manifest.bin index a93c1c45aa25566948bcc5a55db38c51cd822728..d5f14b79b34da516c61d874d7607d165ee08d06a 100644 GIT binary patch delta 14 VcmeyA`6+XQF$W{lW)qHmIsh?;1%3bk delta 14 VcmeyA`6+XQF$W{#W)qHmIsh?&1$_Vj diff --git a/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin b/nvx-rs/tests/fixtures/template/platform-mshv-network/manifest.bin index fe1b9f26609fdcca514e35cbdb45e1a4d098d2cc..02f46c7917a0d33b7ab7a6a62c2ee45fe0218a00 100644 GIT binary patch delta 14 VcmcbddpUQ52p1#MW>KzvIsh$11r7iJ delta 14 VcmcbddpUQ52p1#cW>KzvIsh#`1q}cI diff --git a/nvx-rs/tests/guest_tools.rs b/nvx-rs/tests/guest_tools.rs index 140780cde..01d3b6adf 100644 --- a/nvx-rs/tests/guest_tools.rs +++ b/nvx-rs/tests/guest_tools.rs @@ -1208,13 +1208,12 @@ fn time_packet( rate_deviation: i32, downtime_ns: u64, ) -> Vec { - let mut packet = Vec::with_capacity(32 + ranges.len() * 16 + 64); + let mut packet = Vec::with_capacity(24 + ranges.len() * 16 + 64); packet.extend_from_slice(b"OVR"); - packet.extend_from_slice(&[4, flags, online, ranges.len() as u8, 0]); + packet.extend_from_slice(&[5, flags, online, ranges.len() as u8, 0]); packet.extend_from_slice(&generation.to_le_bytes()); packet.extend_from_slice(&rate_deviation.to_le_bytes()); packet.extend_from_slice(&downtime_ns.to_le_bytes()); - packet.extend_from_slice(&TIME_UTC_NS.to_le_bytes()); for &(start, length) in ranges { packet.extend_from_slice(&start.to_le_bytes()); packet.extend_from_slice(&length.to_le_bytes()); @@ -1223,6 +1222,30 @@ fn time_packet( packet } +/// Returns a time sample of version 2, as OpenVMM encodes it: of `generation`, at host UTC +/// `utc_ns`, with the test-hook flag, the version, flags, status, and reserved byte that +/// `header` gives, guest TSC 123456789 paired within 640 ns, disruption 3, a correction of +/// -6552945 scaled ppm bounded by 66, and no maximum error. +#[cfg(target_arch = "x86_64")] +fn time_sample(header: [u8; 4], generation: u32, utc_ns: u64) -> Vec { + let mut sample = header.to_vec(); + sample.extend_from_slice(&generation.to_le_bytes()); + sample.extend_from_slice(&utc_ns.to_le_bytes()); + sample.extend_from_slice(&123_456_789_u64.to_le_bytes()); + sample.extend_from_slice(&640_u32.to_le_bytes()); + sample.extend_from_slice(&3_u32.to_le_bytes()); + sample.extend_from_slice(&(-6_552_945_i64).to_le_bytes()); + sample.extend_from_slice(&66_u32.to_le_bytes()); + sample.extend_from_slice(&u32::MAX.to_le_bytes()); + sample +} + +/// What the `sample` hook prints for [`time_sample`] of generation 4 at [`TIME_UTC_NS`]. +#[cfg(target_arch = "x86_64")] +const TIME_SAMPLE_OUTPUT: &[u8] = b"ok flags=8 generation=4 status=synchronized \ + utc_ns=1790841600123456789 tsc=123456789 pairing_ns=640 disruption=3 correction=-6552945 \ + correction_bound=66 max_error_us=unknown\n"; + #[cfg(target_arch = "x86_64")] fn time_fixture(directory: &Path, name: &str, contents: impl AsRef<[u8]>) -> PathBuf { let path = directory.join(name); @@ -1260,8 +1283,7 @@ fn time_test_mode_decodes_packets_samples_and_arithmetic() { Some(0), concat!( "ok flags=13 online=4 ranges=0 generation=3 ", - "rate_deviation=-13107200 frequency=13107200 ", - "downtime_ns=30000000000 utc_ns=1790841600123456789 ", + "rate_deviation=-13107200 downtime_ns=30000000000 ", "generation_id=000102030405060708090a0b0c0d0e0f\n" ) .as_bytes() @@ -1311,40 +1333,51 @@ fn time_test_mode_decodes_packets_samples_and_arithmetic() { .any(|window| window == b" ranges=0 ") ); - let mut sample = Vec::new(); - sample.extend_from_slice(&[1, 8]); - sample.extend_from_slice(&0_u16.to_le_bytes()); - sample.extend_from_slice(&4_u32.to_le_bytes()); - sample.extend_from_slice(&TIME_UTC_NS.to_le_bytes()); - let sample = time_fixture(directory.path(), "sample", sample); + let sample = time_fixture( + directory.path(), + "sample", + time_sample([2, 8, 1, 0], 4, TIME_UTC_NS), + ); assert_eq!( time_hook("sample", &sample, &["4"]), - ( - Some(0), - b"ok flags=8 generation=4 utc_ns=1790841600123456789\n".to_vec(), - Vec::new(), - ) + (Some(0), TIME_SAMPLE_OUTPUT.to_vec(), Vec::new()) ); assert_eq!( - run("nvx-time", ["test", "pair", "1000", "3000", "5000"]), + run( + "nvx-time", + [ + "test", + "pair", + "1000000000000000000", + "1000000", + "500", + "999000", + "1002000", + "999999999999000000", + "1004000", + "40", + "2000000000", + "0" + ] + ), ( Some(0), - b"theta_ns=3000 epsilon_ns=1000\n".to_vec(), + b"theta_ns=1002000 epsilon_ns=1524\n".to_vec(), Vec::new() ) ); assert_eq!( - run("nvx-time", ["test", "frequency", "-40000000"]), + run("nvx-time", ["test", "frequency", "39321600", "2100000000"]), ( Some(0), - b"frequency=32768000 ppb=500000\n".to_vec(), + b"frequency=32768000 ppb=500000 limited=1\n".to_vec(), Vec::new() ) ); assert_eq!( run( "nvx-time", - ["test", "correction", "-1128000000", "10", "slow"] + ["test", "correction", "-1128000000", "10", "slow", "0", "0"] ), ( Some(0), @@ -1429,7 +1462,10 @@ fn time_test_mode_checks_logs_timer_state_and_status() { "discontinuities=0\nlast_discontinuity=none\nlast_step_ns=0\n", "last_step_realtime_ns=0\nlast_downtime_ns=0\n", "last_downtime_source=none\nsynchronized=1\noffset_ns=-42\n", - "uncertainty_ns=1800\nfrequency_ppb=0\nsamples=0\n", + "uncertainty_ns=1800\nfrequency_ppb=0\nfrequency_limited=0\n", + "host_status=unknown\nhost_correction_ppb=0\n", + "host_correction_bound_ppb=unknown\nhost_max_error_us=unknown\n", + "host_disruption=0\nsamples=0\n", "rejected_samples=0\nlast_sample_error=none\nviolations=0\n" ) .as_bytes() @@ -1696,7 +1732,8 @@ fn time_test_mode_checks_packet_and_sample_boundaries() { 16, &(2_592_000_u64 * 1_000_000_000 + 1).to_le_bytes(), )); - rejected.push(mutate(24, &0_u64.to_le_bytes())); + // A packet of version 4, which carried host UTC, is refused. + rejected.push(mutate(3, &[4])); rejected.push(time_packet(0x02, 0, &[(0, 0)], 1, 0, 0)); rejected.push(time_packet(0x02, 0, &[(u64::MAX - 4095, 8192)], 1, 0, 0)); for (index, packet) in rejected.iter().enumerate() { @@ -1761,25 +1798,18 @@ fn time_test_mode_checks_packet_and_sample_boundaries() { .starts_with(b"error G_REPAIR_GENERATION ") ); - let sample_path = - |name: &str, version: u8, flags: u8, reserved: u16, generation: u32, utc: u64| { - let mut sample = vec![version, flags]; - sample.extend_from_slice(&reserved.to_le_bytes()); - sample.extend_from_slice(&generation.to_le_bytes()); - sample.extend_from_slice(&utc.to_le_bytes()); - time_fixture(directory.path(), name, sample) - }; - let good = sample_path("sample-good", 1, 8, 0, 4, TIME_UTC_NS); - assert_eq!( - time_hook("sample", &good, &["4"]).1, - b"ok flags=8 generation=4 utc_ns=1790841600123456789\n" - ); + let sample_path = |name: &str, header: [u8; 4], generation: u32, utc: u64| { + time_fixture(directory.path(), name, time_sample(header, generation, utc)) + }; + let good = sample_path("sample-good", [2, 8, 1, 0], 4, TIME_UTC_NS); + assert_eq!(time_hook("sample", &good, &["4"]).1, TIME_SAMPLE_OUTPUT); for (index, path) in [ - sample_path("sample-version", 2, 0, 0, 0, TIME_UTC_NS), - sample_path("sample-flags", 1, 1, 0, 0, TIME_UTC_NS), - sample_path("sample-reserved", 1, 0, 1, 0, TIME_UTC_NS), - sample_path("sample-utc", 1, 0, 0, 0, 0), - sample_path("sample-generation", 1, 0, 0, 3, TIME_UTC_NS), + sample_path("sample-version", [1, 0, 0, 0], 0, TIME_UTC_NS), + sample_path("sample-flags", [2, 1, 0, 0], 0, TIME_UTC_NS), + sample_path("sample-status", [2, 0, 4, 0], 0, TIME_UTC_NS), + sample_path("sample-reserved", [2, 0, 0, 1], 0, TIME_UTC_NS), + sample_path("sample-utc", [2, 0, 0, 0], 0, 0), + sample_path("sample-generation", [2, 0, 0, 0], 3, TIME_UTC_NS), ] .iter() .enumerate() diff --git a/openvmm b/openvmm index 3d7512af6..858b2f93a 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 3d7512af653717aa9123595e993a56cb8587106a +Subproject commit 858b2f93aa78d35df6fdd217322a7f4cfb57b080 diff --git a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh index f646268c8..4708517aa 100644 --- a/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh +++ b/scripts/nvx_tools/microvm_test_scripts/snapshot-core.sh @@ -36,7 +36,7 @@ echo "NVX-SNAPSHOT-CPU-$((process_cpu_restored - process_cpu_before))-$((thread_ echo "NVX-SNAPSHOT-TIMER-WAIT-$((timer_wait_after - timer_wait_before))" generation_id_after="$(/sbin/nvx-time generation-id)" || fail 50 [ "$generation_id_before" != "$generation_id_after" ] || fail 51 -# nvx-time capture consumed restore packet v4 and, for an untiered restore +# nvx-time capture consumed the restore packet and, for an untiered restore # without targets, left its 64 entropy bytes to the caller. The generation ID # is their first 16 bytes. entropy=/run/nvx/restore-entropy diff --git a/scripts/nvx_tools/time_abi.py b/scripts/nvx_tools/time_abi.py index 657ab3137..04a516a48 100644 --- a/scripts/nvx_tools/time_abi.py +++ b/scripts/nvx_tools/time_abi.py @@ -1,4 +1,4 @@ -"""Harness checks of the NVX time ABI v1 guest obligations. +"""Harness checks of the NVX time ABI v2 guest obligations. A conforming guest reports each conformance check with an ``NVX-TIME-ABI`` marker and each runtime violation with an ``NVX-TIME-ABI-VIOLATION`` event, and @@ -794,7 +794,7 @@ def require_status(self, context: str) -> None: def _missing_reason(self) -> str: if self.report_only: return "the guest ran nvx-time in report-only mode" - return "the guest image or OpenVMM does not implement time ABI v1" + return "the guest image or OpenVMM does not implement time ABI v2" def require_boot( self, diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 99f408b73..0e43ae6e0 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -6854,7 +6854,7 @@ def test_snapshot_core_checks_that_restored_processes_cannot_reach_the_ports(sel self.assertEqual(script.count("|| fail 55"), 1) def test_snapshot_core_reads_the_entropy_that_nvx_time_kept(self): - # Restore packet v4 is consumed by nvx-time capture, which leaves the + # The restore packet is consumed by nvx-time capture, which leaves the # entropy of an untiered restore in /run/nvx/restore-entropy. shell = _posix_shell() if shell is None: diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index f5e99d554..55bbe1da6 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -771,6 +771,8 @@ def _write_release_fixture( "distribution": {"name": "nvx", "version": "0.1.0"}, "openvmm": { "microvm_abi_version": 2, + "time_abi_version": 2, + "restore_packet_version": 5, "control_session_protocol_version": 1, "control_contract_revision": "nvx-microvm-v2-control-v2", }, @@ -17494,6 +17496,8 @@ def test_manifest_tracks_every_kernel_patch(self): manifest["openvmm"], { "microvm_abi_version": 2, + "time_abi_version": 2, + "restore_packet_version": 5, "control_session_protocol_version": 1, "control_contract_revision": "nvx-microvm-v2-control-v2", }, @@ -24859,7 +24863,7 @@ def test_snapshot_restore_command_separates_capacity_from_online_target(self): self.assertEqual(command[command.index("--processors") + 1], "8") self.assertEqual(command[command.index("--restore-processors") + 1], "4") - # Restore packet v4 gives every restore fresh entropy. + # The restore packet gives every restore fresh entropy. self.assertNotIn("--restore-entropy", command) def test_snapshot_restore_command_sets_memory_target_separately(self): @@ -24928,7 +24932,7 @@ def test_parses_dd_rates_and_network_gateway(self): self.assertEqual(benchmark.network_gateway("10.0.0.2/24"), "10.0.0.1") def test_benchmark_command_lines_carry_no_clock_tuning(self): - # Under time ABI v1 the hypervisor identity gives Linux its TSC and + # Under the time ABI the hypervisor identity gives Linux its TSC and # LAPIC rates and trust, so the lifecycle line tunes no clock and the # cold-start clocksource variant selects tsc on every backend. for token in ( diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index fe35e6530..879128c76 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -873,7 +873,7 @@ def test_status_queries_require_the_line_of_the_launch_kind(self): with self.assertRaisesRegex( TimeAbiFailure, "did not report the NVX-TIME-ABI boot marker before nvx-time status " - "exited; the guest image or OpenVMM does not implement time ABI v1", + "exited; the guest image or OpenVMM does not implement time ABI v2", ): quiet.feed(b"NVX-TIME-STATUS-EXIT status=0\r\n") From 02d4f932c82cdd82afaf28a467f884c98d29be18 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 11:31:52 -0700 Subject: [PATCH 2/5] doc: specify time ABI 2's time samples and restore packet 5 The architecture guide described time ABI 1: a 16-byte time sample that the guest pairs with its own clock within 1 ms, or 50 us for its discipline, a guest that estimates host UTC's rate itself, and restore packet 4, which carries host UTC latched at its first selection. Describe time ABI 2, which the previous commit pins. 3.2, Clock and time: - Table 3.5: CPUID leaf 0x40000002 reports time ABI 2.0 in EBX. - 3.2.4: the host clock correction of every time sample includes a restore's rate deviation, so that the guest compensates it from its repair on. - 3.2.6, Backend obligations: every backend knows the guest TSC's offset from the host TSC before any processor runs, at cold boot and at every restore, so that the VMM pairs each sample's host UTC with the guest TSC. - 3.2.9, Time samples, rewritten: Figure 3.1 lays out the 48-byte version-2 sample field by field; the text gives the guest's pairing, the bracket check and the uncertainty of the offset, how it computes the adjtimex frequency that makes CLOCK_REALTIME run at host UTC's rate from the correction and the TSC rate, and the 500 ppm limit that it reports. Table 3.10 bounds every use, boot, restore, and the discipline, at 100 us and three attempts. The samples are no paravirtual clock: they feed only the wall-clock discipline, and the TSC stays the guest's clocksource. - 3.2.10, Guest obligations: the time state's new fields, a boot that takes a sample within the bound before readiness, steps to host UTC, and holds the frequency that the host's correction gives, and a discipline that holds that frequency at every accepted sample, so that the kernel's PLL only slews the phase, and returns to its fast cadence when the disruption counter changes. 5.5, Restore protocol: Figure 5.7 lays out restore packet 5, whose header is 24 bytes and carries no UTC, 88 bytes without memory ranges, and flag 3 also covers the host clock rate's test hook. The guest's repair steps take a time sample in the packet's port transaction and set the clock and its rate from it. 3.1, Platform control device, says that time samples pair host UTC with the guest TSC and carry the host's correction, and that the guest checks the pairing against TSC reads around the port write. 1.2, Non-goals, names kvmclock, Hyper-V's reference TSC page, and VMClock among the excluded paravirtual clocks, and says what the time samples feed instead. The build guide's description of nvx-time names restore packet 5 and the version-2 sample parsers and the correction that the discipline applies. Part of #1009 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/build.md | 7 +- doc/design/1.2-non-goals.md | 7 +- doc/design/3.1-platform-control-device.md | 11 +- doc/design/3.2-clock-and-time.md | 177 ++++++++++++++++------ doc/design/5.5-restore-protocol.md | 42 ++--- 5 files changed, 169 insertions(+), 75 deletions(-) diff --git a/doc/build.md b/doc/build.md index 43625cae0..bbc3dec8a 100644 --- a/doc/build.md +++ b/doc/build.md @@ -213,9 +213,10 @@ hypervisor identity, and runs the cross-vCPU warp test that the [time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs skew bound. `nvx-time` implements the time ABI's [guest obligations](design/3.2-clock-and-time.md#3210-guest-obligations): the conformance -checks, the violation watcher, the restore packet v4 and time-sample parsers, -the snapshot agent's time steps, and the wall-clock discipline. Before -shell-ready it only steps the clock to host UTC; the other boot checks run in +checks, the violation watcher, the restore packet v5 and time-sample v2 +parsers, the snapshot agent's time steps, and the wall-clock discipline, which +applies the host's clock correction. Before shell-ready it only steps the +clock to host UTC and sets its rate; the other boot checks run in the background, and the production console shows no time ABI line except a violation event. `nvx-time status` prints every recorded check (boot, capture, and restore) and the runtime state on demand, after waiting for pending diff --git a/doc/design/1.2-non-goals.md b/doc/design/1.2-non-goals.md index b45e19167..35b920ff0 100644 --- a/doc/design/1.2-non-goals.md +++ b/doc/design/1.2-non-goals.md @@ -11,7 +11,12 @@ The architecture deliberately excludes: - General CPU or memory hotplug after readiness. - Restore across hypervisor backends or CPU generations, live migration, and capture-and-continue. -- Scaling of the TSC, trapping of RDTSC, paravirtual clocks, and a virtual PMU. +- Scaling of the TSC, trapping of RDTSC, paravirtual clocks, such as kvmclock, + Hyper-V's reference TSC page, or a VMClock device, and a virtual PMU. The + TSC is the guest's only clocksource; the + [time samples](3.2-clock-and-time.md#329-time-samples) that pair host UTC + with it and publish the host's clock correction feed only the guest's + wall-clock discipline. - Serialization of live network flows, native host handles, or the contents of a live host filesystem export. - Multi-container orchestration, including pod infrastructure, pause diff --git a/doc/design/3.1-platform-control-device.md b/doc/design/3.1-platform-control-device.md index 2f3584378..297350d0b 100644 --- a/doc/design/3.1-platform-control-device.md +++ b/doc/design/3.1-platform-control-device.md @@ -34,8 +34,9 @@ machine omits, and carries: which gives a restored guest what it needs to repair its clocks, processors, and memory, and the fresh random bytes from which it [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh). -- Host time samples, which the guest pairs with its own clock to set and - discipline its wall clock. +- Host time samples, which pair host UTC with the guest TSC and carry the + host's clock correction, from which the guest sets and disciplines its wall + clock. - The generation ID, a random identifier unique to each VM process. Every restored process receives a new one, even when several clones run from one snapshot, so the guest can detect that it has been restored and @@ -55,9 +56,9 @@ queues in guest memory, or interrupt line. Each output byte is one port write, and the guest polls a status port for input. Every access to the channel's ports is a synchronous exit that the VMM completes before the guest continues, so a host time that the VMM records during an access lies between two guest -clock reads taken around it. This bounds the error of -[time samples](3.2-clock-and-time.md#329-time-samples) and of the restore -packet's UTC field. +TSC reads taken around it. The guest checks each +[time sample](3.2-clock-and-time.md#329-time-samples)'s pairing of host UTC +with its TSC against them. As Table 3.2 shows, the byte channel's three ports form one device: a data port, a status and record-selector port, and a time-sample window. diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index ab76b6d4e..2ffbda440 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -71,7 +71,7 @@ through which CPUID reports its results (EAX, EBX, ECX, and EDX). | --- | --- | --- | --- | --- | | `0x40000000` | `0x40000005` (maximum leaf) | `0x7263694d` (`"Micr"`) | `0x666f736f` (`"osof"`) | `0x76482074` (`"t Hv"`) | | `0x40000001` | `0x31237648` (`"Hv#1"`) | 0 | 0 | 0 | -| `0x40000002` | `0x0058564e` (`"NVX"`) | `0x00010000` (time ABI 1.0) | 0 | 0 | +| `0x40000002` | `0x0058564e` (`"NVX"`) | `0x00020000` (time ABI 2.0) | 0 | 0 | | `0x40000003` | `0x00008860` | 0 | 0 | `0x00000100` | | `0x40000004` | 0 | `0xffffffff` | 0 | 0 | | `0x40000005` | `C` | `C` | 0 | 0 | @@ -185,8 +185,9 @@ The boundary is accepted. The manifest records the tolerance, and any other recorded value is rejected. Within the tolerance, the TSC is neither scaled nor trapped, the frequency MSR keeps returning `F_s`, and guest clocks run fast or slow by at most the deviation. The restore packet reports that -deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, so -that the guest can compensate: +deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, and +the host clock correction of every [time sample](#329-time-samples) includes +it, so that the guest compensates it from its repair on: ```text rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s) @@ -254,6 +255,9 @@ must: hypervisor-specific emulator or its saved state. - Pair the first processor's TSC with a host time sample within 100 µs for the capture anchor, keeping the tightest of at most 64 samples. +- Know the guest TSC's offset from the host TSC before any processor runs, at + cold boot and at every restore, so that the VMM pairs each + [time sample](#329-time-samples)'s host UTC with the guest TSC. - Set every instantiated processor's TSC at one host instant and verify it by read-back before any processor runs. - Keep live cross-processor skew within 1 µs after release. @@ -316,40 +320,111 @@ quiesce as elapsed time, as it would a long preemption. ## 3.2.9 Time Samples -Figure 3.1 shows the layout of the 16-byte sample that writing `0xa7` to the -byte-channel selector latches into the time window. +Figure 3.1 shows the layout of the 48-byte sample that writing `0xa7` to the +byte-channel selector latches into the time window. The VMM takes the sample +while it handles the selector write: it reads host UTC, pairs it with the +guest TSC, and reads how the host's time service disciplines host UTC. ```text offset size field - 0 1 version = 1 + 0 1 version = 2 1 1 flags bit 3 mirrors the restore packet's override flag; other bits zero - 2 2 reserved = 0 + 2 1 status host UTC: 0 unknown, 1 synchronized, 2 free-running, 3 unreliable + 3 1 reserved = 0 4 4 generation u32, little-endian, g of this VM process 8 8 utc_ns u64, little-endian, host UTC at the latch + 16 8 tsc u64, little-endian, guest TSC at the instant of utc_ns + 24 4 pairing_ns u32, little-endian, bound on the distance of those instants + 28 4 disruption u32, little-endian, changes with each host clock disruption + 32 8 correction i64, little-endian, ppm scaled by 2^16 + 40 4 correction_bound u32, little-endian, ppm scaled by 2^16, 0xffffffff if unknown + 44 4 max_error_us u32, little-endian, 0xffffffff if unknown ``` **Figure 3.1.** Time-sample layout. -The VMM cannot observe a running processor's TSC on every backend, so the -guest pairs host UTC with its own clock. It reads its realtime clock as `t0` -immediately before the selector write and as `t1` immediately after it. The -host instant lies in `[t0, t1]`, so: +The guest TSC is never scaled, so it is the host TSC plus an offset that is +fixed for the life of a VM process. The backend reads that offset, or +measures it by bracketing a read of the first processor's TSC with host TSC +reads, before any processor runs, at cold boot and at every restore. Each +sample brackets a read of host UTC with host TSC reads, keeping the tightest +of up to 16 brackets, and converts the host TSC to the guest TSC with the +offset. `pairing_ns` adds the offset's uncertainty to half the bracket. + +`correction` is how fast host UTC runs against the guest TSC counted at the +declared rate `F`, minus one, in the kernel's frequency-adjustment unit: + +```text +correction = (1 + c_os) * F / F_d - 1 +``` + +where `c_os` is the time service's correction of the host clock against the +TSC at the native rate `F_d`. On Linux, it is the frequency and tick that +`adjtimex` reports. On Windows, `GetSystemTimeAdjustmentPrecise` reports the +performance counter's frequency as its increment, and the counter's ticks in +each second of host UTC as its adjustment, so `c_os` is the increment over the +adjustment, minus one, while a time service adjusts the clock, and zero +otherwise. After a restore, `F` is `F_s`, so the correction includes the +restore's rate deviation. +`correction_bound` bounds the correction's error: 1 ppb of rounding when the +host clock and `F_d` come from one calibration, as with Linux's `tsc` +clocksource and KVM, or Hyper-V's reference TSC and MSHV or WHP; the kHz to +which one of them is known otherwise; and unknown when the host clock may not +follow the TSC. + +`status` is the host's own account of its clock. On Linux, `adjtimex` +reports a clock fault as unreliable and an unsynchronized clock as +free-running; on Windows, a clock that no time service adjusts is +free-running. `max_error_us` is the maximum error that Linux's time service +sets, and unknown on Windows. `disruption` starts at zero in each VM process +and changes when host UTC steps by at least 1 ms more than the correction +explains, when the host resumes from a suspension, and when the guest TSC's +offset from the host TSC changes. + +These values feed only the guest's wall-clock discipline. The TSC stays the +guest's only clocksource: the time sample is not a paravirtual clock, which +[1.2 Non-Goals](1.2-non-goals.md) excludes, as it excludes a VMClock device. + +The guest checks each sample's pairing against its own TSC. It reads its TSC +as `g0` immediately before the selector write and as `g1` immediately after +it, and then its realtime clock as `r`, in nanoseconds, between two more TSC +reads, whose midpoint is `c` and half distance `w`. The host reads its UTC +while it handles the write, so a sample whose `tsc` lies outside +`[g0 - p, g1 + p]`, where `p` is `pairing_ns` plus the 1 µs skew bound, in +cycles at `F`, is invalid. The kernel counts the TSC at `F_k`, in Hz, `F` +rounded down to the kHz, with its frequency correction `f`, so the guest's +realtime clock at `tsc` is `r` less `d`, the nanoseconds between `tsc` and +`c`: ```text -theta = utc_ns - (t0 + t1) / 2 uncertainty epsilon = (t1 - t0) / 2 +d = (c - tsc) * (1 + f) * 1_000_000_000 / F_k +skew_ns = 1_000 +theta = utc_ns - (r - d) +epsilon = pairing_ns + skew_ns + w * 1_000_000_000 / F_k + |d| * 2000 ppm ``` -The same bracket applies to the UTC sample in the restore packet. -As Table 3.10 shows, the guest accepts a pairing only if `epsilon` is within -the bound for its use. +The skew term bounds a read on another guest CPU than VP 0, whose TSC OpenVMM pairs with host +UTC. The last term bounds how far a slew below the step threshold can move the realtime clock's +rate over that time. The restore packet carries no host UTC: +repair takes a time sample instead. As Table 3.10 shows, the guest accepts a +pairing only if `epsilon` is within the bound for its use. **Table 3.10.** Time-pairing uncertainty bounds. -| Use | Bound | Attempts | When every attempt exceeds the bound | +| Use | Bound | Attempts | When every attempt is invalid or exceeds the bound | | --- | --- | --- | --- | -| Restore repair | 1 ms | 3 | Repair fails, and the guest powers off | -| Initial synchronization at boot | 1 ms | 3 | Conformance fails, and the guest powers off | -| Wall-clock discipline | 50 µs | 3 | The poll is skipped; never fatal | +| Restore repair | 100 µs | 3 | Repair fails, and the guest powers off | +| Initial synchronization at boot | 100 µs | 3 | Conformance fails, and the guest powers off | +| Wall-clock discipline | 100 µs | 3 | The poll is skipped; never fatal | + +The guest's frequency correction runs its wall clock at host UTC's rate: + +```text +f = (1 + correction) * F_k / F - 1 +``` + +The kernel limits `f` to ±500 ppm; the guest holds a correction beyond that +limit at the limit, and reports that it did. ## 3.2.10 Guest Obligations @@ -359,22 +434,26 @@ Production paths keep the console quiet; status goes to a state file that the workload can read through a read-only mount. The state file reports the outcome of each phase's checks, the declared rates, the generation, the last downtime and its source, and the wall clock's discontinuities, -synchronization, offset, uncertainty, and frequency correction. +synchronization, offset, uncertainty, and frequency correction, with whether +the kernel's limit held it, and the status, correction, bounds, and +disruption counter of host UTC that the latest accepted sample published, so +that a workload can see when its host's clock is not synchronized or wanders. **Conformance.** Before readiness, the guest only takes a time sample within -the bound and steps its clock to host UTC, so the workload starts on host -time. It runs every other boot check asynchronously after readiness, starting -at idle priority, so the checks never delay readiness or take a processor from -the workload. The checks cover the identity leaves and explicit zero leaves, -the CPU time bits on every CPU, the identity MSRs, the kernel's platform, -clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence -of forbidden clock parameters among those that the kernel reads from its -command line, stall-detector settings, a running violation watcher, and kernel -W+X integrity. Capture reruns the clocksource, tick-device, and watcher checks. -Restore reruns them too, checks the declared rate in the frequency MSR again, -and checks the identity leaves, the CPU time bits, and the CPU flags of every -processor that it brings online. Every check is fail-fast: a failure powers -the guest off even if the workload is running. +the bound, steps its clock to host UTC, and holds the frequency correction +that the sample's host clock correction gives, so the workload starts on host +time and runs at its rate. It runs every other boot check asynchronously after +readiness, starting at idle priority, so the checks never delay readiness or +take a processor from the workload. The checks cover the identity leaves and +explicit zero leaves, the CPU time bits on every CPU, the identity MSRs, the +kernel's platform, clocksource, and LAPIC-timer selection, CPU flags, tick +devices, the absence of forbidden clock parameters among those that the +kernel reads from its command line, stall-detector settings, a running +violation watcher, and kernel W+X integrity. Capture reruns the clocksource, +tick-device, and watcher checks. Restore reruns them too, checks the declared +rate in the frequency MSR again, and checks the identity leaves, the CPU time +bits, and the CPU flags of every processor that it brings online. Every check +is fail-fast: a failure powers the guest off even if the workload is running. **Violation watcher.** The watcher follows the kernel log from before the boot checks, so no record is missed. TSC instability; clocksource instability, @@ -395,19 +474,25 @@ the guest releases the suppression anyway so that the kernel reports the real stall. **Wall-clock discipline.** The guest keeps its realtime clock on host UTC -through the kernel's phase-locked loop. It polls every 16 seconds for the -first four polls after boot or a restore, then every 64 seconds, and takes up -to three samples per poll. An offset of at least 128 ms is stepped and counted -as a discontinuity; a smaller one is slewed with error bounds derived from the -offset and its uncertainty, which marks the clock synchronized for -applications. The kernel limits the frequency correction to ±500 ppm, of which -the rate tolerance uses at most 250 ppm. The other 250 ppm is the budget for -the host's own clock correction, which the discipline applies too, because -host UTC follows the host's time service: a host whose TSC runs more than -250 ppm from its declared rate against host UTC does not -[qualify](../ci.md#host-qualification) for the contract, and a larger -correction would leave an offset that grows until the discipline steps it. A -sample whose generation differs from the recorded one is discarded, because a +through the kernel's phase-locked loop, with its frequency held: each accepted +sample sets the kernel's frequency correction from the sample's host clock +correction, which carries a restore's rate deviation, and the loop only slews +the phase, so the guest never estimates the host's rate. It polls every 16 +seconds for the first four polls after boot, a restore, or a change of the +sample's disruption counter, then every 64 seconds, and takes up to three +samples per poll. An offset of at least 128 ms is stepped and counted as a +discontinuity; a smaller one is slewed. Either sets error bounds derived from +the offset and its uncertainty, to which the maximum error adds host UTC's +while the host's time service synchronizes it, which marks the clock +synchronized for applications. The kernel limits the frequency correction to +±500 ppm, of which the rate tolerance uses at most 250 ppm. The other 250 ppm +is the budget for the host's own clock correction, which the discipline +applies too, because host UTC follows the host's time service: a host whose TSC +runs more than 250 ppm from its declared rate against host UTC does not +[qualify](../ci.md#host-qualification) for the contract. A correction beyond +the kernel's limit is held at the limit and reported in the state file, so an +offset can grow until the discipline steps it. A sample whose generation +differs from the recorded one is discarded, because a restore happened and restore repair resets the discipline. The discipline follows host wall-clock steps and never powers off the guest. Restore repair and the discipline step the clock through the kernel, so a workload can diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index f7992d794..57a98df15 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -199,7 +199,7 @@ every tier and for untiered snapshots. ```text offset size field 0 3 magic "OVR" - 3 1 version = 4 + 3 1 version = 5 4 1 flags 5 1 online_vp_count 0 = no processor target, else 1, 2, 4, or 8 6 1 memory_range_count 0 unless MEMORY_TARGET @@ -207,9 +207,8 @@ offset size field 8 4 generation u32, little-endian, g >= 1 12 4 rate_deviation i32, little-endian, ppm scaled by 2^16 16 8 downtime_ns u64, little-endian, D - 24 8 utc_ns u64, little-endian, host UTC at selection - 32 16*n ranges n x { u64 gpa_start, u64 length }, LE - 32+16n 64 entropy + 24 16*n ranges n x { u64 gpa_start, u64 length }, LE + 24+16n 64 entropy ``` **Figure 5.7.** Restore packet layout. @@ -221,17 +220,19 @@ offset size field | 0 | `DOWNTIME_UTC` | `D` came from the UTC delta; clear means same-boot host monotonic time | | 1 | `MEMORY_TARGET` | An explicit RAM target was requested; the range count is valid and may be zero | | 2 | `ACK_REQUIRED` | Host input is gated, and the guest must acknowledge through the snapshot port after repair | -| 3 | `TEST_HOOKS` | Host overrides of the downtime, UTC, rate deviation, time samples, or CPU profile are active | +| 3 | `TEST_HOOKS` | Host overrides of the downtime, UTC, rate deviation, host clock rate, time samples, or CPU profile are active | | 4-7 | | Zero; the guest rejects a packet with any of them set | -The UTC field is latched when the guest first selects the packet; every other -field is sealed before the first restored processor runs. A packet without -ranges is 96 bytes. The packet is authoritative, and the status bits mirror -it; the acknowledgement flag spares the guest from inferring gating from the -tier and targets. A VM process exposes at most one packet, and reading its -last byte consumes it, so a guest captured again in the same process never -sees an earlier packet. The guest also ignores a packet whose generation -equals its recorded generation. +Every field is sealed before the first restored processor runs. The packet +carries no host UTC: the guest takes a +[time sample](3.2-clock-and-time.md#329-time-samples) after it, which pairs +host UTC with the guest TSC on the host. A packet without ranges is 88 bytes. +The packet is authoritative, and the status bits mirror it; the +acknowledgement flag spares the guest from inferring gating from the tier and +targets. A VM process exposes at most one packet, and reading its last byte +consumes it, so a guest captured again in the same process never sees an +earlier packet. The guest also ignores a packet whose generation equals its +recorded generation. ## 5.5.6 Generations @@ -255,13 +256,14 @@ runs before the acknowledgement: 1. Read the byte-channel status, whose restore-packet bit is always set after a restore. -2. Bracket the packet selection with realtime clock reads, read the packet - four bytes per access, and validate its magic, version, reserved bits, - counts, generation, and generation ID. -3. Set the wall clock: derive the offset and its uncertainty from the - bracket, take up to two further time samples if the uncertainty exceeds - 1 ms, step the clock by the offset, and set the frequency correction to the - negated rate deviation, clamped to ±500 ppm. +2. Select the packet, read it four bytes per access, and validate its magic, + version, reserved bits, counts, generation, and generation ID. +3. Set the wall clock: in the same port transaction as the packet, take up to + three [time samples](3.2-clock-and-time.md#329-time-samples) of the + packet's generation, until one pairs host UTC with the guest TSC within + 100 µs, step the clock by its offset, and hold the frequency correction + that its host clock correction gives, which carries the rate deviation, + within ±500 ppm. 4. Activate processors and memory as requested. 5. Refresh entropy and identity, as [Entropy Refresh](#558-entropy-refresh) describes. From 7f19c779d3d1ffeb46961587ecb3d1b53148aeed Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 11:33:52 -0700 Subject: [PATCH 3/5] doctor: require a synchronized host clock correction in H4 Since this branch split from dev, H4's rate qualification gained the 250 ppm budget that time ABI 2 leaves for the host's clock correction, a retry when chronyd's skew straddles that limit, a warning above the old 100 ppm bound, the kernel PLL's estimate when chronyd is absent, and the average correction since boot as evidence (#1008). Keep all of those checks and their evidence. Time ABI 2 also changes what H4 must learn directly from the host. Every time sample publishes the host's clock correction, which the guest holds, so a qualified host must have a time service maintaining that correction. H4 now additionally: - Fails unless the host publishes a synchronized correction, read as OpenVMM reads it: on Linux, adjtimex reports neither STA_UNSYNC nor STA_CLOCKERR; on Windows, GetSystemTimeAdjustmentPrecise reports an applied adjustment. - Reports the status, correction, and host UTC's maximum error in its detail and as host_clock_status and host_correction_ppm summary facts. The direct correction is evidence; the independent 250 ppm rate gate continues to protect the guest's correction budget. - Keeps the chronyd estimate and its skew, conclusive re-reads, the kernel PLL fallback, the warning, and the average since boot, instead of replacing them with the instantaneous correction. The Linux reader shares the typed struct timex reader with the PLL and H7. The Windows reader uses the precise API's direction: the performance counter ticks increment times per second, and host UTC counts a second for every adjustment ticks, so a larger adjustment slows UTC. The H4 tests cover synchronized, free-running, unreliable, and unknown host clocks; Linux frequency, tick, and maximum-error mapping; Windows' failed, zero, disabled, and synchronized adjustment states and correction direction; the summary facts; and the interaction with the rate budget, warnings, chronyd re-reads, PLL fallback, and boot average. The CI guide, validate-runner, and its test describe the combined gate. Part of #1009 Refs #1008 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/validate-runner/action.yml | 14 +- doc/ci.md | 2 +- scripts/nvx_tools/doctor.py | 141 +++++++++++++++- scripts/test_nvx_tools.py | 8 +- scripts/test_time_abi.py | 180 ++++++++++++++++++++- 5 files changed, 327 insertions(+), 18 deletions(-) diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index 985e1c0f7..849685be7 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -236,13 +236,13 @@ runs: # Host time. The time ABI host qualification (doc/ci.md, # "Host qualification") checks the backend, the CPU fingerprint and - # generation, and the TSC rate stability on its short CI schedule, and, - # in jobs that downloaded OpenVMM, its CPU profile check (H2) and - # preflight (H3); other jobs qualify without OpenVMM. Jobs whose guests - # run without the microVM scenarios' warp probes add the guest warp probe - # (H6). The host's invariant-TSC flags are evidence only: the guest warp - # probe measures the cross-vCPU skew that a host without one can cause - # (#265). + # generation, and the TSC rate stability and the host's synchronized clock + # correction on its short CI schedule, and, in jobs that downloaded + # OpenVMM, its CPU profile check (H2) and preflight (H3); other jobs + # qualify without OpenVMM. Jobs whose guests run without the microVM + # scenarios' warp probes add the guest warp probe (H6). The host's + # invariant-TSC flags are evidence only: the guest warp probe measures the + # cross-vCPU skew that a host without one can cause (#265). - name: Report host TSC if: runner.os != 'Windows' shell: bash diff --git a/doc/ci.md b/doc/ci.md index 7c47ba1e9..b2b13ff28 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -2052,7 +2052,7 @@ fields, such as the microcode and the invariant-TSC flags, may read `unknown`. | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor | | H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry. Then `openvmm --hypervisor --cpu-fingerprint ` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the pinned profile that `auto` selects from OpenVMM's catalog, the latest revision of the generation's profile, which every backend shares, and prints one `NVX-CPU-PROFILE:` line. The check never falls back to a host profile, so a CPU on which `auto` falls back fails H2. H2 reports the generation and the profile from that line, so a profile that an OpenVMM pin adds qualifies its hosts without an NVX change. H2 requires exit status 0, `status=pass`, the same backend, and a catalog profile of the reported generation, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_HOST_UNKNOWN]` on a CPU that no profile serves or `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. On MSHV and WHP, OpenVMM checks the entries outside the profile on a probe partition configured from the profile, as a cold boot does. With `--no-openvmm`, H2 maps the host with NVX's copy of the catalog instead: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), `emeraldrapids` (6/207, `intel.emeraldrapids.v1`), `alderlake` (6/151 and 6/154, `intel.alderlake.v1`), `milan` (AMD 25/1, `amd.milan.v1`), `genoa` (AMD 25/17, `amd.genoa.v2`), or `turin` (AMD 26/2, `amd.turin.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A unit test keeps the copy, which records the latest revision of each generation's profile, equal to the profiles of the OpenVMM submodule's pinned revision, which it reads from the gitlink's commit, so the NVX CLI jobs validate the CLI after they check out OpenVMM. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew | | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a `cpu_profile` that is a catalog profile, `..v` of any generation but a host profile's `host`, which rejects the host profile that `auto` falls back to on a CPU that no built-in profile serves, and, when H2 runs too, a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` | -| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window must lie within 250 ppm of the rate H3 reports when H3 runs in the same invocation, and H4 reports their deviation to 0.001 ppm. The limit is the guest's budget for the host's clock correction: the guest's [wall-clock discipline](design/3.2-clock-and-time.md#3210-guest-obligations) follows host UTC, so it applies the host's correction on top of a restore's rate deviation, which the time ABI's restore tolerance bounds at 250 ppm, within Linux's frequency limit of ±500 ppm. Beyond the budget the guest doesn't fail: its offset grows to 128 ms, and the discipline steps its clock and counts a discontinuity. A unit test keeps the limit equal to the difference of nvx-rs's frequency limit and its restore deviation limit, and the latter equal to the restore tolerance of the OpenVMM gitlink's commit. The limit doesn't bound the TSC's error alone, as the time ABI v1's 100 ppm did, because no run can tell that error from the wander of the host's reference clock: on `azure-mshv-1`, chronyd's estimate ranged from +2.5 to +189.3 ppm within a day while the TSC held its rate (#1008). Above that old bound, H4 passes with a warning that says how much of the budget the host's correction uses, and that one run can't tell the TSC's error from the reference's wander. On Linux with a synchronized chronyd, that rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the frequency that chronyd estimates from its whole sample history, as `chronyc -c tracking` reports it, and every rate within chronyd's skew of that estimate must lie within the limit. The reference's wander can move the estimate and its skew within seconds, so when that range straddles the limit, H4 re-reads the estimate every 2 s for up to 60 s and uses the first whose range lies wholly within or wholly beyond the limit, with the same samples, since only the estimate changes; a range that straddles the limit throughout fails as inconclusive. The rate against `CLOCK_MONOTONIC` is evidence only, because it also carries the slew with which chronyd removes its current offset, which reaches hundreds of ppm for seconds after the host's clock jumps (#768). Without chronyd's estimate, on a Linux host whose kernel PLL another time service drives, such as systemd-timesyncd or ntpd, and which `adjtimex` reports synchronized, the rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the PLL's frequency, `adjtimex`'s frequency with its tick's, which the PLL keeps apart from the offsets that it slews; H4 never reads it with chronyd, which keeps the PLL off and slews through that frequency. Otherwise the rate is measured against `CLOCK_MONOTONIC` on Linux, and the detail says why neither estimate is available, or against `QueryPerformanceCounter` on Windows. On Linux, H4 also reports as evidence the average correction since boot, the rate of `CLOCK_MONOTONIC_RAW` against `CLOCK_MONOTONIC` over the uptime, and the deviation from the backend's rate that it gives: both clocks exclude suspend, and after a small offset at boot only the time service's gradual adjustments separate them, so the average smooths the reference's wander out without saved state, but slews just after boot or a live migration bias it, so H4 doesn't gate on it. A Linux host's clocksource is recorded as evidence | +| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window must lie within 250 ppm of the rate H3 reports when H3 runs in the same invocation, and H4 reports their deviation to 0.001 ppm. The limit is the guest's budget for the host's clock correction: the guest's [wall-clock discipline](design/3.2-clock-and-time.md#3210-guest-obligations) follows host UTC, so it applies the host's correction on top of a restore's rate deviation, which the time ABI's restore tolerance bounds at 250 ppm, within Linux's frequency limit of ±500 ppm. Beyond the budget the guest doesn't fail: its offset grows to 128 ms, and the discipline steps its clock and counts a discontinuity. A unit test keeps the limit equal to the difference of nvx-rs's frequency limit and its restore deviation limit, and the latter equal to the restore tolerance of the OpenVMM gitlink's commit. The limit doesn't bound the TSC's error alone, as the time ABI v1's 100 ppm did, because no run can tell that error from the wander of the host's reference clock: on `azure-mshv-1`, chronyd's estimate ranged from +2.5 to +189.3 ppm within a day while the TSC held its rate (#1008). Above that old bound, H4 passes with a warning that says how much of the budget the host's correction uses, and that one run can't tell the TSC's error from the reference's wander. H4 also requires the host to publish a synchronized clock correction, read as OpenVMM reads it for every [time sample](design/3.2-clock-and-time.md#329-time-samples): on Linux, `adjtimex` reports neither `STA_UNSYNC` nor `STA_CLOCKERR`; on Windows, `GetSystemTimeAdjustmentPrecise` reports that a time service adjusts the clock. H4 records the status, correction, and host UTC's maximum error; the correction's direct reading is evidence, while the independent rate-budget gate above qualifies the combined rate that the guest must follow. On Linux with a synchronized chronyd, that rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the frequency that chronyd estimates from its whole sample history, as `chronyc -c tracking` reports it, and every rate within chronyd's skew of that estimate must lie within the limit. The reference's wander can move the estimate and its skew within seconds, so when that range straddles the limit, H4 re-reads the estimate every 2 s for up to 60 s and uses the first whose range lies wholly within or wholly beyond the limit, with the same samples, since only the estimate changes; a range that straddles the limit throughout fails as inconclusive. The rate against `CLOCK_MONOTONIC` is evidence only, because it also carries the slew with which chronyd removes its current offset, which reaches hundreds of ppm for seconds after the host's clock jumps (#768). Without chronyd's estimate, on a Linux host whose kernel PLL another time service drives, such as systemd-timesyncd or ntpd, and which `adjtimex` reports synchronized, the rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the PLL's frequency, `adjtimex`'s frequency with its tick's, which the PLL keeps apart from the offsets that it slews; H4 never reads it with chronyd, which keeps the PLL off and slews through that frequency. Otherwise the rate is measured against `CLOCK_MONOTONIC` on Linux, and the detail says why neither estimate is available, or against `QueryPerformanceCounter` on Windows. On Linux, H4 also reports as evidence the average correction since boot, the rate of `CLOCK_MONOTONIC_RAW` against `CLOCK_MONOTONIC` over the uptime, and the deviation from the backend's rate that it gives: both clocks exclude suspend, and after a small offset at boot only the time service's gradual adjustments separate them, so the average smooths the reference's wander out without saved state, but slews just after boot or a live migration bias it, so H4 doesn't gate on it. A Linux host's clocksource is recorded as evidence | | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls | | H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart | | H7 | `adjtimex` reports no `STA_UNSYNC` and no `TIME_ERROR` on Linux; `w32tm /query /status` names a synchronized source on Windows. When H7 can't read either, its detail starts with `cannot verify host UTC synchronization`: `adjtimex` fails or reports a clock state Linux doesn't define, or `w32tm` fails or prints no English `Source` and `Leap Indicator` lines, as in a Windows display language other than English | diff --git a/scripts/nvx_tools/doctor.py b/scripts/nvx_tools/doctor.py index fbb9bc94f..f071df4c1 100644 --- a/scripts/nvx_tools/doctor.py +++ b/scripts/nvx_tools/doctor.py @@ -60,7 +60,7 @@ "H1": "backend device or API", "H2": "CPU fingerprint, generation, and profile", "H3": "OpenVMM time ABI preflight", - "H4": "host TSC rate stability", + "H4": "host TSC rate and clock correction", "H5": "host cross-CPU TSC skew", "H6": "guest warp probe", "H7": "host UTC synchronization", @@ -73,7 +73,9 @@ VERIFY_PREFIX = "NVX-TIME-ABI-VERIFY:" CPU_PROFILE_PREFIX = "NVX-CPU-PROFILE:" RUST_ESCAPE = re.compile(r"\\(u\{[0-9a-fA-F]{1,6}\}|.)") -# H4's rate stability bounds (doc/ci.md, "Host qualification"). +# H4's bounds (doc/ci.md, "Host qualification"): the TSC keeps its rate +# across idle, its rate against host UTC stays within the guest's correction +# budget, and the host publishes a synchronized clock correction. RATE_UNCERTAINTY_PPM = 0.25 RATE_AGREEMENT_PPM = 1.0 # H4's limit is the guest's budget for the host's clock correction. The @@ -137,6 +139,7 @@ class Schedule: ADJTIMEX_TIME_ERROR = 5 ADJTIMEX_STA_PLL = 0x0001 ADJTIMEX_STA_UNSYNC = 0x0040 +ADJTIMEX_STA_CLOCKERR = 0x1000 # adjtimex's frequency is in ppm scaled by 2^16, and its tick is the # microseconds that the clock advances per USER_HZ tick, 100 per second, so # 10,000 leaves the clock's rate alone. @@ -864,6 +867,113 @@ def _adjtimex() -> tuple[int, _Timex]: return state, timex +@dataclass(frozen=True) +class HostClockDiscipline: + """How the host's time service disciplines host UTC, as OpenVMM reads it + and publishes it with each time sample.""" + + # "synchronized", "free-running", "unreliable", or "unknown". + status: str + # How fast host UTC runs against the host's undisciplined clock, minus one. + correction_ppm: float + # The time service's maximum error of host UTC in microseconds, if known. + max_error_us: int | None + # What the host reported. + detail: str + + +def _linux_clock_discipline() -> HostClockDiscipline: + try: + _, timex = _adjtimex() + except ScriptError as error: + return HostClockDiscipline("unknown", 0.0, None, str(error)) + status_bits = int(timex.status) + if status_bits & ADJTIMEX_STA_CLOCKERR: + status = "unreliable" + elif status_bits & ADJTIMEX_STA_UNSYNC: + status = "free-running" + else: + status = "synchronized" + correction_ppm = ( + int(timex.freq) / ADJTIMEX_SCALED_PPM + + (int(timex.tick) - ADJTIMEX_NOMINAL_TICK) * 1e6 / ADJTIMEX_NOMINAL_TICK + ) + maximum = int(timex.maxerror) + return HostClockDiscipline( + status, + correction_ppm, + maximum if maximum >= 0 else None, + f"adjtimex status=0x{status_bits:04x} freq={int(timex.freq)} " + f"tick={int(timex.tick)} maxerror={maximum}", + ) + + +def windows_adjustment_ppm(adjustment: int, increment: int) -> float: + """Return how much faster than the performance counter host UTC runs, in + ppm, under the precise Windows time adjustment that OpenVMM reads. + + The counter ticks ``increment`` times a second, and host UTC counts a + second for every ``adjustment`` of its ticks, so a larger adjustment slows + host UTC. + """ + return (increment - adjustment) / adjustment * 1e6 + + +def _windows_clock_discipline() -> HostClockDiscipline: + if sys.platform != "win32": + return HostClockDiscipline( + "unknown", 0.0, None, "GetSystemTimeAdjustmentPrecise is Windows-only" + ) + adjustment = ctypes.c_uint64() + increment = ctypes.c_uint64() + disabled = ctypes.c_int() + try: + function = ctypes.WinDLL("kernelbase").GetSystemTimeAdjustmentPrecise + function.argtypes = ( + ctypes.POINTER(ctypes.c_uint64), + ctypes.POINTER(ctypes.c_uint64), + ctypes.POINTER(ctypes.c_int), + ) + function.restype = ctypes.c_int + read = bool( + function( + ctypes.byref(adjustment), + ctypes.byref(increment), + ctypes.byref(disabled), + ) + ) + except (AttributeError, OSError) as error: + return HostClockDiscipline( + "unknown", + 0.0, + None, + f"GetSystemTimeAdjustmentPrecise is unavailable: {error}", + ) + detail = ( + f"GetSystemTimeAdjustmentPrecise adjustment={adjustment.value} " + f"increment={increment.value} disabled={disabled.value}" + ) + if not read or increment.value == 0: + return HostClockDiscipline("unknown", 0.0, None, detail) + if disabled.value: + return HostClockDiscipline("free-running", 0.0, None, detail) + if adjustment.value == 0: + return HostClockDiscipline("unknown", 0.0, None, detail) + return HostClockDiscipline( + "synchronized", + windows_adjustment_ppm(adjustment.value, increment.value), + None, + detail, + ) + + +def host_clock_discipline() -> HostClockDiscipline: + """Return the host clock correction that OpenVMM publishes.""" + if host_is_windows(): + return _windows_clock_discipline() + return _linux_clock_discipline() + + @dataclass(frozen=True) class KernelFrequency: """The frequency correction of the kernel's PLL in ppm, positive when it @@ -1130,6 +1240,26 @@ def conclusive(estimate: ChronyFrequency) -> bool: f"guest's {RATE_TOLERANCE_PPM:g} ppm budget, and one run can't " "tell the TSC's error from the reference clock's wander" ) + # Independently of the rate budget above, the guest needs host UTC to have + # a correction maintained by a time service. Read the same host state that + # OpenVMM publishes with every time sample. + discipline = host_clock_discipline() + context.facts["host_clock_status"] = discipline.status + context.facts["host_correction_ppm"] = f"{discipline.correction_ppm:+.3f}" + maximum = ( + "" + if discipline.max_error_us is None + else f", maximum error {discipline.max_error_us} us" + ) + discipline_detail = ( + f"; host UTC is {discipline.status}, correction " + f"{discipline.correction_ppm:+.3f} ppm{maximum} ({discipline.detail})" + ) + if discipline.status != "synchronized": + problems.append( + "the host publishes no synchronized clock correction: host UTC is " + f"{discipline.status}" + ) detail += reread_evidence if isinstance(chrony, ChronyFrequency): detail += ( @@ -1159,12 +1289,13 @@ def conclusive(estimate: ChronyFrequency) -> bool: detail += f", {boot_deviation:.3f} ppm from the backend's rate" detail += " (evidence)" context.facts["boot_correction"] = f"{boot.ppm:+.3f} ppm over {hours:.1f} h" + detail += discipline_detail if clocksource: detail += f"; host clocksource {clocksource} (evidence)" + if problems: + return CheckResult("H4", False, "; ".join(problems) + f"; {detail}") if warning: detail = f"warning: {warning}; {detail}" - if problems: - return CheckResult("H4", False, "; ".join(problems) + f"; {detail}", warning) return CheckResult("H4", True, detail, warning) @@ -1425,6 +1556,8 @@ def summary_markdown( ("rate_agreement_ppm", "Interval rate agreement (ppm)"), ("rate_uncertainty_ppm", "Largest interval uncertainty (ppm)"), ("boot_correction", "Average correction since boot (evidence)"), + ("host_clock_status", "Host UTC"), + ("host_correction_ppm", "Host clock correction (ppm)"), ("host_skew_ns", "Host skew (ns)"), ("guest_warp_ns", "Guest warp offset (ns)"), ("invariant_tsc", "Host invariant TSC (evidence)"), diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 55bbe1da6..c65eecb19 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -13747,10 +13747,10 @@ def test_runners_qualify_their_host_time_before_every_job(self): ).read_text(encoding="utf-8") # The doctor follows the host TSC report on Linux: the backend, the CPU - # fingerprint and generation, and the TSC rate stability. Jobs that - # haven't downloaded OpenVMM skip its CPU profile check (H2) and - # preflight (H3), and only jobs that ask for it run the guest warp - # probe (H6). + # fingerprint and generation, and the TSC rate stability and the host's + # clock correction. Jobs that haven't downloaded OpenVMM skip its CPU + # profile check (H2) and preflight (H3), and only jobs that ask for it + # run the guest warp probe (H6). self.assertLess( validate_runner.index(" - name: Report host TSC\n"), validate_runner.index(" - name: Qualify host time on Linux\n"), diff --git a/scripts/test_time_abi.py b/scripts/test_time_abi.py index 879128c76..88640f62d 100644 --- a/scripts/test_time_abi.py +++ b/scripts/test_time_abi.py @@ -2560,6 +2560,13 @@ def records( result.append(("sample", sample)) return result + synchronized = doctor.HostClockDiscipline( + "synchronized", + 12.5, + 1000, + "adjtimex status=0x2001 freq=819200 tick=10000 maxerror=1000", + ) + def check( context: doctor.DoctorContext, probe_records: list[tuple[str, dict[str, str]]], @@ -2571,6 +2578,7 @@ def check( clock: FakeClock | None = None, kernel: doctor.KernelFrequency | str = PLL_OFF, boot: doctor.BootCorrection | None = None, + discipline: doctor.HostClockDiscipline = synchronized, ) -> doctor.CheckResult: """Run H4; a list of chronyd readings answers its re-reads too.""" clock = clock or FakeClock() @@ -2587,6 +2595,9 @@ def check( ) as frequency, patch.object(doctor, "kernel_frequency", return_value=kernel) as pll, patch.object(doctor, "boot_correction", return_value=boot) as average, + patch.object( + doctor, "host_clock_discipline", return_value=discipline + ) as published, patch.object(doctor.time, "monotonic", clock.monotonic), patch.object(doctor.time, "sleep", clock.sleep), ): @@ -2597,6 +2608,8 @@ def check( # H4 reads the kernel's PLL only without chronyd's estimate. first = readings[0] if readings else chrony self.assertEqual(pll.called, not windows and isinstance(first, str)) + # Every backend reads the correction that OpenVMM publishes. + self.assertTrue(published.called) self.assertEqual( run.call_args.args[1:], ( @@ -2642,6 +2655,14 @@ def ci_context(backend: str = "kvm") -> doctor.DoctorContext: f"no kernel PLL frequency: {PLL_OFF} (evidence)", result.detail, ) + self.assertIn( + "host UTC is synchronized, correction +12.500 ppm, maximum error " + "1000 us (adjtimex status=0x2001 freq=819200 tick=10000 " + "maxerror=1000)", + result.detail, + ) + self.assertEqual(context.facts["host_clock_status"], "synchronized") + self.assertEqual(context.facts["host_correction_ppm"], "+12.500") self.assertEqual(context.facts["rate_agreement_ppm"], "0.125") self.assertEqual(context.facts["rate_deviation_ppm"], "0.013") self.assertEqual(context.facts["rate_reference"], "CLOCK_MONOTONIC") @@ -2667,8 +2688,19 @@ def ci_context(backend: str = "kvm") -> doctor.DoctorContext: resolution_ns=100.0, names=("QueryPerformanceCounter", "QueryPerformanceCounter"), ) - windows = check(ci_context("whp"), qpc, windows=True) + adjusted = doctor.HostClockDiscipline( + "synchronized", + -17.2, + None, + "GetSystemTimeAdjustmentPrecise adjustment=10000172 " + "increment=10000000 disabled=0", + ) + windows = check(ci_context("whp"), qpc, windows=True, discipline=adjusted) self.assertTrue(windows.passed, windows.detail) + self.assertIn( + "host UTC is synchronized, correction -17.200 ppm (", + windows.detail, + ) self.assertIn( "against QueryPerformanceCounter from 3 samples 1 s apart", windows.detail ) @@ -2733,6 +2765,32 @@ def ci_context(backend: str = "kvm") -> doctor.DoctorContext: standalone = check(ci_context(), records([2300000100.0] * 2, interval_s=1.0)) self.assertTrue(standalone.passed, standalone.detail) self.assertIn("not compared with the backend's rate", standalone.detail) + # The correction's status is a gate, while its directly published size + # is evidence: the rate budget above independently qualifies the rate + # against the backend when H3 runs. + steady = records([2300000000.0] * 2, interval_s=1.0) + for status in ("free-running", "unreliable", "unknown"): + with self.subTest(status=status): + context = ci_context("mshv") + context.facts["native_tsc_hz"] = "2300000000" + discipline = doctor.HostClockDiscipline(status, 0.0, None, "evidence") + result = check(context, steady, discipline=discipline) + self.assertFalse(result.passed) + self.assertTrue( + result.detail.startswith( + "the host publishes no synchronized clock correction: host " + f"UTC is {status}; " + ), + result.detail, + ) + self.assertEqual(result.warning, "") + self.assertEqual(context.facts["host_clock_status"], status) + large = doctor.HostClockDiscipline( + "synchronized", 10_000.0, None, "adjtimex tick=10100" + ) + without_h3 = check(ci_context("mshv"), steady, discipline=large) + self.assertTrue(without_h3.passed, without_h3.detail) + self.assertIn("correction +10000.000 ppm", without_h3.detail) # #768: on azure-mshv-1, whose host clock jumps, chronyd slewed # CLOCK_MONOTONIC so that a 2 s window read 946.1 ppm, while the # TSC ran at its usual rate, 44 ppm from the declared one, against @@ -3052,6 +3110,116 @@ def test_chrony_frequency_reads_a_synchronized_tracking_report(self): with patch.object(doctor.shutil, "which", return_value=None): self.assertEqual(doctor.chrony_frequency(), "chronyc is not installed") + def test_host_clock_discipline_maps_the_host_state_as_openvmm_does(self): + def linux(**changes: int) -> doctor.HostClockDiscipline: + fields = { + "status": 0x2001, + "freq": -552_308, + "tick": 10_000, + "maxerror": 1564, + } + fields.update(changes) + timex = types.SimpleNamespace(**fields) + with ( + patch.object(doctor, "_adjtimex", return_value=(0, timex)), + patch.object(doctor, "host_is_windows", return_value=False), + ): + return doctor.host_clock_discipline() + + self.assertEqual( + linux(), + doctor.HostClockDiscipline( + "synchronized", + -552_308 / 65_536, + 1564, + "adjtimex status=0x2001 freq=-552308 tick=10000 maxerror=1564", + ), + ) + # A tick of 10,001 us adds 100 ppm to the published correction. + self.assertAlmostEqual(linux(freq=0, tick=10_001).correction_ppm, 100.0) + self.assertEqual(linux(status=0x2041).status, "free-running") + self.assertEqual(linux(status=0x3001).status, "unreliable") + self.assertEqual(linux(status=0x1041).status, "unreliable") + self.assertIsNone(linux(maxerror=-1).max_error_us) + with ( + patch.object( + doctor, "_adjtimex", side_effect=doctor.ScriptError("no adjtimex") + ), + patch.object(doctor, "host_is_windows", return_value=False), + ): + unknown = doctor.host_clock_discipline() + self.assertEqual((unknown.status, unknown.correction_ppm), ("unknown", 0.0)) + self.assertIn("no adjtimex", unknown.detail) + + adjustment_ppm = doctor.windows_adjustment_ppm + self.assertEqual(adjustment_ppm(10_000_000, 10_000_000), 0.0) + self.assertAlmostEqual(adjustment_ppm(10_000_000, 10_000_100), 10.0) + self.assertAlmostEqual(adjustment_ppm(10_000_000, 9_999_900), -10.0) + # A larger precise adjustment slows host UTC, as measured on the + # Windows runners that motivated the OpenVMM correction. + self.assertAlmostEqual(adjustment_ppm(10_000_501, 10_000_000), -50.0975, 4) + self.assertAlmostEqual(adjustment_ppm(10_000_172, 10_000_000), -17.1997, 4) + + def test_windows_clock_discipline_maps_the_precise_adjustment(self): + class PreciseAdjustment: + argtypes: object = None + restype: object = None + + def __init__( + self, read: int, adjustment: int, increment: int, disabled: int + ) -> None: + self.read = read + self.adjustment = adjustment + self.increment = increment + self.disabled = disabled + + def __call__( + self, adjustment: object, increment: object, disabled: object + ) -> int: + adjustment._obj.value = self.adjustment # type: ignore[attr-defined] + increment._obj.value = self.increment # type: ignore[attr-defined] + disabled._obj.value = self.disabled # type: ignore[attr-defined] + return self.read + + def windows( + read: int, adjustment: int, increment: int, disabled: int + ) -> doctor.HostClockDiscipline: + precise = PreciseAdjustment(read, adjustment, increment, disabled) + library = types.SimpleNamespace(GetSystemTimeAdjustmentPrecise=precise) + with ( + patch.object(doctor.sys, "platform", "win32"), + patch.object( + doctor.ctypes, "WinDLL", return_value=library, create=True + ), + patch.object(doctor, "host_is_windows", return_value=True), + ): + return doctor.host_clock_discipline() + + for reading in ( + windows(0, 10_000_000, 10_000_000, 0), + windows(1, 10_000_000, 0, 0), + windows(1, 0, 10_000_000, 0), + ): + self.assertEqual((reading.status, reading.correction_ppm), ("unknown", 0.0)) + free = windows(1, 10_000_000, 10_000_000, 1) + self.assertEqual((free.status, free.correction_ppm), ("free-running", 0.0)) + synchronized = windows(1, 10_000_172, 10_000_000, 0) + self.assertEqual(synchronized.status, "synchronized") + self.assertAlmostEqual(synchronized.correction_ppm, -17.1997, 4) + with ( + patch.object(doctor.sys, "platform", "win32"), + patch.object( + doctor.ctypes, + "WinDLL", + side_effect=OSError("no precise adjustment"), + create=True, + ), + patch.object(doctor, "host_is_windows", return_value=True), + ): + unknown = doctor.host_clock_discipline() + self.assertEqual(unknown.status, "unknown") + self.assertIn("no precise adjustment", unknown.detail) + def test_kernel_frequency_reads_a_synchronized_kernel_pll(self): # adjtimex's state and the status, freq, and tick of its struct timex. def kernel(state: int, status: int, freq: int, tick: int = 10_000): @@ -3167,6 +3335,13 @@ def records(gaps: tuple[int, int]) -> list[tuple[str, dict[str, str]]]: return_value="chronyc is not installed", ), patch.object(doctor, "kernel_frequency", return_value=PLL_OFF), + patch.object( + doctor, + "host_clock_discipline", + return_value=doctor.HostClockDiscipline( + "synchronized", 0.0, None, "evidence" + ), + ), ): result = doctor.check_rate(context) self.assertTrue(run.called) @@ -3478,7 +3653,8 @@ def warned(context: doctor.DoctorContext) -> doctor.CheckResult: # In GitHub Actions, an annotation repeats it, escaped. annotation = ( "::warning title=Time ABI host qualification (mshv)::H4 host TSC " - "rate stability: the host's clock correction uses 60%25 of the " + "rate and clock correction: the host's clock correction uses " + "60%25 of the " "guest's 250 ppm budget" ) self.assertEqual(lines[1:], [annotation] if annotated else []) From 6d4a96b86d6fc1c14fafba9e137b00b11944abe6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 11:34:54 -0700 Subject: [PATCH 4/5] test-microvm: check that the guest follows the host's clock correction Time ABI 2 publishes the host's clock correction with every sample, and the guest holds it as its frequency correction. Add the `time-host-correction` microVM scenario to check that behavior when host UTC runs at a deliberately different rate from the guest TSC. An unhooked cold boot first reads the host's own correction. OpenVMM's `host-rate-offset-ppm` hook then drives: - a hooked cold boot whose published correction stays below Linux's 500 ppm frequency limit; - a restore at another correction below the limit; and - a restore in the host correction's direction that exceeds the limit, where the guest must hold +/-500 ppm and report frequency_limited=1. The ordinary hooks solve OpenVMM's multiplicative composition of the host and hook rates for targets of 200 and 150 ppm against the host correction's sign. They are clamped to OpenVMM's accepted +/-10,000 ppm hook range; at its edge they use the closest composed correction whose effective Linux correction, after the declared TSC rate's kHz rounding, stays below the limit. A host correction that no accepted hook can keep below it fails before OpenVMM starts with a diagnostic naming both bounds. The limit hook is 900 ppm in the host correction's direction. Each phase requires the test-hook flag and checks that the boot step or restore repair did not retain a stale correction. Up to three pairs of samples pair host UTC and the guest's realtime clock with the guest TSC. Before each pair the guest cancels an earlier poll's kernel phase slew while keeping the frequency held, and brackets the samples with the live discipline state. A changed disruption counter, held correction, or clock-step state discards that window. The checker verifies the first stable pair with its fresh state, within its pairing uncertainty and the allowance for host slews. The limit phase requires the saturated frequency and flag explicitly. Linux counts the declared TSC rate rounded down to the kHz. The checker derives the permitted difference between the host and kernel frequency corrections from each report's tsc_hz and the worst 999 Hz loss, instead of assuming a rate of at least 1 GHz. The regression at 500,000,999 Hz accepts the valid 1,998 ppb difference. Another regression rejects a -10,396 ppm host at that rate: although the largest hook publishes -499.96 ppm, Linux must hold -501.957 ppm. Conversely, a +10,607 ppm host and -10,000 ppm hook publish +500.930 ppm but need only +498.931 ppm from Linux, which both the solver and checker accept. The hook tests cover both +/-10,000 ppm boundaries and reject corrections that cannot be offset. The shard plan runs the scenario on KVM, MSHV, and WHP. Unit tests cover the successful phases, stale and ignored corrections, the clock-rate checks, host slews, disruption and discipline-update retries, saturation, missing or unhooked samples, the full 500 MHz ABI range, OpenVMM's hook range, and the probe, boot, and restore command sequence. Part of #1009 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/ci.md | 43 ++ doc/usage.md | 14 +- nvx-rs/src/guest/sys/discipline.rs | 10 +- nvx-rs/src/guest/time/test_mode.rs | 44 +- scripts/nvx_tools/microvm_shards.json | 32 +- .../time-host-correction.sh.in | 58 ++ scripts/nvx_tools/microvm_tests.py | 560 +++++++++++++++ scripts/test_microvm_tests.py | 635 ++++++++++++++++++ 8 files changed, 1379 insertions(+), 17 deletions(-) create mode 100644 scripts/nvx_tools/microvm_test_scripts/time-host-correction.sh.in diff --git a/doc/ci.md b/doc/ci.md index b2b13ff28..5a8a124ae 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -166,6 +166,49 @@ check on every CPU, a summary with `status=ok`, the requested CPU count, and no failures, and exit status 0; a failure lists each failing check with the guest's detail. The guest command fits on one console line, so the console's echo of it ends before the check prints. +The `time-host-correction` scenario checks that the guest applies the host's +clock correction that OpenVMM publishes with each +[time sample](design/3.2-clock-and-time.md#329-time-samples), through OpenVMM's +`host-rate-offset-ppm` test hook, which publishes host UTC as running that many +ppm faster than it does, with the correction to match, and sets the time +samples' test-hook flag. A one-vCPU cold boot without the hook first reads the +host's own correction from a time sample, which must lack that flag. A one-vCPU +cold boot and a restore of a snapshot captured without the hook then target +corrections of 200 and 150 ppm against the sign of the host's own. The harness +solves OpenVMM's multiplicative composition of the host and hook rates, keeps +the hook within OpenVMM's +/-10,000 ppm range, and requires the correction that +Linux holds after rounding the declared TSC rate to the kHz to stay below its +500 ppm limit; otherwise it fails before OpenVMM starts. A restore with a +900 ppm hook in the direction of the host's own correction publishes one +beyond that limit. These three each print the guest's time state and take up to +three pairs of time samples 5 s apart with `nvx-time sample`, or one sample +when Linux's limit holds the guest's frequency correction, after one that they +discard: the first sample after a restore can read the guest's clock late while +the restored memory that it touches faults in. Before each pair the script reads +the live state, calls `nvx-time test clear-phase` to clear any PLL phase offset +that an earlier discipline poll left with `ADJ_OFFSET | ADJ_NANO`, without +changing `STA_PLL | STA_FREQHOLD` or the held frequency, waits two seconds +for Linux's cached tick length to drop the old phase correction, and reads the +state again; it reads a third state after the pair. A changed accepted-sample +count, +held correction, clock-step state, or sample disruption counter discards and +retries the window. Every sample must carry the test-hook +flag. The correction that the guest's boot +step or restore repair took must lie within 75 ppm, half the 150 ppm of the +smallest hook, of the one that the first stable sample publishes, so that it +isn't a stale one. Between the stable pair, which pairs host UTC and +the guest's clock with the guest TSC, both must run at their corrections +against the TSC counted at the declared rate. The guest's clock must run at the +one that it holds, within 1 ppm beyond the uncertainty of its pairings; a guest +that didn't apply the hook's correction would run 150 ppm or more off it. Host +UTC must run at its published one, within 75 ppm beyond half that correction's +change and the pairing bounds, so that the hook skewed both alike: 75 ppm is +half the 150 ppm that a hook that skewed only one of them would leave, and the +host's time service may slew host UTC between two samples, by up to 25 ppm on +the test VMs measured so far. The 900 ppm restore exceeds Linux's limit +whatever the host's own correction, and the guest must hold the limit and +report `frequency_limited=1`. Both restores must also pass the warp probe and +their restore checks. The `cpu-profile-fallback` scenario covers `--cpu-profile auto`'s fallback to a host profile on a CPU that no built-in profile serves, on runners that built-in profiles serve. It cold boots one vCPU with OpenVMM's diff --git a/doc/usage.md b/doc/usage.md index 01335b5a5..bc27aabd2 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1213,7 +1213,19 @@ backend and splits the backend's suite into COUNT shards. The test hook, which makes OpenVMM's default CPU profile selection, `auto`, fall back to a host profile on a host that a built-in profile serves, checks the fallback's warning, captures the guest, and restores it on the snapshot's host -profile. The `managed-lifecycle` scenario boots a guest whose init hands +profile. The `time-host-correction` scenario reads the host's own clock +correction from a time sample of an unhooked cold boot, then cold boots with +OpenVMM's `host-rate-offset-ppm` test hook. The harness composes the host and +hook rates as OpenVMM does, checks Linux's correction after the TSC rate's kHz +rounding, clears an earlier poll's PLL phase offset through the `nvx-time` +binary that every guest ships, waits out Linux's cached tick length, and retries +a sample pair when host UTC, the accepted-sample count, or the live discipline +state changes. +It checks that the guest holds the published correction and runs its clock at +the hooked rate of host UTC, then restores a snapshot with another rate, and +with one beyond Linux's 500 ppm limit, which the guest must hold at the limit +and report. The +`managed-lifecycle` scenario boots a guest whose init hands the boot to the managed agent, and runs the ignored `nvx-rs` test `tests/managed_lifecycle.rs` with Cargo, which starts OpenVMM with the scenario's command line and drives the guest's control session; the scenario diff --git a/nvx-rs/src/guest/sys/discipline.rs b/nvx-rs/src/guest/sys/discipline.rs index a0f71cc0f..abcb57544 100644 --- a/nvx-rs/src/guest/sys/discipline.rs +++ b/nvx-rs/src/guest/sys/discipline.rs @@ -28,6 +28,9 @@ pub(crate) enum ClockAdjustment { /// Sets the frequency correction and the error estimates, and selects the phase-locked loop /// in nanosecond mode with its frequency held. Frequency(FrequencyHold), + /// Clears the phase offset through the phase-locked loop in nanosecond mode, without changing + /// its status or held frequency correction. + ClearPhase, /// Slews the wall clock by `offset_ns` through the phase-locked loop in nanosecond mode, with /// its frequency held, and sets its time constant, the frequency correction, and the error /// estimates. @@ -63,6 +66,7 @@ impl ClockAdjustment { Self::Query => 0, Self::Step { .. } => libc::ADJ_SETOFFSET | libc::ADJ_NANO, Self::Frequency(_) => HOLD, + Self::ClearPhase => libc::ADJ_OFFSET | libc::ADJ_NANO, Self::Slew { .. } => HOLD | libc::ADJ_OFFSET | libc::ADJ_TIMECONST | libc::ADJ_NANO, } } @@ -70,7 +74,7 @@ impl ClockAdjustment { /// Returns the discipline status that the adjustment sets, or zero for one that sets none. pub(crate) fn status(self) -> i32 { match self { - Self::Query | Self::Step { .. } => 0, + Self::Query | Self::Step { .. } | Self::ClearPhase => 0, Self::Frequency(_) | Self::Slew { .. } => PLL_NANOSECONDS, } } @@ -112,6 +116,7 @@ pub(crate) fn adjust_clock(adjustment: ClockAdjustment) -> Result hold.apply(&mut value), + ClockAdjustment::ClearPhase => {} ClockAdjustment::Slew { offset_ns, time_constant, @@ -147,6 +152,7 @@ mod tests { estimated_error_us: 1, }; let frequency = ClockAdjustment::Frequency(hold); + let clear_phase = ClockAdjustment::ClearPhase; let slew = ClockAdjustment::Slew { offset_ns: 5, time_constant: 4, @@ -155,6 +161,8 @@ mod tests { assert_eq!((step.modes(), step.status()), (0x2100, 0)); // The status holds the frequency, `STA_FREQHOLD`, with the loop and nanoseconds. assert_eq!((frequency.modes(), frequency.status()), (0x001e, 0x2081)); + // Clearing the nanosecond offset changes neither status nor frequency. + assert_eq!((clear_phase.modes(), clear_phase.status()), (0x2001, 0)); assert_eq!((slew.modes(), slew.status()), (0x203f, 0x2081)); assert_eq!( ( diff --git a/nvx-rs/src/guest/time/test_mode.rs b/nvx-rs/src/guest/time/test_mode.rs index 41a15a1c4..7fff82f3f 100644 --- a/nvx-rs/src/guest/time/test_mode.rs +++ b/nvx-rs/src/guest/time/test_mode.rs @@ -9,7 +9,8 @@ //! be `0x`-prefixed hexadecimal, as other guest programs read their arguments. The packet and //! sample hooks read their fixtures as port B delivers a record, zero-filled past its end. //! `console` writes its whole line, as the program writes every event, which is shorter than -//! the 638 bytes after which the C program's buffer cut an asynchronous line. +//! the 638 bytes after which the C program's buffer cut an asynchronous line. `clear-phase` +//! removes an in-flight kernel PLL phase offset without changing its held frequency correction. use std::ffi::{CStr, CString, OsStr}; use std::fs; @@ -65,6 +66,7 @@ pub(super) fn run(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), Comman b"exhaustive-report" => test_exhaustive_report(arguments, out), b"exhaustive-summary" => test_exhaustive_summary(arguments, out), b"timer-list" => test_timer_list(arguments, out), + b"clear-phase" => test_clear_phase(arguments), b"correction" => test_correction(arguments, out), b"frequency" => test_frequency(arguments, out), b"event" => test_event(arguments, out), @@ -408,6 +410,23 @@ fn test_timer_list(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), Comma Ok(()) } +/// Clears the kernel PLL's phase offset without changing its status or held frequency. +fn test_clear_phase(arguments: &[&[u8]]) -> Result<(), CommandError> { + test_clear_phase_with(arguments, |adjustment| { + sys::adjust_clock(adjustment).map(drop) + }) +} + +/// Runs the clear-phase hook through `adjust`, separated so that its privileged operation is +/// testable without changing the host's clock. +fn test_clear_phase_with( + arguments: &[&[u8]], + adjust: impl FnOnce(ClockAdjustment) -> Result<(), Errno>, +) -> Result<(), CommandError> { + exactly::<0>(arguments)?; + adjust(ClockAdjustment::ClearPhase).map_err(|_| CommandError::Operation) +} + /// Plans and prints one step or slew correction, with the `adjtimex` modes and status that the /// system layer would apply, for a discipline that holds a frequency correction and adds a host's /// error. @@ -446,7 +465,7 @@ fn test_correction(arguments: &[&[u8]], out: &mut dyn Write) -> Result<(), Comma hold.estimated_error_us ); } - ClockAdjustment::Query | ClockAdjustment::Frequency(_) => {} + ClockAdjustment::Query | ClockAdjustment::Frequency(_) | ClockAdjustment::ClearPhase => {} } Ok(()) } @@ -847,6 +866,27 @@ mod tests { out } + #[test] + fn clear_phase_changes_only_the_pll_offset() { + let mut adjustment = None; + assert_eq!( + test_clear_phase_with(&[], |value| { + adjustment = Some(value); + Ok(()) + }), + Ok(()) + ); + assert_eq!(adjustment, Some(ClockAdjustment::ClearPhase)); + assert_eq!( + test_clear_phase_with(&[b"extra"], |_| panic!("unexpected adjustment")), + Err(CommandError::Usage) + ); + assert_eq!( + test_clear_phase_with(&[], |_| Err(Errno::EPERM)), + Err(CommandError::Operation) + ); + } + #[test] fn cpuid_values_are_32_bit_decimal_or_hexadecimal() { assert_eq!(cpuid_value(b"0x40000006"), Ok(0x4000_0006)); diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index 39d15771e..23c0968d6 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -63,21 +63,22 @@ "snapshot-tiers": 6.8, "structured-outcome": 0.6, "time-abi-conformance": 0.7, + "time-host-correction": 14.0, "virtio-net": 3.8, "workload-identity": 0.7 }, "shards": [ { - "seconds": 251.4, + "seconds": 258.5, "scenarios": [ "command-line-limit", "console-snapshot", "directional-network-policy", - "endpoint-policy-snapshot", "filesystem-comma-paths", "filesystem-snapshot", "managed-lifecycle", "network-snapshot", + "platform-aggregate-export", "platform-restore-processors", "restore-downtime", "restore-memory", @@ -85,15 +86,17 @@ "smp-snapshot", "snapshot-core", "snapshot-tiers", - "time-abi-conformance" + "time-abi-conformance", + "time-host-correction" ] }, { - "seconds": 251.4, + "seconds": 258.3, "scenarios": [ "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", + "endpoint-policy-snapshot", "filesystem-owner", "filesystem-policy", "filesystem-shares", @@ -105,7 +108,6 @@ "l3-l4-egress-port-ranges", "l3-l4-egress-protocols", "lifecycle", - "platform-aggregate-export", "platform-filesystem", "platform-network-policy", "platform-zero-fill", @@ -161,16 +163,17 @@ "snapshot-tiers": 6.3, "structured-outcome": 0.5, "time-abi-conformance": 0.8, + "time-host-correction": 14.0, "virtio-net": 3.7, "workload-identity": 0.6 }, "shards": [ { - "seconds": 232.5, + "seconds": 239.3, "scenarios": [ "console-snapshot", "directional-network-policy", - "endpoint-policy-snapshot", + "filesystem-comma-paths", "filesystem-shares", "filesystem-snapshot", "network-snapshot", @@ -183,17 +186,18 @@ "smp", "smp-snapshot", "snapshot-core", - "snapshot-tiers" + "snapshot-tiers", + "time-host-correction" ] }, { - "seconds": 232.5, + "seconds": 239.7, "scenarios": [ "command-line-limit", "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", - "filesystem-comma-paths", + "endpoint-policy-snapshot", "filesystem-owner", "filesystem-policy", "guest-boot", @@ -259,15 +263,17 @@ "snapshot-tiers": 12.0, "structured-outcome": 0.8, "time-abi-conformance": 1.6, + "time-host-correction": 14.0, "virtio-net": 4.5, "workload-identity": 1.1 }, "shards": [ { - "seconds": 331.7, + "seconds": 338.7, "scenarios": [ "console-snapshot", "denied-filesystem-paths", + "directional-network-policy", "endpoint-policy-snapshot", "filesystem-policy", "filesystem-snapshot", @@ -286,12 +292,11 @@ ] }, { - "seconds": 331.7, + "seconds": 338.7, "scenarios": [ "command-line-limit", "console-exit", "cpu-profile-fallback", - "directional-network-policy", "filesystem-comma-paths", "filesystem-owner", "filesystem-shares", @@ -310,6 +315,7 @@ "snapshot-core", "structured-outcome", "time-abi-conformance", + "time-host-correction", "virtio-net", "workload-identity" ] diff --git a/scripts/nvx_tools/microvm_test_scripts/time-host-correction.sh.in b/scripts/nvx_tools/microvm_test_scripts/time-host-correction.sh.in new file mode 100644 index 000000000..cd2ac2fd7 --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/time-host-correction.sh.in @@ -0,0 +1,58 @@ +# The time-host-correction scenario's guest report (doc/ci.md): up to +# @ATTEMPTS@ measurement windows of two time samples @SPACING_SECONDS@ seconds +# apart. Each window reads the live time state before and after it clears an +# earlier PLL phase offset and waits out its cached tick length, then after its samples. A changed accepted-sample +# count, disruption counter, held correction, or clock-step state makes the +# script retry. Linux's frequency limit needs only one sample. The first sample after +# a restore can read the guest's clock late in its bracket while the restored +# memory that it touches faults in, so the report discards one sample first. +state=/run/nvx/time/state +state_signature() { + text=$1 + for name in frequency_ppb frequency_limited host_correction_ppb discontinuities last_step_realtime_ns samples; do + value=$(printf '%s\n' "$text" | sed -n "s/.*[[:space:]]$name=\([^ ]*\).*/\1/p") + printf '%s=%s;' "$name" "$value" + done +} +nvx-time sample >/dev/null 2>&1 || true +attempt=1 +index=1 +while [ "$attempt" -le @ATTEMPTS@ ]; do + initial_state=$(tr '\n' ' ' <"$state") + echo "@STATE_PREFIX@attempt=$attempt point=initial $initial_state" + # Clear an earlier daemon poll's PLL phase offset without changing the + # PLL status or held frequency; nvx-time ships in every guest image. + if ! nvx-time test clear-phase >/dev/null 2>&1; then + echo "NVX-TIME-HOST-CORRECTION-ERROR: cannot clear PLL phase offset" + nvx-exit 1 + exit 1 + fi + # Linux caches the old phase correction in tick_length until its next + # kernel-second update. Keep that wait inside the live-state bracket. + sleep @PHASE_SETTLE_SECONDS@ + before_state=$(tr '\n' ' ' <"$state") + echo "@STATE_PREFIX@attempt=$attempt point=before $before_state" + first=$(nvx-time sample 2>&1) + echo "@SAMPLE_PREFIX@index=$index $first" + if grep -qx 'frequency_limited=1' "$state"; then + after_state=$(tr '\n' ' ' <"$state") + echo "@STATE_PREFIX@attempt=$attempt point=after $after_state" + break + fi + sleep @SPACING_SECONDS@ + second=$(nvx-time sample 2>&1) + index=$((index + 1)) + echo "@SAMPLE_PREFIX@index=$index $second" + after_state=$(tr '\n' ' ' <"$state") + echo "@STATE_PREFIX@attempt=$attempt point=after $after_state" + first_disruption=$(printf '%s\n' "$first" | sed -n 's/.*[[:space:]]disruption=\([^ ]*\).*/\1/p') + second_disruption=$(printf '%s\n' "$second" | sed -n 's/.*[[:space:]]disruption=\([^ ]*\).*/\1/p') + initial_signature=$(state_signature "$initial_state") + before_signature=$(state_signature "$before_state") + after_signature=$(state_signature "$after_state") + if [ -n "$first_disruption" ] && [ "$first_disruption" = "$second_disruption" ] && [ "$initial_signature" = "$before_signature" ] && [ "$before_signature" = "$after_signature" ]; then + break + fi + attempt=$((attempt + 1)) + index=$((index + 1)) +done diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index 87087378a..75c839ec9 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -140,6 +140,7 @@ "snapshot-tiers", "structured-outcome", "time-abi-conformance", + "time-host-correction", "virtio-net", "workload-identity", ) @@ -457,6 +458,57 @@ DECLARED_CPU_PROFILE = re.compile( rb'time ABI rates declared [^\r\n]*\bcpu_profile="([a-z0-9.-]+)"' ) +# The time-host-correction scenario's host clock corrections, in ppm, that +# OpenVMM's host-rate-offset-ppm hook makes it publish, against the sign of the +# host's own correction, which an unhooked cold boot's time sample gives: a +# cold boot's and a restore's within the guest's 500 ppm frequency limit, and +# a restore beyond that limit. The hook itself is bounded by OpenVMM's parser. +TIME_HOST_CORRECTION_BOOT_PPM = 200 +TIME_HOST_CORRECTION_RESTORE_PPM = 150 +TIME_HOST_CORRECTION_LIMIT_PPM = 900 +OPENVMM_HOST_RATE_OFFSET_LIMIT_PPM = 10_000 +# Seconds between the guest's two time samples of each phase. +TIME_HOST_CORRECTION_SPACING_SECONDS = 5 +# Seconds after clearing the PLL phase offset: longer than one kernel second even at Linux's +# slowest allowed correction, so the tick length no longer carries the cached phase correction. +TIME_HOST_CORRECTION_PHASE_SETTLE_SECONDS = 2 +# The most measurement windows retried after host UTC reports a disruption. +TIME_HOST_CORRECTION_ATTEMPTS = 3 +# How far, in ppm, the correction that the guest's boot step or restore repair +# took may be from the one that its first sample publishes, as the host's time +# service may change it in between: half the 150 ppm of the smallest hook, by +# which a guest that kept a stale correction would miss it. +TIME_HOST_CORRECTION_HELD_PPM = 75 +# How far, in ppm, the guest's clock may run from the host's correction that it +# holds, against the guest TSC counted at the declared rate, beyond the +# uncertainty of the samples' pairing of the guest's realtime clock with that +# TSC: the kernel runs a frequency correction to a fraction of a ppm, and a +# guest that didn't apply it would run 150 ppm or more off it. +TIME_HOST_CORRECTION_GUEST_PPM = 1 +# How far, in ppm, host UTC may run from its published correction against the +# same TSC, beyond half the correction's change between the samples and their +# pairing bounds: half the 150 ppm of the smallest hook that two samples +# bracket, which a hook that skewed host UTC or its correction alone would +# leave between them. The host's time service may slew host UTC between two +# samples, by up to 25 ppm on the test VMs measured so far, which the guest's +# discipline absorbs at its next poll. +TIME_HOST_CORRECTION_HOST_PPM = 75 +TIME_HOST_CORRECTION_STATE_PREFIX = "NVX-TIME-HOST-CORRECTION-STATE: " +TIME_HOST_CORRECTION_SAMPLE_PREFIX = "NVX-TIME-HOST-CORRECTION-SAMPLE: " +# The time-state fields that must stay fixed over one measurement window. +TIME_HOST_CORRECTION_DISCIPLINE_FIELDS = ( + "frequency_ppb", + "frequency_limited", + "host_correction_ppb", + "discontinuities", + "last_step_realtime_ns", + "samples", +) +TIME_HOST_CORRECTION_COMPLETION_MARKER = b"NVX-TIME-HOST-CORRECTION-OK" +# Linux's limit of the frequency correction, in ppb. +LINUX_FREQUENCY_LIMIT_PPB = 500_000 +# The flag of a time sample that a test hook altered. +TIME_SAMPLE_TEST_HOOKS = 0x08 # The guest finishes a restore after the RCU grace-period release, which gives # up after rcu_cpu_stall_timeout (21 s), and the deferred C7 check. TIME_ABI_RESTORE_FINISH_SECONDS = 30.0 @@ -5119,6 +5171,500 @@ def run_cpu_profile_fallback( ) +def time_host_correction_hook(ppm: int) -> tuple[str, str]: + """Return OpenVMM's test hook that publishes host UTC as running ``ppm`` + faster than it does, with the clock correction to match.""" + return ("--x-time-abi-test-hook", f"host-rate-offset-ppm={ppm}") + + +def time_host_correction_script( + spacing_seconds: int = TIME_HOST_CORRECTION_SPACING_SECONDS, + settle_seconds: int = TIME_HOST_CORRECTION_PHASE_SETTLE_SECONDS, +) -> str: + """Return the guest's report of the time-host-correction scenario: its time + state, then up to ``TIME_HOST_CORRECTION_ATTEMPTS`` pairs of samples + ``spacing_seconds`` apart until host UTC's disruption counter is stable, + unless Linux's limit holds its frequency correction.""" + return _render_script( + "time-host-correction.sh.in", + STATE_PREFIX=TIME_HOST_CORRECTION_STATE_PREFIX, + SAMPLE_PREFIX=TIME_HOST_CORRECTION_SAMPLE_PREFIX, + SPACING_SECONDS=str(spacing_seconds), + PHASE_SETTLE_SECONDS=str(settle_seconds), + ATTEMPTS=str(TIME_HOST_CORRECTION_ATTEMPTS), + ) + + +def _time_host_correction_report( + output: str, *, context: str +) -> tuple[list[dict[str, str]], list[dict[str, str]]]: + """Return the bracket states and samples of every measurement window in + the time-host-correction scenario's guest report.""" + states: list[dict[str, str]] = [] + samples: list[dict[str, str]] = [] + try: + for raw in output.splitlines(): + line = raw.removesuffix("\r") + if line.startswith(TIME_HOST_CORRECTION_STATE_PREFIX): + states.append( + parse_fields(line.removeprefix(TIME_HOST_CORRECTION_STATE_PREFIX)) + ) + elif line.startswith(TIME_HOST_CORRECTION_SAMPLE_PREFIX): + samples.append( + parse_fields(line.removeprefix(TIME_HOST_CORRECTION_SAMPLE_PREFIX)) + ) + except ValueError as error: + raise RuntimeError(f"{context}: malformed guest report: {error}") from error + if not states or not samples: + raise RuntimeError(f"{context}: the guest reported no time state or sample") + return states, samples + + +def _time_host_correction_window( + states: list[dict[str, str]], + samples: list[dict[str, str]], + *, + context: str, + limit: bool, +) -> tuple[dict[str, str], list[dict[str, str]]]: + """Return the state and samples of the first stable measurement window. + + Each window's states must keep the accepted-sample count, held correction, + and clock-step fields, and its samples the host disruption counter. A changed window is retried, + up to ``TIME_HOST_CORRECTION_ATTEMPTS`` times. + """ + for sample in samples: + if "theta_ns" not in sample: + raise RuntimeError( + f"{context}: nvx-time sample failed: " + + " ".join(f"{key}={value}" for key, value in sample.items()) + ) + if len(states) % 3: + raise RuntimeError( + f"{context}: the guest reported {len(states)} window states, not triples" + ) + windows = len(states) // 3 + if not 1 <= windows <= TIME_HOST_CORRECTION_ATTEMPTS: + raise RuntimeError( + f"{context}: the guest reported {windows} measurement windows, not 1 to " + f"{TIME_HOST_CORRECTION_ATTEMPTS}" + ) + for index, (initial, before, after) in enumerate( + zip(states[::3], states[1::3], states[2::3], strict=True), start=1 + ): + if ( + initial.get("attempt") != str(index) + or initial.get("point") != "initial" + or before.get("attempt") != str(index) + or before.get("point") != "before" + or after.get("attempt") != str(index) + or after.get("point") != "after" + ): + raise RuntimeError( + f"{context}: malformed state brackets for measurement window {index}" + ) + state_reports_limit = ( + windows == 1 and len(samples) == 1 and states[2].get("frequency_limited") == "1" + ) + if limit or state_reports_limit: + if windows != 1 or len(samples) != 1: + raise RuntimeError( + f"{context}: the limited phase reported {windows} windows and " + f"{len(samples)} samples, not one of each" + ) + return states[2], samples + if len(samples) != 2 * windows: + raise RuntimeError( + f"{context}: the guest reported {len(samples)} time samples for " + f"{windows} measurement windows, not {2 * windows}" + ) + for window in range(windows): + initial, before, after = states[3 * window : 3 * window + 3] + first, second = samples[2 * window : 2 * window + 2] + missing = [ + field + for field in TIME_HOST_CORRECTION_DISCIPLINE_FIELDS + if any(field not in state for state in (initial, before, after)) + ] + if missing: + raise RuntimeError( + f"{context}: measurement window {window + 1} states lack " + + ", ".join(missing) + ) + signature_initial = tuple( + initial[field] for field in TIME_HOST_CORRECTION_DISCIPLINE_FIELDS + ) + signature_before = tuple( + before[field] for field in TIME_HOST_CORRECTION_DISCIPLINE_FIELDS + ) + signature_after = tuple( + after[field] for field in TIME_HOST_CORRECTION_DISCIPLINE_FIELDS + ) + first_disruption = first.get("disruption") + if ( + signature_initial == signature_before == signature_after + and first_disruption is not None + and first_disruption == second.get("disruption") + ): + return after, [first, second] + raise RuntimeError( + f"{context}: host UTC or its discipline changed in all {windows} " + "measurement windows" + ) + + +def time_host_correction_host(output: str, *, context: str) -> tuple[float, int]: + """Return the host's own clock correction, in ppm, and the declared TSC + rate that the unhooked guest report publishes. + + No test hook may have altered the first sample. The TSC rate determines + whether Linux's kHz rounding would saturate an ordinary hooked phase. + """ + states, samples = _time_host_correction_report(output, context=context) + state, samples = _time_host_correction_window( + states, samples, context=context, limit=False + ) + sample = samples[0] + try: + if "theta_ns" not in sample: + raise RuntimeError( + f"{context}: nvx-time sample failed: " + + " ".join(f"{key}={value}" for key, value in sample.items()) + ) + if int(sample["flags"], 16) & TIME_SAMPLE_TEST_HOOKS: + raise RuntimeError( + f"{context}: time sample flags {sample['flags']} carry the " + "test-hook flag, so a hook altered the host's own correction" + ) + return int(sample["correction"]) / 65_536, int(state["tsc_hz"]) + except (KeyError, ValueError) as error: + raise RuntimeError(f"{context}: malformed guest report: {error!r}") from error + + +def time_host_correction_host_ppm(output: str, *, context: str) -> float: + """Return only the host correction from [`time_host_correction_host`].""" + return time_host_correction_host(output, context=context)[0] + + +def time_host_correction_composed_ppm(host_ppm: float, hook_ppm: int) -> float: + """Return the published correction when OpenVMM composes the host's rate + with ``hook_ppm`` multiplicatively.""" + return host_ppm + hook_ppm + host_ppm * hook_ppm / 1_000_000 + + +def time_host_correction_guest_ppm(correction_ppm: float, tsc_hz: int) -> float: + """Return the correction that Linux must hold after it rounds + ``tsc_hz`` down to the kHz.""" + if tsc_hz <= 0: + raise RuntimeError(f"the declared TSC rate is invalid: {tsc_hz} Hz") + kernel_hz = tsc_hz // 1000 * 1000 + return (1_000_000 + correction_ppm) * kernel_hz / tsc_hz - 1_000_000 + + +def time_host_correction_hooks(host_ppm: float, tsc_hz: int) -> tuple[int, int, int]: + """Return the ``host-rate-offset-ppm`` hooks, in ppm, of the + time-host-correction scenario's cold boot, restore, and limit restore on a + host whose own clock correction is ``host_ppm``. + + The first two aim to make OpenVMM publish corrections of + ``TIME_HOST_CORRECTION_BOOT_PPM`` and ``TIME_HOST_CORRECTION_RESTORE_PPM`` + against the sign of the host's own. OpenVMM composes the two rates + multiplicatively, so solve that composition for the hook. The hooks stay + within OpenVMM's accepted range; near its edge they publish the closest + correction that remains below Linux's 500 ppm limit. A host whose + correction cannot be brought below that limit fails here with an explicit + diagnostic. The last hook extends the host's own correction beyond Linux's + limit. + """ + sign = 1 if host_ppm >= 0 else -1 + denominator = 1_000_000 + host_ppm + if denominator <= 0: + raise RuntimeError(f"the host's own correction is invalid: {host_ppm:+.3f} ppm") + + def ordinary(target_ppm: int) -> int: + target = -sign * target_ppm + hook = round((target - host_ppm) * 1_000_000 / denominator) + hook = max( + -OPENVMM_HOST_RATE_OFFSET_LIMIT_PPM, + min(OPENVMM_HOST_RATE_OFFSET_LIMIT_PPM, hook), + ) + published = time_host_correction_composed_ppm(host_ppm, hook) + guest = time_host_correction_guest_ppm(published, tsc_hz) + if abs(guest) >= LINUX_FREQUENCY_LIMIT_PPB / 1000: + raise RuntimeError( + f"the host's own correction of {host_ppm:+.3f} ppm cannot be " + "composed with Linux's rounded TSC rate below its 500 ppm " + "limit within OpenVMM's host-rate-offset-ppm range of +/-" + f"{OPENVMM_HOST_RATE_OFFSET_LIMIT_PPM:,} ppm (published " + f"{published:+.3f} ppm, guest {guest:+.3f} ppm at {tsc_hz} Hz)" + ) + return hook + + return ( + ordinary(TIME_HOST_CORRECTION_BOOT_PPM), + ordinary(TIME_HOST_CORRECTION_RESTORE_PPM), + sign * TIME_HOST_CORRECTION_LIMIT_PPM, + ) + + +def time_host_correction_frequency_allowance_ppb( + tsc_hz: int, correction_ppb: int +) -> int: + """Return the rounding allowance between the host and kernel corrections. + + Linux counts the declared TSC rate rounded down to the kHz, up to 999 Hz + slow. Scale that worst case at ``tsc_hz`` by the correction, and allow one + more ppb for the correction's integer rounding. The time ABI admits rates + as low as 500 MHz, where this is nearly 2,000 ppb rather than the 1,001 ppb + sufficient at 1 GHz. + """ + if tsc_hz <= 0: + raise ValueError(f"invalid TSC rate {tsc_hz}") + numerator = 999 * (1_000_000_000 + abs(correction_ppb)) + return (numerator + tsc_hz - 1) // tsc_hz + 1 + + +def check_time_host_correction( + output: str, *, context: str, limit: bool = False +) -> str: + """Check the guest's report of the time-host-correction scenario in its + console ``output``, and return a summary of it. + + The correction that the guest's boot step or restore repair took must be + the one that its first sample publishes, give or take the host's own + changes, so that it isn't a stale one. With ``limit``, that correction must + exceed Linux's limit, and the guest must hold the limit and report it. + Otherwise it must lie within the limit, and the guest's frequency + correction must be it, short of the kernel's kHz rounding of the TSC rate; + between its two samples, the guest's clock must run at the correction that + it holds against the guest TSC counted at the declared rate, as each sample + pairs the clock with the TSC, and host UTC, which each sample pairs with + the same TSC, at its published correction, give or take the slews of the + host's time service, so that the hook skewed both alike. A pair whose + disruption counter changes is discarded in favor of one of the bounded retry + windows. Every sample must carry OpenVMM's test-hook flag, so that the hook + reached it.""" + states, samples = _time_host_correction_report(output, context=context) + state, samples = _time_host_correction_window( + states, samples, context=context, limit=limit + ) + try: + for sample in samples: + if "theta_ns" not in sample: + raise RuntimeError( + f"{context}: nvx-time sample failed: " + + " ".join(f"{key}={value}" for key, value in sample.items()) + ) + if int(sample["flags"], 16) & TIME_SAMPLE_TEST_HOOKS == 0: + raise RuntimeError( + f"{context}: time sample flags {sample['flags']} lack the " + "test-hook flag, so OpenVMM's hook did not reach it" + ) + frequency_ppb = int(state["frequency_ppb"]) + correction_ppb = int(state["host_correction_ppb"]) + limited = state["frequency_limited"] == "1" + + def check_held(sample: dict[str, str]) -> float: + published = int(sample["correction"]) / 65_536 + if abs(correction_ppb / 1000 - published) > TIME_HOST_CORRECTION_HELD_PPM: + raise RuntimeError( + f"{context}: the guest holds the host's correction of " + f"{correction_ppb} ppb, but its first stable sample publishes " + f"{published:+.3f} ppm, more than " + f"{TIME_HOST_CORRECTION_HELD_PPM} ppm away: its boot step or " + "restore repair kept a stale correction" + ) + return published + + tsc_hz = int(state["tsc_hz"]) + required_ppm = time_host_correction_guest_ppm(correction_ppb / 1000, tsc_hz) + beyond = abs(required_ppm) > LINUX_FREQUENCY_LIMIT_PPB / 1000 + if limit: + check_held(samples[0]) + if not beyond: + raise RuntimeError( + f"{context}: Linux's required correction of " + f"{required_ppm:+.3f} ppm is within its limit, so this phase " + "does not exercise it" + ) + held = ( + LINUX_FREQUENCY_LIMIT_PPB + if required_ppm > 0 + else (-LINUX_FREQUENCY_LIMIT_PPB) + ) + if not limited or frequency_ppb != held: + raise RuntimeError( + f"{context}: Linux's required correction of " + f"{required_ppm:+.3f} ppm exceeds its limit, but the guest " + f"holds {frequency_ppb} ppb " + f"with frequency_limited={state['frequency_limited']}, not " + f"{held} ppb with frequency_limited=1" + ) + return ( + f"{context}: host_correction_ppb={correction_ppb} " + f"frequency_ppb={frequency_ppb} frequency_limited=1" + ) + if limited or beyond: + raise RuntimeError( + f"{context}: Linux's required correction of {required_ppm:+.3f} " + f"ppm reaches its limit (frequency_limited=" + f"{state['frequency_limited']}), so " + "this phase cannot check the guest's rate" + ) + first, second = samples + published_ppm = check_held(first) + allowance_ppb = time_host_correction_frequency_allowance_ppb( + tsc_hz, correction_ppb + ) + if abs(frequency_ppb - correction_ppb) > allowance_ppb: + raise RuntimeError( + f"{context}: the guest holds a frequency correction of " + f"{frequency_ppb} ppb, not the host's correction of " + f"{correction_ppb} ppb (allowance {allowance_ppb} ppb at " + f"{tsc_hz} Hz)" + ) + # The guest TSC, counted at the declared rate, paces both clocks: each + # sample pairs it with host UTC and, at host UTC less theta, with the + # guest's realtime clock, within its uncertainty less the pairing bound. + elapsed_ns = (int(second["tsc"]) - int(first["tsc"])) * 1e9 / tsc_hz + utc_ns = int(second["utc_ns"]) - int(first["utc_ns"]) + realtime_ns = utc_ns - (int(second["theta_ns"]) - int(first["theta_ns"])) + realtime_error_ns = sum( + int(sample["epsilon_ns"]) - int(sample["pairing_ns"]) for sample in samples + ) + pairing_error_ns = sum(int(sample["pairing_ns"]) for sample in samples) + first_ppm, second_ppm = ( + int(sample["correction"]) / 65_536 for sample in samples + ) + except (KeyError, ValueError, ZeroDivisionError) as error: + raise RuntimeError(f"{context}: malformed guest report: {error!r}") from error + if elapsed_ns <= 0: + raise RuntimeError( + f"{context}: the guest TSC did not advance between the samples" + ) + seconds = elapsed_ns / 1e9 + held_ppm = correction_ppb / 1000 + guest_ppm = (realtime_ns / elapsed_ns - 1) * 1e6 + guest_limit = TIME_HOST_CORRECTION_GUEST_PPM + realtime_error_ns / elapsed_ns * 1e6 + if abs(guest_ppm - held_ppm) > guest_limit: + raise RuntimeError( + f"{context}: the guest's clock ran at {guest_ppm:+.3f} ppm against its " + f"TSC in {seconds:.1f} s, not at the host's correction of " + f"{held_ppm:+.3f} ppm that it holds, within {guest_limit:.3f} ppm" + ) + host_ppm = (utc_ns / elapsed_ns - 1) * 1e6 + published_ppm = (first_ppm + second_ppm) / 2 + host_limit = ( + TIME_HOST_CORRECTION_HOST_PPM + + abs(second_ppm - first_ppm) / 2 + + pairing_error_ns / elapsed_ns * 1e6 + ) + if abs(host_ppm - published_ppm) > host_limit: + raise RuntimeError( + f"{context}: host UTC ran at {host_ppm:+.3f} ppm against the guest TSC " + f"in {seconds:.1f} s, not at its published correction of " + f"{published_ppm:+.3f} ppm, within {host_limit:.3f} ppm" + ) + return ( + f"{context}: host_correction_ppb={correction_ppb} frequency_ppb={frequency_ppb} " + f"guest_ppm={guest_ppm:+.3f} host_ppm={host_ppm:+.3f} " + f"published_ppm={published_ppm:+.3f} in {seconds:.1f} s" + ) + + +def run_time_host_correction( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Check that the guest applies the host's clock correction that OpenVMM + publishes with each time sample, through the ``host-rate-offset-ppm`` test + hook, which publishes host UTC as running faster or slower than it does. + + An unhooked cold boot's time sample gives the host's own correction, from + which ``time_host_correction_hooks`` chooses the hooks. A cold boot with the + hook must hold the published correction from its boot step on, and keep its + clock on the hooked host UTC. A snapshot, captured without the hook, + restored with another rate must hold the destination's correction from its + repair on, and pass the warp probe and its restore checks; restored with a + rate beyond Linux's 500 ppm limit, in the direction of the host's own + correction, it must hold the limit and report it.""" + boot_command = workload_boot_command( + executable, + backend, + kernel, + initrd, + memory_mib, + "quiet loglevel=0", + processors=1, + ) + completion = f"echo {TIME_HOST_CORRECTION_COMPLETION_MARKER.decode()}\nnvx-exit 0\n" + context = "host clock correction probe" + probe = run_guest_script( + boot_command, + time_host_correction_script(spacing_seconds=0) + completion, + TIME_HOST_CORRECTION_COMPLETION_MARKER, + timeout=timeout, + log_path=output_dir / "time-host-correction-probe.log", + ) + host_ppm, tsc_hz = time_host_correction_host(probe["output"], context=context) + boot_ppm, restore_ppm, limit_ppm = time_host_correction_hooks(host_ppm, tsc_hz) + print(f"{context}: the host's own correction is {host_ppm:+.3f} ppm", flush=True) + context = f"host clock correction cold boot (hook {boot_ppm:+d} ppm)" + result = run_guest_script( + [*boot_command, *time_host_correction_hook(boot_ppm)], + time_host_correction_script() + completion, + TIME_HOST_CORRECTION_COMPLETION_MARKER, + timeout=timeout, + log_path=output_dir / "time-host-correction-boot.log", + ) + print(check_time_host_correction(result["output"], context=context), flush=True) + with tempfile.TemporaryDirectory(prefix="nvx-time-host-correction-") as temporary: + snapshot_path = Path(temporary) / "snapshot" + capture_snapshot( + [*boot_command, "--snapshot-destination", str(snapshot_path)], + snapshot_path, + timeout=timeout, + processors=1, + post_restore_script=time_host_correction_script() + post_restore_checks(), + log_path=output_dir / "time-host-correction-capture.log", + ) + for name, ppm, limit in ( + ("restore", restore_ppm, False), + ("limit", limit_ppm, True), + ): + context = f"host clock correction {name} (hook {ppm:+d} ppm)" + restore_command = [ + *snapshot_restore_command( + executable, backend, snapshot_path, processors=1 + ), + *time_host_correction_hook(ppm), + ] + output = _measure_restore( + restore_command, + context=context, + environment=_restore_environment(), + timeout=timeout, + log_path=output_dir / f"time-host-correction-{name}.log", + failure_marker=WARP_PROBE_FAILURE_MARKER, + ) + print( + check_time_host_correction( + output.decode("utf-8", "replace"), context=context, limit=limit + ), + flush=True, + ) + _check_restore_warp(output, processors=1, context=context) + _check_restore_status( + output, restore_command, processors=1, context=context + ) + + def _restore_vp_bindings(output: bytes) -> list[int]: """Return the VP indices bound by one profiled OpenVMM process.""" bound: list[int] = [] @@ -9583,6 +10129,20 @@ def _run_scenarios( timeout=args.timeout, log_path=output_dir / "time-abi-conformance.log", ) + if "time-host-correction" in scenarios: + report.start( + "time-host-correction", + f"Running microVM host clock correction correctness on OpenVMM/{args.backend}", + ) + run_time_host_correction( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "virtio-net" in scenarios: report.start( "virtio-net", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 0e43ae6e0..cc3607432 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -13291,6 +13291,45 @@ def require(path: Path, _description: str) -> Path: ) self.assertEqual(run_guest_boot.call_args.kwargs["memory_mib"], 512) + def test_runner_runs_time_host_correction_on_both_guest_images(self): + def require(path: Path, _description: str) -> Path: + return path + + for guest in ("alpine", "ubuntu"): + descriptor = microvm_tests.guest_descriptor(guest) + with self.subTest(guest=guest), tempfile.TemporaryDirectory() as temporary: + args = nvx.parse_args( + [ + "test-microvm", + "--backend", + "whp", + "--guest", + guest, + "--scenario", + "time-host-correction", + "--output-dir", + temporary, + ] + ) + with ( + patch.object(microvm_tests, "validate_openvmm_test_backend"), + patch.object(microvm_tests, "require_file", side_effect=require), + patch.object( + microvm_tests, "run_time_host_correction" + ) as run_time_host_correction, + ): + self.assertEqual(microvm_tests.run(args), 0) + + run_time_host_correction.assert_called_once() + self.assertEqual( + run_time_host_correction.call_args.args[2], + BuildConstants.BUILD_DIR / descriptor.initramfs_name, + ) + self.assertEqual( + run_time_host_correction.call_args.kwargs["memory_mib"], + descriptor.default_memory_mib, + ) + def test_runner_omits_console_snapshot_from_ubuntu_defaults(self): def require(path: Path, _description: str) -> Path: return path @@ -13806,6 +13845,602 @@ def close(self) -> None: self.assertEqual(log_path.read_bytes(), b"complete raw output\n") +class TimeHostCorrectionTests(unittest.TestCase): + """The time-host-correction scenario and its guest report.""" + + # Host UTC of the first sample, and the declared TSC rate; the samples are + # 5 s of the guest TSC apart. + UTC_NS = 1_790_841_600_000_000_000 + TSC_HZ = 2_000_000_000 + + def report( + self, + *, + frequency_ppb: int = 199_500, + correction_ppb: int = 200_000, + limited: int = 0, + samples: Sequence[tuple[int, int]] = ((1_000, 13_107_200), (1_500, 13_107_200)), + host_ppm: float = 200.0, + flags: str = "0x08", + tsc_hz: int | None = None, + disruptions: Sequence[int] | None = None, + state_windows: Sequence[ + tuple[ + dict[str, str | int], + dict[str, str | int], + dict[str, str | int], + ] + ] + | None = None, + ) -> str: + """Return the console output of a guest report whose state holds + ``frequency_ppb`` for the host's ``correction_ppb``, and whose samples, + 5 s of the guest TSC apart, read ``(theta_ns, correction)`` each, while + host UTC runs ``host_ppm`` against the TSC.""" + tsc_hz = self.TSC_HZ if tsc_hz is None else tsc_hz + state: dict[str, str | int] = { + "version": 1, + "generation": 0, + "tsc_hz": tsc_hz, + "frequency_ppb": frequency_ppb, + "frequency_limited": limited, + "host_status": "synchronized", + "host_correction_ppb": correction_ppb, + "host_correction_bound_ppb": 2, + "discontinuities": 0, + "last_step_realtime_ns": 0, + "samples": 17, + } + disruptions = disruptions or (0,) * len(samples) + sample_lines: list[str] = [] + for index, ((theta, correction), disruption) in enumerate( + zip(samples, disruptions, strict=True) + ): + elapsed_ns = index * 5_000_000_000 + utc = self.UTC_NS + round(elapsed_ns * (1 + host_ppm / 1e6)) + tsc = 1_000_000 + index * 5 * tsc_hz + sample_lines.append( + f"{microvm_tests.TIME_HOST_CORRECTION_SAMPLE_PREFIX}index={index + 1} " + f"flags={flags} generation=0 status=synchronized " + f"utc_ns={utc} tsc={tsc} pairing_ns=640 " + f"disruption={disruption} correction={correction} correction_bound=66 " + f"max_error_us=unknown before={tsc - 100} after={tsc + 100} " + f"realtime_ns={utc - theta} theta_ns={theta} epsilon_ns=700 " + f"frequency=13103527 frequency_limited={limited}" + ) + windows = max(1, (len(samples) + 1) // 2) + if state_windows is None: + state_windows = tuple(({}, {}, {}) for _ in range(windows)) + if len(state_windows) != windows: + raise ValueError( + f"{len(state_windows)} state windows for {windows} windows" + ) + + def format_state(values: dict[str, str | int]) -> str: + return " ".join(f"{name}={value}" for name, value in values.items()) + + lines: list[str] = [] + for window, (initial_changes, before_changes, after_changes) in enumerate( + state_windows, start=1 + ): + initial = state.copy() + initial.update(initial_changes) + before = state.copy() + before.update(before_changes) + after = state.copy() + after.update(after_changes) + lines.append( + f"{microvm_tests.TIME_HOST_CORRECTION_STATE_PREFIX}attempt={window} " + f"point=initial {format_state(initial)}" + ) + lines.append( + f"{microvm_tests.TIME_HOST_CORRECTION_STATE_PREFIX}attempt={window} " + f"point=before {format_state(before)}" + ) + lines.extend(sample_lines[2 * (window - 1) : 2 * window]) + lines.append( + f"{microvm_tests.TIME_HOST_CORRECTION_STATE_PREFIX}attempt={window} " + f"point=after {format_state(after)}" + ) + return "\r\n".join(lines) + "\r\n" + + def check(self, output: str, *, limit: bool = False) -> str: + return microvm_tests.check_time_host_correction( + output, context="phase", limit=limit + ) + + def test_a_guest_that_holds_the_published_correction_passes(self): + # The guest's clock lost 500 ns on host UTC in 5 s, so it ran at + # 199.9 ppm against its TSC, within 1 ppm and the 120 ns of its two + # realtime pairings. + summary = self.check(self.report()) + self.assertEqual( + summary, + "phase: host_correction_ppb=200000 frequency_ppb=199500 " + "guest_ppm=+199.900 host_ppm=+200.000 published_ppm=+200.000 in 5.0 s", + ) + script = microvm_tests.time_host_correction_script() + self.assertNotIn("@", script) + self.assertIn("sleep 5\n", script) + self.assertIn( + "sleep 0\n", microvm_tests.time_host_correction_script(spacing_seconds=0) + ) + self.assertIn('while [ "$attempt" -le 3 ]', script) + self.assertNotIn("/sbin/adjtimex", script) + self.assertLess(script.index("initial_state="), script.index("clear-phase")) + self.assertLess(script.index("clear-phase"), script.index("sleep 2")) + self.assertLess(script.index("sleep 2"), script.index("before_state=")) + self.assertIn( + "sleep 0\n", microvm_tests.time_host_correction_script(settle_seconds=0) + ) + self.assertIn("last_step_realtime_ns samples", script) + self.assertIn("first_disruption=", script) + # A discarded sample precedes the report's samples. + self.assertLess( + script.index("nvx-time sample >/dev/null 2>&1 || true\n"), + script.index("first=$(nvx-time sample 2>&1)"), + ) + + def test_a_sub_ghz_tsc_gets_its_full_khz_rounding_allowance(self): + # The ABI admits 500 MHz. At 500,000,999 Hz, a +200 ppm host + # correction becomes +198,002 ppb after Linux rounds the TSC rate down + # to the kHz: 1,998 ppb away, which the old 1,001 ppb allowance rejected. + tsc_hz = 500_000_999 + report = self.report( + frequency_ppb=198_002, + correction_ppb=200_000, + tsc_hz=tsc_hz, + ) + self.assertIn("frequency_ppb=198002", self.check(report)) + self.assertGreaterEqual( + microvm_tests.time_host_correction_frequency_allowance_ppb(tsc_hz, 200_000), + 1_998, + ) + + def test_a_guest_that_does_not_run_at_the_published_rate_fails(self): + with self.assertRaisesRegex( + RuntimeError, + "holds a frequency correction of 0 ppb, not the host's correction of " + "200000 ppb", + ): + self.check(self.report(frequency_ppb=0)) + # A guest whose clock doesn't run at the frequency correction that it + # holds drifts from host UTC by the hook's rate: 200 ppm over 5 s is + # 1 ms. + with self.assertRaisesRegex( + RuntimeError, + r"the guest's clock ran at -0\.200 ppm against its TSC in 5\.0 s, not at " + r"the host's correction of \+200\.000 ppm that it holds, within 1\.024 ppm", + ): + self.check(self.report(samples=((0, 13_107_200), (1_001_000, 13_107_200)))) + # The limit is 1 ppm plus the uncertainty of the two realtime pairings, + # 0.024 ppm, so a guest that ran 1.2 ppm off fails. + with self.assertRaisesRegex(RuntimeError, r"ran at \+198\.800 ppm"): + self.check(self.report(samples=((0, 13_107_200), (6_000, 13_107_200)))) + + def test_host_utc_must_run_at_its_published_correction(self): + # A hook that skewed the correction but not host UTC leaves the guest, + # which holds the correction, 1 ms behind host UTC in 5 s. + with self.assertRaisesRegex( + RuntimeError, + r"host UTC ran at \+0\.000 ppm against the guest TSC in 5\.0 s, not at its " + r"published correction of \+200\.000 ppm, within 75\.256 ppm", + ): + self.check( + self.report( + samples=((0, 13_107_200), (-1_000_000, 13_107_200)), host_ppm=0.0 + ) + ) + # The time service may slew host UTC between the samples, as the guest's + # discipline absorbs. + slewed = self.report( + samples=((0, 13_107_200), (-350_000, 13_107_200)), host_ppm=130.0 + ) + self.assertIn("host_ppm=+130.000 published_ppm=+200.000", self.check(slewed)) + + def test_a_host_utc_disruption_retries_the_measurement_window(self): + correction = 13_107_200 + samples = ((1_000, correction),) * 4 + retried = self.report( + samples=samples, + disruptions=(0, 1, 1, 1), + ) + self.assertIn( + "guest_ppm=+200.000 host_ppm=+200.000", + self.check(retried), + ) + with self.assertRaisesRegex( + RuntimeError, + "host UTC or its discipline changed in all 3 measurement windows", + ): + self.check( + self.report( + samples=((1_000, correction),) * 6, + disruptions=(0, 1, 1, 2, 2, 3), + ) + ) + + def test_a_discipline_update_retries_with_fresh_state(self): + correction = 16_384_000 + unchanged: dict[str, str | int] = {} + held: dict[str, str | int] = { + "frequency_ppb": 249_500, + "host_correction_ppb": 250_000, + } + retried = self.report( + frequency_ppb=199_500, + correction_ppb=200_000, + samples=((1_000, correction),) * 4, + host_ppm=250.0, + state_windows=((unchanged, unchanged, held), (held, held, held)), + ) + self.assertIn( + "host_correction_ppb=250000 frequency_ppb=249500 " + "guest_ppm=+250.000 host_ppm=+250.000", + self.check(retried), + ) + + def test_a_phase_only_poll_retries_the_measurement_window(self): + correction = 13_107_200 + unchanged: dict[str, str | int] = {} + polled: dict[str, str | int] = {"samples": 18} + for first_window in ( + # A poll during the measurement changes only its accepted-sample count. + (unchanged, unchanged, polled), + # A poll between the initial state and phase cancellation is caught too. + (unchanged, polled, polled), + ): + with self.subTest(first_window=first_window): + retried = self.report( + samples=((1_000, correction),) * 4, + state_windows=(first_window, (polled, polled, polled)), + ) + self.assertIn( + "guest_ppm=+200.000 host_ppm=+200.000", + self.check(retried), + ) + + def test_a_wandering_host_widens_the_allowance(self): + # The host's correction moved from 100 to 300 ppm between the samples: + # host UTC ran at their average, and half the change widens the limit + # by 100 ppm. The guest holds the first sample's 100 ppm. + wandering = ((0, 6_553_600), (500_000, 19_660_800)) + self.assertIn( + "guest_ppm=+100.000 host_ppm=+200.000 published_ppm=+200.000", + self.check( + self.report( + frequency_ppb=99_500, correction_ppb=100_000, samples=wandering + ) + ), + ) + with self.assertRaisesRegex(RuntimeError, r"within 175\.256 ppm"): + self.check( + self.report( + frequency_ppb=99_500, + correction_ppb=100_000, + samples=((0, 6_553_600), (1_400_000, 19_660_800)), + host_ppm=380.0, + ) + ) + + def test_a_stale_held_correction_fails(self): + # A boot step or repair that kept the guest's old correction, here 0, + # runs the guest at it while the samples publish, and host UTC runs at, + # -150 ppm: both rate checks pass, but the held correction is stale. + with self.assertRaisesRegex( + RuntimeError, + r"holds the host's correction of 0 ppb, but its first stable sample publishes " + r"-150\.000 ppm, more than 75 ppm away: its boot step or restore repair " + "kept a stale correction", + ): + self.check( + self.report( + frequency_ppb=0, + correction_ppb=0, + samples=((0, -9_830_400), (-750_000, -9_830_400)), + host_ppm=-150.0, + ) + ) + # The host's own correction may have moved since the boot step or the + # repair: here by 60 ppm, which the guest's discipline follows later. + moved = self.report( + samples=((0, 17_039_360), (300_000, 17_039_360)), host_ppm=260.0 + ) + self.assertIn("guest_ppm=+200.000 host_ppm=+260.000", self.check(moved)) + + def test_the_limit_holds_and_is_reported(self): + limit = self.report( + frequency_ppb=500_000, + correction_ppb=900_000, + limited=1, + samples=((0, 58_982_400),), + ) + self.assertEqual( + self.check(limit, limit=True), + "phase: host_correction_ppb=900000 frequency_ppb=500000 frequency_limited=1", + ) + for frequency_ppb, limited in ((500_000, 0), (499_000, 1)): + with ( + self.subTest(frequency_ppb=frequency_ppb, limited=limited), + self.assertRaisesRegex(RuntimeError, "exceeds its limit"), + ): + self.check( + self.report( + frequency_ppb=frequency_ppb, + correction_ppb=900_000, + limited=limited, + samples=((0, 58_982_400),), + ), + limit=True, + ) + negative = self.report( + frequency_ppb=-500_000, + correction_ppb=-650_000, + limited=1, + samples=((0, -42_598_400),), + ) + self.assertIn("frequency_limited=1", self.check(negative, limit=True)) + # The limit phase must exercise the limit: a host correction of + # -450 ppm and a +900 ppm hook would leave +450 ppm, unclamped. + unclamped = self.report( + frequency_ppb=449_500, + correction_ppb=450_000, + samples=((0, 29_491_200), (0, 29_491_200)), + host_ppm=450.0, + ) + self.assertIn("guest_ppm=+450.000", self.check(unclamped)) + with self.assertRaisesRegex( + RuntimeError, + r"Linux's required correction of \+450\.000 ppm is within its limit, " + "so this phase does not exercise it", + ): + self.check( + self.report( + frequency_ppb=449_500, + correction_ppb=450_000, + samples=((0, 29_491_200),), + host_ppm=450.0, + ), + limit=True, + ) + # And the other phases must not reach it, which leaves no rate to check. + with self.assertRaisesRegex( + RuntimeError, + r"reaches its limit \(frequency_limited=1\), so this phase cannot " + "check the guest's rate", + ): + self.check(limit) + + def test_a_report_without_the_hook_or_a_sample_fails(self): + with self.assertRaisesRegex(RuntimeError, "lack the test-hook flag"): + self.check(self.report(flags="0x00")) + with self.assertRaisesRegex( + RuntimeError, "reported 1 time samples for 1 measurement windows, not 2" + ): + self.check(self.report(samples=((0, 13_107_200),))) + with self.assertRaisesRegex(RuntimeError, "malformed guest report: KeyError"): + self.check(self.report().replace(f"tsc_hz={self.TSC_HZ} ", "")) + with self.assertRaisesRegex(RuntimeError, "no time state or sample"): + self.check("NVX-TIME-HOST-CORRECTION-SAMPLE: index=1 theta_ns=0\n") + failed = ( + f"{microvm_tests.TIME_HOST_CORRECTION_STATE_PREFIX}frequency_ppb=0\n" + f"{microvm_tests.TIME_HOST_CORRECTION_SAMPLE_PREFIX}index=1 " + "nvx-time: portb\n" + ) + with self.assertRaisesRegex(RuntimeError, "phase: "): + self.check(failed) + + def test_the_probe_reads_the_hosts_own_correction(self): + probe = self.phase_report(-12.5, flags="0x00") + self.assertEqual( + microvm_tests.time_host_correction_host_ppm(probe, context="probe"), -12.5 + ) + with self.assertRaisesRegex( + RuntimeError, + "probe: time sample flags 0x08 carry the test-hook flag, so a hook " + "altered the host's own correction", + ): + microvm_tests.time_host_correction_host_ppm(self.report(), context="probe") + failed = ( + f"{microvm_tests.TIME_HOST_CORRECTION_STATE_PREFIX}frequency_ppb=0\n" + f"{microvm_tests.TIME_HOST_CORRECTION_SAMPLE_PREFIX}index=1 error=portb\n" + ) + with self.assertRaisesRegex( + RuntimeError, "probe: nvx-time sample failed: index=1 error=portb" + ): + microvm_tests.time_host_correction_host_ppm(failed, context="probe") + + def test_the_hooks_keep_the_ordinary_corrections_within_the_limit(self): + def hooks(host_ppm: float, tsc_hz: int = self.TSC_HZ) -> tuple[int, int, int]: + return microvm_tests.time_host_correction_hooks(host_ppm, tsc_hz) + + self.assertEqual(hooks(0.0), (-200, -150, 900)) + self.assertEqual(hooks(-12.4), (212, 162, -900)) + self.assertEqual(hooks(400.0), (-600, -550, 900)) + self.assertEqual(hooks(-400.0), (600, 550, -900)) + # At OpenVMM's +/-10,000 ppm hook limit, use the boundary instead of + # generating the invalid -10,200 or +10,200 ppm hooks. + self.assertEqual(hooks(10_000.0), (-10_000, -10_000, 900)) + self.assertEqual(hooks(-10_000.0), (10_000, 10_000, -900)) + for host_ppm in (-499.9, -250.0, -150.0, -0.4, 0.4, 150.0, 250.0, 499.9): + boot, restore, limit = hooks(host_ppm) + for hook, size in ((boot, 200), (restore, 150)): + with self.subTest(host_ppm=host_ppm, hook=hook): + # The published correction has the size, to the hook's + # rounding, against the host's sign, and the hook is at + # least as large. + published = microvm_tests.time_host_correction_composed_ppm( + host_ppm, hook + ) + self.assertLessEqual(abs(abs(published) - size), 0.5) + self.assertLess(published * host_ppm, 0) + self.assertGreaterEqual(abs(hook), size) + with self.subTest(host_ppm=host_ppm, hook=limit): + # The limit phase's hook has the host's sign, and takes the + # published correction beyond Linux's limit. + self.assertGreater(limit * host_ppm, 0) + self.assertGreater( + abs( + microvm_tests.time_host_correction_composed_ppm(host_ppm, limit) + ), + 500, + ) + self.assertLessEqual( + abs(limit), microvm_tests.OPENVMM_HOST_RATE_OFFSET_LIMIT_PPM + ) + # Multiplication makes +10,500 with a -10,000 hook publish +395 + # ppm, so it remains a valid ordinary phase. The negative boundary is + # asymmetric: -10,400 composed with +10,000 publishes -504 ppm. + self.assertEqual(hooks(10_500.0), (-10_000, -10_000, 900)) + self.assertAlmostEqual( + microvm_tests.time_host_correction_composed_ppm(10_500.0, -10_000), + 395.0, + ) + self.assertEqual(hooks(-10_396.0), (10_000, 10_000, -900)) + # At the ABI's minimum TSC rate, Linux's kHz rounding turns the + # otherwise valid -499.96 ppm boundary into -501.957 ppm. + with self.assertRaisesRegex(RuntimeError, "guest -501.957 ppm at 500000999 Hz"): + hooks(-10_396.0, 500_000_999) + # The opposite boundary publishes +500.930 ppm but needs only + # +498.931 ppm from Linux, so both the solver and checker accept it. + tsc_hz = 500_000_999 + self.assertEqual(hooks(10_607.0, tsc_hz), (-10_000, -10_000, 900)) + published = microvm_tests.time_host_correction_composed_ppm(10_607.0, -10_000) + required = microvm_tests.time_host_correction_guest_ppm(published, tsc_hz) + self.assertAlmostEqual(published, 500.93, places=3) + self.assertAlmostEqual(required, 498.931, places=3) + correction = round(published * 65_536) + boundary = self.report( + frequency_ppb=round(required * 1000), + correction_ppb=round(published * 1000), + samples=((1_000, correction), (1_000, correction)), + host_ppm=published, + tsc_hz=tsc_hz, + ) + self.assertIn("frequency_ppb=498931", self.check(boundary)) + for host_ppm in (10_607.0, -10_400.0): + with ( + self.subTest(host_ppm=host_ppm), + self.assertRaisesRegex( + RuntimeError, + "cannot be composed with Linux's rounded TSC rate below its " + "500 ppm limit within OpenVMM's", + ), + ): + hooks(host_ppm) + + def test_the_scenario_hooks_its_boot_and_each_restore(self): + # An unhooked cold boot's sample gives the host's own correction, against + # whose sign the cold boot and the first restore publish 200 and 150 ppm, + # within Linux's limit even on a host at 400 ppm, and with whose sign the + # limit phase's hook extends it past that limit. + for host_ppm, hooks in ( + (0.0, (-200, -150, 900)), + (-100.0, (300, 250, -900)), + (400.0, (-600, -550, 900)), + (-400.0, (600, 550, -900)), + (10_000.0, (-10_000, -10_000, 900)), + (-10_000.0, (10_000, 10_000, -900)), + (10_500.0, (-10_000, -10_000, 900)), + ): + with self.subTest(host_ppm=host_ppm): + self.run_scenario(host_ppm, hooks) + + def phase_report(self, ppm: float, *, flags: str = "0x08") -> str: + """Return the report of a guest that holds the published correction of + ``ppm``, within Linux's limit, and whose clock and host UTC run at it.""" + correction = round(ppm * 65_536) + return self.report( + frequency_ppb=round(ppm * 1000) - 500, + correction_ppb=round(ppm * 1000), + samples=((1_000, correction), (1_000, correction)), + host_ppm=ppm, + flags=flags, + ) + + def run_scenario(self, host_ppm: float, hooks: tuple[int, int, int]) -> None: + commands: list[list[str]] = [] + scripts: list[str] = [] + limit_ppm = hooks[2] + limit_correction = round( + microvm_tests.time_host_correction_composed_ppm(host_ppm, limit_ppm) + * 65_536 + ) + held = 500_000 if limit_correction > 0 else -500_000 + + def published(command: Sequence[str]) -> float: + hook = command[command.index("--x-time-abi-test-hook") + 1] + return microvm_tests.time_host_correction_composed_ppm( + host_ppm, int(hook.removeprefix("host-rate-offset-ppm=")) + ) + + def boot( + command: Sequence[str], script: str, *_args: object, **_kwargs: object + ) -> dict[str, str]: + commands.append(list(command)) + scripts.append(script) + if "--x-time-abi-test-hook" not in command: + return {"output": self.phase_report(host_ppm, flags="0x00")} + return {"output": self.phase_report(published(command))} + + def restore(command: Sequence[str], **_kwargs: object) -> bytes: + commands.append(list(command)) + if f"host-rate-offset-ppm={limit_ppm}" in command: + return self.report( + frequency_ppb=held, + correction_ppb=round(limit_correction * 1000 / 65_536), + limited=1, + samples=((0, limit_correction),), + ).encode() + return self.phase_report(published(command)).encode() + + with ( + tempfile.TemporaryDirectory() as temporary, + patch.object(microvm_tests, "run_guest_script", side_effect=boot), + patch.object(microvm_tests, "capture_snapshot") as capture, + patch.object(microvm_tests, "_measure_restore", side_effect=restore), + patch.object(microvm_tests, "_check_restore_warp") as warp, + patch.object(microvm_tests, "_check_restore_status") as status, + redirect_stdout(io.StringIO()) as stdout, + ): + microvm_tests.run_time_host_correction( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + memory_mib=256, + timeout=60, + output_dir=Path(temporary), + ) + # The probe boots without the hook and takes its samples at once. + self.assertEqual(len(commands), 4) + self.assertNotIn("--x-time-abi-test-hook", commands[0]) + self.assertIn("sleep 0\n", scripts[0]) + self.assertIn("sleep 5\n", scripts[1]) + self.assertEqual( + [ + command[command.index("--x-time-abi-test-hook") + 1] + for command in commands[1:] + ], + [f"host-rate-offset-ppm={hook}" for hook in hooks], + ) + # The capture runs without the hook, and its guest reports again after + # every restore, then runs the restore checks. + captured = capture.call_args.args[0] + self.assertNotIn("--x-time-abi-test-hook", captured) + self.assertTrue( + capture.call_args.kwargs["post_restore_script"].startswith( + microvm_tests.time_host_correction_script() + ) + ) + self.assertIn("--restore-snapshot", commands[2]) + self.assertEqual((warp.call_count, status.call_count), (2, 2)) + self.assertIn( + f"the host's own correction is {host_ppm:+.3f} ppm", stdout.getvalue() + ) + self.assertIn(f"frequency_ppb={held} frequency_limited=1", stdout.getvalue()) + + class DurationReportTests(unittest.TestCase): # 2027-01-15T08:00:00Z on the wall clock, which the tests' monotonic # clock reads as 100. From 468505ed8cbd8ad3344aea6470a31304130967f3 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 17:29:31 -0700 Subject: [PATCH 5/5] doc: map time ABI 2 in the OpenVMM roadmap Record the six commits that the first commit of this series pins as positions 373-378 of the microVM chain, and move the NVX pin to 858b2f93a. Positions 373-378 move NVX's time ABI to version 2: the shared host TSC bracket, the host OS's discipline of host UTC, the publication of host UTC paired with the guest TSC and its correction, the guest TSC's offset on every backend, the version-2 time sample and restore packet 5, and the Guide's description. They change the time-ABI backends, portb device, restore packet, and VM worker that positions 159-269 added, and replayed onto upstream main they conflict in every file that they change except the two that they add. Position 377 also adapts petri's microVM configuration from PR15 and the ttrpc microVM test from PR25. The six commits are microsoft/openvmm#4809's, which microsoft/openvmm#4830 rebases unchanged onto 3d7512af6, position 372 and the current head of feature/microvm/dev since microsoft/openvmm#4816 merged positions 371 and 372. Since they follow that port rather than the unassigned positions 346-370, they start a new unassigned range. Update the scope, physical DAG, after-tip count and table, and remarks. The pin is 221 commits past the mapped tip. Part of #1009 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 47 ++++++++++++++++++++------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index ee3a11ea2..13e776ac7 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -9,15 +9,17 @@ - Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR summary, and commit mapping below cover positions 001-157 and the fixups at positions 334-340, 344, and 345. -- NVX's pinned OpenVMM revision: `3d7512af653717aa9123595e993a56cb8587106a`, position 372. - Positions 371 and 372, which microsoft/openvmm#4816 proposes for `feature/microvm/dev`, are two - commits on position 370, `b58040ee4`, the head of `feature/microvm/dev` since - microsoft/openvmm#4823 merged it. Position 370 is one commit on position 369, `5e0ab772e`, which - microsoft/openvmm#4820 merged with position 368. Positions 368 and 369 are two commits on - position 367, `bcfc47cff`, which microsoft/openvmm#4818 merged with position 366. Positions 366 - and 367 are two commits on position 365, `e4c741e18`, which microsoft/openvmm#4804 merged with - positions 362-364. Positions 362-365 are four commits on position 361, `c32c57d12`, which - microsoft/openvmm#4807 merged with position 360. +- NVX's pinned OpenVMM revision: `858b2f93aa78d35df6fdd217322a7f4cfb57b080`, position 378. + Positions 373-378, which microsoft/openvmm#4830 proposes for `feature/microvm/dev` as the + commits of microsoft/openvmm#4809 rebased, are six commits on position 372, `3d7512af6`, the + head of `feature/microvm/dev` since microsoft/openvmm#4816 merged positions 371 and 372. Those + are two commits on position 370, `b58040ee4`, which microsoft/openvmm#4823 merged. Position 370 + is one commit on position 369, `5e0ab772e`, which microsoft/openvmm#4820 merged with position + 368. Positions 368 and 369 are two commits on position 367, `bcfc47cff`, which + microsoft/openvmm#4818 merged with position 366. Positions 366 and 367 are two commits on + position 365, `e4c741e18`, which microsoft/openvmm#4804 merged with positions 362-364. + Positions 362-365 are four commits on position 361, `c32c57d12`, which microsoft/openvmm#4807 + merged with position 360. Positions 360 and 361 are two commits on position 359, `bd2b27a2e`, which microsoft/openvmm#4794 merged with positions 355-358. Positions 355-359 are five commits on position 354, `832d5bb48`, which microsoft/openvmm#4775 merged with position 353. Positions 353 and 354 are two commits on @@ -31,9 +33,9 @@ microsoft/openvmm#4735 merged with position 339. Positions 339 and 340 are two commits on position 338, `8a0d945ed`, which microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head of the `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin - is 215 commits past the mapped tip on the same first-parent chain. Positions 158-333, 341-343, - and 346-370 are not yet assigned to PRs, and positions 371 and 372 port microsoft/openvmm#4803; - see [Commits after the mapped tip](#commits-after-the-mapped-tip). + is 221 commits past the mapped tip on the same first-parent chain. Positions 158-333, 341-343, + 346-370, and 373-378 are not yet assigned to PRs, and positions 371 and 372 port + microsoft/openvmm#4803; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base. ## Upstream status @@ -325,7 +327,8 @@ The physical branch is one first-parent chain: -> unassigned [341-343] -> PR18 fixups [344-345] -> unassigned [346-370] -> microsoft/openvmm#4803 port [371-372] - -> 3d7512af6 (NVX pin) + -> unassigned [373-378] + -> 858b2f93a (NVX pin) ``` For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main, @@ -347,11 +350,11 @@ PR18's position 111. ## Commits after the mapped tip -The NVX pin extends the chain past position 157 without merges. Of the 215 later commits, -positions 158-333, 341-343, and 346-370 are not yet classified or assigned to PRs, positions -334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02 -fixup, positions 339 and 340 are PR20 and PR23 fixups, positions 344 and 345 are PR18 fixups, -and positions 371 and 372 port microsoft/openvmm#4803. +The NVX pin extends the chain past position 157 without merges. Of the 221 later commits, +positions 158-333, 341-343, 346-370, and 373-378 are not yet classified or assigned to PRs, +positions 334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a +PR02 fixup, positions 339 and 340 are PR20 and PR23 fixups, positions 344 and 345 are PR18 +fixups, and positions 371 and 372 port microsoft/openvmm#4803. This table groups them by topic in chain order; ranges are inclusive, as in the [PR summary](#pr-summary). @@ -407,6 +410,7 @@ This table groups them by topic in chain order; ranges are inclusive, as in the | 368-369 | `342887c99..5e0ab772e` | 2 | The VM time keeper maps a timeout beyond what the host clock can represent to the host clock's latest time: one more than about 584 years after the last start no longer panics the task that arms it, as one armed across a restore to a far VM time panicked the keeper's task at start (microsoft/nvx#1038), and one nearly half the ring after now, which VM time's circular order puts before the start, no longer arms the host timer in the past, which made the keeper spin (microsoft/openvmm#4820) | | 370 | `b58040ee4` | 1 | Explicit `--host-loopback allow` requires an IPv4 network of /29 or larger: OpenVMM refuses it on a /30, which leaves no address at which Consomme presents host peers, at a cold boot and at a restore that installs a new policy on a NIC template, instead of presenting IPv4 host peers at the gateway (microsoft/openvmm#4823) | | 371-372 | `2dec9edcb..3d7512af6` | 2 | The KVP IC VMM test boots Windows Server 2025 instead of 2022, whose COM activation race can lose WMI's server-registration notification, so that the guest answers the test's IP information request with `0x80080005` after 120 seconds (microsoft/nvx#1035); and the check-in gates' x64 Linux KVM job downloads the Server 2025 image and keeps running the test, while the x64 Linux MSHV job excludes it because Server 2025's required initial reset fails there (microsoft/openvmm#4816, which ports microsoft/openvmm#4803) | +| 373-378 | `a34095ab3..858b2f93a` | 6 | NVX time ABI 2: the VMM pairs each time sample's host UTC with the guest TSC, through the guest TSC's offset from the host TSC that every backend measures at every start, and publishes with it the host's clock correction that the host OS reports (`adjtimex`, `GetSystemTimeAdjustmentPrecise`), the bound of its error, the status and maximum error of host UTC, and a disruption counter, in a 48-byte version-2 sample; restore packet 5 drops host UTC, and the `host-rate-offset-ppm` test hook skews the published host clock (microsoft/openvmm#4830, which rebases microsoft/openvmm#4809) | Some of these commits change mapped work rather than add to it. Positions 246-250 remove the legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes @@ -440,6 +444,13 @@ image from the x64 Linux MSHV job, where Server 2025's required initial reset fa onto an upstream main that includes microsoft/openvmm#4803 drops position 371, as the rebases dropped the commits under [Landed upstream](#landed-upstream), and reconciles position 372 with whatever upstream changes for those jobs. +Positions 373-378 move the time ABI of positions 159-269 to version 2: they change the +time-ABI backends of KVM, MSHV, and WHP, the portb device's time sample, the restore packet, and +the microVM worker's time-abi unit, all of which those positions added, so they follow them +upstream. Replayed onto upstream main, they conflict in every file that they change except the +two that they add. Position 377 also adapts petri's microVM configuration, which PR15's position +088 adds, and the ttrpc microVM test, which PR25's position 146 adds, to the portb device's +configuration and restore packet 5. ## PR summary