doctor: gate H4 on the guest's 250 ppm budget for the host's clock correction

doc/ci.md

@@ -325,7 +325,8 @@ and generation, and the TSC rate stability on its short schedule. The microVM
 and platform jobs run it after downloading OpenVMM and add OpenVMM's CPU
 profile check (H2) and preflight (H3), and keep the fingerprint as an artifact
 when the profile check fails; the other jobs pass `--no-openvmm`. This takes a
-few seconds. It reports the CPU generation, the built-in CPU profile that
+few seconds, and up to a minute more when H4 re-reads chronyd's frequency
+estimate. It reports the CPU generation, the built-in CPU profile that
 `auto` selects, and the measured TSC rate in the log and the job summary, and
 fails the job with a stable code when the runner is not qualified, for example
 `E_PROFILE_HOST_UNKNOWN` on an unknown CPU generation, where `auto` would fall
@@ -1696,17 +1697,19 @@ profile, rates, and skew metrics to a file such as `$GITHUB_STEP_SUMMARY`. H4
 and H6 run on long qualification schedules, which take about three
 minutes; `--ci-schedule` selects CI's short ones. A failure that matches a time
 ABI failure code starts its detail with the code in brackets, for example
-`[E_PROFILE_HOST_UNKNOWN]`. A check fails, and never passes, when it can't
-read a fact it gates on: missing or unrecognized tool output fails it, and an
-error in one check fails only that check. Only evidence fields, such as the
-microcode and the invariant-TSC flags, may read `unknown`.
+`[E_PROFILE_HOST_UNKNOWN]`. A check that passes with a warning starts its
+detail with `warning:`, and the summary and, in GitHub Actions, a
+`::warning::` annotation repeat the warning. A check fails, and never passes,
+when it can't read a fact it gates on: missing or unrecognized tool output
+fails it, and an error in one check fails only that check. Only evidence
+fields, such as the microcode and the invariant-TSC flags, may read `unknown`.
 
 | Check | Implementation |
 | --- | --- |
 | H1 | `/dev/kvm` or `/dev/mshv` is readable and writable, and a KVM host has no `/dev/mshv`; on Windows, `WHvGetCapability` reports a hypervisor |
 | H2 | Vendor, family, model, stepping, microcode, and OS build from `/proc/cpuinfo` or the Windows registry. Then `openvmm --hypervisor <backend> --cpu-fingerprint <path>` writes the host's CPU fingerprint (by default `nvx-cpu-fingerprint-<backend>.json` in the probe directory; `--cpu-fingerprint` overrides it), checks it against the pinned profile that `auto` selects from OpenVMM's catalog, the latest revision of the generation's profile, which every backend shares, and prints one `NVX-CPU-PROFILE:` line. The check never falls back to a host profile, so a CPU on which `auto` falls back fails H2. H2 reports the generation and the profile from that line, so a profile that an OpenVMM pin adds qualifies its hosts without an NVX change. H2 requires exit status 0, `status=pass`, the same backend, and a catalog profile of the reported generation, reports the profile and surface digests, and otherwise fails with OpenVMM's code, for example `[E_PROFILE_HOST_UNKNOWN]` on a CPU that no profile serves or `[E_PROFILE_UNSUPPORTED]` naming every unsupported CPUID bit. On MSHV and WHP, OpenVMM checks the entries outside the profile on a probe partition configured from the profile, as a cold boot does. With `--no-openvmm`, H2 maps the host with NVX's copy of the catalog instead: `skylake-sp` (6/85, steppings 0 to 4, `intel.skylake-sp.v1`), `icelake-sp` (6/106, `intel.icelake-sp.v1`), `emeraldrapids` (6/207, `intel.emeraldrapids.v1`), `alderlake` (6/151 and 6/154, `intel.alderlake.v1`), `milan` (AMD 25/1, `amd.milan.v1`), `genoa` (AMD 25/17, `amd.genoa.v2`), or `turin` (AMD 26/2, `amd.turin.v1`). Any other CPU, including Cascade Lake and Cooper Lake, fails with `E_PROFILE_HOST_UNKNOWN`. A unit test keeps the copy, which records the latest revision of each generation's profile, equal to the profiles of the OpenVMM submodule's pinned revision, which it reads from the gitlink's commit, so the NVX CLI jobs validate the CLI after they check out OpenVMM. The host OS's invariant-TSC flags (`constant_tsc nonstop_tsc`, or the CPUID bit on Windows) are recorded as evidence and never fail the check, because they don't decide what a guest observes: Azure WHP hosts show the CPUID bit but cannot offer invariant TSC to partitions, and their guests measure tens of nanoseconds of skew |
 | H3 | `openvmm --x-time-abi-verify` builds the partition and runs the time ABI preflight without running the guest. Its `NVX-TIME-ABI-VERIFY:` line must report `status=ok` for the backend, plausible declared and native TSC rates, the backend's LAPIC rate, and a `cpu_profile` that is a catalog profile, `<vendor>.<generation>.v<revision>` of any generation but a host profile's `host`, which rejects the host profile that `auto` falls back to on a CPU that no built-in profile serves, and, when H2 runs too, a revision of the profile H2 names. A failed preflight reports OpenVMM's code, for example `[E_TSC_SYNC_UNSUPPORTED]` |
-| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window must lie within 100 ppm of the rate H3 reports when H3 runs in the same invocation, and H4 reports their deviation to 0.001 ppm. On Linux with a synchronized chronyd, that rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the frequency that chronyd estimates from its whole sample history, as `chronyc -c tracking` reports it, and every rate within chronyd's skew of that estimate must lie within the limit, so a range that straddles the limit fails as inconclusive; the rate against `CLOCK_MONOTONIC` is evidence only, because it also carries the slew with which chronyd removes its current offset, which reaches hundreds of ppm for seconds after the host's clock jumps (#768). Otherwise the rate is measured against `CLOCK_MONOTONIC` on Linux, and the detail says why chronyd's estimate is missing, or against `QueryPerformanceCounter` on Windows. A Linux host's clocksource is recorded as evidence |
+| H4 | Samples of the TSC against the host's monotonic clocks, with sleeps between them so the host's CPUs idle: 13 samples 10 s apart, or 3 samples 1 s apart with `--ci-schedule`. Each sample reads the TSC between two reads of a clock, keeping the tightest of 64 brackets; its uncertainty is half the bracket plus half the clock's resolution. A clock that returns the same value to consecutive reads is coarser than one read, so the probe takes its smallest step as its resolution: Hyper-V's reference TSC page advances the Linux clocks in 100 ns steps although `clock_getres` reports 1 ns. The interval stability is judged against a clock that time synchronization never steers, `CLOCK_MONOTONIC_RAW` on Linux and `QueryPerformanceCounter` on Windows: every interval between consecutive samples must be conclusive within 0.25 ppm, and the interval rates must agree within 1 ppm. chrony's frequency updates move `CLOCK_MONOTONIC`'s rate by up to several ppm between seconds on the Azure runners, which says nothing about the TSC. The rate over the whole window must lie within 250 ppm of the rate H3 reports when H3 runs in the same invocation, and H4 reports their deviation to 0.001 ppm. The limit is the guest's budget for the host's clock correction: the guest's [wall-clock discipline](design/3.2-clock-and-time.md#3210-guest-obligations) follows host UTC, so it applies the host's correction on top of a restore's rate deviation, which the time ABI's restore tolerance bounds at 250 ppm, within Linux's frequency limit of ±500 ppm. Beyond the budget the guest doesn't fail: its offset grows to 128 ms, and the discipline steps its clock and counts a discontinuity. A unit test keeps the limit equal to the difference of nvx-rs's frequency limit and its restore deviation limit, and the latter equal to the restore tolerance of the OpenVMM gitlink's commit. The limit doesn't bound the TSC's error alone, as the time ABI v1's 100 ppm did, because no run can tell that error from the wander of the host's reference clock: on `azure-mshv-1`, chronyd's estimate ranged from +2.5 to +189.3 ppm within a day while the TSC held its rate (#1008). Above that old bound, H4 passes with a warning that says how much of the budget the host's correction uses, and that one run can't tell the TSC's error from the reference's wander. On Linux with a synchronized chronyd, that rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the frequency that chronyd estimates from its whole sample history, as `chronyc -c tracking` reports it, and every rate within chronyd's skew of that estimate must lie within the limit. The reference's wander can move the estimate and its skew within seconds, so when that range straddles the limit, H4 re-reads the estimate every 2 s for up to 60 s and uses the first whose range lies wholly within or wholly beyond the limit, with the same samples, since only the estimate changes; a range that straddles the limit throughout fails as inconclusive. The rate against `CLOCK_MONOTONIC` is evidence only, because it also carries the slew with which chronyd removes its current offset, which reaches hundreds of ppm for seconds after the host's clock jumps (#768). Without chronyd's estimate, on a Linux host whose kernel PLL another time service drives, such as systemd-timesyncd or ntpd, and which `adjtimex` reports synchronized, the rate is the rate against `CLOCK_MONOTONIC_RAW` corrected by the PLL's frequency, `adjtimex`'s frequency with its tick's, which the PLL keeps apart from the offsets that it slews; H4 never reads it with chronyd, which keeps the PLL off and slews through that frequency. Otherwise the rate is measured against `CLOCK_MONOTONIC` on Linux, and the detail says why neither estimate is available, or against `QueryPerformanceCounter` on Windows. On Linux, H4 also reports as evidence the average correction since boot, the rate of `CLOCK_MONOTONIC_RAW` against `CLOCK_MONOTONIC` over the uptime, and the deviation from the backend's rate that it gives: both clocks exclude suspend, and after a small offset at boot only the time service's gradual adjustments separate them, so the average smooths the reference's wander out without saved state, but slews just after boot or a live migration bias it, so H4 doesn't gate on it. A Linux host's clocksource is recorded as evidence |
 | H5 | Pinned-thread ping-pong rounds over every pair of host CPUs; `max_abs_offset_ns` is at most 1,000, the measurement is conclusive, and no pair stalls |
 | H6 | A microVM with the largest supported vCPU count up to 8 reports a valid `NVX-TIME-ABI` boot line through `nvx-time status` and runs `nvx-time-probe warp` over every CPU pair five times, with all vCPUs halted for 0.1, 1, 5, and 1 s between the runs, so that a host without an invariant TSC corrects the guest TSC as idle host CPUs wake (#265); then a 1-vCPU microVM runs it once. Every run stays within 1,000 ns. `--ci-schedule` runs CI's schedule instead: two runs 1 s apart |
 | H7 | `adjtimex` reports no `STA_UNSYNC` and no `TIME_ERROR` on Linux; `w32tm /query /status` names a synchronized source on Windows. When H7 can't read either, its detail starts with `cannot verify host UTC synchronization`: `adjtimex` fails or reports a clock state Linux doesn't define, or `w32tm` fails or prints no English `Source` and `Leap Indicator` lines, as in a Windows display language other than English |

doc/design/3.2-clock-and-time.md

@@ -401,13 +401,18 @@ to three samples per poll. An offset of at least 128 ms is stepped and counted
 as a discontinuity; a smaller one is slewed with error bounds derived from the
 offset and its uncertainty, which marks the clock synchronized for
 applications. The kernel limits the frequency correction to ±500 ppm, of which
-the rate tolerance uses at most 250 ppm. A sample whose generation differs
-from the recorded one is discarded, because a restore happened and restore
-repair resets the discipline. The discipline follows host wall-clock steps
-and never powers off the guest. Restore repair and the discipline step the
-clock through the kernel, so a workload can observe every step through the
-kernel's clock-set notification, such as a `timerfd` armed with
-`TFD_TIMER_CANCEL_ON_SET`.
+the rate tolerance uses at most 250 ppm. The other 250 ppm is the budget for
+the host's own clock correction, which the discipline applies too, because
+host UTC follows the host's time service: a host whose TSC runs more than
+250 ppm from its declared rate against host UTC does not
+[qualify](../ci.md#host-qualification) for the contract, and a larger
+correction would leave an offset that grows until the discipline steps it. A
+sample whose generation differs from the recorded one is discarded, because a
+restore happened and restore repair resets the discipline. The discipline
+follows host wall-clock steps and never powers off the guest. Restore repair
+and the discipline step the clock through the kernel, so a workload can
+observe every step through the kernel's clock-set notification, such as a
+`timerfd` armed with `TFD_TIMER_CANCEL_ON_SET`.
 
 The [snapshot protocol](5.3-snapshot-protocol.md) and
 [guest repair](5.5-restore-protocol.md#557-guest-repair) define the snapshot

doc/usage.md

@@ -1249,7 +1249,8 @@ python3 scripts/nvx.py doctor
 Qualifies this host for the [time ABI](design/3.2-clock-and-time.md).
 It runs the selected checks in order, prints one
 `NVX-DOCTOR: check=<id> status=<pass|fail> detail="..."` line per check, and
-exits with status 1 if any check fails. A check fails, and never passes, when
+exits with status 1 if any check fails. A check that passes with a warning
+starts its detail with `warning:`. A check fails, and never passes, when
 it can't read a fact it gates on. [CI](ci.md#host-qualification) describes each
 check and the subset that CI runs.