time ABI: publish the host's clock correction with each time sample

.github/actions/validate-runner/action.yml

@@ -236,13 +236,13 @@ runs:
 
     # Host time. The time ABI host qualification (doc/ci.md,
     # "Host qualification") checks the backend, the CPU fingerprint and
-    # generation, and the TSC rate stability on its short CI schedule, and,
-    # in jobs that downloaded OpenVMM, its CPU profile check (H2) and
-    # preflight (H3); other jobs qualify without OpenVMM. Jobs whose guests
-    # run without the microVM scenarios' warp probes add the guest warp probe
-    # (H6). The host's invariant-TSC flags are evidence only: the guest warp
-    # probe measures the cross-vCPU skew that a host without one can cause
-    # (#265).
+    # generation, and the TSC rate stability and the host's synchronized clock
+    # correction on its short CI schedule, and, in jobs that downloaded
+    # OpenVMM, its CPU profile check (H2) and preflight (H3); other jobs
+    # qualify without OpenVMM. Jobs whose guests run without the microVM
+    # scenarios' warp probes add the guest warp probe (H6). The host's
+    # invariant-TSC flags are evidence only: the guest warp probe measures the
+    # cross-vCPU skew that a host without one can cause (#265).
     - name: Report host TSC
       if: runner.os != 'Windows'
       shell: bash

SOURCE-MANIFEST.json

@@ -6,6 +6,8 @@
   },
   "openvmm": {
     "microvm_abi_version": 2,
+    "time_abi_version": 2,
+    "restore_packet_version": 5,
     "control_session_protocol_version": 1,
     "control_contract_revision": "nvx-microvm-v2-control-v2"
   },

doc/build.md

@@ -213,9 +213,10 @@ hypervisor identity, and runs the cross-vCPU warp test that the
 [time ABI](design/3.2-clock-and-time.md#327-cross-processor-consistency) uses to enforce its 1 µs
 skew bound. `nvx-time` implements the time ABI's
 [guest obligations](design/3.2-clock-and-time.md#3210-guest-obligations): the conformance
-checks, the violation watcher, the restore packet v4 and time-sample parsers,
-the snapshot agent's time steps, and the wall-clock discipline. Before
-shell-ready it only steps the clock to host UTC; the other boot checks run in
+checks, the violation watcher, the restore packet v5 and time-sample v2
+parsers, the snapshot agent's time steps, and the wall-clock discipline, which
+applies the host's clock correction. Before shell-ready it only steps the
+clock to host UTC and sets its rate; the other boot checks run in
 the background, and the production console shows no time ABI line except a
 violation event. `nvx-time status` prints every recorded check (boot, capture,
 and restore) and the runtime state on demand, after waiting for pending

doc/design/1.2-non-goals.md

@@ -11,7 +11,12 @@ The architecture deliberately excludes:
 - General CPU or memory hotplug after readiness.
 - Restore across hypervisor backends or CPU generations, live migration, and
   capture-and-continue.
-- Scaling of the TSC, trapping of RDTSC, paravirtual clocks, and a virtual PMU.
+- Scaling of the TSC, trapping of RDTSC, paravirtual clocks, such as kvmclock,
+  Hyper-V's reference TSC page, or a VMClock device, and a virtual PMU. The
+  TSC is the guest's only clocksource; the
+  [time samples](3.2-clock-and-time.md#329-time-samples) that pair host UTC
+  with it and publish the host's clock correction feed only the guest's
+  wall-clock discipline.
 - Serialization of live network flows, native host handles, or the contents of
   a live host filesystem export.
 - Multi-container orchestration, including pod infrastructure, pause

doc/design/3.1-platform-control-device.md

@@ -34,8 +34,9 @@ machine omits, and carries:
   which gives a restored guest what it needs to repair its clocks,
   processors, and memory, and the fresh random bytes from which it
   [refreshes its entropy](5.5-restore-protocol.md#558-entropy-refresh).
-- Host time samples, which the guest pairs with its own clock to set and
-  discipline its wall clock.
+- Host time samples, which pair host UTC with the guest TSC and carry the
+  host's clock correction, from which the guest sets and disciplines its wall
+  clock.
 - The generation ID, a random identifier unique to each VM process. Every
   restored process receives a new one, even when several clones run from one
   snapshot, so the guest can detect that it has been restored and
@@ -55,9 +56,9 @@ queues in guest memory, or interrupt line. Each output byte is one port write,
 and the guest polls a status port for input. Every access to the channel's
 ports is a synchronous exit that the VMM completes before the guest continues,
 so a host time that the VMM records during an access lies between two guest
-clock reads taken around it. This bounds the error of
-[time samples](3.2-clock-and-time.md#329-time-samples) and of the restore
-packet's UTC field.
+TSC reads taken around it. The guest checks each
+[time sample](3.2-clock-and-time.md#329-time-samples)'s pairing of host UTC
+with its TSC against them.
 
 As Table 3.2 shows, the byte channel's three ports form one device: a data
 port, a status and record-selector port, and a time-sample window.

doc/design/3.2-clock-and-time.md

@@ -71,7 +71,7 @@ through which CPUID reports its results (EAX, EBX, ECX, and EDX).
 | --- | --- | --- | --- | --- |
 | `0x40000000` | `0x40000005` (maximum leaf) | `0x7263694d` (`"Micr"`) | `0x666f736f` (`"osof"`) | `0x76482074` (`"t Hv"`) |
 | `0x40000001` | `0x31237648` (`"Hv#1"`) | 0 | 0 | 0 |
-| `0x40000002` | `0x0058564e` (`"NVX"`) | `0x00010000` (time ABI 1.0) | 0 | 0 |
+| `0x40000002` | `0x0058564e` (`"NVX"`) | `0x00020000` (time ABI 2.0) | 0 | 0 |
 | `0x40000003` | `0x00008860` | 0 | 0 | `0x00000100` |
 | `0x40000004` | 0 | `0xffffffff` | 0 | 0 |
 | `0x40000005` | `C` | `C` | 0 | 0 |
@@ -185,8 +185,9 @@ The boundary is accepted. The manifest records the tolerance, and any other
 recorded value is rejected. Within the tolerance, the TSC is neither scaled
 nor trapped, the frequency MSR keeps returning `F_s`, and guest clocks run
 fast or slow by at most the deviation. The restore packet reports that
-deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, so
-that the guest can compensate:
+deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, and
+the host clock correction of every [time sample](#329-time-samples) includes
+it, so that the guest compensates it from its repair on:
 
 ```text
 rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s)
@@ -254,6 +255,9 @@ must:
   hypervisor-specific emulator or its saved state.
 - Pair the first processor's TSC with a host time sample within 100 µs for
   the capture anchor, keeping the tightest of at most 64 samples.
+- Know the guest TSC's offset from the host TSC before any processor runs, at
+  cold boot and at every restore, so that the VMM pairs each
+  [time sample](#329-time-samples)'s host UTC with the guest TSC.
 - Set every instantiated processor's TSC at one host instant and verify it by
   read-back before any processor runs.
 - Keep live cross-processor skew within 1 µs after release.
@@ -316,40 +320,111 @@ quiesce as elapsed time, as it would a long preemption.
 
 ## 3.2.9 Time Samples
 
-Figure 3.1 shows the layout of the 16-byte sample that writing `0xa7` to the
-byte-channel selector latches into the time window.
+Figure 3.1 shows the layout of the 48-byte sample that writing `0xa7` to the
+byte-channel selector latches into the time window. The VMM takes the sample
+while it handles the selector write: it reads host UTC, pairs it with the
+guest TSC, and reads how the host's time service disciplines host UTC.
 
 ```text
 offset size field
-     0    1 version = 1
+     0    1 version = 2
      1    1 flags                bit 3 mirrors the restore packet's override flag; other bits zero
-     2    2 reserved = 0
+     2    1 status               host UTC: 0 unknown, 1 synchronized, 2 free-running, 3 unreliable
+     3    1 reserved = 0
      4    4 generation           u32, little-endian, g of this VM process
      8    8 utc_ns               u64, little-endian, host UTC at the latch
+    16    8 tsc                  u64, little-endian, guest TSC at the instant of utc_ns
+    24    4 pairing_ns           u32, little-endian, bound on the distance of those instants
+    28    4 disruption           u32, little-endian, changes with each host clock disruption
+    32    8 correction           i64, little-endian, ppm scaled by 2^16
+    40    4 correction_bound     u32, little-endian, ppm scaled by 2^16, 0xffffffff if unknown
+    44    4 max_error_us         u32, little-endian, 0xffffffff if unknown
 ```
 
 **Figure 3.1.** Time-sample layout.
 
-The VMM cannot observe a running processor's TSC on every backend, so the
-guest pairs host UTC with its own clock. It reads its realtime clock as `t0`
-immediately before the selector write and as `t1` immediately after it. The
-host instant lies in `[t0, t1]`, so:
+The guest TSC is never scaled, so it is the host TSC plus an offset that is
+fixed for the life of a VM process. The backend reads that offset, or
+measures it by bracketing a read of the first processor's TSC with host TSC
+reads, before any processor runs, at cold boot and at every restore. Each
+sample brackets a read of host UTC with host TSC reads, keeping the tightest
+of up to 16 brackets, and converts the host TSC to the guest TSC with the
+offset. `pairing_ns` adds the offset's uncertainty to half the bracket.
+
+`correction` is how fast host UTC runs against the guest TSC counted at the
+declared rate `F`, minus one, in the kernel's frequency-adjustment unit:
+
+```text
+correction = (1 + c_os) * F / F_d - 1
+```
+
+where `c_os` is the time service's correction of the host clock against the
+TSC at the native rate `F_d`. On Linux, it is the frequency and tick that
+`adjtimex` reports. On Windows, `GetSystemTimeAdjustmentPrecise` reports the
+performance counter's frequency as its increment, and the counter's ticks in
+each second of host UTC as its adjustment, so `c_os` is the increment over the
+adjustment, minus one, while a time service adjusts the clock, and zero
+otherwise. After a restore, `F` is `F_s`, so the correction includes the
+restore's rate deviation.
+`correction_bound` bounds the correction's error: 1 ppb of rounding when the
+host clock and `F_d` come from one calibration, as with Linux's `tsc`
+clocksource and KVM, or Hyper-V's reference TSC and MSHV or WHP; the kHz to
+which one of them is known otherwise; and unknown when the host clock may not
+follow the TSC.
+
+`status` is the host's own account of its clock. On Linux, `adjtimex`
+reports a clock fault as unreliable and an unsynchronized clock as
+free-running; on Windows, a clock that no time service adjusts is
+free-running. `max_error_us` is the maximum error that Linux's time service
+sets, and unknown on Windows. `disruption` starts at zero in each VM process
+and changes when host UTC steps by at least 1 ms more than the correction
+explains, when the host resumes from a suspension, and when the guest TSC's
+offset from the host TSC changes.
+
+These values feed only the guest's wall-clock discipline. The TSC stays the
+guest's only clocksource: the time sample is not a paravirtual clock, which
+[1.2 Non-Goals](1.2-non-goals.md) excludes, as it excludes a VMClock device.
+
+The guest checks each sample's pairing against its own TSC. It reads its TSC
+as `g0` immediately before the selector write and as `g1` immediately after
+it, and then its realtime clock as `r`, in nanoseconds, between two more TSC
+reads, whose midpoint is `c` and half distance `w`. The host reads its UTC
+while it handles the write, so a sample whose `tsc` lies outside
+`[g0 - p, g1 + p]`, where `p` is `pairing_ns` plus the 1 µs skew bound, in
+cycles at `F`, is invalid. The kernel counts the TSC at `F_k`, in Hz, `F`
+rounded down to the kHz, with its frequency correction `f`, so the guest's
+realtime clock at `tsc` is `r` less `d`, the nanoseconds between `tsc` and
+`c`:
 
 ```text
-theta = utc_ns - (t0 + t1) / 2      uncertainty epsilon = (t1 - t0) / 2
+d       = (c - tsc) * (1 + f) * 1_000_000_000 / F_k
+skew_ns = 1_000
+theta   = utc_ns - (r - d)
+epsilon = pairing_ns + skew_ns + w * 1_000_000_000 / F_k + |d| * 2000 ppm
 ```
 
-The same bracket applies to the UTC sample in the restore packet.
-As Table 3.10 shows, the guest accepts a pairing only if `epsilon` is within
-the bound for its use.
+The skew term bounds a read on another guest CPU than VP 0, whose TSC OpenVMM pairs with host
+UTC. The last term bounds how far a slew below the step threshold can move the realtime clock's
+rate over that time. The restore packet carries no host UTC:
+repair takes a time sample instead. As Table 3.10 shows, the guest accepts a
+pairing only if `epsilon` is within the bound for its use.
 
 **Table 3.10.** Time-pairing uncertainty bounds.
 
-| Use | Bound | Attempts | When every attempt exceeds the bound |
+| Use | Bound | Attempts | When every attempt is invalid or exceeds the bound |
 | --- | --- | --- | --- |
-| Restore repair | 1 ms | 3 | Repair fails, and the guest powers off |
-| Initial synchronization at boot | 1 ms | 3 | Conformance fails, and the guest powers off |
-| Wall-clock discipline | 50 µs | 3 | The poll is skipped; never fatal |
+| Restore repair | 100 µs | 3 | Repair fails, and the guest powers off |
+| Initial synchronization at boot | 100 µs | 3 | Conformance fails, and the guest powers off |
+| Wall-clock discipline | 100 µs | 3 | The poll is skipped; never fatal |
+
+The guest's frequency correction runs its wall clock at host UTC's rate:
+
+```text
+f = (1 + correction) * F_k / F - 1
+```
+
+The kernel limits `f` to ±500 ppm; the guest holds a correction beyond that
+limit at the limit, and reports that it did.
 
 ## 3.2.10 Guest Obligations
 
@@ -359,22 +434,26 @@ Production paths keep the console quiet; status goes to a state file that the
 workload can read through a read-only mount. The state file reports the
 outcome of each phase's checks, the declared rates, the generation, the last
 downtime and its source, and the wall clock's discontinuities,
-synchronization, offset, uncertainty, and frequency correction.
+synchronization, offset, uncertainty, and frequency correction, with whether
+the kernel's limit held it, and the status, correction, bounds, and
+disruption counter of host UTC that the latest accepted sample published, so
+that a workload can see when its host's clock is not synchronized or wanders.
 
 **Conformance.** Before readiness, the guest only takes a time sample within
-the bound and steps its clock to host UTC, so the workload starts on host
-time. It runs every other boot check asynchronously after readiness, starting
-at idle priority, so the checks never delay readiness or take a processor from
-the workload. The checks cover the identity leaves and explicit zero leaves,
-the CPU time bits on every CPU, the identity MSRs, the kernel's platform,
-clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence
-of forbidden clock parameters among those that the kernel reads from its
-command line, stall-detector settings, a running violation watcher, and kernel
-W+X integrity. Capture reruns the clocksource, tick-device, and watcher checks.
-Restore reruns them too, checks the declared rate in the frequency MSR again,
-and checks the identity leaves, the CPU time bits, and the CPU flags of every
-processor that it brings online. Every check is fail-fast: a failure powers
-the guest off even if the workload is running.
+the bound, steps its clock to host UTC, and holds the frequency correction
+that the sample's host clock correction gives, so the workload starts on host
+time and runs at its rate. It runs every other boot check asynchronously after
+readiness, starting at idle priority, so the checks never delay readiness or
+take a processor from the workload. The checks cover the identity leaves and
+explicit zero leaves, the CPU time bits on every CPU, the identity MSRs, the
+kernel's platform, clocksource, and LAPIC-timer selection, CPU flags, tick
+devices, the absence of forbidden clock parameters among those that the
+kernel reads from its command line, stall-detector settings, a running
+violation watcher, and kernel W+X integrity. Capture reruns the clocksource,
+tick-device, and watcher checks. Restore reruns them too, checks the declared
+rate in the frequency MSR again, and checks the identity leaves, the CPU time
+bits, and the CPU flags of every processor that it brings online. Every check
+is fail-fast: a failure powers the guest off even if the workload is running.
 
 **Violation watcher.** The watcher follows the kernel log from before the boot
 checks, so no record is missed. TSC instability; clocksource instability,
@@ -395,19 +474,25 @@ the guest releases the suppression anyway so that the kernel reports the real
 stall.
 
 **Wall-clock discipline.** The guest keeps its realtime clock on host UTC
-through the kernel's phase-locked loop. It polls every 16 seconds for the
-first four polls after boot or a restore, then every 64 seconds, and takes up
-to three samples per poll. An offset of at least 128 ms is stepped and counted
-as a discontinuity; a smaller one is slewed with error bounds derived from the
-offset and its uncertainty, which marks the clock synchronized for
-applications. The kernel limits the frequency correction to ±500 ppm, of which
-the rate tolerance uses at most 250 ppm. The other 250 ppm is the budget for
-the host's own clock correction, which the discipline applies too, because
-host UTC follows the host's time service: a host whose TSC runs more than
-250 ppm from its declared rate against host UTC does not
-[qualify](../ci.md#host-qualification) for the contract, and a larger
-correction would leave an offset that grows until the discipline steps it. A
-sample whose generation differs from the recorded one is discarded, because a
+through the kernel's phase-locked loop, with its frequency held: each accepted
+sample sets the kernel's frequency correction from the sample's host clock
+correction, which carries a restore's rate deviation, and the loop only slews
+the phase, so the guest never estimates the host's rate. It polls every 16
+seconds for the first four polls after boot, a restore, or a change of the
+sample's disruption counter, then every 64 seconds, and takes up to three
+samples per poll. An offset of at least 128 ms is stepped and counted as a
+discontinuity; a smaller one is slewed. Either sets error bounds derived from
+the offset and its uncertainty, to which the maximum error adds host UTC's
+while the host's time service synchronizes it, which marks the clock
+synchronized for applications. The kernel limits the frequency correction to
+±500 ppm, of which the rate tolerance uses at most 250 ppm. The other 250 ppm
+is the budget for the host's own clock correction, which the discipline
+applies too, because host UTC follows the host's time service: a host whose TSC
+runs more than 250 ppm from its declared rate against host UTC does not
+[qualify](../ci.md#host-qualification) for the contract. A correction beyond
+the kernel's limit is held at the limit and reported in the state file, so an
+offset can grow until the discipline steps it. A sample whose generation
+differs from the recorded one is discarded, because a
 restore happened and restore repair resets the discipline. The discipline
 follows host wall-clock steps and never powers off the guest. Restore repair
 and the discipline step the clock through the kernel, so a workload can