Skip to content

openvmm: map VM times beyond the host clock to its latest time - #1042

Open
Pedro Henrique Penna (ppenna) wants to merge 2 commits into
devfrom
agents/issue-1038-debug-test-fix-commit
Open

Pedro Henrique Penna (ppenna) wants to merge 2 commits into
devfrom
agents/issue-1038-debug-test-fix-commit

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1038. Pins microsoft/openvmm#4820 at 5e0ab772eeea14f4d0f5ed05474856569358f80e, which makes OpenVMM's VM time keeper map a timeout beyond what the host clock can represent to the host clock's latest time. Such a timeout panicked the task that armed it, as the issue's sequence of public calls panics the keeper's task at start(), and one nearly half the ring ahead of now made the keeper spin. This pull request also maps the pin in the OpenVMM roadmap.

Based on dev at 4434b1af0. microsoft/openvmm#4820 is two commits directly on bcfc47cff, the head of feature/microvm/dev and the revision that dev pins since #1039 merged, and the pin is that pull request's head commit itself, so this branch's two commits are its whole diff.

Change

Commit Files Change
70adf8918 openvmm openvmm bcfc47cff → 5e0ab772e: the two commits of microsoft/openvmm#4820
6d3442d14 doc doc/openvmm-upstream-roadmap.md Records them as positions 368 and 369, unassigned, with a remark that they change only vmcore's VM time keeper, whose vmtime.rs upstream main and the fork base have alike, and apply to upstream main as they are; moves the NVX pin; and records that microsoft/openvmm#4818 merged positions 366 and 367 into feature/microvm/dev

No other NVX change is needed. OpenVMM's control-session protocol vectors and built-in CPU profiles are unchanged at the pin.

OpenVMM changes

VmTime counts 100 ns units in 64 bits and orders them circularly: a time less than half the ring, about 29,000 years, after another follows it. TimerState::timestamp maps a VM time to the host time at which VM time reaches it, from its circular distance after the last start. set_next arms the keeper's host timer with it, and VmTimeAccess::host_time returns it.

  • Saturation (342887c99). pal_async's Instant counts nanoseconds in 64 bits, about 584 years, and its Add panics on overflow, so a timeout more than about 584 years after the start panicked the task that armed it. In the issue's sequence, a timeout armed while VM time is stopped, at 1.8e16, follows the restored time, 1.58e19, by 2.66e18 units, so start() panics the time source's task in TimerState::start. timestamp now adds the distance with Instant::saturating_add, so such a time maps to the host clock's latest time, which the host clock never reaches. Every timer backend already accepts that deadline: a timeout just short of overflowing reached it before, and the epoll, kqueue, IOCP, thread pool, io_uring, and local backends saturate their own conversions of it.
  • Times nearly half the ring ahead (5e0ab772e). Once VM time has run since the start, a time nearly half the ring after now is more than half the ring after the start, which the circular order puts before it, so timestamp mapped it to the start's host time, in the past. The waiters compare with now and keep waiting, so the host timer fired at once, wake re-armed the same deadline, and TimerState::poll looped forever with the timer state's write lock held. timestamp now maps a time that the circular order puts before the start, but after the current time, to the host clock's latest time too. It reads the current time only in that case.

A saved state can carry any VM time, as in the issue. A guest can choose both kinds of timeout where OpenVMM emulates the Hyper-V synthetic timers, as WHP does when it does not offload the enlightenments: a one-shot timer expires at the absolute reference time that the guest writes, up to half the ring after the current one.

The issue suggests saturating the distance at a day instead, after TimerState::poll's default deadline. That stops the panic but makes the keeper spin once VM time has run for a day since the start, since every clamped deadline is then in the past: a TimerState started two days ago with a timeout 1,000 years ahead never returns from poll with that clamp, and returns at once with the saturation.

Validation

  • OpenVMM, on Windows and on the Azure test VM nvx-kvm-test (Azure Linux 3), at each commit: cargo nextest run --profile agent -p vmcore (32 and 33 tests passed) and cargo doc --no-deps -p vmcore. On Windows, also cargo clippy --all-targets -p vmcore -- -D warnings and cargo xtask fmt --fix, which the test VM's toolchain lacks. All clean.
  • Reproduction at bcfc47cff, the previous pin, on both: the issue's sequence panics with "duration too large" in Instant::add, with the issue's backtrace (TimerState::timestamp ← set_next ← wake ← TimerState::start ← SecondaryKeeper::run); a timeout 1,000 years ahead of a running keeper panics in host_time; and a timeout nearly half the ring ahead makes the keeper spend 9.99 s of user CPU time in 10 s, on nvx-kvm-test, until it is killed. Each new test fails without its commit.
  • End to end on the Azure Linux 3 test VMs nvx-kvm-test (Linux/KVM) and nvx-mshv-test (Linux/MSHV), with OpenVMM built at the pin, whose provenance records 5e0ab772e and a clean tree, and the guest artifacts of release v0.1.0-dev.d34d8ab730e5: python3 scripts/nvx.py test-microvm with the default suite, 41 scenarios on the Alpine guest with 1, 2, 4, and 8 vCPUs, among them restore-downtime, snapshot-core, smp-snapshot, console-snapshot, and time-abi-conformance, passed on both backends, all 47 scenario records of each.
  • NVX at 6d3442d14: python scripts\nvx.py verify (consistent); the openvmm_vectors and openvmm_cpu_profiles comparisons with --ignored (3 and 1 passed) against the pinned revision; python scripts\check_markdown_links.py; and git diff --check.
  • Every commit is signed, both these and [DO NOT REVIEW] vmcore: map VM times beyond the host clock to its latest time openvmm#4820's.
  • CI: at head 6d3442d14, every check passed (run 41014162393), the microVM tests on Linux/KVM, Linux/MSHV, and Windows/WHP among them. The first head, bae0fe48e, differed only in the roadmap remark that the review clarified, and its run was superseded. [DO NOT REVIEW] vmcore: map VM times beyond the host clock to its latest time openvmm#4820's checks all passed at 5e0ab772e: the first attempt of its run 41011644930 failed two Windows vmm-tests jobs on failures outside this change, which their rerun passed. Two SNP tests panicked in openhcl_boot, which does not depend on vmcore, with "Imported regions hash mismatch", as that job did in at least seven other pull requests' runs of the past day, and hibernate_token failed with os error 10022 in petri's harness before OpenVMM started.
  • Windows/WHP runs only in CI: nvx-whp-test's SSH host key changed, so it was not used, and the bare-metal WHP host was running another OpenVMM.

Merge microsoft/openvmm#4820 into feature/microvm/dev, by fast-forwarding that branch to 5e0ab772e, before merging this pull request.

Pin microsoft/openvmm#4820 at 5e0ab772eeea14f4d0f5ed05474856569358f80e,
two commits on bcfc47cff, the previous pin, which is the head of
feature/microvm/dev.

OpenVMM's VM time keeper maps a VM time to the host time at which VM
time reaches it, to arm its host timer and for VmTimeAccess::host_time,
from the time's circular distance after the last start. A timeout more
than about 584 years ahead, beyond what the host clock's 64 bits of
nanoseconds can represent, overflowed that host time and panicked the
task that armed it: a timeout armed across a restore to a VM time that
it follows by 2.66e18 units panicked the keeper's task at start(). A
timeout nearly half the ring, about 29,000 years, ahead of now, which
VM time's circular order puts before the start once VM time has run,
mapped to the start's host time, in the past, so the keeper's host timer
fired and was re-armed forever, with the timer state's lock held.

The first commit saturates the host time at the host clock's latest
time, a deadline that every timer backend accepts and never reaches.
The second maps a time that the circular order puts before the start,
but after now, to that latest time too. Each adds tests that fail
without it.

The protocol vectors and built-in CPU profiles that nvx-rs copies are
unchanged.

Fixes #1038

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The roadmap’s reference to “the two commits” is ambiguous and should explicitly identify positions 368 and 369 as independently upstreamable.

1 open issue
What changed in this PR

Updates NVX’s OpenVMM pin to address extreme VM deadlines that could panic the time-keeper task or make it spin, and documents the fixes in the upstream roadmap.

Changes:

  • Pins OpenVMM to 5e0ab772e, which maps unrepresentable deadlines to the host clock’s latest time.
  • Records positions 368–369, refreshes the pin’s lineage and commit counts, and explains the fixes’ independent upstream applicability.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pinned revision, chain mapping, merge history, and VM-time fix descriptions.

Review effort: Max

Comment thread doc/openvmm-upstream-roadmap.md Outdated
Record the two commits that the previous commit pins as positions 368
and 369 of the microVM chain, and move the NVX pin to 5e0ab772e.

Positions 368 and 369 make OpenVMM's VM time keeper map a timeout
beyond what the host clock can represent to the host clock's latest
time: one more than about 584 years after the last start no longer
panics the task that arms it, and one nearly half the ring after now,
which VM time's circular order puts before the start, no longer arms the
host timer in the past. They change only vmcore's vmtime.rs, which
upstream main and the fork base have alike, and apply to upstream main
as they are; say so in the table's remarks. The pin is now 212 commits
past the mapped tip.

Position 367, bcfc47cff, is the head of feature/microvm/dev since
microsoft/openvmm#4818 merged positions 366 and 367. Say so in the
scope, and name microsoft/openvmm#4820 as the proposal of positions 368
and 369.

Refs #1038

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 30, 2026 06:27
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-1038-debug-test-fix-commit branch from bae0fe4 to 6d3442d Compare October 30, 2026 06:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pin changes shared VM timer behavior through unmerged OpenVMM #4820, with exact-head CI and KVM/MSHV/WHP integration validation still pending.

No open issues 🎉

1 resolved since last review

Review effort: Max

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vmtime: armed timeout + restore to a large time panics the keeper task on start()

2 participants