Skip to content

openvmm: run the KVP IC test on Windows Server 2025 - #1036

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/nvx-ci-failure-investigation-624ea356
Oct 30, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/nvx-ci-failure-investigation-624ea356

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1035. Pins microsoft/openvmm#4816 at 3d7512af653717aa9123595e993a56cb8587106a, which ports microsoft/openvmm#4803: OpenVMM's KVP IC test boots Windows Server 2025 instead of Windows Server 2022, whose COM activation race can lose WMI's server-registration notification, so that the guest answers the test's IP information request with 0x80080005 after 120 seconds, as it did in the WHP run of dev at c938ab2. The test's name holds its image, so this also renames it where NVX selects it, and keeps it off NVX's Linux legs.

Based on dev at ce6c0ecfb. microsoft/openvmm#4816 is two commits directly on b58040ee4, the head of feature/microvm/dev since microsoft/openvmm#4823 merged and the revision that dev pins since #1049 merged, and the pin is that pull request's head commit itself, so this branch's two commits are its whole diff.

Change

Commit Files Change
8a00274cb openvmm openvmm, scripts/nvx_tools/ci.py, scripts/nvx_tools/openvmm_test_shards.json, scripts/test_nvx_tools.py, doc/ci.md b58040ee4 → 3d7512af6: the two commits of microsoft/openvmm#4816; the test's new name in the WHP selection and the shards; and the Linux filters' exclusion of every Windows Server Datacenter Core test
78fb9fb32 doc doc/openvmm-upstream-roadmap.md Records them as positions 371 and 372, after #1049's position 370, which microsoft/openvmm#4823 merged; and moves the NVX pin

OpenVMM's control-session protocol vectors and built-in CPU profiles are unchanged at the pin.

OpenVMM changes

  • The port (2dec9edcb). A cherry-pick of vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803's only commit, a73891221, by Steven Malis: the test's image changes from windows_datacenter_core_2022_x64 to windows_datacenter_core_2025_x64. Its patch has the same patch ID as vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803's, 296ebafd9a4e, and ic.rs after the commit is that pull request's, byte for byte. vmm_tests: Fix kvp test flakiness by updating windows version openvmm#4803 is approved but not merged; once it lands upstream, a rebase onto it drops this commit.
  • The Linux KVM image (3d7512af6). Upstream's x64 Linux KVM and MSHV jobs selected the moved test but downloaded only the Server 2022 image. This stages Server 2025 for KVM, where the test passed. On MSHV, two runs on different workers reached the moved test but failed Server 2025's required initial reset with HvCallSetVpRegisters error: Invalid parameter; that job therefore excludes the test and does not download its image. KVM and the Windows-host jobs retain coverage. This changes no generated pipeline.

NVX changes

  • The WHP selection. OPENVMM_WHP_TESTS names multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic instead of the test's Server 2022 name. test-openvmm does not check that each name that it selects exists, so a pin without the rename would have dropped the test from WHP silently.
  • The shards. plan-openvmm-test-shards --seconds …_2025_x64_kvp_ic=57.4 plans them again, as OpenVMM VMM test shards prescribes for a renamed test. The estimate is NVX's median of 34.0 s on Server 2022 times 1.69, the mean ratio of the test's durations on Server 2025 and 2022 on upstream's runners: 44.3 against 23.1-25.5 s on x64-windows-intel, and 36.3 against 23.1-23.7 s on x64-windows-amd. A regeneration from CI's reports will replace it. The plan moves three other tests to balance the shards, which take 128.4 and 128.3 s instead of 119.8 and 123.9 s, so ci.md now gives the latter as the first plan's.
  • The Linux filters. The KVM and MSHV filters select every test whose name contains openvmm and left out Windows guests only by windows_datacenter_core_2022_x64, so they would have selected the renamed test, which no NVX Linux leg runs. They now leave out windows_datacenter_core. At this pin, that newly leaves out only the renamed test and Server 2025 tests that need VBS isolation, which petri finds on no Linux host. test_linux_filters_leave_out_every_windows_server_test checks that neither filter selects a Windows Server test of the WHP selection, while both still select a Linux guest's IC test and the required microVM tests. For it, _nextest_filter_matches learns nextest's substring form of test(), which those filters use.

Validation

For the series rebased onto #1049 and microsoft/openvmm#4823:

For the follow-up final series:

  • At f10e4566d0880e13021067ded5714c087da5451d, python scripts\nvx.py verify, all 15 focused filter/shard tests, the 4 ignored OpenVMM vector/profile comparisons, ruff check, ruff format --check, Linux and Windows pyright, and check_markdown_links.py passed.
  • On the configured nvx-kvm-test Linux host at the preceding series, all 1,492 CLI suite tests, 84 adversarial tests, 4 host-resolver tests, and 25 Specula tests passed. The follow-up base changes do not overlap the PR's executable changes; exact-head CI supplies the complete matrix.
  • On Windows at an earlier rebased head, the same 1,492-test suite's 27 failures were confined to unchanged Linux-script harnesses on this workstation; all changed-area tests passed. test_adversarial.py, test_hosts.py, and every CLI --help check passed.
  • OpenVMM, with Rust 1.95.0: cargo xflowey restore-packages; package-scoped check, clippy with -D warnings, and docs for vmm_tests and flowey_hvlite; 3 flowey_hvlite nextest tests; cargo xflowey regen, which changed nothing; and cargo xtask fmt --fix (clean). The complete local vmm_tests binary needs CI's staged images and hypervisors.
  • OpenVMM's old-head run 40986183074 showed an unrelated vmms.exe crash on TDX and one unrelated SNP OpenHCL boot failure, both of which passed the failed-job rerun. Its moved KVP test passed on Linux KVM, while the MSHV reset failure reproduced twice and led to the filter above.

Previous exact-head NVX run 41072354769 passed at f7f4054ac29994402da0540e82f46e0564e8d565, and previous exact-head OpenVMM run 41012439247, attempt 4, passed at 6b409bd1035678a32faac3761c36e2219f7f8000. Follow-up exact-head OpenVMM run 41098528203, attempt 4, passed at 549a215121f1e691b34234c6ee39a1234fe9ff80; attempts 1-3 hit unrelated SNP VM-start flakes. Follow-up exact-head NVX run 41099118684, attempt 1, passed at f10e4566d0880e13021067ded5714c087da5451d. Rebased exact-head OpenVMM run 41133653556, attempt 1, passed at 3d7512af653717aa9123595e993a56cb8587106a, and rebased exact-head NVX run 41137382366, attempt 1, passed at 78fb9fb3269c6341a9a121021cc7d99209f229a0.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A maintainer needs to confirm exact-head CI for unmerged OpenVMM PR #4816 and the renamed KVP test, which has not run on NVX’s WHP runners.

No open issues 🎉

What changed in this PR

Updates NVX’s OpenVMM integration so the KVP integration-component test boots Windows Server 2025, avoiding the Windows Server 2022 COM activation race described in #1035.

Changes:

  • Updates the WHP test selection and keeps Windows Server tests excluded from Linux backends.
  • Rebalances WHP shards for the renamed test and adds regression coverage for Linux filters.
  • Documents the new OpenVMM pin and clarifies historical shard timings.
File Description
scripts/​test_nvx_tools.py Adds Windows Server exclusion checks and substring-filter support to the test helper.
scripts/​nvx_tools/​openvmm_test_shards.json Renames the KVP test, updates its duration estimate, and rebalances WHP shards.
scripts/​nvx_tools/​ci.py Selects the Server 2025 KVP test and makes Linux exclusions independent of the Server version.
doc/​openvmm-upstream-roadmap.md Records the new pin, upstream port, and Linux image-staging change.
doc/​ci.md Clarifies that the documented shard timings describe the original plan.

Review effort: Max

@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/nvx-ci-failure-investigation-624ea356 branch from 67d0b2e to 14c48b3 Compare October 30, 2026 06:31
Copilot AI max review requested due to automatic review settings October 30, 2026 06:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The 57.4 s test estimate understates NVX’s measured 178.3 s runtime, affecting shard balancing without justification for retaining the lower estimate.

1 open issue

Review effort: Max

@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/nvx-ci-failure-investigation-624ea356 branch from 14c48b3 to 1e25288 Compare October 30, 2026 14:18
Copilot AI max review requested due to automatic review settings October 30, 2026 14:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The roadmap’s #4804 merge description double-counts position 365, so the companion range should be 362–364 rather than 362–365.

1 open issue
1 resolved since last review

Review effort: Max

Comment thread doc/openvmm-upstream-roadmap.md Resolved
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/nvx-ci-failure-investigation-624ea356 branch from 1e25288 to f7f4054 Compare October 30, 2026 14:31
Copilot AI max review requested due to automatic review settings October 30, 2026 14:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pinned OpenVMM PR is still unmerged and exact-head WHP CI remains pending, requiring maintainer confirmation of landing order and CI results before merging.

No open issues 🎉

1 resolved since last review

Review effort: Max

@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/nvx-ci-failure-investigation-624ea356 branch from f7f4054 to f10e456 Compare October 30, 2026 17:28
Copilot AI max review requested due to automatic review settings October 30, 2026 17:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The promoted OpenVMM pin is an unmerged PR head with exact-head OpenVMM and NVX platform CI still pending, requiring confirmation of runtime results and coordination of its upstream landing.

No open issues 🎉

Review effort: Max

Pin microsoft/openvmm#4816 at 3d7512af653717aa9123595e993a56cb8587106a,
two commits on b58040ee4, the previous pin and the head of
feature/microvm/dev since microsoft/openvmm#4823 merged.

OpenVMM's KVP IC test booted Windows Server 2022, whose COM activation
race can lose WMI's server-registration notification, so that the
guest answers the test's IP information request with 0x80080005 after
120 s. The race failed the test in the WHP run of dev at c938ab2. The
first commit ports microsoft/openvmm#4803 unchanged and moves the test
to Windows Server 2025, which has the fix. The second stages the Server
2025 image for the check-in gates' x64 Linux KVM job. It excludes the
test from the MSHV job, where Server 2025's required initial reset
consistently fails with HvCallSetVpRegisters returning Invalid parameter.

The test's name holds its image, so it is now
multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic.
Rename it in OPENVMM_WHP_TESTS, and plan the WHP shards again with an
estimate of 57.4 s for it until CI's reports measure it: NVX's median
of 34.0 s on Server 2022 times 1.69, the mean ratio of the test's
durations on Server 2025 and 2022 on upstream's x64-windows-intel and
x64-windows-amd runners. The plan moves three other tests to balance
the shards, which take 128.4 and 128.3 s instead of 119.8 and 123.9 s,
so ci.md now gives the latter as the first plan's.

The KVM and MSHV filters select every test whose name contains openvmm
and left out Windows guests only by windows_datacenter_core_2022_x64,
so the renamed test would have joined them. Leave out every Windows
Server Datacenter Core test instead, and test that through the
substring form of nextest's test(), which the filters use.

The protocol vectors and built-in CPU profiles that nvx-rs copies are
unchanged.

Refs #1035

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… roadmap

Record the two commits that the previous commit pins as positions 371
and 372 of the microVM chain, and move the NVX pin to 3d7512af6.

Position 371 makes the change of microsoft/openvmm#4803 unchanged: the
KVP IC test boots Windows Server 2025 instead of 2022, whose COM
activation race can fail the guest's answer to the test's IP
information request. Position 372 stages the Server 2025 image for the
check-in gates' x64 Linux KVM job. It excludes the test and its image
from the MSHV job, where Server 2025's required initial reset fails. A
rebase onto an upstream main that includes microsoft/openvmm#4803 drops
position 371 and reconciles position 372 with upstream's changes for
those jobs; say so in the table's remarks. The pin is now 215 commits
past the mapped tip.

Keep the re-checked chain in the scope: microsoft/openvmm#4823 merged
position 370, so position 370, b58040ee4, is now the head of
feature/microvm/dev, and microsoft/openvmm#4816 proposes positions 371
and 372.

Refs #1035

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/nvx-ci-failure-investigation-624ea356 branch from f10e456 to 78fb9fb Compare October 30, 2026 22:19
Copilot AI max review requested due to automatic review settings October 30, 2026 22:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Exact-head OpenVMM and NVX CI remain pending, leaving hardware-backed WHP/KVM/MSHV validation and execution of the renamed Windows Server 2025 KVP test unconfirmed.

No open issues 🎉

Review effort: Max

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Oct 30, 2026
Merged via the queue into dev with commit d6c7eea Oct 30, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants