[Pedro Henrique Penna (ppenna)]

NVX CI stops building and running OpenVMM's unit and VMM tests. A new GitHub-hosted job, openvmm-ci, checks instead that OpenVMM's own CI passed the OpenVMM revision that NVX pins. Developers can still run OpenVMM's tests locally with test-openvmm and test-openvmm-unit.

Based on dev at 23ad26729.

Why

NVX pins feature/microvm/dev, and OpenVMM's own CI already tests that branch:

• .gitmodules tracks feature/microvm/dev, and the openvmm gitlink, 3d7512af6, is the branch's tip (the head of [DO NOT REVIEW] vmm_tests: run the KVP IC test on Windows Server 2025 (port of #4803) openvmm#4816).
• Since OpenVMM commit c2722fbcd (2026-10-21), OpenVMM's checkin gates trigger on feature/microvm/**:
  • openvmm-pr.yaml runs on each pull request into the branch that isn't a draft.
  • openvmm-ci.yaml runs on each push to the branch.
• OpenVMM pull requests land by fast-forward pushes of their heads, so the pinned SHA gets both a PR run and a push run.

The OpenVMM jobs in NVX CI ran tests that OpenVMM's gates also run, and they cost a lot. From 2026-10-21 to 2026-10-31 (1,034 NVX CI runs):

• The five jobs ran in 673 runs:
  • build-openvmm-vmm-tests-linux and build-openvmm-vmm-tests-windows
  • openvmm-vmm-tests-linux and openvmm-vmm-tests-windows (WHP in two shards)
  • openvmm-unit-tests
• They used about 620 runner-hours, 517 of them on the self-hosted KVM, MSHV, and WHP pools.
• They failed 9 times, never because of an OpenVMM defect:
  • 4 times from an NVX script SyntaxError on a dispatched branch;
  • once from a lost WHP runner;
  • 4 times from flaky tests that passed on the same pin in other runs.

Coverage

On the same pin, OpenVMM's gates run every test that NVX's jobs ran, with two differences:

• Emulated Intel VT-d: NVX's KVM leg (61 tests) runs 4 VT-d tests that OpenVMM's AMD KVM job skips. OpenVMM's MSHV and Windows Intel jobs run those tests. NVX's MSHV (60) and WHP (29) selections are subsets of OpenVMM's.
• TTRPC microVM test: NVX's jobs booted NVX's kernel at 1, 2, 4, and 8 vCPUs, while OpenVMM's gates boot OpenVMM's own kernel at 1 vCPU. NVX uses no TTRPC, and the NVX microVM tests boot and restore NVX's kernel at those vCPU counts on every backend.

What changed

Commit	Change
scripts: add check-openvmm-ci to check OpenVMM's CI of a revision	Adds scripts/nvx_tools/openvmm_ci.py and nvx.py check-openvmm-ci, with tests and documentation in doc/usage.md and doc/project-structure.md.
ci: check OpenVMM's own CI instead of running OpenVMM's tests	Replaces the five jobs with openvmm-ci. The change detector's run-openvmm-unit-tests and run-openvmm-vmm-tests outputs become run-openvmm-ci-check, and the required-results policy changes to match. Removes the CI-only plumbing (details below), adds cache saves to build-openvmm, and updates doc/ci.md, doc/usage.md, doc/contribute.md, the Copilot instructions, and the nvx-rs README.
doc: check OpenVMM's CI of the pin when promoting it	Makes the nvx-openvmm-promote skill gather the check's evidence before merging a pin promotion.

The CI-only plumbing that the second commit removes:

• the commands build-openvmm-tests, openvmm-test-cache-key, check-openvmm-tests, reuse-openvmm-tests, and plan-openvmm-test-shards;
• test-openvmm's --content and --shard options;
• openvmm_test_shards.py and openvmm_test_shards.json.

How the check works

The check reads GET /repos/microsoft/openvmm/actions/runs?head_sha=… for the pinned SHA. It keeps only runs of the two gate workflows whose head repository is microsoft/openvmm, then reads each run's latest job attempts.

• Required jobs: six jobs cover NVX's use of OpenVMM:

  • unit tests [x64-linux] and unit tests [x64-windows];
  • run vmm-tests [x64-linux-amd-kvm], [x64-linux-intel-mshv], [x64-windows-intel], and [x64-windows-amd].

  Each must have passed in a run of either workflow. A required name matches one whole comma-separated part of a job name, because the two workflows name their unit-test jobs differently.

• Excluded jobs: the gates' TDX, SEV-SNP, and other OpenHCL or confidential-VM jobs aren't required, because NVX uses none of those features. OpenVMM's push CI on the branch has failed in those jobs since 10-25.

• Verdicts: passed, pending (a job hasn't finished, or a run hasn't listed it yet), failed, or missing.

• Waiting:

  • On pull_request and workflow_dispatch, the job waits up to 300 minutes, checking every 5 minutes, because a pin promotion usually opens before OpenVMM's gates finish (about 1.5 to 2 hours).
  • A required job that failed, or that no run will run, ends the wait at once, because the check can no longer pass. After a rerun of the OpenVMM job passes, rerun openvmm-ci.
  • Merge groups, dev pushes, and scheduled runs check a pin that already passed in its pull request, so they fail at once instead of delaying the queue.
  • While the job may still wait, a failed API call is retried at the next check, at most twice in a row.

• Access: the job uses no secrets and no OpenVMM checkout, so pull requests from forks run it too.

• Triggers: it runs when the pin, ci.yml, nvx.py, or the check's modules change (openvmm_ci.py, nightly_ci.py, common.py, build_constants.py, __init__.py).

• Results policy:

  • Required status check expects the job to succeed when its output is set, and to be skipped otherwise.
  • The development release and the recording of the performance results (performance-record) need it to succeed, or to be skipped because the change selected no check.
  • The single required check aggregates these results, so the rulesets don't need to change.

Reviewers should know

• Token: the job passes ${{ github.token }} to read microsoft/openvmm's public Actions runs. This PR's CI confirmed that the default token can read them: its openvmm-ci job passed in 11 seconds. A fork's pull request gets a read-only token, which reads public data the same way. Without a token, the check works anonymously (I verified this locally) but under a lower rate limit.
• Caches: the removed jobs were the only ones that saved the OpenVMM input and Cargo caches that the OpenVMM binary builds restore. build-openvmm now saves them on a dev push whose build misses the binary cache.
• Runner capacity: the self-hosted pools no longer run the Linux and Windows VMM-test builds or the KVM, MSHV, and WHP VMM-test legs. doc/ci.md updates the runner-capacity and job-graph sections to match.

Validation

• ruff check and ruff format --check, pyright for Linux and for Windows, and check_markdown_links.py pass on every commit.
• All 17 script suites that validate-nvx runs pass locally on Windows: 1,638 tests, 73 skipped. test_adversarial.py (84 tests) and the host-connect tests (4) pass too.
• The first commit passes on its own too: test_nvx_tools.py, test_nightly_ci.py, and the docs tests, 813 tests in all.
• The live check passes on the current pin, 3d7512af6. All six jobs passed in OpenVMM run 41143115866.
• During development, the check also reported pending for an OpenVMM pull request whose gates were still running, and missing for a commit that no gate had tested.
• CI run 41171077133 on the exact head 6c8815c23 passed: 34 jobs succeeded, and the 8 skipped jobs run only on dev pushes, merge groups, or full runs. Its openvmm-ci job found all six required jobs passed for 3d7512af6.

[Copilot]

🟡 Changes recommended

The polling loop delays reporting failed or missing required jobs while other jobs remain pending, potentially for 300 minutes, contrary to documented fail-fast behavior; it needs an immediate return and mixed-result regression coverage.

1 open issue

• Fail fast when any required job fails or is missing · New

What changed in this PR

This PR replaces NVX’s duplicate OpenVMM test jobs with a check of upstream CI results for the pinned revision, reducing runner usage while retaining local test commands.

Changes:

• Adds check-openvmm-ci to verify six required upstream jobs, with polling and evidence summaries.
• Updates CI selection, required results, release gates, and cache saves.
• Removes CI-only test packaging and sharding tools, and documents the new workflow.

File	Description
scripts/​nvx.py	Registers the upstream check and removes obsolete CLI options.
scripts/​nvx_tools/​shard_plans.py	Updates the docstring to reflect its remaining microVM use.
scripts/​nvx_tools/​openvmm_test_shards.py	Removes the OpenVMM test shard planner.
scripts/​nvx_tools/​openvmm_test_shards.json	Removes the OpenVMM shard configuration.
scripts/​nvx_tools/​openvmm_ci.py	Adds upstream job verification, polling, and reporting.
scripts/​nvx_tools/​ci.py	Updates change selection and required results; removes test handoff code.
nvx-rs/​README.md	Updates guest-image and test-selection documentation.
doc/​usage.md	Documents the new command and removes retired command details.
doc/​project-structure.md	Adds the upstream checker to the module listing.
doc/​contribute.md	Clarifies upstream test ownership and local commands.
doc/​ci.md	Documents upstream verification and the revised CI structure.
.github/​workflows/​ci.yml	Replaces five test jobs with the upstream CI check.
.github/​skills/​nvx-openvmm-promote/​SKILL.md	Adds upstream CI evidence to pin-promotion guidance.
.github/​copilot-instructions.md	Directs OpenVMM test validation to upstream CI.
.github/​actions/​build-openvmm/​action.yml	Saves input and Cargo caches after uncached builds on dev pushes.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The sole replacement gate still needs an exact-head NVX CI run to verify cross-repository OpenVMM Actions access using the restricted GITHUB_TOKEN, including on fork PRs.

No open issues 🎉

1 resolved since last review

• Fail fast when any required job fails or is missing

_{Review effort: Balanced}