From 72aa678b44dffb9a3e5fe8f85740d2e0685d72cd Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 21:06:10 -0700 Subject: [PATCH 1/3] scripts: add check-openvmm-ci to check OpenVMM's CI of a revision NVX pins a revision of feature/microvm/dev in microsoft/openvmm, whose checkin gates test it: since OpenVMM's c2722fbcd of 2026-10-21, the OpenVMM PR workflow runs them on each pull request into the branch that is ready for review, and the OpenVMM CI workflow on each push to it. Nothing on OpenVMM's side requires them to pass, though: the branch has no required status check, a pull request lands by a push of its head, and every OpenVMM CI run of the branch from 2026-10-25 to 2026-10-30 failed in the VMM tests of OpenHCL and confidential VMs on its TDX and SEV-SNP hosts, which NVX does not use. A run's conclusion therefore says little about the jobs that matter to NVX. nvx.py check-openvmm-ci reads the runs of those two workflows whose head is, in microsoft/openvmm, the revision that the checkout's index pins or the one that --revision names, with the latest attempt of each of their jobs. It requires the jobs that cover NVX's use of OpenVMM to have passed in at least one of those runs: unit tests [x64-linux] and [x64-windows], and run vmm-tests [x64-linux-amd-kvm], [x64-linux-intel-mshv], [x64-windows-intel], and [x64-windows-amd]. It ignores the runs of pull requests from forks, which run the fork's copy of the workflows. A job is pending while a run that has not finished may still pass it, since a run creates a job only once the jobs that it needs finish; failed when every run that ran it finished it without passing; and missing when no run will run it. With --wait-minutes, the command looks again every five minutes while a job is pending, and after a look that failed, at most twice in a row, for at most that long. It prints each verdict with the job that decided it, appends the verdicts as a table to the --summary file, such as the step summary, and fails with a GitHub error for each job that did not pass. It reads the public repository through the nightly report's GitHub client, with GH_TOKEN or GITHUB_TOKEN when one is set. For the current pin, 3d7512af6, the pull request's and the branch's runs both passed all six jobs; for a pull request whose Windows VMM tests still ran, the check reported those two as pending, and for a commit that no run tested, all six as missing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/project-structure.md | 1 + doc/usage.md | 32 ++ scripts/nvx.py | 7 + scripts/nvx_tools/openvmm_ci.py | 347 ++++++++++++++++++++ scripts/test_nvx_tools.py | 565 ++++++++++++++++++++++++++++++++ 5 files changed, 952 insertions(+) create mode 100644 scripts/nvx_tools/openvmm_ci.py diff --git a/doc/project-structure.md b/doc/project-structure.md index 1e5e5f4b8..3c90763ff 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -140,6 +140,7 @@ nvx/ | | |-- microvm_tests.py NVX microVM correctness tests | | |-- nightly_ci.py Nightly CI failure report | | |-- nvx_rs.py Finds and builds nvx-rs, and runs it for the microVM tests +| | |-- openvmm_ci.py Check that OpenVMM's own CI passed the pinned revision | | |-- openvmm_process.py Foreground OpenVMM process control for tests | | |-- performance.py CI performance collection, persistence, and gates | | |-- performance_history.py Nightly persistence of the results that dev pushes recorded diff --git a/doc/usage.md b/doc/usage.md index 712aa7ea0..d7f2e0e4d 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -433,6 +433,7 @@ is authorized for the organization when it enforces single sign-on. | `report-nightly-ci` | Open, update, or close the report of failed nightly CI runs. | | `cancel-superseded-merge-groups` | Cancel the CI runs of the merge groups that the merge queue replaced. | | `persist-recorded-performance` | Persist the performance results that the CI runs of `dev` pushes recorded, in `dev`'s first-parent order. | +| `check-openvmm-ci` | Check that OpenVMM's own CI passed the pinned OpenVMM revision. | | `check-static-elf` | Check that executables are static ELF files without an interpreter or an executable stack. | | `test-openvmm-unit` | Run the OpenVMM workspace unit and documentation tests. | | `build-openvmm-tests` | Build OpenVMM Petri VMM tests without running them. | @@ -745,6 +746,37 @@ with each run, `--summary` writes the outcome to a Markdown file, and file, when the command persisted any. It exits with status 1 when it refused a run's results or stopped, after persisting the others. +### `check-openvmm-ci` + +```text +python3 scripts/nvx.py check-openvmm-ci + [--revision SHA] + [--wait-minutes MINUTES] + [--summary PATH] +``` + +Checks that OpenVMM's own CI passed the OpenVMM revision that the checkout's +index pins, or the full commit SHA that `--revision` names. +It reads the runs of the OpenVMM PR +and OpenVMM CI workflows of `microsoft/openvmm` whose head is the revision, +with the latest attempt of each of their jobs, and requires each job that +covers NVX's use of OpenVMM, the x86-64 Linux and Windows unit tests and the +VMM tests on KVM, MSHV, and Windows, to have passed in at least one of those +runs. It ignores the runs of a pull request from a fork, which run the fork's +copy of the workflows, and the workflows' other jobs. A job is pending while a +run that has not finished may still pass it, failed when every run that ran it +finished it without passing, and missing when no run will run it. With +`--wait-minutes`, the command looks again every five minutes while a job is +pending and none failed or is missing, for at most that long; by default it +looks once. While it may still wait, it also looks again, with a GitHub +warning, after a look that failed, as when GitHub's API is briefly unavailable, +but a third failure in a row ends it. It prints each job's verdict with the job +that decided it, appends the verdicts as a table to the Markdown file that +`--summary` names, such as `$GITHUB_STEP_SUMMARY`, and fails, with a GitHub +error for each, unless every job passed. It reads a token from `GH_TOKEN` or +`GITHUB_TOKEN` when one is set; the repository is public, so it needs none, but +GitHub allows fewer requests without one. + ### `check-static-elf` ```text diff --git a/scripts/nvx.py b/scripts/nvx.py index ada0ae81a..5e6520581 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -99,6 +99,7 @@ configure_shard_parser as configure_microvm_shard_parser, ) from nvx_tools.nightly_ci import configure_parser as configure_nightly_ci_parser +from nvx_tools.openvmm_ci import configure_parser as configure_openvmm_ci_parser from nvx_tools.openvmm_test_shards import ( configure_parser as configure_openvmm_test_shard_parser, ) @@ -760,6 +761,12 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) configure_performance_history_parser(recorded_performance) + openvmm_ci = subparsers.add_parser( + "check-openvmm-ci", + help="check that OpenVMM's own CI passed the pinned OpenVMM revision", + ) + configure_openvmm_ci_parser(openvmm_ci) + static_elf = subparsers.add_parser( "check-static-elf", help=( diff --git a/scripts/nvx_tools/openvmm_ci.py b/scripts/nvx_tools/openvmm_ci.py new file mode 100644 index 000000000..138f80a4f --- /dev/null +++ b/scripts/nvx_tools/openvmm_ci.py @@ -0,0 +1,347 @@ +"""Evidence that OpenVMM's own CI passed the OpenVMM revision that NVX pins. + +NVX pins a revision of `feature/microvm/dev` in microsoft/openvmm, whose +checkin gates test it: the OpenVMM PR workflow on each pull request into the +branch that is ready for review, and the OpenVMM CI workflow on each push to +it. ``nvx.py check-openvmm-ci`` requires the jobs of those gates that cover +NVX's use of OpenVMM, the x86-64 Linux and Windows unit tests and the VMM tests +on KVM, MSHV, and Windows, to have passed for the pinned revision in either +workflow, and waits while they run. The gates' other jobs, such as the VMM tests +on TDX and SEV-SNP hosts, test OpenHCL and confidential VMs, which NVX does not +use. +""" + +from __future__ import annotations + +import argparse +import os +import re +import time +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol, cast + +from .common import ScriptError, git_output +from .nightly_ci import GITHUB_API_URL, GitHubClient + +OPENVMM_REPOSITORY = "microsoft/openvmm" +# The workflows that run OpenVMM's checkin gates, by their paths. +OPENVMM_CI_WORKFLOWS = ( + ".github/workflows/openvmm-pr.yaml", + ".github/workflows/openvmm-ci.yaml", +) +# The jobs of the checkin gates that must pass, each as one of the +# comma-separated parts of a job's name: the OpenVMM PR workflow names its +# Linux unit test job "clippy [macos], unit tests [x64-linux]", and the OpenVMM +# CI workflow "clippy [x64-linux, macos], unit tests [x64-linux]". +OPENVMM_CI_REQUIRED_JOBS = ( + "unit tests [x64-linux]", + "unit tests [x64-windows]", + "run vmm-tests [x64-linux-amd-kvm]", + "run vmm-tests [x64-linux-intel-mshv]", + "run vmm-tests [x64-windows-intel]", + "run vmm-tests [x64-windows-amd]", +) +# How long the check waits between two looks at runs that have not finished. +# OpenVMM's gates take one and a half to two hours, so a look every five +# minutes adds little to a wait, and keeps a wait's requests, three a look, +# well below the rate limit that every job of a repository's runs shares. +POLL_SECONDS = 300 +# A look that fails, as when GitHub's API is briefly unavailable, is retried +# at the next poll, at most this many times in a row, while the check may +# still wait; a failure that persists ends the check. +FAILED_LOOK_RETRIES = 2 +_REVISION = re.compile(r"[0-9a-f]{40}") + + +class OpenVMMCiApi(Protocol): + """The GitHub REST operation that the check uses.""" + + def get_all( + self, + path: str, + params: Mapping[str, str] | None = None, + *, + key: str | None = None, + deadline: float | None = None, + ) -> list[object]: ... + + +@dataclass(frozen=True) +class OpenVMMJob: + """The latest attempt of a job of a run of OpenVMM's checkin gates.""" + + name: str + status: str + conclusion: str | None + url: str + + @property + def passed(self) -> bool: + return self.status == "completed" and self.conclusion == "success" + + +@dataclass(frozen=True) +class OpenVMMRun: + """A run of OpenVMM's checkin gates for a revision, with its jobs.""" + + url: str + finished: bool + jobs: tuple[OpenVMMJob, ...] + + +@dataclass(frozen=True) +class Verdict: + """Whether a required job passed for the revision: `passed`, `pending` + while a run may still pass it, `failed` when every run that ran it + finished it without passing, or `missing` when no run will run it.""" + + job: str + state: str + detail: str + + +def covers(name: str, job: str) -> bool: + """Return whether a job of the checkin gates named `name` runs `job`.""" + return job in (part.strip() for part in name.split(",")) + + +def _text(document: Mapping[str, object], field: str) -> str: + value = document.get(field) + if not isinstance(value, str): + raise ScriptError(f"GitHub returned a workflow run or job without {field}") + return value + + +def openvmm_ci_runs(api: OpenVMMCiApi, revision: str) -> list[OpenVMMRun]: + """Return the runs of OpenVMM's checkin gates whose head is `revision` + in OpenVMM's repository, with the latest attempt of each of their jobs.""" + runs: list[OpenVMMRun] = [] + for document in api.get_all( + f"/repos/{OPENVMM_REPOSITORY}/actions/runs", + {"head_sha": revision}, + key="workflow_runs", + ): + if not isinstance(document, Mapping): + raise ScriptError("GitHub returned an invalid workflow run") + run = cast(Mapping[str, object], document) + head_repository = run.get("head_repository") + # A pull request from a fork runs the fork's copy of the workflow. + if ( + str(run.get("path", "")).partition("@")[0] not in OPENVMM_CI_WORKFLOWS + or run.get("head_sha") != revision + or not isinstance(head_repository, Mapping) + or cast(Mapping[str, object], head_repository).get("full_name") + != OPENVMM_REPOSITORY + ): + continue + identifier = run.get("id") + if not isinstance(identifier, int) or isinstance(identifier, bool): + raise ScriptError("GitHub returned a workflow run without an id") + jobs: list[OpenVMMJob] = [] + for item in api.get_all( + f"/repos/{OPENVMM_REPOSITORY}/actions/runs/{identifier}/jobs", + {"filter": "latest"}, + key="jobs", + ): + if not isinstance(item, Mapping): + raise ScriptError("GitHub returned an invalid job") + job = cast(Mapping[str, object], item) + conclusion = job.get("conclusion") + jobs.append( + OpenVMMJob( + name=_text(job, "name"), + status=_text(job, "status"), + conclusion=conclusion if isinstance(conclusion, str) else None, + url=str(job.get("html_url", "")), + ) + ) + runs.append( + OpenVMMRun( + url=str(run.get("html_url", "")), + finished=run.get("status") == "completed", + jobs=tuple(jobs), + ) + ) + return runs + + +def assess_openvmm_ci( + runs: Sequence[OpenVMMRun], + required: Sequence[str] = OPENVMM_CI_REQUIRED_JOBS, +) -> list[Verdict]: + """Return the verdict on each required job: passed if any run passed it. + A run creates a job only when the jobs that it needs finish, so a run that + has not finished may still run a job that it does not list yet.""" + verdicts: list[Verdict] = [] + for job in required: + matches = [ + found for run in runs for found in run.jobs if covers(found.name, job) + ] + passed = [found for found in matches if found.passed] + if passed: + verdicts.append(Verdict(job, "passed", passed[0].url)) + continue + unfinished = [found for found in matches if found.status != "completed"] + unlisted = [ + run + for run in runs + if not run.finished + and not any(covers(found.name, job) for found in run.jobs) + ] + if unfinished or unlisted: + verdicts.append( + Verdict( + job, + "pending", + unfinished[0].url if unfinished else unlisted[0].url, + ) + ) + elif matches: + verdicts.append( + Verdict( + job, + "failed", + ", ".join( + f"{found.conclusion or found.status} {found.url}" + for found in matches + ), + ) + ) + else: + verdicts.append(Verdict(job, "missing", "no run of the gates ran it")) + return verdicts + + +def _annotation(text: str) -> str: + """Escape text for the message of a GitHub Actions workflow command.""" + return text.replace("%", "%25").replace("\r", "%0D").replace("\n", "%0A") + + +def check_openvmm_ci( + api: OpenVMMCiApi, + revision: str, + *, + wait_seconds: float, + sleep: Callable[[float], None] = time.sleep, + clock: Callable[[], float] = time.monotonic, +) -> list[Verdict]: + """Return the verdict on each required job for `revision`, looking again + while any is pending and none failed or is missing, or after a look that + failed, until `wait_seconds` run out.""" + deadline = clock() + wait_seconds + failures = 0 + while True: + try: + verdicts = assess_openvmm_ci(openvmm_ci_runs(api, revision)) + except (ScriptError, OSError) as error: + failures += 1 + if failures > FAILED_LOOK_RETRIES or clock() + POLL_SECONDS > deadline: + raise + print( + "::warning::" + + _annotation( + f"could not read OpenVMM's CI of {revision}, so the check " + f"looks again: {error}" + ), + flush=True, + ) + sleep(POLL_SECONDS) + continue + failures = 0 + pending = [verdict.job for verdict in verdicts if verdict.state == "pending"] + # A job that failed, or that no run will run, fails the check whatever + # the pending jobs do, so the check does not wait for them. + if ( + not pending + or any(verdict.state in ("failed", "missing") for verdict in verdicts) + or clock() + POLL_SECONDS > deadline + ): + return verdicts + print( + f">> waiting for OpenVMM's CI of {revision}: {', '.join(pending)}", + flush=True, + ) + sleep(POLL_SECONDS) + + +def _pinned_revision() -> str: + """Return the OpenVMM revision that the checkout's index pins.""" + return git_output("rev-parse", ":openvmm") + + +def _summary(revision: str, verdicts: Sequence[Verdict]) -> str: + rows = "".join( + f"| `{verdict.job}` | {verdict.state} | {verdict.detail} |\n" + for verdict in verdicts + ) + return ( + f"### OpenVMM CI of `{revision}`\n" + "| Job | Result | Evidence |\n" + "| --- | --- | --- |\n" + f"{rows}" + ) + + +def command_check_openvmm_ci(args: argparse.Namespace) -> None: + revision = args.revision or _pinned_revision() + if not _REVISION.fullmatch(revision): + raise ScriptError(f"{revision!r} is no full OpenVMM commit SHA") + # The repository is public, so any token reads its runs, and none does + # too, under a lower rate limit. + token = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN") or None + api = GitHubClient(token, os.environ.get("GITHUB_API_URL") or GITHUB_API_URL) + verdicts = check_openvmm_ci(api, revision, wait_seconds=args.wait_minutes * 60) + for verdict in verdicts: + print(f"{verdict.state}: {verdict.job}: {verdict.detail}") + if args.summary is not None: + with args.summary.open("a", encoding="utf-8", newline="") as summary: + summary.write(_summary(revision, verdicts)) + failures = [verdict for verdict in verdicts if verdict.state != "passed"] + if failures: + for verdict in failures: + print( + "::error::" + + _annotation( + f"OpenVMM's CI has not passed {verdict.job} " + f"({verdict.state}: {verdict.detail})" + ) + ) + raise ScriptError( + f"OpenVMM's CI has not passed {len(failures)} required job(s) for " + f"{revision}" + ) + + +def _minutes(value: str) -> int: + if not value.isdigit(): + raise argparse.ArgumentTypeError("must be a whole number of minutes") + return int(value) + + +def configure_parser(parser: argparse.ArgumentParser) -> None: + parser.description = ( + "Check that OpenVMM's checkin gates passed the jobs that cover NVX's use " + "of OpenVMM for the pinned OpenVMM revision." + ) + parser.add_argument( + "--revision", + metavar="SHA", + help="the full OpenVMM commit to check (default: the revision that the " + "checkout's index pins)", + ) + parser.add_argument( + "--wait-minutes", + type=_minutes, + default=0, + metavar="MINUTES", + help="how long to wait for runs that may still pass a job (default: 0)", + ) + parser.add_argument( + "--summary", + type=Path, + metavar="PATH", + help="Markdown file, such as $GITHUB_STEP_SUMMARY, to append the verdicts to", + ) + parser.set_defaults(handler=command_check_openvmm_ci) diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index c0da5d57f..6a18ae616 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -58,6 +58,7 @@ microvm_durations, microvm_shards, nvx_rs, + openvmm_ci, openvmm_process, openvmm_test_shards, performance, @@ -1573,6 +1574,33 @@ def test_release_commands_keep_their_cli_contract(self): ): nvx.parse_args(arguments) + openvmm_ci_check = nvx.parse_args(["check-openvmm-ci"]) + self.assertIsNone(openvmm_ci_check.revision) + self.assertEqual(openvmm_ci_check.wait_minutes, 0) + self.assertIsNone(openvmm_ci_check.summary) + self.assertIs(openvmm_ci_check.handler, openvmm_ci.command_check_openvmm_ci) + openvmm_ci_check = nvx.parse_args( + [ + "check-openvmm-ci", + "--revision", + "a" * 40, + "--wait-minutes", + "300", + "--summary", + "summary.md", + ] + ) + self.assertEqual(openvmm_ci_check.revision, "a" * 40) + self.assertEqual(openvmm_ci_check.wait_minutes, 300) + self.assertEqual(openvmm_ci_check.summary, Path("summary.md")) + for value in ("-1", "1.5", "an hour", ""): + with ( + self.subTest(wait_minutes=value), + patch("sys.stderr", new_callable=io.StringIO), + self.assertRaises(SystemExit), + ): + nvx.parse_args(["check-openvmm-ci", "--wait-minutes", value]) + def test_openvmm_help_describes_petri_vmm_tests(self): with ( patch("sys.stdout", new_callable=io.StringIO) as output, @@ -7704,6 +7732,543 @@ def test_merge_groups_cancel_the_runs_of_superseded_groups(self): ) +_OPENVMM_REVISION = "3d7512af653717aa9123595e993a56cb8587106a" + + +def _openvmm_run( + run_id: int, + workflow: str = "openvmm-pr.yaml", + *, + status: str = "completed", + head_sha: str = _OPENVMM_REVISION, + head_repository: str = "microsoft/openvmm", +) -> dict[str, Any]: + return { + "id": run_id, + "path": f".github/workflows/{workflow}", + "status": status, + "head_sha": head_sha, + "head_repository": {"full_name": head_repository}, + "html_url": f"https://github.com/microsoft/openvmm/actions/runs/{run_id}", + } + + +def _openvmm_job( + run_id: int, + name: str, + status: str = "completed", + conclusion: str | None = "success", +) -> dict[str, Any]: + return { + "name": name, + "status": status, + "conclusion": conclusion, + "html_url": f"https://github.com/microsoft/openvmm/actions/runs/{run_id}/job/1", + } + + +def _openvmm_gate_jobs( + run_id: int, conclusion: str | None = "success", status: str = "completed" +) -> list[dict[str, Any]]: + """Return a run's jobs that run OpenVMM's required jobs, under the names + that the OpenVMM CI workflow gives them, and a job that none requires.""" + return [ + _openvmm_job(run_id, "clippy [x64-linux, macos], unit tests [x64-linux]"), + _openvmm_job(run_id, "clippy [x64-windows], unit tests [x64-windows]"), + *( + _openvmm_job(run_id, f"run vmm-tests [{target}]", status, conclusion) + for target in ( + "x64-linux-amd-kvm", + "x64-linux-intel-mshv", + "x64-windows-intel", + "x64-windows-amd", + ) + ), + _openvmm_job( + run_id, "run vmm-tests [x64-windows-intel-tdx]", conclusion="failure" + ), + ] + + +_OpenVMMLook = tuple[ + Sequence[Mapping[str, Any]], Mapping[int, Sequence[Mapping[str, Any]]] +] + + +class _FakeOpenVMMCiApi: + """Answers the check's requests from the runs of _OPENVMM_REVISION and the + jobs of each run, one look after another, raises each look that is an + exception instead, and records each request.""" + + def __init__(self, *looks: _OpenVMMLook | Exception) -> None: + self.looks = list(looks) + self.look = -1 + self.requests: list[str] = [] + + def get_all( + self, + path: str, + params: Mapping[str, str] | None = None, + *, + key: str | None = None, + deadline: float | None = None, + ) -> list[object]: + assert deadline is None + if path == "/repos/microsoft/openvmm/actions/runs": + assert params == {"head_sha": _OPENVMM_REVISION}, params + assert key == "workflow_runs", key + self.look = min(self.look + 1, len(self.looks) - 1) + self.requests.append("runs") + look = self.looks[self.look] + if isinstance(look, Exception): + raise look + return list(look[0]) + match = re.fullmatch(r"/repos/microsoft/openvmm/actions/runs/(\d+)/jobs", path) + assert match is not None, path + assert params == {"filter": "latest"} and key == "jobs", (params, key) + self.requests.append(f"jobs {match.group(1)}") + look = self.looks[self.look] + assert not isinstance(look, Exception) + return list(look[1][int(match.group(1))]) + + +class OpenVMMCiTests(unittest.TestCase): + """The check that OpenVMM's own CI passed the pinned OpenVMM revision + (doc/usage.md#check-openvmm-ci).""" + + def test_a_required_job_is_a_whole_part_of_a_job_name(self): + for name, job, covered in ( + ("clippy [macos], unit tests [x64-linux]", "unit tests [x64-linux]", True), + ( + "clippy [x64-linux, macos], unit tests [x64-linux]", + "unit tests [x64-linux]", + True, + ), + ( + "clippy [x64-linux-musl, misc nostd], unit tests [x64-linux-musl]", + "unit tests [x64-linux]", + False, + ), + ( + "run vmm-tests [x64-windows-intel]", + "run vmm-tests [x64-windows-intel]", + True, + ), + ( + "run vmm-tests [x64-windows-intel-tdx]", + "run vmm-tests [x64-windows-intel]", + False, + ), + ( + "run vmm-perf [x64-linux-amd-kvm]", + "run vmm-tests [x64-linux-amd-kvm]", + False, + ), + ): + with self.subTest(name=name, job=job): + self.assertEqual(openvmm_ci.covers(name, job), covered) + # Each required job is one of the x86-64 legs that cover what NVX + # uses of OpenVMM: the unit tests and the VMM tests of each backend. + self.assertEqual( + openvmm_ci.OPENVMM_CI_REQUIRED_JOBS, + ( + "unit tests [x64-linux]", + "unit tests [x64-windows]", + "run vmm-tests [x64-linux-amd-kvm]", + "run vmm-tests [x64-linux-intel-mshv]", + "run vmm-tests [x64-windows-intel]", + "run vmm-tests [x64-windows-amd]", + ), + ) + + def test_reads_only_the_checkin_gates_of_the_revision_in_openvmm(self): + api = _FakeOpenVMMCiApi( + ( + [ + _openvmm_run(1, "openvmm-ci.yaml", status="in_progress"), + _openvmm_run(2), + # Other workflows, a fork's copy of the gates, and a run + # that GitHub lists for another head. + _openvmm_run(3, "openvmm-pr-release.yaml"), + _openvmm_run(4, "labeler.yml"), + _openvmm_run(5, head_repository="contributor/openvmm"), + _openvmm_run(6, head_sha="b" * 40), + ], + { + 1: [_openvmm_job(1, "run vmm-tests [x64-linux-amd-kvm]")], + 2: [ + _openvmm_job( + 2, "xtask fmt (windows)", "in_progress", conclusion=None + ) + ], + }, + ) + ) + + runs = openvmm_ci.openvmm_ci_runs(api, _OPENVMM_REVISION) + + self.assertEqual(api.requests, ["runs", "jobs 1", "jobs 2"]) + self.assertEqual( + runs, + [ + openvmm_ci.OpenVMMRun( + url="https://github.com/microsoft/openvmm/actions/runs/1", + finished=False, + jobs=( + openvmm_ci.OpenVMMJob( + name="run vmm-tests [x64-linux-amd-kvm]", + status="completed", + conclusion="success", + url="https://github.com/microsoft/openvmm/actions/runs/1/job/1", + ), + ), + ), + openvmm_ci.OpenVMMRun( + url="https://github.com/microsoft/openvmm/actions/runs/2", + finished=True, + jobs=( + openvmm_ci.OpenVMMJob( + name="xtask fmt (windows)", + status="in_progress", + conclusion=None, + url="https://github.com/microsoft/openvmm/actions/runs/2/job/1", + ), + ), + ), + ], + ) + # GitHub's answers must have the shape that the check reads. + for runs in (cast(Any, [1]), [{**_openvmm_run(1), "id": "1"}]): + with self.subTest(runs=runs), self.assertRaises(common.ScriptError): + openvmm_ci.openvmm_ci_runs( + _FakeOpenVMMCiApi((runs, {})), _OPENVMM_REVISION + ) + + @staticmethod + def _clock() -> tuple[Callable[[float], None], Callable[[], float], list[float]]: + """Return a sleep that advances a clock instead, the clock, and the + list of the sleeps.""" + now = [0.0] + sleeps: list[float] = [] + + def sleep(seconds: float) -> None: + sleeps.append(seconds) + now[0] += seconds + + return sleep, lambda: now[0], sleeps + + def test_any_run_that_passed_a_required_job_passes_it(self): + def job(name: str, status: str, conclusion: str | None) -> Any: + return openvmm_ci.OpenVMMJob(name, status, conclusion, f"url {name}") + + def run(finished: bool, *jobs: Any) -> Any: + return openvmm_ci.OpenVMMRun(f"run {finished}", finished, tuple(jobs)) + + # A pull request's run failed what the push's run passed, a job runs, + # a job failed, and a job that no run ran. + verdicts = openvmm_ci.assess_openvmm_ci( + [ + run( + True, + job("passed elsewhere", "completed", "failure"), + job("failed", "completed", "failure"), + job("failed", "completed", "cancelled"), + ), + run( + True, + job("passed elsewhere", "completed", "success"), + job("running", "in_progress", None), + ), + ], + ("passed elsewhere", "running", "failed", "absent"), + ) + self.assertEqual( + verdicts, + [ + openvmm_ci.Verdict( + "passed elsewhere", "passed", "url passed elsewhere" + ), + openvmm_ci.Verdict("running", "pending", "url running"), + openvmm_ci.Verdict( + "failed", "failed", "failure url failed, cancelled url failed" + ), + openvmm_ci.Verdict("absent", "missing", "no run of the gates ran it"), + ], + ) + # A run that has not finished may still create a job that it does not + # list yet, once the jobs that the job needs finish. + self.assertEqual( + openvmm_ci.assess_openvmm_ci([run(False)], ("absent",)), + [openvmm_ci.Verdict("absent", "pending", "run False")], + ) + self.assertEqual( + openvmm_ci.assess_openvmm_ci([], ("absent",)), + [openvmm_ci.Verdict("absent", "missing", "no run of the gates ran it")], + ) + + def test_check_waits_while_a_required_job_may_still_pass(self): + running = ( + [_openvmm_run(7, status="in_progress")], + {7: _openvmm_gate_jobs(7, conclusion=None, status="in_progress")}, + ) + passed = ( + [_openvmm_run(7), _openvmm_run(8, "openvmm-ci.yaml")], + {7: _openvmm_gate_jobs(7), 8: _openvmm_gate_jobs(8, "failure")}, + ) + api = _FakeOpenVMMCiApi(running, running, passed) + sleep, clock, sleeps = self._clock() + + with contextlib.redirect_stdout(io.StringIO()) as output: + verdicts = openvmm_ci.check_openvmm_ci( + api, + _OPENVMM_REVISION, + wait_seconds=600, + sleep=sleep, + clock=clock, + ) + + self.assertEqual(sleeps, [openvmm_ci.POLL_SECONDS] * 2) + self.assertEqual(api.requests.count("runs"), 3) + self.assertEqual({verdict.state for verdict in verdicts}, {"passed"}) + self.assertEqual( + [verdict.job for verdict in verdicts], + list(openvmm_ci.OPENVMM_CI_REQUIRED_JOBS), + ) + self.assertIn( + f">> waiting for OpenVMM's CI of {_OPENVMM_REVISION}: " + "run vmm-tests [x64-linux-amd-kvm], ", + output.getvalue(), + ) + + def test_check_stops_waiting_once_a_job_cannot_pass_or_time_runs_out(self): + running = ( + [_openvmm_run(7, status="in_progress")], + {7: _openvmm_gate_jobs(7, conclusion=None, status="in_progress")}, + ) + failed = ([_openvmm_run(7)], {7: _openvmm_gate_jobs(7, "failure")}) + nothing: _OpenVMMLook = ([], {}) + for look, wait_seconds, polls in ( + # A failed or missing job never passes, so the check doesn't wait. + (failed, 600, 1), + (nothing, 600, 1), + # Pending jobs wait for as long as another look fits in the time. + (running, 0, 1), + (running, openvmm_ci.POLL_SECONDS * 2.5, 3), + ): + api = _FakeOpenVMMCiApi(look) + sleep, clock, _ = self._clock() + with ( + self.subTest(look=look, wait_seconds=wait_seconds), + contextlib.redirect_stdout(io.StringIO()), + ): + verdicts = openvmm_ci.check_openvmm_ci( + api, + _OPENVMM_REVISION, + wait_seconds=wait_seconds, + sleep=sleep, + clock=clock, + ) + self.assertEqual(api.requests.count("runs"), polls) + self.assertNotIn( + "passed", + { + verdict.state + for verdict in verdicts + if verdict.job.startswith("run vmm-tests") + }, + ) + + # A job that failed fails the check whatever the jobs that still run + # do, so the check doesn't wait for them either. + failing: _OpenVMMLook = ( + [_openvmm_run(7, status="in_progress")], + { + 7: [ + {**job, "conclusion": "failure"} + if "unit tests [x64-windows]" in job["name"] + else job + for job in _openvmm_gate_jobs( + 7, conclusion=None, status="in_progress" + ) + ] + }, + ) + api = _FakeOpenVMMCiApi(failing) + sleep, clock, sleeps = self._clock() + with contextlib.redirect_stdout(io.StringIO()) as output: + verdicts = openvmm_ci.check_openvmm_ci( + api, _OPENVMM_REVISION, wait_seconds=600, sleep=sleep, clock=clock + ) + self.assertEqual(sleeps, []) + self.assertEqual(api.requests.count("runs"), 1) + self.assertNotIn(">> waiting", output.getvalue()) + self.assertEqual( + {verdict.job: verdict.state for verdict in verdicts}, + { + "unit tests [x64-linux]": "passed", + "unit tests [x64-windows]": "failed", + **{ + job: "pending" + for job in openvmm_ci.OPENVMM_CI_REQUIRED_JOBS + if job.startswith("run vmm-tests") + }, + }, + ) + + def test_check_looks_again_after_a_failed_look_while_it_may_wait(self): + passed: _OpenVMMLook = ([_openvmm_run(7)], {7: _openvmm_gate_jobs(7)}) + # GitHub's message may hold a newline, which would end a warning. + unavailable = common.ScriptError( + "GitHub GET /repos/microsoft/openvmm/actions/runs failed:\ntimed out" + ) + api = _FakeOpenVMMCiApi(unavailable, unavailable, passed) + sleep, clock, sleeps = self._clock() + with contextlib.redirect_stdout(io.StringIO()) as output: + verdicts = openvmm_ci.check_openvmm_ci( + api, _OPENVMM_REVISION, wait_seconds=3600, sleep=sleep, clock=clock + ) + self.assertEqual(sleeps, [openvmm_ci.POLL_SECONDS] * 2) + self.assertEqual({verdict.state for verdict in verdicts}, {"passed"}) + warnings = [ + line + for line in output.getvalue().splitlines() + if line.startswith("::warning::") + ] + self.assertEqual( + warnings, + [ + f"::warning::could not read OpenVMM's CI of {_OPENVMM_REVISION}, so " + "the check looks again: GitHub GET /repos/microsoft/openvmm/actions/" + "runs failed:%0Atimed out" + ] + * 2, + ) + # A failure that persists, or that leaves no time to look again, ends + # the check. + cases: tuple[tuple[tuple[_OpenVMMLook | Exception, ...], float, int], ...] = ( + ( + (unavailable,) * (openvmm_ci.FAILED_LOOK_RETRIES + 1) + (passed,), + 3600, + openvmm_ci.FAILED_LOOK_RETRIES + 1, + ), + ((unavailable, passed), 0, 1), + ) + for looks, wait_seconds, polls in cases: + api = _FakeOpenVMMCiApi(*looks) + sleep, clock, _ = self._clock() + with self.subTest(looks=len(looks), wait_seconds=wait_seconds): + with ( + contextlib.redirect_stdout(io.StringIO()), + self.assertRaises(common.ScriptError) as raised, + ): + openvmm_ci.check_openvmm_ci( + api, + _OPENVMM_REVISION, + wait_seconds=wait_seconds, + sleep=sleep, + clock=clock, + ) + self.assertIs(raised.exception, unavailable) + self.assertEqual(api.requests.count("runs"), polls) + + def test_command_reports_each_verdict_and_fails_unless_every_job_passed(self): + passed = ([_openvmm_run(7)], {7: _openvmm_gate_jobs(7)}) + failed = ([_openvmm_run(7)], {7: _openvmm_gate_jobs(7, "failure")}) + for look, succeeds in ((passed, True), (failed, False)): + with ( + self.subTest(succeeds=succeeds), + tempfile.TemporaryDirectory() as temporary, + ): + summary = Path(temporary) / "summary.md" + args = nvx.parse_args( + [ + "check-openvmm-ci", + "--revision", + _OPENVMM_REVISION, + "--summary", + os.fspath(summary), + ] + ) + api = _FakeOpenVMMCiApi(look) + with ( + patch.object( + openvmm_ci, "GitHubClient", return_value=api + ) as client, + patch.dict( + os.environ, + { + "GH_TOKEN": "token", + "GITHUB_API_URL": "https://api.example", + }, + ), + contextlib.redirect_stdout(io.StringIO()) as output, + ): + if succeeds: + args.handler(args) + else: + with self.assertRaisesRegex( + common.ScriptError, + f"OpenVMM's CI has not passed 4 required job\\(s\\) for " + f"{_OPENVMM_REVISION}", + ): + args.handler(args) + client.assert_called_once_with("token", "https://api.example") + lines = output.getvalue().splitlines() + errors = [line for line in lines if line.startswith("::error::")] + self.assertEqual(len(errors), 0 if succeeds else 4) + for line in errors: + self.assertRegex( + line, + r"^::error::OpenVMM's CI has not passed run vmm-tests " + r"\[[a-z0-9-]+\] \(failed: failure https://\S+\)$", + ) + self.assertIn( + "passed: unit tests [x64-linux]: " + "https://github.com/microsoft/openvmm/actions/runs/7/job/1", + lines, + ) + text = summary.read_text(encoding="utf-8") + self.assertTrue( + text.startswith( + f"### OpenVMM CI of `{_OPENVMM_REVISION}`\n" + "| Job | Result | Evidence |\n" + ) + ) + self.assertIn( + "| `run vmm-tests [x64-windows-amd]` | " + f"{'passed' if succeeds else 'failed'} |", + text, + ) + + def test_command_checks_the_pin_with_any_token(self): + args = nvx.parse_args(["check-openvmm-ci"]) + api = _FakeOpenVMMCiApi(([_openvmm_run(7)], {7: _openvmm_gate_jobs(7)})) + with ( + patch.object( + openvmm_ci, "git_output", return_value=_OPENVMM_REVISION + ) as git_output, + patch.object(openvmm_ci, "GitHubClient", return_value=api) as client, + patch.dict(os.environ, {"GH_TOKEN": "", "GITHUB_TOKEN": ""}), + contextlib.redirect_stdout(io.StringIO()), + ): + args.handler(args) + git_output.assert_called_once_with("rev-parse", ":openvmm") + # The repository is public, so the check reads it without a token too. + self.assertIsNone(client.call_args.args[0]) + for revision in ("3d7512af6", "main", _OPENVMM_REVISION.upper()): + with ( + self.subTest(revision=revision), + patch.object(openvmm_ci, "GitHubClient") as client, + self.assertRaisesRegex( + common.ScriptError, "is no full OpenVMM commit SHA" + ), + ): + openvmm_ci.command_check_openvmm_ci( + nvx.parse_args(["check-openvmm-ci", "--revision", revision]) + ) + client.assert_not_called() + + class CiConfigurationTests(unittest.TestCase): def test_ci_needs_reference_declared_jobs(self): workflow = ( From 51f8f3b18e8b239975293170e6a72c317a69f567 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 21:54:54 -0700 Subject: [PATCH 2/3] ci: check OpenVMM's own CI instead of running OpenVMM's tests Five jobs built and ran OpenVMM's unit tests and a selection of its VMM tests on the NVX runners: build-openvmm-vmm-tests-linux and -windows, openvmm-vmm-tests-linux and -windows, the latter in two shards, and openvmm-unit-tests. From 2026-10-21 to 2026-10-31 they ran in 673 runs and took about 620 runner-hours, 517 of them on the KVM, MSHV, and WHP runners. They failed 9 times, never for a defect in OpenVMM: 4 times for an NVX script error on a dispatched branch, once for a WHP runner that lost its connection, and 4 times for flaky tests that passed on the same OpenVMM revision in other runs. OpenVMM's own checkin gates run every test that those jobs ran, on OpenVMM's runners, on each pull request into feature/microvm/dev that is ready for review and on each push to it, but for two differences. OpenVMM's KVM job, on AMD hosts, skips the 4 emulated Intel VT-d tests that NVX's KVM job ran, which OpenVMM's MSHV and Windows Intel jobs run. And the gates' TTRPC microVM test boots OpenVMM's own kernel at one processor, where NVX's jobs booted NVX's kernel at one, two, four, and eight; NVX uses no TTRPC, and the NVX microVM tests boot and restore NVX's kernel at those processor counts on every backend. CI therefore runs none of OpenVMM's tests. The GitHub-hosted openvmm-ci job runs check-openvmm-ci on the pinned revision instead. It reads only the public repository's runs, so it needs neither an OpenVMM checkout nor the deploy key, and pull requests from forks run it too. A pull request, whose pin promotion usually starts before OpenVMM's gates finish with the pin, which takes them one and a half to two hours, and a manual run wait up to 300 minutes for the gates. A merge group, a dev push, and a scheduled run check a pin whose pull request passed the check, so they fail at once rather than hold up the merge groups and dev pushes that queue behind them. The change detector replaces run-openvmm-unit-tests and run-openvmm-vmm-tests with run-openvmm-ci-check, which a change to the OpenVMM pin, ci.yml, nvx.py, or the check's modules sets. Required status check expects openvmm-ci to succeed whenever the output is set and to be skipped otherwise, and the development release and the recording of the performance results require it to succeed, or to be skipped because the change selected no check. The guest artifacts are built only for the jobs that boot them. The VMM tests' build-only content, its cache, its sharding, and its test duration reports served only those jobs, so build-openvmm-tests, openvmm-test-cache-key, check-openvmm-tests, reuse-openvmm-tests, plan-openvmm-test-shards, test-openvmm's --content and --shard, and the partition of the WHP tests' shards go too. test-openvmm and test-openvmm-unit still build and run OpenVMM's tests on a developer's host. The removed jobs were the only ones that saved the OpenVMM input and Cargo caches that the OpenVMM binary builds restore, so build-openvmm now saves them when a dev push's build misses its binary cache. doc/ci.md replaces the sections on the VMM tests, their cache, and their shards with one on OpenVMM CI, and doc/usage.md, doc/contribute.md, doc/project-structure.md, the Copilot instructions, and the nvx-rs README follow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/build-openvmm/action.yml | 69 + .github/copilot-instructions.md | 5 +- .github/workflows/ci.yml | 929 +---- doc/ci.md | 616 +-- doc/contribute.md | 8 +- doc/project-structure.md | 2 +- doc/usage.md | 246 +- nvx-rs/README.md | 6 +- scripts/nvx.py | 146 +- scripts/nvx_tools/ci.py | 1029 +---- scripts/nvx_tools/openvmm_ci.py | 9 +- scripts/nvx_tools/openvmm_test_shards.json | 207 -- scripts/nvx_tools/openvmm_test_shards.py | 587 --- scripts/nvx_tools/shard_plans.py | 4 +- scripts/test_nvx_tools.py | 3917 +++----------------- 15 files changed, 769 insertions(+), 7011 deletions(-) delete mode 100644 scripts/nvx_tools/openvmm_test_shards.json delete mode 100644 scripts/nvx_tools/openvmm_test_shards.py diff --git a/.github/actions/build-openvmm/action.yml b/.github/actions/build-openvmm/action.yml index 7bf5381c0..a271a26bc 100644 --- a/.github/actions/build-openvmm/action.yml +++ b/.github/actions/build-openvmm/action.yml @@ -52,6 +52,7 @@ runs: echo "path=${cache_tools}" >> "${GITHUB_OUTPUT}" - name: Restore OpenVMM inputs on Linux + id: restore-openvmm-inputs-linux if: runner.os != 'Windows' uses: actions/cache/restore@v5 env: @@ -65,6 +66,7 @@ runs: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - name: Restore OpenVMM inputs on Windows + id: restore-openvmm-inputs-windows if: runner.os == 'Windows' uses: actions/cache/restore@v5 with: @@ -192,3 +194,70 @@ runs: build/openvmm.provenance.json ${{ inputs.binary-path }} key: openvmm-binary-v6-${{ inputs.backend }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('rust-toolchain.toml') }}-${{ steps.openvmm.outputs.sha }} + + # No other job builds OpenVMM, since OpenVMM's own CI runs its tests + # (doc/ci.md#openvmm-ci), so a dev push whose build missed the binary + # cache saves the inputs and the Cargo cache that the builds restore. + - name: Save OpenVMM inputs on Linux + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + runner.os != 'Windows' && + steps.restore-openvmm-binary-linux.outputs.cache-hit != 'true' && + steps.restore-openvmm-inputs-linux.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + openvmm/flowey-persist/flowey_lib_common__download_gh_release + openvmm/flowey-persist/flowey_lib_common__cache + key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-${{ steps.openvmm.outputs.sha }} + + - name: Save OpenVMM inputs on Windows + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + runner.os == 'Windows' && + steps.restore-openvmm-binary-windows.outputs.cache-hit != 'true' && + steps.restore-openvmm-inputs-windows.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + with: + path: | + openvmm/flowey-persist/flowey_lib_common__download_gh_release + openvmm/flowey-persist/flowey_lib_common__cache + key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-${{ steps.openvmm.outputs.sha }} + + - name: Save shared Cargo cache on Linux + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + runner.os != 'Windows' && + steps.restore-openvmm-binary-linux.outputs.cache-hit != 'true' && + steps.restore-cargo-linux.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + env: + PATH: ${{ steps.cache-tools-linux.outputs.path }} + with: + path: | + ${{ format('{0}/cargo-home/registry', runner.temp) }} + ${{ format('{0}/cargo-home/git', runner.temp) }} + key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('openvmm/Cargo.lock') }} + + - name: Save shared Cargo cache on Windows + if: >- + success() && + github.event_name == 'push' && + github.ref == 'refs/heads/dev' && + runner.os == 'Windows' && + steps.restore-openvmm-binary-windows.outputs.cache-hit != 'true' && + steps.restore-cargo-windows.outputs.cache-hit != 'true' + uses: actions/cache/save@v5 + with: + path: | + ${{ format('{0}/cargo-home/registry', runner.temp) }} + ${{ format('{0}/cargo-home/git', runner.temp) }} + key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('openvmm/Cargo.lock') }} diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 09b48a386..4ac376ee7 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -73,8 +73,9 @@ - The session ends 59 minutes after it starts, setup included, on 4 vCPUs. Run the narrowest affected check first. Rebuild OpenVMM, which takes about 5 minutes and 4 GB, only when the `openvmm` gitlink changed or the cached binary is missing, - and guest artifacts only when their inputs changed. Leave `test-openvmm-unit`, - `test-openvmm`, `build-guest --guest all`, `verify-guest-determinism`, and + and guest artifacts only when their inputs changed. Leave `test-openvmm-unit` + and `test-openvmm` to OpenVMM's own CI, whose results NVX CI checks for the + pin, and `build-guest --guest all`, `verify-guest-determinism`, and `benchmark` to CI. - `nvx-rs/target/release/nvx-rs run` and `nvx-rs/target/release/nvx-rs sandbox` run microVMs and sandboxes. Build `nvx-rs` there, in about a minute, with diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9e40d61d3..2c8973591 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -458,8 +458,7 @@ jobs: outputs: run-guest-artifacts: ${{ steps.changes.outputs.run-guest-artifacts }} run-openvmm-binaries: ${{ steps.changes.outputs.run-openvmm-binaries }} - run-openvmm-unit-tests: ${{ steps.changes.outputs.run-openvmm-unit-tests }} - run-openvmm-vmm-tests: ${{ steps.changes.outputs.run-openvmm-vmm-tests }} + run-openvmm-ci-check: ${{ steps.changes.outputs.run-openvmm-ci-check }} run-microvm-tests: ${{ steps.changes.outputs.run-microvm-tests }} run-benchmarks: ${{ steps.changes.outputs.run-benchmarks }} steps: @@ -491,8 +490,8 @@ jobs: set -euo pipefail outputs=( - run-guest-artifacts run-openvmm-binaries run-openvmm-unit-tests - run-openvmm-vmm-tests run-microvm-tests run-benchmarks + run-guest-artifacts run-openvmm-binaries run-openvmm-ci-check + run-microvm-tests run-benchmarks ) run_everything() { @@ -549,7 +548,7 @@ jobs: # The kernel and the Alpine initramfs, which every job that boots a guest # reads, build beside the Ubuntu and Azure Linux images, which only the # microVM test and package jobs read, so that a rebuild of those images - # delays neither the OpenVMM VMM tests nor the debug-kernel jobs. + # delays neither the default microVM suite nor the debug-kernel jobs. artifacts: name: Build kernel and Alpine artifacts needs: [openvmm-changes] @@ -910,13 +909,18 @@ jobs: secrets: OPENVMM_DEPLOY_KEY: ${{ secrets.OPENVMM_DEPLOY_KEY }} - # The KVM and MSHV VMM tests run the same Linux build, which boots no guest, - # so one job builds it beside the guest artifacts, on a KVM runner whose - # sccache holds it, and hands its test content to both backends' jobs. It - # restores the content instead while nothing that determines it changes - # (doc/ci.md#openvmm-vmm-test-cache). - build-openvmm-vmm-tests-linux: - name: OpenVMM vmm-tests build / Linux + # OpenVMM's own checkin gates test the revision that NVX pins, on each pull + # request into feature/microvm/dev and on each push to it, so CI runs none of + # OpenVMM's tests: this job checks that those gates passed the jobs that + # cover NVX's use of OpenVMM for the pinned revision (doc/ci.md#openvmm-ci). + # It reads only the public OpenVMM repository's runs, so it needs neither an + # OpenVMM checkout nor a deploy key, and a pull request from a fork runs it + # too. A pull request or a manual run waits while the gates still run, so + # that a pin promotion passes once OpenVMM's CI does. Every other event + # checks a pin that passed the check before it merged, so it fails at once + # rather than hold up the dev pushes and merge groups that queue behind it. + openvmm-ci: + name: OpenVMM CI of the pinned revision needs: [openvmm-changes] if: >- ${{ @@ -926,833 +930,34 @@ jobs: github.event_name != 'pull_request' || github.event.pull_request.draft == false ) && - ( - github.event_name != 'pull_request' || - github.event.pull_request.head.repo.full_name == github.repository - ) && ( needs.openvmm-changes.result != 'success' || - needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' + needs.openvmm-changes.outputs.run-openvmm-ci-check == 'true' ) }} - runs-on: [self-hosted, linux, kvm, virtual-machine, general] - timeout-minutes: 90 - steps: - - name: Checkout - uses: actions/checkout@v5 - with: - persist-credentials: false - - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: kvm - - # The key covers everything that determines the test content and reads - # the OpenVMM revision from the gitlink, so it needs no OpenVMM checkout. - - name: Prepare OpenVMM test cache key - id: openvmm-tests-cache - shell: bash - run: python3 scripts/nvx.py openvmm-test-cache-key --backend kvm --backend mshv - - - name: Prepare Linux cache tools - id: cache-tools-linux - shell: bash - run: | - set -euo pipefail - cache_tools="${RUNNER_TEMP}/nvx-gzip-cache-tools" - mkdir -p "${cache_tools}" - ln -sf "$(command -v tar)" "${cache_tools}/tar" - ln -sf "$(command -v gzip)" "${cache_tools}/gzip" - echo "path=${cache_tools}" >> "${GITHUB_OUTPUT}" - - # Each dev push that builds the tests saves them under the key, its run, - # and its attempt, and the restore takes the newest of those entries. - # Full runs build the tests from scratch, so that a breakage of the - # build that the key does not cover shows within a day. A failed - # restore only costs the build, like a miss. - - name: Restore OpenVMM tests on Linux - if: >- - github.event_name == 'pull_request' || - github.event_name == 'merge_group' || - github.event_name == 'push' - continue-on-error: true - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: ${{ runner.temp }}/openvmm-vmm-tests-linux - key: ${{ steps.openvmm-tests-cache.outputs.key }} - restore-keys: | - ${{ steps.openvmm-tests-cache.outputs.key }}- - - # The reuse keeps only the content that a fresh build makes, and removes - # a refused entry or a partial restore, so that the build starts empty. - - name: Reuse restored OpenVMM tests on Linux - id: reuse-openvmm-tests-linux - shell: bash - run: >- - python3 scripts/nvx.py reuse-openvmm-tests - --backend kvm --backend mshv - --content "${RUNNER_TEMP}/openvmm-vmm-tests-linux" - - - name: Checkout OpenVMM - id: openvmm - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: ./.github/actions/checkout-openvmm - with: - ssh-key: ${{ secrets.OPENVMM_DEPLOY_KEY }} - - - name: Set up retrying curl - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: ./.github/actions/setup-curl - - - name: Restore OpenVMM inputs on Linux - id: restore-openvmm-inputs-linux - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - restore-keys: | - openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Restore shared Cargo cache on Linux - id: restore-openvmm-cargo-linux - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - restore-keys: | - openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Prepare sccache - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: ./.github/actions/sccache - with: - operation: start - - - name: Build OpenVMM tests on Linux - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - shell: bash - env: - GH_TOKEN: ${{ github.token }} - run: >- - python3 scripts/nvx.py build-openvmm-tests - --backend kvm --backend mshv - --output "${RUNNER_TEMP}/openvmm-vmm-tests-linux" - - - name: Report sccache - if: always() && steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: ./.github/actions/sccache - with: - operation: stop - - # A build that a later reuse would refuse fails here rather than reach - # the cache or the test jobs. - - name: Check built OpenVMM tests on Linux - if: steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - shell: bash - run: >- - python3 scripts/nvx.py check-openvmm-tests - --backend kvm --backend mshv - --content "${RUNNER_TEMP}/openvmm-vmm-tests-linux" - - # include-hidden-files keeps the content's .config/nextest.toml. - - name: Upload OpenVMM tests - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-tests-linux - path: ${{ runner.temp }}/openvmm-vmm-tests-linux - include-hidden-files: true - if-no-files-found: error - overwrite: true - retention-days: 1 - - - name: Save OpenVMM tests on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-linux.outputs.reused != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: ${{ runner.temp }}/openvmm-vmm-tests-linux - key: ${{ steps.openvmm-tests-cache.outputs.key }}-${{ github.run_id }}-${{ github.run_attempt }} - - - name: Save OpenVMM inputs on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-linux.outputs.reused != 'true' && - steps.restore-openvmm-inputs-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - - - name: Save shared Cargo cache on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-linux.outputs.reused != 'true' && - steps.restore-openvmm-cargo-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - - openvmm-vmm-tests-linux: - name: OpenVMM vmm-tests / ${{ matrix.name }} - needs: [artifacts, build-openvmm-vmm-tests-linux, openvmm-changes] - if: >- - ${{ - always() && - !cancelled() && - ( - github.event_name != 'pull_request' || - github.event.pull_request.draft == false - ) && - ( - github.event_name != 'pull_request' || - github.event.pull_request.head.repo.full_name == github.repository - ) && - needs.artifacts.result == 'success' && - needs.build-openvmm-vmm-tests-linux.result == 'success' && - ( - needs.openvmm-changes.result != 'success' || - needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' - ) - }} - runs-on: ${{ fromJSON(matrix.runner) }} - timeout-minutes: 90 - strategy: - fail-fast: false - matrix: - include: - - name: Linux / KVM - backend: kvm - runner: '["self-hosted", "linux", "kvm", "virtual-machine", "general"]' - - name: Linux / MSHV - backend: mshv - runner: '["self-hosted", "linux", "mshv", "virtual-machine", "general"]' + runs-on: ubuntu-latest + timeout-minutes: 320 steps: - name: Checkout uses: actions/checkout@v5 with: persist-credentials: false - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: ${{ matrix.backend }} - - - name: Set up retrying curl - uses: ./.github/actions/setup-curl - - # The VMM tests boot only the kernel and the Alpine initramfs. - - name: Download guest kernel - uses: actions/download-artifact@v8 - with: - name: guest-kernel - path: build - - - name: Download Alpine initramfs - uses: actions/download-artifact@v8 - with: - name: guest-alpine - path: build - - - name: Download OpenVMM tests - uses: actions/download-artifact@v8 - with: - name: openvmm-vmm-tests-linux - path: ${{ runner.temp }}/openvmm-vmm-tests - - - name: Run OpenVMM tests on Linux + - name: Check OpenVMM's CI of the pinned revision shell: bash env: GH_TOKEN: ${{ github.token }} + WAIT_MINUTES: >- + ${{ + ( + github.event_name == 'pull_request' || + github.event_name == 'workflow_dispatch' + ) && '300' || '0' + }} run: >- - python3 scripts/nvx.py test-openvmm - --backend "${{ matrix.backend }}" - --content "${RUNNER_TEMP}/openvmm-vmm-tests" - - - name: Upload OpenVMM test diagnostics - if: failure() - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-tests-${{ runner.os }}-${{ matrix.backend }}-${{ github.run_id }}-${{ github.run_attempt }} - path: ${{ runner.temp }}/openvmm-vmm-tests/test_results - if-no-files-found: warn - retention-days: 7 - - # Each job and attempt keeps the duration of every test that it ran - # (doc/usage.md#test-duration-report), whether its tests pass or fail. - # A cancelled run usually ends before it writes them, and the step then - # only warns. - - name: Upload OpenVMM test durations - if: always() - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-test-durations-${{ runner.os }}-${{ matrix.backend }}-${{ github.run_id }}-${{ github.run_attempt }} - path: ${{ runner.temp }}/openvmm-vmm-tests/test_results/openvmm-test-durations.json - if-no-files-found: warn - retention-days: 30 - - # The WHP VMM tests boot no guest while they build either, so a WHP runner, - # whose sccache holds the Windows test build, builds them beside the guest - # artifacts and hands their test content to the shards of the WHP job. As on - # Linux, it restores the content instead while nothing that determines it - # changes. - build-openvmm-vmm-tests-windows: - name: OpenVMM vmm-tests build / Windows - needs: [openvmm-changes] - if: >- - ${{ - always() && - !cancelled() && - ( - github.event_name != 'pull_request' || - github.event.pull_request.draft == false - ) && - ( - github.event_name != 'pull_request' || - github.event.pull_request.head.repo.full_name == github.repository - ) && - ( - needs.openvmm-changes.result != 'success' || - needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' - ) - }} - runs-on: [self-hosted, windows, whp, virtual-machine, general] - timeout-minutes: 90 - steps: - - name: Checkout - uses: actions/checkout@v5 - with: - persist-credentials: false - - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: whp - - - name: Prepare OpenVMM test cache key - id: openvmm-tests-cache - shell: powershell - run: python scripts\nvx.py openvmm-test-cache-key --backend whp - - - name: Prepare Windows cache tools - shell: powershell - run: python scripts\nvx.py setup-cross-os-cache - - - name: Restore OpenVMM tests on Windows - if: >- - github.event_name == 'pull_request' || - github.event_name == 'merge_group' || - github.event_name == 'push' - continue-on-error: true - uses: actions/cache/restore@v5 - with: - path: ${{ runner.temp }}/openvmm-vmm-tests-windows - key: ${{ steps.openvmm-tests-cache.outputs.key }} - restore-keys: | - ${{ steps.openvmm-tests-cache.outputs.key }}- - - - name: Reuse restored OpenVMM tests on Windows - id: reuse-openvmm-tests-windows - shell: powershell - run: >- - python scripts\nvx.py reuse-openvmm-tests - --backend whp - --content "$env:RUNNER_TEMP\openvmm-vmm-tests-windows" - - - name: Checkout OpenVMM - id: openvmm - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: ./.github/actions/checkout-openvmm - with: - ssh-key: ${{ secrets.OPENVMM_DEPLOY_KEY }} - - - name: Set up retrying curl - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: ./.github/actions/setup-curl - - - name: Restore OpenVMM inputs on Windows - id: restore-openvmm-inputs-windows - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: actions/cache/restore@v5 - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - restore-keys: | - openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Restore shared Cargo cache on Windows - id: restore-openvmm-cargo-windows - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: actions/cache/restore@v5 - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - restore-keys: | - openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Prepare sccache - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: ./.github/actions/sccache - with: - operation: start - - - name: Build OpenVMM tests on Windows - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - shell: powershell - env: - GH_TOKEN: ${{ github.token }} - run: >- - python scripts\nvx.py build-openvmm-tests - --backend whp - --output "$env:RUNNER_TEMP\openvmm-vmm-tests-windows" - - - name: Report sccache - if: always() && steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: ./.github/actions/sccache - with: - operation: stop - - - name: Check built OpenVMM tests on Windows - if: steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - shell: powershell - run: >- - python scripts\nvx.py check-openvmm-tests - --backend whp - --content "$env:RUNNER_TEMP\openvmm-vmm-tests-windows" - - # include-hidden-files keeps the content's .config/nextest.toml. - - name: Upload OpenVMM tests - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-tests-windows - path: ${{ runner.temp }}/openvmm-vmm-tests-windows - include-hidden-files: true - if-no-files-found: error - overwrite: true - retention-days: 1 - - - name: Save OpenVMM tests on Windows - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-windows.outputs.reused != 'true' - uses: actions/cache/save@v5 - with: - path: ${{ runner.temp }}/openvmm-vmm-tests-windows - key: ${{ steps.openvmm-tests-cache.outputs.key }}-${{ github.run_id }}-${{ github.run_attempt }} - - - name: Save OpenVMM inputs on Windows - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-windows.outputs.reused != 'true' && - steps.restore-openvmm-inputs-windows.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ steps.openvmm.outputs.sha }} - - - name: Save shared Cargo cache on Windows - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - steps.reuse-openvmm-tests-windows.outputs.reused != 'true' && - steps.restore-openvmm-cargo-windows.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-vmm-tests-${{ hashFiles('openvmm/Cargo.lock') }} - - # The pinned OpenVMM's vmm-tests-run-target checks for Hyper-V on Windows - # whatever tests it runs, which the WHP runners' Network Service account - # cannot pass, so test-openvmm runs the content's nextest archive itself, as - # the runner would after that check (doc/ci.md#openvmm-vmm-tests). The tests - # run in shards that nextest runs in about equal times, one leg of this - # matrix each, as scripts/nvx_tools/openvmm_test_shards.json splits them - # (doc/ci.md#openvmm-vmm-test-shards). A leg fails unless that file puts - # every WHP test in exactly one shard and splits them into as many shards as - # the matrix runs. The job's result, which the required status check, the - # release, and the performance persistence read, succeeds only when every - # shard does. - openvmm-vmm-tests-windows: - name: OpenVMM vmm-tests / Windows / WHP / Shard ${{ matrix.shard }} of ${{ strategy.job-total }} - needs: [artifacts, build-openvmm-vmm-tests-windows, openvmm-changes] - if: >- - ${{ - always() && - !cancelled() && - ( - github.event_name != 'pull_request' || - github.event.pull_request.draft == false - ) && - ( - github.event_name != 'pull_request' || - github.event.pull_request.head.repo.full_name == github.repository - ) && - needs.artifacts.result == 'success' && - needs.build-openvmm-vmm-tests-windows.result == 'success' && - ( - needs.openvmm-changes.result != 'success' || - needs.openvmm-changes.outputs.run-openvmm-vmm-tests == 'true' - ) - }} - runs-on: [self-hosted, windows, whp, virtual-machine, general] - timeout-minutes: 90 - # A failed shard does not cancel the others, which keep their diagnostics - # and durations; GitHub expands the matrix after it evaluates the - # condition, so the shards run together or not at all. - strategy: - fail-fast: false - matrix: - shard: [1, 2] - steps: - - name: Checkout - uses: actions/checkout@v5 - with: - persist-credentials: false - - - name: Validate runner - uses: ./.github/actions/validate-runner - with: - backend: whp - - # The VMM tests boot only the kernel and the Alpine initramfs. - - name: Download guest kernel - uses: actions/download-artifact@v8 - with: - name: guest-kernel - path: build - - - name: Download Alpine initramfs - uses: actions/download-artifact@v8 - with: - name: guest-alpine - path: build - - - name: Download OpenVMM tests - uses: actions/download-artifact@v8 - with: - name: openvmm-vmm-tests-windows - path: ${{ runner.temp }}/openvmm-vmm-tests - - - name: Run OpenVMM tests on Windows - shell: powershell - run: >- - python scripts\nvx.py test-openvmm - --backend whp - --shard "${{ matrix.shard }}/${{ strategy.job-total }}" - --content "$env:RUNNER_TEMP\openvmm-vmm-tests" - - - name: Upload OpenVMM test diagnostics - if: failure() - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-tests-${{ runner.os }}-whp-shard-${{ matrix.shard }}-${{ github.run_id }}-${{ github.run_attempt }} - path: ${{ runner.temp }}/openvmm-vmm-tests/test_results - if-no-files-found: warn - retention-days: 7 - - - name: Upload OpenVMM test durations - if: always() - uses: actions/upload-artifact@v7 - with: - name: openvmm-vmm-test-durations-${{ runner.os }}-whp-shard-${{ matrix.shard }}-${{ github.run_id }}-${{ github.run_attempt }} - path: ${{ runner.temp }}/openvmm-vmm-tests/test_results/openvmm-test-durations.json - if-no-files-found: warn - retention-days: 30 - - openvmm-unit-tests: - name: OpenVMM unit tests / ${{ matrix.name }} - needs: [openvmm-changes] - if: >- - ${{ - always() && - !cancelled() && - ( - github.event_name != 'pull_request' || - github.event.pull_request.draft == false - ) && - ( - github.event_name != 'pull_request' || - github.event.pull_request.head.repo.full_name == github.repository - ) && - ( - needs.openvmm-changes.result != 'success' || - needs.openvmm-changes.outputs.run-openvmm-unit-tests == 'true' - ) - }} - runs-on: ${{ fromJSON(matrix.runner) }} - timeout-minutes: 90 - strategy: - fail-fast: false - matrix: - # The suites run once per OS and need no hypervisor: the tests that - # need /dev/kvm or /dev/mshv are ignored. The Linux leg runs on a - # GitHub-hosted runner, which keeps no sccache but still finishes - # before the WHP lanes, and leaves the MSHV runners to the jobs that - # need them. The Windows leg stays on a WHP runner, which takes half - # as long as a GitHub-hosted Windows runner (doc/ci.md). - include: - - name: Linux - runner: '["ubuntu-latest"]' - - name: Windows - backend: whp - runner: '["self-hosted", "windows", "whp", "virtual-machine", "general"]' - steps: - - name: Checkout - uses: actions/checkout@v5 - with: - persist-credentials: false - - - name: Validate runner - if: runner.environment == 'self-hosted' - uses: ./.github/actions/validate-runner - with: - backend: ${{ matrix.backend }} - - # A GitHub-hosted runner has rustup, but neither the Rust release nor - # the cargo-nextest that the runner setup scripts pin. As on the - # self-hosted runners, Cargo's home is in the job's temporary - # directory, where the shared Cargo cache restores the registry, and - # incremental compilation is off, since no target directory survives - # the job. - - name: Install the pinned Rust release and cargo-nextest - if: runner.environment == 'github-hosted' - shell: bash - run: | - set -euo pipefail - toolchain=$(sed -n 's/^channel = "\([^"]*\)".*/\1/p' rust-toolchain.toml) - nextest=$(sed -n 's/^cargo_nextest\.version=//p' scripts/setup/tool-versions.conf) - test -n "${toolchain}" && test -n "${nextest}" - rustup toolchain install "${toolchain}" --profile minimal --no-self-update - export CARGO_HOME="${RUNNER_TEMP}/cargo-home" - mkdir -p "${CARGO_HOME}/bin" - curl --fail --location --silent --show-error --retry 5 --retry-all-errors \ - "https://get.nexte.st/${nextest}/linux" | tar -xzf - -C "${CARGO_HOME}/bin" - export PATH="${CARGO_HOME}/bin:${PATH}" - { - echo "CARGO_HOME=${CARGO_HOME}" - echo "CARGO_INCREMENTAL=0" - } >> "${GITHUB_ENV}" - echo "${CARGO_HOME}/bin" >> "${GITHUB_PATH}" - rust_version=$(rustc --version) - echo "${rust_version}" - if [[ "${rust_version}" != "rustc ${toolchain} "* ]]; then - echo "::error::${rust_version} does not match Rust ${toolchain}, which rust-toolchain.toml pins" >&2 - exit 1 - fi - cargo --version - nextest_version=$(cargo nextest --version) - echo "${nextest_version}" - if [[ "${nextest_version}" != "cargo-nextest ${nextest} "* ]]; then - echo "::error::${nextest_version} does not match cargo-nextest ${nextest}, which scripts/setup/tool-versions.conf pins" >&2 - exit 1 - fi - echo "CPUs: $(nproc); $(grep -m 1 '^model name' /proc/cpuinfo)" - free -h - df -h . - - - name: Checkout OpenVMM - id: openvmm - uses: ./.github/actions/checkout-openvmm - with: - ssh-key: ${{ secrets.OPENVMM_DEPLOY_KEY }} - - - name: Set up retrying curl - uses: ./.github/actions/setup-curl - - - name: Prepare Windows cache tools - if: runner.os == 'Windows' - shell: powershell - run: python scripts\nvx.py setup-cross-os-cache - - - name: Prepare Linux cache tools - id: cache-tools-linux - if: runner.os != 'Windows' - shell: bash - run: | - set -euo pipefail - cache_tools="${RUNNER_TEMP}/nvx-gzip-cache-tools" - mkdir -p "${cache_tools}" - ln -sf "$(command -v tar)" "${cache_tools}/tar" - ln -sf "$(command -v gzip)" "${cache_tools}/gzip" - echo "path=${cache_tools}" >> "${GITHUB_OUTPUT}" - - - name: Restore OpenVMM inputs on Linux - id: restore-openvmm-inputs-linux - if: runner.os != 'Windows' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ steps.openvmm.outputs.sha }} - restore-keys: | - openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Restore OpenVMM inputs on Windows - id: restore-openvmm-inputs-windows - if: runner.os == 'Windows' - uses: actions/cache/restore@v5 - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ steps.openvmm.outputs.sha }} - restore-keys: | - openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Restore shared Cargo cache on Linux - id: restore-openvmm-cargo-linux - if: runner.os != 'Windows' - uses: actions/cache/restore@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ hashFiles('openvmm/Cargo.lock') }} - restore-keys: | - openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- - - - name: Restore shared Cargo cache on Windows - id: restore-openvmm-cargo-windows - if: runner.os == 'Windows' - uses: actions/cache/restore@v5 - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ hashFiles('openvmm/Cargo.lock') }} - restore-keys: | - openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}- - - # Only the self-hosted runners keep an sccache between jobs. - - name: Prepare sccache - if: runner.environment == 'self-hosted' - uses: ./.github/actions/sccache - with: - operation: start - - - name: Run OpenVMM unit tests on Linux - if: runner.os != 'Windows' - shell: bash - run: python3 scripts/nvx.py test-openvmm-unit - - - name: Run OpenVMM unit tests on Windows - if: runner.os == 'Windows' - shell: powershell - run: python scripts\nvx.py test-openvmm-unit - - - name: Report sccache - if: always() && runner.environment == 'self-hosted' - uses: ./.github/actions/sccache - with: - operation: stop - - - name: Save OpenVMM inputs on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os != 'Windows' && - steps.restore-openvmm-inputs-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ steps.openvmm.outputs.sha }} - - - name: Save OpenVMM inputs on Windows - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os == 'Windows' && - steps.restore-openvmm-inputs-windows.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - with: - path: | - openvmm/flowey-persist/flowey_lib_common__download_gh_release - openvmm/flowey-persist/flowey_lib_common__cache - key: openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ steps.openvmm.outputs.sha }} - - - name: Save shared Cargo cache on Linux - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os != 'Windows' && - steps.restore-openvmm-cargo-linux.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - env: - PATH: ${{ steps.cache-tools-linux.outputs.path }} - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ hashFiles('openvmm/Cargo.lock') }} - - - name: Save shared Cargo cache on Windows - if: >- - success() && - github.event_name == 'push' && - github.ref == 'refs/heads/dev' && - runner.os == 'Windows' && - steps.restore-openvmm-cargo-windows.outputs.cache-hit != 'true' - uses: actions/cache/save@v5 - with: - path: | - ${{ format('{0}/cargo-home/registry', runner.temp) }} - ${{ format('{0}/cargo-home/git', runner.temp) }} - key: openvmm-cargo-v1-${{ runner.os }}-${{ runner.arch }}-unit-tests-${{ hashFiles('openvmm/Cargo.lock') }} + python3 scripts/nvx.py check-openvmm-ci + --wait-minutes "${WAIT_MINUTES}" + --summary "${GITHUB_STEP_SUMMARY}" # Each backend runs its default suite in parallel shards of about equal # duration, one leg of this matrix each, as @@ -2326,11 +1531,7 @@ jobs: - nvx-cli-linux - nvx-cli-windows - openvmm-changes - - build-openvmm-vmm-tests-linux - - build-openvmm-vmm-tests-windows - - openvmm-vmm-tests-linux - - openvmm-vmm-tests-windows - - openvmm-unit-tests + - openvmm-ci - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp @@ -2346,8 +1547,8 @@ jobs: - package-kvm - package-mshv - package-whp - # The VMM-test jobs must all succeed, or all be skipped because the change - # selects no VMM tests: a failed or unexpectedly skipped one blocks this. + # The OpenVMM CI check must succeed, or be skipped because the change + # selects no check: a failed or unexpectedly skipped one blocks this. if: >- ${{ always() && @@ -2362,25 +1563,13 @@ jobs: needs.nvx-cli-linux.result == 'success' && needs.nvx-cli-windows.result == 'success' && ( - ( - needs.build-openvmm-vmm-tests-linux.result == 'success' && - needs.build-openvmm-vmm-tests-windows.result == 'success' && - needs.openvmm-vmm-tests-linux.result == 'success' && - needs.openvmm-vmm-tests-windows.result == 'success' - ) || + needs.openvmm-ci.result == 'success' || ( needs.openvmm-changes.result == 'success' && - needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' && - needs.build-openvmm-vmm-tests-linux.result == 'skipped' && - needs.build-openvmm-vmm-tests-windows.result == 'skipped' && - needs.openvmm-vmm-tests-linux.result == 'skipped' && - needs.openvmm-vmm-tests-windows.result == 'skipped' + needs.openvmm-changes.outputs.run-openvmm-ci-check != 'true' && + needs.openvmm-ci.result == 'skipped' ) ) && - ( - needs.openvmm-unit-tests.result == 'success' || - needs.openvmm-unit-tests.result == 'skipped' - ) && ( needs.nvx-microvm-tests-kvm.result == 'success' || needs.nvx-microvm-tests-kvm.result == 'skipped' @@ -2490,11 +1679,7 @@ jobs: - build-openvmm-linux-gnu - build-openvmm-linux-musl - build-openvmm-windows-msvc - - build-openvmm-vmm-tests-linux - - build-openvmm-vmm-tests-windows - - openvmm-vmm-tests-linux - - openvmm-vmm-tests-windows - - openvmm-unit-tests + - openvmm-ci - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp @@ -2531,8 +1716,7 @@ jobs: }} RUN_GUEST_ARTIFACTS: ${{ needs.openvmm-changes.outputs.run-guest-artifacts }} RUN_OPENVMM_BINARIES: ${{ needs.openvmm-changes.outputs.run-openvmm-binaries }} - RUN_OPENVMM_UNIT_TESTS: ${{ needs.openvmm-changes.outputs.run-openvmm-unit-tests }} - RUN_OPENVMM_VMM_TESTS: ${{ needs.openvmm-changes.outputs.run-openvmm-vmm-tests }} + RUN_OPENVMM_CI_CHECK: ${{ needs.openvmm-changes.outputs.run-openvmm-ci-check }} RUN_MICROVM_TESTS: ${{ needs.openvmm-changes.outputs.run-microvm-tests }} RUN_BENCHMARKS: ${{ needs.openvmm-changes.outputs.run-benchmarks }} QUALITY_RESULT: ${{ needs.quality.result }} @@ -2549,11 +1733,7 @@ jobs: BUILD_LINUX_GNU_RESULT: ${{ needs.build-openvmm-linux-gnu.result }} BUILD_LINUX_MUSL_RESULT: ${{ needs.build-openvmm-linux-musl.result }} BUILD_WINDOWS_MSVC_RESULT: ${{ needs.build-openvmm-windows-msvc.result }} - BUILD_VMM_TESTS_LINUX_RESULT: ${{ needs.build-openvmm-vmm-tests-linux.result }} - BUILD_VMM_TESTS_WINDOWS_RESULT: ${{ needs.build-openvmm-vmm-tests-windows.result }} - VMM_TESTS_LINUX_RESULT: ${{ needs.openvmm-vmm-tests-linux.result }} - VMM_TESTS_WINDOWS_RESULT: ${{ needs.openvmm-vmm-tests-windows.result }} - UNIT_TESTS_RESULT: ${{ needs.openvmm-unit-tests.result }} + OPENVMM_CI_RESULT: ${{ needs.openvmm-ci.result }} MICROVM_KVM_RESULT: ${{ needs.nvx-microvm-tests-kvm.result }} MICROVM_MSHV_RESULT: ${{ needs.nvx-microvm-tests-mshv.result }} MICROVM_WHP_RESULT: ${{ needs.nvx-microvm-tests-whp.result }} @@ -2576,8 +1756,7 @@ jobs: --same-repository "${SAME_REPOSITORY}" --run-guest-artifacts "${RUN_GUEST_ARTIFACTS}" --run-openvmm-binaries "${RUN_OPENVMM_BINARIES}" - --run-openvmm-unit-tests "${RUN_OPENVMM_UNIT_TESTS}" - --run-openvmm-vmm-tests "${RUN_OPENVMM_VMM_TESTS}" + --run-openvmm-ci-check "${RUN_OPENVMM_CI_CHECK}" --run-microvm-tests "${RUN_MICROVM_TESTS}" --run-benchmarks "${RUN_BENCHMARKS}" @@ -2594,11 +1773,7 @@ jobs: - nvx-cli-linux - nvx-cli-windows - openvmm-changes - - build-openvmm-vmm-tests-linux - - build-openvmm-vmm-tests-windows - - openvmm-vmm-tests-linux - - openvmm-vmm-tests-windows - - openvmm-unit-tests + - openvmm-ci - nvx-microvm-tests-kvm - nvx-microvm-tests-mshv - nvx-microvm-tests-whp @@ -2614,8 +1789,8 @@ jobs: - package-kvm - package-mshv - package-whp - # The VMM-test jobs must all succeed, or all be skipped because the change - # selects no VMM tests: a failed or unexpectedly skipped one blocks this. + # The OpenVMM CI check must succeed, or be skipped because the change + # selects no check: a failed or unexpectedly skipped one blocks this. if: >- ${{ always() && @@ -2628,25 +1803,13 @@ jobs: needs.nvx-cli-linux.result == 'success' && needs.nvx-cli-windows.result == 'success' && ( - ( - needs.build-openvmm-vmm-tests-linux.result == 'success' && - needs.build-openvmm-vmm-tests-windows.result == 'success' && - needs.openvmm-vmm-tests-linux.result == 'success' && - needs.openvmm-vmm-tests-windows.result == 'success' - ) || + needs.openvmm-ci.result == 'success' || ( needs.openvmm-changes.result == 'success' && - needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' && - needs.build-openvmm-vmm-tests-linux.result == 'skipped' && - needs.build-openvmm-vmm-tests-windows.result == 'skipped' && - needs.openvmm-vmm-tests-linux.result == 'skipped' && - needs.openvmm-vmm-tests-windows.result == 'skipped' + needs.openvmm-changes.outputs.run-openvmm-ci-check != 'true' && + needs.openvmm-ci.result == 'skipped' ) ) && - ( - needs.openvmm-unit-tests.result == 'success' || - needs.openvmm-unit-tests.result == 'skipped' - ) && ( needs.nvx-microvm-tests-kvm.result == 'success' || needs.nvx-microvm-tests-kvm.result == 'skipped' diff --git a/doc/ci.md b/doc/ci.md index 1d48df7d9..b4afdf1e1 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -14,31 +14,12 @@ its performance series on the same VM (see [Benchmarks](benchmarks.md#ci-collection)). Before it measures, each platform job also checks that its runner is still the host that the series' history comes from (see -[Expected benchmark hosts](benchmarks.md#expected-benchmark-hosts)). The OpenVMM VMM tests download only the NVX -Linux-direct kernel and Alpine initramfs (see -[Guest artifact handoff](#guest-artifact-handoff)) and use them to -exercise OpenVMM's Linux MP-table lifecycle, TTRPC, and snapshot contracts on -each backend. `openvmm-unit-tests` runs the OpenVMM unit and documentation -tests once per operating system, on a GitHub-hosted Ubuntu runner and a WHP -runner. The tests that need `/dev/kvm` or `/dev/mshv` are ignored, so the KVM -and MSHV runners used to run the same 4,820 tests, and the Linux leg needs no -hypervisor runner; see -[Job graph and runner capacity](#job-graph-and-runner-capacity). -Failed VMM-test jobs, which [OpenVMM VMM tests](#openvmm-vmm-tests) describes, -upload Petri's `test_results` directory, -including guest and VMM logs, screenshots, and watchdog inspection data. -These seven-day artifacts are named -`openvmm-vmm-tests--[-shard-]--`, where -each shard of the WHP tests adds `-shard-` (see -[OpenVMM VMM test shards](#openvmm-vmm-test-shards)), so a successful rerun -does not replace the failed attempt's diagnostics. Each job collects them from -`/openvmm-vmm-tests/test_results`, in the test content that it -runs. Every VMM-test job also uploads the -[test duration report](usage.md#test-duration-report) of its run, whether its -tests pass or fail, as the 30-day -`openvmm-vmm-test-durations--[-shard-]--` -artifact. A job that is cancelled usually ends before its run writes one, so -its upload step only warns. +[Expected benchmark hosts](benchmarks.md#expected-benchmark-hosts)). CI runs +none of OpenVMM's own tests: OpenVMM's checkin gates test the pinned revision +of `feature/microvm/dev`, on each pull request into the branch and on each push +to it, on OpenVMM's own runners, and the GitHub-hosted `openvmm-ci` job checks +that they passed the jobs that cover NVX's use of OpenVMM; see +[OpenVMM CI](#openvmm-ci). Two jobs per backend exercise Linux, SMP, virtio, sandbox, and snapshot behavior through the public OpenVMM CLI, on the NVX Linux kernel plus the selected Alpine, Ubuntu, or Azure Linux initramfs. Both run on the backend's @@ -343,11 +324,9 @@ ABI, never-restored guests on such an MSHV runner VM, an Ice Lake 8370C, hit cross-vCPU TSC warps when an idle host CPU woke (#211, #265), which is why the probe schedules include idle gaps. Under the time ABI, its guests' warp probes measure at most 36 ns, so `validate-runner` only notes a missing -`nonstop_tsc`, and the Linux runner setup script only warns. The OpenVMM -vmm-tests run no warp probe: they boot OpenVMM's own test guests, whose -verdicts don't depend on the skew bound. Runner labels do not encode the -generation; per-PR CI captures and restores on one runner, so generations -never mix. +`nonstop_tsc`, and the Linux runner setup script only warns. Runner labels do +not encode the generation; per-PR CI captures and restores on one runner, so +generations never mix. The warp probe replaced the `restore-tsc-sync` scenario, its test-only `clearcpuid=tsc_adjust` kernel option, the guest's scan of the kernel log for @@ -491,7 +470,8 @@ run and for a managed sandbox whose configuration must persist the mode. The `fi `test-microvm` likewise requires guest root and a foreign guest identity to fail with `EPERM` on the runners, and a root-owned export to be rejected before boot; on Windows, `--mount-owner caller` must be rejected before boot. -The OpenVMM unit tests check the host identity that each request runs as, +OpenVMM's unit tests, which its own CI runs (see [OpenVMM CI](#openvmm-ci)), +check the host identity that each request runs as, including guest root squashed to an export owner other than OpenVMM. Where OpenVMM can drop its supplementary groups, the scenario instead requires files that guest root creates to belong to the share owner rather than to @@ -619,15 +599,9 @@ when the `nvx-rs` build of its operating system ends. When `distro-artifacts`, and `debug-kernel` restore or build the guest artifacts on GitHub-hosted runners (see [Guest artifact handoff](#guest-artifact-handoff)), the three OpenVMM binary -producers restore or build OpenVMM on a KVM, an MSHV, and a WHP runner, -`build-openvmm-vmm-tests-linux` and `build-openvmm-vmm-tests-windows` restore -or build the Linux VMM tests on a KVM runner and the WHP ones on a WHP runner -(see [OpenVMM VMM test cache](#openvmm-vmm-test-cache)), and -`openvmm-unit-tests` starts on a GitHub-hosted and a WHP runner. -`openvmm-vmm-tests-linux` waits for `artifacts`, which builds the kernel and the -Alpine initramfs, and the Linux VMM-test build, and `openvmm-vmm-tests-windows`, -whose legs run the [shards](#openvmm-vmm-test-shards) of the WHP tests, for -`artifacts` and the Windows VMM-test build. Each microVM test job waits for +producers restore or build OpenVMM on a KVM, an MSHV, and a WHP runner, and +`openvmm-ci` checks OpenVMM's own CI of the pinned revision on a GitHub-hosted +runner (see [OpenVMM CI](#openvmm-ci)). Each microVM test job waits for `artifacts`, its OpenVMM producer, and the `nvx-rs` build of its operating system, and each platform job for `artifacts` and its OpenVMM producer alone. Beside each platform job, a package job on a GitHub-hosted runner of the @@ -703,8 +677,8 @@ images and `nvx-rs` only to package the release after they measured. The package jobs now package it beside them on GitHub-hosted runners, and the platform jobs wait only for the kernel and the Alpine initramfs, which they boot, and their OpenVMM producer (#982). In the 89 successful first-attempt runs of the same -window that restored the OpenVMM VMM tests from the -[cache](#openvmm-vmm-test-cache), `Platform / Windows / WHP / Virtual machine` +window that restored the OpenVMM VMM tests from their cache, +`Platform / Windows / WHP / Virtual machine` waited in 34 for `distro-artifacts`, whose cold builds of the Ubuntu and Azure Linux images ended 5.5 to 7.4 minutes into the run, and it finished last in 61. Packaging and the images' downloads also took 1.7 of the median 5.4 @@ -730,48 +704,18 @@ full pull-request runs. Replaying the observed arrivals and durations with dedicated benchmark runners reduced WHP platform wait to about p90 3.6 and at most 8.4 minutes. -The WHP jobs are the longest: the Windows VMM-test build and the OpenVMM unit -tests compile OpenVMM on Windows for about 15 and 13 minutes, so overlapping -runs queue on WHP first. Dedicating the benchmark runner takes it out of its -pool's general capacity. In the replay, which left four general WHP runners, +The WHP jobs were the longest: until CI stopped running OpenVMM's tests (see +[OpenVMM CI](#openvmm-ci)), the Windows build of OpenVMM's VMM tests and its +unit tests compiled OpenVMM on Windows for about 15 and 13 minutes, so +overlapping runs queued on WHP first. Dedicating the benchmark runner takes it +out of its pool's general capacity. In the replay, which left four general WHP +runners, the WHP non-benchmark p90 wait rose from about three to five minutes, while the KVM and MSHV non-benchmark waits stayed near zero. The WHP pool has since grown to six general runners, more than the five that served these jobs before the split. A change that adds a job or a shard needs general runners to run it in parallel; otherwise it only queues. -The OpenVMM unit tests need no hypervisor. In October 2026 (#697), on -GitHub-hosted runners, which have four vCPUs and keep no sccache, the Linux -suite took 12 to 18 minutes from scratch on four runners, against 6.5 on an -MSHV runner, and the Windows suite took 32 minutes from scratch and 21 with a -warm sccache, against 15 on a WHP runner. Every hosted run passed the same -5,294 Linux or 4,933 Windows tests as the self-hosted runners. The Linux leg -therefore runs on `ubuntu-latest`: even at its slowest, it would have finished -before the rest of the run in each of the 208 successful runs that ran the -suite from October 8 to 21, and it leaves about 900 MSHV runner-minutes a week, -about 15% of that pool's busy time, to the jobs that need a hypervisor. It -installs the pinned Rust release and cargo-nextest, keeps Cargo's home in the -job's temporary directory, as the self-hosted runners do, and skips -`validate-runner` and sccache, which only they have. The Windows leg stays on -WHP: a GitHub-hosted runner would have delayed 69% of the runs that ran it, and -still 13% with a warm sccache, which would take over 2 GiB of the repository's -10 GB Actions cache, already full. Larger GitHub-hosted runners, which public -repositories pay for, or a self-hosted Windows runner without a hypervisor -would cost about as much as another general WHP runner. - -Since the `nvx-rs` handoff and the package jobs shorten the rest of the runs -that restore the VMM tests from the [cache](#openvmm-vmm-test-cache), the -Linux leg, which took a median 17.0 minutes from October 23 to 28, 2026, ends -last in most of those that run the unit tests, 26 of the 89 such runs of that -window (#982). On an MSHV runner with a warm sccache, it would take about 6.5 -minutes, but the replay below estimates that those runs would then end with -the Windows leg, which took 14.2 minutes on a WHP runner, at a median of 15.9 -minutes instead of 18.1, while its 6.5 minutes on MSHV's general runners, -which also run the pull requests' debug kernel, would make 177 instead of 149 -of the 287 runs that used them wait 2 minutes or more for one. The Linux leg -therefore stays on `ubuntu-latest`, and the OpenVMM unit tests remain the -bound of the runs that restore the VMM tests and run them. - The WHP microVM tests bounded every run that skipped both OpenVMM suites, so each backend runs them in two jobs, and the first of them in shards (see [MicroVM test shards](#microvm-test-shards)). In reference run @@ -784,29 +728,12 @@ general runner per backend for about three to five minutes. The default suite waits only for `artifacts`, so a cold build of the Ubuntu or Azure Linux images, about 4 minutes, delays only the guest and sandbox job, which takes about as long as a shard of the default suite. A `dev` push, full run, or -merge group that runs the VMM tests then needs five of KVM's four general -runners at once: one for the Linux VMM-test build and then its KVM leg, and -four for the two shards, the guest and sandbox job, and the debug-kernel job, -so the last of them to start waits for the first to end, about three minutes -later; one that skips the VMM tests fills the four. A pull request, which runs -the debug kernel on MSHV, needs at most the four. MSHV's four hold the two -shards, the guest and sandbox job, and the MSHV VMM-test leg, which starts -once the Linux VMM-test build ends, before the guest and sandbox job ends in -nearly every run that restores the VMM tests from the -[cache](#openvmm-vmm-test-cache). A pull request, `dev` push, or full run that -runs the VMM tests and the debug kernel on MSHV then needs five of them at -once too; one that skips the VMM tests fills the four. On WHP, a run -needs at most seven of the six general runners at once: one for the OpenVMM unit -tests, up to four for the microVM jobs, and one for the Windows VMM-test build -and then two for the [shards of the WHP VMM tests](#openvmm-vmm-test-shards). -The build ended after the last WHP microVM job in 10 of the 13 runs since -[#783](https://github.com/microsoft/nvx/pull/783) that ran both, so the shards -usually take runners that the microVM jobs freed, and a run then uses at most -six. A run that restores the Windows build from the -[cache](#openvmm-vmm-test-cache) instead ends it within about a minute, so its -shards run beside the microVM jobs. A `dev` push can then need all seven, so -the last job to start waits for one to end; a pull request or merge group needs -at most six. When a pool has fewer +merge group then fills KVM's four general runners with the two shards, the +guest and sandbox job, and the debug-kernel job, and a pull request, which +runs the debug kernel on MSHV, needs three of them. MSHV's four hold the two +shards, the guest and sandbox job, and, for a pull request, `dev` push, or +full run, the debug-kernel job. On WHP, a run needs at most four of the six +general runners at once, for the microVM jobs. When a pool has fewer free general runners than a backend's microVM jobs, the jobs that start last wait for the first to end, so the backend's microVM tests take up to about twice as long as one of its jobs, still less than the unsharded suite. @@ -835,7 +762,10 @@ would have raised that pool's waits to 131 runs, at a p90 of 10.5 minutes instead of 8.8, and the median of the runs that build the VMM tests and run the unit tests to 24.4 minutes, against 23.7 with the jobs on MSHV. A fifth general KVM runner would then lower KVM's waits to 79 runs and a p90 of 3.9 -minutes, so it is the next step if KVM queueing still bounds the runs. +minutes, so it is the next step if KVM queueing still bounds the runs. Since +CI stopped running OpenVMM's tests (see [OpenVMM CI](#openvmm-ci)), the Linux +VMM-test build and the KVM and MSHV VMM-test legs no longer take those pools' +general runners either. With the `nvx-rs` handoff, the package jobs, and the pull requests' debug kernel on MSHV, the replay, with the six general WHP runners of today, @@ -848,8 +778,10 @@ in nearly all of the former. The runs that restored the VMM tests from the cache would have taken a median of 14.2 minutes without queueing, against 16.5, and 13.8 when they ran no OpenVMM unit tests, ending with the KVM VMM-test leg, the WHP platform job, or, after a cold build of the Ubuntu and -Azure Linux images, a guest and sandbox job. Those that ran the unit tests end -with them, as described above, at a median of 18.1 minutes. +Azure Linux images, a guest and sandbox job. Those that ran the unit tests +would have ended with their Linux leg, which took a median 17.0 minutes on +`ubuntu-latest`, at a median of 18.1 minutes. Since CI stopped running +OpenVMM's tests, every run is of the first two kinds. ## MicroVM test shards @@ -917,8 +849,7 @@ about as long as the guest and sandbox jobs beside them, which took 3.7 to shard would take about 3.2, 3.0, and 4.0 minutes. On WHP it would end before the guest and sandbox job, so it would not shorten the backend's tests, and on KVM and MSHV, whose four general runners the two shards, the guest and sandbox -job, and the VMM-test leg or the debug-kernel job fill, it would mostly wait -for a runner. +job, and the debug-kernel job fill, it would mostly wait for a runner. A sharded run fails before it runs a scenario unless the file puts every default scenario in exactly one shard of every backend, with its duration, @@ -970,16 +901,14 @@ components that its steps read, and waits only for the jobs that upload them: | Artifact | Files in `build/` | Producer | Jobs that download it | | --- | --- | --- | --- | -| `guest-kernel` | `vmlinux`, `vmlinux.config`, `vmlinux.provenance.json` | `artifacts` | VMM tests, microVM tests, debug-kernel jobs, platform jobs, package jobs | -| `guest-alpine` | `initramfs.cpio.gz`, its `.packages.json`, `initramfs.provenance.json` | `artifacts` | VMM tests, microVM tests, debug-kernel jobs, platform jobs, package jobs | +| `guest-kernel` | `vmlinux`, `vmlinux.config`, `vmlinux.provenance.json` | `artifacts` | microVM tests, debug-kernel jobs, platform jobs, package jobs | +| `guest-alpine` | `initramfs.cpio.gz`, its `.packages.json`, `initramfs.provenance.json` | `artifacts` | microVM tests, debug-kernel jobs, platform jobs, package jobs | | `guest-ubuntu` | `initramfs-ubuntu.cpio.gz` and its `.packages.json` | `distro-artifacts` | microVM guest and sandbox jobs, package jobs | | `guest-ubuntu-distro` | `ubuntu-distro.erofs` and its `.manifest.json`, and `ubuntu-smoke-scratch.ext4`, the sandbox scratch template | `distro-artifacts` | microVM guest and sandbox jobs, package jobs | | `guest-azurelinux` | `initramfs-azurelinux.cpio.gz` and its `.packages.json` | `distro-artifacts` | microVM guest and sandbox jobs, package jobs | | `guest-debug-kernel` | `vmlinux-debug`, `vmlinux-debug.config`, `vmlinux-debug.provenance.json` | `debug-kernel` | debug-kernel jobs | -The OpenVMM VMM tests boot only the kernel and the Alpine initramfs, and their -Linux build boots no guest, so it downloads no component. The -debug-kernel jobs run the Alpine scenarios on the debug kernel and qualify +The debug-kernel jobs run the Alpine scenarios on the debug kernel and qualify their runner, through the doctor's preflight (H3), with the kernel and the Alpine initramfs. The microVM default suites boot only those two, and the microVM guest and sandbox jobs boot every guest and run sandboxes from the @@ -1014,8 +943,8 @@ saves those that miss, as one job did before. The Ubuntu determinism check, the provenance that each build records, and the cache keys are unchanged. A cold `artifacts` job, with a warm kernel, takes about 1.3 minutes, so a miss of the Ubuntu or Azure Linux images, whose build and determinism check take -about 4 minutes, no longer delays the VMM tests, which are the longest job of -most runs that select them. `distro-artifacts` builds the base and guest-binary +about 4 minutes, delays neither the default microVM suites nor the debug-kernel +jobs. `distro-artifacts` builds the base and guest-binary stages of the guest Docker build again, about 45 seconds of GitHub-hosted time, and its consumers wait for it about as long as they waited for the single guest artifact job before. From October 19 to 21, 2026, before the guest image keys @@ -1202,13 +1131,12 @@ checks, but the rehearsal of the [guest artifact reuse](#guest-artifact-reuse), which the pull request's run made, and with the debug kernel on KVM instead of MSHV, so that each change runs it on both before it merges: each suite runs only when one of its inputs changed, the platform and package jobs and the -performance regression gate after them included, and the OpenVMM VMM tests -are restored from the [cache](#openvmm-vmm-test-cache) that `dev` pushes -save. The two `nvx-rs` build jobs run too: each hands its executable to the -microVM and package jobs of its operating system and to the CLI job that tests -it. No hypervisor job waits for the CLI tests, and the platform jobs wait for -neither `nvx-rs` nor the distro artifacts that only the packages ship. A group -publishes no development release and persists no performance results, +performance regression gate after them included. The two `nvx-rs` build jobs +run too: each hands its executable to the microVM and package jobs of its +operating system and to the CLI job that tests it. No hypervisor job waits for +the CLI tests, and the platform jobs wait for neither `nvx-rs` nor the distro +artifacts that only the packages ship. A group publishes no development +release and persists no performance results, which the `dev` push of the merged commit does, and saves no cache: GitHub scopes a cache entry to the ref that saved it, and no other run can restore an entry of a group's branch, which GitHub deletes as soon as it merges or drops @@ -1216,8 +1144,7 @@ the group. A group's run can restore the entries of `dev` alone, not those of its pull request's runs, so, like the `dev` push that merges the pull request, it installs the guest artifacts that the pull request's run built instead of building them again, when they check out (see -[Guest artifact reuse](#guest-artifact-reuse)); a pull request that changes -the key of the OpenVMM VMM tests, though, builds them in its group too. Each +[Guest artifact reuse](#guest-artifact-reuse)). Each group's run has a concurrency group of its own, keyed by the run rather than the group's ref, so it neither waits for nor cancels another: a pull request that leaves the queue and returns to it on the same base gets its earlier @@ -1484,291 +1411,73 @@ from its main thread. The `validate-runner` action of the runner's next job ends the OpenVMM processes and removes the scratch directories that still escape all of this. -## OpenVMM VMM tests - -OpenVMM's `vmm-tests-run` builds the tests that a filter selects and then runs -them. In build-only mode, it instead writes everything that the tests need to -a test content directory, from which `vmm-tests-run-target` runs a filter's -tests on another host: the tests' nextest archive, OpenVMM, the guest agents, -the test firmware and kernels, and the `flowey_hvlite` that runs them. The -content is relocatable, because `vmm-tests-run-target` remaps the archive's -workspace to the directory, which also serves as the repository root whose -nextest configuration the tests read. The KVM and MSHV tests run the same -`x86_64-unknown-linux-gnu` build: their jobs resolved the same artifacts, and -the GNU OpenVMM that a KVM runner builds already runs the MSHV microVM tests -on the same Azure Linux 3 release. CI therefore builds the tests once per -operating system and hands each build to the jobs of its backends: - -- `build-openvmm-vmm-tests-linux` runs - [`build-openvmm-tests`](usage.md#build-openvmm-tests) for KVM and MSHV, - which builds the union of their filters, on a general KVM runner, whose - sccache holds the Linux test build. It restores, and on `dev` pushes saves, - the `vmm-tests` OpenVMM input and Cargo caches, and uploads the content as - the one-day `openvmm-vmm-tests-linux` artifact, which keeps the content's - hidden `.config` and replaces an earlier attempt's content. It needs no - guest artifacts, so it starts with the OpenVMM binary producers instead of - waiting for `artifacts`. While nothing that determines the content changes, - it restores the content from the - [OpenVMM VMM test cache](#openvmm-vmm-test-cache) instead of building it. -- `openvmm-vmm-tests-linux` runs a KVM and an MSHV leg once the build and - `artifacts` succeed. Each downloads the content, the kernel, and the Alpine - initramfs, and runs [`test-openvmm`](usage.md#test-openvmm) on the content, - which runs its backend's own filter with the content's - `vmm-tests-run-target`. The legs check out no OpenVMM, restore no cache, and - compile nothing, and a rerun of a failed leg downloads the build of the - earlier attempt instead of rebuilding. -- `build-openvmm-vmm-tests-windows` builds the WHP tests the same way on a - general WHP runner, whose sccache holds the Windows test build, with the - Windows `vmm-tests` caches, and uploads the content as the one-day - `openvmm-vmm-tests-windows` artifact. It also starts with the OpenVMM binary - producers, and also restores the content from the cache instead of - building it while nothing that determines it changes. -- `openvmm-vmm-tests-windows` runs the WHP tests, in the - [shards](#openvmm-vmm-test-shards) of its matrix, once that build and - `artifacts` succeed. Like a Linux leg, each shard downloads the content, the - kernel, and the Alpine initramfs, runs `test-openvmm` on the content, and - checks out, restores, and compiles nothing. On Windows, though, the pinned - `vmm-tests-run-target` checks for Hyper-V whatever tests it runs: it - requires `Get-VM` to succeed and the `AllowFirmwareLoadFromFile` and - `EnableAdditionalComPorts` registry values to be set, or administrator - rights to set them. The WHP runners run jobs as Network Service, which is - not a Hyper-V administrator, on hosts whose setup sets neither value. No WHP - test needs Hyper-V, so `vmm-tests-run`, which derives the dependencies from - the selected tests, checked nothing when this job built and ran them in one - step. `test-openvmm` therefore takes the runner's remaining steps itself: - it downloads the content's one image, `sample-vmgs.vhd`, from OpenVMM's - public test image container, sets the environment that Petri reads, and - runs the archive's tests with the runner setup's `cargo-nextest`, as the - runner does with the release that it downloads. The job needs neither the - curl shim nor a GitHub token, which only flowey's downloads use. - -The content's manifest, `nvx-openvmm-tests.json`, records the OpenVMM revision, -the Rust release, the target and the build options, each backend's filter, the -runner and its options, and every file with its size and SHA-256, so that a -test job refuses content built from another OpenVMM revision or for another -filter, and so that the content describes the inputs on which reusing it would -depend. The builds and their test jobs run together: a run that selects the -VMM tests, and so builds the guest artifacts, runs all four VMM jobs, and any -other run skips all four. `Required status check` expects -exactly that, and `Publish development release` and `Persist performance -baseline` require the four jobs to succeed, or all four to be skipped -because the change selected no VMM tests, so that a failed or unexpectedly -skipped build or test blocks them. The WHP job succeeds only when each of its -shards does. - -## OpenVMM VMM test cache - -What determines the VMM tests' build changes far less often than the tests -run: replaying the change classifier over the 42 `dev` merges through -[#659](https://github.com/microsoft/nvx/pull/659), the VMM tests would have run -on 33 of them, but the OpenVMM pin changed in none. Each producer therefore -caches its test content across runs, and restores it instead of building it -while nothing that determines it changes -([#692](https://github.com/microsoft/nvx/issues/692)). - -[`openvmm-test-cache-key`](usage.md#openvmm-test-cache-key) keys the content -of a producer's backends as `openvmm-vmm-tests-vVERSION-TARGET-REVISION-DIGEST`. -TARGET is the build's target, `linux-x64` or `windows-x64`; REVISION is the -OpenVMM revision that the checkout's gitlink pins; and DIGEST is the SHA-256 -of every field of the content's manifest that describes what makes a clean -build of that revision: the manifest's format, the Rust release that -`rust-toolchain.toml` pins, the target, the `vmm-tests-run` build options, a -release build without optional features, and each backend's filter and their -union. A change to any of them, such as a pin promotion, a new Rust release, -or a test that a filter adds, therefore builds the tests again. VERSION, -`OPENVMM_TEST_CACHE_VERSION` in [`ci.py`](../scripts/nvx_tools/ci.py), covers -what none of them captures: increment it when what a build makes, or what a -reuse checks, changes otherwise, such as the environment in which -`build-openvmm-tests` builds the tests. Like the OpenVMM binary cache, the key -leaves out the C toolchains and other tools of the runners, so a runner -upgrade reuses the content built before it until the key changes, and the -full runs below check the build every day. - -On a push, a pull request, or a merge group, a producer computes the key before -it checks out OpenVMM and restores the newest entry of the key into the -directory that it uploads. [`reuse-openvmm-tests`](usage.md#reuse-openvmm-tests) keeps that -content only if it is exactly what a fresh build of the key makes: its -manifest must record every field that the key digests, for a clean checkout, -and the runner and options that its run script runs, which the command reads -from the script again, and the directory must hold exactly the files that the -manifest lists, with their sizes, SHA-256, and executable bits. The test jobs -therefore run the same nextest archive with the same options and filters, and -so exactly the tests, that a fresh build would give them. On such a hit, the -producer skips the OpenVMM checkout, the retrying curl, the OpenVMM input and -Cargo caches, sccache, and the build, and uploads the restored content as the -same-run artifact that the test jobs download as before; the test jobs never -read the cache. Otherwise, the reuse removes whatever the restore left, with a -warning annotation when it refuses an entry or a partial restore, and the -producer builds the tests as before. The restore step continues on error, so -a miss, an eviction, a failed restore, or a refused entry only costs the -build. - -After a build, [`check-openvmm-tests`](usage.md#check-openvmm-tests) fails the -producer unless the content passes the same check, so that neither the cache -nor a test job gets a build that a reuse would refuse. Only `dev` pushes save -the content, as they save the other OpenVMM caches, and only after a build: -under the key followed by the run ID and attempt, so that each build makes an -entry of its own, and the restore, which takes the newest entry whose key -starts with the key, takes the latest build. An entry that a later reuse -refuses, for example after a change to the check that kept the version, -therefore blocks nothing: the next `dev` push builds the tests and saves a -newer entry. Pull requests and merge groups restore the entries that `dev` -pushes save but save none, so a pull request that changes the key builds the -tests in each of its runs and in its merge group's. Scheduled and manually -dispatched full runs neither restore nor save the content: they build the -tests every day, so that a breakage of the build that the key doesn't cover, -such as a toolchain change on the runners, shows within a day rather than at -the next pin promotion. - -This change's validation run, -[39988095577](https://github.com/microsoft/nvx/actions/runs/39988095577), saved -its builds from its pull request through a temporary change. The 228.5 MiB of -Linux content made a 124.0 MiB entry, which the KVM runners compress with gzip, -as they do their other caches, and the 132.6 MiB of Windows content a 37.3 MiB -entry, which the WHP runners compress with zstd: 161.3 MiB a key, 1.6% of the -repository's 10 GB of Actions caches. Those were full, at 10.4 GB in 89 -entries on October 23, 2026, so each new entry displaces the least recently -used ones. These were stale: the guest image entries of merged pull requests, -and the OpenVMM input and Cargo entries of the previous OpenVMM pin, about -2.4 GiB, which no run restores any more. A hit, in turn, restores neither the -`vmm-tests` OpenVMM input entry, about 945 MiB per operating system, nor its -Cargo entry, 161 MiB on Linux and 261 MiB on Windows, so those age out between -pin promotions as well. On a miss, the key, the restore, and the two checks -added under five seconds to each producer, whose builds took 340 and 712 -seconds, and a `dev` push's save adds about 10 and 4 seconds more. Rerun on -other runners, the producers hit, and took 20 and 36 seconds instead of 6.5 -and 14.2 minutes: 3 seconds to restore each entry, at about 118 and 25 MB/s, -under 1 and 2 seconds for the reuse's check, and 8 and 7 seconds to upload the -content. The test jobs then ran the same 61 KVM, 60 MSHV, and 15 and 14 WHP -tests, all passing, from the restored content as from the build. - -## OpenVMM VMM test shards - -`nvx.py test-openvmm --shard INDEX/COUNT` runs one of COUNT shards of a -backend's VMM tests. The partition is -[`openvmm_test_shards.json`](../scripts/nvx_tools/openvmm_test_shards.json): -for each sharded backend, the test threads with which nextest runs its tests, -the median duration of each test that its filter selects, and the tests of -each shard, with the time that nextest takes to run them. A shard runs the -backend's filter and its own tests both as its filter, so it never runs a -test that the backend's filter leaves out. Only the WHP filter names each of -its tests; the KVM and MSHV filters select theirs by pattern, so no partition -can list them, and only the WHP tests have shards. - -nextest runs a job's tests in parallel, so unlike the scenarios of the -[microVM shards](#microvm-test-shards), which run one after another, a shard -does not take the sum of its tests' durations. In the WHP jobs of the 124 -runs that the file lists, from October 22 and 23, 2026, the 29 tests took a -median 278 seconds, although their median durations add up to 769, and each -test's duration varied little between runs. nextest starts the tests in name -order, each as soon as it has the test threads that it requires, and never -one before an earlier one. OpenVMM's nextest configuration has a VMM test -require 2 threads, and a `heavy` test 12, so with the eight test threads that -nextest runs by default on the WHP runners' eight logical processors, four -tests run at a time, and each of the two `boot_heavy` tests runs alone: it -waits for every earlier test to end, and every later test waits for it. A -model of that scheduling gives each of the 124 runs' wall time to within 0.2 -seconds from its tests' durations, and -[`plan-openvmm-test-shards`](usage.md#plan-openvmm-test-shards) balances the -shards' modeled times. Two shards of equal sums of durations would take 177 -and 175 seconds, because each would hold a heavy test that stalls its other -tests. The first plan instead put both heavy tests with shorter tests in one -shard, which took 119.8 seconds, and the two FreeBSD tests, the longest at -87.9 and 83.9 seconds, with the PCIe tests in the other, which took 123.9. A -test checks the model's test threads and the path of the JUnit report that -`test-openvmm` reads against OpenVMM's pinned nextest configuration. - -A shard job spends a median 35 seconds beside its tests in the runs since -[#783](https://github.com/microsoft/nvx/pull/783): 24 to set up, of which the -content's download takes 2 to 5, about 6 in `test-openvmm` around the tests, -mostly for nextest to extract the archive and list the tests, and 5 to end. -Two shards therefore take about 2.7 minutes each instead of the single job's -5.1, at about the same WHP runner time, since the shards also stall less -behind the heavy tests. Applied to each of the 124 runs' own durations, the -longer shard takes a median 125.8 seconds, a p90 131.3, and at most 154.6, -against the 278.0, 287.6, and 309.4 that the unsharded runs took. In the 14 -first-attempt runs since #783 that ran the VMM tests, all from October 23, two -shards would have ended the WHP lane a median 2.5 minutes sooner, 20.1 -minutes after the run's first job started instead of 22.5. A third shard -would take 88 seconds, the floor that the longest test sets, and so save at -most another 36 seconds for another runner and another setup. The KVM lane, -though, the Linux build and the 10.5-minute KVM leg, ended a median 20.4 -minutes after the run's first job started, and would have ended after the -two shards in 8 of those 14 runs, so a third shard would have shortened the -VMM tests of at most six of them, by at most 0.6 minutes. A fourth would save -nothing. - -The Linux legs are not sharded. Their filters select tests by pattern, which -a partition could list only from each archive's tests at run time. In run -[39948600574](https://github.com/microsoft/nvx/actions/runs/39948600574), the -KVM leg's 61 tests took 611 seconds, four of them 190 to 267 seconds each. In -four of the eight runs above in which the KVM lane would have ended last, the -Linux build or the KVM leg had waited three minutes or more for one of KVM's -four general runners, so a second KVM leg would often have waited for one too. -Every VMM-test job uploads its test duration report, so that later runs show -whether the KVM lane ends last often enough, and by enough, to shard it too. - -CI runs each shard in a job of its own. `openvmm-vmm-tests-windows` is a matrix -of the WHP shards, which passes each leg its `matrix.shard` and the matrix's -`strategy.job-total` as `--shard INDEX/COUNT`, on WHP's `general` runners. -Each leg downloads the content of the one Windows build, which its rerun -downloads again instead of rebuilding. A leg fails before it runs a test -unless the file puts each test of every sharded filter in exactly one shard, -with its duration, and splits the WHP tests into the run's COUNT shards, and a -test checks that the matrix runs exactly the shards of the partition, so that -the legs run every WHP test exactly once. A shard that fails does not cancel -the other (`fail-fast: false`), so each keeps its diagnostics and its test -duration report under names of its own, and *Re-run failed jobs* reruns only -the shards that failed. For a `dev` run, use that partial rerun only under the -[procedure above](#run-concurrency). GitHub expands the matrix only once the -job's condition holds, so the shards run together or not at all, and the -job's fork, draft, change, and cancellation conditions are those of the -single job before. The job's result, which `Required status check`, `Publish -development release`, and `Record performance results` read, succeeds only -when every shard succeeds, so a shard that fails or is cancelled blocks them. - -The first plan took the medians of the durations that nextest printed in the -WHP jobs of the 124 successful runs that the file lists, from 03:01 UTC on -October 22 to 12:01 UTC on October 23, 2026, since no job recorded a test -duration report before the shards, with the WHP runners' eight test threads. -A sharded run fails before it runs a test unless the file puts each test of -every sharded filter in exactly one shard, with its duration, and splits the -backend's tests into the run's COUNT shards. Tests in -`scripts/test_nvx_tools.py` check the same, that the file is what the planner -makes of its own durations, and that the shards' filters select each test of -the WHP filter exactly once, so a change that adds a test to the WHP filter, -as [#135](https://github.com/microsoft/nvx/issues/135) would, removes one, or -renames one fails them until it plans the shards again: - -1. Download the test duration reports of recent CI runs that ran the WHP - tests, such as the last ten: - - ```bash - gh run download RUN_ID --repo microsoft/nvx \ - --pattern 'openvmm-vmm-test-durations-*' --dir build/vmm-durations/RUN_ID - ``` - -2. Plan the shards from them: - - ```bash - python3 scripts/nvx.py plan-openvmm-test-shards --report build/vmm-durations - ``` - - A test that no report measures keeps its duration in the file. A new test - has none, so give it an estimate, such as - `--seconds ttrpc::test_ttrpc_uefi_boot=9.4`, which the report of a local - `test-openvmm --content` run gives. A regeneration from CI's reports - replaces the estimate once CI has run the test. - -3. Commit the file with the change. The planner prints the shards with the - times that nextest takes to run them, and the tests that changed shard. A - test stays in its shard unless moving it, or swapping it for a test of - another shard, shortens the longest shard by a second or more. To change - the number of shards, pass `--shards whp=COUNT` and change the matrix of - `openvmm-vmm-tests-windows` in `ci.yml` to list the shards from 1 to COUNT. - The planner notes reports whose runners have another number of logical - processors than the file's test threads; pass `--test-threads whp=COUNT` to - plan for them. +## OpenVMM CI + +NVX pins a revision of `feature/microvm/dev` in `microsoft/openvmm` (see the +[OpenVMM upstream roadmap](openvmm-upstream-roadmap.md)), whose own checkin +gates test it: since OpenVMM's `c2722fbcd` of October 21, 2026, the OpenVMM PR +workflow runs them on each pull request into the branch that is ready for +review, and the OpenVMM CI workflow on each push to it. On OpenVMM's own +runners, they run OpenVMM's unit and documentation tests on x86-64 and AArch64 +Linux, Linux musl, and Windows, and its VMM tests on Windows Intel and AMD +hosts, where the WHP tests run, on Linux KVM and MSHV hosts, and on TDX, +SEV-SNP, and AArch64 hosts. CI therefore runs none of OpenVMM's tests. Until +then, five jobs built and ran OpenVMM's unit tests and a selection of its VMM +tests on the NVX runners: from October 21 to 31, 2026, they ran in 673 runs, +took about 620 runner-hours, 517 of them on the KVM, MSHV, and WHP runners, and +failed 9 times, never for a defect in OpenVMM. They also ended last in many of +the runs that ran them (see +[Job graph and runner capacity](#job-graph-and-runner-capacity)). OpenVMM's +gates run every test +that those jobs ran, but for two differences. OpenVMM's KVM job, on AMD hosts, +skips the 4 emulated Intel VT-d tests that NVX's KVM job ran, which OpenVMM's +MSHV and Windows Intel jobs run. And the gates' TTRPC microVM test boots +OpenVMM's own kernel at one processor, where NVX's jobs booted NVX's kernel at +one, two, four, and eight; NVX uses no TTRPC, and the NVX microVM tests boot +and restore NVX's kernel at those processor counts on every backend. +[`test-openvmm`](usage.md#test-openvmm) and +[`test-openvmm-unit`](usage.md#test-openvmm-unit) still run OpenVMM's tests on +a developer's host. + +Nothing on OpenVMM's side requires its gates to pass: `feature/microvm/dev` +has no required status check, and an OpenVMM pull request lands by a push of +its head (see the +[promotion skill](../.github/skills/nvx-openvmm-promote/SKILL.md)). Nor is +the outcome of a run a gate: from October 25 to 30, 2026, every OpenVMM CI run +of the branch failed, in the VMM tests of OpenHCL and confidential VMs on its +TDX and SEV-SNP hosts, which NVX does not use. The `openvmm-ci` job +therefore runs [`check-openvmm-ci`](usage.md#check-openvmm-ci), which +requires the jobs of the gates that cover NVX's use of OpenVMM, +`unit tests [x64-linux]`, `unit tests [x64-windows]`, and the +`run vmm-tests` jobs of `x64-linux-amd-kvm`, `x64-linux-intel-mshv`, +`x64-windows-intel`, and `x64-windows-amd`, to have passed for the pinned +revision, in a run of either workflow whose head is the revision in +`microsoft/openvmm`. A pull request's run tests GitHub's merge of the pull +request into the branch, which holds the tree of the head while the head +descends from the branch, as landing it requires. The job reads the pin from +the checkout's index and the public repository's runs with the run's token, +so it needs neither an OpenVMM checkout nor a deploy key, runs on a +GitHub-hosted runner, and runs for pull requests from forks too. + +A pull request that changes the pin usually starts before OpenVMM's gates +finish with the pinned revision, which takes them one and a half to two hours. +Its `openvmm-ci` job, and that of a manual run, therefore looks again every +five minutes while a required job may still pass, for up to 300 minutes, so +that the pin promotion passes once OpenVMM's CI does; a look takes about three +requests of the rate limit that the jobs of the repository's runs share. It +also looks again after a look that failed, at most twice in a row. It fails at +once when a required job failed, after which a rerun of the OpenVMM job and +then of the `openvmm-ci` job passes it, or when no run will run one, as for a +revision that no pull request into the branch that is ready for review holds. +A merge group, a `dev` push, and a scheduled run check a pin whose pull +request passed the check, so they fail at once rather than hold up the merge +groups and `dev` pushes that queue behind them. + +The [change classification](#change-classification) runs the job when the pin +or the check changes, and every full run runs it. `Required status check` +expects it to succeed then and to be skipped otherwise, and +`Publish development release` and `Record performance results` require it to +succeed, or to be skipped because the change selected no check. ## Change classification @@ -1802,30 +1511,21 @@ other path selects each suite that takes it as an input: request or merge group sets it only when one of these inputs changed, but a push sets it for every change but documentation; see [Development releases and performance history](#development-releases-and-performance-history). -- `run-openvmm-unit-tests` and `run-openvmm-vmm-tests` enable the OpenVMM - suites. A push, pull request, or merge group sets each only when one of that - suite's own inputs changed. The unit tests depend on the OpenVMM pin, the - Rust pin, the runner setup scripts, `ci.yml`, the actions that their job - uses, `nvx.py`, and the modules that `ci.py` imports. The VMM tests also - boot the kernel and Alpine initramfs, so they add every input of those - artifacts' cache keys and the `build-guest-artifacts` action, and run their - WHP tests in [shards](#openvmm-vmm-test-shards), so they add the modules of - the commands that their jobs run and the files that those read, such as the - partition of the shards. Of `nvx-rs`, those are only the inputs - of the guest binaries, which `GuestRustBuildConstants.SOURCE_PATTERN` - matches, so a change to its host side or tests runs the NVX microVM tests - and the platform and package jobs but not the VMM tests. +- `run-openvmm-ci-check` enables the [OpenVMM CI](#openvmm-ci) check. A push, + pull request, or merge group sets it only when the OpenVMM pin, `ci.yml`, + `nvx.py`, or one of the CLI modules that the check runs changed: it builds + nothing and runs no test, so neither the Rust pin nor the runners' setup is + one of its inputs. - `run-guest-artifacts` enables `artifacts`, `distro-artifacts`, and `debug-kernel`, which build the guest artifacts and the CI debug kernel, and `run-openvmm-binaries` the three OpenVMM binary producers. Neither has a use of its own, so the classifier sets each exactly when a job that uses its - artifacts runs: the guest artifacts for the OpenVMM VMM tests, the NVX - microVM tests, or the platform and package jobs, and the OpenVMM binaries - for the microVM tests or the platform and package jobs. Every input of an - artifact is an input of each job that uses it. Every input of the VMM tests is - also an input of the microVM tests, which read the Ubuntu and Azure Linux - images of - `distro-artifacts`, so the three guest artifact jobs share the output. The + artifacts runs: the guest artifacts for the NVX microVM tests or the + platform and package jobs, and the OpenVMM binaries for the same jobs. Every + input of an artifact is an input of each job that uses it. Each of those + suites also runs a job that reads the Ubuntu and Azure Linux images of + `distro-artifacts`, the microVM guest and sandbox jobs or the package jobs, + so the three guest artifact jobs share the output. The debug kernel is built with the guest artifacts, as before, because the development release and the recording of the performance results require its build too. @@ -1839,48 +1539,42 @@ not alone make it a suite input. `microvm_tests.py` and the modules that only it uses, including `microvm_durations.py`, with the `microvm_test_scripts/` that it runs in guests and the partition of its [shards](#microvm-test-shards), -`microvm_shards.json`, are inputs of the microVM tests alone. -`openvmm_test_shards.py`, with the partition of the -[VMM test shards](#openvmm-vmm-test-shards), `openvmm_test_shards.json`, and -`shard_plans.py`, which both partitions' planners share, are inputs of the -VMM tests and, as every input of theirs is, of the microVM tests, but not of -the platform or package jobs. `performance.py` selects the platform and package -jobs alone. `nightly_ci.py`, which only the nightly report workflow runs, +`microvm_shards.json`, are inputs of the microVM tests alone, as is +`shard_plans.py`, the planner that the partition's command uses. +`performance.py` selects the platform and package jobs alone. `crate_release.py`, which only the crate release jobs run, `merge_queue.py`, which only the job that cancels superseded merge-group runs uses, and `performance_history.py`, which only the job that persists the performance -results that the `dev` pushes recorded runs, are inputs of neither. Modules -that both kinds +results that the `dev` pushes recorded runs, are inputs of neither, and +`openvmm_ci.py` and `nightly_ci.py`, which the nightly report workflow runs +and whose GitHub client `openvmm_ci.py` and `performance_history.py` use, are +inputs of the OpenVMM CI check alone. Modules that both kinds of job run, such as `benchmark.py`, which `microvm_tests.py` imports, `doctor.py`, which every runner validation runs, and `release.py`, whose `verify` command the OpenVMM builds run, select both. -A push, pull request, or merge group that changes neither OpenVMM suite's -inputs, such as one that changes only the microVM tests, the benchmarks, the -Ubuntu or Azure Linux guests, or the host side of `nvx-rs`, therefore skips the -jobs of both OpenVMM suites, which occupy the WHP runners longest. A pull +A push, pull request, or merge group that changes neither the OpenVMM pin nor +the check, such as one that changes only the microVM tests, the benchmarks, +the guests, or `nvx-rs`, therefore skips the OpenVMM CI check. A pull request or merge group that changes only the microVM tests, such as their scenarios or the runtime adapter, also skips the platform and package jobs. The platform jobs wait for their pool's single benchmark runner. A change only to the performance tooling, such as `performance.py`, the `run-benchmark` action, or `data/`, skips the microVM tests. One that changes the input of no -suite, such as the CLI tests, nightly reporting, crate publication, the -adversarial campaigns, or the GitHub-hosted checks, runs only the GitHub-hosted -jobs. +suite, such as the CLI tests, crate publication, the adversarial campaigns, +or the GitHub-hosted checks, runs only the GitHub-hosted jobs. The nightly schedule runs at 05:17 UTC, and a manual workflow dispatch starts the same full run. Both select the full matrix before the detector examines a comparison base. Each full run has a separate concurrency key, so it neither cancels nor queues behind a `dev` push. Full runs build, test, benchmark, and package, but publish no development release, record no -performance results, and save no caches, and they build the OpenVMM VMM tests -without restoring them from the -[OpenVMM VMM test cache](#openvmm-vmm-test-cache). A full run of `dev` +performance results, and save no caches. A full run of `dev` persists the performance results that the `dev` pushes recorded instead (see [Development releases and performance history](#development-releases-and-performance-history)), and publishes the `nvx` crate to crates.io when the crate changed (see [crates.io releases](#cratesio-releases)). Only `dev` pushes save the -OpenVMM caches, and only pushes and pull +OpenVMM binary caches, and only pushes and pull requests save the guest artifact and pip caches, through the `save-cache` input of the [`build-guest-artifacts`](../.github/actions/build-guest-artifacts/action.yml) @@ -1900,20 +1594,18 @@ expects every scheduled job to succeed and every other job to be skipped. As in `ci.yml`, it schedules a job only when the jobs that build the artifacts that it uses are scheduled too. Tests in `scripts/test_nvx_tools.py` derive each suite's inputs and fail when -a pattern misses one. The OpenVMM suites' come from the actions that their -jobs use, the modules that `ci.py` imports, and the guest cache keys, -including the files of the guest binaries' source identity, and the VMM -tests' also from the modules of the `nvx.py` commands that their jobs run, -with the files that those read. The inputs of the microVM tests and the +a pattern misses one. The OpenVMM CI check's come from the modules of the +`nvx.py` command that its job runs, with every module that those import, and +the pin. The inputs of the microVM tests and the platform and package jobs come from those jobs and the jobs that build their artifacts: the reusable workflows and actions that these reach, the files that the cache keys there hash and the source identity's, the CLI modules of the `nvx.py` commands that they run, which the tests map from `nvx.py`'s parser, with every module that those import and every file that those read beside them, and the `nvx` crate, from which the `nvx-rs` build -jobs build the executable. The tests also check that no other file of `nvx-rs` -selects the VMM tests or changes a guest image key, while every one selects the -NVX microVM tests and the platform and package jobs. +jobs build the executable. The tests also check that no file of `nvx-rs` but +the inputs of the guest binaries changes a guest image key, while every one +selects the NVX microVM tests and the platform and package jobs. ### Development releases and performance history @@ -2258,7 +1950,7 @@ install it with its OpenVMM targets. `validate-runner` disables rustup's automatic installation and fails a job before it builds anything when its runner lacks the release, naming the setup script to rerun. The OpenVMM binary cache key includes a hash of the file, so a new release rebuilds OpenVMM, and a change -to the file runs the OpenVMM test matrix. +to the file runs the NVX microVM tests and the platform jobs. To move the pin, first confirm that `guest_test_uefi` links with the new release: @@ -2285,9 +1977,11 @@ Every self-hosted runner keeps a persistent compiles through it with incremental compilation disabled. CI builds in clean target directories, so each runner compiles a crate only when its own cache lacks it, and loses the cache when its VM is reimaged, as by an image upgrade -of its scale set. The OpenVMM jobs that run on these runners start sccache and -report its statistics at the end of the job through the -[`sccache`](../.github/actions/sccache/action.yml) action. +of its scale set. The OpenVMM binary builds, the only CI jobs that compile +OpenVMM, start sccache and report its statistics at the end of the job through +the [`sccache`](../.github/actions/sccache/action.yml) action. They restore +flowey's downloads and the Cargo registry from caches that the builds of `dev` +pushes save when they miss the binary cache. sccache keys a Rust compilation on its arguments, its `CARGO_*` environment, and the contents of its sources and of each library that it names with @@ -2298,8 +1992,8 @@ with the time of the link and a random PDB signature. Every Windows build therefore missed the cache for each crate that uses a proc macro directly, several times per job because flowey builds `openvmm` and `pipette` in target directories of their own: 243 of the 1,421 cacheable compilations of a WHP -VMM-test job, and 286 of the 1,318 of a Windows unit-test job (#693). On -Windows, `build-openvmm-tests`, `test-openvmm`, and `test-openvmm-unit` +VMM-test job, and 286 of the 1,318 of a Windows unit-test job, when CI still +ran OpenVMM's tests (#693). On Windows, `test-openvmm` and `test-openvmm-unit` therefore append `/Brepro` to `_LINK_` when they build, which makes `link.exe` derive both from the image's contents. diff --git a/doc/contribute.md b/doc/contribute.md index d5c8198cb..af400caff 100644 --- a/doc/contribute.md +++ b/doc/contribute.md @@ -94,7 +94,11 @@ systems, so add a new script or test module to those lists. ## Test ownership -OpenVMM owns control-plane coverage. Run its unit and documentation tests with +OpenVMM owns control-plane coverage. Its own CI runs its unit, +documentation, and VMM tests on each pull request into the pinned branch and +on each push to it, and NVX CI checks that they passed the pinned revision +instead of running them (see [OpenVMM CI](ci.md#openvmm-ci)). Run its unit and +documentation tests locally with `scripts/nvx.py test-openvmm-unit` and its VMM tests with `scripts/nvx.py test-openvmm --backend BACKEND`. The VMM tests boot OpenVMM's own test guests, except the management-RPC (TTRPC) lifecycle, SMP, @@ -131,7 +135,7 @@ session between the host and the guest, belong in the [runtime adapter](../tests/runtime-adapter/README.md), whose consolidated suite the microVM jobs run on each backend with `scripts/nvx.py test-runtime-adapter --backend BACKEND`. [Continuous integration](ci.md) -describes what the OpenVMM VMM tests and the NVX microVM suite cover. +describes how CI checks OpenVMM's own CI and what the NVX microVM suite covers. ## Test expectations diff --git a/doc/project-structure.md b/doc/project-structure.md index 3c90763ff..d7e4b946c 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -121,7 +121,7 @@ nvx/ | | |-- build.py Artifact build workflows | | |-- build_config.py Per-invocation build configuration | | |-- build_constants.py Grouped build pins, paths, and fixed defaults -| | |-- ci.py CI change classification, required jobs, and OpenVMM tests +| | |-- ci.py CI change classification, required jobs, and local OpenVMM test runs | | |-- collect_alpine_sources.py Alpine source collection | | |-- collect_ubuntu_sources.py Ubuntu source collection | | |-- common.py Shared process and configuration helpers diff --git a/doc/usage.md b/doc/usage.md index d7f2e0e4d..fb970aa33 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -436,12 +436,7 @@ is authorized for the organization when it enforces single sign-on. | `check-openvmm-ci` | Check that OpenVMM's own CI passed the pinned OpenVMM revision. | | `check-static-elf` | Check that executables are static ELF files without an interpreter or an executable stack. | | `test-openvmm-unit` | Run the OpenVMM workspace unit and documentation tests. | -| `build-openvmm-tests` | Build OpenVMM Petri VMM tests without running them. | -| `openvmm-test-cache-key` | Print the key under which CI caches the OpenVMM VMM tests that `build-openvmm-tests` builds. | -| `check-openvmm-tests` | Check that a directory holds the OpenVMM VMM tests that `build-openvmm-tests` builds from this checkout. | -| `reuse-openvmm-tests` | Keep OpenVMM VMM tests that a cache restored if `build-openvmm-tests` builds them from this checkout, and remove them otherwise. | | `test-openvmm` | Run OpenVMM Petri VMM tests. | -| `plan-openvmm-test-shards` | Split the OpenVMM VMM tests into shards that nextest runs in balanced times for CI. | | `test-microvm` | Run NVX Linux and device correctness tests through OpenVMM. | | `plan-microvm-shards` | Split the default microVM suite into shards of balanced scenario durations for CI. | | `doctor` | Qualify this host for the NVX time ABI. | @@ -603,16 +598,17 @@ python3 scripts/nvx.py classify-ci-changes Reads one changed repository path per line by default, or NUL-delimited paths with `--null`, and prints the change detector's -`run-guest-artifacts`, `run-openvmm-binaries`, `run-openvmm-unit-tests`, -`run-openvmm-vmm-tests`, `run-microvm-tests`, and `run-benchmarks` outputs as +`run-guest-artifacts`, `run-openvmm-binaries`, `run-openvmm-ci-check`, +`run-microvm-tests`, and `run-benchmarks` outputs as `name=true` or `name=false` lines. Documentation selects no suite. Pull requests, merge groups, and pushes run each test suite only when one of its -own inputs changed. A pull request or merge group runs the platform benchmarks -and package jobs only when one of their inputs changed, but a push runs them -for every change but documentation, because it publishes the development -release and records the performance history. The guest artifacts and the OpenVMM -binaries are built exactly when a job that uses them runs. The summary file -receives a Markdown +own inputs changed, and check OpenVMM's own CI only when the OpenVMM pin or +the code that checks it changed. A pull request or merge group runs the +platform benchmarks and package jobs only when one of their inputs changed, +but a push runs them for every change but documentation, because it publishes +the development release and records the performance history. The guest +artifacts and the OpenVMM binaries are built exactly when a job that uses them +runs. The summary file receives a Markdown explanation of each decision. CI runs the full matrix instead when the command fails or omits an output; see [Change classification](ci.md#change-classification). @@ -625,8 +621,7 @@ python3 scripts/nvx.py check-required-ci --same-repository {false,true} --run-guest-artifacts VALUE --run-openvmm-binaries VALUE - --run-openvmm-unit-tests VALUE - --run-openvmm-vmm-tests VALUE + --run-openvmm-ci-check VALUE --run-microvm-tests VALUE --run-benchmarks VALUE ``` @@ -756,8 +751,8 @@ python3 scripts/nvx.py check-openvmm-ci ``` Checks that OpenVMM's own CI passed the OpenVMM revision that the checkout's -index pins, or the full commit SHA that `--revision` names. -It reads the runs of the OpenVMM PR +index pins, or the full commit SHA that `--revision` names, as +[OpenVMM CI](ci.md#openvmm-ci) describes. It reads the runs of the OpenVMM PR and OpenVMM CI workflows of `microsoft/openvmm` whose head is the revision, with the latest attempt of each of their jobs, and requires each job that covers NVX's use of OpenVMM, the x86-64 Linux and Windows unit tests and the @@ -1028,218 +1023,25 @@ Afterward, runs the workspace doctests with Cargo. On Windows, it appends can reuse the crates that use them; see [Compilation cache](ci.md#compilation-cache). -### `build-openvmm-tests` - -```text -python3 scripts/nvx.py build-openvmm-tests - --backend {kvm,mshv,whp} [--backend {kvm,mshv,whp} ...] - --output PATH -``` - -Builds the VMM tests that [`test-openvmm`](#test-openvmm) runs for each -`--backend`, without running them, into the new or empty directory that -`--output` names, so that another host can run them without building OpenVMM. -The backends must run on this host's operating system: Linux builds the KVM -and MSHV tests, and Windows the WHP tests. The command runs OpenVMM's -`vmm-tests-run` in build-only mode on the union of the backends' filters, for -the `linux-x64` or `windows-x64` target, since NVX supports only x86-64 hosts, -as a release build without optional features whose run script uses the -nextest CI profile, which puts the tests' nextest archive, OpenVMM, the guest -agents, the test firmware and kernels, and a `vmm-tests-run-target` runner into -the directory. It then reads the runner's options from the run script that the -build writes, and records them in the directory's `nvx-openvmm-tests.json` -with the OpenVMM revision and whether its checkout was clean, the Rust -release, the target, the `vmm-tests-run` build options, each backend's filter, -and every file's size, SHA-256, and executable bit. The tests boot NVX's kernel -and initramfs only when they run, so the build needs no guest artifacts. On -Windows, it appends `/Brepro` to `_LINK_`, as `test-openvmm` does. - -### `openvmm-test-cache-key` - -```text -python3 scripts/nvx.py openvmm-test-cache-key - --backend {kvm,mshv,whp} [--backend {kvm,mshv,whp} ...] -``` - -Prints the key under which CI caches the test content that -[`build-openvmm-tests`](#build-openvmm-tests) builds for the backends from this -checkout (see [OpenVMM VMM test cache](ci.md#openvmm-vmm-test-cache)): -`openvmm-vmm-tests-vVERSION-TARGET-REVISION-DIGEST`. VERSION is the -cache's own version, which changes when what a build makes changes in a way -that nothing else in the key captures. TARGET is the build's target, REVISION -the OpenVMM revision that the checkout's index pins, and DIGEST the SHA-256 of -the fields of the manifest of a clean build of that revision that describe -what built it: the manifest's format, the Rust release, the target, the build -options, and each backend's filter and their union. The backends must run on -this host's operating system, as for `build-openvmm-tests`, and the command -needs neither the OpenVMM submodule's files nor Cargo. In GitHub Actions, it -also gives its step the `key` output. - -### `check-openvmm-tests` - -```text -python3 scripts/nvx.py check-openvmm-tests - --backend {kvm,mshv,whp} [--backend {kvm,mshv,whp} ...] - --content PATH -``` - -Fails unless the directory that `--content` names holds exactly the test -content of the backends' cache key: a manifest that records every field that -[`openvmm-test-cache-key`](#openvmm-test-cache-key) digests, for a clean -OpenVMM checkout, and the runner and options that the content's run script -runs, and exactly the files that the manifest lists, with their sizes, -SHA-256, and executable bits. CI checks each build this way before it caches -it. - -### `reuse-openvmm-tests` - -```text -python3 scripts/nvx.py reuse-openvmm-tests - --backend {kvm,mshv,whp} [--backend {kvm,mshv,whp} ...] - --content PATH -``` - -Keeps the directory that `--content` names, into which a cache restored test -content, if it passes the checks of -[`check-openvmm-tests`](#check-openvmm-tests), and otherwise removes it with a -warning that gives the reason, so that a build can take its place. A -directory that doesn't exist, as after a cache miss, needs no removal. The -command fails only when it cannot compute the key or remove the directory, -never because of the content. In GitHub Actions, it gives its step the -`reused` output, `true` or `false`. - ### `test-openvmm` ```text python3 scripts/nvx.py test-openvmm --backend {kvm,mshv,whp} - [--content PATH] - [--shard INDEX/COUNT] -``` - -Builds and runs OpenVMM's checkout-owned VMM tests. The Linux-direct microVM -TTRPC test boots NVX's `build/vmlinux` and `build/initramfs.cpio.gz`, so build -the guest first; the remaining test artifacts are produced by OpenVMM itself. -On Windows, it appends `/Brepro` to `_LINK_`, as `test-openvmm-unit` does. - -`--shard INDEX/COUNT`, such as `1/2`, runs shard INDEX of the COUNT shards of -the backend's tests, as -[`openvmm_test_shards.json`](../scripts/nvx_tools/openvmm_test_shards.json) -splits them: the tests of the backend's filter that the shard lists, which the -command runs with the backend's filter and the shard's tests both as its -filter. CI runs each shard of the WHP tests in a job of its own (see -[OpenVMM VMM test shards](ci.md#openvmm-vmm-test-shards)). Only the WHP filter -names each of its tests, so only the WHP tests have shards. The command fails -before it runs a test unless that file puts each test of every sharded filter -in exactly one shard, with its duration, and splits the backend's tests into -COUNT shards. - -With `--content`, the command builds nothing: it runs the backend's tests from -a directory that [`build-openvmm-tests`](#build-openvmm-tests) built for that -backend, so the host needs neither the OpenVMM checkout nor its Rust targets. -It refuses a directory built from another OpenVMM revision than the checkout -pins or for another filter than the backend's, and one that lacks a file of -the build or holds a truncated one, and it restores the executable bits that -an artifact download drops. It still needs the guest artifacts above, and -Petri writes the test results to `test_results` in the directory, where the -command also writes the [test duration report](#test-duration-report). On -Linux, it runs the tests with the directory's `vmm-tests-run-target` runner and -the options that its manifest records. - -On Windows, the pinned OpenVMM's runner checks for Hyper-V whatever tests it -runs: the account must be able to run `Get-VM`, and the -`AllowFirmwareLoadFromFile` and `EnableAdditionalComPorts` values must be set -under `HKLM\Software\Microsoft\Windows NT\CurrentVersion\Virtualization`. No WHP -test needs Hyper-V, so the command takes the runner's remaining steps itself. -It downloads the images that the recorded options name into `images` in the -directory, or into the directory that `VMM_TEST_IMAGES` names, unless that -already holds an image at the size that OpenVMM declares. It then sets the -environment that Petri reads and runs the backend's filter, or the shard's, -from the directory's nextest archive with `cargo-nextest`, which must be on -`PATH`, the directory's nextest configuration, the profile that the options -select, and the directory as the workspace. It refuses the options that only -the runner carries out, such as prep steps, the test IGVM agent, hardware -isolation, and repeated runs, and an image that it cannot download. - -#### Test duration report - -A run of a test content directory records how long each test that nextest ran -took in `openvmm-test-durations.json` in the directory's `test_results`, as -the run ends, whether its tests passed or not, and [CI](ci.md) uploads the -reports of its VMM-test jobs. The command reads the durations from the JUnit -report that OpenVMM's nextest configuration writes for the `ci` profile, which -the content's runs use, under the workspace that a run remaps to the -directory: `target/nextest/ci/junit.xml`. It removes both reports of an -earlier run before it runs the tests, so that a run whose nextest writes no -JUnit report leaves no test duration report either. A run that is -interrupted, as when CI cancels its job, stops nextest before it writes the -JUnit report, which it writes only as its run ends, and so usually leaves no -test duration report. A report that it cannot write only warns, so that it -never hides the run's own outcome. The report is a JSON object: - -| Field | Value | -| --- | --- | -| `schema`, `schema_version` | `nvx-openvmm-test-durations` and `1`. The version changes only when a reader of the previous version would misread a report, not when a field is added. | -| `backend` | The `--backend` | -| `shard` | The `--shard`, as `{"index": INDEX, "count": COUNT}`, or `null` for a run of all of the backend's tests | -| `filter` | The nextest filter that the run ran | -| `logical_processors` | The host's logical processors, one for each of nextest's default test threads, or `null` if the OS doesn't tell | -| `github` | In GitHub Actions, the job's `repository`, `workflow`, `run_id`, `run_attempt`, `job`, `event_name`, `ref`, `sha`, `runner_name`, and `runner_os`, as strings from the `GITHUB_` and `RUNNER_` variables of those names; otherwise `null` | -| `tests` | Each test case of the JUnit report, in its order: its `binary` and `test` names, the `seconds` that it ran, to the millisecond, and its `outcome`: `passed`, `failed`, `flaky` for a test that passed only when nextest retried it, or `skipped` | - -### `plan-openvmm-test-shards` - -```text -python3 scripts/nvx.py plan-openvmm-test-shards - [--report PATH]... - [--seconds TEST=SECONDS]... - [--shards BACKEND=COUNT]... - [--test-threads BACKEND=COUNT]... - [--partition PATH] ``` -Plans the shards of the OpenVMM VMM tests, which CI runs in parallel (see -[OpenVMM VMM test shards](ci.md#openvmm-vmm-test-shards)), and writes them to -[`openvmm_test_shards.json`](../scripts/nvx_tools/openvmm_test_shards.json), -or to `--partition`. The file holds, for each sharded backend, the test -threads with which nextest runs its tests, the median duration of each of the -tests that its filter selects, the CI runs whose reports gave them, and each -shard's tests with the time that nextest takes to run them. Only a backend -whose filter names each of its tests, which only the WHP filter does, can have -shards. - -`--report` reads [test duration reports](#test-duration-report), or every -report in a directory, such as the one that `gh run download` fills with a -run's `openvmm-vmm-test-durations-*` artifacts. Each backend's duration of a -test is the median, to a tenth of a second, of the reports' times of the test -when it passed, in runs of a shard or of all of the backend's tests. A test -that no report measures keeps its duration in the file. `--seconds` sets a -test's duration on every backend instead, such as an estimate of a new test -that no report measures yet; the command fails when a test of a sharded filter -has no duration. - -nextest runs a shard's tests in parallel, so a shard does not take the sum of -their durations. The command models how nextest runs them: in name order, each -as soon as it has the test threads that it requires, and never one before an -earlier one. A test requires the threads that OpenVMM's nextest configuration -gives it, 2, or 12 for a `heavy` test and 32 for a `very_heavy` one, and -nextest gives a test that requires more than it runs all of its threads, so -that the test runs alone and the tests after it wait for it. A backend's test -threads are those that the file records, which `--test-threads` changes, one -for each logical processor of its runners; the command notes the reports whose -runners have another number. - -Each backend keeps its number of shards unless `--shards` changes it, and a -new backend needs both `--shards` and `--test-threads`. Each test starts in its -shard of the file, and a test that has none goes, longest first, to the shard -that nextest takes the least time to run so far. A shard that is still empty, -as one that `--shards` adds can be, takes the longest test of the longest -shard that has more than one. The command then moves a test to another shard, -or swaps two tests of different shards, while that shortens the longest shard -by at least a second, comparing the next longest where the longest stay alike, -and takes the best such change each time; no change empties a shard. It prints -each backend's shards with the times that nextest takes to run them and all of -its tests, and the tests that changed shard. +Builds and runs the OpenVMM checkout's VMM tests that the backend's filter +selects, with OpenVMM's `vmm-tests-run`. The Linux-direct microVM TTRPC test +boots NVX's `build/vmlinux` and `build/initramfs.cpio.gz` at one, two, four, +and eight processors, so build the guest first; the remaining test artifacts +are produced by OpenVMM itself. On Windows, it appends `/Brepro` to `_LINK_`, +as `test-openvmm-unit` does. No WHP test needs Hyper-V, so `vmm-tests-run`, +which derives what a run needs from the tests that it selects, needs no +Hyper-V administration on a WHP host. + +CI runs neither this command nor `test-openvmm-unit`: OpenVMM's own CI runs +OpenVMM's tests, and CI checks that it passed the pinned revision with +[`check-openvmm-ci`](#check-openvmm-ci); see [OpenVMM CI](ci.md#openvmm-ci). ### `test-microvm` diff --git a/nvx-rs/README.md b/nvx-rs/README.md index d5420b233..c5e1de274 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -2555,9 +2555,9 @@ and every file under `src/` but the host side and the test modules. The host side is `src/host/`, `src/testing/`, and the `nvx-rs` and `nvx-fake-openvmm` binaries; the test modules are the `tests.rs` files, the `tests` directories, and the managed agent's `loopback` harness. CI keys the guest images on those -files and runs the OpenVMM VMM tests, which boot the Alpine image, only when -one of them changes, while every change to the crate runs the NVX microVM -tests; see [Continuous integration](../doc/ci.md#change-classification). Keep +files, so only a change to one of them rebuilds the images, while every change +to the crate runs the NVX microVM tests; see +[Continuous integration](../doc/ci.md#change-classification). Keep host-only code under `src/host/` and test-only modules in `tests.rs` files or `tests` directories. The build fails when Cargo's dep-info lists another file, which `GuestRustBuildConstants` in diff --git a/scripts/nvx.py b/scripts/nvx.py index 5e6520581..a6c7174ab 100644 --- a/scripts/nvx.py +++ b/scripts/nvx.py @@ -52,14 +52,10 @@ OPENVMM_TEST_BACKENDS, REQUIRED_CI_RESULT_ENVIRONMENTS, CiChanges, - build_openvmm_tests, - check_openvmm_tests, classify_ci_changes, describe_ci_changes, kernel_patch_identity, - openvmm_test_cache_key, required_ci_failures, - reuse_openvmm_tests, run_openvmm_tests, run_openvmm_unit_tests, setup_cross_os_cache, @@ -100,10 +96,6 @@ ) from nvx_tools.nightly_ci import configure_parser as configure_nightly_ci_parser from nvx_tools.openvmm_ci import configure_parser as configure_openvmm_ci_parser -from nvx_tools.openvmm_test_shards import ( - configure_parser as configure_openvmm_test_shard_parser, -) -from nvx_tools.openvmm_test_shards import select_shard as select_openvmm_test_shard from nvx_tools.performance import configure_parser as configure_performance_parser from nvx_tools.performance_history import ( configure_parser as configure_performance_history_parser, @@ -117,7 +109,6 @@ from nvx_tools.runtime_adapter_tests import ( configure_parser as configure_runtime_adapter_test_parser, ) -from nvx_tools.shard_plans import parse_shard from nvx_tools.time_abi import ( benchmark_cpu_profile_refusal, openvmm_cpu_generations, @@ -294,8 +285,7 @@ def command_check_required_ci(args: argparse.Namespace) -> None: changes=CiChanges( run_guest_artifacts=args.run_guest_artifacts == "true", run_openvmm_binaries=args.run_openvmm_binaries == "true", - run_openvmm_unit_tests=args.run_openvmm_unit_tests == "true", - run_openvmm_vmm_tests=args.run_openvmm_vmm_tests == "true", + run_openvmm_ci_check=args.run_openvmm_ci_check == "true", run_microvm_tests=args.run_microvm_tests == "true", run_benchmarks=args.run_benchmarks == "true", ), @@ -311,29 +301,8 @@ def command_check_static_elf(args: argparse.Namespace) -> None: check_static_executables(args.paths) -def command_build_openvmm_tests(args: argparse.Namespace) -> None: - build_openvmm_tests(args.backend, args.output) - - -def command_openvmm_test_cache_key(args: argparse.Namespace) -> None: - openvmm_test_cache_key(args.backend, os.environ) - - -def command_check_openvmm_tests(args: argparse.Namespace) -> None: - check_openvmm_tests(args.backend, args.content) - - -def command_reuse_openvmm_tests(args: argparse.Namespace) -> None: - reuse_openvmm_tests(args.backend, args.content, os.environ) - - def command_test_openvmm(args: argparse.Namespace) -> None: - shard = ( - None - if args.shard is None - else select_openvmm_test_shard(args.backend, args.shard) - ) - run_openvmm_tests(args.backend, args.content, shard=shard) + run_openvmm_tests(args.backend) def command_test_openvmm_unit(_: argparse.Namespace) -> None: @@ -730,10 +699,7 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: "--run-openvmm-binaries", choices=("false", "true"), required=True ) required_ci.add_argument( - "--run-openvmm-unit-tests", choices=("false", "true"), required=True - ) - required_ci.add_argument( - "--run-openvmm-vmm-tests", choices=("false", "true"), required=True + "--run-openvmm-ci-check", choices=("false", "true"), required=True ) required_ci.add_argument( "--run-microvm-tests", choices=("false", "true"), required=True @@ -783,85 +749,6 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: ) openvmm_unit_tests.set_defaults(handler=command_test_openvmm_unit) - openvmm_test_build = subparsers.add_parser( - "build-openvmm-tests", - help="build OpenVMM Petri VMM tests without running them", - ) - openvmm_test_build.add_argument( - "--backend", - choices=OPENVMM_TEST_BACKENDS, - action="append", - required=True, - help="build this backend's tests; repeat for each backend of this host's OS", - ) - openvmm_test_build.add_argument( - "--output", - type=Path, - required=True, - help="empty or new directory to build the test content into", - ) - openvmm_test_build.set_defaults(handler=command_build_openvmm_tests) - - openvmm_test_cache_key = subparsers.add_parser( - "openvmm-test-cache-key", - help=( - "print the key under which CI caches the OpenVMM VMM tests that " - "build-openvmm-tests builds" - ), - ) - openvmm_test_cache_key.add_argument( - "--backend", - choices=OPENVMM_TEST_BACKENDS, - action="append", - required=True, - help="key this backend's tests; repeat for each backend that the build builds", - ) - openvmm_test_cache_key.set_defaults(handler=command_openvmm_test_cache_key) - - openvmm_test_check = subparsers.add_parser( - "check-openvmm-tests", - help=( - "check that a directory holds the OpenVMM VMM tests that " - "build-openvmm-tests builds from this checkout" - ), - ) - openvmm_test_check.add_argument( - "--backend", - choices=OPENVMM_TEST_BACKENDS, - action="append", - required=True, - help="check for this backend's tests; repeat for each backend of the build", - ) - openvmm_test_check.add_argument( - "--content", - type=Path, - required=True, - help="directory of the test content to check", - ) - openvmm_test_check.set_defaults(handler=command_check_openvmm_tests) - - openvmm_test_reuse = subparsers.add_parser( - "reuse-openvmm-tests", - help=( - "keep OpenVMM VMM tests that a cache restored if build-openvmm-tests " - "builds them from this checkout, and remove them otherwise" - ), - ) - openvmm_test_reuse.add_argument( - "--backend", - choices=OPENVMM_TEST_BACKENDS, - action="append", - required=True, - help="reuse this backend's tests; repeat for each backend of the build", - ) - openvmm_test_reuse.add_argument( - "--content", - type=Path, - required=True, - help="directory into which the cache restored the test content", - ) - openvmm_test_reuse.set_defaults(handler=command_reuse_openvmm_tests) - openvmm_tests = subparsers.add_parser( "test-openvmm", help="run OpenVMM Petri VMM tests", @@ -871,35 +758,8 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: choices=OPENVMM_TEST_BACKENDS, required=True, ) - openvmm_tests.add_argument( - "--content", - type=Path, - help=( - "run the tests from the directory that build-openvmm-tests built, " - "instead of building them" - ), - ) - openvmm_tests.add_argument( - "--shard", - type=parse_shard, - metavar="INDEX/COUNT", - help=( - "run shard INDEX of the COUNT shards of the backend's tests, as " - "scripts/nvx_tools/openvmm_test_shards.json splits them, such as " - "1/2; COUNT must be their number of shards" - ), - ) openvmm_tests.set_defaults(handler=command_test_openvmm) - openvmm_test_shards = subparsers.add_parser( - "plan-openvmm-test-shards", - help=( - "split the OpenVMM VMM tests into shards that nextest runs in " - "balanced times for CI" - ), - ) - configure_openvmm_test_shard_parser(openvmm_test_shards) - microvm_tests = subparsers.add_parser( "test-microvm", help="run NVX-owned OpenVMM microVM correctness tests", diff --git a/scripts/nvx_tools/ci.py b/scripts/nvx_tools/ci.py index 08f7292f7..b99984a51 100644 --- a/scripts/nvx_tools/ci.py +++ b/scripts/nvx_tools/ci.py @@ -3,25 +3,16 @@ from __future__ import annotations import hashlib -import json -import math import os import re -import shlex import shutil -import stat -import sys import zipfile from collections.abc import Mapping, Sequence from dataclasses import dataclass -from pathlib import Path, PurePosixPath -from typing import cast -from xml.etree import ElementTree +from pathlib import Path from .build_constants import ( AlpineBuildConstants, - BuildConstants, - GuestRustBuildConstants, KernelBuildConstants, OpenVMMBuildConstants, ZstdBuildConstants, @@ -29,89 +20,15 @@ from .common import ( ScriptError, artifact_path, - bytes_to_mib, download, - github_job, - is_link_or_junction, - openvmm_git_state, - path_exists, require_file, require_success, require_tool, run_capture, run_checked, - sha256_file, - strict_json_object, ) OPENVMM_TEST_BACKENDS = ("kvm", "mshv", "whp") -# The operating system that builds and runs each backend's VMM tests. -OPENVMM_TEST_OPERATING_SYSTEMS = {"kvm": "linux", "mshv": "linux", "whp": "windows"} -# `build-openvmm-tests` describes the test content that it builds in this file, -# which `test-openvmm --content` reads. Format 2 added the target, the build -# options, and the SHA-256 of each file. -OPENVMM_TEST_CONTENT_MANIFEST = "nvx-openvmm-tests.json" -OPENVMM_TEST_CONTENT_FORMAT = 2 -# The vmm-tests-run target of each operating system's content. NVX supports -# only x86-64 hosts, and the WHP tests run only windows-x64 content, so the -# build names its target rather than taking the host's. -OPENVMM_TEST_TARGETS = {"linux": "linux-x64", "windows": "windows-x64"} -# The vmm-tests-run options that build the content: a release build without -# optional features, whose run script runs the tests with the nextest CI -# profile and without the interactive VHD prompt. -OPENVMM_TEST_BUILD_OPTIONS = ( - "--release", - "--ci-profile", - "--skip-vhd-prompt", - "--build-only", -) -# CI caches the test content of each operating system across runs under a key -# of everything that determines it (doc/ci.md#openvmm-vmm-test-cache): this -# version, the target, the OpenVMM revision, and the SHA-256 of every field of -# the content's manifest that describes what built it. Increment the version -# when what a build makes, or what a reuse of the content checks, changes in a -# way that none of those fields captures, such as the environment in which the -# tests build. -OPENVMM_TEST_CACHE_VERSION = 1 -# The script that `vmm-tests-run --build-only` writes to run the content. -OPENVMM_TEST_RUN_SCRIPTS = {"linux": "run.sh", "windows": "run.ps1"} -# The nextest archive of the tests, in the content's directory of its target. -OPENVMM_TEST_ARCHIVE_NAME = "vmm_tests.tar.zst" -# Petri writes the results of a content run into this directory of the -# content, which also holds the run's test duration report. -OPENVMM_TEST_RESULTS = "test_results" -# A content run takes the remapped workspace from the content, so nextest -# writes the JUnit report that OpenVMM's nextest configuration gives the ci -# profile, which the content's runs use, under the content's target directory. -OPENVMM_TEST_JUNIT = PurePosixPath("target", "nextest", "ci", "junit.xml") -# `test-openvmm --content` records each test's duration in this report in its -# results, from which `plan-openvmm-test-shards` plans the shards -# (doc/usage.md#test-duration-report). -OPENVMM_TEST_DURATIONS_NAME = "openvmm-test-durations.json" -OPENVMM_TEST_DURATIONS_SCHEMA = "nvx-openvmm-test-durations" -# Changes only when a reader of the previous version would misread a report; -# added fields keep it. -OPENVMM_TEST_DURATIONS_SCHEMA_VERSION = 1 -# The images that the pinned vmm-tests-run-target downloads before it runs the -# tests, by the name that a run script passes with --artifacts, with the file -# and the size that petri_artifacts_vmm_test declares for each. They come from -# OpenVMM's public test image container, from which Petri streams every other -# image on demand. -OPENVMM_TEST_IMAGE_URL = "https://hvlitetestvhds.blob.core.windows.net/vhds" -OPENVMM_TEST_IMAGES = {"VmgsWithBootEntry": ("sample-vmgs.vhd", 4194816)} -# The vmm-tests-run-target options that the pinned run scripts pass, and -# whether each takes a value. The consumer passes its own --dir and --filter. -OPENVMM_TEST_RUN_TARGET_OPTIONS = { - "--target": True, - "--repetitions": True, - "--artifacts": True, - "--prep-steps": True, - "--skip-vhd-prompt": False, - "--ci-profile": False, - "--no-reuse-prepped-vhds": False, - "--needs-hardware-isolation": False, - "--needs-igvm-agent": False, -} REQUIRED_CI_RESULT_ENVIRONMENTS = { "quality": "QUALITY_RESULT", "nvx-rs": "NVX_RS_RESULT", @@ -127,11 +44,7 @@ "build-openvmm-linux-gnu": "BUILD_LINUX_GNU_RESULT", "build-openvmm-linux-musl": "BUILD_LINUX_MUSL_RESULT", "build-openvmm-windows-msvc": "BUILD_WINDOWS_MSVC_RESULT", - "build-openvmm-vmm-tests-linux": "BUILD_VMM_TESTS_LINUX_RESULT", - "build-openvmm-vmm-tests-windows": "BUILD_VMM_TESTS_WINDOWS_RESULT", - "openvmm-vmm-tests-linux": "VMM_TESTS_LINUX_RESULT", - "openvmm-vmm-tests-windows": "VMM_TESTS_WINDOWS_RESULT", - "openvmm-unit-tests": "UNIT_TESTS_RESULT", + "openvmm-ci": "OPENVMM_CI_RESULT", "nvx-microvm-tests-kvm": "MICROVM_KVM_RESULT", "nvx-microvm-tests-mshv": "MICROVM_MSHV_RESULT", "nvx-microvm-tests-whp": "MICROVM_WHP_RESULT", @@ -163,19 +76,10 @@ # GitHub-hosted, once per OS, after the nvx-rs build of its OS; no hypervisor # job waits for it. REQUIRED_CI_CLI_JOBS = ("nvx-cli-linux", "nvx-cli-windows") -REQUIRED_CI_OPENVMM_TEST_JOBS = ("openvmm-unit-tests",) -# OpenVMM VMM tests boot the NVX guest artifacts, so they need the job that -# builds them. -REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS = ( - "openvmm-vmm-tests-linux", - "openvmm-vmm-tests-windows", -) -# Each builds, without the guest artifacts, the VMM tests that the test jobs of -# its OS run, so it runs whenever a change selects the VMM tests. -REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS = ( - "build-openvmm-vmm-tests-linux", - "build-openvmm-vmm-tests-windows", -) +# GitHub-hosted; checks that OpenVMM's own CI passed the pinned revision +# (doc/ci.md#openvmm-ci), which needs no secret, so it runs for pull requests +# from forks too. +REQUIRED_CI_OPENVMM_CI_JOB = "openvmm-ci" # Each backend runs the default suite in one job and, beside it, the guest # smoke and sandbox steps that follow the suite in another. REQUIRED_CI_MICROVM_TEST_JOBS = ( @@ -247,8 +151,7 @@ CI_CHANGE_OUTPUTS = ( "run-guest-artifacts", "run-openvmm-binaries", - "run-openvmm-unit-tests", - "run-openvmm-vmm-tests", + "run-openvmm-ci-check", "run-microvm-tests", "run-benchmarks", ) @@ -270,7 +173,7 @@ r"scripts/nvx\.py$|" r"scripts/nvx_tools/(?!(adversarial|adversarial_broker|adversarial_executor|" r"crate_release|create_linux_source_archive|merge_queue|nightly_ci|" - r"performance|performance_history)\.py$|adversarial_cases/))" + r"openvmm_ci|performance|performance_history)\.py$|adversarial_cases/))" ) # Inputs of the platform jobs, which benchmark each backend, of the package # jobs, which package its release, and of the regression gate that reads the @@ -292,39 +195,20 @@ r"scripts/nvx\.py$|" r"scripts/nvx_tools/(?!(adversarial|adversarial_broker|adversarial_executor|" r"crate_release|create_linux_source_archive|managed_exec_tests|merge_queue|" - r"microvm_durations|microvm_shards|microvm_tests|nightly_ci|" - r"openvmm_process|openvmm_test_shards|performance_history|" - r"runtime_adapter_tests|sandbox_lifecycle_tests|sandbox_proxy_tests|" - r"sandbox_ram_upper_tests|shard_plans)\.py$|" - r"(microvm|openvmm_test)_shards\.json$|adversarial_cases/|" + r"microvm_durations|microvm_shards|microvm_tests|nightly_ci|openvmm_ci|" + r"openvmm_process|performance_history|runtime_adapter_tests|" + r"sandbox_lifecycle_tests|sandbox_proxy_tests|sandbox_ram_upper_tests|" + r"shard_plans)\.py$|" + r"microvm_shards\.json$|adversarial_cases/|" r"microvm_test_scripts/))" ) -# Inputs of the OpenVMM unit and documentation tests: the pinned revision, the -# Rust release, the runner setup, and the code and workflow that run them. -OPENVMM_UNIT_TEST_INPUT_PATHS = re.compile( - r"^(openvmm($|/)|rust-toolchain\.toml$|scripts/setup/|" - r"\.github/workflows/ci\.yml$|" - r"\.github/actions/(checkout-openvmm|sccache|setup-curl|validate-runner)/|" - r"scripts/nvx\.py$|scripts/nvx_tools/(__init__|build_constants|ci|common)\.py$)" -) -# The OpenVMM VMM tests also boot the NVX kernel and Alpine initramfs, so they -# add the inputs of those artifacts' cache keys and of the job that builds them, -# and the modules and the partition that split the WHP tests into shards. -# Of nvx-rs, the Alpine key covers only the files that the guest binaries' build -# reads, through their source identity, so a change to its host side or tests -# leaves the VMM tests out. -OPENVMM_VMM_TEST_INPUT_PATHS = re.compile( - r"^(openvmm($|/)|rust-toolchain\.toml$|scripts/setup/|" - r"\.github/workflows/ci\.yml$|" - r"\.github/actions/(build-guest-artifacts|checkout-openvmm|sccache|" - r"setup-curl|validate-runner)/|" - r"scripts/nvx\.py$|" +# Inputs of the check that OpenVMM's own CI passed the pinned revision +# (doc/ci.md#openvmm-ci): the pin, and the workflow and the CLI modules that +# run the check. +OPENVMM_CI_CHECK_INPUT_PATHS = re.compile( + r"^(openvmm($|/)|\.github/workflows/ci\.yml$|scripts/nvx\.py$|" r"scripts/nvx_tools/" - r"(__init__|build|build_config|build_constants|ci|common|elf|" - r"guest_binaries|guests|openvmm_test_shards|shard_plans)\.py$|" - r"scripts/nvx_tools/openvmm_test_shards\.json$|" - r"SOURCE-MANIFEST\.json$|docker/Dockerfile$|\.dockerignore$|kernel/|guest/(alpine|common)/|" - rf"(?:{GuestRustBuildConstants.SOURCE_PATTERN.pattern})$)" + r"(__init__|build_constants|common|nightly_ci|openvmm_ci)\.py$)" ) @@ -334,8 +218,7 @@ class CiChanges: run_guest_artifacts: bool run_openvmm_binaries: bool - run_openvmm_unit_tests: bool - run_openvmm_vmm_tests: bool + run_openvmm_ci_check: bool run_microvm_tests: bool run_benchmarks: bool @@ -346,8 +229,7 @@ def outputs(self) -> dict[str, bool]: ( self.run_guest_artifacts, self.run_openvmm_binaries, - self.run_openvmm_unit_tests, - self.run_openvmm_vmm_tests, + self.run_openvmm_ci_check, self.run_microvm_tests, self.run_benchmarks, ), @@ -398,7 +280,8 @@ def classify_ci_changes(event_name: str, changed_files: Sequence[str]) -> CiChan their inputs changed, but a push runs them for every change but documentation, so that every such `dev` commit publishes its development release and records its performance results. The guest artifacts and the - OpenVMM binaries are built exactly when a job that uses them runs. + OpenVMM binaries are built exactly when a job that uses them runs, and + OpenVMM's own CI is checked whenever the pin or the check changes. """ if event_name not in CI_CLASSIFIED_EVENTS: raise ValueError(f"unsupported CI event {event_name!r}") @@ -409,7 +292,6 @@ def classify_ci_changes(event_name: str, changed_files: Sequence[str]) -> CiChan def changed(pattern: re.Pattern[str]) -> bool: return any(pattern.match(path) for path in inputs) - run_openvmm_vmm_tests = changed(OPENVMM_VMM_TEST_INPUT_PATHS) run_microvm_tests = changed(MICROVM_TEST_INPUT_PATHS) # An empty diff runs the platform and package jobs, as it ran every # workload before the classifier selected them separately. @@ -419,12 +301,9 @@ def changed(pattern: re.Pattern[str]) -> bool: or (event_name == "push" and bool(inputs)) ) return CiChanges( - run_guest_artifacts=run_openvmm_vmm_tests - or run_microvm_tests - or run_benchmarks, + run_guest_artifacts=run_microvm_tests or run_benchmarks, run_openvmm_binaries=run_microvm_tests or run_benchmarks, - run_openvmm_unit_tests=changed(OPENVMM_UNIT_TEST_INPUT_PATHS), - run_openvmm_vmm_tests=run_openvmm_vmm_tests, + run_openvmm_ci_check=changed(OPENVMM_CI_CHECK_INPUT_PATHS), run_microvm_tests=run_microvm_tests, run_benchmarks=run_benchmarks, ) @@ -488,7 +367,6 @@ def users_reason(users: dict[str, bool]) -> str: changes.run_guest_artifacts, users_reason( { - "the OpenVMM VMM tests": changes.run_openvmm_vmm_tests, "the NVX microVM tests": changes.run_microvm_tests, "the platform benchmarks and packages": changes.run_benchmarks, }, @@ -505,17 +383,10 @@ def users_reason(users: dict[str, bool]) -> str: ), ), section( - "OpenVMM unit tests", - changes.run_openvmm_unit_tests, + "OpenVMM CI check", + changes.run_openvmm_ci_check, inputs_reason( - changes.run_openvmm_unit_tests, OPENVMM_UNIT_TEST_INPUT_PATHS - ), - ), - section( - "OpenVMM VMM tests", - changes.run_openvmm_vmm_tests, - inputs_reason( - changes.run_openvmm_vmm_tests, OPENVMM_VMM_TEST_INPUT_PATHS + changes.run_openvmm_ci_check, OPENVMM_CI_CHECK_INPUT_PATHS ), ), section( @@ -576,6 +447,10 @@ def required_ci_expected_results( and run_benchmarks else "skipped" ), + # The check needs no secret, so a pull request from a fork runs it too. + REQUIRED_CI_OPENVMM_CI_JOB: ( + "success" if changes.run_openvmm_ci_check else "skipped" + ), } expected.update( { @@ -593,38 +468,6 @@ def required_ci_expected_results( for job in REQUIRED_CI_BUILD_JOBS } ) - expected.update( - { - job: ( - "success" - if repository_jobs_enabled and changes.run_openvmm_unit_tests - else "skipped" - ) - for job in REQUIRED_CI_OPENVMM_TEST_JOBS - } - ) - expected.update( - { - job: ( - "success" - if repository_jobs_enabled and changes.run_openvmm_vmm_tests - else "skipped" - ) - for job in REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS - } - ) - expected.update( - { - job: ( - "success" - if repository_jobs_enabled - and changes.run_openvmm_vmm_tests - and run_guest_artifacts - else "skipped" - ) - for job in REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS - } - ) expected.update( { job: ( @@ -750,9 +593,8 @@ def _exclude_openvmm_tests( ) # ttrpc::test_ttrpc_interface stays on the Linux backends: it boots Linux # pipette, which flowey can build only on a Linux host. No test needs Hyper-V, -# which flowey reads from "hyperv" in a test's name, because test-openvmm runs -# the WHP tests that a build hands over without vmm-tests-run-target, which -# checks for Hyper-V on Windows. +# which flowey reads from "hyperv" in a test's name, so that vmm-tests-run +# requires no Hyper-V on the WHP host. OPENVMM_WHP_TESTS = ( "multiarch::hibernate::openvmm_uefi_x64_guest_test_x64_hibernate_halts", "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2022_x64_timesync_ic", @@ -795,50 +637,11 @@ def _join_openvmm_tests( return f"({test_filter}){exclusions}" -def _select_openvmm_tests( - tests: tuple[str, ...], - excluded_tests: tuple[str, ...], -) -> tuple[str, ...]: - return tuple(test for test in tests if test not in excluded_tests) - - OPENVMM_TEST_FILTERS = { "kvm": OPENVMM_KVM_TEST_FILTER, "mshv": OPENVMM_MSHV_TEST_FILTER, "whp": _join_openvmm_tests(OPENVMM_WHP_TESTS, OPENVMM_WHP_EXCLUDED_TESTS), } -# The tests that a backend's filter selects by their exact names, which is -# what a partition of its tests into shards lists. The KVM and MSHV filters -# select tests by pattern, so only the WHP tests have shards -# (doc/ci.md#openvmm-vmm-test-shards). -OPENVMM_TEST_SELECTIONS = { - "whp": _select_openvmm_tests(OPENVMM_WHP_TESTS, OPENVMM_WHP_EXCLUDED_TESTS), -} - - -@dataclass(frozen=True) -class OpenVMMTestShard: - """Shard INDEX of the COUNT shards of a backend's VMM tests, with the - tests that it runs.""" - - index: int - count: int - tests: tuple[str, ...] - - -def openvmm_test_filter(backend: str, shard: OpenVMMTestShard | None = None) -> str: - """Return the filter of the backend's tests, or of those of its tests that - a shard runs: only tests that the backend's filter selects, whatever the - shard lists.""" - test_filter = OPENVMM_TEST_FILTERS[backend] - if shard is None: - return test_filter - if not shard.tests: - raise ScriptError( - f"shard {shard.index} of {shard.count} of the {backend.upper()} " - "OpenVMM tests runs no test" - ) - return f"({test_filter}) & {_join_openvmm_tests(shard.tests)}" def validate_openvmm_test_backend(backend: str) -> None: @@ -1026,21 +829,9 @@ def _openvmm_microvm_test_guest() -> tuple[Path, Path]: return kernel, initrd -def run_openvmm_tests( - backend: str, - content: Path | None = None, - *, - shard: OpenVMMTestShard | None = None, -) -> None: - """Run the backend's VMM tests, or those of a shard of them, building them - unless ``content`` names a test content directory that - ``build_openvmm_tests`` built.""" +def run_openvmm_tests(backend: str) -> None: + """Build and run the backend's VMM tests.""" validate_openvmm_test_backend(backend) - test_filter = openvmm_test_filter(backend, shard) - if content is not None: - _run_openvmm_test_content(backend, content, shard, test_filter) - return - require_file( OpenVMMBuildConstants.DIRECTORY / "Cargo.toml", "initialized OpenVMM submodule" ) @@ -1064,7 +855,7 @@ def run_openvmm_tests( "--ci-profile", "--skip-vhd-prompt", "--filter", - test_filter, + OPENVMM_TEST_FILTERS[backend], ] if os.name == "nt": command.extend( @@ -1078,752 +869,6 @@ def run_openvmm_tests( run_checked(command, cwd=OpenVMMBuildConstants.DIRECTORY, env=rust_environment) -def openvmm_test_build_filter(backends: Sequence[str]) -> str: - """Return the filter that selects every test of the backends' filters.""" - filters = [ - OPENVMM_TEST_FILTERS[backend] - for backend in OPENVMM_TEST_BACKENDS - if backend in backends - ] - if len(filters) == 1: - return filters[0] - return " | ".join(f"({test_filter})" for test_filter in filters) - - -def _openvmm_test_content_backends(backends: Sequence[str]) -> tuple[str, ...]: - if not backends: - raise ScriptError("select at least one OpenVMM test backend") - for backend in backends: - if backend not in OPENVMM_TEST_BACKENDS: - choices = ", ".join(OPENVMM_TEST_BACKENDS) - raise ScriptError( - f"unsupported OpenVMM test backend {backend!r}; choose {choices}" - ) - host = "windows" if os.name == "nt" else "linux" - for backend in backends: - if OPENVMM_TEST_OPERATING_SYSTEMS[backend] != host: - raise ScriptError( - f"{backend.upper()} OpenVMM tests build on " - f"{OPENVMM_TEST_OPERATING_SYSTEMS[backend].capitalize()}" - ) - return tuple(backend for backend in OPENVMM_TEST_BACKENDS if backend in backends) - - -def _content_relative_path(value: object) -> PurePosixPath | None: - """Return a path inside a test content directory, or None if ``value`` - names none. - - Manifests and run scripts name files with forward slashes, so a backslash - or a drive, which Windows would resolve elsewhere, names no such path. - """ - if not isinstance(value, str) or "\\" in value or ":" in value: - return None - path = PurePosixPath(value) - if not path.parts or path.is_absolute() or ".." in path.parts: - return None - return path - - -def _openvmm_test_run_target(script: Path, build_filter: str) -> tuple[str, list[str]]: - """Return the runner and the options other than --dir and --filter of the - vmm-tests-run-target command that a build-only run script runs. - - The pinned flowey writes the command after a `cd` to the script's - directory, as its arguments joined with spaces, and quotes only the filter. - """ - text = require_file(script, "OpenVMM test run script").read_text(encoding="utf-8") - quoted_filter = f" --filter '{build_filter}'" - _, separator, command = text.strip().partition(" ; ") - if not separator or command.count(quoted_filter) != 1: - raise ScriptError(f"{script} does not run the filter that it was built for") - tokens = command.replace(quoted_filter, "", 1).split() - if tokens[1:4] != ["pipeline", "run", "vmm-tests-run-target"]: - raise ScriptError(f"{script} does not run vmm-tests-run-target") - runner = tokens[0] - runner_path = _content_relative_path(runner[2:].replace("\\", "/")) - if not runner.startswith(("./", ".\\")) or runner_path is None: - raise ScriptError(f"{script} runs {runner}, outside its directory") - arguments: list[str] = [] - directory: str | None = None - options = iter(tokens[4:]) - for option in options: - if option == "--dir": - directory = next(options, None) - continue - takes_value = OPENVMM_TEST_RUN_TARGET_OPTIONS.get(option) - if takes_value is None: - raise ScriptError(f"{script} passes unsupported option {option}") - arguments.append(option) - if takes_value: - value = next(options, None) - if value is None or value.startswith("--"): - raise ScriptError(f"{script} passes {option} without a value") - arguments.append(value) - if directory != "." or "--target" not in arguments: - raise ScriptError(f"{script} does not run the tests in its own directory") - return runner_path.as_posix(), arguments - - -def _openvmm_test_content_files(directory: Path) -> list[dict[str, object]]: - files: list[dict[str, object]] = [] - paths = { - path.relative_to(directory).as_posix(): path for path in directory.rglob("*") - } - for relative, path in sorted(paths.items()): - metadata = path.lstat() - if stat.S_ISDIR(metadata.st_mode) or relative == OPENVMM_TEST_CONTENT_MANIFEST: - continue - if not stat.S_ISREG(metadata.st_mode): - raise ScriptError(f"OpenVMM test content has a non-regular file {relative}") - files.append( - { - "path": relative, - "size": metadata.st_size, - # Artifact uploads drop file modes, so the consumer restores - # these. - "executable": os.name != "nt" and bool(metadata.st_mode & 0o111), - "sha256": sha256_file(path), - } - ) - return files - - -def _openvmm_gitlink() -> str: - """Return the OpenVMM revision that the checkout's index pins.""" - gitlink = run_capture( - ["git", "-C", BuildConstants.REPO_ROOT, "rev-parse", ":openvmm"] - ) - require_success(gitlink, "OpenVMM gitlink query") - return gitlink.stdout.decode("ascii").strip() - - -def _read_openvmm_test_manifest(path: Path) -> dict[str, object]: - try: - document = cast( - object, - json.loads( - path.read_text(encoding="utf-8"), - object_pairs_hook=strict_json_object, - ), - ) - except (UnicodeDecodeError, json.JSONDecodeError) as error: - raise ScriptError(f"{path} is not valid JSON: {error}") from error - if not isinstance(document, dict): - raise ScriptError(f"{path} is not a JSON object") - return cast(dict[str, object], document) - - -def _openvmm_test_content_inputs( - selected: Sequence[str], revision: str, clean: bool -) -> dict[str, object]: - """Return the fields of the manifest of the content that - ``build_openvmm_tests`` builds for the selected backends from an OpenVMM - checkout at ``revision`` that describe what built it: everything that - determines the content but the build itself.""" - operating_system = OPENVMM_TEST_OPERATING_SYSTEMS[selected[0]] - return { - "format": OPENVMM_TEST_CONTENT_FORMAT, - "openvmm_revision": revision, - "openvmm_clean": clean, - "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, - "operating_system": operating_system, - "target": OPENVMM_TEST_TARGETS[operating_system], - "build_options": list(OPENVMM_TEST_BUILD_OPTIONS), - "filters": {backend: OPENVMM_TEST_FILTERS[backend] for backend in selected}, - "build_filter": openvmm_test_build_filter(selected), - } - - -def build_openvmm_tests(backends: Sequence[str], output: Path) -> None: - """Build, without running them, the VMM tests of every named backend into - a test content directory that ``run_openvmm_tests`` runs on another host. - - The directory holds what OpenVMM's `vmm-tests-run-target` needs, and a - manifest that binds it to the OpenVMM revision, the Rust release, the - target, the build options, and the backends' filters that built it, and - lists the size and SHA-256 of each of its files. - """ - selected = _openvmm_test_content_backends(backends) - require_file( - OpenVMMBuildConstants.DIRECTORY / "Cargo.toml", "initialized OpenVMM submodule" - ) - if path_exists(output) and (not output.is_dir() or any(output.iterdir())): - raise ScriptError(f"{output} is not an empty directory") - cargo = require_tool("cargo") - rustup = require_tool("rustup") - revision, clean = openvmm_git_state(OpenVMMBuildConstants.DIRECTORY) - operating_system = OPENVMM_TEST_OPERATING_SYSTEMS[selected[0]] - target = OPENVMM_TEST_TARGETS[operating_system] - build_filter = openvmm_test_build_filter(selected) - if "'" in build_filter: - raise ScriptError("OpenVMM run scripts cannot quote a filter with a quote") - - environment = _prepare_openvmm_test_environment(selected, rustup) - run_checked( - [cargo, "xflowey", "restore-packages", "--no-compat-igvm"], - cwd=OpenVMMBuildConstants.DIRECTORY, - env=environment, - ) - output.mkdir(parents=True, exist_ok=True) - run_checked( - [ - cargo, - "xflowey", - "vmm-tests-run", - "--target", - target, - *OPENVMM_TEST_BUILD_OPTIONS, - "--filter", - build_filter, - "--dir", - os.fspath(output.resolve()), - ], - cwd=OpenVMMBuildConstants.DIRECTORY, - env=environment, - ) - - script = output / OPENVMM_TEST_RUN_SCRIPTS[operating_system] - runner, arguments = _openvmm_test_run_target(script, build_filter) - if arguments[arguments.index("--target") + 1] != target: - raise ScriptError(f"{script} does not run the {target} tests that it built") - files = _openvmm_test_content_files(output) - manifest = { - **_openvmm_test_content_inputs(selected, revision, clean), - "runner": runner, - "arguments": arguments, - "files": files, - } - (output / OPENVMM_TEST_CONTENT_MANIFEST).write_text( - json.dumps(manifest, indent=2) + "\n", encoding="utf-8" - ) - size = sum(cast(int, entry["size"]) for entry in files) - print( - f"Built the {', '.join(backend.upper() for backend in selected)} OpenVMM " - f"tests into {output}: {len(files)} files, {bytes_to_mib(size):.1f} MiB" - ) - - -@dataclass(frozen=True) -class _OpenVMMTestCacheEntry: - """The content of some backends that CI caches: a clean build of the - OpenVMM revision that the checkout pins, the fields of its manifest that - describe what built it, and its cache key.""" - - backends: tuple[str, ...] - inputs: Mapping[str, object] - key: str - - def describe(self) -> str: - return ", ".join(backend.upper() for backend in self.backends) - - -def _openvmm_test_cache_entry(backends: Sequence[str]) -> _OpenVMMTestCacheEntry: - selected = _openvmm_test_content_backends(backends) - inputs = _openvmm_test_content_inputs(selected, _openvmm_gitlink(), True) - canonical = json.dumps(inputs, sort_keys=True, separators=(",", ":")) - digest = hashlib.sha256(canonical.encode("utf-8")).hexdigest() - return _OpenVMMTestCacheEntry( - backends=selected, - inputs=inputs, - key=( - f"openvmm-vmm-tests-v{OPENVMM_TEST_CACHE_VERSION}-{inputs['target']}-" - f"{inputs['openvmm_revision']}-{digest}" - ), - ) - - -def _set_step_output(environment: Mapping[str, str], name: str, value: str) -> None: - """Give the GitHub Actions step that runs the command an output.""" - path = environment.get("GITHUB_OUTPUT") - if path: - with Path(path).open("a", encoding="utf-8", newline="\n") as stream: - stream.write(f"{name}={value}\n") - - -def openvmm_test_cache_key( - backends: Sequence[str], environment: Mapping[str, str] | None = None -) -> str: - """Print and return the key under which CI caches the content that - ``build_openvmm_tests`` builds for the backends from this checkout, and - give the step that asks for it the `key` output.""" - key = _openvmm_test_cache_entry(backends).key - print(key) - _set_step_output(environment or {}, "key", key) - return key - - -def _brief(value: object) -> str: - text = json.dumps(value) - return text if len(text) <= 80 else f"{text[:77]}..." - - -def _openvmm_test_file_refusal( - listed: object, files: Sequence[Mapping[str, object]] -) -> str | None: - """Return how a manifest's list of files differs from the files of its - content, as _openvmm_test_content_files lists them, or None if it lists - exactly those files.""" - if listed == files: - return None - present = {str(entry["path"]): entry for entry in files} - recorded = { - str(cast(dict[str, object], entry).get("path")): cast(dict[str, object], entry) - for entry in (cast(list[object], listed) if isinstance(listed, list) else []) - if isinstance(entry, dict) - } - missing = sorted(recorded.keys() - present.keys()) - if missing: - return f"it lacks {missing[0]}, which its manifest lists" - unlisted = sorted(present.keys() - recorded.keys()) - if unlisted: - return f"its manifest does not list {unlisted[0]}" - for path, entry in sorted(present.items()): - for field in ("size", "sha256", "executable"): - if recorded[path].get(field) != entry[field]: - return f"the {field} of {path} differs from its manifest's" - return "its manifest does not list exactly its files" - - -def _openvmm_test_content_refusal( - entry: _OpenVMMTestCacheEntry, content: Path -) -> str | None: - """Return why ``content`` does not hold the test content that ``entry`` - describes, or None if it holds exactly that: a manifest that records the - entry's fields, and the runner and options that the content's run script - runs, and exactly the files that the manifest lists, with their sizes, - SHA-256, and executable bits.""" - path = content / OPENVMM_TEST_CONTENT_MANIFEST - if not path.is_file(): - return f"it has no {OPENVMM_TEST_CONTENT_MANIFEST}" - try: - manifest = _read_openvmm_test_manifest(path) - for field, expected in entry.inputs.items(): - if manifest.get(field) != expected: - return ( - f"its manifest records the {field} {_brief(manifest.get(field))}" - f", not {_brief(expected)}" - ) - operating_system = cast(str, entry.inputs["operating_system"]) - script = content / OPENVMM_TEST_RUN_SCRIPTS[operating_system] - runner, arguments = _openvmm_test_run_target( - script, cast(str, entry.inputs["build_filter"]) - ) - if arguments[arguments.index("--target") + 1] != entry.inputs["target"]: - return f"{script.name} runs the tests of another target" - if manifest.get("runner") != runner or manifest.get("arguments") != arguments: - return ( - f"its manifest records another runner or other options than " - f"{script.name} runs" - ) - files = _openvmm_test_content_files(content) - except (OSError, ScriptError, UnicodeDecodeError) as error: - return str(error) - return _openvmm_test_file_refusal(manifest.get("files"), files) - - -def check_openvmm_tests(backends: Sequence[str], content: Path) -> None: - """Fail unless ``content`` holds exactly the content that - ``build_openvmm_tests`` builds for the backends from this checkout, which - CI caches under ``openvmm_test_cache_key``.""" - entry = _openvmm_test_cache_entry(backends) - refusal = _openvmm_test_content_refusal(entry, content) - if refusal is not None: - raise ScriptError( - f"{content} does not hold the {entry.describe()} OpenVMM tests of " - f"cache key {entry.key}: {refusal}" - ) - print( - f">> {content} holds the {entry.describe()} OpenVMM tests of cache key " - f"{entry.key}" - ) - - -def _annotation(text: str) -> str: - return text.replace("%", "%25").replace("\r", "%0D").replace("\n", "%0A") - - -def reuse_openvmm_tests( - backends: Sequence[str], - content: Path, - environment: Mapping[str, str] | None = None, -) -> bool: - """Keep ``content``, which a cache restored, if it holds exactly the - content that ``build_openvmm_tests`` builds for the backends from this - checkout, and remove whatever it holds otherwise, so that a build can take - its place. Report the outcome, warn of a refusal, give the step the - `reused` output, and return whether the content is reused.""" - entry = _openvmm_test_cache_entry(backends) - if not path_exists(content): - reused = False - print( - f">> {content} holds no {entry.describe()} OpenVMM tests of cache key " - f"{entry.key} to reuse" - ) - else: - refusal = _openvmm_test_content_refusal(entry, content) - reused = refusal is None - if reused: - print( - f">> reusing the {entry.describe()} OpenVMM tests of cache key " - f"{entry.key} in {content}" - ) - else: - detail = ( - f"{content} does not hold the {entry.describe()} OpenVMM tests of " - f"cache key {entry.key}: {refusal}. Removed it, so that the tests " - "build instead." - ) - print(f"::warning title=OpenVMM test cache::{_annotation(detail)}") - if content.is_dir() and not is_link_or_junction(content): - shutil.rmtree(content) - else: - content.unlink() - _set_step_output(environment or {}, "reused", "true" if reused else "false") - return reused - - -@dataclass(frozen=True) -class _OpenVMMTestContent: - """What runs the tests of a checked test content directory.""" - - runner: Path - arguments: tuple[str, ...] - files: tuple[PurePosixPath, ...] - - -def _openvmm_test_content(content: Path, backend: str) -> _OpenVMMTestContent: - """Check that the content runs this checkout's tests of the backend, and - return its runner, its vmm-tests-run-target options, and its files.""" - path = require_file( - content / OPENVMM_TEST_CONTENT_MANIFEST, "OpenVMM test content manifest" - ) - manifest = _read_openvmm_test_manifest(path) - if manifest.get("format") != OPENVMM_TEST_CONTENT_FORMAT: - raise ScriptError( - f"{path} has format {manifest.get('format')!r}, " - f"expected {OPENVMM_TEST_CONTENT_FORMAT}" - ) - revision = _openvmm_gitlink() - if manifest.get("openvmm_revision") != revision: - raise ScriptError( - f"{path} describes tests of OpenVMM " - f"{manifest.get('openvmm_revision')}, but this checkout pins {revision}" - ) - filters = manifest.get("filters") - if ( - not isinstance(filters, dict) - or cast(dict[str, object], filters).get(backend) - != OPENVMM_TEST_FILTERS[backend] - ): - raise ScriptError( - f"{path} describes no tests of this checkout's {backend.upper()} filter" - ) - - runner = _content_relative_path(manifest.get("runner")) - arguments = manifest.get("arguments") - files = manifest.get("files") - if ( - runner is None - or not isinstance(arguments, list) - or not all(isinstance(item, str) for item in cast(list[object], arguments)) - or not isinstance(files, list) - ): - raise ScriptError(f"{path} is malformed") - listed: list[PurePosixPath] = [] - for entry in cast(list[object], files): - details = cast(dict[str, object], entry) if isinstance(entry, dict) else {} - relative = _content_relative_path(details.get("path")) - if relative is None: - raise ScriptError(f"{path} lists a file outside its directory") - file = content.joinpath(*relative.parts) - if not file.is_file() or file.stat().st_size != details.get("size"): - raise ScriptError(f"OpenVMM test content in {content} lacks {relative}") - # Artifact downloads drop the file modes that the build recorded. - if details.get("executable") is True: - file.chmod(file.stat().st_mode | 0o111) - listed.append(relative) - return _OpenVMMTestContent( - runner=content.joinpath(*runner.parts), - arguments=tuple(cast(list[str], arguments)), - files=tuple(listed), - ) - - -def _run_openvmm_test_content( - backend: str, - content: Path, - shard: OpenVMMTestShard | None, - test_filter: str, -) -> None: - kernel, initrd = _openvmm_microvm_test_guest() - root = content.resolve() - built = _openvmm_test_content(root, backend) - - environment = os.environ.copy() - environment["RUSTUP_TOOLCHAIN"] = OpenVMMBuildConstants.RUST_TOOLCHAIN - _set_openvmm_test_cache(environment, (backend,)) - environment["OPENVMM_MICROVM_TEST_KERNEL"] = os.fspath(kernel.resolve()) - environment["OPENVMM_MICROVM_TEST_INITRD"] = os.fspath(initrd.resolve()) - # An earlier run's reports must not pass for this run's. - junit = root.joinpath(*OPENVMM_TEST_JUNIT.parts) - durations = root / OPENVMM_TEST_RESULTS / OPENVMM_TEST_DURATIONS_NAME - for report in (junit, durations): - report.unlink(missing_ok=True) - try: - if OPENVMM_TEST_OPERATING_SYSTEMS[backend] == "windows": - _run_openvmm_test_archive(root, built, environment, test_filter) - else: - run_checked( - [ - built.runner, - "pipeline", - "run", - "vmm-tests-run-target", - *built.arguments, - "--dir", - os.fspath(root), - "--filter", - test_filter, - ], - cwd=root, - env=environment, - ) - finally: - _record_openvmm_test_durations(junit, durations, backend, shard, test_filter) - - -def _openvmm_test_junit_durations(junit: Path) -> list[dict[str, object]]: - """Return each test case of a nextest JUnit report: its binary, its test, - the seconds that it ran, and its outcome, which is passed, failed, flaky - for a test that passed only when retried, or skipped.""" - tests: list[dict[str, object]] = [] - for _, element in ElementTree.iterparse(junit): - if element.tag != "testcase": - continue - children = {child.tag for child in element} - if children & {"failure", "error"}: - outcome = "failed" - elif "skipped" in children: - outcome = "skipped" - elif children & {"flakyFailure", "flakyError"}: - outcome = "flaky" - else: - outcome = "passed" - name = element.get("name") - seconds = float(element.get("time", "nan")) - if not name or not math.isfinite(seconds) or seconds < 0: - raise ValueError(f"{junit} has a test case without a name or a time") - tests.append( - { - "binary": element.get("classname"), - "test": name, - "seconds": round(seconds, 3), - "outcome": outcome, - } - ) - # The cases keep the output of the tests, which the report drops. - element.clear() - return tests - - -def _record_openvmm_test_durations( - junit: Path, - report: Path, - backend: str, - shard: OpenVMMTestShard | None, - test_filter: str, -) -> None: - """Write the test duration report of a content run from the JUnit report - of its nextest run, if it wrote one, as the run ends, whether its tests - passed or not. A report that cannot be written only warns, so that it - never hides the run's own outcome.""" - if not junit.is_file(): - print(f">> recorded no test durations: nextest wrote no {junit}", flush=True) - return - try: - tests = _openvmm_test_junit_durations(junit) - document = { - "schema": OPENVMM_TEST_DURATIONS_SCHEMA, - "schema_version": OPENVMM_TEST_DURATIONS_SCHEMA_VERSION, - "backend": backend, - "shard": ( - None if shard is None else {"index": shard.index, "count": shard.count} - ), - "filter": test_filter, - # nextest runs as many test threads by default. - "logical_processors": os.cpu_count(), - "github": github_job(os.environ), - "tests": tests, - } - report.parent.mkdir(parents=True, exist_ok=True) - report.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") - except (OSError, ValueError, ElementTree.ParseError) as error: - print( - f"warning: could not record the test durations in {report}: {error}", - file=sys.stderr, - flush=True, - ) - return - print(f">> recorded the durations of {len(tests)} tests in {report}", flush=True) - - -@dataclass(frozen=True) -class _OpenVMMTestArchiveRun: - """What a run script's vmm-tests-run-target options ask of a run.""" - - profile: str - images: tuple[str, ...] - reuse_prepped_vhds: bool - - -def _openvmm_test_archive_run(arguments: Sequence[str]) -> _OpenVMMTestArchiveRun: - """Return what the vmm-tests-run-target options of Windows test content ask - of a run of its archive, refusing any that only the runner carries out.""" - target: str | None = None - profile = "default" - images: list[str] = [] - reuse_prepped_vhds = True - options = iter(arguments) - for option in options: - takes_value = OPENVMM_TEST_RUN_TARGET_OPTIONS.get(option) - if takes_value is None: - raise ScriptError( - f"OpenVMM test content passes unsupported option {option}" - ) - value = next(options, None) if takes_value else "" - if value is None or value.startswith("--"): - raise ScriptError(f"OpenVMM test content passes {option} without a value") - if option == "--target": - target = value - elif option == "--repetitions": - if value != "1": - raise ScriptError( - f"OpenVMM test content repeats its tests {value} times; " - "test-openvmm runs them once" - ) - elif option == "--artifacts": - images.extend(value.split(",")) - elif option == "--ci-profile": - profile = "ci" - elif option == "--no-reuse-prepped-vhds": - reuse_prepped_vhds = False - elif option != "--skip-vhd-prompt": - raise ScriptError( - f"OpenVMM test content passes {option}, which only its " - "vmm-tests-run-target carries out" - ) - if target != "windows-x64": - raise ScriptError( - f"OpenVMM test content targets {target}; test-openvmm runs the " - "archive of windows-x64 content" - ) - return _OpenVMMTestArchiveRun( - profile=profile, - images=tuple(images), - reuse_prepped_vhds=reuse_prepped_vhds, - ) - - -def _download_openvmm_test_images(names: Sequence[str], directory: Path) -> None: - """Download each named image into the directory, as vmm-tests-run-target - does, unless the directory holds it at its declared size.""" - for name in names: - image = OPENVMM_TEST_IMAGES.get(name) - if image is None: - known = ", ".join(sorted(OPENVMM_TEST_IMAGES)) - raise ScriptError( - f"OpenVMM test content needs the {name} image; " - f"test-openvmm downloads only {known}" - ) - filename, size = image - path = directory / filename - if path.is_file() and path.stat().st_size == size: - continue - print(f">> downloading {name} to {path}", flush=True) - download(f"{OPENVMM_TEST_IMAGE_URL}/{filename}", path) - actual = path.stat().st_size - if actual != size: - path.unlink() - raise ScriptError( - f"{filename} has {actual} bytes, but OpenVMM declares {size}" - ) - - -def _run_openvmm_test_archive( - root: Path, - content: _OpenVMMTestContent, - environment: dict[str, str], - test_filter: str, -) -> None: - """Run a filter of the WHP tests, or of a shard of them, from the nextest - archive of Windows test content, as its vmm-tests-run-target does after it - checks the host. - - The pinned vmm-tests-run-target requires Hyper-V on every Windows target, - whatever tests it runs: `Get-VM` must succeed and two registry values must - be set, or the account must be an administrator, which the WHP runners' - Network Service account is not. The WHP filter selects no Hyper-V test, so - `vmm-tests-run`, which derives the dependencies from the selected tests, - checks nothing there. This takes the runner's remaining steps: it downloads - the images that the content names, sets the environment that Petri reads, - and runs the archive's tests with cargo-nextest, the content's nextest - configuration, and the content as the workspace. - """ - run = _openvmm_test_archive_run(content.arguments) - archives = [ - path for path in content.files if path.name == OPENVMM_TEST_ARCHIVE_NAME - ] - if len(archives) != 1: - raise ScriptError( - f"OpenVMM test content in {root} holds {len(archives)} " - f"{OPENVMM_TEST_ARCHIVE_NAME} archives, expected 1" - ) - configuration = PurePosixPath(".config", "nextest.toml") - if configuration not in content.files: - raise ScriptError(f"OpenVMM test content in {root} lacks {configuration}") - nextest = require_tool("cargo-nextest") - - # The runner downloads the images where VMM_TEST_IMAGES names, by default - # in the content, and recreates the test results that Petri writes. - images = Path(os.environ.get("VMM_TEST_IMAGES") or root / "images").resolve() - _download_openvmm_test_images(run.images, images) - results = root / OPENVMM_TEST_RESULTS - if path_exists(results): - shutil.rmtree(results) - results.mkdir() - petri = { - "VMM_TESTS_CONTENT_DIR": os.fspath(root), - "TEST_OUTPUT_PATH": os.fspath(results), - "VMM_TEST_IMAGES": os.fspath(images), - # The runner sets these whatever its caller's environment holds. - "RUST_LOG": "trace", - "RUST_BACKTRACE": "1", - } - if run.reuse_prepped_vhds: - petri["PETRI_REUSE_PREPPED_VHDS"] = "1" - environment.update(petri) - command = [ - nextest, - "nextest", - "run", - "--profile", - run.profile, - "--config-file", - os.fspath(root.joinpath(*configuration.parts)), - "--workspace-remap", - os.fspath(root), - "--archive-file", - os.fspath(root.joinpath(*archives[0].parts)), - "--filter-expr", - test_filter, - ] - # Log what the runner logs before it runs the tests. - for name, value in sorted(petri.items()): - print(f">> {name}={value}", flush=True) - print(f">> {shlex.join(command)}", flush=True) - run_checked(command, cwd=root, env=environment) - - def setup_cross_os_cache() -> None: github_path_value = os.environ.get("GITHUB_PATH") runner_temp_value = os.environ.get("RUNNER_TEMP") diff --git a/scripts/nvx_tools/openvmm_ci.py b/scripts/nvx_tools/openvmm_ci.py index 138f80a4f..2b0dae2a1 100644 --- a/scripts/nvx_tools/openvmm_ci.py +++ b/scripts/nvx_tools/openvmm_ci.py @@ -3,12 +3,13 @@ NVX pins a revision of `feature/microvm/dev` in microsoft/openvmm, whose checkin gates test it: the OpenVMM PR workflow on each pull request into the branch that is ready for review, and the OpenVMM CI workflow on each push to -it. ``nvx.py check-openvmm-ci`` requires the jobs of those gates that cover +it. NVX therefore runs none of OpenVMM's unit or VMM tests itself. +``nvx.py check-openvmm-ci`` instead requires the jobs of those gates that cover NVX's use of OpenVMM, the x86-64 Linux and Windows unit tests and the VMM tests on KVM, MSHV, and Windows, to have passed for the pinned revision in either workflow, and waits while they run. The gates' other jobs, such as the VMM tests on TDX and SEV-SNP hosts, test OpenHCL and confidential VMs, which NVX does not -use. +use. See doc/ci.md, "OpenVMM CI". """ from __future__ import annotations @@ -310,7 +311,7 @@ def command_check_openvmm_ci(args: argparse.Namespace) -> None: ) raise ScriptError( f"OpenVMM's CI has not passed {len(failures)} required job(s) for " - f"{revision}" + f"{revision}; see doc/ci.md, 'OpenVMM CI'" ) @@ -323,7 +324,7 @@ def _minutes(value: str) -> int: def configure_parser(parser: argparse.ArgumentParser) -> None: parser.description = ( "Check that OpenVMM's checkin gates passed the jobs that cover NVX's use " - "of OpenVMM for the pinned OpenVMM revision." + "of OpenVMM for the pinned OpenVMM revision (doc/ci.md, 'OpenVMM CI')." ) parser.add_argument( "--revision", diff --git a/scripts/nvx_tools/openvmm_test_shards.json b/scripts/nvx_tools/openvmm_test_shards.json deleted file mode 100644 index 5e9a367d9..000000000 --- a/scripts/nvx_tools/openvmm_test_shards.json +++ /dev/null @@ -1,207 +0,0 @@ -{ - "schema": "nvx-openvmm-test-shards", - "schema_version": 1, - "runs": [ - "39713968358-1", - "39715958615-1", - "39716028034-1", - "39718025956-1", - "39720512693-1", - "39720626810-1", - "39722903378-1", - "39724121642-1", - "39725436024-1", - "39728229888-1", - "39730546661-1", - "39741214955-1", - "39748010375-1", - "39761820480-1", - "39765239533-1", - "39766030609-1", - "39766509178-1", - "39768324116-1", - "39770674661-1", - "39771039182-1", - "39771927579-1", - "39774840679-1", - "39775417095-1", - "39778084391-1", - "39779640214-1", - "39783750996-1", - "39785367867-1", - "39785687509-1", - "39788615461-1", - "39791725058-1", - "39792079980-1", - "39801222132-1", - "39804644654-1", - "39807968692-1", - "39810434285-1", - "39812919464-1", - "39814103094-1", - "39815307317-1", - "39816544623-1", - "39816751386-1", - "39817733469-1", - "39823426649-1", - "39825190876-1", - "39826000015-1", - "39826588708-1", - "39828359160-1", - "39830043548-1", - "39830463671-1", - "39830463671-2", - "39831856144-1", - "39833953233-1", - "39833953233-2", - "39836564329-1", - "39841208939-1", - "39848435878-1", - "39851502361-1", - "39854426558-1", - "39855161942-1", - "39857700369-1", - "39862961113-1", - "39863120108-1", - "39866075964-1", - "39866347297-1", - "39868696884-1", - "39868778944-1", - "39869628717-1", - "39870083243-1", - "39871252155-1", - "39872708539-1", - "39873201397-1", - "39873201397-2", - "39874730678-1", - "39875487073-1", - "39876782765-1", - "39876913472-1", - "39880936108-1", - "39885427627-1", - "39890286133-1", - "39890691692-1", - "39893418176-1", - "39893426769-1", - "39894193779-1", - "39894345948-1", - "39894653320-1", - "39894764614-1", - "39895930233-1", - "39896939957-1", - "39897516736-1", - "39898205771-1", - "39899320658-1", - "39899489736-1", - "39900515697-1", - "39900789578-1", - "39901517651-1", - "39902391325-1", - "39902391325-2", - "39902787227-1", - "39904491775-1", - "39905888230-1", - "39906871272-1", - "39906929824-1", - "39907015479-1", - "39909405359-1", - "39909557950-1", - "39911726969-1", - "39915908920-1", - "39916457661-1", - "39916499031-1", - "39918603473-1", - "39929511926-1", - "39934424243-1", - "39936291356-1", - "39937204135-1", - "39938176292-1", - "39938567427-1", - "39940176775-1", - "39940877632-1", - "39940877632-2", - "39943384839-1", - "39944037966-1", - "39944477453-1", - "39945048295-1", - "39945048295-2", - "39948600574-1" - ], - "backends": { - "whp": { - "test_threads": 8, - "seconds": { - "multiarch::hibernate::openvmm_uefi_x64_guest_test_x64_hibernate_halts": 3.9, - "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2022_x64_timesync_ic": 32.3, - "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic": 57.4, - "multiarch::openvmm_pcat_x64_freebsd_13_2_x64_boot_no_agent": 87.9, - "multiarch::openvmm_pcat_x64_freebsd_13_2_x64_iso_boot_no_agent": 83.9, - "multiarch::openvmm_pcat_x64_windows_datacenter_core_2022_x64_boot": 29.5, - "multiarch::openvmm_pcat_x64_windows_datacenter_core_2022_x64_boot_heavy": 24.5, - "multiarch::openvmm_uefi_x64_frontpage": 2.6, - "multiarch::openvmm_uefi_x64_guest_test_x64_guest_test_uefi": 9.7, - "multiarch::openvmm_uefi_x64_ubuntu_2504_server_x64_secure_boot_mismatched_template": 3.6, - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot": 22.0, - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot_heavy": 21.1, - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot_nvme": 36.2, - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_secure_boot": 30.4, - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_secure_boot_mismatched_template": 2.9, - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_hotplug": 54.2, - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_nvme_boot": 41.4, - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_root_emulation_multi_segment": 33.6, - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_root_emulation_single_segment": 41.2, - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_switches": 30.1, - "multiarch::vmgs::openvmm_uefi_x64_ubuntu_2504_server_x64_invalid_boot_entries": 3.3, - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_clear_vmgs": 28.9, - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_default_boot": 29.4, - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_invalid_boot_entries": 2.8, - "ttrpc::test_ttrpc_microvm_linux_direct_lifecycle_and_snapshot": 18.7, - "ttrpc::test_ttrpc_uefi_boot": 9.4, - "x86_64::microvm::openvmm_linux_x64_phase_1_lifecycle": 15.4, - "x86_64::openvmm_uefi_x64_guest_test_x64_crash_dump_on_triple_fault": 9.9, - "x86_64::openvmm_uefi_x64_windows_datacenter_core_2022_x64_battery_capacity": 26.6 - }, - "shards": [ - { - "seconds": 128.4, - "tests": [ - "multiarch::hibernate::openvmm_uefi_x64_guest_test_x64_hibernate_halts", - "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2022_x64_timesync_ic", - "multiarch::openvmm_pcat_x64_windows_datacenter_core_2022_x64_boot", - "multiarch::openvmm_pcat_x64_windows_datacenter_core_2022_x64_boot_heavy", - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot_heavy", - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_secure_boot_mismatched_template", - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_root_emulation_single_segment", - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_switches", - "multiarch::vmgs::openvmm_uefi_x64_ubuntu_2504_server_x64_invalid_boot_entries", - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_clear_vmgs", - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_default_boot", - "multiarch::vmgs::openvmm_uefi_x64_windows_datacenter_core_2022_x64_invalid_boot_entries", - "ttrpc::test_ttrpc_microvm_linux_direct_lifecycle_and_snapshot", - "x86_64::microvm::openvmm_linux_x64_phase_1_lifecycle", - "x86_64::openvmm_uefi_x64_guest_test_x64_crash_dump_on_triple_fault" - ] - }, - { - "seconds": 128.3, - "tests": [ - "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic", - "multiarch::openvmm_pcat_x64_freebsd_13_2_x64_boot_no_agent", - "multiarch::openvmm_pcat_x64_freebsd_13_2_x64_iso_boot_no_agent", - "multiarch::openvmm_uefi_x64_frontpage", - "multiarch::openvmm_uefi_x64_guest_test_x64_guest_test_uefi", - "multiarch::openvmm_uefi_x64_ubuntu_2504_server_x64_secure_boot_mismatched_template", - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot", - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_boot_nvme", - "multiarch::openvmm_uefi_x64_windows_datacenter_core_2022_x64_secure_boot", - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_hotplug", - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_nvme_boot", - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_root_emulation_multi_segment", - "ttrpc::test_ttrpc_uefi_boot", - "x86_64::openvmm_uefi_x64_windows_datacenter_core_2022_x64_battery_capacity" - ] - } - ] - } - } -} diff --git a/scripts/nvx_tools/openvmm_test_shards.py b/scripts/nvx_tools/openvmm_test_shards.py deleted file mode 100644 index 03364036c..000000000 --- a/scripts/nvx_tools/openvmm_test_shards.py +++ /dev/null @@ -1,587 +0,0 @@ -"""The shards of the OpenVMM VMM tests, which CI runs in parallel. - -doc/ci.md#openvmm-vmm-test-shards describes the partition, how this module -plans it from test duration reports, and when to plan it again. -""" - -from __future__ import annotations - -import argparse -import heapq -import json -import statistics -from collections.abc import Collection, Iterable, Mapping, Sequence -from dataclasses import dataclass -from pathlib import Path -from typing import cast - -from . import shard_plans -from .ci import ( - OPENVMM_TEST_BACKENDS, - OPENVMM_TEST_DURATIONS_NAME, - OPENVMM_TEST_DURATIONS_SCHEMA, - OPENVMM_TEST_DURATIONS_SCHEMA_VERSION, - OPENVMM_TEST_SELECTIONS, - OpenVMMTestShard, -) -from .common import ScriptError, strict_json_object - -PARTITION_PATH = Path(__file__).with_name("openvmm_test_shards.json") -SCHEMA = "nvx-openvmm-test-shards" -SCHEMA_VERSION = 1 -PLAN_ADVICE = ( - "plan the shards again with `nvx.py plan-openvmm-test-shards` " - "(doc/ci.md#openvmm-vmm-test-shards)" -) -# OpenVMM's .config/nextest.toml has a VMM test require 32 of nextest's test -# threads when its name holds very_heavy, 12 when it holds heavy, and 2 -# otherwise. nextest gives a test that requires more threads than it runs all -# of them, so that the test runs alone. -THREADS_REQUIRED = (("very_heavy", 32), ("heavy", 12)) -DEFAULT_THREADS_REQUIRED = 2 - - -def threads_required(test: str, test_threads: int) -> int: - """Return how many of nextest's test threads a VMM test takes.""" - required = next( - (threads for marker, threads in THREADS_REQUIRED if marker in test), - DEFAULT_THREADS_REQUIRED, - ) - return min(required, test_threads) - - -def run_tenths( - tests: Iterable[str], tenths: Mapping[str, int], test_threads: int -) -> int: - """Return how long nextest takes to run the tests, in tenths of a second, - from the duration of each. - - nextest starts the tests in name order, each as soon as it has the test - threads that the test requires, and never one before an earlier one: a - test that takes every thread waits for each earlier test to end, and each - later test waits for it. - """ - running: list[tuple[int, int]] = [] - free = test_threads - now = 0 - for test in sorted(tests): - required = threads_required(test, test_threads) - while free < required: - ended, threads = heapq.heappop(running) - now = max(now, ended) - free += threads - heapq.heappush(running, (now + tenths[test], required)) - free -= required - return max((ended for ended, _ in running), default=0) - - -@dataclass(frozen=True) -class BackendShards: - """One backend's partition of its tests: the test threads with which - nextest runs them, one for each logical processor of the backend's - runners, each test's median duration in seconds, and the tests of each - shard, in shard order.""" - - test_threads: int - seconds: Mapping[str, float] - shards: tuple[tuple[str, ...], ...] - - def run_seconds(self, tests: Iterable[str]) -> float: - """Return how long nextest takes to run some of the tests.""" - durations = { - test: shard_plans.tenths(duration) - for test, duration in self.seconds.items() - } - return run_tenths(tests, durations, self.test_threads) / 10 - - def shard_seconds(self) -> tuple[float, ...]: - """Return how long nextest takes to run each shard's tests.""" - return tuple(self.run_seconds(shard) for shard in self.shards) - - -@dataclass(frozen=True) -class Partition: - """The shards of each sharded backend, and the CI runs, as - `-`, whose reports gave the durations.""" - - runs: tuple[str, ...] - backends: Mapping[str, BackendShards] - - -def _backend_shards(value: object) -> BackendShards | None: - document = shard_plans.json_object(value, ("test_threads", "seconds", "shards")) - if document is None: - return None - test_threads = document["test_threads"] - raw_seconds = document["seconds"] - raw_shards = document["shards"] - if ( - isinstance(test_threads, bool) - or not isinstance(test_threads, int) - or test_threads < 1 - or not isinstance(raw_seconds, dict) - or not isinstance(raw_shards, list) - ): - return None - seconds: dict[str, float] = {} - for test, duration in cast(dict[str, object], raw_seconds).items(): - number = shard_plans.json_number(duration) - if number is None or number < 0: - return None - seconds[test] = number - shards: list[tuple[str, ...]] = [] - for raw_shard in cast(list[object], raw_shards): - shard = shard_plans.json_object(raw_shard, ("seconds", "tests")) - if shard is None or shard_plans.json_number(shard["seconds"]) is None: - return None - tests = shard_plans.json_strings(shard["tests"]) - if tests is None: - return None - shards.append(tests) - return BackendShards(test_threads, seconds, tuple(shards)) - - -def load_partition(path: Path = PARTITION_PATH) -> Partition: - """Read a partition, failing unless its structure is valid; whether it - partitions each sharded backend's tests is check_partition()'s question.""" - try: - text = path.read_text(encoding="utf-8") - except FileNotFoundError: - raise ScriptError(f"{path} does not exist; {PLAN_ADVICE}") from None - try: - value: object = json.loads(text, object_pairs_hook=strict_json_object) - except ValueError as error: - raise ScriptError(f"{path} is not valid JSON: {error}") from error - document = shard_plans.json_object( - value, ("schema", "schema_version", "runs", "backends") - ) - if ( - document is None - or document["schema"] != SCHEMA - or document["schema_version"] != SCHEMA_VERSION - ): - raise ScriptError( - f"{path} is not an {SCHEMA} document of schema version {SCHEMA_VERSION}" - ) - runs = shard_plans.json_strings(document["runs"]) - raw_backends = document["backends"] - if runs is None or not isinstance(raw_backends, dict): - raise ScriptError(f"{path} has malformed runs or backends") - backends: dict[str, BackendShards] = {} - for backend, raw in cast(dict[str, object], raw_backends).items(): - shards = _backend_shards(raw) - if shards is None: - raise ScriptError(f"{path} has a malformed partition of {backend}") - backends[backend] = shards - return Partition(runs, backends) - - -def partition_problems(partition: Partition) -> list[str]: - """Return how a partition fails to put each test that a sharded backend's - filter selects in exactly one of the backend's shards, with its - duration.""" - problems: list[str] = [] - for backend, shards in partition.backends.items(): - selection = OPENVMM_TEST_SELECTIONS.get(backend) - if selection is None: - problems.append(f"{backend} has no filter that names each of its tests") - continue - if not shards.shards: - problems.append(f"{backend} has no shards") - expected = set(selection) - placements: dict[str, list[int]] = {} - for index, shard in enumerate(shards.shards, 1): - if not shard: - problems.append(f"{backend} shard {index} is empty") - for test in shard: - placements.setdefault(test, []).append(index) - for test in selection: - found = placements.get(test, []) - if not found: - problems.append(f"{backend}: {test} is in no shard") - elif len(found) > 1: - problems.append( - f"{backend}: {test} is in more than one shard: " - + ", ".join(str(index) for index in found) - ) - for test in sorted(set(placements) - expected): - problems.append(f"{backend}: {test} is not a test of its filter") - for test in selection: - if test not in shards.seconds: - problems.append(f"{backend}: {test} has no duration") - for test in sorted(set(shards.seconds) - expected): - problems.append( - f"{backend}: {test} has a duration but is not a test of its filter" - ) - return problems - - -def check_partition(partition: Partition, path: Path = PARTITION_PATH) -> None: - """Fail unless the partition puts each test that a sharded backend's - filter selects in exactly one of the backend's shards.""" - problems = partition_problems(partition) - if problems: - raise ScriptError( - f"{path} does not put each test of a sharded OpenVMM filter in " - f"exactly one shard: {'; '.join(problems)}; {PLAN_ADVICE}" - ) - - -def select_shard( - backend: str, - shard: tuple[int, int], - path: Path = PARTITION_PATH, -) -> OpenVMMTestShard: - """Return shard INDEX of the COUNT shards of the backend's tests, with its - tests in the order of the backend's filter, once the partition is checked - and has COUNT shards for the backend.""" - partition = load_partition(path) - check_partition(partition, path) - index, count = shard - shards = partition.backends.get(backend) - if shards is None: - raise ScriptError( - f"{path} does not shard the {backend.upper()} tests, which CI runs " - "in one job (doc/ci.md#openvmm-vmm-test-shards)" - ) - if count != len(shards.shards): - raise ScriptError( - f"{path} splits the {backend.upper()} tests into " - f"{len(shards.shards)} shards, but this run is shard {index} of " - f"{count}; CI must run one job for each shard " - "(doc/ci.md#openvmm-vmm-test-shards)" - ) - selected = set(shards.shards[index - 1]) - return OpenVMMTestShard( - index, - count, - tuple(test for test in OPENVMM_TEST_SELECTIONS[backend] if test in selected), - ) - - -def read_reports( - paths: Iterable[Path], -) -> tuple[dict[str, dict[str, list[float]]], set[str], dict[str, set[int]]]: - """Return each backend's samples of the duration of each of its tests, - the CI runs that they come from, and the logical processors of the - runners that took them. - - A sample is one report's time of a test that passed, in a run of the - backend's tests or of a shard of them. - """ - samples: dict[str, dict[str, list[float]]] = {} - runs: set[str] = set() - processors: dict[str, set[int]] = {} - documents: set[str] = set() - for path in shard_plans.report_paths(paths, OPENVMM_TEST_DURATIONS_NAME): - try: - value: object = json.loads( - path.read_text(encoding="utf-8"), object_pairs_hook=strict_json_object - ) - except ValueError as error: - raise ScriptError(f"{path} is not valid JSON: {error}") from error - document = cast(dict[str, object], value) if isinstance(value, dict) else {} - if ( - document.get("schema") != OPENVMM_TEST_DURATIONS_SCHEMA - or document.get("schema_version") != OPENVMM_TEST_DURATIONS_SCHEMA_VERSION - ): - raise ScriptError( - f"{path} is not a test duration report of schema version " - f"{OPENVMM_TEST_DURATIONS_SCHEMA_VERSION}" - ) - backend = document.get("backend") - tests = document.get("tests") - if backend not in OPENVMM_TEST_BACKENDS or not isinstance(tests, list): - raise ScriptError(f"{path} has a malformed backend or tests") - backend = cast(str, backend) - # The same report found twice counts once. - identity = json.dumps(document, sort_keys=True) - if identity in documents: - continue - documents.add(identity) - measured = False - for raw_test in cast(list[object], tests): - test = ( - cast(dict[str, object], raw_test) if isinstance(raw_test, dict) else {} - ) - name = test.get("test") - seconds = shard_plans.json_number(test.get("seconds")) - if not isinstance(name, str) or seconds is None or seconds < 0: - raise ScriptError(f"{path} has a malformed test") - if test.get("outcome") != "passed": - continue - samples.setdefault(backend, {}).setdefault(name, []).append(seconds) - measured = True - if not measured: - continue - github = document.get("github") - if isinstance(github, dict): - job = cast(dict[str, object], github) - run_id, attempt = job.get("run_id"), job.get("run_attempt") - if isinstance(run_id, str) and isinstance(attempt, str): - runs.add(f"{run_id}-{attempt}") - logical = document.get("logical_processors") - if isinstance(logical, int) and not isinstance(logical, bool) and logical > 0: - processors.setdefault(backend, set()).add(logical) - return samples, runs, processors - - -def plan_partition( - previous: Partition | None, - *, - samples: Mapping[str, Mapping[str, Sequence[float]]], - runs: Collection[str], - estimates: Mapping[str, float], - counts: Mapping[str, int], - test_threads: Mapping[str, int], -) -> Partition: - """Plan the shards of each backend that `previous` shards, or that - `counts` or `test_threads` names. - - A test's duration is its estimate, if given, or else the median of the - backend's samples, to a tenth of a second, or else its duration in - `previous`. A backend keeps its number of shards and its test threads - unless `counts` or `test_threads` changes them, and its tests start in - their shards of `previous`. A shard takes as long as nextest takes to run - its tests with the backend's test threads (run_tenths), which the plan - balances with shard_plans.plan_shards. - """ - backends: dict[str, BackendShards] = {} - missing: list[str] = [] - planned = [ - backend - for backend in OPENVMM_TEST_BACKENDS - if (previous is not None and backend in previous.backends) - or backend in counts - or backend in test_threads - ] - if not planned: - raise ScriptError( - "the partition shards no backend; pass --shards BACKEND=COUNT" - ) - for backend in planned: - selection = OPENVMM_TEST_SELECTIONS.get(backend) - if selection is None: - raise ScriptError( - f"the {backend.upper()} filter selects its tests by pattern, so no " - "partition can list them; only the tests of " - + ", ".join(name.upper() for name in OPENVMM_TEST_SELECTIONS) - + " have shards" - ) - old = previous.backends.get(backend) if previous is not None else None - measured = samples.get(backend, {}) - seconds: dict[str, float] = {} - for test in selection: - if test in estimates: - seconds[test] = round(estimates[test], 1) - elif measured.get(test): - seconds[test] = round(statistics.median(measured[test]), 1) - elif old is not None and test in old.seconds: - seconds[test] = old.seconds[test] - else: - missing.append(f"{test} on {backend}") - count = counts.get(backend, len(old.shards) if old is not None else 0) - if count == 0: - raise ScriptError( - f"the partition has no {backend} shards; pass --shards {backend}=COUNT" - ) - threads = test_threads.get(backend, old.test_threads if old is not None else 0) - if threads == 0: - raise ScriptError( - f"the partition has no {backend} test threads; pass " - f"--test-threads {backend}=COUNT" - ) - if missing: - continue - durations = { - test: shard_plans.tenths(duration) for test, duration in seconds.items() - } - backends[backend] = BackendShards( - threads, - seconds, - shard_plans.plan_shards( - seconds, - count, - old.shards if old is not None else (), - cost=lambda tests, durations=durations, threads=threads: run_tenths( - tests, durations, threads - ), - noun="tests", - ), - ) - if missing: - raise ScriptError( - "no report measures, and the partition has no duration of, " - + ", ".join(missing) - + "; pass --report with a report that measures them, or --seconds " - "TEST=SECONDS with an estimate" - ) - planned_runs = ( - tuple(sorted(runs, key=shard_plans.run_order)) - if samples - else (previous.runs if previous is not None else ()) - ) - return Partition(planned_runs, backends) - - -def partition_json(partition: Partition) -> str: - """Return a partition as its file holds it.""" - document = { - "schema": SCHEMA, - "schema_version": SCHEMA_VERSION, - "runs": list(partition.runs), - "backends": { - backend: { - "test_threads": shards.test_threads, - "seconds": { - test: shards.seconds[test] for test in sorted(shards.seconds) - }, - "shards": [ - {"seconds": total, "tests": list(shard)} - for shard, total in zip( - shards.shards, shards.shard_seconds(), strict=True - ) - ], - } - for backend, shards in partition.backends.items() - }, - } - return json.dumps(document, indent=2) + "\n" - - -def describe_partition(previous: Partition | None, partition: Partition) -> str: - """Return each backend's shards, with how long nextest takes to run them - and all of the backend's tests, and the tests that changed shard.""" - lines: list[str] = [] - for backend, shards in partition.backends.items(): - lines.append( - f"{backend}: {len(shards.shards)} shards of {len(shards.seconds)} " - f"tests, which take {shards.run_seconds(shards.seconds):.1f} s " - f"unsharded with {shards.test_threads} test threads" - ) - for index, (shard, total) in enumerate( - zip(shards.shards, shards.shard_seconds(), strict=True), 1 - ): - lines.append(f" shard {index}: {total:.1f} s, {len(shard)} tests") - old = previous.backends.get(backend) if previous is not None else None - if old is None: - continue - before = { - test: index for index, shard in enumerate(old.shards, 1) for test in shard - } - for index, shard in enumerate(shards.shards, 1): - for test in shard: - if before.get(test) != index: - origin = before.get(test) - lines.append( - f" {test}: " - + (f"shard {origin}" if origin else "new") - + f" -> shard {index}" - ) - return "\n".join(lines) - - -def configure_parser(parser: argparse.ArgumentParser) -> None: - """Configure `plan-openvmm-test-shards`.""" - parser.description = ( - "Plan the shards of the OpenVMM VMM tests, which CI runs in parallel, " - "from test duration reports (doc/ci.md#openvmm-vmm-test-shards)." - ) - parser.add_argument( - "--report", - type=Path, - action="append", - default=[], - metavar="PATH", - help=( - f"an {OPENVMM_TEST_DURATIONS_NAME} or a directory to search for " - "them, such as one that gh run download filled; repeatable" - ), - ) - parser.add_argument( - "--seconds", - type=shard_plans.named_seconds("TEST", "ttrpc::test_ttrpc_uefi_boot=9.4"), - action="append", - default=[], - metavar="TEST=SECONDS", - help=( - "use this duration of TEST on every backend, such as an estimate " - "of a test that no report measures; repeatable" - ), - ) - parser.add_argument( - "--shards", - type=shard_plans.backend_count, - action="append", - default=[], - metavar="BACKEND=COUNT", - help=( - "split BACKEND's tests into COUNT shards instead of as many as the " - "partition has; repeatable" - ), - ) - parser.add_argument( - "--test-threads", - type=shard_plans.backend_count, - action="append", - default=[], - metavar="BACKEND=COUNT", - help=( - "model nextest's runs of BACKEND's tests with COUNT test threads, " - "the logical processors of its runners, instead of as many as the " - "partition has; repeatable" - ), - ) - parser.add_argument( - "--partition", - type=Path, - default=PARTITION_PATH, - metavar="PATH", - help="the partition to update (default: the one that CI runs)", - ) - parser.set_defaults(handler=run) - - -def run(args: argparse.Namespace) -> int: - path: Path = args.partition - estimates = dict(cast(list[tuple[str, float]], args.seconds)) - tests = { - test for selection in OPENVMM_TEST_SELECTIONS.values() for test in selection - } - unknown = sorted(set(estimates) - tests) - if unknown: - raise ScriptError( - "--seconds names tests that no sharded filter selects: " - + ", ".join(unknown) - ) - previous = load_partition(path) if path.exists() else None - reports = cast(list[Path], args.report) - samples, runs, processors = read_reports(reports) - if reports and not samples: - raise ScriptError( - "no test duration report measures a test that passed in " - + ", ".join(str(report) for report in reports) - ) - partition = plan_partition( - previous, - samples=samples, - runs=runs, - estimates=estimates, - counts=dict(cast(list[tuple[str, int]], args.shards)), - test_threads=dict(cast(list[tuple[str, int]], args.test_threads)), - ) - check_partition(partition, path) - path.write_text(partition_json(partition), encoding="utf-8", newline="\n") - print(describe_partition(previous, partition)) - for backend, shards in partition.backends.items(): - others = sorted(processors.get(backend, set()) - {shards.test_threads}) - if others: - print( - f"note: runners of the {backend} reports have " - + ", ".join(str(count) for count in others) - + f" logical processors, but the plan models {shards.test_threads} " - f"test threads; pass --test-threads {backend}=COUNT to model another " - "number" - ) - print(f"Wrote {path}") - return 0 diff --git a/scripts/nvx_tools/shard_plans.py b/scripts/nvx_tools/shard_plans.py index 9bc3d06b9..94bf39067 100644 --- a/scripts/nvx_tools/shard_plans.py +++ b/scripts/nvx_tools/shard_plans.py @@ -1,9 +1,7 @@ """Balanced shards of a suite that CI splits across parallel jobs. `microvm_shards` plans the shards of the default microVM suite, whose -scenarios run one after another, and `openvmm_test_shards` those of the -OpenVMM VMM tests, which nextest runs in parallel; each passes the time that -a shard takes. +scenarios run one after another, and passes the time that a shard takes. """ from __future__ import annotations diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 6a18ae616..ddf22a59b 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -60,12 +60,10 @@ nvx_rs, openvmm_ci, openvmm_process, - openvmm_test_shards, performance, performance_history, release, runtime_adapter_tests, - shard_plans, ubuntu, ) from nvx_tools.build_constants import ( # noqa: E402 @@ -1197,9 +1195,9 @@ def test_check_required_ci_passes_each_flag_to_its_suite(self): # flag passed to the wrong suite would fail the first or pass the # second. for changes in ( - ci.CiChanges(True, True, True, True, True, True), - ci.CiChanges(True, True, False, False, True, False), - ci.CiChanges(True, True, False, True, False, True), + ci.CiChanges(True, True, True, True, True), + ci.CiChanges(True, True, False, True, False), + ci.CiChanges(True, True, True, False, True), ): results = ci.required_ci_expected_results( "pull_request", same_repository=True, changes=changes @@ -1465,114 +1463,27 @@ def test_release_commands_keep_their_cli_contract(self): openvmm_tests = nvx.parse_args(["test-openvmm", "--backend", "mshv"]) self.assertEqual(openvmm_tests.backend, "mshv") - self.assertIsNone(openvmm_tests.content) - self.assertIsNone(openvmm_tests.shard) self.assertIs(openvmm_tests.handler, nvx.command_test_openvmm) - openvmm_content_tests = nvx.parse_args( - ["test-openvmm", "--backend", "kvm", "--content", "content"] - ) - self.assertEqual(openvmm_content_tests.content, Path("content")) - self.assertEqual( - nvx.parse_args( - ["test-openvmm", "--backend", "whp", "--shard", "2/2"] - ).shard, - (2, 2), - ) - for value in ("0/2", "3/2", "1", "a/b", "01/2"): - with ( - self.subTest(shard=value), - patch("sys.stderr", new_callable=io.StringIO), - self.assertRaises(SystemExit), - ): - nvx.parse_args(["test-openvmm", "--backend", "whp", "--shard", value]) - - planned_shards = nvx.parse_args( - [ - "plan-openvmm-test-shards", - "--report", - "reports", - "--seconds", - "ttrpc::test_ttrpc_uefi_boot=9.5", - "--shards", - "whp=3", - "--test-threads", - "whp=16", - "--partition", - "shards.json", - ] - ) - self.assertEqual(planned_shards.report, [Path("reports")]) - self.assertEqual(planned_shards.seconds, [("ttrpc::test_ttrpc_uefi_boot", 9.5)]) - self.assertEqual(planned_shards.shards, [("whp", 3)]) - self.assertEqual(planned_shards.test_threads, [("whp", 16)]) - self.assertEqual(planned_shards.partition, Path("shards.json")) - self.assertIs(planned_shards.handler, openvmm_test_shards.run) - for option, value in ( - ("--seconds", "ttrpc::test_ttrpc_uefi_boot"), - ("--seconds", "=9"), - ("--seconds", "test=-1"), - ("--shards", "whp=0"), - ("--shards", "arm=2"), - ("--test-threads", "whp"), - ): - with ( - self.subTest(option=option, value=value), - patch("sys.stderr", new_callable=io.StringIO), - self.assertRaises(SystemExit), - ): - nvx.parse_args(["plan-openvmm-test-shards", option, value]) - - openvmm_test_build = nvx.parse_args( - [ - "build-openvmm-tests", - "--backend", - "kvm", - "--backend", - "mshv", - "--output", - "content", - ] - ) - self.assertEqual(openvmm_test_build.backend, ["kvm", "mshv"]) - self.assertEqual(openvmm_test_build.output, Path("content")) - self.assertIs(openvmm_test_build.handler, nvx.command_build_openvmm_tests) - with ( - patch("sys.stderr", new_callable=io.StringIO), - self.assertRaises(SystemExit), - ): - nvx.parse_args(["build-openvmm-tests", "--output", "content"]) - - cache_key = nvx.parse_args( - ["openvmm-test-cache-key", "--backend", "kvm", "--backend", "mshv"] - ) - self.assertEqual(cache_key.backend, ["kvm", "mshv"]) - self.assertIs(cache_key.handler, nvx.command_openvmm_test_cache_key) - for command, handler in ( - ("check-openvmm-tests", nvx.command_check_openvmm_tests), - ("reuse-openvmm-tests", nvx.command_reuse_openvmm_tests), - ): - with self.subTest(command=command): - parsed = nvx.parse_args( - [command, "--backend", "whp", "--content", "content"] - ) - self.assertEqual(parsed.backend, ["whp"]) - self.assertEqual(parsed.content, Path("content")) - self.assertIs(parsed.handler, handler) + # CI checks OpenVMM's own CI of the pin instead of building and running + # OpenVMM's tests, so the commands that handed its test builds and + # shards between jobs are gone (doc/ci.md#openvmm-ci). for arguments in ( - ["openvmm-test-cache-key"], - ["openvmm-test-cache-key", "--backend", "arm"], - ["check-openvmm-tests", "--backend", "whp"], - ["check-openvmm-tests", "--content", "content"], - ["reuse-openvmm-tests", "--backend", "whp"], - ["reuse-openvmm-tests", "--content", "content"], + ["test-openvmm", "--backend", "kvm", "--content", "content"], + ["test-openvmm", "--backend", "whp", "--shard", "2/2"], + ["build-openvmm-tests", "--backend", "kvm", "--output", "content"], + ["openvmm-test-cache-key", "--backend", "kvm"], + ["check-openvmm-tests", "--backend", "whp", "--content", "content"], + ["reuse-openvmm-tests", "--backend", "whp", "--content", "content"], + ["plan-openvmm-test-shards", "--report", "reports"], ): with ( self.subTest(arguments=arguments), patch("sys.stderr", new_callable=io.StringIO), - self.assertRaises(SystemExit), + self.assertRaises(SystemExit) as exit_context, ): nvx.parse_args(arguments) + self.assertEqual(exit_context.exception.code, 2) openvmm_ci_check = nvx.parse_args(["check-openvmm-ci"]) self.assertIsNone(openvmm_ci_check.revision) @@ -3179,9 +3090,9 @@ def test_openvmm_tests_define_each_backend_filter(self): ) for required_test in ci.OPENVMM_REQUIRED_MICROVM_TESTS: self.assertIn(required_test, ci.OPENVMM_WHP_TESTS) - # test-openvmm runs the WHP content without vmm-tests-run-target, which - # checks for Hyper-V on Windows, so no WHP test may need Hyper-V, as - # flowey reads from "hyperv" in a test's name (#690). + # vmm-tests-run checks for Hyper-V when a selected test needs it, as + # flowey reads from "hyperv" in a test's name, so no WHP test may need + # Hyper-V, which WHP hosts need not run (#690). for test in ci.OPENVMM_WHP_TESTS: with self.subTest(whp_test=test): self.assertNotIn("hyperv", test) @@ -3214,2335 +3125,109 @@ def test_openvmm_tests_define_each_backend_filter(self): ci.OPENVMM_WHP_EXCLUDED_TESTS, ) self.assertNotIn( - "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_nvme_boot", - ci.OPENVMM_WHP_EXCLUDED_TESTS, - ) - self.assertTrue( - set(ci.OPENVMM_WHP_EXCLUDED_TESTS).issubset(ci.OPENVMM_WHP_TESTS) - ) - self.assertEqual( - ci.OPENVMM_TEST_FILTERS["whp"], - ci._join_openvmm_tests( - ci.OPENVMM_WHP_TESTS, - ci.OPENVMM_WHP_EXCLUDED_TESTS, - ), - ) - self.assertEqual( - ci._join_openvmm_tests( - ("suite::boot", "suite::boot_heavy"), - ("suite::boot_heavy",), - ), - ( - "(test(/^suite::boot$/) | test(/^suite::boot_heavy$/))" - " & !test(/^suite::boot_heavy$/)" - ), - ) - - def test_linux_filters_leave_out_every_windows_server_test(self): - # The Linux legs run Linux guests. A test's name holds the year of its - # Windows Server image, so a test that moves to another image, as the - # KVP IC test moved from 2022 to 2025 (#1035), must stay out of them. - windows_tests = [ - test for test in ci.OPENVMM_WHP_TESTS if "windows_datacenter_core" in test - ] - self.assertIn( - "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic", - windows_tests, - ) - for backend in ("kvm", "mshv"): - test_filter = ci.OPENVMM_TEST_FILTERS[backend] - for test in windows_tests: - with self.subTest(backend=backend, test=test): - self.assertFalse(_nextest_filter_matches(test_filter, test)) - for test in ( - "multiarch::ic::openvmm_uefi_x64_ubuntu_2504_server_x64_timesync_ic", - *ci.OPENVMM_REQUIRED_MICROVM_TESTS, - ): - with self.subTest(backend=backend, test=test): - self.assertTrue(_nextest_filter_matches(test_filter, test)) - - def test_openvmm_tests_reject_unknown_backend(self): - with self.assertRaisesRegex(common.ScriptError, "unsupported.*backend"): - ci.run_openvmm_tests("unknown") - - def test_openvmm_test_build_filter_selects_every_backends_tests(self): - kvm = ci.OPENVMM_TEST_FILTERS["kvm"] - mshv = ci.OPENVMM_TEST_FILTERS["mshv"] - # One Linux build serves both Linux backends: a test of either - # backend's filter matches the union, which nextest reads as "or". - self.assertEqual( - ci.openvmm_test_build_filter(("mshv", "kvm")), - f"({kvm}) | ({mshv})", - ) - for backend, test_filter in ci.OPENVMM_TEST_FILTERS.items(): - with self.subTest(backend=backend): - self.assertEqual(ci.openvmm_test_build_filter((backend,)), test_filter) - # The run script quotes the filter without escaping quotes. - self.assertNotIn("'", test_filter) - self.assertEqual( - set(ci.OPENVMM_TEST_OPERATING_SYSTEMS), set(ci.OPENVMM_TEST_BACKENDS) - ) - - @staticmethod - def _openvmm_run_script(operating_system: str, test_filter: str) -> str: - """Return the run script that the pinned flowey writes for a build.""" - if operating_system == "windows": - prefix = "cd $PSScriptRoot ; .\\x86_64-pc-windows-msvc\\flowey_hvlite.exe" - target = "windows-x64" - else: - prefix = ( - 'cd "$(dirname "${BASH_SOURCE[0]}")" ; ' - "./x86_64-unknown-linux-gnu/flowey_hvlite" - ) - target = "linux-x64" - return ( - f"{prefix} pipeline run vmm-tests-run-target --target {target} " - f"--dir . --filter '{test_filter}' --repetitions 1 " - "--artifacts VmgsWithBootEntry --skip-vhd-prompt --ci-profile" - ) - - def _build_openvmm_test_content( - self, - root: Path, - backends: Sequence[str], - *, - revision: str = "a" * 40, - clean: bool = True, - script: str | None = None, - ) -> tuple[Path, MagicMock, MagicMock, str]: - """Build the backends' test content into ``root / "content"`` with - build_openvmm_tests, with an OpenVMM checkout at ``revision`` and a - flowey that writes a runner, a kernel, the nextest configuration, and - ``script`` or the run script that the pinned flowey writes. Return the - content, the mocks of openvmm_git_state and run_checked, and what the - build printed.""" - operating_system = ci.OPENVMM_TEST_OPERATING_SYSTEMS[backends[0]] - build_filter = ci.openvmm_test_build_filter(backends) - runner = ( - "x86_64-pc-windows-msvc/flowey_hvlite.exe" - if operating_system == "windows" - else "x86_64-unknown-linux-gnu/flowey_hvlite" - ) - targets = sorted( - { - target - for backend in backends - for target in OpenVMMBuildConstants.TEST_RUST_TARGETS[backend] - } - ) - installed_targets = common.CommandResult( - args=("rustup", "target", "list"), - returncode=0, - stdout="".join(f"{target}\n" for target in targets).encode(), - stderr=b"", - ) - - def build(command: list[str], **_: object) -> None: - if "--build-only" not in command: - return - content = Path(command[command.index("--dir") + 1]) - files = { - runner: b"runner", - "x86_64/vmlinux": b"kernel", - ".config/nextest.toml": b"[profile.ci]\n", - } - for relative, data in files.items(): - path = content / relative - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(data) - path.chmod(0o755 if relative != ".config/nextest.toml" else 0o644) - run_script = content / ci.OPENVMM_TEST_RUN_SCRIPTS[operating_system] - run_script.write_text( - script or self._openvmm_run_script(operating_system, build_filter), - encoding="utf-8", - ) - run_script.chmod(0o755) - - openvmm = root / "openvmm" - openvmm.mkdir(exist_ok=True) - (openvmm / "Cargo.toml").touch() - output = root / "content" - with ( - patch.object(OpenVMMBuildConstants, "DIRECTORY", openvmm), - patch.object(ci, "require_tool", side_effect=["cargo", "rustup"]), - patch.object(ci, "run_capture", return_value=installed_targets), - patch.object( - ci, "openvmm_git_state", return_value=(revision, clean) - ) as git_state, - patch.object(ci, "run_checked", side_effect=build) as run_checked, - patch.dict(os.environ, {"RUNNER_TEMP": os.fspath(root)}), - patch("sys.stdout", new_callable=io.StringIO) as stdout, - ): - ci.build_openvmm_tests(backends, output) - return output, git_state, run_checked, stdout.getvalue() - - def test_openvmm_test_content_paths_stay_inside_the_directory(self): - for value in ("x86_64/vmlinux", ".config/nextest.toml", "run.sh"): - with self.subTest(value=value): - self.assertEqual(ci._content_relative_path(value), PurePosixPath(value)) - # A backslash or a drive would name another path on Windows. - for value in ( - None, - 1, - "", - ".", - "/escape", - "../escape", - "target/../../escape", - "..\\escape", - "target\\..\\..\\escape", - "C:\\escape", - "C:escape", - "C:/escape", - "\\\\server\\share\\escape", - "target/file:stream", - ): - with self.subTest(value=value): - self.assertIsNone(ci._content_relative_path(value)) - - def test_openvmm_test_run_scripts_name_the_runner_and_its_options(self): - test_filter = ci.openvmm_test_build_filter(("kvm", "mshv")) - options = [ - "--target", - "linux-x64", - "--repetitions", - "1", - "--artifacts", - "VmgsWithBootEntry", - "--skip-vhd-prompt", - "--ci-profile", - ] - with tempfile.TemporaryDirectory() as temporary: - script = Path(temporary) / "run" - for operating_system, runner, target in ( - ("linux", "x86_64-unknown-linux-gnu/flowey_hvlite", "linux-x64"), - ( - "windows", - "x86_64-pc-windows-msvc/flowey_hvlite.exe", - "windows-x64", - ), - ): - with self.subTest(operating_system=operating_system): - script.write_text( - self._openvmm_run_script(operating_system, test_filter) + "\n", - encoding="utf-8", - ) - self.assertEqual( - ci._openvmm_test_run_target(script, test_filter), - (runner, ["--target", target, *options[2:]]), - ) - - valid = self._openvmm_run_script("linux", test_filter) - for name, text, message in ( - ("another filter", valid, "does not run the filter"), - ("no cd", valid.split(" ; ", 1)[1], "does not run the filter"), - ( - "another pipeline", - valid.replace("vmm-tests-run-target", "vmm-tests-run"), - "does not run vmm-tests-run-target", - ), - ( - "absolute runner", - valid.replace("./x86_64", "/x86_64"), - "outside its directory", - ), - ( - "parent runner", - valid.replace("./x86_64", "./../x86_64"), - "outside its directory", - ), - ( - "Windows parent runner", - self._openvmm_run_script("windows", test_filter).replace( - ".\\x86_64", ".\\..\\x86_64" - ), - "outside its directory", - ), - ( - "drive runner", - self._openvmm_run_script("windows", test_filter).replace( - ".\\x86_64", ".\\C:\\x86_64" - ), - "outside its directory", - ), - ( - "unknown option", - valid + " --install-missing-deps", - "unsupported option --install-missing-deps", - ), - ( - "missing value", - valid + " --repetitions", - "--repetitions without a value", - ), - ( - "another directory", - valid.replace("--dir .", "--dir /tmp"), - "does not run the tests in its own directory", - ), - ( - "no target", - valid.replace("--target linux-x64 ", ""), - "does not run the tests in its own directory", - ), - ): - with self.subTest(case=name): - script.write_text(text, encoding="utf-8") - expected_filter = ( - ci.OPENVMM_TEST_FILTERS["kvm"] - if name == "another filter" - else test_filter - ) - with self.assertRaisesRegex(common.ScriptError, message): - ci._openvmm_test_run_target(script, expected_filter) - - def test_build_openvmm_tests_builds_only_and_describes_the_content(self): - backends = ("whp",) if os.name == "nt" else ("mshv", "kvm") - operating_system = "windows" if os.name == "nt" else "linux" - build_filter = ci.openvmm_test_build_filter(backends) - runner = ( - "x86_64-pc-windows-msvc/flowey_hvlite.exe" - if os.name == "nt" - else "x86_64-unknown-linux-gnu/flowey_hvlite" - ) - - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - openvmm = root / "openvmm" - revision = "a" * 40 - output, git_state, run_checked, printed = self._build_openvmm_test_content( - root, backends, revision=revision - ) - - git_state.assert_called_once_with(openvmm) - restore, tests = run_checked.call_args_list - self.assertEqual( - restore.args[0], - ["cargo", "xflowey", "restore-packages", "--no-compat-igvm"], - ) - target = "windows-x64" if os.name == "nt" else "linux-x64" - self.assertEqual(target, ci.OPENVMM_TEST_TARGETS[operating_system]) - # The build names its target and its build mode, without optional - # features. - self.assertEqual( - tests.args[0], - [ - "cargo", - "xflowey", - "vmm-tests-run", - "--target", - target, - "--release", - "--ci-profile", - "--skip-vhd-prompt", - "--build-only", - "--filter", - build_filter, - "--dir", - os.fspath(output.resolve()), - ], - ) - for command in (restore, tests): - self.assertEqual(command.kwargs["cwd"], openvmm) - self.assertEqual( - command.kwargs["env"]["RUSTUP_TOOLCHAIN"], - OpenVMMBuildConstants.RUST_TOOLCHAIN, - ) - # Windows links the build reproducibly for sccache (#693). - if os.name == "nt": - self.assertIn( - ci.MSVC_REPRODUCIBLE_LINK_OPTION, - command.kwargs["env"]["_LINK_"].split(), - ) - # The build boots no NVX guest, so it needs no guest artifacts. - self.assertNotIn("OPENVMM_MICROVM_TEST_KERNEL", tests.kwargs["env"]) - - manifest = json.loads( - (output / ci.OPENVMM_TEST_CONTENT_MANIFEST).read_text(encoding="utf-8") - ) - script = ci.OPENVMM_TEST_RUN_SCRIPTS[operating_system] - executable = os.name != "nt" - - def digest(data: bytes) -> str: - return hashlib.sha256(data).hexdigest() - - self.assertEqual( - manifest, - { - "format": 2, - "openvmm_revision": revision, - "openvmm_clean": True, - "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, - "operating_system": operating_system, - "target": target, - "build_options": [ - "--release", - "--ci-profile", - "--skip-vhd-prompt", - "--build-only", - ], - "filters": { - backend: ci.OPENVMM_TEST_FILTERS[backend] - for backend in sorted(backends) - }, - "build_filter": build_filter, - "runner": runner, - "arguments": [ - "--target", - target, - "--repetitions", - "1", - "--artifacts", - "VmgsWithBootEntry", - "--skip-vhd-prompt", - "--ci-profile", - ], - "files": sorted( - ( - { - "path": ".config/nextest.toml", - "size": 13, - "executable": False, - "sha256": digest(b"[profile.ci]\n"), - }, - { - "path": script, - "size": (output / script).stat().st_size, - "executable": executable, - "sha256": digest((output / script).read_bytes()), - }, - { - "path": runner, - "size": 6, - "executable": executable, - "sha256": digest(b"runner"), - }, - { - "path": "x86_64/vmlinux", - "size": 6, - "executable": executable, - "sha256": digest(b"kernel"), - }, - ), - key=lambda entry: str(entry["path"]), - ), - }, - ) - self.assertIn("4 files", printed) - - def test_build_openvmm_tests_refuse_a_run_script_of_another_target(self): - backends = ("whp",) if os.name == "nt" else ("kvm", "mshv") - operating_system = ci.OPENVMM_TEST_OPERATING_SYSTEMS[backends[0]] - target = ci.OPENVMM_TEST_TARGETS[operating_system] - other = "linux-x64" if target == "windows-x64" else "windows-x64" - script = self._openvmm_run_script( - operating_system, ci.openvmm_test_build_filter(backends) - ).replace(f"--target {target} ", f"--target {other} ") - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - with self.assertRaisesRegex( - common.ScriptError, f"does not run the {target} tests that it built" - ): - self._build_openvmm_test_content(root, backends, script=script) - self.assertFalse( - (root / "content" / ci.OPENVMM_TEST_CONTENT_MANIFEST).exists() - ) - - @staticmethod - def _openvmm_gitlink(revision: str) -> common.CommandResult: - return common.CommandResult( - args=("git", "rev-parse", ":openvmm"), - returncode=0, - stdout=f"{revision}\n".encode(), - stderr=b"", - ) - - def test_openvmm_test_cache_key_covers_every_input_of_the_content(self): - # #692: CI caches each OS's build under a key of everything that - # determines it, so that a change to any of them builds the tests - # again, and only such a change does. - backends = ("whp",) if os.name == "nt" else ("kvm", "mshv") - operating_system = ci.OPENVMM_TEST_OPERATING_SYSTEMS[backends[0]] - target = ci.OPENVMM_TEST_TARGETS[operating_system] - revision = "a" * 40 - - def key( - selected: Sequence[str] = backends, - gitlink: str = revision, - environment: Mapping[str, str] | None = None, - ) -> str: - with ( - patch.object( - ci, "run_capture", return_value=self._openvmm_gitlink(gitlink) - ) as run_capture, - patch("sys.stdout", new_callable=io.StringIO) as stdout, - ): - value = ci.openvmm_test_cache_key(selected, environment) - # The key needs only the checkout's gitlink, not OpenVMM's files. - run_capture.assert_called_once_with( - ["git", "-C", BuildConstants.REPO_ROOT, "rev-parse", ":openvmm"] - ) - self.assertEqual(stdout.getvalue(), f"{value}\n") - return value - - base = key() - self.assertRegex( - base, rf"^openvmm-vmm-tests-v1-{target}-{revision}-[0-9a-f]{{64}}$" - ) - # GitHub rejects longer keys and keys with a comma. - self.assertLessEqual(len(base), 512) - self.assertNotIn(",", base) - self.assertEqual(key(tuple(reversed(backends))), base) - keys = {base} - for name, change, gitlink in ( - ("OpenVMM revision", contextlib.nullcontext(), "b" * 40), - ( - "cache version", - patch.object(ci, "OPENVMM_TEST_CACHE_VERSION", 2), - revision, - ), - ( - "content format", - patch.object(ci, "OPENVMM_TEST_CONTENT_FORMAT", 3), - revision, - ), - ( - "Rust release", - patch.object(OpenVMMBuildConstants, "RUST_TOOLCHAIN", "1.0.0"), - revision, - ), - ( - "target", - patch.dict(ci.OPENVMM_TEST_TARGETS, {operating_system: "other-x64"}), - revision, - ), - ( - "build options", - patch.object( - ci, - "OPENVMM_TEST_BUILD_OPTIONS", - (*ci.OPENVMM_TEST_BUILD_OPTIONS, "--copy-extras"), - ), - revision, - ), - ( - "filter", - patch.dict(ci.OPENVMM_TEST_FILTERS, {backends[0]: "all()"}), - revision, - ), - ): - with self.subTest(input=name), change: - changed = key(gitlink=gitlink) - self.assertNotIn(changed, keys) - keys.add(changed) - # A build of fewer backends has a key of its own. - if len(backends) > 1: - self.assertNotIn(key(backends[:1]), keys) - with tempfile.TemporaryDirectory() as temporary: - output = Path(temporary) / "github-output" - self.assertEqual( - key(environment={"GITHUB_OUTPUT": os.fspath(output)}), base - ) - self.assertEqual(output.read_text(encoding="utf-8"), f"key={base}\n") - - def test_openvmm_test_cache_takes_only_a_clean_build_of_its_key(self): - # #692: CI checks a build before it caches it, and reuses restored - # content only when the same check passes, so that a hit runs exactly - # the tests that a fresh build runs. A fresh build passes; any - # difference in what the key digests, in the runner and options that - # the run script runs, or in the files fails, and the reuse then - # removes the content, so that the tests build in its place. - backends = ("whp",) if os.name == "nt" else ("kvm", "mshv") - names = ", ".join(backend.upper() for backend in backends) - operating_system = ci.OPENVMM_TEST_OPERATING_SYSTEMS[backends[0]] - target = ci.OPENVMM_TEST_TARGETS[operating_system] - script = ci.OPENVMM_TEST_RUN_SCRIPTS[operating_system] - runner = ( - "x86_64-pc-windows-msvc/flowey_hvlite.exe" - if operating_system == "windows" - else "x86_64-unknown-linux-gnu/flowey_hvlite" - ) - revision = "a" * 40 - - def set_field(field: str, value: object) -> Callable[[Path], None]: - def edit(content: Path) -> None: - path = content / ci.OPENVMM_TEST_CONTENT_MANIFEST - manifest = json.loads(path.read_text(encoding="utf-8")) - manifest[field] = value - path.write_text(json.dumps(manifest), encoding="utf-8") - - return edit - - def write(relative: str, data: bytes) -> Callable[[Path], object]: - return lambda content: (content / relative).write_bytes(data) - - def replace_in_script(old: str, new: str) -> Callable[[Path], None]: - def edit(content: Path) -> None: - path = content / script - text = path.read_text(encoding="utf-8") - self.assertIn(old, text) - path.write_text(text.replace(old, new), encoding="utf-8") - - return edit - - def run( - content: Path, gitlink: str = revision - ) -> tuple[str | None, bool, str, str]: - """Check and then reuse the content, and return the check's error, - whether the reuse kept it, what both printed, and the step - output.""" - output = content.parent / "github-output" - output.unlink(missing_ok=True) - with ( - patch.object( - ci, "run_capture", return_value=self._openvmm_gitlink(gitlink) - ), - patch("sys.stdout", new_callable=io.StringIO) as stdout, - ): - try: - ci.check_openvmm_tests(backends, content) - except common.ScriptError as error: - refusal: str | None = str(error) - else: - refusal = None - reused = ci.reuse_openvmm_tests( - backends, content, {"GITHUB_OUTPUT": os.fspath(output)} - ) - return refusal, reused, stdout.getvalue(), output.read_text("utf-8") - - with tempfile.TemporaryDirectory() as temporary: - content, *_ = self._build_openvmm_test_content( - Path(temporary), backends, revision=revision - ) - with ( - patch.object( - ci, "run_capture", return_value=self._openvmm_gitlink(revision) - ), - patch("sys.stdout", new_callable=io.StringIO), - ): - key = ci.openvmm_test_cache_key(backends) - refusal, reused, printed, output = run(content) - self.assertIsNone(refusal) - self.assertTrue(reused) - self.assertEqual(output, "reused=true\n") - self.assertIn( - f"{content} holds the {names} OpenVMM tests of cache key {key}", - printed, - ) - self.assertIn(f"reusing the {names} OpenVMM tests of cache key", printed) - self.assertNotIn("::warning", printed) - self.assertTrue((content / ci.OPENVMM_TEST_CONTENT_MANIFEST).is_file()) - - # A cache miss leaves nothing to reuse or to warn of. - missing = Path(temporary) / "missing" - refusal, reused, printed, output = run(missing) - self.assertIn("has no nvx-openvmm-tests.json", refusal or "") - self.assertFalse(reused) - self.assertEqual(output, "reused=false\n") - self.assertIn("holds no", printed) - self.assertNotIn("::warning", printed) - - cases: list[tuple[str, Callable[[Path], object], str, str]] = [ - ( - "no manifest", - lambda content: (content / ci.OPENVMM_TEST_CONTENT_MANIFEST).unlink(), - revision, - f"it has no {ci.OPENVMM_TEST_CONTENT_MANIFEST}", - ), - ( - "malformed manifest", - write(ci.OPENVMM_TEST_CONTENT_MANIFEST, b"{"), - revision, - "is not valid JSON", - ), - ( - "OpenVMM revision", - lambda _: None, - "b" * 40, - f'the openvmm_revision "{revision}", not "{"b" * 40}"', - ), - ( - "dirty checkout", - set_field("openvmm_clean", False), - revision, - "the openvmm_clean false, not true", - ), - ("content format", set_field("format", 1), revision, "the format 1, not 2"), - ( - "Rust release", - set_field("rust_toolchain", "1.0.0"), - revision, - 'the rust_toolchain "1.0.0"', - ), - ( - "target", - set_field("target", "other-x64"), - revision, - 'the target "other-x64"', - ), - ( - "build options", - set_field("build_options", ["--build-only"]), - revision, - 'the build_options ["--build-only"]', - ), - ( - "filter", - set_field("filters", {backends[0]: "all()"}), - revision, - f'the filters {{"{backends[0]}": "all()"}}', - ), - ( - "run script options", - replace_in_script( - "--ci-profile", "--ci-profile --no-reuse-prepped-vhds" - ), - revision, - f"another runner or other options than {script} runs", - ), - ( - "run script target", - replace_in_script(f"--target {target} ", "--target other-x64 "), - revision, - f"{script} runs the tests of another target", - ), - ( - "missing file", - lambda content: (content / "x86_64" / "vmlinux").unlink(), - revision, - "it lacks x86_64/vmlinux, which its manifest lists", - ), - ( - "unlisted file", - write("stray", b"stray"), - revision, - "its manifest does not list stray", - ), - ( - "changed file", - write("x86_64/vmlinux", b"KERNEL"), - revision, - "the sha256 of x86_64/vmlinux differs", - ), - ( - "truncated file", - write("x86_64/vmlinux", b"kern"), - revision, - "the size of x86_64/vmlinux differs", - ), - ] - if os.name != "nt": - - def drop_executable_bit(content: Path) -> None: - (content / runner).chmod(0o644) - - def link(content: Path) -> None: - (content / "link").symlink_to("x86_64/vmlinux") - - cases += [ - ( - "executable bit", - drop_executable_bit, - revision, - f"the executable of {runner} differs", - ), - ("symbolic link", link, revision, "a non-regular file link"), - ] - for name, edit, gitlink, message in cases: - with self.subTest(case=name), tempfile.TemporaryDirectory() as temporary: - content, *_ = self._build_openvmm_test_content( - Path(temporary), backends, revision=revision - ) - edit(content) - refusal, reused, printed, output = run(content, gitlink) - self.assertIn(message, refusal or "") - self.assertIn( - f"does not hold the {names} OpenVMM tests of cache key", - refusal or "", - ) - self.assertFalse(reused) - self.assertFalse(common.path_exists(content)) - self.assertEqual(output, "reused=false\n") - self.assertIn("::warning title=OpenVMM test cache::", printed) - self.assertIn(message, printed) - - # The reuse also removes a file where it expects the directory. - with tempfile.TemporaryDirectory() as temporary: - content = Path(temporary) / "content" - content.write_bytes(b"content") - refusal, reused, printed, _ = run(content) - self.assertFalse(reused) - self.assertFalse(common.path_exists(content)) - self.assertIn("::warning title=OpenVMM test cache::", printed) - - def test_build_openvmm_tests_refuse_other_hosts_and_used_directories(self): - other = ("kvm",) if os.name == "nt" else ("whp",) - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - openvmm = root / "openvmm" - openvmm.mkdir() - (openvmm / "Cargo.toml").touch() - used = root / "used" - used.mkdir() - (used / "stale").touch() - native = ("whp",) if os.name == "nt" else ("kvm",) - for backends, output, message in ( - ((), root / "content", "at least one"), - (("unknown",), root / "content", "unsupported OpenVMM test backend"), - (other, root / "content", "tests build on"), - (native, used, "not an empty directory"), - (native, used / "stale", "not an empty directory"), - ): - with ( - self.subTest(backends=backends, output=output.name), - patch.object(OpenVMMBuildConstants, "DIRECTORY", openvmm), - patch.object(ci, "run_checked") as run_checked, - self.assertRaisesRegex(common.ScriptError, message), - ): - ci.build_openvmm_tests(backends, output) - run_checked.assert_not_called() - - def _openvmm_test_content( - self, content: Path, backend: str, revision: str - ) -> tuple[Path, list[str]]: - runner = content / "target" / "flowey_hvlite" - runner.parent.mkdir(parents=True) - runner.write_bytes(b"runner") - runner.chmod(0o644) - arguments = ["--target", "linux-x64", "--ci-profile"] - manifest = { - "format": ci.OPENVMM_TEST_CONTENT_FORMAT, - "openvmm_revision": revision, - "openvmm_clean": True, - "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, - "operating_system": ci.OPENVMM_TEST_OPERATING_SYSTEMS[backend], - "target": "linux-x64", - "build_options": list(ci.OPENVMM_TEST_BUILD_OPTIONS), - "filters": {backend: ci.OPENVMM_TEST_FILTERS[backend]}, - "build_filter": ci.OPENVMM_TEST_FILTERS[backend], - "runner": "target/flowey_hvlite", - "arguments": arguments, - "files": [ - { - "path": "target/flowey_hvlite", - "size": 6, - "executable": True, - "sha256": hashlib.sha256(b"runner").hexdigest(), - } - ], - } - (content / ci.OPENVMM_TEST_CONTENT_MANIFEST).write_text( - json.dumps(manifest), encoding="utf-8" - ) - return runner, arguments - - def test_openvmm_tests_run_built_content_with_its_runner(self): - revision = "b" * 40 - gitlink = common.CommandResult( - args=("git", "rev-parse", ":openvmm"), - returncode=0, - stdout=f"{revision}\n".encode(), - stderr=b"", - ) - for backend in ("kvm", "mshv"): - with ( - self.subTest(backend=backend), - tempfile.TemporaryDirectory() as temporary, - ): - root = Path(temporary) - content = root / "content" - content.mkdir() - runner, arguments = self._openvmm_test_content( - content, backend, revision - ) - kernel = root / "vmlinux" - initrd = root / "initramfs.cpio.gz" - kernel.touch() - initrd.touch() - - with ( - patch.object(ci, "validate_openvmm_test_backend") as validate, - patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), - patch.object( - ci, "run_capture", return_value=gitlink - ) as run_capture, - patch.object(ci, "require_tool") as require_tool, - patch.object(ci, "download") as download, - patch.object(ci, "run_checked") as run_checked, - patch.dict(os.environ, {"RUNNER_TEMP": os.fspath(root)}), - ): - ci.run_openvmm_tests(backend, content) - - validate.assert_called_once_with(backend) - run_capture.assert_called_once_with( - ["git", "-C", BuildConstants.REPO_ROOT, "rev-parse", ":openvmm"] - ) - # The runner downloads the images and cargo-nextest itself. - require_tool.assert_not_called() - download.assert_not_called() - run_checked.assert_called_once() - command = run_checked.call_args.args[0] - resolved = content.resolve() - self.assertEqual( - command, - [ - resolved / "target" / "flowey_hvlite", - "pipeline", - "run", - "vmm-tests-run-target", - *arguments, - "--dir", - os.fspath(resolved), - "--filter", - ci.OPENVMM_TEST_FILTERS[backend], - ], - ) - self.assertEqual(run_checked.call_args.kwargs["cwd"], resolved) - environment = run_checked.call_args.kwargs["env"] - self.assertEqual( - environment["RUSTUP_TOOLCHAIN"], - OpenVMMBuildConstants.RUST_TOOLCHAIN, - ) - self.assertEqual( - environment["OPENVMM_MICROVM_TEST_KERNEL"], - os.fspath(kernel.resolve()), - ) - self.assertEqual( - environment["OPENVMM_MICROVM_TEST_INITRD"], - os.fspath(initrd.resolve()), - ) - self.assertEqual( - environment["XDG_CACHE_HOME"], os.fspath(root / "openvmm-cache") - ) - for name in ("VMM_TESTS_CONTENT_DIR", "TEST_OUTPUT_PATH"): - self.assertNotIn(name, environment) - if os.name != "nt": - # The artifact download dropped the runner's executable bit. - self.assertTrue(runner.stat().st_mode & 0o100) - - _OPENVMM_WINDOWS_TEST_ARGUMENTS: tuple[str, ...] = ( - "--target", - "windows-x64", - "--repetitions", - "1", - "--artifacts", - "VmgsWithBootEntry", - "--skip-vhd-prompt", - "--ci-profile", - ) - - def _openvmm_windows_test_content( - self, - content: Path, - revision: str, - arguments: tuple[str, ...] = _OPENVMM_WINDOWS_TEST_ARGUMENTS, - files: tuple[str, ...] = ( - ".config/nextest.toml", - "x86_64-pc-windows-msvc/flowey_hvlite.exe", - "x86_64-pc-windows-msvc/vmm_tests.tar.zst", - ), - ) -> None: - """Write WHP test content as the pinned flowey lays it out on Windows.""" - for relative in files: - path = content / relative - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(relative.encode()) - manifest = { - "format": ci.OPENVMM_TEST_CONTENT_FORMAT, - "openvmm_revision": revision, - "openvmm_clean": True, - "rust_toolchain": OpenVMMBuildConstants.RUST_TOOLCHAIN, - "operating_system": "windows", - "target": "windows-x64", - "build_options": list(ci.OPENVMM_TEST_BUILD_OPTIONS), - "filters": {"whp": ci.OPENVMM_TEST_FILTERS["whp"]}, - "build_filter": ci.OPENVMM_TEST_FILTERS["whp"], - "runner": "x86_64-pc-windows-msvc/flowey_hvlite.exe", - "arguments": list(arguments), - "files": [ - { - "path": relative, - "size": len(relative), - "executable": False, - "sha256": hashlib.sha256(relative.encode()).hexdigest(), - } - for relative in files - ], - } - (content / ci.OPENVMM_TEST_CONTENT_MANIFEST).write_text( - json.dumps(manifest), encoding="utf-8" - ) - - def _run_openvmm_windows_test_content( - self, - root: Path, - content: Path, - revision: str, - environment: Mapping[str, str] | None = None, - download: Callable[..., None] | None = None, - run_checked: MagicMock | None = None, - shard: ci.OpenVMMTestShard | None = None, - ) -> tuple[MagicMock, MagicMock, MagicMock, str]: - """Run WHP content, or a shard of it, on any host, and return the mocks - of the tools that the run resolves, of its downloads, and of the - commands that it runs, and what it prints.""" - gitlink = common.CommandResult( - args=("git", "rev-parse", ":openvmm"), - returncode=0, - stdout=f"{revision}\n".encode(), - stderr=b"", - ) - kernel = root / "vmlinux" - initrd = root / "initramfs.cpio.gz" - kernel.touch() - initrd.touch() - _, size = ci.OPENVMM_TEST_IMAGES["VmgsWithBootEntry"] - commands = run_checked or MagicMock() - - def write_image(_url: str, destination: Path) -> None: - destination.parent.mkdir(parents=True, exist_ok=True) - destination.write_bytes(b"\0" * size) - - with ( - patch.object(ci, "validate_openvmm_test_backend"), - patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), - patch.object(ci, "run_capture", return_value=gitlink), - patch.object( - ci, "require_tool", return_value="cargo-nextest" - ) as require_tool, - patch.object( - ci, "download", side_effect=download or write_image - ) as downloads, - patch.object(ci, "run_checked", commands), - patch.dict(os.environ, {"RUNNER_TEMP": os.fspath(root)}), - patch("sys.stdout", new_callable=io.StringIO) as stdout, - ): - os.environ.pop("VMM_TEST_IMAGES", None) - os.environ.update(environment or {}) - ci.run_openvmm_tests("whp", content, shard=shard) - return require_tool, downloads, commands, stdout.getvalue() - - def test_openvmm_tests_run_windows_content_from_its_archive(self): - # The pinned vmm-tests-run-target requires Hyper-V on Windows whatever - # tests it runs, which the WHP runners' Network Service account lacks, - # so the WHP content runs as the runner runs it after that check - # (#690): with its images, Petri's environment, and cargo-nextest. - revision = "e" * 40 - filename, size = ci.OPENVMM_TEST_IMAGES["VmgsWithBootEntry"] - self.assertEqual((filename, size), ("sample-vmgs.vhd", 4194816)) - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - content = root / "content" - content.mkdir() - self._openvmm_windows_test_content(content, revision) - # A rerun replaces the results of an earlier one. - stale = content / "test_results" / "stale.log" - stale.parent.mkdir() - stale.touch() - - require_tool, downloads, run_checked, output = ( - self._run_openvmm_windows_test_content( - root, - content, - revision, - # The runner sets both whatever its caller's are. - {"RUST_LOG": "info", "RUST_BACKTRACE": "0"}, - ) - ) - - resolved = content.resolve() - images = resolved / "images" - require_tool.assert_called_once_with("cargo-nextest") - downloads.assert_called_once_with( - f"https://hvlitetestvhds.blob.core.windows.net/vhds/{filename}", - images / filename, - ) - run_checked.assert_called_once() - self.assertEqual( - run_checked.call_args.args[0], - [ - "cargo-nextest", - "nextest", - "run", - "--profile", - "ci", - "--config-file", - os.fspath(resolved / ".config" / "nextest.toml"), - "--workspace-remap", - os.fspath(resolved), - "--archive-file", - os.fspath( - resolved / "x86_64-pc-windows-msvc" / "vmm_tests.tar.zst" - ), - "--filter-expr", - ci.OPENVMM_TEST_FILTERS["whp"], - ], - ) - self.assertEqual(run_checked.call_args.kwargs["cwd"], resolved) - environment = run_checked.call_args.kwargs["env"] - for name, value in { - "VMM_TESTS_CONTENT_DIR": os.fspath(resolved), - "TEST_OUTPUT_PATH": os.fspath(resolved / "test_results"), - "VMM_TEST_IMAGES": os.fspath(images), - "PETRI_REUSE_PREPPED_VHDS": "1", - "RUST_LOG": "trace", - "RUST_BACKTRACE": "1", - "RUSTUP_TOOLCHAIN": OpenVMMBuildConstants.RUST_TOOLCHAIN, - "OPENVMM_MICROVM_TEST_KERNEL": os.fspath((root / "vmlinux").resolve()), - "OPENVMM_MICROVM_TEST_INITRD": os.fspath( - (root / "initramfs.cpio.gz").resolve() - ), - }.items(): - with self.subTest(variable=name): - self.assertEqual(environment[name], value) - # As for the runner, Petri keeps its Windows cache directories. - self.assertNotIn("XDG_CACHE_HOME", environment) - self.assertEqual(list((resolved / "test_results").iterdir()), []) - self.assertEqual((images / filename).stat().st_size, size) - # Like the runner, the log shows the download, Petri's - # environment, and the command, in the order that they happen, - # and then whether the run's test durations were recorded. - lines = output.splitlines() - self.assertEqual( - lines[0], - f">> downloading VmgsWithBootEntry to {images / filename}", - ) - self.assertIn(f">> VMM_TESTS_CONTENT_DIR={resolved}", lines) - self.assertIn(">> PETRI_REUSE_PREPPED_VHDS=1", lines) - self.assertEqual( - lines[-2:], - [ - f">> {shlex.join(run_checked.call_args.args[0])}", - ">> recorded no test durations: nextest wrote no " - f"{resolved / 'target' / 'nextest' / 'ci' / 'junit.xml'}", - ], - ) - - # A JUnit report as nextest writes it for OpenVMM's ci profile: a test - # that passed, one that failed, one that passed only when retried, and - # one that it skipped. - _OPENVMM_JUNIT = """ - - - - output that the report drops - - - assertion failed - - - the first try failed - - - - - - -""" - _OPENVMM_JUNIT_TESTS = [ - { - "binary": "vmm_tests::tests", - "test": "multiarch::openvmm_uefi_x64_frontpage", - "seconds": 2.548, - "outcome": "passed", - }, - { - "binary": "vmm_tests::tests", - "test": "ttrpc::test_ttrpc_uefi_boot", - "seconds": 9.695, - "outcome": "failed", - }, - { - "binary": "vmm_tests::tests", - "test": "x86_64::openvmm_uefi_x64_guest_test_x64_crash_dump_on_triple_fault", - "seconds": 9.939, - "outcome": "flaky", - }, - { - "binary": "vmm_tests::tests", - "test": "multiarch::openvmm_uefi_x64_guest_test_x64_guest_test_uefi", - "seconds": 0.0, - "outcome": "skipped", - }, - ] - - def test_openvmm_content_runs_record_the_durations_of_their_tests(self): - # #691: each content run, of a shard or of all of a backend's tests, - # turns the JUnit report of its nextest run into a test duration - # report in its results, whether its tests pass or not, from which - # plan-openvmm-test-shards plans the shards. - revision = "e" * 40 - shard = ci.OpenVMMTestShard( - 1, - 2, - ("multiarch::openvmm_uefi_x64_frontpage", "ttrpc::test_ttrpc_uefi_boot"), - ) - for in_github, failure in ( - (True, None), - (False, common.ScriptError("nextest reports a failed test")), - ): - with ( - self.subTest(in_github=in_github, failure=failure), - tempfile.TemporaryDirectory() as temporary, - ): - root = Path(temporary) - content = root / "content" - content.mkdir() - self._openvmm_windows_test_content(content, revision) - - def run( - _command: list[str], - failure: common.ScriptError | None = failure, - **arguments: Any, - ) -> None: - junit = Path(arguments["cwd"]).joinpath( - *ci.OPENVMM_TEST_JUNIT.parts - ) - junit.parent.mkdir(parents=True) - junit.write_text(self._OPENVMM_JUNIT, encoding="utf-8") - if failure is not None: - raise failure - - environment = { - "GITHUB_ACTIONS": "true" if in_github else "false", - "GITHUB_RUN_ID": "39948600574", - "GITHUB_RUN_ATTEMPT": "2", - "RUNNER_NAME": "azure-windows-8", - } - run_checked = MagicMock(side_effect=run) - output = "" - with ( - patch.object(ci.os, "cpu_count", return_value=8), - contextlib.ExitStack() as stack, - ): - if failure is not None: - stack.enter_context( - self.assertRaisesRegex( - common.ScriptError, "nextest reports a failed test" - ) - ) - output = self._run_openvmm_windows_test_content( - root, - content, - revision, - environment, - run_checked=run_checked, - shard=shard, - )[3] - - # The shard runs only its tests of the backend's filter. - test_filter = ci.openvmm_test_filter("whp", shard) - self.assertEqual( - run_checked.call_args.args[0][-2:], ["--filter-expr", test_filter] - ) - report = ( - content.resolve() / "test_results" / ci.OPENVMM_TEST_DURATIONS_NAME - ) - self.assertEqual( - json.loads(report.read_text(encoding="utf-8")), - { - "schema": "nvx-openvmm-test-durations", - "schema_version": 1, - "backend": "whp", - "shard": {"index": 1, "count": 2}, - "filter": test_filter, - "logical_processors": 8, - "github": common.github_job({**os.environ, **environment}), - "tests": self._OPENVMM_JUNIT_TESTS, - }, - ) - if failure is None: - self.assertEqual( - output.splitlines()[-1], - f">> recorded the durations of 4 tests in {report}", - ) - - def test_openvmm_content_runs_drop_an_earlier_runs_durations(self): - # A run whose nextest writes no JUnit report, as when it cannot start, - # leaves no test duration report, not even an earlier run's. - revision = "b" * 40 - gitlink = common.CommandResult( - args=("git", "rev-parse", ":openvmm"), - returncode=0, - stdout=f"{revision}\n".encode(), - stderr=b"", - ) - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - content = root / "content" - content.mkdir() - self._openvmm_test_content(content, "kvm", revision) - junit = content.joinpath(*ci.OPENVMM_TEST_JUNIT.parts) - junit.parent.mkdir(parents=True) - junit.write_text(self._OPENVMM_JUNIT, encoding="utf-8") - report = content / "test_results" / ci.OPENVMM_TEST_DURATIONS_NAME - report.parent.mkdir() - report.write_text("{}", encoding="utf-8") - kernel = root / "vmlinux" - initrd = root / "initramfs.cpio.gz" - kernel.touch() - initrd.touch() - with ( - patch.object(ci, "validate_openvmm_test_backend"), - patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), - patch.object(ci, "run_capture", return_value=gitlink), - patch.object(ci, "run_checked") as run_checked, - patch("sys.stdout", new_callable=io.StringIO) as stdout, - ): - ci.run_openvmm_tests("kvm", content) - - # The Linux runner runs the backend's filter. - self.assertEqual( - run_checked.call_args.args[0][-2:], - ["--filter", ci.OPENVMM_TEST_FILTERS["kvm"]], - ) - self.assertFalse(junit.exists()) - self.assertFalse(report.exists()) - self.assertEqual( - stdout.getvalue().splitlines()[-1], - ">> recorded no test durations: nextest wrote no " - f"{content.resolve().joinpath(*ci.OPENVMM_TEST_JUNIT.parts)}", - ) - - def test_openvmm_content_runs_only_warn_about_a_malformed_junit_report(self): - revision = "e" * 40 - for name, junit_text, message in ( - ("truncated", "", - "has a test case without a name or a time", - ), - ( - "negative time", - "", - "has a test case without a name or a time", - ), - ): - with ( - self.subTest(junit=name), - tempfile.TemporaryDirectory() as temporary, - ): - root = Path(temporary) - content = root / "content" - content.mkdir() - self._openvmm_windows_test_content(content, revision) - - def run( - _command: list[str], junit_text: str = junit_text, **arguments: Any - ) -> None: - junit = Path(arguments["cwd"]).joinpath( - *ci.OPENVMM_TEST_JUNIT.parts - ) - junit.parent.mkdir(parents=True) - junit.write_text(junit_text, encoding="utf-8") - - with patch("sys.stderr", new_callable=io.StringIO) as stderr: - self._run_openvmm_windows_test_content( - root, content, revision, run_checked=MagicMock(side_effect=run) - ) - self.assertIn( - "warning: could not record the test durations in ", - stderr.getvalue(), - ) - self.assertIn(message, stderr.getvalue()) - self.assertFalse( - (content / "test_results" / ci.OPENVMM_TEST_DURATIONS_NAME).exists() - ) - - def test_openvmm_tests_reuse_windows_content_images_of_the_declared_size(self): - revision = "e" * 40 - filename, size = ci.OPENVMM_TEST_IMAGES["VmgsWithBootEntry"] - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - cache = (root / "images").resolve() - # VMM_TEST_IMAGES moves the images, as it moves the runner's. - for cached, downloaded in ((size, False), (size - 1, True)): - with self.subTest(cached=cached): - content = root / f"content-{cached}" - content.mkdir() - self._openvmm_windows_test_content(content, revision) - cache.mkdir(exist_ok=True) - (cache / filename).write_bytes(b"\0" * cached) - - _, downloads, run_checked, _ = ( - self._run_openvmm_windows_test_content( - root, - content, - revision, - {"VMM_TEST_IMAGES": os.fspath(cache)}, - ) - ) - - self.assertEqual(downloads.called, downloaded) - self.assertEqual((cache / filename).stat().st_size, size) - self.assertEqual( - run_checked.call_args.kwargs["env"]["VMM_TEST_IMAGES"], - os.fspath(cache), - ) - self.assertFalse(common.path_exists(content / "images")) - - # A download of another size than OpenVMM declares fails the run. - content = root / "content-truncated" - content.mkdir() - self._openvmm_windows_test_content(content, revision) - - def truncated(_url: str, destination: Path) -> None: - destination.parent.mkdir(parents=True, exist_ok=True) - destination.write_bytes(b"\0" * (size - 1)) - - with self.assertRaisesRegex( - common.ScriptError, - f"{filename} has {size - 1} bytes, but OpenVMM declares {size}", - ): - self._run_openvmm_windows_test_content( - root, content, revision, download=truncated - ) - self.assertFalse(common.path_exists(content / "images" / filename)) - - def test_openvmm_tests_refuse_windows_content_that_only_its_runner_runs(self): - revision = "f" * 40 - options: list[str] = list(self._OPENVMM_WINDOWS_TEST_ARGUMENTS) - archive = "x86_64-pc-windows-msvc/vmm_tests.tar.zst" - runner = "x86_64-pc-windows-msvc/flowey_hvlite.exe" - configuration = ".config/nextest.toml" - - def replace(option: str, value: str) -> tuple[str, ...]: - changed: list[str] = list(options) - changed[changed.index(option) + 1] = value - return tuple(changed) - - for name, arguments, files, message in ( - ( - "prep steps", - (*options, "--prep-steps", "standard"), - None, - "passes --prep-steps, which only its vmm-tests-run-target", - ), - ( - "IGVM agent", - (*options, "--needs-igvm-agent"), - None, - "passes --needs-igvm-agent, which only", - ), - ( - "hardware isolation", - (*options, "--needs-hardware-isolation"), - None, - "passes --needs-hardware-isolation, which only", - ), - ( - "repetitions", - replace("--repetitions", "2"), - None, - "repeats its tests 2 times", - ), - ( - "Linux target", - replace("--target", "linux-x64"), - None, - "targets linux-x64", - ), - ("no target", tuple(options[2:]), None, "targets None"), - ( - "unknown image", - replace("--artifacts", "VmgsWithBootEntry,VmgsWithoutBootEntry"), - None, - "needs the VmgsWithoutBootEntry image", - ), - ( - "unknown option", - (*options, "--install-missing-deps"), - None, - "unsupported option --install-missing-deps", - ), - ( - "missing value", - (*options, "--artifacts"), - None, - "passes --artifacts without a value", - ), - ( - "no archive", - tuple(options), - (configuration, runner), - "holds 0 vmm_tests.tar.zst archives, expected 1", - ), - ( - "two archives", - tuple(options), - ( - configuration, - runner, - archive, - "x86_64-unknown-linux-gnu/vmm_tests.tar.zst", - ), - "holds 2 vmm_tests.tar.zst archives, expected 1", - ), - ( - "no nextest configuration", - tuple(options), - (runner, archive), - "lacks .config/nextest.toml", - ), - ): - with ( - self.subTest(case=name), - tempfile.TemporaryDirectory() as temporary, - ): - root = Path(temporary) - content = root / "content" - content.mkdir() - if files is None: - self._openvmm_windows_test_content(content, revision, arguments) - else: - self._openvmm_windows_test_content( - content, revision, arguments, files - ) - run_checked = MagicMock() - with self.assertRaisesRegex(common.ScriptError, message): - self._run_openvmm_windows_test_content( - root, content, revision, run_checked=run_checked - ) - run_checked.assert_not_called() - - def test_openvmm_tests_refuse_content_that_is_not_this_checkouts(self): - backend = "whp" if os.name == "nt" else "kvm" - other_backend = "kvm" if os.name == "nt" else "mshv" - revision = "c" * 40 - gitlink = common.CommandResult( - args=("git", "rev-parse", ":openvmm"), - returncode=0, - stdout=f"{revision}\n".encode(), - stderr=b"", - ) - - def edit(manifest: dict[str, object], change: str, content: Path) -> None: - if change == "format": - # Content of the previous format, such as an earlier run built. - manifest["format"] = ci.OPENVMM_TEST_CONTENT_FORMAT - 1 - elif change == "revision": - manifest["openvmm_revision"] = "d" * 40 - elif change == "backend": - manifest["filters"] = { - other_backend: ci.OPENVMM_TEST_FILTERS[other_backend] - } - elif change == "filter": - manifest["filters"] = {backend: "all()"} - elif change == "runner": - manifest["runner"] = "../flowey_hvlite" - elif change == "arguments": - manifest["arguments"] = ["--target", 1] - elif change == "outside": - manifest["files"] = [{"path": "../escape", "size": 0}] - elif change == "windows outside": - manifest["files"] = [{"path": "..\\escape", "size": 0}] - elif change == "drive": - manifest["files"] = [{"path": "C:escape", "size": 0}] - elif change == "missing": - (content / "target" / "flowey_hvlite").unlink() - elif change == "truncated": - (content / "target" / "flowey_hvlite").write_bytes(b"run") - - for change, message in ( - ("format", "has format 1, expected 2"), - ("revision", f"of OpenVMM d{{40}}, but this checkout pins {revision}"), - ("backend", "no tests of this checkout's"), - ("filter", "no tests of this checkout's"), - ("runner", "is malformed"), - ("arguments", "is malformed"), - ("outside", "lists a file outside its directory"), - ("windows outside", "lists a file outside its directory"), - ("drive", "lists a file outside its directory"), - ("missing", "lacks target/flowey_hvlite"), - ("truncated", "lacks target/flowey_hvlite"), - ): - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - content = root / "content" - content.mkdir() - self._openvmm_test_content(content, backend, revision) - manifest_path = content / ci.OPENVMM_TEST_CONTENT_MANIFEST - manifest = json.loads(manifest_path.read_text(encoding="utf-8")) - edit(manifest, change, content) - manifest_path.write_text(json.dumps(manifest), encoding="utf-8") - kernel = root / "vmlinux" - initrd = root / "initramfs.cpio.gz" - kernel.touch() - initrd.touch() - with ( - self.subTest(change=change), - patch.object(ci, "artifact_path", side_effect=[kernel, initrd]), - patch.object(ci.os, "access", return_value=True), - patch.object(ci.Path, "exists", return_value=False), - patch.object(ci, "run_capture", return_value=gitlink), - patch.object(ci, "run_checked") as run_checked, - self.assertRaisesRegex(common.ScriptError, message), - ): - ci.run_openvmm_tests(backend, content) - run_checked.assert_not_called() - - -def _nextest_filter_matches(expression: str, test: str) -> bool: - """Return whether a nextest filter that OpenVMM's test filters spell, - with test(/REGEX/), test(SUBSTRING), !, &, |, and parentheses, selects a - test's name.""" - tokens = re.findall( - r"test\(/(?:[^/\\]|\\.)*/\)|test\([^/)][^)]*\)|[!&|()]", expression - ) - if "".join(tokens) != "".join(expression.split()): - raise AssertionError(f"unsupported filter {expression!r}") - position = 0 - - def take() -> str: - nonlocal position - position += 1 - return tokens[position - 1] - - def peek() -> str | None: - return tokens[position] if position < len(tokens) else None - - def primary() -> bool: - token = take() - if token == "!": - return not primary() - if token == "(": - value = union() - if take() != ")": - raise AssertionError(f"unbalanced filter {expression!r}") - return value - if token.startswith("test(/"): - return re.search(token[6:-2], test) is not None - # Without slashes, test() matches a substring of the name. - return token[5:-1] in test - - def intersection() -> bool: - value = primary() - while peek() == "&": - take() - value = primary() and value - return value - - def union() -> bool: - value = intersection() - while peek() == "|": - take() - value = intersection() or value - return value - - selected = union() - if position != len(tokens): - raise AssertionError(f"unbalanced filter {expression!r}") - return selected - - -class OpenVMMTestShardTests(unittest.TestCase): - """The checked-in partition of the WHP VMM tests into the shards that CI - runs in parallel, the model of nextest's runs with which the planner - balances them, the planner, and test-openvmm --shard (#691).""" - - def _partition( - self, shards: tuple[tuple[str, ...], ...] | None = None, **changes: Any - ) -> openvmm_test_shards.Partition: - """Return the checked-in partition with other WHP shards or fields.""" - partition = openvmm_test_shards.load_partition() - whp = partition.backends["whp"] - backend = openvmm_test_shards.BackendShards( - changes.get("test_threads", whp.test_threads), - changes.get("seconds", whp.seconds), - whp.shards if shards is None else shards, - ) - return openvmm_test_shards.Partition( - partition.runs, {**changes.get("backends", {}), "whp": backend} - ) - - def test_partition_puts_each_whp_test_in_exactly_one_shard(self): - partition = openvmm_test_shards.load_partition() - selection = ci.OPENVMM_TEST_SELECTIONS["whp"] - self.assertEqual(openvmm_test_shards.partition_problems(partition), []) - # Only the WHP filter names each of its tests. - self.assertEqual(tuple(partition.backends), ("whp",)) - self.assertEqual(tuple(ci.OPENVMM_TEST_SELECTIONS), ("whp",)) - self.assertTrue(partition.runs) - shards = partition.backends["whp"] - self.assertGreaterEqual(len(shards.shards), 2) - self.assertTrue(all(shards.shards)) - self.assertEqual( - sorted(test for shard in shards.shards for test in shard), - sorted(selection), - ) - self.assertEqual(set(shards.seconds), set(selection)) - # The WHP runners' logical processors. - self.assertEqual(shards.test_threads, 8) - - def test_partition_is_the_balanced_plan_of_its_durations(self): - # The file is the planner's output for its own durations: no move or - # swap shortens the longest modeled shard by MINIMUM_GAIN_TENTHS, and - # each shard's duration is its modeled run. - partition = openvmm_test_shards.load_partition() - self.assertEqual( - openvmm_test_shards.PARTITION_PATH.read_text(encoding="utf-8"), - openvmm_test_shards.partition_json(partition), - ) - self.assertEqual( - openvmm_test_shards.plan_partition( - partition, - samples={}, - runs=(), - estimates={}, - counts={}, - test_threads={}, - ), - partition, - ) - # Each shard runs in less than half of the unsharded tests' time. - whp = partition.backends["whp"] - unsharded = whp.run_seconds(whp.seconds) - for seconds in whp.shard_seconds(): - self.assertLess(seconds, unsharded / 2) - - def test_shards_run_exactly_the_tests_of_the_unsharded_filter(self): - # Every test that the WHP filter selects runs in exactly one shard, - # and a shard runs no test that the filter leaves out, such as an - # excluded test or a Linux backend's test. - partition = openvmm_test_shards.load_partition() - count = len(partition.backends["whp"].shards) - unsharded = ci.OPENVMM_TEST_FILTERS["whp"] - candidates = { - *ci.OPENVMM_WHP_TESTS, - "ttrpc::test_ttrpc_interface", - "multiarch::openvmm_uefi_x64_frontpage_extra", - "prefix::multiarch::openvmm_uefi_x64_frontpage", - } - filters: list[str] = [] - for index in range(1, count + 1): - shard = openvmm_test_shards.select_shard("whp", (index, count)) - self.assertEqual((shard.index, shard.count), (index, count)) - # In the filter's order. - self.assertEqual( - list(shard.tests), - [ - test - for test in ci.OPENVMM_TEST_SELECTIONS["whp"] - if test in shard.tests - ], - ) - test_filter = ci.openvmm_test_filter("whp", shard) - self.assertTrue(test_filter.startswith(f"({unsharded}) & (")) - filters.append(test_filter) - for test in sorted(candidates): - with self.subTest(test=test): - selected = _nextest_filter_matches(unsharded, test) - self.assertEqual(selected, test in ci.OPENVMM_TEST_SELECTIONS["whp"]) - self.assertEqual( - sum( - _nextest_filter_matches(test_filter, test) - for test_filter in filters - ), - 1 if selected else 0, - ) - self.assertEqual(ci.openvmm_test_filter("whp"), unsharded) - with self.assertRaisesRegex(common.ScriptError, "shard 1 of 2 .* runs no test"): - ci.openvmm_test_filter("whp", ci.OpenVMMTestShard(1, 2, ())) - - def test_shard_fails_before_any_test_unless_each_test_is_in_one_shard(self): - partition = openvmm_test_shards.load_partition() - first, second, *_ = partition.backends["whp"].shards - whp = partition.backends["whp"] - cases: list[tuple[str, openvmm_test_shards.Partition, str, str, str]] = [ - ( - "test in two shards", - self._partition((first, (*second, first[0]))), - "whp", - "2/2", - rf"whp: {re.escape(first[0])} is in more than one shard: 1, 2", - ), - ( - "test in no shard", - self._partition((first[1:], second)), - "whp", - "1/2", - rf"whp: {re.escape(first[0])} is in no shard", - ), - ( - "test outside the filter", - self._partition((first, (*second, "ttrpc::test_ttrpc_interface"))), - "whp", - "1/2", - "whp: ttrpc::test_ttrpc_interface is not a test of its filter", - ), - ( - "empty shard", - self._partition((first, second, ())), - "whp", - "1/3", - "whp shard 3 is empty", - ), - ( - "test without a duration", - self._partition( - seconds={ - test: seconds - for test, seconds in whp.seconds.items() - if test != first[0] - } - ), - "whp", - "1/2", - rf"whp: {re.escape(first[0])} has no duration", - ), - ( - "pattern filter", - self._partition(backends={"kvm": whp}), - "whp", - "1/2", - "kvm has no filter that names each of its tests", - ), - # CI runs fewer or more jobs than the WHP tests have shards. - ("fewer jobs", partition, "whp", "1/1", "splits the WHP tests into 2"), - ("more jobs", partition, "whp", "3/3", "this run is shard 3 of 3"), - ("unsharded backend", partition, "kvm", "1/2", "does not shard the KVM"), - ] - for name, changed, backend, spec, message in cases: - with ( - self.subTest(case=name), - patch.object( - openvmm_test_shards, "load_partition", return_value=changed - ), - patch.object(nvx, "run_openvmm_tests") as run, - self.assertRaisesRegex(common.ScriptError, message) as raised, - ): - nvx.command_test_openvmm( - nvx.parse_args( - [ - "test-openvmm", - "--backend", - backend, - "--shard", - spec, - "--content", - "content", - ] - ) - ) - run.assert_not_called() - if name not in ("fewer jobs", "more jobs", "unsharded backend"): - self.assertIn("plan-openvmm-test-shards", str(raised.exception)) - - def test_test_openvmm_runs_the_shard_that_it_selects(self): - partition = openvmm_test_shards.load_partition() - whp = partition.backends["whp"] - for index, tests in enumerate(whp.shards, 1): - with ( - self.subTest(shard=index), - patch.object(nvx, "run_openvmm_tests") as run, - ): - nvx.command_test_openvmm( - nvx.parse_args( - [ - "test-openvmm", - "--backend", - "whp", - "--shard", - f"{index}/{len(whp.shards)}", - "--content", - "content", - ] - ) - ) - run.assert_called_once_with( - "whp", - Path("content"), - shard=ci.OpenVMMTestShard( - index, - len(whp.shards), - tuple( - test - for test in ci.OPENVMM_TEST_SELECTIONS["whp"] - if test in tests - ), - ), - ) - with patch.object(nvx, "run_openvmm_tests") as run: - nvx.command_test_openvmm( - nvx.parse_args(["test-openvmm", "--backend", "kvm"]) - ) - run.assert_called_once_with("kvm", None, shard=None) - - def test_model_runs_tests_in_name_order_with_the_threads_they_require(self): - threads = openvmm_test_shards.threads_required - self.assertEqual(threads("multiarch::boot", 8), 2) - self.assertEqual(threads("multiarch::boot_heavy", 8), 8) - self.assertEqual(threads("multiarch::boot_heavy", 16), 12) - self.assertEqual(threads("multiarch::boot_very_heavy", 8), 8) - self.assertEqual(threads("multiarch::boot_very_heavy", 64), 32) - - def run(seconds: Mapping[str, int], test_threads: int = 8) -> float: - return ( - openvmm_test_shards.run_tenths( - seconds, - {test: value * 10 for test, value in seconds.items()}, - test_threads, - ) - / 10 - ) - - # Four tests of two threads each run at once on eight threads; the - # fifth starts when the first of them ends. - self.assertEqual(run({"a": 10, "b": 4, "c": 10, "d": 10, "e": 5}), 10) - self.assertEqual(run({"a": 10, "b": 4, "c": 10, "d": 10, "e": 7}), 11) - self.assertEqual(run({}), 0) - # A heavy test takes every thread, so it waits for each earlier test - # and each later test waits for it: nextest never starts a test - # before an earlier one, even where it would fit. - self.assertEqual(run({"a": 10, "b_heavy": 5, "c": 3}), 18) - self.assertEqual(run({"a_heavy": 5, "b": 10, "c": 3}), 15) - # With sixteen threads, a heavy test leaves four for others. - self.assertEqual(run({"a": 10, "b_heavy": 5, "c": 3}, 16), 10) - - def test_plan_balances_the_modeled_runs_rather_than_the_sums(self): - # Balancing the sums gives each shard a heavy test, which stalls each - # shard's other tests; the model puts both into one shard. - seconds = { - "a_heavy": 20.0, - "b_heavy": 20.0, - "c": 40.0, - "d": 40.0, - "e": 40.0, - "f": 40.0, - } - durations = {test: shard_plans.tenths(value) for test, value in seconds.items()} - - def cost(tests: Sequence[str]) -> int: - return openvmm_test_shards.run_tenths(tests, durations, 8) - - by_sums = shard_plans.plan_shards(seconds, 2) - self.assertEqual([cost(shard) for shard in by_sums], [600, 600]) - planned = shard_plans.plan_shards(seconds, 2, cost=cost, noun="tests") - self.assertEqual(planned, (("c", "d", "e", "f"), ("a_heavy", "b_heavy"))) - self.assertEqual([cost(shard) for shard in planned], [400, 400]) - with self.assertRaisesRegex(common.ScriptError, "cannot split 6 tests"): - shard_plans.plan_shards(seconds, 7, cost=cost, noun="tests") - - def test_plan_shards_only_backends_whose_filters_name_their_tests(self): - partition = openvmm_test_shards.load_partition() - common_arguments: dict[str, Any] = { - "samples": {}, - "runs": (), - "estimates": {}, - } - for counts, test_threads, message in ( - ({"kvm": 2}, {}, "the KVM filter selects its tests by pattern"), - ({}, {"mshv": 8}, "the MSHV filter selects its tests by pattern"), - ): - with ( - self.subTest(counts=counts, test_threads=test_threads), - self.assertRaisesRegex(common.ScriptError, message), - ): - openvmm_test_shards.plan_partition( - partition, - counts=counts, - test_threads=test_threads, - **common_arguments, - ) - # A new partition needs its number of shards and of test threads. - whp = partition.backends["whp"] - estimates = dict(whp.seconds) - for counts, test_threads, message in ( - ({}, {}, "shards no backend; pass --shards BACKEND=COUNT"), - ({"whp": 2}, {}, "pass --test-threads whp=COUNT"), - ({}, {"whp": 8}, "pass --shards whp=COUNT"), - ): - with ( - self.subTest(counts=counts, test_threads=test_threads), - self.assertRaisesRegex(common.ScriptError, message), - ): - openvmm_test_shards.plan_partition( - None, - samples={}, - runs=(), - estimates=estimates, - counts=counts, - test_threads=test_threads, - ) - # Without a duration of each test, the plan names the missing ones. - with self.assertRaisesRegex( - common.ScriptError, - r"no report measures, and the partition has no duration of, " - r"ttrpc::test_ttrpc_uefi_boot on whp; pass --report", - ): - openvmm_test_shards.plan_partition( - None, - samples={}, - runs=(), - estimates={ - test: value - for test, value in estimates.items() - if test != "ttrpc::test_ttrpc_uefi_boot" - }, - counts={"whp": 2}, - test_threads={"whp": 8}, - ) - planned = openvmm_test_shards.plan_partition( - None, - samples={}, - runs=(), - estimates=estimates, - counts={"whp": 2}, - test_threads={"whp": 8}, - ) - self.assertEqual(openvmm_test_shards.partition_problems(planned), []) - self.assertEqual(planned.runs, ()) - - def _write_report( - self, - directory: Path, - tests: Sequence[tuple[str, float, str]], - *, - run: str | None = "39948600574-1", - backend: str = "whp", - processors: int | None = 8, - ) -> Path: - """Write the test duration report that a content run writes from - nextest's JUnit report of the tests, with their seconds and outcome.""" - junit = directory / "junit.xml" - cases: list[str] = [] - for test, seconds, outcome in tests: - child = { - "passed": "", - "failed": "", - "flaky": "", - "skipped": "", - }[outcome] - cases.append( - f'{child}' - ) - directory.mkdir(parents=True, exist_ok=True) - junit.write_text( - "" + "".join(cases) + "", - encoding="utf-8", - ) - report = directory / ci.OPENVMM_TEST_DURATIONS_NAME - run_id, _, attempt = (run or "").partition("-") - environment = { - "GITHUB_ACTIONS": "true" if run else "false", - "GITHUB_RUN_ID": run_id, - "GITHUB_RUN_ATTEMPT": attempt, - } - with ( - patch.dict(os.environ, environment), - patch.object(ci.os, "cpu_count", return_value=processors), - patch("sys.stdout", new_callable=io.StringIO), - ): - ci._record_openvmm_test_durations( - junit, report, backend, None, ci.OPENVMM_TEST_FILTERS[backend] - ) - return report - - def test_reports_give_samples_of_the_tests_that_passed(self): - frontpage = "multiarch::openvmm_uefi_x64_frontpage" - uefi = "ttrpc::test_ttrpc_uefi_boot" - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - first = self._write_report( - root / "1", - [(frontpage, 2.5, "passed"), (uefi, 9.0, "failed")], - run="101-1", - ) - self._write_report( - root / "2" / "shard-1", - [(frontpage, 3.0, "passed"), (uefi, 9.5, "passed")], - run="102-2", - processors=16, - ) - # A flaky or skipped test gives no sample, and a report that gives - # none adds no run. - self._write_report( - root / "3", - [(frontpage, 30.0, "flaky"), (uefi, 0.0, "skipped")], - run="103-1", - ) - # A local run's report adds samples but no run. - self._write_report(root / "4", [(frontpage, 2.0, "passed")], run=None) - self._write_report( - root / "5", [(frontpage, 40.0, "passed")], run="105-1", backend="kvm" - ) - samples, runs, processors = openvmm_test_shards.read_reports( - # The same report found twice counts once. - [root, first] - ) - self.assertEqual( - samples, - { - "whp": {frontpage: [2.5, 3.0, 2.0], uefi: [9.5]}, - "kvm": {frontpage: [40.0]}, - }, - ) - self.assertEqual(runs, {"101-1", "102-2", "105-1"}) - self.assertEqual(processors, {"whp": {8, 16}, "kvm": {8}}) - - def test_reports_that_are_not_test_duration_reports_fail(self): - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - report = root / ci.OPENVMM_TEST_DURATIONS_NAME - valid = json.loads( - self._write_report( - root / "valid", [("ttrpc::test_ttrpc_uefi_boot", 9.0, "passed")] - ).read_text(encoding="utf-8") - ) - for name, text, message in ( - ("not JSON", "{", "is not valid JSON"), - ("not an object", "[]", "is not a test duration report"), - ( - "other schema", - json.dumps({**valid, "schema": "nvx-microvm-durations"}), - "is not a test duration report of schema version 1", - ), - ( - "other version", - json.dumps({**valid, "schema_version": 2}), - "is not a test duration report", - ), - ( - "unknown backend", - json.dumps({**valid, "backend": "hvf"}), - "has a malformed backend or tests", - ), - ( - "malformed test", - json.dumps({**valid, "tests": [{"test": "x"}]}), - "has a malformed test", - ), - ): - with self.subTest(case=name): - report.write_text(text, encoding="utf-8") - with self.assertRaisesRegex(common.ScriptError, message): - openvmm_test_shards.read_reports([report]) - with self.assertRaisesRegex( - common.ScriptError, "is neither a report nor a directory" - ): - openvmm_test_shards.read_reports([root / "missing"]) - - def test_plan_command_writes_the_partition_from_reports(self): - partition = openvmm_test_shards.load_partition() - whp = partition.backends["whp"] - first, second = whp.shards - moved = first[0] - with tempfile.TemporaryDirectory() as temporary: - root = Path(temporary) - path = root / "shards.json" - path.write_text( - openvmm_test_shards.partition_json(partition), encoding="utf-8" - ) - # Reports of two runs whose runners have sixteen logical - # processors, in which one test of the first shard takes far - # longer than its median. - for run in ("201-1", "202-1"): - self._write_report( - root / "reports" / run, - [ - (test, 300.0 if test == moved else seconds, "passed") - for test, seconds in whp.seconds.items() - ], - run=run, - processors=16, - ) - with patch("sys.stdout", new_callable=io.StringIO) as stdout: - self.assertEqual( - nvx.main( - [ - "plan-openvmm-test-shards", - "--report", - str(root / "reports"), - "--partition", - str(path), - ] - ), - 0, - ) - planned = openvmm_test_shards.load_partition(path) - self.assertEqual(openvmm_test_shards.partition_problems(planned), []) - self.assertEqual(planned.runs, ("201-1", "202-1")) - self.assertEqual(planned.backends["whp"].seconds[moved], 300.0) - # The plan keeps its test threads but notes the reports'. - self.assertEqual(planned.backends["whp"].test_threads, 8) - output = stdout.getvalue() - self.assertIn("whp: 2 shards of 29 tests", output) - self.assertIn( - "note: runners of the whp reports have 16 logical processors, " - "but the plan models 8 test threads", - output, - ) - self.assertTrue(output.rstrip().endswith(f"Wrote {path}")) - self.assertNotEqual(planned.backends["whp"].shards, (first, second)) - - # Estimates must name tests of a sharded filter. - with self.assertRaisesRegex( - common.ScriptError, "--seconds names tests that no sharded filter" - ): - arguments = nvx.parse_args( - [ - "plan-openvmm-test-shards", - "--seconds", - "ttrpc::test_ttrpc_interface=10", - "--partition", - str(path), - ] - ) - arguments.handler(arguments) - # Reports that measure no test that passed fail the plan. - empty = root / "empty" - self._write_report(empty, [("ttrpc::test_ttrpc_uefi_boot", 9.0, "failed")]) - with self.assertRaisesRegex( - common.ScriptError, "no test duration report measures a test" - ): - arguments = nvx.parse_args( - [ - "plan-openvmm-test-shards", - "--report", - str(empty), - "--partition", - str(path), - ] - ) - arguments.handler(arguments) - - def test_load_partition_rejects_malformed_documents(self): - valid: dict[str, Any] = json.loads( - openvmm_test_shards.PARTITION_PATH.read_text(encoding="utf-8") + "multiarch::pcie::openvmm_uefi_x64_windows_datacenter_core_2022_x64_pcie_nvme_boot", + ci.OPENVMM_WHP_EXCLUDED_TESTS, ) - whp: dict[str, Any] = valid["backends"]["whp"] - cases: list[tuple[str, str | dict[str, Any], str]] = [ - ("not JSON", "{", "is not valid JSON"), - ( - "other schema", - {**valid, "schema": "nvx-microvm-shards"}, - "is not an nvx-openvmm-test-shards document", - ), - ("runs", {**valid, "runs": [1]}, "malformed runs or backends"), - ( - "test threads", - {**valid, "backends": {"whp": {**whp, "test_threads": 0}}}, - "malformed partition of whp", - ), - ( - "boolean test threads", - {**valid, "backends": {"whp": {**whp, "test_threads": True}}}, - "malformed partition of whp", + self.assertTrue( + set(ci.OPENVMM_WHP_EXCLUDED_TESTS).issubset(ci.OPENVMM_WHP_TESTS) + ) + self.assertEqual( + ci.OPENVMM_TEST_FILTERS["whp"], + ci._join_openvmm_tests( + ci.OPENVMM_WHP_TESTS, + ci.OPENVMM_WHP_EXCLUDED_TESTS, ), - ( - "negative duration", - { - **valid, - "backends": { - "whp": {**whp, "seconds": {**whp["seconds"], "x": -1}} - }, - }, - "malformed partition of whp", + ) + self.assertEqual( + ci._join_openvmm_tests( + ("suite::boot", "suite::boot_heavy"), + ("suite::boot_heavy",), ), ( - "shard fields", - { - **valid, - "backends": { - "whp": {**whp, "shards": [{"seconds": 1, "scenarios": []}]} - }, - }, - "malformed partition of whp", + "(test(/^suite::boot$/) | test(/^suite::boot_heavy$/))" + " & !test(/^suite::boot_heavy$/)" ), + ) + + def test_linux_filters_leave_out_every_windows_server_test(self): + # The Linux legs run Linux guests. A test's name holds the year of its + # Windows Server image, so a test that moves to another image, as the + # KVP IC test moved from 2022 to 2025 (#1035), must stay out of them. + windows_tests = [ + test for test in ci.OPENVMM_WHP_TESTS if "windows_datacenter_core" in test ] - for name, document, message in cases: - with ( - self.subTest(case=name), - tempfile.TemporaryDirectory() as temporary, + self.assertIn( + "multiarch::ic::openvmm_uefi_x64_windows_datacenter_core_2025_x64_kvp_ic", + windows_tests, + ) + for backend in ("kvm", "mshv"): + test_filter = ci.OPENVMM_TEST_FILTERS[backend] + for test in windows_tests: + with self.subTest(backend=backend, test=test): + self.assertFalse(_nextest_filter_matches(test_filter, test)) + for test in ( + "multiarch::ic::openvmm_uefi_x64_ubuntu_2504_server_x64_timesync_ic", + *ci.OPENVMM_REQUIRED_MICROVM_TESTS, ): - path = Path(temporary) / "shards.json" - path.write_text( - document if isinstance(document, str) else json.dumps(document), - encoding="utf-8", - ) - with self.assertRaisesRegex(common.ScriptError, message): - openvmm_test_shards.load_partition(path) - with ( - tempfile.TemporaryDirectory() as temporary, - self.assertRaisesRegex( - common.ScriptError, "does not exist; plan the shards again" - ), - ): - openvmm_test_shards.load_partition(Path(temporary) / "missing.json") - - def test_model_matches_the_pinned_nextest_configuration(self): - # The planner models nextest with the test threads that OpenVMM's - # nextest configuration requires of each VMM test, and test-openvmm - # reads the JUnit report that it writes for the ci profile. Both are - # read from the gitlink's commit, not from the submodule's working - # tree, which can hold another revision. - root = BuildConstants.REPO_ROOT - submodule = root / "openvmm" - if not (submodule / ".git").exists(): - self.skipTest("the OpenVMM submodule is not initialized") + with self.subTest(backend=backend, test=test): + self.assertTrue(_nextest_filter_matches(test_filter, test)) - def git(directory: Path, *arguments: str) -> str | None: - result = subprocess.run( - ["git", "-C", str(directory), *arguments], - capture_output=True, - encoding="utf-8", - check=False, - ) - return result.stdout if result.returncode == 0 else None + def test_openvmm_tests_reject_unknown_backend(self): + with self.assertRaisesRegex(common.ScriptError, "unsupported.*backend"): + ci.run_openvmm_tests("unknown") - pin = (git(root, "rev-parse", ":openvmm") or "").strip() - if not pin or git(submodule, "cat-file", "-e", f"{pin}^{{commit}}") is None: - self.skipTest( - f"the OpenVMM submodule lacks its pinned revision {pin or '(unknown)'}" - ) - configuration = git(submodule, "show", f"{pin}:.config/nextest.toml") - assert configuration is not None - sections: list[tuple[str, dict[str, str]]] = [] - for line in configuration.splitlines(): - line = line.split("#", 1)[0].strip() - if not line: - continue - if line.startswith("["): - sections.append((line, {})) - elif sections: - key, _, value = line.partition("=") - sections[-1][1][key.strip()] = value.strip() - overrides = [ - (fields.get("filter"), int(fields["threads-required"])) - for header, fields in sections - if header == "[[profile.default.overrides]]" - and "threads-required" in fields - ] - self.assertEqual( - overrides, - [ - ("'package(~vmm_tests) and test(very_heavy)'", 32), - ( - "'package(~vmm_tests) and test(heavy) and not (test(very_heavy))'", - 12, - ), - ("'package(~vmm_tests)'", 2), - ], - ) - self.assertEqual( - openvmm_test_shards.THREADS_REQUIRED, (("very_heavy", 32), ("heavy", 12)) - ) - self.assertEqual(openvmm_test_shards.DEFAULT_THREADS_REQUIRED, 2) - # The ci profile, which inherits the default profile's overrides, - # requires no other threads and keeps nextest's default test - # threads, one for each logical processor. - for header, fields in sections: - with self.subTest(section=header): - if header.startswith("[[profile.ci."): - self.assertNotIn("threads-required", fields) - self.assertNotIn("test-threads", fields) - junit = dict(sections)["[profile.ci.junit]"] - self.assertEqual(junit["path"], '"junit.xml"') - self.assertEqual( - ci.OPENVMM_TEST_JUNIT, PurePosixPath("target", "nextest", "ci", "junit.xml") - ) + +def _nextest_filter_matches(expression: str, test: str) -> bool: + """Return whether a nextest filter that OpenVMM's test filters spell, + with test(/REGEX/), test(SUBSTRING), !, &, |, and parentheses, selects a + test's name.""" + tokens = re.findall( + r"test\(/(?:[^/\\]|\\.)*/\)|test\([^/)][^)]*\)|[!&|()]", expression + ) + if "".join(tokens) != "".join(expression.split()): + raise AssertionError(f"unsupported filter {expression!r}") + position = 0 + + def take() -> str: + nonlocal position + position += 1 + return tokens[position - 1] + + def peek() -> str | None: + return tokens[position] if position < len(tokens) else None + + def primary() -> bool: + token = take() + if token == "!": + return not primary() + if token == "(": + value = union() + if take() != ")": + raise AssertionError(f"unbalanced filter {expression!r}") + return value + if token.startswith("test(/"): + return re.search(token[6:-2], test) is not None + # Without slashes, test() matches a substring of the name. + return token[5:-1] in test + + def intersection() -> bool: + value = primary() + while peek() == "&": + take() + value = primary() and value + return value + + def union() -> bool: + value = intersection() + while peek() == "|": + take() + value = intersection() or value + return value + + selected = union() + if position != len(tokens): + raise AssertionError(f"unbalanced filter {expression!r}") + return selected _REUSE_REPOSITORY = "microsoft/nvx" @@ -8440,12 +6125,6 @@ def load(name: str) -> Any: "setup-cross-os-cache": set[str](), "verify": set[str](), "test-openvmm": kernel_alpine, - # The VMM tests' build, its cache key, and the checks of what it - # builds or a cache restores boot no guest. - "build-openvmm-tests": set[str](), - "openvmm-test-cache-key": set[str](), - "check-openvmm-tests": set[str](), - "reuse-openvmm-tests": set[str](), # The runtime adapter's VM scenarios boot what nvx finds. "test-runtime-adapter": kernel_alpine, # A sandbox boots the Alpine control initramfs under its layers. @@ -8632,7 +6311,6 @@ def consume( self.assertEqual( consumers, { - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS, @@ -8669,21 +6347,8 @@ def consume( {name for name, files in components.items() if files & read}, ) self.assertIn("guest-ubuntu-distro", downloaded) - # The VMM tests' build downloads no guest artifact, none of its steps - # reads one, and it waits for no job that builds one. - for job_name in ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS: - job = jobs[job_name] - context = {"runner.os": "Linux"} - with self.subTest(build=job_name): - self.assertIsNone(consume(job["steps"], context)) - self.assertEqual( - set[str]().union(*(reads(step, context) for step in job["steps"])), - set(), - ) - self.assertFalse(set(job["needs"]) & set(producers.values())) - # test-openvmm boots the kernel and the Alpine initramfs, whether it - # builds the tests or runs the content of a build. + # test-openvmm boots the kernel and the Alpine initramfs. requested: list[str] = [] class Stop(Exception): @@ -8696,22 +6361,16 @@ def artifact(name: str) -> Path: def present(path: Path, _description: str) -> Path: return path - for content, stop in ( - (None, "_prepare_openvmm_test_environment"), - (Path("content"), "_openvmm_test_content"), + with ( + patch.object(ci, "validate_openvmm_test_backend"), + patch.object(ci, "require_file", side_effect=present), + patch.object(ci, "require_tool", return_value="tool"), + patch.object(ci, "artifact_path", side_effect=artifact), + patch.object(ci, "_prepare_openvmm_test_environment", side_effect=Stop), + self.assertRaises(Stop), ): - requested.clear() - with ( - self.subTest(content=content), - patch.object(ci, "validate_openvmm_test_backend"), - patch.object(ci, "require_file", side_effect=present), - patch.object(ci, "require_tool", return_value="tool"), - patch.object(ci, "artifact_path", side_effect=artifact), - patch.object(ci, stop, side_effect=Stop), - self.assertRaises(Stop), - ): - ci.run_openvmm_tests("kvm", content) - self.assertEqual(set(requested), commands["test-openvmm"]) + ci.run_openvmm_tests("kvm") + self.assertEqual(set(requested), commands["test-openvmm"]) def test_development_release_requires_successful_crate_checks(self): workflow = ( @@ -9035,14 +6694,9 @@ def test_required_ci_result_policy(self): | set(ci.REQUIRED_CI_CLI_JOBS) ) builds = set(ci.REQUIRED_CI_BUILD_JOBS) - openvmm_tests = set(ci.REQUIRED_CI_OPENVMM_TEST_JOBS) - # The VMM tests' build boots no guest, so it runs whenever a change - # selects the VMM tests, which the classifier only does together with - # the guest artifacts that the tests boot. - openvmm_test_builds = set(ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS) - openvmm_artifact_tests = openvmm_test_builds | set( - ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS - ) + # The check of OpenVMM's CI needs no secret and no artifact, so it runs + # whenever its flag is set, for a pull request from a fork too. + openvmm_ci = {ci.REQUIRED_CI_OPENVMM_CI_JOB} microvm_tests = set(ci.REQUIRED_CI_MICROVM_TEST_JOBS) debug_kernel_pull_request = set(ci.REQUIRED_CI_MICROVM_DEBUG_JOBS) debug_kernel_merge_group = set(ci.REQUIRED_CI_MICROVM_DEBUG_MERGE_GROUP_JOBS) @@ -9061,50 +6715,37 @@ def test_required_ci_result_policy(self): microvm_merge_group = microvm_tests | debug_kernel_merge_group microvm_full = microvm_tests | debug_kernel_pull_request | debug_kernel_push full_matrix = ( - repository - | artifacts - | builds - | openvmm_tests - | openvmm_artifact_tests - | microvm_full - | platforms + repository | artifacts | builds | openvmm_ci | microvm_full | platforms ) gate = {"performance-gate"} # Event, same repository, the change detector's outputs (guest - # artifacts, OpenVMM binaries, OpenVMM unit tests, OpenVMM VMM tests, - # NVX microVM tests, platform benchmarks and packages), and the jobs - # that must - # succeed; every other required job must be skipped. - everything = (True, True, True, True, True, True) + # artifacts, OpenVMM binaries, OpenVMM CI check, NVX microVM tests, + # platform benchmarks and packages), and the jobs that must succeed; + # every other required job must be skipped. + everything = (True, True, True, True, True) cases = ( # Documentation, or a change to no suite's inputs. - ("pull_request", True, (False,) * 6, repository), + ("pull_request", True, (False,) * 5, repository), # Only the benchmarks or the packages change. ( "pull_request", True, - (True, True, False, False, False, True), + (True, True, False, False, True), repository | artifacts | rehearsal | builds | platforms | gate, ), # Only the microVM tests change. ( "pull_request", True, - (True, True, False, False, True, False), + (True, True, False, True, False), repository | artifacts | rehearsal | builds | microvm_pull_request, ), - # Only the OpenVMM suites, which build OpenVMM themselves. + # Only the OpenVMM CI check, which needs nothing built. ( "pull_request", True, - (False, False, True, False, False, False), - repository | openvmm_tests, - ), - ( - "pull_request", - True, - (True, False, False, True, False, False), - repository | artifacts | rehearsal | openvmm_artifact_tests, + (False, False, True, False, False), + repository | openvmm_ci, ), ( "pull_request", @@ -9114,35 +6755,20 @@ def test_required_ci_result_policy(self): | artifacts | rehearsal | builds - | openvmm_tests - | openvmm_artifact_tests - | microvm_pull_request - | platforms - | gate, - ), - # A pull request that changes no input of either OpenVMM suite. - ( - "pull_request", - True, - (True, True, False, False, True, True), - repository - | artifacts - | rehearsal - | builds + | openvmm_ci | microvm_pull_request | platforms | gate, ), - # Guest inputs reach the VMM tests but not the unit tests. + # A pull request that changes neither the pin nor the check. ( "pull_request", True, - (True, True, False, True, True, True), + (True, True, False, True, True), repository | artifacts | rehearsal | builds - | openvmm_artifact_tests | microvm_pull_request | platforms | gate, @@ -9152,50 +6778,55 @@ def test_required_ci_result_policy(self): ( "pull_request", True, - (False, True, True, True, True, True), - repository | builds | openvmm_tests | openvmm_test_builds, + (False, True, True, True, True), + repository | builds | openvmm_ci, ), ( "pull_request", True, - (True, False, False, True, True, True), - repository | artifacts | rehearsal | openvmm_artifact_tests, + (True, False, False, True, True), + repository | artifacts | rehearsal, ), - ("pull_request", False, (False,) * 6, always_successful), + ("pull_request", False, (False,) * 5, always_successful), ( "pull_request", False, - (True, True, False, False, False, True), + (True, True, False, False, True), always_successful | artifacts, ), ( "pull_request", False, - (False, False, True, False, False, False), - always_successful, + (False, False, True, False, False), + always_successful | openvmm_ci, + ), + ( + "pull_request", + False, + everything, + always_successful | artifacts | openvmm_ci, ), - ("pull_request", False, everything, always_successful | artifacts), # A merge group runs a same-repository pull request's jobs, whatever # repository its pull requests come from, but not the rehearsal of # the guest artifact reuse, which its pull request's run made. - ("merge_group", True, (False,) * 6, repository), + ("merge_group", True, (False,) * 5, repository), ( "merge_group", True, - (True, True, False, False, False, True), + (True, True, False, False, True), repository | artifacts | builds | platforms | gate, ), ( "merge_group", True, - (True, True, False, False, True, False), + (True, True, False, True, False), repository | artifacts | builds | microvm_merge_group, ), ( "merge_group", True, - (True, False, False, True, False, False), - repository | artifacts | openvmm_artifact_tests, + (False, False, True, False, False), + repository | openvmm_ci, ), ( "merge_group", @@ -9204,8 +6835,7 @@ def test_required_ci_result_policy(self): repository | artifacts | builds - | openvmm_tests - | openvmm_artifact_tests + | openvmm_ci | microvm_merge_group | platforms | gate, @@ -9217,36 +6847,34 @@ def test_required_ci_result_policy(self): repository | artifacts | builds - | openvmm_tests - | openvmm_artifact_tests + | openvmm_ci | microvm_merge_group | platforms | gate, ), - ("push", True, (False,) * 6, repository), + ("push", True, (False,) * 5, repository), ( "push", True, - (True, True, False, False, False, True), + (True, True, False, False, True), repository | artifacts | builds | platforms, ), ( "push", True, - (True, True, False, False, True, True), + (True, True, False, True, True), repository | artifacts | builds | microvm_full | platforms, ), ( "push", True, - (False, False, True, False, False, False), - repository | openvmm_tests, + (False, False, True, False, False), + repository | openvmm_ci, ), ("push", True, everything, full_matrix), ("schedule", False, everything, full_matrix), ("workflow_dispatch", False, everything, full_matrix), ) - for event_name, same_repository, flags, successful_jobs in cases: changes = ci.CiChanges(*flags) with self.subTest( @@ -9353,8 +6981,8 @@ def test_ci_jobs_follow_their_change_detector_output(self): # true, or when the detector did not succeed, so that a failed # classification runs the full matrix. Required status check passes # every output to check-required-ci, the release and the performance - # persistence read whether the VMM tests run, and no other job reads - # one. + # persistence read whether the OpenVMM CI check runs, and no other job + # reads one. workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") @@ -9364,14 +6992,7 @@ def test_ci_jobs_follow_their_change_detector_output(self): ci.REQUIRED_CI_DEBUG_KERNEL_JOB: "run-guest-artifacts", ci.REQUIRED_CI_GUEST_ARTIFACT_REUSE_JOB: "run-guest-artifacts", **dict.fromkeys(ci.REQUIRED_CI_BUILD_JOBS, "run-openvmm-binaries"), - **dict.fromkeys(ci.REQUIRED_CI_OPENVMM_TEST_JOBS, "run-openvmm-unit-tests"), - **dict.fromkeys( - ( - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - ), - "run-openvmm-vmm-tests", - ), + ci.REQUIRED_CI_OPENVMM_CI_JOB: "run-openvmm-ci-check", **dict.fromkeys( ( *ci.REQUIRED_CI_MICROVM_TEST_JOBS, @@ -9403,9 +7024,8 @@ def test_ci_jobs_follow_their_change_detector_output(self): re.findall(r"needs\.openvmm-changes\.outputs\.([a-z-]+)", job), [output], ) - # The release and the recording of the performance results accept - # skipped VMM-test jobs only when the change selected no VMM tests - # (#690). + # The release and the recording of the performance results accept a + # skipped OpenVMM CI check only when the change selected none (#690). gates = ("release", "performance-record") for job_name in gates: with self.subTest(job=job_name): @@ -9414,7 +7034,7 @@ def test_ci_jobs_follow_their_change_detector_output(self): r"needs\.openvmm-changes\.outputs\.([a-z-]+)", _workflow_job(workflow, job_name), ), - ["run-openvmm-vmm-tests"], + ["run-openvmm-ci-check"], ) jobs = re.findall( r"^ ([A-Za-z0-9_-]+):$", workflow.split("\njobs:\n", 1)[1], re.MULTILINE @@ -9726,9 +7346,6 @@ def test_ci_repository_jobs_run_for_every_non_pull_request_event(self): *ci.REQUIRED_CI_NVX_RS_BUILD_JOBS, *ci.REQUIRED_CI_CLI_JOBS, *ci.REQUIRED_CI_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_TEST_JOBS, - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_TEST_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, *ci.REQUIRED_CI_PLATFORM_JOBS, @@ -9737,6 +7354,11 @@ def test_ci_repository_jobs_run_for_every_non_pull_request_event(self): for job_name in gated_jobs: with self.subTest(job=job_name): self.assertIn(repository_gate, _workflow_job(workflow, job_name)) + # The check of OpenVMM's CI needs no secret, so a pull request from a + # fork runs it too. + self.assertNotIn( + repository_gate, _workflow_job(workflow, ci.REQUIRED_CI_OPENVMM_CI_JOB) + ) self.assertNotIn("github.event_name == 'push' ||", workflow) for job_name in ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS: with self.subTest(job=job_name): @@ -9857,8 +7479,7 @@ def test_ci_full_runs_do_not_publish_record_or_save_caches(self): self.assertEqual( writes, { - "workflows/ci.yml": 10, - "actions/build-openvmm/action.yml": 2, + "actions/build-openvmm/action.yml": 6, "actions/build-guest-artifacts/action.yml": 6, "actions/check-quality/action.yml": 1, }, @@ -9880,23 +7501,23 @@ def test_ci_full_runs_do_not_publish_record_or_save_caches(self): with self.subTest(call=name): self.assertEqual(arguments.get("save-cache"), save_cache) # No other push-only job condition is allowed in this workflow: only - # its ten cache saves, the release, performance persistence, the four - # save-cache arguments, the two reuse arguments, and the two restores - # of the OpenVMM VMM tests, which full runs skip, name the push event. - self.assertEqual(workflow.count("github.event_name == 'push'"), 20) + # the release, performance persistence, the four save-cache arguments, + # and the two reuse arguments name the push event; ci.yml itself writes + # no cache. + self.assertEqual(workflow.count("github.event_name == 'push'"), 8) def test_ci_change_classification_selects_suites_by_input(self): - guest, binaries, _, vmm, microvm, benchmarks = ci.CI_CHANGE_OUTPUTS + guest, binaries, openvmm_ci, microvm, benchmarks = ci.CI_CHANGE_OUTPUTS def select(*outputs: str) -> ci.CiChanges: return ci.CiChanges(*(output in outputs for output in ci.CI_CHANGE_OUTPUTS)) everything = select(*ci.CI_CHANGE_OUTPUTS) - without_unit_tests = select(guest, binaries, vmm, microvm, benchmarks) - vmm_and_microvm = select(guest, binaries, vmm, microvm) microvm_and_benchmarks = select(guest, binaries, microvm, benchmarks) microvm_only = select(guest, binaries, microvm) benchmarks_only = select(guest, binaries, benchmarks) + openvmm_ci_only = select(openvmm_ci) + openvmm_ci_and_benchmarks = select(guest, binaries, openvmm_ci, benchmarks) # Each class of changed path, the outputs that a pull request selects, # and those that a push selects when they differ: a push runs the # platform and package jobs, and the jobs that build their artifacts, @@ -9915,60 +7536,52 @@ def select(*outputs: str) -> ci.CiChanges: select(), benchmarks_only, ), - # Inputs of every suite: the OpenVMM pin, the Rust pin, the - # runners, the workflow, the actions that every suite's jobs use, - # the CLI, and the modules that the OpenVMM suites run. + # Inputs of every suite and of the OpenVMM CI check: the OpenVMM + # pin, the workflow, the CLI, and the modules that the check runs. (("openvmm",), everything, None), - (("rust-toolchain.toml",), everything, None), - (("scripts/setup/setup-windows-whp.ps1",), everything, None), - (("scripts/setup/azure/build_package.py",), everything, None), ((".github/workflows/ci.yml",), everything, None), - ((".github/actions/validate-runner/action.yml",), everything, None), - ((".github/actions/checkout-openvmm/action.yml",), everything, None), - ((".github/actions/sccache/action.yml",), everything, None), - ((".github/actions/setup-curl/curl-shim.rs",), everything, None), (("scripts/nvx.py",), everything, None), - (("scripts/nvx_tools/ci.py",), everything, None), + (("scripts/nvx_tools/__init__.py",), everything, None), + (("scripts/nvx_tools/build_constants.py",), everything, None), (("scripts/nvx_tools/common.py",), everything, None), - # Inputs of the kernel and Alpine initramfs that the VMM tests - # boot, and of the other guests. - ((".dockerignore",), without_unit_tests, None), - (("docker/Dockerfile",), without_unit_tests, None), - (("SOURCE-MANIFEST.json",), without_unit_tests, None), - (("scripts/nvx_tools/build.py",), without_unit_tests, None), - (("scripts/nvx_tools/guests.py",), without_unit_tests, None), - (("kernel/config-microvm",), without_unit_tests, None), - (("kernel/patches/0001-example.patch",), without_unit_tests, None), - (("guest/common/init",), without_unit_tests, None), - (("guest/alpine/nvx-container-enter",), without_unit_tests, None), + # Inputs of every suite but the check, which builds nothing: the + # Rust pin, the runners, the actions that the suites' jobs use, and + # the CLI modules that they run. + (("rust-toolchain.toml",), microvm_and_benchmarks, None), + (("scripts/setup/setup-windows-whp.ps1",), microvm_and_benchmarks, None), + (("scripts/setup/azure/build_package.py",), microvm_and_benchmarks, None), ( - (".github/actions/build-guest-artifacts/action.yml",), - without_unit_tests, + (".github/actions/validate-runner/action.yml",), + microvm_and_benchmarks, None, ), - # Of nvx-rs, the guest images, and so the VMM tests, take only the - # inputs of the guest binaries. - (("nvx-rs/src/lib.rs",), without_unit_tests, None), - (("nvx-rs/Cargo.lock",), without_unit_tests, None), - (("nvx-rs/src/guest/exit.rs",), without_unit_tests, None), - (("nvx-rs/src/bin/nvx-time.rs",), without_unit_tests, None), - # The shards of the WHP VMM tests, and the planner that the - # microVM shards share, are inputs of the VMM tests and so of the - # microVM tests, but not of the platform or package jobs (#691). ( - ("scripts/nvx_tools/openvmm_test_shards.py",), - vmm_and_microvm, - without_unit_tests, + (".github/actions/checkout-openvmm/action.yml",), + microvm_and_benchmarks, + None, ), + ((".github/actions/sccache/action.yml",), microvm_and_benchmarks, None), ( - ("scripts/nvx_tools/openvmm_test_shards.json",), - vmm_and_microvm, - without_unit_tests, + (".github/actions/setup-curl/curl-shim.rs",), + microvm_and_benchmarks, + None, ), + (("scripts/nvx_tools/ci.py",), microvm_and_benchmarks, None), + # Inputs of the guests, which the microVM tests and the platform + # jobs boot. + ((".dockerignore",), microvm_and_benchmarks, None), + (("docker/Dockerfile",), microvm_and_benchmarks, None), + (("SOURCE-MANIFEST.json",), microvm_and_benchmarks, None), + (("scripts/nvx_tools/build.py",), microvm_and_benchmarks, None), + (("scripts/nvx_tools/guests.py",), microvm_and_benchmarks, None), + (("kernel/config-microvm",), microvm_and_benchmarks, None), + (("kernel/patches/0001-example.patch",), microvm_and_benchmarks, None), + (("guest/common/init",), microvm_and_benchmarks, None), + (("guest/alpine/nvx-container-enter",), microvm_and_benchmarks, None), ( - ("scripts/nvx_tools/shard_plans.py",), - vmm_and_microvm, - without_unit_tests, + (".github/actions/build-guest-artifacts/action.yml",), + microvm_and_benchmarks, + None, ), (("guest/ubuntu/nvx-bashrc",), microvm_and_benchmarks, None), (("ubuntu/packages.lock.json",), microvm_and_benchmarks, None), @@ -10000,12 +7613,17 @@ def select(*outputs: str) -> ci.CiChanges: ), (("scripts/nvx_tools/release.py",), microvm_and_benchmarks, None), ((".gitattributes",), microvm_and_benchmarks, None), - # The rest of nvx-rs: the microVM tests run the nvx-rs CLI, and - # every package ships it. + # Every nvx-rs change: the microVM tests run the nvx-rs CLI, every + # package ships it, and the guest images carry its guest binaries. + (("nvx-rs/src/lib.rs",), microvm_and_benchmarks, None), + (("nvx-rs/Cargo.lock",), microvm_and_benchmarks, None), + (("nvx-rs/src/guest/exit.rs",), microvm_and_benchmarks, None), + (("nvx-rs/src/bin/nvx-time.rs",), microvm_and_benchmarks, None), (("nvx-rs/src/host/cli/args.rs",), microvm_and_benchmarks, None), (("nvx-rs/src/bin/nvx-rs.rs",), microvm_and_benchmarks, None), (("nvx-rs/tests/cli.rs",), microvm_and_benchmarks, None), - # Inputs of the microVM tests alone. + # Inputs of the microVM tests alone, such as the planner of their + # shards (#691). ( ("tests/runtime-adapter/src/lib.rs",), microvm_only, @@ -10031,6 +7649,11 @@ def select(*outputs: str) -> ci.CiChanges: microvm_only, microvm_and_benchmarks, ), + ( + ("scripts/nvx_tools/shard_plans.py",), + microvm_only, + microvm_and_benchmarks, + ), ( ("scripts/nvx_tools/openvmm_process.py",), microvm_only, @@ -10046,6 +7669,18 @@ def select(*outputs: str) -> ci.CiChanges: microvm_only, microvm_and_benchmarks, ), + # Inputs of the OpenVMM CI check alone: the module that checks, and + # the GitHub client that it shares with the nightly report. + ( + ("scripts/nvx_tools/openvmm_ci.py",), + openvmm_ci_only, + openvmm_ci_and_benchmarks, + ), + ( + ("scripts/nvx_tools/nightly_ci.py",), + openvmm_ci_only, + openvmm_ci_and_benchmarks, + ), # Inputs of the platform jobs, package jobs, and regression gate alone. (("scripts/nvx_tools/performance.py",), benchmarks_only, None), ((".github/workflows/run-platform.yml",), benchmarks_only, None), @@ -10084,7 +7719,6 @@ def select(*outputs: str) -> ci.CiChanges: select(), benchmarks_only, ), - (("scripts/nvx_tools/nightly_ci.py",), select(), benchmarks_only), (("scripts/test_nightly_ci.py",), select(), benchmarks_only), ((".github/workflows/nightly-ci-report.yml",), select(), benchmarks_only), # Only the merge groups' GitHub-hosted job that cancels the runs @@ -10101,9 +7735,17 @@ def select(*outputs: str) -> ci.CiChanges: # A change selects the union of its paths' suites. ( ("scripts/nvx_tools/microvm_tests.py", "kernel/config-microvm"), - without_unit_tests, + microvm_and_benchmarks, None, ), + ( + ( + "scripts/nvx_tools/openvmm_ci.py", + "scripts/nvx_tools/microvm_tests.py", + ), + select(guest, binaries, openvmm_ci, microvm), + everything, + ), ( ( "scripts/nvx_tools/microvm_tests.py", @@ -10163,9 +7805,7 @@ def test_ci_change_classification_builds_artifacts_for_the_jobs_that_use_them(se with self.subTest(event_name=event_name, files=changed_files): self.assertEqual( changes.run_guest_artifacts, - changes.run_openvmm_vmm_tests - or changes.run_microvm_tests - or changes.run_benchmarks, + changes.run_microvm_tests or changes.run_benchmarks, ) self.assertEqual( changes.run_openvmm_binaries, @@ -10175,9 +7815,9 @@ def test_ci_change_classification_builds_artifacts_for_the_jobs_that_use_them(se def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(self): # The microVM jobs run the runtime adapter's suite on every backend, so # a change to the adapter, its suite, or the command that runs it alone - # runs them. Neither OpenVMM suite builds or boots the adapter, and - # neither the benchmarks nor the packages use it, so a pull request - # skips the platform and package jobs. + # runs them. The OpenVMM CI check reads none of them, and neither the + # benchmarks nor the packages use the adapter, so a pull request skips + # the platform and package jobs. workflow = ( BuildConstants.REPO_ROOT / ".github" @@ -10207,12 +7847,10 @@ def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(sel ): with self.subTest(changed_files=changed_files): changes = ci.classify_ci_changes("pull_request", changed_files) - self.assertEqual( - changes, ci.CiChanges(True, True, False, False, True, False) - ) + self.assertEqual(changes, ci.CiChanges(True, True, False, True, False)) self.assertEqual( ci.classify_ci_changes("push", changed_files), - ci.CiChanges(True, True, False, False, True, True), + ci.CiChanges(True, True, False, True, True), ) expected = ci.required_ci_expected_results( "pull_request", same_repository=True, changes=changes @@ -10223,9 +7861,7 @@ def test_ci_change_classification_runs_microvm_tests_for_the_runtime_adapter(sel ): self.assertEqual(expected[job], "success", job) for job in ( - *ci.REQUIRED_CI_OPENVMM_TEST_JOBS, - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, + ci.REQUIRED_CI_OPENVMM_CI_JOB, *ci.REQUIRED_CI_PLATFORM_JOBS, *ci.REQUIRED_CI_PACKAGE_JOBS, "performance-gate", @@ -10373,30 +8009,8 @@ def imported_modules(module: str) -> set[str]: pending.append(name) return modules - command_inputs = { - "scripts/nvx.py", - "scripts/nvx_tools/__init__.py", - ".github/workflows/ci.yml", - *(f"scripts/nvx_tools/{name}.py" for name in imported_modules("ci")), - } - self.assertLessEqual( - {"ci", "common", "build_constants"}, imported_modules("ci") - ) - unit_inputs = command_inputs | { - f".github/actions/{name}/action.yml" - for name in local_actions(_workflow_job(workflow, "openvmm-unit-tests")) - } - vmm_inputs = command_inputs | { - f".github/actions/{name}/action.yml" - for job in ( - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - "artifacts", - ) - for name in local_actions(_workflow_job(workflow, job)) - } - # Every cache key input of the kernel and Alpine initramfs that the - # VMM tests boot, and of every guest that the microVM tests boot. + # Every cache key input of every guest that the microVM tests and the + # platform jobs boot. guest_action = ( github / "actions" / "build-guest-artifacts" / "action.yml" ).read_text(encoding="utf-8") @@ -10427,8 +8041,6 @@ def imported_modules(module: str) -> set[str]: path.relative_to(BuildConstants.REPO_ROOT).as_posix() for path in guest_binaries.guest_binary_source_files() } - vmm_inputs |= guest_inputs["KERNEL"] | guest_inputs["ALPINE"] - vmm_inputs |= guest_binary_inputs # The NVX microVM tests and the platform and package jobs take every # input of the @@ -10622,8 +8234,6 @@ def job_inputs( "scripts/nvx_tools/microvm_shards.py", "scripts/nvx_tools/microvm_shards.json", "scripts/nvx_tools/microvm_tests.py", - "scripts/nvx_tools/openvmm_test_shards.py", - "scripts/nvx_tools/openvmm_test_shards.json", "scripts/nvx_tools/shard_plans.py", ): self.assertNotIn(path, benchmark_inputs) @@ -10632,54 +8242,34 @@ def job_inputs( self.assertNotIn("scripts/nvx_tools/crate_release.py", inputs) self.assertNotIn("scripts/nvx_tools/merge_queue.py", inputs) self.assertNotIn("scripts/nvx_tools/performance_history.py", inputs) - # The VMM-test jobs also take the modules of the nvx.py commands that - # they run, with every file that those read beside them, such as the - # partition of the WHP tests' shards (#691). - vmm_commands = { - command - for job in ( - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - ) - for command in re.findall( - r"nvx\.py\s+([a-z][a-z-]*)", _workflow_job(workflow, job) - ) - } - self.assertLessEqual({"build-openvmm-tests", "test-openvmm"}, vmm_commands) - vmm_inputs = vmm_inputs.union(*map(command_files, vmm_commands)) + self.assertNotIn("scripts/nvx_tools/openvmm_ci.py", inputs) + # The OpenVMM CI check reads the pin from the checkout's index, and + # its command's modules read the public OpenVMM repository's runs. + openvmm_ci_inputs = job_inputs(ci.REQUIRED_CI_OPENVMM_CI_JOB) | {"openvmm"} self.assertLessEqual( { - "scripts/nvx_tools/openvmm_test_shards.py", - "scripts/nvx_tools/openvmm_test_shards.json", - "scripts/nvx_tools/shard_plans.py", + "scripts/nvx_tools/openvmm_ci.py", + "scripts/nvx_tools/nightly_ci.py", + "scripts/nvx_tools/common.py", }, - vmm_inputs, + openvmm_ci_inputs, ) for pattern, inputs in ( - (ci.OPENVMM_UNIT_TEST_INPUT_PATHS, unit_inputs), - (ci.OPENVMM_VMM_TEST_INPUT_PATHS, vmm_inputs), + (ci.OPENVMM_CI_CHECK_INPUT_PATHS, openvmm_ci_inputs), (ci.MICROVM_TEST_INPUT_PATHS, microvm_inputs), (ci.BENCHMARK_INPUT_PATHS, benchmark_inputs), ): for path in sorted(inputs): with self.subTest(pattern=pattern.pattern, path=path): self.assertIsNotNone(pattern.match(path)) - # Every change that runs the VMM tests runs the microVM tests too, which - # read the Ubuntu and Azure Linux images, so distro-artifacts shares the - # selection of the other guest artifact jobs. - for path in sorted(vmm_inputs): - with self.subTest(vmm_input=path): - self.assertIsNotNone(ci.MICROVM_TEST_INPUT_PATHS.match(path)) - def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): + def test_only_guest_binary_inputs_rebuild_the_guest_images(self): # The guest images carry only the Rust guest binaries of nvx-rs, so a - # change to a file that their build reads rebuilds every guest image and - # runs the OpenVMM VMM tests, which boot the Alpine image, and no other - # nvx-rs change does either. The NVX microVM tests run the nvx-rs CLI, - # and every package ships it, so every nvx-rs change runs them and the - # platform and package jobs, and so the jobs that build their artifacts, - # which - # restore unchanged guest images from the cache. + # change to a file that their build reads rebuilds every guest image, + # and no other nvx-rs change does. The NVX microVM tests run the nvx-rs + # CLI, and every package ships it, so every nvx-rs change runs them and + # the platform and package jobs, and so the jobs that build their + # artifacts, which restore unchanged guest images from the cache. root = BuildConstants.REPO_ROOT identity = { path.relative_to(root).as_posix() @@ -10697,10 +8287,12 @@ def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): with self.subTest(path=path): self.assertIsNotNone(ci.MICROVM_TEST_INPUT_PATHS.match(path)) self.assertIsNotNone(ci.BENCHMARK_INPUT_PATHS.match(path)) - if path in identity: - self.assertIsNotNone(ci.OPENVMM_VMM_TEST_INPUT_PATHS.match(path)) - else: - self.assertIsNone(ci.OPENVMM_VMM_TEST_INPUT_PATHS.match(path)) + if path in crate: + self.assertIsNone(ci.OPENVMM_CI_CHECK_INPUT_PATHS.match(path)) + self.assertEqual( + GuestRustBuildConstants.SOURCE_PATTERN.fullmatch(path) is not None, + path in identity, + ) # A new module counts unless it belongs to the host side or the tests, # and a file beside an input doesn't. for path, selected in ( @@ -10716,7 +8308,8 @@ def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): self.assertIsNotNone(ci.MICROVM_TEST_INPUT_PATHS.match(path)) self.assertIsNotNone(ci.BENCHMARK_INPUT_PATHS.match(path)) self.assertEqual( - ci.OPENVMM_VMM_TEST_INPUT_PATHS.match(path) is not None, selected + GuestRustBuildConstants.SOURCE_PATTERN.fullmatch(path) is not None, + selected, ) # The guest image keys hash no nvx-rs file and end with the identity. @@ -10798,7 +8391,7 @@ def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): changes = ci.classify_ci_changes(event_name, paths) self.assertEqual( changes, - ci.CiChanges(True, True, False, guest, True, True), + ci.CiChanges(True, True, False, True, True), ) expected = ci.required_ci_expected_results( event_name, same_repository=True, changes=changes @@ -10812,13 +8405,9 @@ def test_only_guest_binary_inputs_rebuild_guests_and_run_vmm_tests(self): *ci.REQUIRED_CI_PACKAGE_JOBS, ): self.assertEqual(expected[job], "success", job) - for job in ( - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - ): - self.assertEqual( - expected[job], "success" if guest else "skipped", job - ) + self.assertEqual( + expected[ci.REQUIRED_CI_OPENVMM_CI_JOB], "skipped" + ) with self.subTest(changed=paths): contents = {path: (checkout / path).read_bytes() for path in paths} for path, data in contents.items(): @@ -10869,16 +8458,14 @@ def test_ci_change_summary_explains_each_decision(self): [ "Guest artifacts", "OpenVMM binaries", - "OpenVMM unit tests", - "OpenVMM VMM tests", + "OpenVMM CI check", "NVX microVM tests", "Platform benchmarks and packages", ], ) self.assertIn( - "### Guest artifacts: run\nJobs that use them run: the OpenVMM VMM " - "tests, the NVX microVM tests, and the platform benchmarks and " - "packages.\n", + "### Guest artifacts: run\nJobs that use them run: the NVX microVM " + "tests and the platform benchmarks and packages.\n", summary, ) self.assertIn( @@ -10886,19 +8473,19 @@ def test_ci_change_summary_explains_each_decision(self): "tests and the platform benchmarks and packages.\n", summary, ) - self.assertIn("### OpenVMM unit tests: skip\nNo input changed.\n", summary) - self.assertIn( - "### OpenVMM VMM tests: run\nInputs changed: " - "`guest/common/init`, `guest/common/nvx-hostmount`, " - "`guest/common/nvx-snapshot`, and 1 more.\n", - summary, - ) + self.assertIn("### OpenVMM CI check: skip\nNo input changed.\n", summary) self.assertIn( "### NVX microVM tests: run\nInputs changed: " "`scripts/nvx_tools/microvm_tests.py`, `guest/common/init`, " "`guest/common/nvx-hostmount`, and 2 more.\n", summary, ) + pin = ci.describe_ci_changes( + "pull_request", + ["openvmm"], + ci.classify_ci_changes("pull_request", ["openvmm"]), + ) + self.assertIn("### OpenVMM CI check: run\nInputs changed: `openvmm`.\n", pin) push = ci.describe_ci_changes( "push", changed_files, ci.classify_ci_changes("push", changed_files) ) @@ -10940,7 +8527,7 @@ def describe(event_name: str, files: list[str]) -> str: "### Guest artifacts: skip\nNo job that uses them runs.\n", documentation ) self.assertIn( - "### OpenVMM VMM tests: skip\nOnly documentation files changed.\n", + "### OpenVMM CI check: skip\nOnly documentation files changed.\n", documentation, ) self.assertIn( @@ -10983,8 +8570,7 @@ def test_classify_ci_changes_command_prints_every_output(self): [ "run-guest-artifacts=true", "run-openvmm-binaries=true", - "run-openvmm-unit-tests=false", - "run-openvmm-vmm-tests=false", + "run-openvmm-ci-check=false", "run-microvm-tests=true", "run-benchmarks=false", ], @@ -11025,8 +8611,7 @@ def test_classify_ci_changes_command_reads_null_delimited_paths(self): [ "run-guest-artifacts=true", "run-openvmm-binaries=true", - "run-openvmm-unit-tests=false", - "run-openvmm-vmm-tests=true", + "run-openvmm-ci-check=false", "run-microvm-tests=true", "run-benchmarks=true", ], @@ -11117,10 +8702,10 @@ def git(*arguments: str, directory: Path = root) -> str: for line in output.read_text(encoding="utf-8").splitlines() ) # A kernel patch is an input of every suite but the - # OpenVMM unit tests, and documentation of none. + # OpenVMM CI check, and documentation of none. expected = { output: str( - test_input and output != "run-openvmm-unit-tests" + test_input and output != "run-openvmm-ci-check" ).lower() for output in ci.CI_CHANGE_OUTPUTS } @@ -11160,8 +8745,7 @@ def test_ci_change_detector_runs_the_full_matrix_unless_classified(self): classified = { "run-guest-artifacts": "true", "run-openvmm-binaries": "true", - "run-openvmm-unit-tests": "false", - "run-openvmm-vmm-tests": "false", + "run-openvmm-ci-check": "false", "run-microvm-tests": "true", "run-benchmarks": "false", } @@ -11323,26 +8907,10 @@ def test_flowey_downloads_use_retrying_curl(self): self.assertIn('Join-Path $ShimDirectory "curl.exe"', action) self.assertIn("Get-Command rustc.exe", action) self.assertNotIn("curl.cmd", action) - # Every job that runs flowey retries its curl downloads. The WHP - # VMM-test job runs no flowey: test-openvmm runs its content's archive - # itself and downloads the content's image with Python. - for job_name, uses in ( - *((job, 1) for job in ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS), - ("openvmm-vmm-tests-linux", 1), - ("openvmm-vmm-tests-windows", 0), - ("openvmm-unit-tests", 1), - ): - with self.subTest(job_name=job_name): - self.assertEqual( - _workflow_job(workflow, job_name).count( - "uses: ./.github/actions/setup-curl" - ), - uses, - ) - self.assertEqual( - set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS), - {"openvmm-vmm-tests-linux", "openvmm-vmm-tests-windows"}, - ) + # Every job that runs flowey retries its curl downloads: the OpenVMM + # binary builds, through build-openvmm, and the microVM tests. No job + # of ci.yml runs flowey itself, since OpenVMM's own CI runs its tests. + self.assertNotIn("uses: ./.github/actions/setup-curl", workflow) self.assertEqual( microvm_workflow.count("uses: ./.github/actions/setup-curl"), 1, @@ -11456,19 +9024,49 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): / "action.yml" ).read_text(encoding="utf-8") - for configuration in (workflow, build_action): - self.assertIn( - "openvmm/flowey-persist/flowey_lib_common__download_gh_release", - configuration, - ) - self.assertIn( - "openvmm/flowey-persist/flowey_lib_common__cache", - configuration, - ) - self.assertIn( - "openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-", - configuration, - ) + # The binary builds cache the downloads of flowey, which only they run, + # and the Cargo registry. A dev push whose build missed the binary + # cache saves both, under the keys and paths that the builds restore, + # since no other job builds OpenVMM. + self.assertIn( + "openvmm/flowey-persist/flowey_lib_common__download_gh_release", + build_action, + ) + self.assertIn("openvmm/flowey-persist/flowey_lib_common__cache", build_action) + self.assertIn( + "openvmm-inputs-v1-${{ runner.os }}-${{ runner.arch }}-", build_action + ) + self.assertNotIn("flowey-persist", workflow) + for system in ("Linux", "Windows"): + for name, restore_id in ( + ("OpenVMM inputs", f"restore-openvmm-inputs-{system.lower()}"), + ("shared Cargo cache", f"restore-cargo-{system.lower()}"), + ): + with self.subTest(system=system, cache=name): + restore = _composite_action_step( + build_action, f"Restore {name} on {system}" + ) + save = _composite_action_step( + build_action, f"Save {name} on {system}" + ) + self.assertEqual(_yaml_field(restore, "id"), [restore_id]) + for field in ("key", "path"): + self.assertEqual( + _yaml_field(save, field), _yaml_field(restore, field) + ) + self.assertEqual( + _expression_conjuncts(_yaml_scalar(save, "if")), + [ + "success()", + "github.event_name == 'push'", + "github.ref == 'refs/heads/dev'", + f"runner.os {'!=' if system == 'Linux' else '=='} " + "'Windows'", + f"steps.restore-openvmm-binary-{system.lower()}.outputs." + "cache-hit != 'true'", + f"steps.{restore_id}.outputs.cache-hit != 'true'", + ], + ) producers = { "build-openvmm-linux-gnu": ( "kvm", @@ -11690,52 +9288,13 @@ def test_ci_shares_openvmm_inputs_and_binary_artifacts(self): build_workflow.count("uses: ./.github/actions/sccache"), 2, ) - for job_name, cache_namespace in ( - ("build-openvmm-vmm-tests-linux", "vmm-tests"), - ("build-openvmm-vmm-tests-windows", "vmm-tests"), - ("openvmm-unit-tests", "unit-tests"), - ): - with self.subTest(job_name=job_name): - job = _workflow_job(workflow, job_name) - self.assertEqual( - job.count("uses: ./.github/actions/sccache"), - 2, - ) - self.assertIn( - f"key: openvmm-inputs-v1-${{{{ runner.os }}}}-" - f"${{{{ runner.arch }}}}-{cache_namespace}-" - "${{ steps.openvmm.outputs.sha }}", - job, - ) - self.assertIn( - f"key: openvmm-cargo-v1-${{{{ runner.os }}}}-" - f"${{{{ runner.arch }}}}-{cache_namespace}-" - "${{ hashFiles('openvmm/Cargo.lock') }}", - job, - ) - self.assertIn( - "restore-keys: |\n" - " openvmm-inputs-v1-${{ runner.os }}-" - "${{ runner.arch }}-", - job, - ) - self.assertIn( - "restore-keys: |\n" - " openvmm-cargo-v1-${{ runner.os }}-" - "${{ runner.arch }}-", - job, - ) - # The VMM-test jobs run the test content of their OS's build and - # compile nothing, so they neither start sccache nor restore OpenVMM's - # inputs. - for job_name in ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS: - with self.subTest(consumer=job_name): - consumer_job = _workflow_job(workflow, job_name) - self.assertNotIn("uses: ./.github/actions/sccache", consumer_job) - self.assertNotIn( - "uses: ./.github/actions/checkout-openvmm", consumer_job - ) - self.assertNotIn("actions/cache/", consumer_job) + # Only the binary builds compile OpenVMM, through build-openvmm: + # OpenVMM's own CI runs its tests (doc/ci.md#openvmm-ci), so no job of + # ci.yml starts sccache or caches OpenVMM's inputs itself. + self.assertNotIn("uses: ./.github/actions/sccache", workflow) + self.assertNotIn("actions/cache/", workflow) + self.assertNotIn("openvmm-inputs-v1-", workflow) + self.assertNotIn("openvmm-cargo-v1-", workflow) self.assertNotIn("uses: actions/cache@v5", workflow) self.assertNotIn("uses: actions/cache@v5", build_workflow) self.assertNotIn("uses: actions/cache@v5", build_action) @@ -12812,633 +10371,67 @@ def test_benchmarks_rely_on_the_microvm_correctness_jobs(self): job, ) - def test_ci_runs_openvmm_vmm_tests_per_backend_and_unit_tests_per_os(self): - workflow = ( - BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" - ).read_text(encoding="utf-8") - build_job = _workflow_job(workflow, "build-openvmm-vmm-tests-linux") - windows_build_job = _workflow_job(workflow, "build-openvmm-vmm-tests-windows") - linux_job = _workflow_job(workflow, "openvmm-vmm-tests-linux") - windows_job = _workflow_job(workflow, "openvmm-vmm-tests-windows") - unit_tests_job = _workflow_job(workflow, "openvmm-unit-tests") - - def legs(job: str) -> list[tuple[str, str, str]]: - return re.findall( - r"^ - name: (.+)\n" - r" backend: (\S+)\n" - r" runner: '(.+)'$", - job, - re.MULTILINE, - ) - - self.assertEqual( - legs(linux_job), - [ - ( - "Linux / KVM", - "kvm", - '["self-hosted", "linux", "kvm", "virtual-machine", "general"]', - ), - ( - "Linux / MSHV", - "mshv", - '["self-hosted", "linux", "mshv", "virtual-machine", "general"]', - ), - ], - ) - self.assertIn(" name: OpenVMM vmm-tests / ${{ matrix.name }}\n", linux_job) - # One KVM runner builds the tests of both Linux backends. - self.assertIn(" name: OpenVMM vmm-tests build / Linux\n", build_job) - self.assertIn( - " runs-on: [self-hosted, linux, kvm, virtual-machine, general]\n", - build_job, - ) - self.assertIn(" backend: kvm\n", build_job) - # A general WHP runner builds the WHP tests, and others run their - # shards. - self.assertIn( - " name: OpenVMM vmm-tests build / Windows\n", windows_build_job - ) - self.assertIn( - " name: OpenVMM vmm-tests / Windows / WHP / " - "Shard ${{ matrix.shard }} of ${{ strategy.job-total }}\n", - windows_job, - ) - for job in (windows_build_job, windows_job): - self.assertIn( - " runs-on: [self-hosted, windows, whp, virtual-machine, general]\n", - job, - ) - self.assertIn(" backend: whp\n", job) - # The unit tests that need /dev/kvm or /dev/mshv are ignored, so they - # run once per OS, and Linux needs no hypervisor runner (#697). - self.assertEqual( - yaml.safe_load(workflow)["jobs"]["openvmm-unit-tests"]["strategy"][ - "matrix" - ]["include"], - [ - {"name": "Linux", "runner": '["ubuntu-latest"]'}, - { - "name": "Windows", - "backend": "whp", - "runner": ( - '["self-hosted", "windows", "whp", "virtual-machine", ' - '"general"]' - ), - }, - ], - ) - self.assertIn( - """ - name: Run OpenVMM unit tests on Linux - if: runner.os != 'Windows' - shell: bash - run: python3 scripts/nvx.py test-openvmm-unit""", - unit_tests_job, - ) - self.assertIn( - """ - name: Run OpenVMM unit tests on Windows - if: runner.os == 'Windows' - shell: powershell - run: python scripts\\nvx.py test-openvmm-unit""", - unit_tests_job, - ) - self.assertEqual( - build_job.count( - " python3 scripts/nvx.py build-openvmm-tests\n" - " --backend kvm --backend mshv\n" - ' --output "${RUNNER_TEMP}/openvmm-vmm-tests-linux"\n' - ), - 1, - ) - self.assertEqual( - linux_job.count( - " python3 scripts/nvx.py test-openvmm\n" - ' --backend "${{ matrix.backend }}"\n' - ' --content "${RUNNER_TEMP}/openvmm-vmm-tests"\n' - ), - 1, - ) - self.assertEqual( - windows_build_job.count( - " python scripts\\nvx.py build-openvmm-tests\n" - " --backend whp\n" - ' --output "$env:RUNNER_TEMP\\openvmm-vmm-tests-windows"\n' - ), - 1, - ) - self.assertEqual( - windows_job.count( - " python scripts\\nvx.py test-openvmm\n" - " --backend whp\n" - ' --shard "${{ matrix.shard }}/${{ strategy.job-total }}"\n' - ' --content "$env:RUNNER_TEMP\\openvmm-vmm-tests"\n' - ), - 1, - ) - for job in (build_job, windows_build_job, linux_job, windows_job): - self.assertNotIn("test-openvmm-unit", job) - for job in (build_job, windows_build_job): - self.assertNotIn("nvx.py test-openvmm", job) - self.assertEqual(unit_tests_job.count("scripts/nvx.py test-openvmm-unit"), 1) - self.assertEqual( - unit_tests_job.count("scripts\\nvx.py test-openvmm-unit"), - 1, - ) - self.assertNotIn("test-openvmm --backend", unit_tests_job) - - def test_ci_runs_the_linux_openvmm_unit_tests_on_a_github_hosted_runner(self): - # The Linux leg needs no hypervisor, so it runs on a GitHub-hosted - # runner, which has neither the self-hosted runners' validated - # toolchain nor their persistent sccache (#697). It installs the pinned - # Rust release and cargo-nextest itself, and keeps Cargo's home where - # the shared Cargo cache restores the registry. - workflow = ( + def test_ci_checks_openvmm_ci_instead_of_running_openvmm_tests(self): + workflow_text = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" ).read_text(encoding="utf-8") - job_name = "openvmm-unit-tests" - job = _workflow_job(workflow, job_name) - self.assertIn(" runs-on: ${{ fromJSON(matrix.runner) }}\n", job) - self.assertIn(" timeout-minutes: 90\n", job) - - for step_name, condition in ( - ("Validate runner", "runner.environment == 'self-hosted'"), - ("Prepare sccache", "runner.environment == 'self-hosted'"), - ("Report sccache", "always() && runner.environment == 'self-hosted'"), - ): - with self.subTest(step=step_name): - step = _workflow_step(workflow, job_name, step_name) - self.assertEqual(_yaml_scalar(step, "if"), condition) - - install_name = "Install the pinned Rust release and cargo-nextest" - install = _workflow_step(workflow, job_name, install_name) - self.assertEqual( - _yaml_scalar(install, "if"), "runner.environment == 'github-hosted'" - ) - for fragment in ( - "rust-toolchain.toml", - 'rustup toolchain install "${toolchain}" --profile minimal' - " --no-self-update", - "s/^cargo_nextest\\.version=//p' scripts/setup/tool-versions.conf", - '"https://get.nexte.st/${nextest}/linux"', - 'export CARGO_HOME="${RUNNER_TEMP}/cargo-home"', - 'echo "CARGO_HOME=${CARGO_HOME}"', - 'echo "CARGO_INCREMENTAL=0"', - 'echo "${CARGO_HOME}/bin" >> "${GITHUB_PATH}"', - 'if [[ "${rust_version}" != "rustc ${toolchain} "* ]]; then', - 'if [[ "${nextest_version}" != "cargo-nextest ${nextest} "* ]]; then', + workflow = yaml.safe_load(workflow_text) + # OpenVMM's own checkin gates test the pinned revision + # (doc/ci.md#openvmm-ci), so no job builds or runs OpenVMM's tests. + for command in ( + "test-openvmm", + "test-openvmm-unit", + "build-openvmm-tests", + "check-openvmm-tests", + "reuse-openvmm-tests", + "openvmm-test-cache-key", ): - with self.subTest(fragment=fragment): - self.assertIn(fragment, install) + with self.subTest(command=command): + self.assertNotRegex(workflow_text, rf"nvx\.py {command}(?![\w-])") + job = workflow["jobs"]["openvmm-ci"] + text = _workflow_job(workflow_text, "openvmm-ci") + self.assertEqual(job["name"], "OpenVMM CI of the pinned revision") + self.assertEqual(job["needs"], ["openvmm-changes"]) + self.assertEqual(job["runs-on"], "ubuntu-latest") self.assertIn( - "${{ format('{0}/cargo-home/registry', runner.temp) }}", - _workflow_step(workflow, job_name, "Restore shared Cargo cache on Linux"), - ) - steps = [ - _yaml_scalar(step, "name") for step in _workflow_steps(workflow, job_name) - ] - self.assertLess( - steps.index(install_name), - steps.index("Run OpenVMM unit tests on Linux"), - ) - # The step takes its pins from inputs of the suite, so a change to - # either runs it. - for path in ("rust-toolchain.toml", "scripts/setup/tool-versions.conf"): - with self.subTest(path=path): - self.assertIsNotNone(ci.OPENVMM_UNIT_TEST_INPUT_PATHS.match(path)) - - def test_ci_hands_each_vmm_test_build_to_the_jobs_of_its_os(self): - # Each OS's build boots no guest, so it starts beside the guest - # artifacts; each backend's job runs its filter from the build of its - # OS, a same-run artifact, once both are ready (#690). - workflow = ( - BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" - ).read_text(encoding="utf-8") - handoffs = { - "Linux": ("build-openvmm-vmm-tests-linux", "openvmm-vmm-tests-linux"), - "Windows": ( - "build-openvmm-vmm-tests-windows", - "openvmm-vmm-tests-windows", - ), - } - for system, (build_name, test_name) in handoffs.items(): - build_job = _workflow_job(workflow, build_name) - test_job = _workflow_job(workflow, test_name) - build_steps = [ - _yaml_scalar(step, "name") - for step in _workflow_steps(workflow, build_name) - ] - test_steps = [ - _yaml_scalar(step, "name") - for step in _workflow_steps(workflow, test_name) - ] - # Each build uploads its content under its own name. - artifact = build_name.removeprefix("build-") - with self.subTest(system=system): - self.assertIn(" needs: [openvmm-changes]\n", build_job) - self.assertNotIn("artifacts", build_job.split(" steps:", 1)[0]) - # A change that selects the VMM tests also builds the guest - # artifacts, so the build needs no other output to run - # exactly with its tests. - self.assertIn( - " (\n" - " needs.openvmm-changes.result != 'success' ||\n" - " needs.openvmm-changes.outputs.run-openvmm-vmm-tests" - " == 'true'\n" - " )\n", - build_job, - ) - self.assertIn( - f" needs: [artifacts, {build_name}, openvmm-changes]\n", - test_job, - ) - self.assertIn( - " needs.artifacts.result == 'success' &&\n", test_job - ) - self.assertIn( - f" needs.{build_name}.result == 'success' &&\n", test_job - ) - # The test jobs of each OS wait for neither job of the other. - for other, (other_build, other_test) in handoffs.items(): - if other != system: - self.assertNotIn(other_build, test_job) - self.assertNotIn(other_test, test_job) - - upload = _workflow_step(workflow, build_name, "Upload OpenVMM tests") - self.assertIn("uses: actions/upload-artifact@v7", upload) - self.assertIn(f" name: {artifact}\n", upload) - self.assertIn( - f" path: ${{{{ runner.temp }}}}/{artifact}\n", - upload, - ) - # The content's nextest configuration is under .config. - self.assertIn(" include-hidden-files: true\n", upload) - self.assertIn(" if-no-files-found: error\n", upload) - # A rerun of the build replaces the earlier attempt's content. - self.assertIn(" overwrite: true\n", upload) - self.assertIn(" retention-days: 1\n", upload) - self.assertLess( - build_steps.index(f"Build OpenVMM tests on {system}"), - build_steps.index("Upload OpenVMM tests"), - ) - - download = _workflow_step(workflow, test_name, "Download OpenVMM tests") - self.assertIn("uses: actions/download-artifact@v8", download) - self.assertIn(f" name: {artifact}\n", download) - self.assertIn( - " path: ${{ runner.temp }}/openvmm-vmm-tests\n", download - ) - for step in ( - "Download guest kernel", - "Download Alpine initramfs", - "Download OpenVMM tests", - ): - self.assertLess( - test_steps.index(step), - test_steps.index(f"Run OpenVMM tests on {system}"), - ) - - # Every VMM-test job must succeed, or all must be skipped because the - # change selects no VMM tests, before a release or persistence. - jobs = ( - "build-openvmm-vmm-tests-linux", - "build-openvmm-vmm-tests-windows", - "openvmm-vmm-tests-linux", - "openvmm-vmm-tests-windows", - ) - self.assertEqual( - set(jobs), - { - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - }, - ) - gate = ( - " (\n" - " (\n" - + "".join( - f" needs.{job}.result == 'success'" - + (" &&\n" if job != jobs[-1] else "\n") - for job in jobs - ) - + " ) ||\n" - " (\n" - " needs.openvmm-changes.result == 'success' &&\n" - " needs.openvmm-changes.outputs.run-openvmm-vmm-tests != 'true' &&\n" - + "".join( - f" needs.{job}.result == 'skipped'" - + (" &&\n" if job != jobs[-1] else "\n") - for job in jobs - ) - + " )\n" - " ) &&\n" - ) - for consumer in ("release", "performance-record"): - job = _workflow_job(workflow, consumer) - with self.subTest(consumer=consumer): - self.assertIn(" - openvmm-changes\n", job) - for name in jobs: - self.assertIn(f" - {name}\n", job) - self.assertEqual(job.count(gate), 1) - self.assertNotIn("needs.openvmm-vmm-tests.result", job) - - def test_ci_reuses_the_cached_vmm_test_build_of_each_os(self): - # #692: each VMM-test build restores the newest cached content of its - # key and builds the tests only when the reuse refuses what the - # restore left. Only a dev push saves, after a build that passes the - # check, under the key, its run, and its attempt; full runs neither - # restore nor save; and the test jobs take only the same-run artifact, - # which test_ci_shares_openvmm_inputs_and_binary_artifacts checks. - jobs = yaml.safe_load( - (BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml").read_text( - encoding="utf-8" - ) - )["jobs"] - producers = { - "Linux": ( - "build-openvmm-vmm-tests-linux", - "python3 scripts/nvx.py", - "--backend kvm --backend mshv", - '"${RUNNER_TEMP}/openvmm-vmm-tests-linux"', - "Prepare Linux cache tools", - ), - "Windows": ( - "build-openvmm-vmm-tests-windows", - "python scripts\\nvx.py", - "--backend whp", - '"$env:RUNNER_TEMP\\openvmm-vmm-tests-windows"', - "Prepare Windows cache tools", - ), + "needs.openvmm-changes.outputs.run-openvmm-ci-check == 'true'", text + ) + # It reads only the public OpenVMM repository's runs, so it needs + # neither an OpenVMM checkout nor a secret, and a pull request from a + # fork runs it too. + self.assertNotIn("secrets.", text) + self.assertNotIn("checkout-openvmm", text) + self.assertNotIn("head.repo.full_name", text) + checkout, check = job["steps"] + self.assertEqual(checkout["uses"], "actions/checkout@v5") + self.assertEqual(checkout["with"], {"persist-credentials": False}) + self.assertEqual(check["env"]["GH_TOKEN"], "${{ github.token }}") + self.assertEqual( + check["run"], + 'python3 scripts/nvx.py check-openvmm-ci --wait-minutes "${WAIT_MINUTES}" ' + '--summary "${GITHUB_STEP_SUMMARY}"', + ) + # A pull request, whose pin promotion passes once OpenVMM's CI does, + # and a manual run wait for the gates; the other events check pins + # that passed before they merged, and the dev pushes and merge groups + # that queue behind them must not wait for theirs. + waits = { + event: _interpolate_expressions( + check["env"]["WAIT_MINUTES"], {"github": {"event_name": event}}, {} + ) + for event in ci.CI_EVENTS } - key = "${{ steps.openvmm-tests-cache.outputs.key }}" - dev_push = ( - "success() && github.event_name == 'push' && github.ref == 'refs/heads/dev'" - ) - - def words(text: object) -> str: - return " ".join(str(text).split()) - - for system, (job_name, nvx_py, backends, content, tools) in producers.items(): - order = [step["name"] for step in jobs[job_name]["steps"]] - steps = {step["name"]: step for step in jobs[job_name]["steps"]} - path = f"${{{{ runner.temp }}}}/openvmm-vmm-tests-{system.lower()}" - reuse_id = f"reuse-openvmm-tests-{system.lower()}" - built = f"steps.{reuse_id}.outputs.reused != 'true'" - with self.subTest(job=job_name): - # The key covers exactly the backends of the build. - key_step = steps["Prepare OpenVMM test cache key"] - self.assertEqual(key_step["id"], "openvmm-tests-cache") - self.assertEqual( - words(key_step["run"]), - f"{nvx_py} openvmm-test-cache-key {backends}", - ) - self.assertIn( - f"build-openvmm-tests {backends} --output {content}", - words(steps[f"Build OpenVMM tests on {system}"]["run"]), - ) - - # Pushes, pull requests, and merge groups restore the newest - # entry of the key into the directory that the job uploads. - restore = steps[f"Restore OpenVMM tests on {system}"] - self.assertEqual(restore["uses"], "actions/cache/restore@v5") - self.assertEqual( - restore["if"], - "github.event_name == 'pull_request' || " - "github.event_name == 'merge_group' || " - "github.event_name == 'push'", - ) - self.assertEqual( - restore["with"], - {"path": path, "key": key, "restore-keys": f"{key}-\n"}, - ) - # A failed restore falls back to the build, as a miss does. - self.assertIs(restore["continue-on-error"], True) - self.assertEqual(steps["Upload OpenVMM tests"]["with"]["path"], path) - self.assertNotIn("if", steps["Upload OpenVMM tests"]) - - # Every run decides on the reuse, which removes whatever it - # refuses, and only a refusal runs the steps of the build. - reuse = steps[f"Reuse restored OpenVMM tests on {system}"] - self.assertEqual(reuse["id"], reuse_id) - self.assertNotIn("if", reuse) - self.assertNotIn("continue-on-error", reuse) - self.assertEqual( - words(reuse["run"]), - f"{nvx_py} reuse-openvmm-tests {backends} --content {content}", - ) - for name in ( - "Checkout OpenVMM", - "Set up retrying curl", - f"Restore OpenVMM inputs on {system}", - f"Restore shared Cargo cache on {system}", - "Prepare sccache", - f"Build OpenVMM tests on {system}", - f"Check built OpenVMM tests on {system}", - ): - self.assertEqual(steps[name]["if"], built, name) - self.assertEqual(steps["Report sccache"]["if"], f"always() && {built}") - - # A build that a reuse would refuse fails the job. - check = steps[f"Check built OpenVMM tests on {system}"] - self.assertNotIn("continue-on-error", check) - self.assertEqual( - words(check["run"]), - f"{nvx_py} check-openvmm-tests {backends} --content {content}", - ) - - # Only a dev push that built the tests saves them, as an entry - # of its own. - save = steps[f"Save OpenVMM tests on {system}"] - self.assertEqual(save["uses"], "actions/cache/save@v5") - self.assertEqual(words(save["if"]), f"{dev_push} && {built}") - self.assertEqual( - save["with"], - { - "path": path, - "key": f"{key}-${{{{ github.run_id }}}}-" - "${{ github.run_attempt }}", - }, - ) - for name in ( - f"Save OpenVMM inputs on {system}", - f"Save shared Cargo cache on {system}", - ): - self.assertTrue( - words(steps[name]["if"]).startswith(f"{dev_push} && {built}"), - name, - ) - # The Linux caches compress with gzip, as the job's others do. - if system == "Linux": - for step in (restore, save): - self.assertEqual( - step["env"], - {"PATH": "${{ steps.cache-tools-linux.outputs.path }}"}, - ) - - for earlier, later in itertools.pairwise( - ( - "Validate runner", - "Prepare OpenVMM test cache key", - tools, - f"Restore OpenVMM tests on {system}", - f"Reuse restored OpenVMM tests on {system}", - "Checkout OpenVMM", - f"Build OpenVMM tests on {system}", - f"Check built OpenVMM tests on {system}", - "Upload OpenVMM tests", - f"Save OpenVMM tests on {system}", - ) - ): - self.assertLess(order.index(earlier), order.index(later)) - - def test_ci_runs_the_whp_vmm_tests_in_their_shards(self): - # #691: the WHP tests run in the shards that openvmm_test_shards.json - # plans, one leg of the WHP job's matrix each, from the content of - # the one Windows build, so that every WHP test runs exactly once. - # The job's result, which the required status check, the release, - # and the persistence read, fails when a shard fails. The Linux legs - # are not sharded. - workflow = yaml.safe_load( - (BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml").read_text( - encoding="utf-8" - ) - ) - jobs = workflow["jobs"] - job = jobs["openvmm-vmm-tests-windows"] - partition = openvmm_test_shards.load_partition() - count = len(partition.backends["whp"].shards) - # A failed shard does not cancel the others, which keep their - # diagnostics and durations; GitHub expands the matrix after it - # evaluates the condition, so the shards run together or not at all. - self.assertEqual( - job["strategy"], - {"fail-fast": False, "matrix": {"shard": list(range(1, count + 1))}}, - ) - self.assertNotIn("matrix.", job["if"]) - self.assertNotIn("strategy.", job["if"]) - self.assertEqual( - job["name"], - "OpenVMM vmm-tests / Windows / WHP / " - "Shard ${{ matrix.shard }} of ${{ strategy.job-total }}", - ) - steps = {step["name"]: step for step in job["steps"]} - run = steps["Run OpenVMM tests on Windows"]["run"] - ran: list[str] = [] - diagnostics: set[str] = set() - for index in range(1, count + 1): - resolved = { - "${{ matrix.shard }}": str(index), - "${{ strategy.job-total }}": str(count), - "${{ runner.os }}": "Windows", - "${{ github.run_id }}": "39948600574", - "${{ github.run_attempt }}": "1", - } - - def resolve(text: str, resolved: dict[str, str] = resolved) -> str: - for expression, value in resolved.items(): - text = text.replace(expression, value) - self.assertNotIn("${{", text) - return text - - args = nvx.parse_args( - shlex.split(resolve(run).split("nvx.py ", 1)[1].replace("\\", "/")) - ) - self.assertEqual( - (args.command, args.backend, args.shard), - ("test-openvmm", "whp", (index, count)), - ) - self.assertEqual(args.content, Path("$env:RUNNER_TEMP/openvmm-vmm-tests")) - ran.extend(openvmm_test_shards.select_shard("whp", args.shard).tests) - diagnostics.add( - resolve(steps["Upload OpenVMM test diagnostics"]["with"]["name"]) - ) - self.assertEqual(sorted(ran), sorted(ci.OPENVMM_TEST_SELECTIONS["whp"])) - # Each shard keeps its failure diagnostics under a name of its own. self.assertEqual( - diagnostics, + waits, { - f"openvmm-vmm-tests-Windows-whp-shard-{index}-39948600574-1" - for index in range(1, count + 1) + "pull_request": "300", + "merge_group": "0", + "push": "0", + "schedule": "0", + "workflow_dispatch": "300", }, ) - # Every shard downloads the same content, which one job builds. - self.assertEqual( - steps["Download OpenVMM tests"]["with"]["name"], "openvmm-vmm-tests-windows" - ) - linux = jobs["openvmm-vmm-tests-linux"] - self.assertEqual( - [leg["backend"] for leg in linux["strategy"]["matrix"]["include"]], - ["kvm", "mshv"], - ) - self.assertNotIn("--shard", json.dumps(linux["steps"])) - self.assertNotIn("Shard", linux["name"]) - - def test_ci_keeps_the_test_durations_of_every_vmm_test_job(self): - # Each VMM-test job, every WHP shard and Linux leg, keeps the test - # duration report of its run whether it passes, fails, or is - # cancelled, under a name of its own in each attempt (#691). - jobs = yaml.safe_load( - (BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml").read_text( - encoding="utf-8" - ) - )["jobs"] - self.assertEqual( - set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS), - {"openvmm-vmm-tests-linux", "openvmm-vmm-tests-windows"}, - ) - names: set[str] = set() - for job_name, system, legs in ( - ( - "openvmm-vmm-tests-linux", - "Linux", - [{"matrix.backend": backend} for backend in ("kvm", "mshv")], - ), - ( - "openvmm-vmm-tests-windows", - "Windows", - [{"matrix.shard": str(index)} for index in (1, 2)], - ), - ): - steps = [step["name"] for step in jobs[job_name]["steps"]] - upload = next( - step - for step in jobs[job_name]["steps"] - if step["name"] == "Upload OpenVMM test durations" - ) - with self.subTest(job=job_name): - self.assertEqual(upload["if"], "always()") - self.assertEqual(upload["uses"], "actions/upload-artifact@v7") - self.assertEqual( - upload["with"]["path"], - "${{ runner.temp }}/openvmm-vmm-tests/test_results/" - + ci.OPENVMM_TEST_DURATIONS_NAME, - ) - self.assertEqual(upload["with"]["if-no-files-found"], "warn") - self.assertEqual(upload["with"]["retention-days"], 30) - self.assertLess( - steps.index(f"Run OpenVMM tests on {system}"), - steps.index("Upload OpenVMM test durations"), - ) - for leg in legs: - for attempt in (1, 2): - name = upload["with"]["name"] - for expression, value in { - **leg, - "runner.os": system, - "github.run_id": "39948600574", - "github.run_attempt": str(attempt), - }.items(): - name = name.replace(f"${{{{ {expression} }}}}", value) - with self.subTest(job=job_name, leg=leg, attempt=attempt): - self.assertNotIn("${{", name) - self.assertNotIn(name, names) - names.add(name) - self.assertLessEqual( - { - "openvmm-vmm-test-durations-Linux-kvm-39948600574-1", - "openvmm-vmm-test-durations-Windows-whp-shard-2-39948600574-2", - }, - names, - ) + self.assertGreater(job["timeout-minutes"], 300) def test_ci_measures_each_performance_series_on_its_benchmark_runner(self): # Each backend's pool mixes VMs whose results differ by more than the @@ -13614,12 +10607,12 @@ def test_ci_validates_the_cli_once_per_os_after_checking_out_openvmm(self): *ci.REQUIRED_CI_MICROVM_DEBUG_JOBS, *ci.REQUIRED_CI_MICROVM_DEBUG_PUSH_JOBS, *ci.REQUIRED_CI_PLATFORM_JOBS, - *ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS, - *ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS, - "openvmm-unit-tests", ): with self.subTest(job=job_name): self.assertNotIn("nvx-cli", _workflow_job(workflow, job_name)) + self.assertNotIn( + "nvx-cli", _workflow_job(workflow, ci.REQUIRED_CI_OPENVMM_CI_JOB) + ) for consumer in ("release", "performance-record", "required-status-check"): job = _workflow_job(workflow, consumer) for cli in ci.REQUIRED_CI_CLI_JOBS: @@ -14065,42 +11058,6 @@ def test_ci_packages_each_platform_beside_its_benchmarks(self): if consumer != "required-status-check": self.assertIn(f"needs.{package_job}.result == 'success'", job) - def test_ci_preserves_failed_openvmm_test_diagnostics(self): - workflow = ( - BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" - ).read_text(encoding="utf-8") - step_name = " - name: Upload OpenVMM test diagnostics" - # Each backend's artifact names its OS, backend, run, and attempt, and - # each WHP shard's its shard too (#691), and holds the test_results - # directory that Petri writes in the content directory that its job - # runs. - for job_name, backend, system in ( - ("openvmm-vmm-tests-linux", "${{ matrix.backend }}", "Linux"), - ("openvmm-vmm-tests-windows", "whp-shard-${{ matrix.shard }}", "Windows"), - ): - with self.subTest(job=job_name): - job = _workflow_job(workflow, job_name) - self.assertIn(step_name, job) - upload = job.split(step_name, 1)[1].split("\n - name:", 1)[0] - - self.assertIn("\n if: failure()", upload) - self.assertIn("uses: actions/upload-artifact@v7", upload) - self.assertIn( - f"name: openvmm-vmm-tests-${{{{ runner.os }}}}-{backend}-" - "${{ github.run_id }}-${{ github.run_attempt }}\n", - upload, - ) - self.assertIn( - "path: ${{ runner.temp }}/openvmm-vmm-tests/test_results\n", - upload, - ) - self.assertIn("if-no-files-found: warn", upload) - self.assertIn("retention-days: 7", upload) - self.assertLess( - job.index(f" - name: Run OpenVMM tests on {system}"), - job.index(step_name), - ) - def test_runner_setup_pins_and_validates_sccache(self): workflow = ( BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" @@ -14976,14 +11933,14 @@ def test_ci_takes_rust_toolchain_from_repository_pin(self): 4, ) - # A new release reruns every suite that it builds. + # A new release reruns every suite that it builds; OpenVMM's own CI + # builds its tests with OpenVMM's pin. for pattern in ( ci.MICROVM_TEST_INPUT_PATHS, ci.BENCHMARK_INPUT_PATHS, - ci.OPENVMM_UNIT_TEST_INPUT_PATHS, - ci.OPENVMM_VMM_TEST_INPUT_PATHS, ): self.assertIsNotNone(pattern.match("rust-toolchain.toml")) + self.assertIsNone(ci.OPENVMM_CI_CHECK_INPUT_PATHS.match("rust-toolchain.toml")) def test_release_actions_use_deterministic_immutable_tooling(self): package_action = ( @@ -20668,48 +17625,6 @@ def stage_binaries(output: Path) -> guest_binaries.GuestBinaryBuildResult: ) self.assertNotIn("/usr/sbin/nvx-hostmount", init_script) - def test_openvmm_ci_downloads_guest_artifacts(self): - workflow = ( - BuildConstants.REPO_ROOT / ".github" / "workflows" / "ci.yml" - ).read_text(encoding="utf-8") - # Only the jobs that run the VMM tests wait for the guest artifacts, - # which the microVM TTRPC test boots; the Linux build does not. - for job_name in ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS: - with self.subTest(job=job_name): - openvmm_tests = _workflow_job(workflow, job_name) - self.assertRegex( - openvmm_tests, - r"\n needs: \[artifacts, [a-z, -]*openvmm-changes\]", - ) - self.assertIn("needs.artifacts.result == 'success'", openvmm_tests) - # The VMM tests boot only the kernel and the Alpine initramfs. - self.assertEqual( - re.findall( - r"^ - name: (Download .+)\n" - r" uses: actions/download-artifact@v8\n" - r" with:\n" - r" name: (\S+)\n" - r" path: build$", - openvmm_tests, - re.MULTILINE, - ), - [ - ("Download guest kernel", "guest-kernel"), - ("Download Alpine initramfs", "guest-alpine"), - ], - ) - self.assertNotIn("guest-artifacts", openvmm_tests) - self.assertNotIn("distro-artifacts", openvmm_tests) - for job_name in ci.REQUIRED_CI_OPENVMM_TEST_BUILD_JOBS: - with self.subTest(job=job_name): - self.assertNotIn( - "actions/download-artifact", _workflow_job(workflow, job_name) - ) - self.assertEqual( - set(ci.REQUIRED_CI_OPENVMM_ARTIFACT_TEST_JOBS), - {"openvmm-vmm-tests-linux", "openvmm-vmm-tests-windows"}, - ) - def test_apk_add_uses_host_ca_bundle_without_overriding_configuration(self): root = Path("root") with ( From 6c8815c23c3723c9aba45bce4151a16ae3880b90 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Fri, 30 Oct 2026 21:06:36 -0700 Subject: [PATCH 3/3] doc: check OpenVMM's CI of the pin when promoting it NVX CI no longer runs OpenVMM's tests: its openvmm-ci job requires OpenVMM's checkin gates to have passed the x86-64 Linux and Windows unit tests and the KVM, MSHV, and Windows VMM tests for the pinned revision, in an OpenVMM PR or OpenVMM CI run whose head is that revision in microsoft/openvmm. An OpenVMM pull request gets those runs only once it is ready for review and its branch is in microsoft/openvmm. The promotion skill now says so, and records that evidence with check-openvmm-ci before relying on NVX CI: a pending job means that OpenVMM's CI still runs, and a failed one needs a rerun or a fix in OpenVMM. It also says that the pull request's openvmm-ci job waits up to 300 minutes for OpenVMM's CI of the pin, and that after a failed OpenVMM job, that job and then openvmm-ci need a rerun. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/skills/nvx-openvmm-promote/SKILL.md | 23 ++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/.github/skills/nvx-openvmm-promote/SKILL.md b/.github/skills/nvx-openvmm-promote/SKILL.md index 9c7d6b667..f68993841 100644 --- a/.github/skills/nvx-openvmm-promote/SKILL.md +++ b/.github/skills/nvx-openvmm-promote/SKILL.md @@ -291,6 +291,24 @@ unit-test, and formatting gates required by the nested instructions before relyi NVX CI. For a pin-only promotion of an already reviewed source, record existing exact- head OpenVMM evidence and do not claim unexecuted local checks. +NVX CI runs none of OpenVMM's tests. Its `openvmm-ci` job requires OpenVMM's checkin +gates to have passed the x86-64 Linux and Windows unit tests and the KVM, MSHV, and +Windows VMM tests for the pinned SHA, in an OpenVMM PR or OpenVMM CI run whose head is +that SHA in `microsoft/openvmm`; see [OpenVMM CI](../../../doc/ci.md#openvmm-ci). The +OpenVMM pull request gets those runs only once it is ready for review, from a branch +of `microsoft/openvmm`. Record that evidence with the platform launcher: + +```bash +python3 scripts/nvx.py check-openvmm-ci --revision +``` + +```powershell +python scripts\nvx.py check-openvmm-ci --revision +``` + +A pending job means that OpenVMM's CI still runs; a failed one needs a rerun or a fix +in OpenVMM before the promotion can pass. + Choose additional NVX runtime checks from the changed behavior and its dependencies. Use `nvx-run` for a smoke test, `nvx-debug` for a failure, and `nvx-benchmark` only when performance evidence is required. Report unavailable KVM, MSHV, or WHP coverage @@ -319,7 +337,10 @@ Ask for Draft or Ready state before creation. Ready pull requests start the comp [NVX matrix](../../workflows/ci.yml); changing an already active pull request to Draft does not cancel its current run. After each push, follow only the workflow run whose head SHA matches the current pull-request head. Treat older runs as superseded and use -`nvx-ci-investigation` for a failed or unexpectedly cancelled current-head run. +`nvx-ci-investigation` for a failed or unexpectedly cancelled current-head run. The +run's `openvmm-ci` job waits up to 300 minutes while OpenVMM's CI of the pin runs; when +it fails because a required OpenVMM job failed, rerun that job in OpenVMM, then the +`openvmm-ci` job. `.gitmodules` tracks `feature/microvm/dev`, so the pinned revision must be on that branch before the NVX pull request merges. `microsoft/openvmm` allows only squash merges, which