From 3f972577217a7e0283fee3c51c6c875b3cdc569a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 14:25:03 -0700 Subject: [PATCH 1/3] setup: merge the Linux setup scripts into setup-linux.sh setup-linux-mshv.sh prepared a Linux/MSHV development host, and setup-linux-runner.sh a Linux/KVM or Linux/MSHV GitHub Actions runner. Each carried its own package lists, manifest reader, rustup, Rust, and cargo-nextest installation, device access rules, and host checks, and the copies had drifted: the development script's apt and dnf lists lacked diffutils, and only the runner's check mode checked its Rust targets. scripts/setup/setup-linux.sh replaces both, with the mode structure of setup-windows-whp.ps1: - Development-host mode, the default, prepares an existing checkout for the user who runs it, as setup-linux-mshv.sh did: Docker with Buildx, the user's pinned Rust release, musl target, and cargo-nextest, Docker and device group membership with the reconnect exit (status 20), the NVX build and dry run, and --check-only, --skip-build, --workspace, --guest-bundle, and --bundle-only. It may run as root or prompt for sudo, as before. - --runner-only selects runner mode, which keeps everything that setup-linux-runner.sh did: the nvx-runner account without sudo or Docker access, the root-owned trusted toolchain with the pinned sccache, registration with exactly one role and the additional labels, the service and its environment, migration of a legacy runner, writable state confined to _work, and check mode. It needs no checkout, still refuses to run as root, and never lets sudo prompt. - --backend kvm or mshv is required in both modes, so development-host mode now prepares Linux/KVM hosts too, and its dry run uses the selected hypervisor. - Each mode rejects the other's options, and the options are validated before the script reads a registration token from standard input, reads the manifest, or inspects the host, so a misplaced option never waits for a token. - One implementation installs the packages, to which each mode adds its own: Docker for a development host, or the runner's .NET dependencies on Azure Linux. One downloads and verifies release artifacts; one installs rustup, Rust, each mode's targets, and cargo-nextest, as the user or as root into the trusted toolchain; one writes the device's udev rule and grants the group; and one checks the platform, commands, Python, Rust, targets, cargo-nextest, and device access, as the user or as nvx-runner. Runner mode runs cargo as root only to install cargo-nextest, whose configuration discovery starts at CARGO_HOME. It checks the installed release by running cargo-nextest itself, as the administrator, so no configuration from the current directory, such as a cargo alias, runs as root. The output markers stay as they were: NVX_SETUP_COMPLETE, NVX_SETUP_CHECK, NVX_SETUP_RECONNECT_REQUIRED, NVX_GUEST_BUNDLE, and NVX_RUNNER_SETUP_COMPLETE, which runner check mode prints as before. Development-host check mode now also requires the musl target that the mode installs. The development script's second `tdnf install docker-cli` is gone: the first transaction already installs docker-cli, which provides moby-cli on Azure Linux 3. A missing --workspace now names the path that the script could not find instead of an empty one. The Azure VMSS bootstrap packages setup-linux.sh and runs it with --runner-only. check-quality lints the new script, and validate-runner names it in its hint. The setup README describes both modes with examples and maps the former invocations to the new ones: `setup-linux-mshv.sh ARGS` becomes `setup-linux.sh --backend mshv ARGS`, and `setup-linux-runner.sh ARGS` becomes `setup-linux.sh --runner-only ARGS`. Its runner examples now run the staged script with sh, which the non-executable file requires. The tests follow the single script: they run its manifest reader, read each mode's pins and Rust targets, select runner labels and roles, defer unnamed runner checks, and read the manifest before the host. New tests check that both modes install the same build dependencies plus their own packages, that each mode rejects the other's options before it reads the manifest, and that root runs cargo only to install. Fixes #128 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/actions/check-quality/action.yml | 6 +- .github/actions/validate-runner/action.yml | 2 +- doc/project-structure.md | 3 +- doc/setup.md | 17 +- scripts/setup/README.md | 106 ++- scripts/setup/azure/README.md | 7 +- scripts/setup/azure/build_package.py | 4 +- scripts/setup/azure/linux/bootstrap-runner.sh | 5 +- scripts/setup/setup-linux-mshv.sh | 472 ------------ .../{setup-linux-runner.sh => setup-linux.sh} | 696 ++++++++++++++---- scripts/setup/tool-versions.conf | 9 +- scripts/test_nightly_ci.py | 2 +- scripts/test_nvx_tools.py | 80 +- scripts/test_setup_azure.py | 10 +- scripts/test_tool_versions.py | 357 +++++++-- 15 files changed, 977 insertions(+), 799 deletions(-) delete mode 100644 scripts/setup/setup-linux-mshv.sh rename scripts/setup/{setup-linux-runner.sh => setup-linux.sh} (63%) diff --git a/.github/actions/check-quality/action.yml b/.github/actions/check-quality/action.yml index 3201f92c3..74d8e1230 100644 --- a/.github/actions/check-quality/action.yml +++ b/.github/actions/check-quality/action.yml @@ -48,8 +48,7 @@ runs: guest/common/nvx-container-launch-azurelinux \ guest/common/nvx-sandbox-smoke guest/common/nvx-snapshot \ guest/common/nvx-virtio-restore-probe \ - scripts/setup/setup-linux-mshv.sh \ - scripts/setup/setup-linux-runner.sh + scripts/setup/setup-linux.sh ${SHELLCHECK} --shell=bash \ .github/actions/run-benchmark/forward-signals.sh \ .github/specula/setup-runner.sh guest/ubuntu/nvx-bashrc \ @@ -70,8 +69,7 @@ runs: guest/common/nvx-container-launch-azurelinux \ guest/common/nvx-sandbox-smoke guest/common/nvx-snapshot \ guest/common/nvx-virtio-restore-probe \ - scripts/setup/setup-linux-mshv.sh \ - scripts/setup/setup-linux-runner.sh + scripts/setup/setup-linux.sh ${SHFMT} -d -ln bash -i 4 -ci \ .github/actions/run-benchmark/forward-signals.sh \ .github/specula/setup-runner.sh guest/ubuntu/nvx-bashrc \ diff --git a/.github/actions/validate-runner/action.yml b/.github/actions/validate-runner/action.yml index bf95777a4..985e1c0f7 100644 --- a/.github/actions/validate-runner/action.yml +++ b/.github/actions/validate-runner/action.yml @@ -42,7 +42,7 @@ runs: test -n "${toolchain}" if ! rustup toolchain list | awk -v name="${toolchain}" \ 'index($1, name "-") == 1 { found = 1 } END { exit !found }'; then - echo "::error::Rust ${toolchain}, which rust-toolchain.toml pins, is not installed; rerun scripts/setup/setup-linux-runner.sh" >&2 + echo "::error::Rust ${toolchain}, which rust-toolchain.toml pins, is not installed; rerun scripts/setup/setup-linux.sh --runner-only" >&2 exit 1 fi rust_version=$(rustc --version) diff --git a/doc/project-structure.md b/doc/project-structure.md index 85d24e17d..636eca87c 100644 --- a/doc/project-structure.md +++ b/doc/project-structure.md @@ -149,8 +149,7 @@ nvx/ | | |-- time_abi.py Time ABI harness checks and CPU profile guidance | | `-- ubuntu.py Verified Ubuntu rootfs and EROFS preparation | |-- setup/ Host and Actions runner bootstrap scripts -| | |-- setup-linux-mshv.sh Linux/MSHV development-host bootstrap -| | |-- setup-linux-runner.sh Linux/KVM or Linux/MSHV Actions runner bootstrap +| | |-- setup-linux.sh Linux/KVM or Linux/MSHV development-host and Actions runner bootstrap | | |-- setup-windows-whp.ps1 Windows/WHP development-host and Actions runner bootstrap | | `-- tool-versions.conf Shared tool release and checksum pins | |-- check_markdown_links.py Markdown link, anchor, and doc/ path check run by CI diff --git a/doc/setup.md b/doc/setup.md index 4fa17dcd2..94706430e 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -70,7 +70,7 @@ rustup toolchain install "$toolchain" --profile minimal rustup target add --toolchain "$toolchain" x86_64-unknown-linux-musl ``` -The Linux runner setup scripts and Copilot setup install this target +Both modes of the Linux setup script, and Copilot setup, install this target automatically. `build-essential` remains necessary for the kernel and as Rust's linker driver; no separate guest libc compiler is required. @@ -140,9 +140,12 @@ SDK, and Docker Desktop using Linux containers. ## Automated environment bootstrap The scripts in [`scripts/setup`](../scripts/setup/README.md) prepare Linux/KVM, Linux/MSHV, and -Windows/WHP development or GitHub Actions hosts. They install the platform build prerequisites, -configure hypervisor access, and provide non-mutating validation modes. Runner registration -tokens are read from standard input so they do not appear in command history. +Windows/WHP development or GitHub Actions hosts. On Linux, `setup-linux.sh` configures a +development host for the selected backend by default and an Actions runner with +`--runner-only`, as `setup-windows-whp.ps1` does on Windows with `-RunnerOnly`. They install the +platform build prerequisites, configure hypervisor access, and provide non-mutating validation +modes. Runner registration tokens are read from standard input so they do not appear in command +history. Linux can also emit a revision-bound, checksummed guest artifact bundle for Windows. The Windows script verifies that bundle before copying it into the checkout and building OpenVMM natively. @@ -221,7 +224,7 @@ shellcheck --shell=sh \ guest/common/nvx-container-launch-azurelinux \ guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ - scripts/setup/setup-linux-mshv.sh scripts/setup/setup-linux-runner.sh + scripts/setup/setup-linux.sh shellcheck --shell=bash \ .github/specula/setup-runner.sh guest/ubuntu/nvx-bashrc python3 -m pyright --pythonplatform Linux @@ -236,7 +239,7 @@ shfmt -d -ln posix -i 4 -ci \ guest/common/nvx-container-launch-azurelinux \ guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ - scripts/setup/setup-linux-mshv.sh scripts/setup/setup-linux-runner.sh + scripts/setup/setup-linux.sh shfmt -d -ln bash -i 4 -ci \ .github/specula/setup-runner.sh guest/ubuntu/nvx-bashrc python3 scripts/check_markdown_links.py @@ -282,7 +285,7 @@ shfmt -w -ln posix -i 4 -ci \ guest/common/nvx-container-launch-azurelinux \ guest/common/nvx-sandbox-smoke \ guest/common/nvx-snapshot guest/common/nvx-virtio-restore-probe \ - scripts/setup/setup-linux-mshv.sh scripts/setup/setup-linux-runner.sh + scripts/setup/setup-linux.sh shfmt -w -ln bash -i 4 -ci \ .github/specula/setup-runner.sh guest/ubuntu/nvx-bashrc ``` diff --git a/scripts/setup/README.md b/scripts/setup/README.md index 535f890e8..d1917b74a 100644 --- a/scripts/setup/README.md +++ b/scripts/setup/README.md @@ -1,34 +1,80 @@ # Environment bootstrap -These scripts prepare an existing NVX checkout on a supported host: +These scripts prepare a supported host for NVX: -- `setup-linux-mshv.sh` configures Linux/MSHV build and device prerequisites. -- `setup-windows-whp.ps1` configures Windows/WHP build prerequisites. -- `setup-linux-runner.sh` configures a Linux/KVM or Linux/MSHV Actions runner. +- `setup-linux.sh` configures a Linux/KVM or Linux/MSHV host. +- `setup-windows-whp.ps1` configures a Windows/WHP host. + +Each script has two modes. By default, it prepares a development host: an +existing NVX checkout from which a developer builds, tests, and runs NVX. +Runner-only mode, which `--runner-only` on Linux and `-RunnerOnly` on Windows +select, prepares a persistent GitHub Actions runner for NVX's CI jobs instead, +and needs no checkout. For Azure VMSS bootstrapping, see [Azure provisioning](azure/README.md). The scripts in this directory remain usable on supported hosts outside Azure. -They do not create machines or initialize a checkout. The runner modes consume -a short-lived registration token from standard input and do not store repository -credentials in the scripts. Initialize the OpenVMM submodule before using the -development-host modes. +They do not create machines or initialize a checkout. Runner-only mode +consumes a short-lived registration token from standard input and does not +store repository credentials in the scripts. Initialize the OpenVMM submodule +before using development-host mode. Each script reads the versions, release artifacts, and checksums of the tools that it installs from [`tool-versions.conf`](tool-versions.conf) in its own directory, so stage the manifest next to a script that runs outside a checkout; see [Tool versions](#tool-versions). +## Linux modes + +`setup-linux.sh` requires `--backend kvm` or `--backend mshv` in both modes. It +supports distributions with `apt-get`, `dnf`, or `tdnf`, such as Ubuntu and +Azure Linux. Both modes install the same native build dependencies, the +pinned rustup, Rust release, and cargo-nextest, and a udev rule that lets only +the backend's group open its device, `/dev/kvm` or `/dev/mshv`; on MSHV, they +load `mshv_root` when the device is missing. They differ in everything else: + +| | Development host (default) | Runner (`--runner-only`) | +| --- | --- | --- | +| Runs as | Root, or a user whose `sudo` may prompt | The SSH administrator with passwordless `sudo`, never root | +| NVX checkout | Required: the one that contains the script, or `--workspace PATH` | Not used | +| Rust tools | The current user's, with the musl target of the guest binaries | Root-owned and read-only to jobs under `/opt/nvx`, with OpenVMM's test targets and the pinned sccache | +| Docker | Installs Docker with Buildx and adds the user to `docker` | Keeps `nvx-runner` out of `docker` | +| Backend device | Adds the user to the backend's group | Makes the backend's group the only supplementary group of `nvx-runner` | +| Completes with | A build of NVX and its guests, unless `--skip-build`, and `NVX_SETUP_COMPLETE=1` | With `--runner-name`, a registered runner service, and `NVX_RUNNER_SETUP_COMPLETE=1` | +| `--check-only` prints | `NVX_SETUP_CHECK=ok` | `NVX_RUNNER_SETUP_COMPLETE=1` | + +Each mode rejects the other's options, so a misplaced option fails before the +script inspects or changes the host. [Linux development +hosts](#linux-development-hosts) and [GitHub Actions +runners](#github-actions-runners) describe the modes. + +### Migrating from the former Linux scripts + +`setup-linux.sh` replaces `setup-linux-mshv.sh` and `setup-linux-runner.sh`. +Stage it with `tool-versions.conf` instead of them, and translate their +invocations: + +| Former invocation | Replacement | +| --- | --- | +| `setup-linux-mshv.sh [OPTIONS]` | `setup-linux.sh --backend mshv [OPTIONS]` | +| `setup-linux-runner.sh --backend BACKEND [OPTIONS]` | `setup-linux.sh --runner-only --backend BACKEND [OPTIONS]` | + +Their options, output markers, and exit statuses are otherwise unchanged. +Development-host mode also accepts `--backend kvm` now, and its check mode +also requires the musl target that it installs. A runner that the former +script configured needs no migration: runner-only mode manages the same +account, toolchain, installation, and service. + ## GitHub Actions runners Generate a repository runner registration token on an authenticated workstation, then stream it to the target without placing it in shell history. On Linux: ```bash -scp scripts/setup/setup-linux-runner.sh scripts/setup/tool-versions.conf HOST:/tmp/ +scp scripts/setup/setup-linux.sh scripts/setup/tool-versions.conf HOST:/tmp/ gh api --method POST repos/microsoft/nvx/actions/runners/registration-token \ --jq .token | - ssh HOST '/tmp/setup-linux-runner.sh --backend kvm --runner-name azure-kvm-1 --runner-token-stdin' + ssh HOST 'sh /tmp/setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-1 --runner-token-stdin' ``` For Windows/WHP, stage `setup-windows-whp.ps1` and `tool-versions.conf` in one @@ -105,7 +151,8 @@ registration; a subsequent named pass accepts that directory but refuses unrelated partial runner installations. Rustup bootstrap binaries are versioned and SHA-256 verified before execution; Linux provisioning also installs `zstd` for native Actions cache archives. -Both runner setup scripts install a pinned, SHA-256-verified `sccache` binary. +Both scripts' runner-only modes install a pinned, SHA-256-verified `sccache` +binary. Runner services use a persistent `_work/_sccache` directory with a 10-GiB limit, disable Cargo incremental compilation, and expose `sccache` through `RUSTC_WRAPPER`. CI uses clean Cargo target directories and reports per-job @@ -153,8 +200,8 @@ branch. Validate an installed runner without changing the host: ```bash -sh scripts/setup/setup-linux-runner.sh \ - --backend kvm --runner-name azure-kvm-1 --check-only +sh scripts/setup/setup-linux.sh \ + --runner-only --backend kvm --runner-name azure-kvm-1 --check-only ``` ```powershell @@ -164,7 +211,7 @@ sh scripts/setup/setup-linux-runner.sh \ ### Rust toolchain -The runner scripts install the Rust release that `rust.toolchain` in the +Runner-only mode installs the Rust release that `rust.toolchain` in the manifest pins, which equals the release in the repository's [`rust-toolchain.toml`](../../rust-toolchain.toml), with its OpenVMM targets, into the read-only trusted toolchain, and check mode requires exactly @@ -176,9 +223,9 @@ through `rust-toolchain.toml`. Stage the updated script and manifest, then run that pass and validate the runner: ```bash -scp scripts/setup/setup-linux-runner.sh scripts/setup/tool-versions.conf HOST:/tmp/ -ssh HOST 'sh /tmp/setup-linux-runner.sh --backend kvm' -ssh HOST 'sh /tmp/setup-linux-runner.sh --backend kvm --runner-name azure-kvm-1 --check-only' +scp scripts/setup/setup-linux.sh scripts/setup/tool-versions.conf HOST:/tmp/ +ssh HOST 'sh /tmp/setup-linux.sh --runner-only --backend kvm' +ssh HOST 'sh /tmp/setup-linux.sh --runner-only --backend kvm --runner-name azure-kvm-1 --check-only' ``` ```powershell @@ -189,17 +236,19 @@ ssh HOST powershell.exe -NoProfile -ExecutionPolicy Bypass ` -File C:\setup-windows-whp.ps1 -RunnerOnly -RunnerName nvx-windows-3 -CheckOnly ``` -## Linux / MSHV +## Linux development hosts -The Linux script supports distributions with `apt-get`, `dnf`, or `tdnf`. It -installs native build dependencies and the Rust and cargo-nextest releases -that the manifest pins. It installs Docker with Buildx and configures -group-based access to Docker and `/dev/mshv`. +Development-host mode prepares a Linux/KVM or Linux/MSHV checkout for the user +who runs the script. It installs native build dependencies, Docker with Buildx, +and the Rust and cargo-nextest releases that the manifest pins, and configures +group-based access to Docker and to the backend's device. Its udev rule lets +only the backend's group open the device, so another user of the host needs +that group too, even where the device was open to everyone before. -Run the complete bootstrap from the checkout: +Run the complete bootstrap from the checkout, naming the host's backend: ```bash -sh scripts/setup/setup-linux-mshv.sh +sh scripts/setup/setup-linux.sh --backend mshv ``` Changing group membership exits with status 20. Reconnect and run the script @@ -208,16 +257,17 @@ again so the new membership applies. Validate dependencies without changing the host or requiring build artifacts: ```bash -sh scripts/setup/setup-linux-mshv.sh --check-only --skip-build +sh scripts/setup/setup-linux.sh --backend kvm --check-only --skip-build ``` -Omit `--skip-build` to also validate the built artifacts and generated MSHV -command. Use `--workspace PATH` when the script is outside the checkout. +Omit `--skip-build` to also validate the built artifacts and the command that +NVX generates for the backend. Use `--workspace PATH` when the script is +outside the checkout. To produce a revision-bound guest bundle for a Windows build: ```bash -sh scripts/setup/setup-linux-mshv.sh \ +sh scripts/setup/setup-linux.sh --backend mshv \ --guest-bundle /path/to/nvx-guest-artifacts ``` diff --git a/scripts/setup/azure/README.md b/scripts/setup/azure/README.md index 56e3b6565..766c6f86b 100644 --- a/scripts/setup/azure/README.md +++ b/scripts/setup/azure/README.md @@ -2,8 +2,9 @@ These Azure-specific adapters configure **existing** NVX CI virtual machine scale sets. They package the provider-neutral -[`setup-linux-runner.sh`](../setup-linux-runner.sh) or -[`setup-windows-whp.ps1`](../setup-windows-whp.ps1) together with the adjacent +[`setup-linux.sh`](../setup-linux.sh), which the Linux bootstrap runs in +runner-only mode, or [`setup-windows-whp.ps1`](../setup-windows-whp.ps1) +together with the adjacent [`tool-versions.conf`](../tool-versions.conf). That manifest and [`rust-toolchain.toml`](../../../rust-toolchain.toml) must pin the same release. Do not maintain a second copy of the runner setup or its tool versions here. @@ -13,7 +14,7 @@ Do not maintain a second copy of the runner setup or its tool versions here. | `linux/bootstrap-runner.sh` | Validate Azure instance identity, prepare the attached disk and backend, derive a runner name, and register using managed identity and Key Vault. | | `linux/block-runner-metadata.sh` | Block the dedicated runner account from Azure instance metadata with a verified, reboot-persistent IPv4 owner rule. | | `linux/setup-data-disk.sh` | Find an Azure managed disk by LUN, preserve recognized ext4 data, and initialize only a blank or owned incomplete disk. | -| `linux/setup-mshv.sh`, `linux/verify-mshv.sh` | Optional Azure Linux 3 Dom0 boot preparation and post-boot verification. This differs from the checkout/build helper `../setup-linux-mshv.sh`. | +| `linux/setup-mshv.sh`, `linux/verify-mshv.sh` | Optional Azure Linux 3 Dom0 boot preparation and post-boot verification. This differs from `../setup-linux.sh`, which requires an MSHV kernel that is already running. | | `windows/bootstrap-runner.ps1` | Validate instance identity; set up disk, WHP/Hyper-V features, reboot continuation, and the Network Service runner. | | `windows/setup-data-disk.ps1` | Safely prepare an Azure managed NTFS data disk and handle temporary-drive/pagefile conflicts. | | `windows/vmss-bootstrap.ps1`, `windows/install-package.ps1` | Run through Custom Script, configure OpenSSH, verify and unpack the source bundle, and start Windows bootstrap. | diff --git a/scripts/setup/azure/build_package.py b/scripts/setup/azure/build_package.py index 5a1dc44e9..54bacc7b2 100644 --- a/scripts/setup/azure/build_package.py +++ b/scripts/setup/azure/build_package.py @@ -418,7 +418,7 @@ def load_config(path: str | Path) -> LinuxConfig | WindowsConfig: def source(path: str) -> bytes: - if path in {"setup-linux-runner.sh", "setup-windows-whp.ps1", "tool-versions.conf"}: + if path in {"setup-linux.sh", "setup-windows-whp.ps1", "tool-versions.conf"}: location = SETUP / path elif path == "rust-toolchain.toml": location = REPO / path @@ -466,7 +466,7 @@ def linux_payload( ) -> tuple[bytes, dict[str, bytes], dict[str, bytes]]: files = { "config.json": configuration_bytes(config), - "setup-linux-runner.sh": source("setup-linux-runner.sh"), + "setup-linux.sh": source("setup-linux.sh"), "tool-versions.conf": source("tool-versions.conf"), "setup-data-disk.sh": source("linux/setup-data-disk.sh"), "bootstrap-runner.sh": source("linux/bootstrap-runner.sh"), diff --git a/scripts/setup/azure/linux/bootstrap-runner.sh b/scripts/setup/azure/linux/bootstrap-runner.sh index 728364941..e9f3312fe 100644 --- a/scripts/setup/azure/linux/bootstrap-runner.sh +++ b/scripts/setup/azure/linux/bootstrap-runner.sh @@ -186,6 +186,7 @@ fi } setup_arguments=( + --runner-only --backend "$backend" --repository-url "https://github.com/$repository" --runner-directory "$runner_directory" @@ -193,7 +194,7 @@ setup_arguments=( if [ -n "$extra_runner_labels" ]; then setup_arguments+=(--extra-runner-labels "$extra_runner_labels") fi -sudo -u "$admin_username" -H sh "$setup_root/setup-linux-runner.sh" "${setup_arguments[@]}" +sudo -u "$admin_username" -H sh "$setup_root/setup-linux.sh" "${setup_arguments[@]}" metadata_service=nvx-runner-metadata-block.service systemctl enable "$metadata_service" @@ -246,7 +247,7 @@ if [ "$benchmark_runner" = true ]; then registration_arguments+=(--benchmark-runner) fi printf '%s\n' "$runner_token" | - sudo -u "$admin_username" -H sh "$setup_root/setup-linux-runner.sh" \ + sudo -u "$admin_username" -H sh "$setup_root/setup-linux.sh" \ "${registration_arguments[@]}" --runner-token-stdin runner_token= diff --git a/scripts/setup/setup-linux-mshv.sh b/scripts/setup/setup-linux-mshv.sh deleted file mode 100644 index a3fffe7ee..000000000 --- a/scripts/setup/setup-linux-mshv.sh +++ /dev/null @@ -1,472 +0,0 @@ -#!/bin/sh - -set -eu - -PYTHON_MINIMUM_VERSION=3.10.0 - -script_directory=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) -workspace=$(CDPATH='' cd -- "${script_directory}/../.." && pwd) -# The versions, artifacts, and checksums of the tools that this script -# installs; stage the manifest next to the script. -tool_manifest=${script_directory}/tool-versions.conf -tool_platform=linux-x86_64 -guest_bundle= -bundle_only=false -check_only=false -skip_build=false -reconnect_required=false - -usage() { - cat <&2 - exit 1 -} - -require_command() { - command -v "$1" >/dev/null 2>&1 || die "required command not found: $1" -} - -version_at_least() { - [ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | head -n 1)" = "$2" ] -} - -tool_manifest_error() { - die "invalid tool manifest ${tool_manifest}:${tool_manifest_line}: $*" -} - -# Reads the tool manifest as data, without sourcing or evaluating it, so a -# fresh host needs nothing beyond the shell. Every line must be well formed. -load_tool_manifest() { - [ -f "$tool_manifest" ] || die "tool manifest not found: ${tool_manifest}" - carriage_return=$(printf '\r') - blank=$(printf ' \t') - tool_manifest_entries= - tool_manifest_line=0 - while IFS= read -r line || [ -n "$line" ]; do - tool_manifest_line=$((tool_manifest_line + 1)) - line=${line%"$carriage_return"} - case "$line" in - '' | '#'*) continue ;; - *["$blank"]*) tool_manifest_error "expected KEY=VALUE without spaces" ;; - *=*) ;; - *) tool_manifest_error "expected KEY=VALUE without spaces" ;; - esac - key=${line%%=*} - value=${line#*=} - tool=${key%%.*} - case "$tool" in - '' | [!a-z]* | *[!a-z0-9_]*) tool_manifest_error "unsupported key: ${key}" ;; - esac - case "${key#"$tool"}" in - .version | .toolchain) field=version ;; - .artifacts.*.name | .artifacts.*.sha256) - field=${key##*.} - platform=${key#"$tool".artifacts.} - platform=${platform%.*} - case "$platform" in - '' | *[!a-z0-9_-]*) tool_manifest_error "unsupported key: ${key}" ;; - linux-x86_64 | windows-x86_64) ;; - *) tool_manifest_error "unsupported platform: ${platform}" ;; - esac - ;; - *) tool_manifest_error "unsupported key: ${key}" ;; - esac - case "${tool_manifest_entries} " in - *" ${key}="*) tool_manifest_error "duplicate key: ${key}" ;; - esac - case "$field" in - version) - case "$value" in - *[!0-9.]* | .* | *. | *..* | *.*.*.*) value= ;; - *.*.*) ;; - *) value= ;; - esac - ;; - sha256) - case "$value" in - *[!0-9a-f]*) value= ;; - esac - [ "${#value}" -eq 64 ] || value= - ;; - name) - name=$value - while :; do - case "$name" in - *'{version}'*) name=${name%%'{version}'*}v${name#*'{version}'} ;; - *) break ;; - esac - done - case "$name" in - '' | /* | */ | *//* | .* | */.* | *[!A-Za-z0-9._/-]*) value= ;; - esac - ;; - esac - [ -n "$value" ] || - tool_manifest_error "invalid value for ${key}: ${line#*=}" - tool_manifest_entries="${tool_manifest_entries} ${key}=${value}" - done <"$tool_manifest" -} - -# Sets tool_value to the value of a key in the loaded tool manifest. -tool_manifest_value() { - case "${tool_manifest_entries} " in - *" ${1}="*) ;; - *) die "tool manifest ${tool_manifest} does not define ${1}" ;; - esac - tool_value=${tool_manifest_entries#*" ${1}="} - tool_value=${tool_value%%" "*} -} - -# Sets tool_version, tool_artifact, and tool_sha256 to a tool's release and -# the name and checksum of its artifact for tool_platform. -tool_manifest_artifact() { - tool_manifest_value "${1}.version" - tool_version=$tool_value - tool_manifest_value "${1}.artifacts.${tool_platform}.name" - tool_artifact= - while :; do - case "$tool_value" in - *'{version}'*) - tool_artifact=${tool_artifact}${tool_value%%'{version}'*}${tool_version} - tool_value=${tool_value#*'{version}'} - ;; - *) break ;; - esac - done - tool_artifact=${tool_artifact}${tool_value} - tool_manifest_value "${1}.artifacts.${tool_platform}.sha256" - tool_sha256=$tool_value -} - -# Pins every tool that this script installs before it inspects or changes -# the host. -read_tool_versions() { - load_tool_manifest - tool_manifest_value rust.toolchain - RUST_TOOLCHAIN=$tool_value - tool_manifest_value cargo_nextest.version - CARGO_NEXTEST_VERSION=$tool_value - tool_manifest_artifact rustup - RUSTUP_VERSION=$tool_version - RUSTUP_ARTIFACT=$tool_artifact - RUSTUP_SHA256=$tool_sha256 -} - -run_as_root() { - if [ "$(id -u)" -eq 0 ]; then - "$@" - else - sudo "$@" - fi -} - -install_packages() { - if command -v apt-get >/dev/null 2>&1; then - run_as_root apt-get update - run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y \ - bc binutils bison build-essential ca-certificates cmake cpio curl \ - docker-buildx docker.io flex git gzip iproute2 iptables \ - libarchive-tools libelf-dev libssl-dev make ninja-build patch perl \ - pkg-config protobuf-compiler python3 rsync tar util-linux xz-utils \ - zstd - elif command -v tdnf >/dev/null 2>&1; then - run_as_root tdnf install -y \ - bc binutils bison ca-certificates cmake cpio curl diffutils \ - docker-buildx docker-cli elfutils-libelf-devel findutils flex gcc \ - gcc-c++ git glibc-devel glibc-iconv gzip iproute iptables \ - kernel-headers libarchive libarchive-devel make moby-engine \ - ninja-build openssl openssl-devel patch perl perl-FindBin \ - perl-IPC-Cmd perl-Time-Piece perl-lib pkgconf pkgconf-pkg-config \ - protobuf python3 rsync shadow-utils tar \ - util-linux which xz zstd - run_as_root tdnf install -y docker-cli - elif command -v dnf >/dev/null 2>&1; then - run_as_root dnf install -y \ - bc binutils bison ca-certificates cmake cpio curl elfutils-libelf-devel \ - findutils flex gcc gcc-c++ git glibc-devel gzip iproute iptables \ - kernel-headers libarchive libarchive-devel make moby-engine \ - ninja-build openssl openssl-devel patch perl pkgconf \ - pkgconf-pkg-config protobuf-compiler python3 rsync shadow-utils tar \ - util-linux which xz zstd docker-buildx - else - die "supported package manager not found (apt-get, dnf, or tdnf)" - fi -} - -install_rust_tools() { - if ! command -v rustup >/dev/null 2>&1; then - require_command curl - installer=$(mktemp "${TMPDIR:-/tmp}/rustup-init-${RUSTUP_VERSION}.XXXXXX") - trap 'rm -f "$installer"' 0 HUP INT TERM - installer_url=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/${RUSTUP_ARTIFACT} - curl --fail --proto '=https' --tlsv1.2 --silent --show-error \ - --output "$installer" "$installer_url" - printf '%s %s\n' "$RUSTUP_SHA256" "$installer" | sha256sum --check - - chmod 0755 "$installer" - "$installer" -y --no-modify-path --profile minimal \ - --default-toolchain "$RUST_TOOLCHAIN" - rm -f "$installer" - trap - 0 HUP INT TERM - # shellcheck disable=SC1091 - . "${HOME}/.cargo/env" - fi - - rustup toolchain install "$RUST_TOOLCHAIN" --profile minimal \ - --no-self-update - rustup target add --toolchain "$RUST_TOOLCHAIN" \ - x86_64-unknown-linux-musl - rust_version=$(RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN rustc --version | awk '{print $2}') - [ "$rust_version" = "$RUST_TOOLCHAIN" ] || - die "Rust ${RUST_TOOLCHAIN} is required, found ${rust_version}" - rustup target list --installed --toolchain "$RUST_TOOLCHAIN" | - grep -Fxq x86_64-unknown-linux-musl || - die "Rust target x86_64-unknown-linux-musl is not installed" - if ! command -v cargo-nextest >/dev/null 2>&1 || - ! cargo nextest --version | grep -Fq "cargo-nextest ${CARGO_NEXTEST_VERSION}"; then - cargo +"$RUST_TOOLCHAIN" install --locked cargo-nextest \ - --version "$CARGO_NEXTEST_VERSION" - fi -} - -configure_docker_access() { - require_command systemctl - run_as_root systemctl enable --now docker - getent group docker >/dev/null 2>&1 || - die "the Docker package did not create its access group" - - user_name=${SUDO_USER:-$(id -un)} - if [ "$user_name" != root ] && - ! id -nG "$user_name" | tr ' ' '\n' | grep -Fxq docker; then - run_as_root usermod -aG docker "$user_name" - reconnect_required=true - fi -} - -configure_mshv_access() { - if [ ! -e /dev/mshv ]; then - run_as_root modprobe mshv_root 2>/dev/null || - die "/dev/mshv is unavailable and mshv_root could not be loaded" - fi - [ -c /dev/mshv ] || die "/dev/mshv is not a character device" - - if ! getent group mshv >/dev/null 2>&1; then - run_as_root groupadd --system mshv - fi - run_as_root sh -c \ - 'printf "%s\n" '\''KERNEL=="mshv", GROUP="mshv", MODE="0660"'\'' > /etc/udev/rules.d/70-mshv.rules' - run_as_root udevadm control --reload-rules - run_as_root udevadm trigger --name-match=mshv - - user_name=${SUDO_USER:-$(id -un)} - if [ "$user_name" != root ] && - ! id -nG "$user_name" | tr ' ' '\n' | grep -Fxq mshv; then - run_as_root usermod -aG mshv "$user_name" - reconnect_required=true - fi - run_as_root chgrp mshv /dev/mshv - run_as_root chmod 0660 /dev/mshv -} - -check_environment() { - [ "$(uname -s)" = Linux ] || die "this script requires Linux" - [ "$(uname -m)" = x86_64 ] || die "this script requires x86_64" - [ -f "${workspace}/scripts/nvx.py" ] || - die "NVX checkout not found at ${workspace}" - [ -f "${workspace}/openvmm/Cargo.toml" ] || - die "OpenVMM submodule is not initialized" - [ -c /dev/mshv ] || die "/dev/mshv is not a character device" - [ -r /dev/mshv ] && [ -w /dev/mshv ] || - die "current session cannot access /dev/mshv" - - for command_name in python3 git curl rustup cargo cargo-nextest gcc make ld \ - bison flex cpio gzip sha256sum tar xz zstd; do - require_command "$command_name" - done - python_version=$(python3 -c \ - 'import sys; print(".".join(map(str, sys.version_info[:3])))') - version_at_least "$python_version" "$PYTHON_MINIMUM_VERSION" || - die "Python ${PYTHON_MINIMUM_VERSION} or newer is required" - rust_version=$(RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN rustc --version | awk '{print $2}') - [ "$rust_version" = "$RUST_TOOLCHAIN" ] || - die "Rust ${RUST_TOOLCHAIN} is required, found ${rust_version}" - cargo nextest --version | grep -Fq "cargo-nextest ${CARGO_NEXTEST_VERSION}" || - die "cargo-nextest ${CARGO_NEXTEST_VERSION} is not installed" - require_command docker - docker version >/dev/null 2>&1 || - die "Docker is not available to the current session" - docker buildx version >/dev/null 2>&1 || die "Docker Buildx is not installed" - - ( - cd "$workspace" - python3 scripts/nvx.py verify - ) -} - -check_build() { - ( - cd "$workspace" - RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN cargo build --release --locked --features cli \ - --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target - # Packaging and the adversarial executor take only an nvx-rs whose - # provenance binds it to the checkout's sources. - RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN NVX_RS=nvx-rs/target/release/nvx-rs \ - python3 scripts/nvx.py record-nvx-rs-provenance - nvx-rs/target/release/nvx-rs run --hypervisor mshv --dry-run - ) -} - -build_nvx() { - require_command docker - docker version >/dev/null 2>&1 || die "Docker is not available to the current session" - docker buildx version >/dev/null 2>&1 || die "Docker Buildx is not installed" - ( - cd "$workspace" - RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN python3 scripts/nvx.py build --guest all - ) -} - -check_guest_bundle() { - bundle_path=$1 - [ -d "$bundle_path" ] || die "guest bundle not found: $bundle_path" - for name in vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ - initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ - initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ - ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ - initramfs-azurelinux.cpio.gz.packages.json REVISION SHA256SUMS; do - [ -f "${bundle_path}/${name}" ] || - die "missing guest bundle file: ${bundle_path}/${name}" - done - ( - cd "$bundle_path" - sha256sum --check SHA256SUMS - ) || die "guest bundle checksum validation failed" - bundle_revision=$(cat "${bundle_path}/REVISION") - workspace_revision=$(git -C "$workspace" rev-parse HEAD) - [ "$bundle_revision" = "$workspace_revision" ] || - die "guest bundle revision does not match the NVX checkout" -} - -create_guest_bundle() { - bundle_path=$1 - if [ -e "$bundle_path" ] && - find "$bundle_path" -mindepth 1 -print -quit | grep -q .; then - die "guest bundle directory is not empty: $bundle_path" - fi - mkdir -p "$bundle_path" - for name in vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ - initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ - initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ - ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ - initramfs-azurelinux.cpio.gz.packages.json; do - source_path="${workspace}/build/${name}" - [ -f "$source_path" ] || die "missing guest artifact: $source_path" - cp "$source_path" "$bundle_path" - done - git -C "$workspace" rev-parse HEAD >"${bundle_path}/REVISION" - ( - cd "$bundle_path" - sha256sum vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ - initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ - initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ - ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ - initramfs-azurelinux.cpio.gz.packages.json REVISION >SHA256SUMS - ) - check_guest_bundle "$bundle_path" -} - -while [ "$#" -gt 0 ]; do - case "$1" in - --workspace) - [ "$#" -ge 2 ] || die "--workspace requires a path" - workspace=$2 - shift 2 - ;; - --guest-bundle) - [ "$#" -ge 2 ] || die "--guest-bundle requires a path" - guest_bundle=$2 - shift 2 - ;; - --bundle-only) - bundle_only=true - shift - ;; - --check-only) - check_only=true - shift - ;; - --skip-build) - skip_build=true - shift - ;; - -h | --help) - usage - exit 0 - ;; - *) - die "unknown option: $1" - ;; - esac -done - -read_tool_versions -workspace=$(CDPATH='' cd -- "$workspace" 2>/dev/null && pwd) || - die "workspace directory not found: $workspace" - -if [ "$bundle_only" = true ]; then - [ -n "$guest_bundle" ] || die "--bundle-only requires --guest-bundle" - [ "$check_only" = false ] || - die "--bundle-only and --check-only cannot be combined" - create_guest_bundle "$guest_bundle" - printf 'NVX_GUEST_BUNDLE=%s\n' "$guest_bundle" - exit 0 -fi - -if [ "$check_only" = true ]; then - check_environment - if [ "$skip_build" = false ]; then - check_build - fi - if [ -n "$guest_bundle" ]; then - check_guest_bundle "$guest_bundle" - fi - printf 'NVX_SETUP_CHECK=ok\n' - exit 0 -fi - -if [ "$(id -u)" -ne 0 ]; then - require_command sudo -fi -install_packages -install_rust_tools -configure_docker_access -configure_mshv_access -if [ "$reconnect_required" = true ]; then - printf '%s\n' \ - 'NVX_SETUP_RECONNECT_REQUIRED=1' \ - 'Reconnect so Docker/MSHV group membership applies, then rerun this script.' - exit 20 -fi -if [ "$skip_build" = false ]; then - build_nvx - check_build -fi -if [ -n "$guest_bundle" ]; then - create_guest_bundle "$guest_bundle" -fi -check_environment -printf 'NVX_SETUP_COMPLETE=1\n' diff --git a/scripts/setup/setup-linux-runner.sh b/scripts/setup/setup-linux.sh similarity index 63% rename from scripts/setup/setup-linux-runner.sh rename to scripts/setup/setup-linux.sh index c2bba779a..33885d7c1 100644 --- a/scripts/setup/setup-linux-runner.sh +++ b/scripts/setup/setup-linux.sh @@ -2,6 +2,8 @@ set -eu +PYTHON_MINIMUM_VERSION=3.10.0 + script_directory=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) # The versions, artifacts, and checksums of the tools that this script # installs; stage the manifest next to the script. @@ -9,17 +11,28 @@ tool_manifest=${script_directory}/tool-versions.conf tool_platform=linux-x86_64 backend= -benchmark_runner=false check_only=false +runner_only=false +# The last option given that only development-host or only runner mode takes. +development_option= +runner_option= + +workspace=${script_directory}/../.. +guest_bundle= +bundle_only=false +skip_build=false +reconnect_required=false + +benchmark_runner=false configure_runner=false repository_url=https://github.com/microsoft/nvx -legacy_runner_directory=${HOME}/actions-runner runner_directory=/opt/nvx-runner runner_directory_is_default=true runner_name= extra_runner_labels= runner_service_account=nvx-runner runner_token= +runner_token_stdin=false runner_package_directory= removed_service_name= trusted_tool_root=/opt/nvx @@ -31,11 +44,31 @@ runner_service_path=${trusted_cargo_home}/bin:/usr/local/sbin:/usr/local/bin:/us usage() { cat </dev/null 2>&1; then + set -- bc binutils bison build-essential ca-certificates cmake cpio curl \ + diffutils flex git gzip iproute2 iptables libarchive-tools libelf-dev \ + libssl-dev make ninja-build patch perl pkg-config protobuf-compiler \ + python3 rsync tar util-linux xz-utils zstd + if [ "$runner_only" = false ]; then + set -- "$@" docker-buildx docker.io + fi run_as_root apt-get update - run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y \ - bc binutils bison build-essential ca-certificates cmake cpio curl diffutils \ - flex git gzip iproute2 iptables \ - libarchive-tools libelf-dev libssl-dev make ninja-build patch perl \ - pkg-config protobuf-compiler python3 rsync tar util-linux xz-utils \ - zstd + run_as_root env DEBIAN_FRONTEND=noninteractive apt-get install -y "$@" elif command -v tdnf >/dev/null 2>&1; then - run_as_root tdnf install -y \ - bc binutils bison ca-certificates cmake cpio curl diffutils \ - elfutils-libelf-devel findutils flex gcc \ - gcc-c++ git glibc-devel glibc-iconv gzip icu iproute iptables \ - kernel-headers libarchive libarchive-devel lttng-ust make \ - ninja-build openssl openssl-devel patch perl perl-FindBin \ - perl-IPC-Cmd perl-Time-Piece perl-lib pkgconf pkgconf-pkg-config \ - protobuf python3 rsync shadow-utils tar \ - util-linux which xz zstd - elif command -v dnf >/dev/null 2>&1; then - run_as_root dnf install -y \ - bc binutils bison ca-certificates cmake cpio curl diffutils \ + set -- bc binutils bison ca-certificates cmake cpio curl diffutils \ elfutils-libelf-devel findutils flex gcc gcc-c++ git glibc-devel \ - gzip iproute iptables kernel-headers libarchive libarchive-devel \ - make ninja-build openssl openssl-devel patch perl \ - pkgconf pkgconf-pkg-config protobuf-compiler python3 rsync \ - shadow-utils tar util-linux which xz zstd + glibc-iconv gzip iproute iptables kernel-headers libarchive \ + libarchive-devel make ninja-build openssl openssl-devel patch perl \ + perl-FindBin perl-IPC-Cmd perl-Time-Piece perl-lib pkgconf \ + pkgconf-pkg-config protobuf python3 rsync shadow-utils tar util-linux \ + which xz zstd + if [ "$runner_only" = true ]; then + set -- "$@" icu lttng-ust + else + set -- "$@" docker-buildx docker-cli moby-engine + fi + run_as_root tdnf install -y "$@" + elif command -v dnf >/dev/null 2>&1; then + set -- bc binutils bison ca-certificates cmake cpio curl diffutils \ + elfutils-libelf-devel findutils flex gcc gcc-c++ git glibc-devel gzip \ + iproute iptables kernel-headers libarchive libarchive-devel make \ + ninja-build openssl openssl-devel patch perl pkgconf \ + pkgconf-pkg-config protobuf-compiler python3 rsync shadow-utils tar \ + util-linux which xz zstd + if [ "$runner_only" = false ]; then + set -- "$@" docker-buildx moby-engine + fi + run_as_root dnf install -y "$@" else die "supported package manager not found (apt-get, dnf, or tdnf)" fi } -install_rust_tools() { - rustup=${trusted_cargo_home}/bin/rustup - cargo=${trusted_cargo_home}/bin/cargo - rustc=${trusted_cargo_home}/bin/rustc - nextest=${trusted_cargo_home}/bin/cargo-nextest - sccache=${trusted_cargo_home}/bin/sccache - run_as_root mkdir -p "$trusted_cargo_home" "$trusted_rustup_home" - - if [ ! -x "$rustup" ]; then - installer=${trusted_tool_root}/rustup-init-${RUSTUP_VERSION} - installer_url=https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/${RUSTUP_ARTIFACT} - run_as_root curl --fail --location --proto '=https' --tlsv1.2 \ - --silent --show-error --output "$installer" "$installer_url" - printf '%s %s\n' "$RUSTUP_SHA256" "$installer" | - run_as_root sha256sum --check - - run_as_root chmod 0755 "$installer" - run_as_root env \ - CARGO_HOME="$trusted_cargo_home" \ - RUSTUP_HOME="$trusted_rustup_home" \ - "$installer" -y --no-modify-path --profile minimal \ - --default-toolchain "$RUST_TOOLCHAIN" - run_as_root rm -f "$installer" +# Installs the pinned rustup with the pinned Rust release as its default +# toolchain. +install_rustup() { + if [ "$runner_only" = true ]; then + installer_directory=$trusted_tool_root + else + installer_directory=${TMPDIR:-/tmp} fi + installer=$(run_as_tool_owner mktemp \ + "${installer_directory}/rustup-init-${RUSTUP_VERSION}.XXXXXX") + trap 'run_as_tool_owner rm -f "$installer"' 0 HUP INT TERM + download_verified \ + "https://static.rust-lang.org/rustup/archive/${RUSTUP_VERSION}/${RUSTUP_ARTIFACT}" \ + "$RUSTUP_SHA256" "$installer" + run_as_tool_owner chmod 0755 "$installer" + run_rust_tool "$installer" -y --no-modify-path --profile minimal \ + --default-toolchain "$RUST_TOOLCHAIN" + run_as_tool_owner rm -f "$installer" + trap - 0 HUP INT TERM +} - run_as_root env \ - CARGO_HOME="$trusted_cargo_home" \ - RUSTUP_HOME="$trusted_rustup_home" \ - "$rustup" toolchain install "$RUST_TOOLCHAIN" --profile minimal \ - --no-self-update - run_as_root env \ - CARGO_HOME="$trusted_cargo_home" \ - RUSTUP_HOME="$trusted_rustup_home" \ - "$rustup" target add --toolchain "$RUST_TOOLCHAIN" \ - x86_64-unknown-none \ - x86_64-unknown-linux-musl \ - x86_64-unknown-uefi - - rust_version=$(run_as_root env \ - CARGO_HOME="$trusted_cargo_home" \ - RUSTUP_HOME="$trusted_rustup_home" \ - RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ - "$rustc" --version | awk '{print $2}') - [ "$rust_version" = "$RUST_TOOLCHAIN" ] || - die "Rust ${RUST_TOOLCHAIN} is required, found ${rust_version}" - if [ ! -x "$nextest" ] || - ! "$nextest" --version | grep -Fq "cargo-nextest ${CARGO_NEXTEST_VERSION}"; then - run_as_root env \ - CARGO_HOME="$trusted_cargo_home" \ - RUSTUP_HOME="$trusted_rustup_home" \ - "$cargo" +"$RUST_TOOLCHAIN" install --locked --force cargo-nextest \ - --version "$CARGO_NEXTEST_VERSION" - fi +# Installs the pinned sccache release into the trusted toolchain. +install_sccache() { + sccache=${trusted_cargo_home}/bin/sccache if [ ! -x "$sccache" ] || ! "$sccache" --version | grep -Fq "sccache ${SCCACHE_VERSION}"; then ( @@ -653,13 +785,9 @@ install_rust_tools() { archive=${package_directory}/sccache.tar.gz root=${package_directory}/root trap 'run_as_root rm -rf "$package_directory"' 0 HUP INT TERM - run_as_root curl --fail --location --proto '=https' --tlsv1.2 \ - --silent --show-error \ - --output "$archive" \ - "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/${SCCACHE_ARTIFACT}" - actual_sha256=$(run_as_root sha256sum "$archive" | awk '{print $1}') - [ "$actual_sha256" = "$SCCACHE_SHA256" ] || - die "sccache archive checksum mismatch: ${actual_sha256}" + download_verified \ + "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/${SCCACHE_ARTIFACT}" \ + "$SCCACHE_SHA256" "$archive" run_as_root mkdir -p "$root" run_as_root tar --extract --gzip \ --file "$archive" \ @@ -670,8 +798,56 @@ install_rust_tools() { run_as_root install -m 0755 "$extracted" "$sccache" ) fi - run_as_root chown -R root:root "$trusted_tool_root" - run_as_root chmod -R go-w "$trusted_tool_root" +} + +# Installs the pinned Rust release with this mode's targets and the pinned +# cargo-nextest: for the current user on a development host, or, for runner +# jobs, with the pinned sccache as a toolchain that only root can modify. +install_rust_tools() { + rust_bin= + if [ "$runner_only" = true ]; then + rust_bin=${trusted_cargo_home}/bin/ + run_as_root mkdir -p "$trusted_cargo_home" "$trusted_rustup_home" + fi + if ! command -v "${rust_bin}rustup" >/dev/null 2>&1; then + install_rustup + if [ "$runner_only" = false ]; then + # shellcheck disable=SC1091 + . "${HOME}/.cargo/env" + fi + fi + + run_rust_tool "${rust_bin}rustup" toolchain install "$RUST_TOOLCHAIN" \ + --profile minimal --no-self-update + with_rust_targets run_rust_tool "${rust_bin}rustup" target add \ + --toolchain "$RUST_TOOLCHAIN" + rust_version=$(run_rust_tool env RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ + "${rust_bin}rustc" --version | awk '{print $2}') + [ "$rust_version" = "$RUST_TOOLCHAIN" ] || + die "Rust ${RUST_TOOLCHAIN} is required, found ${rust_version}" + # Run cargo-nextest itself, as the invoking user: cargo reads the + # configuration of the current directory, such as its aliases, which + # another user, such as a runner's job, may control. + if ! "${rust_bin}cargo-nextest" --version 2>/dev/null | + grep -Fq "cargo-nextest ${CARGO_NEXTEST_VERSION}"; then + run_rust_tool "${rust_bin}cargo" +"$RUST_TOOLCHAIN" install --locked \ + --force cargo-nextest --version "$CARGO_NEXTEST_VERSION" + fi + if [ "$runner_only" = true ]; then + install_sccache + run_as_root chown -R root:root "$trusted_tool_root" + run_as_root chmod -R go-w "$trusted_tool_root" + fi +} + +# Starts Docker and lets the user who runs this script build guest artifacts +# with it. +configure_docker_access() { + require_command systemctl + run_as_root systemctl enable --now docker + getent group docker >/dev/null 2>&1 || + die "the Docker package did not create its access group" + grant_group docker } restrict_docker_access() { @@ -681,6 +857,9 @@ restrict_docker_access() { fi } +# Lets only the backend's group open the backend's device, now and after every +# boot, and makes that group the runner service account's only supplementary +# group, or adds the user who runs this script to it. configure_backend_access() { device=/dev/${backend} module=${backend} @@ -701,7 +880,11 @@ configure_backend_access() { run_as_root udevadm trigger --name-match="$backend" run_as_root chgrp "$backend" "$device" run_as_root chmod 0660 "$device" - run_as_root usermod -G "$backend" "$runner_service_account" + if [ "$runner_only" = true ]; then + run_as_root usermod -G "$backend" "$runner_service_account" + else + grant_group "$backend" + fi } install_runner() { @@ -769,34 +952,64 @@ install_runner() { configure_runner_service } -check_environment() { +check_platform() { [ "$(uname -s)" = Linux ] || die "this script requires Linux" [ "$(uname -m)" = x86_64 ] || die "this script requires x86_64" - for command_name in python3 git curl diff rustup cargo cargo-nextest sccache gcc make ld \ - bison flex cpio gzip sha256sum tar xz zstd systemctl; do +} + +# Checks the Python release and the commands that both modes need, and those +# that the caller names. +check_host_tools() { + for command_name in python3 git curl rustup cargo cargo-nextest gcc make ld \ + bison flex cpio gzip sha256sum tar xz zstd "$@"; do require_command "$command_name" done - python_version=$(python3 -c \ 'import sys; print(".".join(map(str, sys.version_info[:3])))') - version_at_least "$python_version" 3.10.0 || - die "Python 3.10.0 or newer is required" - rust_version=$(run_as_runner env RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ + version_at_least "$python_version" "$PYTHON_MINIMUM_VERSION" || + die "Python ${PYTHON_MINIMUM_VERSION} or newer is required" +} + +# Checks that installed_targets lists every target given. +check_rust_targets() { + for target in "$@"; do + printf '%s\n' "$installed_targets" | grep -Fxq "$target" || + die "Rust target ${target} is not installed" + done +} + +# Checks that whoever builds NVX on this host runs the pinned Rust release, +# with this mode's targets, and the pinned cargo-nextest. +check_rust_tools() { + rust_version=$(run_as_builder env RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ rustc --version | awk '{print $2}') [ "$rust_version" = "$RUST_TOOLCHAIN" ] || die "Rust ${RUST_TOOLCHAIN} is required, found ${rust_version}" - run_as_runner cargo nextest --version | + run_as_builder cargo nextest --version | grep -Fq "cargo-nextest ${CARGO_NEXTEST_VERSION}" || die "cargo-nextest ${CARGO_NEXTEST_VERSION} is not installed" - installed_targets=$(run_as_runner env RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ + installed_targets=$(run_as_builder env RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" \ rustup target list --installed) - for target in \ - x86_64-unknown-none \ - x86_64-unknown-linux-musl \ - x86_64-unknown-uefi; do - printf '%s\n' "$installed_targets" | grep -Fxq "$target" || - die "Rust target ${target} is not installed" - done + with_rust_targets check_rust_targets +} + +# Checks that whoever builds NVX on this host can open the backend's device. +check_backend_access() { + device=/dev/${backend} + [ -c "$device" ] || die "${device} is not a character device" + if [ "$runner_only" = true ]; then + run_as_runner sh -c "test -r ${device} && test -w ${device}" || + die "${runner_service_account} cannot access ${device}" + else + [ -r "$device" ] && [ -w "$device" ] || + die "current session cannot access ${device}" + fi +} + +check_runner_host() { + check_platform + check_host_tools diff sccache systemctl + check_rust_tools run_as_runner sccache --version | grep -Fq "sccache ${SCCACHE_VERSION}" || die "sccache ${SCCACHE_VERSION} is not installed" @@ -806,9 +1019,7 @@ check_environment() { die "runner service account can modify trusted Cargo" run_as_runner test ! -w "$trusted_rustup_home" || die "runner service account can modify trusted Rustup state" - run_as_runner sh -c \ - "test -r /dev/${backend} && test -w /dev/${backend}" || - die "${runner_service_account} cannot access /dev/${backend}" + check_backend_access if run_as_runner sudo -n true >/dev/null 2>&1; then die "runner service account has passwordless sudo access" fi @@ -927,6 +1138,102 @@ check_environment() { fi } +# Checks that the current session can build guest artifacts with Docker and +# Buildx. +check_docker() { + require_command docker + docker version >/dev/null 2>&1 || + die "Docker is not available to the current session" + docker buildx version >/dev/null 2>&1 || die "Docker Buildx is not installed" +} + +check_development_host() { + check_platform + [ -f "${workspace}/scripts/nvx.py" ] || + die "NVX checkout not found at ${workspace}" + [ -f "${workspace}/openvmm/Cargo.toml" ] || + die "OpenVMM submodule is not initialized" + check_backend_access + check_host_tools + check_rust_tools + check_docker + + ( + cd "$workspace" + python3 scripts/nvx.py verify + ) +} + +check_build() { + ( + cd "$workspace" + RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN cargo build --release --locked --features cli \ + --manifest-path nvx-rs/Cargo.toml --target-dir nvx-rs/target + # Packaging and the adversarial executor take only an nvx-rs whose + # provenance binds it to the checkout's sources. + RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN NVX_RS=nvx-rs/target/release/nvx-rs \ + python3 scripts/nvx.py record-nvx-rs-provenance + nvx-rs/target/release/nvx-rs run --hypervisor "$backend" --dry-run + ) +} + +build_nvx() { + check_docker + ( + cd "$workspace" + RUSTUP_TOOLCHAIN=$RUST_TOOLCHAIN python3 scripts/nvx.py build --guest all + ) +} + +check_guest_bundle() { + bundle_path=$1 + [ -d "$bundle_path" ] || die "guest bundle not found: $bundle_path" + for name in vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ + initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ + initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ + ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ + initramfs-azurelinux.cpio.gz.packages.json REVISION SHA256SUMS; do + [ -f "${bundle_path}/${name}" ] || + die "missing guest bundle file: ${bundle_path}/${name}" + done + ( + cd "$bundle_path" + sha256sum --check SHA256SUMS + ) || die "guest bundle checksum validation failed" + bundle_revision=$(cat "${bundle_path}/REVISION") + workspace_revision=$(git -C "$workspace" rev-parse HEAD) + [ "$bundle_revision" = "$workspace_revision" ] || + die "guest bundle revision does not match the NVX checkout" +} + +create_guest_bundle() { + bundle_path=$1 + if [ -e "$bundle_path" ] && + find "$bundle_path" -mindepth 1 -print -quit | grep -q .; then + die "guest bundle directory is not empty: $bundle_path" + fi + mkdir -p "$bundle_path" + for name in vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ + initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ + initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ + ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ + initramfs-azurelinux.cpio.gz.packages.json; do + source_path="${workspace}/build/${name}" + [ -f "$source_path" ] || die "missing guest artifact: $source_path" + cp "$source_path" "$bundle_path" + done + git -C "$workspace" rev-parse HEAD >"${bundle_path}/REVISION" + ( + cd "$bundle_path" + sha256sum vmlinux vmlinux.config vmlinux.provenance.json initramfs.cpio.gz \ + initramfs.cpio.gz.packages.json initramfs.provenance.json initramfs-ubuntu.cpio.gz \ + initramfs-ubuntu.cpio.gz.packages.json ubuntu-distro.erofs \ + ubuntu-distro.erofs.manifest.json initramfs-azurelinux.cpio.gz \ + initramfs-azurelinux.cpio.gz.packages.json REVISION >SHA256SUMS + ) + check_guest_bundle "$bundle_path" +} + while [ "$#" -gt 0 ]; do case "$1" in --backend) @@ -934,18 +1241,51 @@ while [ "$#" -gt 0 ]; do backend=$2 shift 2 ;; + --check-only) + check_only=true + shift + ;; + --workspace) + [ "$#" -ge 2 ] || die "--workspace requires a path" + development_option=$1 + workspace=$2 + shift 2 + ;; + --guest-bundle) + [ "$#" -ge 2 ] || die "--guest-bundle requires a path" + development_option=$1 + guest_bundle=$2 + shift 2 + ;; + --bundle-only) + development_option=$1 + bundle_only=true + shift + ;; + --skip-build) + development_option=$1 + skip_build=true + shift + ;; + --runner-only) + runner_only=true + shift + ;; --runner-name) [ "$#" -ge 2 ] || die "--runner-name requires a value" + runner_option=$1 runner_name=$2 configure_runner=true shift 2 ;; --benchmark-runner) + runner_option=$1 benchmark_runner=true shift ;; --extra-runner-labels) [ "$#" -ge 2 ] || die "--extra-runner-labels requires a value" + runner_option=$1 extra_runner_labels=$2 [ -n "$extra_runner_labels" ] || die "--extra-runner-labels requires a nonempty value" @@ -953,23 +1293,21 @@ while [ "$#" -gt 0 ]; do ;; --repository-url) [ "$#" -ge 2 ] || die "--repository-url requires a value" + runner_option=$1 repository_url=$2 shift 2 ;; --runner-directory) [ "$#" -ge 2 ] || die "--runner-directory requires a value" + runner_option=$1 runner_directory=$2 runner_directory_is_default=false shift 2 ;; --runner-token-stdin) + runner_option=$1 + runner_token_stdin=true configure_runner=true - IFS= read -r runner_token || die "could not read runner token from stdin" - runner_token=$(printf '%s' "$runner_token" | tr -d '\r\n') - shift - ;; - --check-only) - check_only=true shift ;; -h | --help) @@ -982,51 +1320,87 @@ while [ "$#" -gt 0 ]; do esac done -case "$backend" in - kvm | mshv) ;; - *) die "--backend must be kvm or mshv" ;; -esac -read_tool_versions -report_invariant_tsc /proc/cpuinfo -[ "$(id -u)" -ne 0 ] || die "run this script as the SSH administrator, not root" -require_command sudo -sudo -n true || die "passwordless sudo is required" -if [ "$configure_runner" = true ]; then - [ -n "$runner_name" ] || die "--runner-name is required to configure a runner" - validate_runner_name "$runner_name" +validate_options +# Only a valid runner invocation consumes the token, so a misplaced option +# never waits for one. +if [ "$runner_token_stdin" = true ]; then + IFS= read -r runner_token || die "could not read runner token from stdin" + runner_token=$(printf '%s' "$runner_token" | tr -d '\r\n') fi -[ "$benchmark_runner" = false ] || [ -n "$runner_name" ] || - die "--benchmark-runner requires --runner-name" -runner_cargo_home=${runner_directory}/_work/_temp/cargo-home -runner_sccache_dir=${runner_directory}/_work/_sccache -runner_labels_file=${runner_directory}/.nvx-labels -expected_runner_labels=linux,${backend},virtual-machine -# CI's platform jobs use a dedicated benchmark runner. Every other runner uses -# the complementary general role so unrelated jobs cannot occupy that runner. -if [ "$benchmark_runner" = true ]; then - expected_runner_labels=${expected_runner_labels},benchmark -else - expected_runner_labels=${expected_runner_labels},general +read_tool_versions + +if [ "$runner_only" = true ]; then + report_invariant_tsc /proc/cpuinfo + [ "$(id -u)" -ne 0 ] || die "run this script as the SSH administrator, not root" + require_command sudo + sudo -n true || die "passwordless sudo is required" + legacy_runner_directory=${HOME}/actions-runner + runner_cargo_home=${runner_directory}/_work/_temp/cargo-home + runner_sccache_dir=${runner_directory}/_work/_sccache + runner_labels_file=${runner_directory}/.nvx-labels + + if [ "$check_only" = false ]; then + install_packages + configure_runner_account + install_rust_tools + restrict_docker_access + configure_backend_access + if [ "$configure_runner" = true ]; then + install_runner + fi + elif ! getent passwd "$runner_service_account" >/dev/null 2>&1; then + die "runner service account is not configured: ${runner_service_account}" + fi + + export PATH="$runner_service_path" + export CARGO_HOME="$runner_cargo_home" + export RUSTUP_HOME="$trusted_rustup_home" + check_runner_host + printf 'NVX_RUNNER_SETUP_COMPLETE=1\n' + exit 0 fi -if [ -n "$extra_runner_labels" ]; then - append_runner_labels "$extra_runner_labels" + +resolved_workspace=$(CDPATH='' cd -- "$workspace" 2>/dev/null && pwd) || + die "workspace directory not found: $workspace" +workspace=$resolved_workspace + +if [ "$bundle_only" = true ]; then + create_guest_bundle "$guest_bundle" + printf 'NVX_GUEST_BUNDLE=%s\n' "$guest_bundle" + exit 0 fi -if [ "$check_only" = false ]; then - install_packages - configure_runner_account - install_rust_tools - restrict_docker_access - configure_backend_access - if [ "$configure_runner" = true ]; then - install_runner +if [ "$check_only" = true ]; then + check_development_host + if [ "$skip_build" = false ]; then + check_build fi -elif ! getent passwd "$runner_service_account" >/dev/null 2>&1; then - die "runner service account is not configured: ${runner_service_account}" + if [ -n "$guest_bundle" ]; then + check_guest_bundle "$guest_bundle" + fi + printf 'NVX_SETUP_CHECK=ok\n' + exit 0 fi -export PATH="$runner_service_path" -export CARGO_HOME="$runner_cargo_home" -export RUSTUP_HOME="$trusted_rustup_home" -check_environment -printf 'NVX_RUNNER_SETUP_COMPLETE=1\n' +if [ "$(id -u)" -ne 0 ]; then + require_command sudo +fi +install_packages +install_rust_tools +configure_docker_access +configure_backend_access +if [ "$reconnect_required" = true ]; then + printf '%s\n' \ + 'NVX_SETUP_RECONNECT_REQUIRED=1' \ + "Reconnect so the docker and ${backend} group memberships apply, then rerun this script." + exit 20 +fi +if [ "$skip_build" = false ]; then + build_nvx + check_build +fi +if [ -n "$guest_bundle" ]; then + create_guest_bundle "$guest_bundle" +fi +check_development_host +printf 'NVX_SETUP_COMPLETE=1\n' diff --git a/scripts/setup/tool-versions.conf b/scripts/setup/tool-versions.conf index e94695401..d5141fa04 100644 --- a/scripts/setup/tool-versions.conf +++ b/scripts/setup/tool-versions.conf @@ -1,9 +1,8 @@ # Canonical versions, artifacts, and SHA-256 checksums of the tools that the -# NVX setup scripts install. setup-linux-runner.sh, setup-linux-mshv.sh, -# setup-windows-whp.ps1, and the Specula runner setup read this file as data -# and never source or evaluate it, so none of them repeats its pins. Stage it -# next to a setup script that runs outside a checkout; README.md describes -# how to upgrade a tool. +# NVX setup scripts install. setup-linux.sh, setup-windows-whp.ps1, and the +# Specula runner setup read this file as data and never source or evaluate +# it, so none of them repeats its pins. Stage it next to a setup script that +# runs outside a checkout; README.md describes how to upgrade a tool. # # Each line is blank, a comment that starts with #, or KEY=VALUE without # spaces. TOOL.version and TOOL.toolchain are exact MAJOR.MINOR.PATCH diff --git a/scripts/test_nightly_ci.py b/scripts/test_nightly_ci.py index e81f94b9b..dc80d2451 100644 --- a/scripts/test_nightly_ci.py +++ b/scripts/test_nightly_ci.py @@ -488,7 +488,7 @@ def test_setup_step_failures_are_infrastructure_failures(self): failure = self.classify( [ "Rust 1.0.0, which rust-toolchain.toml pins, is not " - "installed; rerun scripts/setup/setup-linux-runner.sh", + "installed; rerun scripts/setup/setup-linux.sh --runner-only", "Process completed with exit code 1.", ], steps=(("Set up job", "success"), (step, "failure")), diff --git a/scripts/test_nvx_tools.py b/scripts/test_nvx_tools.py index 2ec35e074..04e6b6737 100644 --- a/scripts/test_nvx_tools.py +++ b/scripts/test_nvx_tools.py @@ -8700,7 +8700,7 @@ def job_inputs( if name == "validate-runner": inputs |= { "rust-toolchain.toml", - "scripts/setup/setup-linux-runner.sh", + "scripts/setup/setup-linux.sh", ".gitattributes", } commands: set[str] = set() @@ -12048,7 +12048,7 @@ def test_runner_setup_pins_and_validates_sccache(self): BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-windows-whp.ps1" ).read_text(encoding="utf-8") linux_setup = ( - BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" ).read_text(encoding="utf-8") manifest = ( BuildConstants.REPO_ROOT / "scripts" / "setup" / "tool-versions.conf" @@ -12088,9 +12088,9 @@ def test_runner_setup_pins_and_validates_sccache(self): def test_linux_unnamed_setup_defers_existing_runner_validation(self): linux_setup = ( - BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" ).read_text(encoding="utf-8") - environment_check = linux_setup.split("check_environment() {\n", 1)[1].split( + environment_check = linux_setup.split("check_runner_host() {\n", 1)[1].split( "\n}\n", 1 )[0] runner_checks = environment_check.split( @@ -12146,20 +12146,17 @@ def test_linux_unnamed_setup_defers_existing_runner_validation(self): self.assertIn("work-checked", result.stdout) def test_linux_setup_installs_openvmm_perl_modules(self): - setup_directory = BuildConstants.REPO_ROOT / "scripts" / "setup" - configurations = ( - (setup_directory / "setup-linux-runner.sh").read_text(encoding="utf-8"), - (setup_directory / "setup-linux-mshv.sh").read_text(encoding="utf-8"), - ) - - for configuration in configurations: - for package in ( - "perl-FindBin", - "perl-IPC-Cmd", - "perl-Time-Piece", - "perl-lib", - ): - self.assertIn(package, configuration) + configuration = ( + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" + ).read_text(encoding="utf-8") + + for package in ( + "perl-FindBin", + "perl-IPC-Cmd", + "perl-Time-Piece", + "perl-lib", + ): + self.assertIn(package, configuration) def test_windows_runner_requires_inbox_pcat_firmware(self): windows_setup = ( @@ -12191,7 +12188,7 @@ def test_linux_runners_report_the_invariant_tsc_as_evidence(self): / "action.yml" ).read_text(encoding="utf-8") linux_setup = ( - BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" ).read_text(encoding="utf-8") # Neither CI nor provisioning rejects a host without an invariant @@ -12246,7 +12243,7 @@ def test_linux_runners_report_the_invariant_tsc_as_evidence(self): not invariant, ) self.assertNotIn("::error::", result.stdout + result.stderr) - with self.subTest(flags=flags, check="setup-linux-runner"): + with self.subTest(flags=flags, check="setup-linux"): result = subprocess.run( [ "sh", @@ -12651,7 +12648,7 @@ def test_runners_qualify_their_host_time_before_every_job(self): def test_runner_setups_install_backend_native_openvmm_targets(self): linux_setup = ( - BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" ).read_text(encoding="utf-8") windows_setup = ( BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-windows-whp.ps1" @@ -12677,10 +12674,11 @@ def test_linux_runner_setup_selects_exactly_one_runner_role(self): # Provisioning replaces a runner's registration, so the benchmark # runner keeps the role that CI requires only when setup registers it. linux_setup = ( - BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux-runner.sh" + BuildConstants.REPO_ROOT / "scripts" / "setup" / "setup-linux.sh" ).read_text(encoding="utf-8") self.assertIn( " --benchmark-runner)\n" + " runner_option=$1\n" " benchmark_runner=true\n" " shift\n" " ;;\n", @@ -12694,21 +12692,40 @@ def test_linux_runner_setup_selects_exactly_one_runner_role(self): ' die "GitHub Actions runner labels are not validated"', linux_setup, ) - start = linux_setup.index('\n[ "$benchmark_runner" = false ]') - end = linux_setup.index("\nfi\n", start) + len("\nfi\n") - labels = linux_setup[start:end] + + def function(name: str) -> str: + body = linux_setup.split(f"\n{name}() {{\n", 1)[1].split("\n}\n", 1)[0] + return f"{name}() {{\n{body}\n}}\n" + + self.assertIn( + "\n expected_runner_labels=linux,${backend},virtual-machine\n", + function("set_expected_runner_labels"), + ) self.assertIn( - "\nexpected_runner_labels=linux,${backend},virtual-machine\n", labels + "\n set_expected_runner_labels\n", function("validate_options") ) if os.name != "posix": return harness = ( "set -eu\n" - "die() { printf 'error: %s\\n' \"$*\" >&2; exit 1; }\n" - "backend=$1\nrunner_name=$2\nbenchmark_runner=$3\n" - "runner_directory=/opt/nvx-runner\n" - f"{labels}" + + "".join( + function(name) + for name in ( + "die", + "validate_runner_name", + "append_runner_labels", + "set_expected_runner_labels", + "validate_options", + ) + ) + + "backend=$1\nrunner_name=$2\nbenchmark_runner=$3\n" + "configure_runner=false\n" + '[ -z "$runner_name" ] || configure_runner=true\n' + "runner_only=true\ncheck_only=false\nbundle_only=false\n" + "guest_bundle=\nextra_runner_labels=\n" + "development_option=\nrunner_option=\n" + "validate_options\n" 'printf "%s\\n" "$expected_runner_labels"\n' ) for backend, runner_name, benchmark_runner, expected in ( @@ -12831,8 +12848,7 @@ def test_repository_pins_one_exact_rust_toolchain(self): [toolchain], ) for path, reader in ( - (setup / "setup-linux-runner.sh", "\n RUST_TOOLCHAIN=$tool_value\n"), - (setup / "setup-linux-mshv.sh", "\n RUST_TOOLCHAIN=$tool_value\n"), + (setup / "setup-linux.sh", "\n RUST_TOOLCHAIN=$tool_value\n"), ( setup / "setup-windows-whp.ps1", '\n$RustToolchain = Get-ToolManifestValue $ToolManifest "rust.toolchain"\n', diff --git a/scripts/test_setup_azure.py b/scripts/test_setup_azure.py index 70722272f..864576411 100644 --- a/scripts/test_setup_azure.py +++ b/scripts/test_setup_azure.py @@ -375,7 +375,7 @@ def test_linux_files_round_trip(self): set(names), { "config.json", - "setup-linux-runner.sh", + "setup-linux.sh", "tool-versions.conf", "bootstrap-runner.sh", "block-runner-metadata.sh", @@ -402,8 +402,12 @@ def test_linux_files_round_trip(self): self.assertIn(b"Requires=%s", bootstrap) self.assertIn(b"ExecStartPre=+%s", bootstrap) self.assertEqual(json.loads(names["config.json"]), config) + self.assertEqual(names["setup-linux.sh"], builder.source("setup-linux.sh")) + # Both setup passes configure a runner host, not a development host. + self.assertIn(b"setup_arguments=(\n --runner-only\n", bootstrap) self.assertEqual( - names["setup-linux-runner.sh"], builder.source("setup-linux-runner.sh") + bootstrap.count(b'sh "$setup_root/setup-linux.sh"'), + 2, ) self.assertEqual( names["tool-versions.conf"], builder.source("tool-versions.conf") @@ -462,7 +466,7 @@ def test_mshv_modes(self): set(manifest["guestFiles"]) - { "config.json", - "setup-linux-runner.sh", + "setup-linux.sh", "tool-versions.conf", "setup-data-disk.sh", "bootstrap-runner.sh", diff --git a/scripts/test_tool_versions.py b/scripts/test_tool_versions.py index 01db11bda..f71945f0b 100644 --- a/scripts/test_tool_versions.py +++ b/scripts/test_tool_versions.py @@ -7,8 +7,9 @@ accept, reject, and resolve manifests exactly as the reference reader below does, that each setup script reads the manifest before it inspects or changes the host, that no setup lets rustup update itself, and that no script, -workflow, or guide hard-codes a pin. Runner-label selection and fresh-install -directory checks are exercised without provisioning a host. +workflow, or guide hard-codes a pin. Runner-label selection, the options that +each mode of the Linux setup accepts, and fresh-install directory checks are +exercised without provisioning a host. """ from __future__ import annotations @@ -25,8 +26,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1] SETUP = REPO_ROOT / "scripts" / "setup" MANIFEST = SETUP / "tool-versions.conf" -LINUX_RUNNER = SETUP / "setup-linux-runner.sh" -LINUX_MSHV = SETUP / "setup-linux-mshv.sh" +LINUX = SETUP / "setup-linux.sh" WINDOWS = SETUP / "setup-windows-whp.ps1" SPECULA = REPO_ROOT / ".github" / "specula" / "setup-runner.sh" COPILOT_SETUP = REPO_ROOT / ".github" / "workflows" / "copilot-setup-steps.yml" @@ -567,20 +567,11 @@ def _assert_reader_output(self, output: str, native_paths: bool) -> None: ) index += 1 - def test_linux_setups_share_one_manifest_reader(self): - runner = LINUX_RUNNER.read_text(encoding="utf-8") - mshv = LINUX_MSHV.read_text(encoding="utf-8") - for name in _SHELL_READER_FUNCTIONS: - with self.subTest(function=name): - self.assertEqual( - _shell_function(runner, name), _shell_function(mshv, name) - ) - def test_shell_reader_matches_the_reference_reader(self): shells = _posix_shells() if not shells: self.skipTest("a POSIX shell is unavailable") - source = LINUX_RUNNER.read_text(encoding="utf-8") + source = LINUX.read_text(encoding="utf-8") functions = "".join( _shell_function(source, name) for name in _SHELL_READER_FUNCTIONS ) @@ -620,7 +611,7 @@ def test_powershell_reader_matches_the_reference_reader(self): self.assertEqual(result.returncode, 0, result.stderr) self._assert_reader_output(result.stdout, native_paths=True) - def test_linux_setups_pin_their_tools_to_the_manifest(self): + def test_linux_setup_pins_its_tools_to_the_manifest(self): shells = _posix_shells() if not shells: self.skipTest("a POSIX shell is unavailable") @@ -639,29 +630,32 @@ def test_linux_setups_pin_their_tools_to_the_manifest(self): pins[f"{prefix}_VERSION"] = artifact.version pins[f"{prefix}_ARTIFACT"] = artifact.name pins[f"{prefix}_SHA256"] = artifact.sha256 - mshv_pins = { + development_pins = { name: value for name, value in pins.items() if not name.startswith(("SCCACHE_", "RUNNER_")) } incomplete = _with_line(text, "sccache.artifacts.linux-x86_64.sha256", None) - for script, expected in ((LINUX_RUNNER, pins), (LINUX_MSHV, mshv_pins)): - source = script.read_text(encoding="utf-8") - # The scripts install the x86_64 Linux builds of each tool. - self.assertIn("\ntool_platform=linux-x86_64\n", source) - self.assertIn( - "\ntool_manifest=${script_directory}/tool-versions.conf\n", source - ) - functions = "".join( - _shell_function(source, name) - for name in (*_SHELL_READER_FUNCTIONS, "read_tool_versions") - ) + source = LINUX.read_text(encoding="utf-8") + # The script installs the x86_64 Linux builds of each tool. + self.assertIn("\ntool_platform=linux-x86_64\n", source) + self.assertIn( + "\ntool_manifest=${script_directory}/tool-versions.conf\n", source + ) + functions = "".join( + _shell_function(source, name) + for name in (*_SHELL_READER_FUNCTIONS, "read_tool_versions") + ) + for mode, runner_only, expected in ( + ("runner", "true", pins), + ("development", "false", development_pins), + ): report = "".join( f"printf '%s=%s\\n' {name} \"${name}\"\n" for name in expected ) for case, manifest in (("canonical", text), ("incomplete", incomplete)): - with self.subTest(script=script.name, case=case): - path = _write_case(self.root / script.stem / case, manifest) + with self.subTest(mode=mode, case=case): + path = _write_case(self.root / mode / case, manifest) result = subprocess.run( [ *shells[0], @@ -669,6 +663,7 @@ def test_linux_setups_pin_their_tools_to_the_manifest(self): "set -eu\n" f"{functions}tool_manifest=$1\n" "tool_platform=linux-x86_64\n" + f"runner_only={runner_only}\n" f"read_tool_versions\n{report}", "pins", path.as_posix(), @@ -678,9 +673,9 @@ def test_linux_setups_pin_their_tools_to_the_manifest(self): timeout=60, check=False, ) - # Only the runner installs sccache, so only it needs the + # Only runner mode installs sccache, so only it needs the # sccache checksum. - if case == "incomplete" and script == LINUX_RUNNER: + if case == "incomplete" and mode == "runner": self.assertEqual(result.returncode, 1) self.assertEqual( _error(result.stderr), @@ -693,12 +688,52 @@ def test_linux_setups_pin_their_tools_to_the_manifest(self): [f"{name}={value}" for name, value in expected.items()], ) - def test_linux_setups_install_the_pinned_musl_target(self): - for script in (LINUX_RUNNER, LINUX_MSHV): - with self.subTest(script=script.name): - source = script.read_text(encoding="utf-8") - self.assertIn('target add --toolchain "$RUST_TOOLCHAIN"', source) - self.assertIn("x86_64-unknown-linux-musl", source) + def test_linux_setup_runs_cargo_as_root_only_to_install(self): + # cargo install starts its configuration discovery at CARGO_HOME, but + # other commands read the current directory's configuration, such as + # aliases, which another user may control. + source = LINUX.read_text(encoding="utf-8") + invocations = re.findall(r'run_rust_tool "\$\{rust_bin\}cargo"[^\n]*', source) + self.assertTrue(invocations) + for invocation in invocations: + with self.subTest(invocation=invocation): + self.assertIn(' +"$RUST_TOOLCHAIN" install ', invocation) + + def test_linux_setup_installs_the_pinned_musl_target_in_both_modes(self): + source = LINUX.read_text(encoding="utf-8") + # Installation and check mode take each mode's targets from one place. + self.assertIn( + 'with_rust_targets run_rust_tool "${rust_bin}rustup" target add \\\n' + ' --toolchain "$RUST_TOOLCHAIN"\n', + source, + ) + self.assertIn(" with_rust_targets check_rust_targets\n", source) + shells = _posix_shells() + if not shells: + self.skipTest("a POSIX shell is unavailable") + function = _shell_function(source, "with_rust_targets") + for runner_only, expected in ( + ( + "true", + "x86_64-unknown-none x86_64-unknown-linux-musl x86_64-unknown-uefi", + ), + ("false", "x86_64-unknown-linux-musl"), + ): + with self.subTest(runner_only=runner_only): + result = subprocess.run( + [ + *shells[0], + "-c", + f"set -eu\n{function}runner_only={runner_only}\n" + "with_rust_targets echo\n", + ], + capture_output=True, + text=True, + timeout=60, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout.strip(), expected) def test_windows_setup_pins_its_tools_to_the_manifest(self): powershells = _powershells() @@ -778,16 +813,10 @@ def test_setups_never_let_rustup_update_itself(self): # rustup exits, so the setup's next rustup command can find it missing. for script, install, invocation in ( ( - LINUX_RUNNER, - " toolchain install ", - '"$rustup" toolchain install "$RUST_TOOLCHAIN" --profile minimal \\\n' - " --no-self-update\n", - ), - ( - LINUX_MSHV, + LINUX, " toolchain install ", - 'rustup toolchain install "$RUST_TOOLCHAIN" --profile minimal \\\n' - " --no-self-update\n", + 'run_rust_tool "${rust_bin}rustup" toolchain install "$RUST_TOOLCHAIN" \\\n' + " --profile minimal --no-self-update\n", ), ( SPECULA, @@ -808,23 +837,23 @@ def test_setups_never_let_rustup_update_itself(self): self.assertIn(invocation, source) def test_setups_read_the_manifest_before_they_inspect_the_host(self): - runner = LINUX_RUNNER.read_text(encoding="utf-8") - self.assertEqual(runner.count("\nread_tool_versions\n"), 1) - for later in ( - "\nreport_invariant_tsc /proc/cpuinfo\n", - '\nsudo -n true || die "passwordless sudo is required"\n', - "\n install_packages\n", - ): - self.assertLess(runner.index("\nread_tool_versions\n"), runner.index(later)) - mshv = LINUX_MSHV.read_text(encoding="utf-8") - self.assertEqual(mshv.count("\nread_tool_versions\n"), 1) + linux = LINUX.read_text(encoding="utf-8") + self.assertEqual(linux.count("\nread_tool_versions\n"), 1) + load = linux.index("\nread_tool_versions\n") + # Validating the options inspects nothing but the options. + self.assertLess(linux.index("\nvalidate_options\n"), load) for later in ( - "\nworkspace=$(CDPATH='' cd -- \"$workspace\" 2>/dev/null && pwd) ||\n", + # Runner mode. + "\n report_invariant_tsc /proc/cpuinfo\n", + '\n sudo -n true || die "passwordless sudo is required"\n', + "\n install_packages\n", + # Development-host mode. + "\nresolved_workspace=$(CDPATH='' cd -- \"$workspace\" 2>/dev/null && pwd) ||\n", '\nif [ "$bundle_only" = true ]; then\n', '\nif [ "$check_only" = true ]; then\n', "\ninstall_packages\n", ): - self.assertLess(mshv.index("\nread_tool_versions\n"), mshv.index(later)) + self.assertLess(load, linux.index(later)) windows = WINDOWS.read_text(encoding="utf-8") load = windows.index("\n$ToolManifest = Read-ToolManifest ") for later in ( @@ -854,22 +883,26 @@ def _staged_failures(self) -> list[tuple[str, str | None, tuple[str, ...]]]: self.assertEqual(len(failures), 3) return failures - def test_staged_linux_setups_reject_a_bad_manifest_before_anything_else(self): + def test_staged_linux_setup_rejects_a_bad_manifest_before_anything_else(self): shells = _posix_shells() if not shells: self.skipTest("a POSIX shell is unavailable") # A missing workspace must not hide a bad manifest. missing = (self.root / "no-workspace").as_posix() - for script, arguments in ( - (LINUX_RUNNER, ("--backend", "kvm", "--check-only")), - (LINUX_MSHV, ("--check-only", "--skip-build", "--workspace", missing)), + for mode, arguments in ( + ("runner", ("--runner-only", "--backend", "kvm", "--check-only")), + ( + "development", + ("--backend", "mshv", "--check-only", "--skip-build", "--workspace") + + (missing,), + ), ): for case, text, error in self._staged_failures(): - with self.subTest(script=script.name, case=case): - directory = self.root / "staged" / script.stem / case + with self.subTest(mode=mode, case=case): + directory = self.root / "staged" / LINUX.stem / mode / case _write_case(directory, text) - staged = directory / script.name - shutil.copyfile(script, staged) + staged = directory / LINUX.name + shutil.copyfile(LINUX, staged) result = subprocess.run( [*shells[0], staged.as_posix(), *arguments], stdin=subprocess.DEVNULL, @@ -883,6 +916,119 @@ def test_staged_linux_setups_reject_a_bad_manifest_before_anything_else(self): self.assertEqual(len(result.stderr.splitlines()), 1, result.stderr) self.assertEqual(_error(result.stderr), error) + def test_staged_linux_setup_rejects_options_of_the_other_mode(self): + shells = _posix_shells() + if not shells: + self.skipTest("a POSIX shell is unavailable") + # Without a manifest next to the script, options that the script + # accepts stop it at the manifest, which it reads right after them. + directory = self.root / "options" + directory.mkdir() + staged = directory / LINUX.name + shutil.copyfile(LINUX, staged) + runner_options = ( + ("--runner-name", "nvx-1"), + ("--benchmark-runner",), + ("--extra-runner-labels", "pool"), + ("--repository-url", "https://github.com/microsoft/nvx"), + ("--runner-directory", "/opt/nvx-runner"), + ("--runner-token-stdin",), + ) + development_options = ( + ("--workspace", "."), + ("--guest-bundle", "bundle"), + ("--bundle-only",), + ("--skip-build",), + ) + cases: list[tuple[tuple[str, ...], str | None]] = [ + ((), "--backend must be kvm or mshv"), + (("--backend", "whp"), "--backend must be kvm or mshv"), + (("--runner-only",), "--backend must be kvm or mshv"), + *( + (("--backend", "kvm", *option), f"{option[0]} requires --runner-only") + for option in runner_options + ), + *( + ( + ("--runner-only", "--backend", "mshv", *option), + f"{option[0]} cannot be combined with --runner-only", + ) + for option in development_options + ), + ( + ("--backend", "mshv", "--bundle-only"), + "--bundle-only requires --guest-bundle", + ), + ( + ("--backend", "mshv", "--bundle-only", "--guest-bundle", "bundle") + + ("--check-only",), + "--bundle-only and --check-only cannot be combined", + ), + ( + ("--runner-only", "--backend", "kvm", "--runner-token-stdin"), + "--runner-name is required to configure a runner", + ), + ( + ("--runner-only", "--backend", "kvm", "--runner-name", "bad name"), + "invalid runner name: bad name", + ), + ( + ("--runner-only", "--backend", "kvm", "--benchmark-runner"), + "--benchmark-runner requires --runner-name", + ), + ( + ("--runner-only", "--backend", "kvm", "--runner-name", "nvx-1") + + ("--extra-runner-labels", "general"), + "runner role label must be selected with --benchmark-runner: general", + ), + # The token is read only after the options are valid. + ( + ("--runner-only", "--backend", "kvm", "--runner-name", "nvx-1") + + ("--runner-token-stdin",), + "could not read runner token from stdin", + ), + (("--backend", "kvm"), None), + ( + ( + "--backend", + "mshv", + "--check-only", + "--skip-build", + "--workspace", + ".", + ), + None, + ), + (("--backend", "mshv", "--bundle-only", "--guest-bundle", "bundle"), None), + (("--runner-only", "--backend", "mshv", "--check-only"), None), + ( + ("--runner-only", "--backend", "kvm", "--runner-name", "nvx-1") + + ("--benchmark-runner", "--extra-runner-labels", "pool") + + ("--runner-token-stdin",), + None, + ), + ] + for arguments, error in cases: + with self.subTest(arguments=arguments): + # Only a valid invocation that reads a token gets one, so an + # invalid one that read standard input would report its end. + token = error is None and "--runner-token-stdin" in arguments + result = subprocess.run( + [*shells[0], staged.as_posix(), *arguments], + input="token\n" if token else "", + capture_output=True, + text=True, + timeout=60, + check=False, + ) + self.assertEqual(result.returncode, 1, result.stderr) + self.assertEqual(result.stdout, "") + self.assertEqual(len(result.stderr.splitlines()), 1, result.stderr) + if error is None: + self.assertEqual(_error(result.stderr), ("missing",)) + else: + self.assertEqual(result.stderr, f"error: {error}\n") + def test_staged_windows_setup_rejects_a_bad_manifest_before_anything_else(self): powershells = _powershells() if not powershells: @@ -926,7 +1072,7 @@ def test_staged_windows_setup_rejects_a_bad_manifest_before_anything_else(self): class ToolManifestConsumerTests(unittest.TestCase): def test_consumers_read_the_manifest(self): - for path in (LINUX_RUNNER, LINUX_MSHV, WINDOWS, SPECULA, COPILOT_SETUP): + for path in (LINUX, WINDOWS, SPECULA, COPILOT_SETUP): with self.subTest(path=path.name): self.assertIn("tool-versions.conf", path.read_text(encoding="utf-8")) @@ -1091,20 +1237,79 @@ def test_copilot_setup_installs_the_manifest_nextest(self): ) +class LinuxSetupTests(unittest.TestCase): + def test_linux_setup_installs_the_packages_of_each_mode(self): + shells = _posix_shells() + if os.name != "posix" or not shells: + self.skipTest("the harness needs a POSIX shell and executable stubs") + source = LINUX.read_text(encoding="utf-8") + harness = ( + "set -eu\n" + + "".join( + _shell_function(source, name) for name in ("die", "install_packages") + ) + + 'run_as_root() { printf "%s\\n" "$*"; }\n' + + "runner_only=$1\ninstall_packages\n" + ) + installers = { + "apt-get": "env DEBIAN_FRONTEND=noninteractive apt-get install -y ", + "tdnf": "tdnf install -y ", + "dnf": "dnf install -y ", + } + docker = { + "apt-get": ["docker-buildx", "docker.io"], + "tdnf": ["docker-buildx", "docker-cli", "moby-engine"], + "dnf": ["docker-buildx", "moby-engine"], + } + # Azure Linux has both tdnf and dnf, and the script uses tdnf there. + for managers in (("apt-get",), ("tdnf", "dnf"), ("dnf",)): + manager = managers[0] + base: set[tuple[str, ...]] = set() + for runner_only in ("false", "true"): + with ( + self.subTest(manager=manager, runner_only=runner_only), + tempfile.TemporaryDirectory() as temporary, + ): + for name in managers: + stub = Path(temporary) / name + stub.write_text("#!/bin/sh\n", encoding="utf-8") + stub.chmod(0o755) + result = subprocess.run( + [*shells[0], "-c", harness, "packages", runner_only], + env={"PATH": temporary}, + capture_output=True, + text=True, + timeout=30, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + *before, install = result.stdout.splitlines() + self.assertEqual( + before, ["apt-get update"] if manager == "apt-get" else [] + ) + self.assertTrue(install.startswith(installers[manager])) + packages = install.removeprefix(installers[manager]).split() + extra: list[str] = [] + if runner_only == "false": + extra = docker[manager] + elif manager == "tdnf": + extra = ["icu", "lttng-ust"] + self.assertEqual(packages[len(packages) - len(extra) :], extra) + base.add(tuple(packages[: len(packages) - len(extra)])) + # Both modes install the same build dependencies. + self.assertEqual(len(base), 1, base) + + class RunnerSetupTests(unittest.TestCase): def test_linux_extra_labels_keep_required_labels(self): shells = _posix_shells() if not shells: self.skipTest("a POSIX shell is unavailable") - source = LINUX_RUNNER.read_text(encoding="utf-8") + source = LINUX.read_text(encoding="utf-8") functions = "".join( - _shell_function(source, name) for name in ("die", "append_runner_labels") - ) - start = source.index( - "\nexpected_runner_labels=linux,${backend},virtual-machine\n" + _shell_function(source, name) + for name in ("die", "append_runner_labels", "set_expected_runner_labels") ) - end = source.index('\nif [ "$check_only" = false ]; then', start) - labels = source[start:end] self.assertIn("--extra-runner-labels)", source) for backend in ("kvm", "mshv"): for benchmark in ("false", "true"): @@ -1124,7 +1329,7 @@ def test_linux_extra_labels_keep_required_labels(self): script = ( f"set -eu\n{functions}" "backend=$1\nbenchmark_runner=$2\nextra_runner_labels=$3\n" - f"{labels}" + "set_expected_runner_labels\n" 'printf "%s\\n" "$expected_runner_labels"\n' ) result = subprocess.run( @@ -1149,7 +1354,7 @@ def test_linux_extra_labels_reject_duplicates_and_invalid_values(self): shells = _posix_shells() if not shells: self.skipTest("a POSIX shell is unavailable") - source = LINUX_RUNNER.read_text(encoding="utf-8") + source = LINUX.read_text(encoding="utf-8") script = ( "set -eu\n" + "".join( From 105a4887b8fde19473fcdb19a7472c087190b389 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 14:25:59 -0700 Subject: [PATCH 2/3] setup: keep the Docker that a Linux development host has Development-host mode installed the distribution's Docker packages on every host, as setup-linux-mshv.sh did. On Ubuntu, docker.io conflicts with Docker's own packages, so apt removes docker-ce, docker-ce-cli, and containerd.io to install it: a simulated `apt-get install docker-buildx docker.io` on an Ubuntu 26.04 host that runs docker-ce plans exactly those removals. Azure Linux has no docker-ce, so the Linux/MSHV development hosts never met this, but development-host mode now prepares Linux/KVM hosts too, which commonly run Docker's own packages. Install Docker with Buildx only on a development host that has no docker command, and keep any Docker installation that the host has. The mode still enables the Docker service and grants the docker group, and its check still requires Docker and Buildx, so a kept installation without Buildx stops setup with "Docker Buildx is not installed". Runner mode installs no Docker, as before. The package test also runs each mode on a host that has Docker, and the usage text and the setup README describe the condition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/setup/README.md | 20 ++++++---- scripts/setup/setup-linux.sh | 32 ++++++++++------ scripts/test_tool_versions.py | 70 +++++++++++++++++++---------------- 3 files changed, 71 insertions(+), 51 deletions(-) diff --git a/scripts/setup/README.md b/scripts/setup/README.md index d1917b74a..b9c389fec 100644 --- a/scripts/setup/README.md +++ b/scripts/setup/README.md @@ -38,7 +38,7 @@ load `mshv_root` when the device is missing. They differ in everything else: | Runs as | Root, or a user whose `sudo` may prompt | The SSH administrator with passwordless `sudo`, never root | | NVX checkout | Required: the one that contains the script, or `--workspace PATH` | Not used | | Rust tools | The current user's, with the musl target of the guest binaries | Root-owned and read-only to jobs under `/opt/nvx`, with OpenVMM's test targets and the pinned sccache | -| Docker | Installs Docker with Buildx and adds the user to `docker` | Keeps `nvx-runner` out of `docker` | +| Docker | Installs Docker with Buildx unless the host has a Docker CLI, and adds the user to `docker` | Keeps `nvx-runner` out of `docker` | | Backend device | Adds the user to the backend's group | Makes the backend's group the only supplementary group of `nvx-runner` | | Completes with | A build of NVX and its guests, unless `--skip-build`, and `NVX_SETUP_COMPLETE=1` | With `--runner-name`, a registered runner service, and `NVX_RUNNER_SETUP_COMPLETE=1` | | `--check-only` prints | `NVX_SETUP_CHECK=ok` | `NVX_RUNNER_SETUP_COMPLETE=1` | @@ -60,10 +60,11 @@ invocations: | `setup-linux-runner.sh --backend BACKEND [OPTIONS]` | `setup-linux.sh --runner-only --backend BACKEND [OPTIONS]` | Their options, output markers, and exit statuses are otherwise unchanged. -Development-host mode also accepts `--backend kvm` now, and its check mode -also requires the musl target that it installs. A runner that the former -script configured needs no migration: runner-only mode manages the same -account, toolchain, installation, and service. +Development-host mode also accepts `--backend kvm` now, keeps a Docker +installation that the host already has, and its check mode also requires the +musl target that it installs. A runner that the former script configured needs +no migration: runner-only mode manages the same account, toolchain, +installation, and service. ## GitHub Actions runners @@ -241,9 +242,12 @@ ssh HOST powershell.exe -NoProfile -ExecutionPolicy Bypass ` Development-host mode prepares a Linux/KVM or Linux/MSHV checkout for the user who runs the script. It installs native build dependencies, Docker with Buildx, and the Rust and cargo-nextest releases that the manifest pins, and configures -group-based access to Docker and to the backend's device. Its udev rule lets -only the backend's group open the device, so another user of the host needs -that group too, even where the device was open to everyone before. +group-based access to Docker and to the backend's device. A host that already +has a Docker CLI keeps its Docker, such as Docker's own `docker-ce`, which the +distribution's packages would replace; install Buildx for it if it lacks the +plugin. The udev rule lets only the backend's group open the device, so +another user of the host needs that group too, even where the device was open +to everyone before. Run the complete bootstrap from the checkout, naming the host's backend: diff --git a/scripts/setup/setup-linux.sh b/scripts/setup/setup-linux.sh index 33885d7c1..a236ff275 100644 --- a/scripts/setup/setup-linux.sh +++ b/scripts/setup/setup-linux.sh @@ -49,12 +49,13 @@ Usage: $0 --backend kvm|mshv [options] Bootstrap a Linux host for NVX. Development-host mode, the default, prepares an existing NVX checkout for the -current user: it installs the build dependencies, Docker with Buildx, and the -pinned Rust tools, grants the user Docker and backend device access, and builds -and validates NVX. Runner mode prepares a host for NVX GitHub Actions jobs and -needs no checkout: it installs the build dependencies and a read-only Rust -toolchain, and, with --runner-name, registers a runner whose jobs run as the -dedicated ${runner_service_account} account, without sudo or Docker access. +current user: it installs the build dependencies, the pinned Rust tools, and, +unless the host has Docker, Docker with Buildx, grants the user Docker and +backend device access, and builds and validates NVX. Runner mode prepares a +host for NVX GitHub Actions jobs and needs no checkout: it installs the build +dependencies and a read-only Rust toolchain, and, with --runner-name, +registers a runner whose jobs run as the dedicated ${runner_service_account} +account, without sudo or Docker access. Options: --backend BACKEND Virtualization backend: kvm or mshv @@ -708,17 +709,24 @@ configure_runner_service() { run_as_root systemctl start "$service_name" } +# Development hosts build guest artifacts with Docker and Buildx. The +# distribution's Docker packages can replace another Docker installation, such +# as Docker's own docker-ce, so a host that has a Docker CLI keeps its Docker. +needs_docker_packages() { + [ "$runner_only" = false ] && ! command -v docker >/dev/null 2>&1 +} + # Installs the build dependencies of both modes and each mode's own: Docker -# with Buildx, with which a development host builds guest artifacts, or, with -# tdnf, the Actions runner's dependencies, which installdependencies.sh does -# not install on Azure Linux. +# with Buildx on a development host that lacks Docker, or, with tdnf, the +# Actions runner's dependencies, which installdependencies.sh does not install +# on Azure Linux. install_packages() { if command -v apt-get >/dev/null 2>&1; then set -- bc binutils bison build-essential ca-certificates cmake cpio curl \ diffutils flex git gzip iproute2 iptables libarchive-tools libelf-dev \ libssl-dev make ninja-build patch perl pkg-config protobuf-compiler \ python3 rsync tar util-linux xz-utils zstd - if [ "$runner_only" = false ]; then + if needs_docker_packages; then set -- "$@" docker-buildx docker.io fi run_as_root apt-get update @@ -733,7 +741,7 @@ install_packages() { which xz zstd if [ "$runner_only" = true ]; then set -- "$@" icu lttng-ust - else + elif needs_docker_packages; then set -- "$@" docker-buildx docker-cli moby-engine fi run_as_root tdnf install -y "$@" @@ -744,7 +752,7 @@ install_packages() { ninja-build openssl openssl-devel patch perl pkgconf \ pkgconf-pkg-config protobuf-compiler python3 rsync shadow-utils tar \ util-linux which xz zstd - if [ "$runner_only" = false ]; then + if needs_docker_packages; then set -- "$@" docker-buildx moby-engine fi run_as_root dnf install -y "$@" diff --git a/scripts/test_tool_versions.py b/scripts/test_tool_versions.py index f71945f0b..df14aad71 100644 --- a/scripts/test_tool_versions.py +++ b/scripts/test_tool_versions.py @@ -1246,7 +1246,8 @@ def test_linux_setup_installs_the_packages_of_each_mode(self): harness = ( "set -eu\n" + "".join( - _shell_function(source, name) for name in ("die", "install_packages") + _shell_function(source, name) + for name in ("die", "needs_docker_packages", "install_packages") ) + 'run_as_root() { printf "%s\\n" "$*"; }\n' + "runner_only=$1\ninstall_packages\n" @@ -1266,36 +1267,43 @@ def test_linux_setup_installs_the_packages_of_each_mode(self): manager = managers[0] base: set[tuple[str, ...]] = set() for runner_only in ("false", "true"): - with ( - self.subTest(manager=manager, runner_only=runner_only), - tempfile.TemporaryDirectory() as temporary, - ): - for name in managers: - stub = Path(temporary) / name - stub.write_text("#!/bin/sh\n", encoding="utf-8") - stub.chmod(0o755) - result = subprocess.run( - [*shells[0], "-c", harness, "packages", runner_only], - env={"PATH": temporary}, - capture_output=True, - text=True, - timeout=30, - check=False, - ) - self.assertEqual(result.returncode, 0, result.stderr) - *before, install = result.stdout.splitlines() - self.assertEqual( - before, ["apt-get update"] if manager == "apt-get" else [] - ) - self.assertTrue(install.startswith(installers[manager])) - packages = install.removeprefix(installers[manager]).split() - extra: list[str] = [] - if runner_only == "false": - extra = docker[manager] - elif manager == "tdnf": - extra = ["icu", "lttng-ust"] - self.assertEqual(packages[len(packages) - len(extra) :], extra) - base.add(tuple(packages[: len(packages) - len(extra)])) + for has_docker in (False, True): + with ( + self.subTest( + manager=manager, + runner_only=runner_only, + has_docker=has_docker, + ), + tempfile.TemporaryDirectory() as temporary, + ): + commands = (*managers, *(("docker",) if has_docker else ())) + for name in commands: + stub = Path(temporary) / name + stub.write_text("#!/bin/sh\n", encoding="utf-8") + stub.chmod(0o755) + result = subprocess.run( + [*shells[0], "-c", harness, "packages", runner_only], + env={"PATH": temporary}, + capture_output=True, + text=True, + timeout=30, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + *before, install = result.stdout.splitlines() + self.assertEqual( + before, ["apt-get update"] if manager == "apt-get" else [] + ) + self.assertTrue(install.startswith(installers[manager])) + packages = install.removeprefix(installers[manager]).split() + extra: list[str] = [] + if runner_only == "false" and not has_docker: + # A host that has Docker keeps it. + extra = docker[manager] + elif runner_only == "true" and manager == "tdnf": + extra = ["icu", "lttng-ust"] + self.assertEqual(packages[len(packages) - len(extra) :], extra) + base.add(tuple(packages[: len(packages) - len(extra)])) # Both modes install the same build dependencies. self.assertEqual(len(base), 1, base) From 5d06cba0234fc1cb0c321381f897456ae9fea6ac Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Thu, 29 Oct 2026 15:21:56 -0700 Subject: [PATCH 3/3] setup: run the Linux runner setup's Python as root in isolated mode Runner mode runs five Python programs as root: three read the runner's .runner file, one sets its disableUpdate, and one writes svc.sh from the runner's template. They ran as `python3 -c` and `python3 -`, which put the current directory first on sys.path, so a json.py or pathlib.py there replaced the standard module. setup-linux-runner.sh did the same. When the administrator runs setup from a directory that another user can write, such as a checkout in a runner's _work directory, a job could plant such a module and run code as root: with a json.py that records its uid in the current directory, check mode of the former script ran it as uid 0 on a Linux/MSHV test host. Run them with -I, Python's isolated mode, which leaves the current directory, the user's site directory, and the PYTHON* variables out of the search. The same check mode no longer imports the planted module. A test requires -I for every Python that runs as root, as another requires root to run cargo only to install. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/setup/setup-linux.sh | 10 +++++----- scripts/test_tool_versions.py | 11 +++++++++++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/scripts/setup/setup-linux.sh b/scripts/setup/setup-linux.sh index a236ff275..df7b9d418 100644 --- a/scripts/setup/setup-linux.sh +++ b/scripts/setup/setup-linux.sh @@ -559,7 +559,7 @@ install_or_verify_runner_package() { refresh_runner_service_script() { [ -n "$removed_service_name" ] || return 0 - run_as_root python3 - "$runner_directory" "$removed_service_name" "$runner_name" <<'PY' + run_as_root python3 -I - "$runner_directory" "$removed_service_name" "$runner_name" <<'PY' import pathlib import sys @@ -602,7 +602,7 @@ configure_runner_account() { set_runner_disable_update() { validate_runner_state_paths - run_as_root python3 -c ' + run_as_root python3 -I -c ' import json import pathlib import sys @@ -908,7 +908,7 @@ install_runner() { configured_runner_name= if run_as_root test -f "${runner_directory}/.runner"; then - configured_runner_name=$(run_as_root python3 -c \ + configured_runner_name=$(run_as_root python3 -I -c \ 'import json, sys; print(json.load(open(sys.argv[1], encoding="utf-8-sig"))["agentName"])' \ "${runner_directory}/.runner") fi @@ -1047,13 +1047,13 @@ check_runner_host() { return 0 fi if [ "$configure_runner" = true ]; then - configured_runner_name=$(run_as_root python3 -c \ + configured_runner_name=$(run_as_root python3 -I -c \ 'import json, sys; print(json.load(open(sys.argv[1], encoding="utf-8-sig"))["agentName"])' \ "${runner_directory}/.runner") [ "$configured_runner_name" = "$runner_name" ] || die "configured runner is ${configured_runner_name}, expected ${runner_name}" fi - runner_disable_update=$(run_as_root python3 -c \ + runner_disable_update=$(run_as_root python3 -I -c \ 'import json, sys; print(json.load(open(sys.argv[1], encoding="utf-8-sig")).get("disableUpdate"))' \ "${runner_directory}/.runner") [ "$runner_disable_update" = True ] || diff --git a/scripts/test_tool_versions.py b/scripts/test_tool_versions.py index df14aad71..d7eca760f 100644 --- a/scripts/test_tool_versions.py +++ b/scripts/test_tool_versions.py @@ -699,6 +699,17 @@ def test_linux_setup_runs_cargo_as_root_only_to_install(self): with self.subTest(invocation=invocation): self.assertIn(' +"$RUST_TOOLCHAIN" install ', invocation) + def test_linux_setup_runs_python_as_root_in_isolated_mode(self): + # Without -I, Python searches the current directory first, so a + # module there that another user wrote, such as a json.py, would run + # as root. + source = LINUX.read_text(encoding="utf-8") + invocations = re.findall(r"run_as_root python3[^\n]*", source) + self.assertTrue(invocations) + for invocation in invocations: + with self.subTest(invocation=invocation): + self.assertTrue(invocation.startswith("run_as_root python3 -I ")) + def test_linux_setup_installs_the_pinned_musl_target_in_both_modes(self): source = LINUX.read_text(encoding="utf-8") # Installation and check mode take each mode's targets from one place.