From 32ab168fafaf3eba91cdae199df51c822aa1b5d5 Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 5 Oct 2026 20:49:50 -0400 Subject: [PATCH 1/2] docs(runner): record R7 privilege-shape observations from a VM node (RIG-3723) Ran R7's six items with controls on an isolated rootful k3s node in a nested-KVM VM. The run does not meet R7's real-hardware bar, so R7 stays open. S1 shows the spec as written does not boot: passt and virtiofsd also need a user-namespaced pod with unmasked /proc, AppArmor unconfined, a node sysctl, and a world-rw /dev/kvm, which defeats the kvm-gid grant. Section Privilege shape is unchanged pending that decision. Spec-impact: none Refs RIG-3723 Co-authored-by: Matt Wilkinson --- .../spike-findings.md | 147 +++++++++++++++++- 1 file changed, 143 insertions(+), 4 deletions(-) diff --git a/docs/designs/platform/compass-runner-containerization/spike-findings.md b/docs/designs/platform/compass-runner-containerization/spike-findings.md index 05aa371f3..41b9a65d3 100644 --- a/docs/designs/platform/compass-runner-containerization/spike-findings.md +++ b/docs/designs/platform/compass-runner-containerization/spike-findings.md @@ -1,10 +1,9 @@ # Privilege-shape spike findings Findings for R7 of -[runner containerization](design.md). **Not yet run** — this file is the -recorded shape of the verification so the questions and their negative controls -are fixed before anyone runs them, rather than being chosen after seeing -results. +[runner containerization](design.md). Run 2026-10-05; results are under +[Results](#results). The item plan below was fixed before the run and is kept +unchanged so the controls stay visibly chosen in advance. **This spike narrows a pod spec; it does not ask whether containerization works.** The composition already boots on Linux with `/dev/kvm` — the @@ -120,3 +119,143 @@ Each item records: what ran, the observed result, the control's result, and whether the contract changed. An item whose control was not run is reported as **not verified**, never as passing — a positive without its control is the failure mode this file exists to prevent. + +## Results + +**Status: R7 is not complete.** These are observations on a nested-KVM VM, not +the real-hardware run R7 requires, so no item below closes an R3 or R4 gate. +They do show that the opening hypothesis, that every answer costs at most a +grant the record already names, **did not hold**: S1 needs grants and a node +setting outside §Privilege shape, and one of them conflicts with the S3 grant. +The record must be reconciled, and every item re-run on the target node, before +R3 encodes the pod spec. + +### Environment + +- **Node:** an isolated single-node k3s in an Ubuntu 24.04 VM with nested KVM. + The S1, S2, and S3 controls show it enforces the device cgroup, seccomp, and + AppArmor. Version strings were not captured with the evidence. +- **Image:** the runner image built from compass `9a39795`, imported into the + node as `localhost/compass-runner:r7`. +- **Boot harness:** the repo's `TestNetOnlyBootSmoke` (passt only) and + `TestFullBoot` (passt + virtiofsd + cloud-hypervisor), built from the same + commit and run as pid 1 under the pod spec under test. +- **Device plugin:** `ghcr.io/squat/generic-device-plugin` advertising + `devic.es/kvm` for the existing `/dev/kvm`; it does not change the mode. +- **Node `/dev/kvm`:** `crw-rw---- 0:993` unless a run states otherwise; each + evidence block records the mode and sysctl it ran under. +- **Rootless kind was tried first and discarded.** hostPath control devices + opened there, so it did not enforce the device cgroup. That run's output was + not kept with this evidence: **not verified**. + +### Summary + +| Item | Observed on the VM | Implication for the record | +| --- | --- | --- | +| S1 | Boots only with AppArmor unconfined, a user-namespaced pod with unmasked `/proc`, a node sysctl, and node `/dev/kvm` at `0666` | §Privilege shape must widen; open decision | +| S2 | hostPath `open()` fails `EPERM`; privileged control opens | Supports the inference | +| S3 | Without the kvm gid, `open()` fails `EACCES`; with it, opens | Supports the grant, but it is inert under S1's user namespace | +| S4 | `RuntimeDefault` fails; the custom `Localhost` profile boots | Points to OQ-2 closing: ship the profile | +| S5 | Guest RAM is charged to the pod as `shmem` | Supports the model | +| S6 | Zero processes stranded after a pid 1 kill | Supports the model (pidfile reap not verified) | + +### S1 — the assembled spec + +**Result: the §Privilege shape as written does not boot.** Every launch fails +in the sandbox setup of passt or virtiofsd, before cloud-hypervisor starts. +Both tests PASS only with the full set below. Each row removes one grant from +that set, with the node sysctl at `0`: + +| Removed | Observed | +| --- | --- | +| `Localhost` seccomp (to `RuntimeDefault`) | passt: `Couldn't create user namespace: Operation not permitted` | +| `appArmorProfile: Unconfined` (to `RuntimeDefault`) | passt: `Failed to remount /: Permission denied` | +| `procMount: Unmasked` (to `Default`) | NetOnly PASS; FullBoot fails, virtiofsd: `Error entering sandbox: MountProc(... Operation not permitted)` | +| `hostUsers: false` | Not removable alone: the API rejects `Unmasked` without it. Removing both gives the `MountProc` failure above | +| node sysctl to `1` | passt: `Failed to detach isolating namespaces: Operation not permitted`, even with every pod grant | + +- **AppArmor** is a confinement layer the record does not name. The + container runtime's default profile denies the `mount` passt's sandbox + performs inside its own user namespace. +- **Masked `/proc`:** virtiofsd's `--sandbox=namespace` mounts a fresh `/proc` + in its namespace. The kernel refuses that while the container's `/proc` has + masked or read-only submounts, and only `procMount: Unmasked` removes them. +- **Node sysctl:** Ubuntu 24.04 ships + `kernel.apparmor_restrict_unprivileged_userns=1`, and the working set needs + `0`. That is an operator node-provisioning requirement on this image; other + node images may not carry the setting. +- **No Linux capability and no `privileged: true` was needed.** The pod stays + `drop: ["ALL"]`, `runAsNonRoot`, `allowPrivilegeEscalation: false`. The + container-vs-host ruling stands. + +**Open interaction (S1 × S3):** under `hostUsers: false`, the injected +`/dev/kvm` shows as `65534:65534` in the pod (`crw-rw----`). The host kvm gid +is outside the pod's id mapping, so with `supplementalGroups: [993]` `open()` +still fails `EACCES`, and the full set fails `/dev/kvm is not openable` with +the node device at `0660`. The passing runs used the node device at `0666`. +Measured: a user-namespaced pod booted only with a world-rw node device. A +device plugin that sets the mode per pod is untested. This is a design +decision, and §Privilege shape stays unchanged until it is made. + +### S2 — hostPath char device + +`/dev/kvm` and the control `/dev/fuse` were mounted as hostPath `CharDevice` +into an unprivileged pod (uid 10001, groups include 993, `drop: ["ALL"]`, no +device plugin). In the pod `/dev/kvm` was `crw-rw---- 0:993` and `/dev/fuse` +`crw-rw-rw-`. Both `open()` calls failed with **`EPERM`** (`Operation not +permitted`). A world-rw mode failing rules out DAC, so this is the cgroup +device controller. **Control:** the same hostPath in a `privileged: true` pod +opened. The record's `[INFERENCE]` holds: hostPath is non-functional, not +merely a worse posture. + +### S3 — the kvm gid + +With device-plugin injection (`devic.es/kvm: 1`), non-root uid 10001, and the +device `crw-rw---- 0:993` in the pod: + +- With `supplementalGroups: [993]`: opens. +- **Control**, without it: fails **`EACCES`** (`Permission denied`), so a DAC + denial, not the cgroup. + +The grant is necessary under the node's `0660` mode, but inert under S1's +`hostUsers: false` (see above). + +### S4 — seccomp (OQ-2) + +`RuntimeDefault` fails at either sysctl value: passt reports `Couldn't create +user namespace: Operation not permitted`. No syscall trace was captured, so +the exact denied syscall is **not verified**. The working profile is the +container runtime's default profile plus an unconditional allow of `unshare`, +`mount`, `umount2`, `pivot_root`, `setns`, `clone`, and `clone3`, staged at +`/var/lib/kubelet/seccomp/profiles/compass-runner.json`. On this node the +`Localhost` profile and its on-node staging are required; OQ-2 closes once the +real-node run agrees. This spike did not minimize the allow list; R3 should +narrow it with the same remove-one test. + +### S5 — memory cgroup + +Pods ran the full S1 set with a 2048 MiB guest. The harness wrote 900 MiB to +guest `/tmp` over the guest-control `Exec` RPC (`dd` reported +`943718400 bytes ... copied`). Pod cgroup values: + +| Pod | `memory.current` | `shmem` | +| --- | --- | --- | +| Control: no session (runner exits; a `sleep` sidecar holds the pod) | <1 MiB | 0 MiB | +| Guest booted, RAM not written | 229 MiB | 206 MiB | +| Guest booted, 900 MiB written | 1119 MiB | 1089 MiB | + +The control is a pod-level baseline only: the Runner binary was not running +in it. The rise of 890 MiB (883 MiB shmem) tracks the guest write. The guest's +`memfd:ch_ram` mapping (`--memory shared=on`) is charged to the pod as +`shmem`, and only as pages are touched. §Pod resources holds. Size requests +and limits to configured guest totals; a fresh guest under-reports until it +uses its RAM. + +### S6 — container restart + +**Control:** before the kill, the node showed cloud-hypervisor, passt, and two +virtiofsd processes from the runner image. pid 1 was killed via +`crictl stop -t 0`; the container exited with code 137. **After the kill: zero** +cloud-hypervisor, virtiofsd, or passt processes on the node. The pid namespace +teardown reaps them. The stale-pidfile reap on restart was **not verified**: +the harness used an `emptyDir` runtime dir and is not the Runner binary. From 1724ae69d534d45515efa7f39fa1f674d040b686 Mon Sep 17 00:00:00 2001 From: mintaka Date: Mon, 5 Oct 2026 23:38:05 -0400 Subject: [PATCH 2/2] docs(runner): mark the userns sysctl image default unverified (RIG-3723) Refs RIG-3723 Co-authored-by: Matt Wilkinson --- .../compass-runner-containerization/spike-findings.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/designs/platform/compass-runner-containerization/spike-findings.md b/docs/designs/platform/compass-runner-containerization/spike-findings.md index 41b9a65d3..04e6f31be 100644 --- a/docs/designs/platform/compass-runner-containerization/spike-findings.md +++ b/docs/designs/platform/compass-runner-containerization/spike-findings.md @@ -180,10 +180,10 @@ that set, with the node sysctl at `0`: - **Masked `/proc`:** virtiofsd's `--sandbox=namespace` mounts a fresh `/proc` in its namespace. The kernel refuses that while the container's `/proc` has masked or read-only submounts, and only `procMount: Unmasked` removes them. -- **Node sysctl:** Ubuntu 24.04 ships - `kernel.apparmor_restrict_unprivileged_userns=1`, and the working set needs - `0`. That is an operator node-provisioning requirement on this image; other - node images may not carry the setting. +- **Node sysctl:** with `kernel.apparmor_restrict_unprivileged_userns=1` passt + fails even under every pod grant; the working set needs `0`. The harness set + the value before each run, so the Ubuntu 24.04 image default is **not + verified**. If the target node image enables it, operators must set `0`. - **No Linux capability and no `privileged: true` was needed.** The pod stays `drop: ["ALL"]`, `runAsNonRoot`, `allowPrivilegeEscalation: false`. The container-vs-host ruling stands.