Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,8 @@ capacity sizing); this record fixes the *shape*.
§Privilege shape.
- `spec.nodeName` via `fieldRef` into the environment, so the Runner can
identify its node.
- A scrape annotation for the metrics endpoint.
- No scrape annotation: the Runner pushes metrics over OTLP
(`OTEL_EXPORTER_OTLP_ENDPOINT`) and serves no scrape endpoint.
- **Liveness probe: conservative, or absent.** A probe-driven container
restart is the same full-session teardown as a rollout (§Privilege shape,
restart semantics) — pid 1 dies and every session on the node dies with
Expand Down
6 changes: 3 additions & 3 deletions runner-image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@
# proving the loader comes from the closure rather than the base.
#
# The pinned digest is the nonroot variant (uid/gid 65532). The Runner needs no
# root: /dev/kvm access is granted by supplementary group at the pod layer.
# Pinned by digest, never by tag — GHCR-style tag mutability gives no
# immutability guarantee.
# root: the device plugin grants the cgroup allowance and node mode 0666 grants
# DAC access. Pinned by digest, never by tag — GHCR-style tag mutability gives
# no immutability guarantee.
FROM gcr.io/distroless/static-debian12@sha256:c0f429e16b13e583da7e5a6ec20dd656d325d88e6819cafe0adb0828976529dc

# The realised nix closure: the Runner binary, the KVM userland and the guest
Expand Down
45 changes: 45 additions & 0 deletions tools/runner-manifests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Runner manifests

Render Kubernetes objects from an operator values file:

```sh
bun run render.ts values.example.json
```

The image must be a `repo@sha256:<digest>` reference. Replace the example node
selector, taint, host paths, Server address, and token Secret before applying.

## KVM device delivery

- Install an operator-selected, digest-pinned device plugin that advertises the
configured `kvmResourceName` for `/dev/kvm`. The plugin injects the device and
its cgroup device allowance; no `/dev` hostPath is used.
- Set node `/dev/kvm` mode to `0666`. User-namespaced pods see the device as
`65534:65534`, so a host `kvm` group grant does not work.

## Other node prerequisites

- Enable Kubernetes user-namespace support.
- Set `kernel.apparmor_restrict_unprivileged_userns=0` where the node image
enables that restriction.
- Stage `seccomp/compass-runner.json` at the kubelet seccomp root under the
configured `seccompProfilePath` before starting the DaemonSet. It is the
profile measured to boot, not yet minimized: `clone`, `clone3`, `unshare`
and `mount` are allowed without argument filters. Narrow it with a
remove-one test on a real node before production use.
- Provide the session-volume host tree on a filesystem configured for project
quotas. Provide the separate runtime host tree for stale-session reaping.
Own both by uid and gid 65532: user-namespaced pods mount them idmapped. Not
yet verified on a real node.
- Create the `runnerTokenSecret` with the key named in the values file.

**Single-node only for now.** A Runner token is minted for one runner ID, and
each pod enrolls as its node name. One shared Secret therefore enrolls only
the node whose name matches the token. Per-node token delivery is not
designed yet.

Metrics: set `OTEL_EXPORTER_OTLP_ENDPOINT` to push them. The Runner serves no
scrape endpoint.

The Runner does not call the Kubernetes API. Its ServiceAccount disables
credential automount, and no Role is rendered.
4 changes: 4 additions & 0 deletions tools/runner-manifests/biome.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"extends": "//",
"linter": { "rules": { "suspicious": { "noConsole": "off" } } }
}
14 changes: 14 additions & 0 deletions tools/runner-manifests/moon.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# yaml-language-server: $schema=https://moonrepo.dev/schemas/project.json
#
# runner-manifests: values-driven DaemonSet rendering and behavior tests.
layer: 'tool'
language: 'typescript'
tags: ['bun', 'ci-group.bun']

tasks:
typecheck:
command: 'bunx tsc --noEmit'
deps: ['install']
inputs: ['*.ts', 'tsconfig.json', '/tsconfig.base.json', 'package.json', '/bun.lock']
test:
inputs: ['*.ts', 'tsconfig.json', '/tsconfig.base.json', 'package.json', '/bun.lock', 'seccomp/*.json', '../../runner-image/Dockerfile']
14 changes: 14 additions & 0 deletions tools/runner-manifests/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"name": "@compass/runner-manifests",
"private": true,
"type": "module",
"description": "Render operator-configured Kubernetes manifests for compass-runner.",
"module": "render.ts",
"bin": {
"runner-manifests-render": "./render.ts"
},
"devDependencies": {
"@types/bun": "catalog:",
"typescript": "catalog:"
}
}
Loading
Loading