Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ jobs:
- run: python3 bench/eatflow/test_agentbench.py
- run: python3 bench/eatflow/test_pairs.py
- run: python3 bench/eatflow/test_solo.py
- run: tests/xhost_tools_test.sh
- run: python3 bench/eatflow/test_h4345.py
- run: python3 bench/models/test_export_qat_ov.py
- run: python3 bench/models/test_qat_grid.py
Expand Down
53 changes: 51 additions & 2 deletions EXPERIMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -1735,7 +1735,7 @@ entries that ask for another, Xeon/AMX, XMX, non-throttling or quiet host, and t
below showed the cause: without C1 the agent pod stayed Pending. C1 changes when the kubelet
evicts, nothing else, and applies to both arms alike. X1's 97% guard stays.
5. **Pins.** k3d v5.9.0 and kubectl v1.35.2 in `~/k3dgpu-xhost/bin`, each checked against its
upstream checksum; node image `cryptoandcoffee/k3d-gpu:0.1.61` (the launcher's default at this pre-registration's base, set explicitly with `K3D_GPU_IMAGE` or `IMAGE` in every command,
upstream checksum (kubectl v1.37.2 for the rows [amendment K1](#cross-host-amendment-k1-kubectl-v1372-in-the-cross-host-workspace-2026-10-29) moves); node image `cryptoandcoffee/k3d-gpu:0.1.61` (the launcher's default at this pre-registration's base, set explicitly with `K3D_GPU_IMAGE` or `IMAGE` in every command,
k3s v1.37.1; a change: the originals ran 0.1.28 (H25 run 4), 0.1.55 (H34) and 0.1.59 (H28),
with the agent-manifest code unchanged since H25 run 4); OVMS `openvino/model_server@sha256:a928903f...76fc58` (2026.4.1, as every
original row); `load.py`'s stress-ng pin. Models are downloaded on the host at the original
Expand Down Expand Up @@ -1801,6 +1801,36 @@ failure C1 fixes. Smoke 2 (22:14-22:20Z, with C1): `up` 48 s, the agent pod Read
C1 as `--k3s-arg ...@server:*`) had no taint after 90 s. No inference request was sent in either.
Smoke records: `bench/xhost/results/2026-10-20/smoke{1,2}/`.

### Cross-host amendment K1: kubectl v1.37.2 in the cross-host workspace (2026-10-29)

Recorded 2026-10-29, before any H7x hold and before H28x's run 2. Mechanics only. No hypothesis,
arm, model, node image, prediction, outcome table, scorer, threshold, cap or order changes. It
is pre-registered by its push time.

1. **Why.** Rule 5 pins kubectl v1.35.2 in `~/k3dgpu-xhost/bin`, and `bench/xhost/hold.sh` puts
that directory first on `PATH`. The node image these rows run (0.1.61) serves Kubernetes
v1.37.1 (k3s v1.37.1+k3s1). Kubernetes' version skew policy supports kubectl within one minor
version of kube-apiserver, so v1.35 against v1.37 is outside it. No hold has failed from this.
Read on 2026-10-29: kubectl v1.37.2 (2026-10-28, `dl.k8s.io/release/stable.txt`) is the newest
release; k3d v5.9.0 is still its newest and stays. Crescent amendment E4 (PR #238) makes the
same move for shared rule 5's workspace, `~/k3dgpu-bench/bin`; this amendment covers the
cross-host workspace, which E4 does not touch.
2. **Pins.** k3d v5.9.0 (unchanged) and kubectl v1.37.2 (linux/amd64 sha256 `c0448752...`).
`bench/xhost/tools.sh`, new, installs exactly these two into `~/k3dgpu-xhost/bin`, each checked
against its upstream checksum file (k3d's `checksums.txt`, kubectl's `.sha256`), and prints
both versions. It touches nothing else. Rule 5's tools had no script in this repo until now.
3. **When.** `tools.sh` runs once on crescent from a clean checkout at or after this amendment's
merge, with the bench lock free and no cross-host row between holds. Its output goes into the
next cross-host row's records.
4. **Rows.** A row runs all its holds on the kubectl in place at its first measuring hold. Rows
whose first measuring hold starts after `tools.sh` has run use v1.37.2: H28x's run 2, if PR
#208's amendment R1 allows it, and H7x (PRs #196 and #235), if its pre-registration merges
under these rules. H25x, H27x and H34x have run and keep what they ran; H25x and H34x record
kubectl v1.35.2. A smoke, or a hold that measured nothing (H28x run 1), does not fix a row's
tools.
5. **Not changed.** The node image, the models, OVMS, `load.py`'s stress-ng pin and every other
item of rule 5.

### H25x run (Xeon 6741P, 2026-10-21)

```text
Expand Down Expand Up @@ -4181,7 +4211,8 @@ re-graded here.
whose need never fits before the campaign ends is INCONCLUSIVE (not run), with the
readings recorded. Everything this campaign creates on wildct is deleted at the end
except the run records, which are copied back into the repo first.
7. **Pins.** k3d v5.9.0, helm v4.3.1, kubectl v1.37.1 (each the newest release on 2026-10-20)
7. **Pins.** k3d v5.9.0, helm v4.3.1, kubectl v1.37.1 (each the newest release on 2026-10-20;
kubectl v1.37.2 from [amendment P1](#wildct-amendment-p1-kubectl-v1372-2026-10-29))
in `~/k3dgpu-bench-wildct/bin`, checked against their upstream checksums
(`bench/wildct/install.sh`); k3d-gpu: the newest release at the first hold (the first one
carrying PR #180: 0.1.65 or later), its `.deb` checked against the release's `SHA256SUMS` and unpacked in
Expand Down Expand Up @@ -4282,6 +4313,24 @@ request, discarded and disclosed). Then H52w (1 hold), H18w (1), H25w (1), H34w
repeat a failed hold once. At most 12 holds in all. A protocol change is a dated amendment
merged before the data it governs.

### Wildct amendment P1: kubectl v1.37.2 (2026-10-29)

Recorded 2026-10-29, before any wildct hold (none has run; `smoke1` was discarded). Mechanics only.
No row's hypothesis, prediction, arm, model, statistic, threshold, cap, disk need, hold count or
order changes.

Rule W7 pins kubectl v1.37.1, the newest release on 2026-10-20. Kubernetes v1.37.2 came out on
2026-10-28 and is the newest now (`dl.k8s.io/release/stable.txt`, read 2026-10-29); k3d v5.9.0
and helm v4.3.1 are still the newest of theirs. W7 now reads kubectl v1.37.2.
`bench/wildct/install.sh` fetches it and checks it against its `.sha256` file (linux/amd64
`c0448752...`), as before. The node images serve Kubernetes v1.37.1 (k3s v1.37.1+k3s1), so
client and server share a minor version, as before. `install.sh` runs again before
the first hold anyway, to install W7's newest k3d-gpu release (0.1.75 or later, not the 0.1.71
installed on 2026-10-21), and that run installs kubectl v1.37.2. So every wildct row (H52w,
H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, and H59w if it is pre-registered under W7) runs on
it. If wildct amendment R1 (PR #226) merges, its sentence that keeps kubectl v1.37.1 is superseded
by this amendment. Each row's Reproduce header names the kubectl that ran.

### H5 amendment A1: stage-1 harness and definitions (2026-10-19)

Written 2026-10-19T22:30-07:00, before any H5 measurement. It fixes what the
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -573,7 +573,7 @@ name in place of `gpu`):
```bash
docker exec k3d-gpu-server-0 touch /var/lib/rancher/k3s/server/manifests/nri-balloons.yaml.skip
helm upgrade --install nri-resource-policy-balloons nri-resource-policy-balloons \
--repo https://containers.github.io/nri-plugins --version v0.14.0 \
--repo https://containers.github.io/nri-plugins --version v0.15.0 \
--namespace kube-system -f values.yaml --force # Helm 4: --force-conflicts
```

Expand Down Expand Up @@ -713,7 +713,7 @@ documents GPU support for NVIDIA only.
### Node Feature Discovery

Every cluster also runs [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery)
v0.19.0 with Intel's
v0.20.1 with Intel's
[NodeFeatureRules](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/v0.37.1/deployments/nfd/overlays/node-feature-rules)
v0.37.1, baked into the image (namespace `node-feature-discovery`). About 30 s
after `up`, nodes carry the host's CPU, kernel and PCI features. On an
Expand Down Expand Up @@ -1485,7 +1485,7 @@ not switch the Intel GPU path.
### Agent sandboxes on E-cores

`k3d-gpu sandbox up` installs [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
v1.0.7 (controller image pinned by digest) and a `python-ecores` SandboxTemplate
v1.0.8 (controller image pinned by digest) and a `python-ecores` SandboxTemplate
plus a one-replica warm pool in namespace `agent-sandbox`. Sandbox pods carry
`k3d-gpu.io/cores: efficient` and request one CPU, so on a hybrid host untrusted
agent tool execution runs on E-cores, apart from the P-cores that serve inference.
Expand All @@ -1506,7 +1506,7 @@ EOF
```

The sandbox serves `POST /execute {"command": "..."}` on port 8888
(`registry.k8s.io/agent-sandbox/python-runtime-sandbox:v1.0.7`, pinned by digest).
(`registry.k8s.io/agent-sandbox/python-runtime-sandbox:v1.0.8`, pinned by digest).

What works on k3d, and what does not:

Expand Down
14 changes: 7 additions & 7 deletions bench/wildct/install.sh
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
#!/usr/bin/env bash
# install.sh K3D_GPU_VERSION: the wildct campaign's tools, pinned and checksum-checked, into
# ~/k3dgpu-bench-wildct (EXPERIMENTS.md "Wildct pre-registration", rule W7); nothing
# system-wide. k3d v5.9.0, helm v4.3.1 and kubectl v1.37.1 (each the newest release on
# 2026-10-20) into bin/; the k3d-gpu release's .deb, checked against the release's
# SHA256SUMS, unpacked into k3d-gpu/ (k3d-gpu/usr/bin/k3d-gpu with K3D_GPU_SHARE pointing at
# k3d-gpu/usr/share/k3d-gpu); huggingface_hub 1.21.0 (crescent's pin) in hfvenv/. Prints
# every version and checksum.
# system-wide. k3d v5.9.0 and helm v4.3.1 (each the newest release on 2026-10-20) and kubectl
# v1.37.2 (the newest on 2026-10-29, wildct amendment P1) into bin/; the k3d-gpu release's
# .deb, checked against the release's SHA256SUMS, unpacked into k3d-gpu/
# (k3d-gpu/usr/bin/k3d-gpu with K3D_GPU_SHARE pointing at k3d-gpu/usr/share/k3d-gpu);
# huggingface_hub 1.21.0 (crescent's pin) in hfvenv/. Prints every version and checksum.
set -euo pipefail
V=${1:?usage: install.sh K3D_GPU_VERSION}
W=${HOME}/k3dgpu-bench-wildct
Expand All @@ -23,8 +23,8 @@ check 0403a88ae20913d11b10a2467b50087721fea7581736da6b9d3992f5357fe54b helm.tgz
tar -xzf helm.tgz linux-amd64/helm
install -m 0755 linux-amd64/helm "${W}/bin/helm"

get https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/kubectl kubectl
get https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/kubectl.sha256 kubectl.sha256
get https://dl.k8s.io/release/v1.37.2/bin/linux/amd64/kubectl kubectl
get https://dl.k8s.io/release/v1.37.2/bin/linux/amd64/kubectl.sha256 kubectl.sha256
check "$(cat kubectl.sha256)" kubectl
install -m 0755 kubectl "${W}/bin/kubectl"

Expand Down
21 changes: 21 additions & 0 deletions bench/xhost/tools.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# tools.sh: the cross-host rows' k3d and kubectl in ~/k3dgpu-xhost/bin (EXPERIMENTS.md
# "Cross-host pre-registration" rule 5, as cross-host amendment K1 moves it), each checked
# against its upstream checksum file. Installs nothing else and touches no other file. Run on
# crescent from a clean checkout of this repo, with the bench lock free. Prints both versions.
set -euo pipefail
X=${HOME}/k3dgpu-xhost
K3D_V=v5.9.0 KUBECTL_V=v1.37.2
mkdir -p "${X}/bin"
DL=$(mktemp -d "${X}/tools-dl.XXXXXX")
trap 'rm -rf "${DL}"' EXIT
cd "${DL}"
curl -fsSLo k3d "https://github.com/k3d-io/k3d/releases/download/${K3D_V}/k3d-linux-amd64"
curl -fsSLo k3d.sums "https://github.com/k3d-io/k3d/releases/download/${K3D_V}/checksums.txt"
curl -fsSLo kubectl "https://dl.k8s.io/release/${KUBECTL_V}/bin/linux/amd64/kubectl"
curl -fsSLo kubectl.sha256 "https://dl.k8s.io/release/${KUBECTL_V}/bin/linux/amd64/kubectl.sha256"
echo "$(grep 'k3d-linux-amd64$' k3d.sums | awk '{print $1}') k3d" | sha256sum -c
echo "$(cat kubectl.sha256) kubectl" | sha256sum -c
install -m755 k3d kubectl "${X}/bin/"
"${X}/bin/k3d" version | head -1
"${X}/bin/kubectl" version --client | head -1
4 changes: 2 additions & 2 deletions docs/intel.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ with detection, enablement and what k3d-gpu does with each, see the
manifests into k3s's auto-deploy directory:
[intel-device-plugins-for-kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes)
`0.37.1` (GPU, NPU, DSA, IAA, QAT), `intel/gaudi-device-plugin:1.24.2-481` and
[nri-plugins](https://github.com/containers/nri-plugins/releases/tag/v0.14.0)
balloons `v0.14.0`. With the default image nothing is `kubectl apply`'d at
[nri-plugins](https://github.com/containers/nri-plugins/releases/tag/v0.15.0)
balloons `v0.15.0`. With the default image nothing is `kubectl apply`'d at
run time. If a custom `K3D_GPU_IMAGE` lacks a baked DaemonSet, `up` applies
the manifest from `K3D_GPU_SHARE`; for NVIDIA with no local manifest it falls
back to the upstream `main` branch manifest from GitHub, which is unpinned.
Expand Down
2 changes: 1 addition & 1 deletion docs/intel/upstream-eat.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ EAS (main e8f3ebc; one concern per PR, in this order)
├── roles/kubernetes/tasks/apply_node_topology.yaml 0004
├── docs/customize/node_topology.md 0004 "label by hand" text replaced
├── docs/deploy/topologies.md 0004
├── roles/nri_cpu_balloons/defaults/main.yaml 0003 nri_cpu_balloons_chart_version: "v0.14.0"
├── roles/nri_cpu_balloons/defaults/main.yaml 0003 nri_cpu_balloons_chart_version: "v0.15.0"
├── roles/nri_cpu_balloons/tasks/install_advanced.yaml 0003
├── docs/deploy/k3d.md 0005 new page "Single-Node Dev on k3d"
└── docs/README.md 0005 +1 row under Deploy
Expand Down
12 changes: 6 additions & 6 deletions images.lock
Original file line number Diff line number Diff line change
Expand Up @@ -388,28 +388,28 @@
"tag": "v3.15.0"
},
{
"digest": "sha256:f254384c24605436ce35d7077427559e24bda8f5691cff5a54a4c2e030d81083",
"digest": "sha256:e1d96312487eba7d8f001e848a9d4e644b7deea8c2a4564ccc318abb729be731",
"name": "registry.k8s.io/agent-sandbox/agent-sandbox-controller",
"sources": [
"share/agent-sandbox.yml"
],
"tag": "v1.0.7"
"tag": "v1.0.8"
},
{
"digest": "sha256:fc92654c97fe5dff2a7935a845820981026c075d0c1014e5c2c569f5e58204f1",
"digest": "sha256:41da84809426098cb04cdafbb41855f15e3604f7455ef53816bad666e532e787",
"name": "registry.k8s.io/agent-sandbox/python-runtime-sandbox",
"sources": [
"share/agent-sandbox-k3d.yml"
],
"tag": "v1.0.7"
"tag": "v1.0.8"
},
{
"digest": "sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2",
"digest": "sha256:6fc5c9af9176e1acb5ce57a4a1d0c0d3eb25cd34f5bdd0efefebf0efe45a8379",
"name": "registry.k8s.io/nfd/node-feature-discovery",
"sources": [
"share/node-feature-discovery.yml"
],
"tag": "v0.19.0"
"tag": "v0.20.1"
},
{
"digest": "sha256:5d064465d751a73a729e7ecdc32c0ba5de0df1d305debb60a3d9a85b4fd3a967",
Expand Down
Loading
Loading