eas: EAS and EAT config from cluster facts, the pins file, fixture tests by 88plug · Pull Request #251 · 88plug/k3d-gpu

88plug · 2026-10-29T21:10:12Z

This is the config part of full EAS support. It turns read-only facts about a k3d-gpu cluster into the configuration Intel's Enterprise AI Solutions installer (es_auto_installer.sh, existing_kubernetes mode) needs, for the inference layer and for EAT 2.0 through install agent-toolkit. It does not deploy anything: eas up (bastion, facts gathering, install) comes in a later part and consumes these files.

It is stacked on #248 (the patch overlay), which adds tests/eas_test.sh and its ci.yml line. This PR adds a config section to that file, so the two do not conflict. Its base is the #248 branch, and it will retarget to main once #248 merges.

What it adds

integrations/eas/eas-config.py takes the facts JSON (schema k3d-gpu.eas-facts/v1, documented in its docstring), pins.json and the flags. It is pure, uses only the Python standard library, and writes:

k3d-gpu-overrides.yaml: ansible extra vars, passed last to install so they win. Strings are quoted, while booleans and lists keep their YAML types, because ansible-core 2.19 rejects a quoted "false" in a conditional.
global_config.edit.yaml: existing_kubernetes: "/kube/config". _resolve_kubeconfig reads this key from global_config.yaml only.
models.edit.yaml: defaults.require_amx and any KV cache change, for env/k3d-gpu/models.yaml, which model-manager reads. Entries match a catalog entry by name; env merges and args_append goes after the catalog's args.
repos.<layer>.yaml: the layer manifest's revs as commit SHAs.
choices.txt: one line per choice, <fact> -> <choice> [<repo>@<sha7> <path>:<lines>]. --explain prints each source as a full permalink.

A refusal exits 3, prints REFUSE: lines and leaves none of these files in --out. Bad input, including a floating ref, exits 2.

integrations/eas/pins.json (k3d-gpu.eas-pins/v1, checked 2026-10-29) holds flat rows {profile, name, repo, ref, sha, date, source}:

Profile main (the default): EAS main 97f6157 (dgpu: per-id Battlemage kernel minimum; Alchemist, Flex, Max detection #69), EI main 3e8a244 (feat: NPU firmware/user-space check, render group, embeddings recipe #44), EAT release-2.0.0 036b662 (the build: devcontainer and make check running every repo validator #19 merge). The EAT tree is identical to agentic-toolkit-migration fb59459, which EAS main names at repos.agent-toolkit.yaml#L21. I compared the two trees file by file.
Profile v1.2.0-rc.1: EAS 5c27fb3, EI release-1.1 170eb3a, EAT 036b662, as in the release notes.
Profile bastion, for the bastion part:
- kubectl v1.37.2 (the minor of the Dockerfile's K3S_TAG)
- helm v3.22.1 (newest v3)
- yq v4.54.1
- kubernetes.core 6.3.0 (EAS's exact pin)
- the newest Python packages inside EAS's requirements.txt ranges, each with its wheel sha256: ansible 12.3.0, kubernetes 35.0.0, jsonpatch 1.38, jmespath 1.1.0
- ubuntu:24.04 at the same digest as integrations/eat/bastion.Dockerfile

Every head was re-read with git ls-remote and gh api just before the commit. Tool versions came from the GitHub releases API and PyPI.

Choices and their sources

Each of the 36 sources was read at its permalink. A script printed every cited line range from local copies of the pinned trees, to confirm each citation says what the choice claims.

istio_platform follows containerd's CNI bin_dirs: k3d for k3d-gpu's node image (["/usr/bin"], read from a live node, which needs overlay 0002), k3s for stock k3s, and "" for Istio's default paths. Any other layout is refused.
metallb_enabled: false while ServiceLB is on. EAS's MetalLB role looks only for MetalLB and kube-vip (install.yaml#L23-L31).
kubernetes_cpu_policy is best-effort in three cases:
- a balloons DaemonSet from a k3s addon is present (k3d-gpu's baked policy; Gate 2 probe 6)
- Docker is rootless (k3s disables NRI in a user namespace, templates.go#L322-L325)
- the node has no NRI socket
Otherwise it is nri-balloons, with a line flagging overlay 0003. Ownership comes from the objectset.rio.cattle.io/owner-gvk annotation, not from Helm labels: a live k3d-gpu node showed that the baked balloons DaemonSet also carries the chart's app.kubernetes.io/managed-by: Helm label and meta.helm.sh/release-name annotation.
nfd_enabled: false next to k3d-gpu's NFD, because EI's role looks only for a Helm release named node-feature-discovery (install.yaml#L15-L25). It stays true only for that release; a Helm release of another name (NFD's own docs use nfd) gets false too.
require_amx follows the cpu-cpuid.AMXBF16 label. Without AMX, VLLM_CPU_SGL_KERNEL is left as shipped.
The CIDRs come from k3s.io/node-args, or the k3s defaults at v1.37.1+k3s1. kata_containers_enabled: false is set on BYO. kubernetes_accelerator: cpu always; a GPU or Gaudi gets the note "serve the device with k3d-gpu agent up".
On the agent-toolkit layer:
- agent_sandbox_enabled: true.
- The model is EAT's default when the node has the 131 GiB EAT documents and k3d-gpu fit agrees. Otherwise it is the first documented 64 GiB fallback that fits, else [] with the reason. --model overrides the pick, with warnings.
- Guardrails and the hybrid router keep EAT's defaults.
The KV cache: EI renders the catalog's memory: as both the pod's request and its limit (llm-inference-service.yaml), while fit sizes against the host. So fit may lower the catalog's VLLM_CPU_KVCACHE_SPACE (40 GiB for all three candidates, at both EI pins), never raise it. A lowered cache also carries the --max-model-len fit sized it for: the catalog sets none, so vLLM would serve the model's own context (131072 tokens for Llama-3.1-8B-Instruct, 16 GiB of KV cache for one request) and refuse to start when one request does not fit.

Refusals:

ServiceLB disabled
EAT 1.x namespaces (genai-gateway, ingress-nginx)
k3d-gpu's own agent-sandbox, found by its agent-sandbox namespace, or any version other than EAT's v0.5.0
an unknown CNI layout
--layer erag: ERAG's chatqna_intel_gpu is EXPERIMENTAL and needs kubernetes_accelerator=intel-gpu, and EI 3e8a244's model_manager/runtimes holds only openvino and vllm

Tests

Test first: the config section and its fixtures were committed failing (5 passed, 30 failed) before the generator.

There are 19 golden fixtures under tests/fixtures/eas/config:

the desktop i9-13900T, a Xeon with AMX and an external policy, and a Gaudi box
ServiceLB disabled, k3d-gpu's sandbox v1.0.8, and EAT 1.x namespaces
rootless Docker, NRI off, and a tiny host
--layer erag, the v1.2.0-rc.1 profile, and the inference layer
--model cases, including one fit rejects
an EAS re-run over its own Helm releases with custom CIDRs
stock k3s, Istio-default and unknown CNI layouts
fit failing for one candidate

The section also checks:

the generator runs under python3 -S
floating refs and malformed facts exit 2
a refusal or a second run leaves no stale output
--explain, the permalink form of every source, and the pins shape
every lowered KV cache carries fit's own --max-model-len (16384 in a mutated fit document), and a fit KV cache without one exits 2
NFD from a Helm release other than node-feature-discovery gets nfd_enabled: false

A mutation pass of 19 mutants over the generator was run against the config section. All 19 were caught.

Run on this branch:

tests/eas_test.sh: 223 passed (overlay 87, config 136)
tests/ci_wiring_test.sh: 70 passed
tests/claims_check.sh: clean
python3 tests/ledger_drift_check.py: agrees
tests/eat_test.sh: 64 passed
tests/image_pins_test.sh: 43 passed
shellcheck v0.11.0 and actionlint 1.7.12: clean

No ledger row: these are fixture tests with no functional run. The scored run is F3, pre-registered in #249.

For the stacked docs PR (#249)

docs/intel/eas.md there describes this generator. A few points differ from what is built here:

The KV cache is capped at the catalog value, and a lowered one carries fit's --max-model-len as args_append in models.edit.yaml.
pins.json rows carry a profile field.
Two more refusals exist: an unknown CNI layout, and an agent-sandbox version that is neither k3d-gpu's nor EAT's.
best-effort is also chosen when the node has no NRI socket.
Pinning agent_sandbox_router_image and agent_sandbox_python_runtime_image by digest is not in this part; the design leaves those digests to the bastion part.

Not verified

No cluster was run. The facts format (node-args, DaemonSet ownership annotations, CNI bin_dirs, the NRI socket) was read from one live k3d-gpu v1.37.1 node.
The EAT 2.0 layer has never run on k3s or k3d.
Whether the chosen overrides install cleanly end to end is for the later wiring smoke and F3.

…s the fix integrations/eas/overlay.py decides, per row of integrations/eas/overlay.json and at one EAS commit, whether a k3s/k3d patch is skipped (not needed on this cluster), dropped because upstream merged it (git apply -R --check), dropped because upstream fixed it another way (a probe matches and the knob the generator writes is still there), applied (first variant whose git apply --check passes), or a STOP that names the upstream change that moved the hunk and says to regenerate. Every decision prints as `fact -> choice [source]`, with --explain giving permalinks. Decisions read the commit through a scratch index, so they do not change once the overlay is applied; --apply writes the working tree only when nothing STOPs, and a re-run is a no-op. Patches are looked up in integrations/eas/patches, then in the upstream drafts, so the drafts stay the only copy. The one new patch is the release-1.2 form of 0006, where the EnvoyProxy task is still in roles/envoy_gateway/tasks/install.yaml. tests/eas_test.sh (section overlay) runs it on synthetic git repos built from byte-identical copies of the EAS files the patches touch, at main 97f6157 and release-1.2 5c27fb3. apply-check.sh gains --overlay, which checks the recorded decisions on a real clone and prints them for branch tips and open EAS PRs #50, #52 and #61.

tests/eas_test.sh gains a config section, next to the overlay section, that runs integrations/eas/eas-config.py on nineteen golden fixtures under tests/fixtures/eas/config. Each fixture is a facts.json and an args line, and the expected k3d-gpu-overrides.yaml, global_config.edit.yaml, models.edit.yaml, repos.<layer>.yaml and choices.txt must come out byte for byte. Refusal cases must exit 3, print the expected reasons and write nothing. The fixtures cover: - the desktop (hybrid, baked balloons, no AMX, Arc GPU, 125 GiB, 32 CPUs) - a Xeon with AMX and no NRI policy, and a Gaudi box - ServiceLB disabled, k3d-gpu's own agent-sandbox, and EAT 1.x namespaces - rootless Docker, NRI off, and a host where no model fits - the erag layer, the v1.2.0-rc.1 profile, and the inference layer - --model overrides, including one fit rejects - a re-run over EAS's own Helm releases with custom CIDRs - stock k3s, Istio's default, and unknown CNI layouts - fit failing for one candidate The balloons and NFD DaemonSet metadata, the k3s.io/node-args format and the containerd CNI bin_dirs come from a live k3d-gpu v1.37.1 node. The k3d-gpu balloons DaemonSet carries the chart's Helm labels as well as the k3s addon annotation, so ownership is read from the annotation. The section also checks that the generator needs only the Python standard library, that floating refs and malformed facts exit 2, and that a refusal or a second run leaves no stale output. It checks that --explain prints permalinks, that every citable source is a permalink at a full commit SHA, and the pins.json shape. It fails now (5 passed, 30 failed): the generator and pins.json do not exist yet. The overlay section is unchanged and passes.

…ream SHAs integrations/eas/eas-config.py turns read-only facts about a k3d-gpu cluster into what Intel's Enterprise AI Solutions installer (es_auto_installer.sh, existing_kubernetes mode) needs: - ansible extra vars, passed last so they win - the existing_kubernetes key for global_config.yaml - edits to models.yaml, the file model-manager reads - the layer manifest's revs, as commit SHAs - one line per choice: "<fact> -> <choice> [<source>]" Each source is a permalink at the tree it was read from: EAS 97f6157, EI 3e8a244, EAT 036b662, k3s v1.37.1+k3s1 356c025, vLLM v0.28.0 2cf0a69, and this repo's Gate 2 probe records. --explain prints the permalinks. The generator is pure (no cluster access) and uses only the Python standard library. Strings are quoted, while booleans and lists keep their YAML types, because ansible-core 2.19 rejects a quoted "false" in a conditional. The facts schema is in the docstring for the part that gathers them. Choices: - istio_platform from containerd's CNI bin_dirs: k3d for k3d-gpu's node image, which needs overlay 0002. - metallb_enabled false while ServiceLB is on. - kubernetes_cpu_policy best-effort when a k3s-addon balloons DaemonSet is present, Docker is rootless or containerd NRI is off. Otherwise nri-balloons, flagged as needing overlay 0003. Overlay.py makes that decision from this policy. - nfd_enabled false next to k3d-gpu's NFD. - require_amx from the AMXBF16 label. - The k3s CIDRs. - kata off on a bring-your-own cluster. - kubernetes_accelerator cpu, with a note to serve any GPU or Gaudi through k3d-gpu agent up. On the agent-toolkit layer: - agent_sandbox_enabled. - The model: EAT's default when the node has the 131 GiB EAT documents and k3d-gpu fit agrees, else the first documented 64 GiB fallback that fits, else none. --model overrides the pick. - The KV cache from fit, but only below the catalog's value. The catalog's memory is the pod's limit, and fit sizes against the host. Refusals exit 3 and leave nothing in --out: - ServiceLB disabled - EAT 1.x namespaces - k3d-gpu's own agent-sandbox, or any version other than EAT's v0.5.0 - a CNI layout no Istio profile matches - the erag layer: ERAG's Intel GPU flavour is experimental, and EI has no vllm-xpu runtime Floating refs exit 2: only a pins profile or a full SHA is accepted. integrations/eas/pins.json (k3d-gpu.eas-pins/v1, checked 2026-10-29) holds flat rows {profile, name, repo, ref, sha, date, source}: - profile main: EAS main 97f6157, EI main 3e8a244, and EAT release-2.0.0 036b662, the same tree as the agentic-toolkit-migration branch EAS main names - profile v1.2.0-rc.1: EAS 5c27fb3, EI release-1.1 170eb3a and EAT 036b662, as in the release notes - the bastion tools: kubectl v1.37.2, helm v3.22.1, yq v4.54.1, kubernetes.core 6.3.0, ansible 12.3.0, kubernetes 35.0.0, jsonpatch 1.38, jmespath 1.1.0, and ubuntu:24.04 by digest Every head was re-read with git ls-remote and gh api before this commit. A mutation pass of 19 mutants over the generator is killed by the config section. Both sections pass (220 checks).

fit sizes the KV cache for its own context, the --max-model-len in its flags (32768 at most). The generator takes only VLLM_CPU_KVCACHE_SPACE from fit, and EI's catalog sets no --max-model-len for qwen2-5-coder-14b, qwen2-5-coder-7b or llama-3-1-8b, so vLLM serves the model's own context. Llama-3.1-8B-Instruct's is 131072 tokens (max_position_embeddings), which at 128 KiB a token needs 16 GiB of KV cache for one request. vLLM v0.28.0 sets the KV cache from VLLM_CPU_KVCACHE_SPACE (vllm/platforms/cpu.py#L165-L169 at 2cf0a69) and will not start when one request at max_model_len does not fit it (vllm/v1/core/kv_cache_utils.py#L747-L767). The fit-missing and rootless-docker goldens pick llama-3-1-8b with an 8 GiB KV cache, which vLLM refuses. The four goldens that lower the KV cache now also carry fit's --max-model-len, as args_append on the model's entry in models.edit.yaml, with a choice line citing vLLM's check. The config section checks that every lowered KV cache has one, that the value comes from fit (16384 in a mutated fit document), and that a fit document with a KV cache and no --max-model-len exits 2. EI's NFD role skips the install only when it finds a Helm release named node-feature-discovery (roles/nfd/tasks/install.yaml#L15-L25 at 3e8a244). NFD's own docs install it as release nfd in the same namespace (docs/deployment/helm.md#L41 at kubernetes-sigs/node-feature-discovery b8c0974). The generator called any Helm release EI's own and kept nfd_enabled: true, so EI would install a second NFD next to it. The new check expects nfd_enabled: false for release nfd. Config section: 116 passed, 20 failed, each on these cases. The overlay section passes (87).

… only by name When k3d-gpu fit lowers a model's VLLM_CPU_KVCACHE_SPACE below the catalog's 40 GiB, the generator now also writes the --max-model-len fit sized that cache for, as args_append on the model's entry in models.edit.yaml, with a choice line citing vLLM's check (vllm/v1/core/kv_cache_utils.py#L747-L767 at 2cf0a69). Without it EI's catalog, which sets no --max-model-len for these models, leaves vLLM on the model's own context: 131072 tokens for Llama-3.1-8B-Instruct, 16 GiB of KV cache for one request, so an 8 GiB cache from fit stops vLLM v0.28.0 at startup. A fit document with a KV cache and no --max-model-len is malformed input (exit 2): fit's vLLM CPU flags always carry both (console/fit.py _flags). nfd_enabled stays true only when the NFD DaemonSet belongs to the Helm release node-feature-discovery, the one name EI's NFD role checks before it installs (roles/nfd/tasks/install.yaml#L15-L25 at 3e8a244). NFD from a Helm release of another name, such as nfd from NFD's own docs, now gets nfd_enabled: false, as k3d-gpu's baked NFD does. The docstring states how models.edit.yaml merges: defaults merge, entries match by name, env merges and args_append goes after the catalog's args. tests/eas_test.sh: 223 passed (overlay 87, config 136). ci_wiring_test.sh 70, eat_test.sh 64 and image_pins_test.sh 43 pass; claims_check.sh is clean and ledger_drift_check.py agrees; shellcheck v0.11.0 is clean.

Lands #248, #251, #250, #249, #247 and #243 in one merge.

88plug added 3 commits October 29, 2026 13:52

88plug mentioned this pull request Oct 29, 2026

docs: k3d-gpu eas, EAT 2.0 through EAS, upstream re-read; ledger F3 pre-registration #249

Merged

88plug added 2 commits October 29, 2026 14:30

88plug changed the base branch from claude/eas-eas-overlay to main October 30, 2026 00:17

88plug mentioned this pull request Oct 30, 2026

batch: land #248, #251, #250, #249, #247, #243 #262

Merged

88plug merged commit eff0826 into main Oct 30, 2026
1 check passed

88plug added a commit that referenced this pull request Oct 30, 2026

batch: land #248, #251, #250, #249, #247, #243 (#262)

6659ace

Lands #248, #251, #250, #249, #247 and #243 in one merge.

88plug mentioned this pull request Oct 30, 2026

Pin classes, pin-guard, scoped images-lock, tagged refs and tool pins out of workflows #265

Open

Navigation Menu

Repository navigation

eas: EAS and EAT config from cluster facts, the pins file, fixture tests - #251

eas: EAS and EAT config from cluster facts, the pins file, fixture tests#251
88plug merged 5 commits into
mainfrom
claude/eas-eas-config

88plug commented Oct 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

88plug commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it adds

Choices and their sources

Tests

For the stacked docs PR (#249)

Not verified

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

88plug commented Oct 29, 2026 •

edited

Loading