Skip to content

eas: k3d-gpu eas up, down, status and check (EAS with EAT 2.0 on a k3d-gpu cluster) - #260

Open
88plug wants to merge 13 commits into
mainfrom
claude/eas-eas-cli
Open

88plug wants to merge 13 commits into
mainfrom
claude/eas-eas-cli

Conversation

@88plug

@88plug 88plug commented Oct 29, 2026

Copy link
Copy Markdown
Owner

k3d-gpu eas up | down | status | check runs Intel's own installer for Enterprise AI Solutions (es_auto_installer.sh from intel/enterprise-ai-solutions) against a k3d-gpu cluster in its bring-your-own-cluster mode (existing_kubernetes). It installs the platform and inference layers, and with --agent-toolkit EAT 2.0 on top (es_auto_installer.sh install agent-toolkit). The installer runs in a pinned bastion container, never on the host.

This is the CLI part of full EAS support. It is stacked on #251 (eas-config, the choice generator), which is stacked on #248 (the patch overlay); the base of this PR is claude/eas-eas-config. The docs part is #249.

What eas up does

  1. Reads facts, read-only, into facts.json in the k3d-gpu.eas-facts/v1 schema that eas-config.py documents: nodes, namespaces, DaemonSets, the agent-sandbox CRD and controller image, the server version, rootless docker, the node container's NRI socket (docker exec ... test -S /var/run/nri/nri.sock), containerd's CNI binDirs (docker exec ... crictl info), MemAvailable, inotify limits, and k3d-gpu fit --engine vllm --device cpu --json for each of eas-config.py's model candidates. The candidates are read from eas-config.py's own CANDIDATES, so the two cannot drift.
  2. Resolves refs. A pins.json profile (--eas-ref v1.2.0-rc.1) or a full SHA goes to eas-config.py as given. Any other --eas-ref, --ei-ref or --eat-ref becomes a commit through git ls-remote (a branch before a tag, as EAS's _checkout_rev resolves; an annotated tag is peeled), once, and the result is printed.
  3. Runs eas-config.py. Exit 3 is a refusal: nothing is changed. The commits it resolved (the # eas|ei|eat lines of choices.txt) are recorded in pins.resolved.json.
  4. With --dry-run, clones EAS at the commit into the state dir and prints overlay.py's decisions (no --apply), then stops. It runs only get, version and config view against kubectl, and only info and the two read-only execs against docker.
  5. Otherwise: builds or reuses the bastion, tagged by the sha256 of its Dockerfile and lock, and runs integrations/eas/eas.sh up <layer> in it. That script clones EAS and checks out the commit (forced, cleaned), runs overlay.py --apply with kubernetes_cpu_policy from the overrides and the node count, and sets each manifest rev from repos.<layer>.yaml. It then runs init <layer> --env k3d-gpu (with --upgrade on a re-run, so a moved pin is accepted), merges global_config.edit.yaml and models.edit.yaml into env/k3d-gpu/, and runs install <layer> --env k3d-gpu -- -e @k3d-gpu-images.yaml -e @k3d-gpu-overrides.yaml [your args]. Extra vars load last in es_auto_installer.sh main() (#L1432-L1466), so the overrides win. configure is never run. When the cluster is absent, eas up runs k3d-gpu up first.
  6. Prints the gateway: the LoadBalancer IP of eg-gateway's Service, found by its gateway.envoyproxy.io/owning-gateway-name=eg-gateway label as EAS's own docs do. With the agent toolkit it also prints a ready curl --resolve litellm.<domain>:443:<ip> line. /etc/hosts is never edited.

eas down first sets the finalizers of every AIGatewayRoute to [] (EAS issue #71, open: the teardown hangs on them), then runs teardown platform --env k3d-gpu --force. The cluster stays. eas status prints the installed layer, the pins, the overlay decisions, EAS's own status, the gateway, and the pods that are not Ready. eas check refuses unless the agent toolkit is installed, since only that layer brings LiteLLM. When it is, it runs integrations/eat/eat-check.sh in a new EAT_MODE=eas:

  • the gateway is eg-gateway's address with Host litellm.<base_domain_name>;
  • a LiteLLM virtual key is minted from Secret litellm/litellm-master-key with POST /key/generate and deleted at exit (POST /key/delete), pass or fail;
  • the model defaults to the first chat model /v1/models lists;
  • the sandbox step is unchanged.

eat-check.sh now gives curl its keys as header files and its bodies on stdin in both modes, so no key is on an argument list. k3d-gpu sandbox up refuses while EAS's agent-sandbox-system/sandbox-router-deployment exists.

Choices that differ from the design, with the reason

  • The bastion runs as root (--user 0:0), not as the host uid:gid. es_auto_installer.sh ensure_sudo runs sudo -k 2>/dev/null under set -euo pipefail before every install, teardown and status, unless EUID is 0 (#L866-L875). As a non-root uid in an image without sudo, that function exits the installer with 127 before any playbook. I checked this by running the function verbatim with no sudo on PATH: rc=127. With sudo installed, an arbitrary host uid has no passwd entry, so sudo -k fails too. The container is throwaway; eas.sh gives the state dir back to the caller's uid:gid when it exits (EAS_OWNER). The bastion also sets git config --system safe.directory '*', because the checkout belongs to that uid and git refuses a repository another uid owns. docs: k3d-gpu eas, EAT 2.0 through EAS, upstream re-read; ledger F3 pre-registration #249's "your uid and gid" line should say this.
  • The packages ship share/k3d-gpu/integrations/{eas,eat/upstream/patches,eat/eat-check.sh}, not share/eas. overlay.py resolves overlay.json's patch_dirs (integrations/eas/patches, integrations/eat/upstream/patches) against the directory above integrations/. Keeping the repo layout lets the overlay and the upstream patches stay one copy. For the same reason the bastion mounts the whole integrations/ dir read-only at /k3d-gpu/integrations. eas_dir looks at K3D_GPU_EAS_DIR, then the checkout's integrations/eas, then ${SHARE_DIR}/integrations/eas. nfpm, the formula, nix and the AUR recipe all ship it, and packaging/check.sh nfpm installs and removes the deb and the rpm with it.
  • No --layer platform. eas-config.py takes inference, agent-toolkit and erag, and platform comes with either of the first two. --layer erag reaches eas-config.py, which refuses it with its reason.
  • The image pins are a separate extra-vars file. images.yaml pins agent_sandbox_router_image (:latest-main) and agent_sandbox_python_runtime_image (v0.5.0) by digest, read from the registries on 2026-10-29. It is passed before k3d-gpu-overrides.yaml, so eas-config's output still wins. EAT substitutes these values whole into the router manifest and the default SandboxTemplate (install.yaml#L131-L190), so a name:tag@sha256 reaches the pods as written. scripts/images-lock now collects integrations/eas (the bastion's FROM and every <name>_image: extra var).
  • models.edit.yaml's args_append (added to eas: EAS and EAT config from cluster facts, the pins file, fixture tests #251 after this branch started) goes after the catalog entry's args. Each flag it names, with its value, is first removed from the current args, because init never reseeds models.yaml. A second eas up therefore replaces --max-model-len instead of repeating it, and no args_append key reaches the catalog. I ran this with yq v4.54.1 on EI 3e8a244's real models.yaml, twice: 20 args both times, --max-model-len 32768 once.
  • known-issues.json's --skip-models row: nothing needs replacing on the inference layer, because EI's seeded config.yaml already sets llm_services_deploy_models: false (config.yaml#L94-L98). The agent toolkit's model deploys whatever that key says (install.yaml#L143-L161), so the row names -- -e '{"llm_services_autodeploy_models": []}', as docs: k3d-gpu eas, EAT 2.0 through EAS, upstream re-read; ledger F3 pre-registration #249 does.

Bastion pins

These match #251's pins.json bastion profile, and tests/eas_test.sh cli asserts that they do.

What Pin Source
Base ubuntu:24.04@sha256:fe815609… the EAT bastion's digest, still the tag's digest on 2026-10-29
kubectl v1.37.2, sha256 c0448752… dl.k8s.io .sha256; the minor of the Dockerfile's K3S_TAG v1.37.1-k3s1
helm v3.22.1, sha256 896cf636… get.helm.sh .sha256sum; the newest v3
yq v4.54.1, sha256 8e34fc29… the SHA-256 column of yq's checksums file
kubernetes.core 6.3.0, sha256 86a2a898… Galaxy's artifact sha256. It is installed with --force over the 6.2.0 copy that ansible 12.3.0 bundles, in the venv's purelib, where _ensure_collections looks (#L832-L847)
Python requirements.lock, 22 packages, every one hashed pip-compile --generate-hashes in the bastion's own base image (Python 3.12) from EAS requirements.txt at 97f6157: ansible 12.3.0, kubernetes 35.0.0, jsonpatch 1.38, jmespath 1.1.0. lock.sh regenerates it

I built the image locally (docker build, rc 0). Inside it, kubectl reports v1.37.2, helm v3.22.1+gd6c1a0e, yq v4.54.1, ansible-core 2.19.14, and kubernetes.core in the venv's purelib is 6.3.0.

Tests

The tests were written first: commit eas: tests for ... (failing). Results on the final commit:

  • tests/eas_test.sh: 353 passed, 0 failed (overlay, config and the new cli section, 130 checks). The cli section runs the real eas-config.py on the facts the launcher gathers, so a facts-schema mismatch fails the test. It stubs docker, kubectl, k3d, git, yq, k3d-gpu fit and overlay.py.
  • I applied 21 mutations to the launcher and to eas.sh, and the cli section failed on each. Examples: no --user 0:0, kubeconfig mounted rw, a secret passed by value, no finalizer clear, the dry run installing, overrides not last, no overlay, no env edits, no manifest pins, teardown without --force, a floating checkout, a floating ref passed to eas-config, a lost NRI or CNI fact, the sandbox refusal removed.
  • tests/eat_test.sh 86/0, tests/packaging_test.sh 150/0, tests/image_pins_test.sh 44/0, scripts/images-lock check and pins ok, tests/audit_test.sh 49/0 (goldens updated: images.lock grew from 55 to 57 images), tests/ci_wiring_test.sh 70/0, tests/kube_context_test.sh 38/0, tests/remote_test.sh 114/0, tests/quickstart_test.sh 174/0, tests/fuzz/props.py pass, tests/claims_check.sh clean, tests/ledger_drift_check.py 41 files agree, tests/platforms_doc_test.sh 195/0. On the pre-stack branch: tests/detect_test.sh 1185/0, tests/experimental_test.sh 122/0, tests/image_test.sh 26/0, tests/mcp_test.sh 72/0.
  • shellcheck v0.11.0 at warning severity is clean on every changed shell file.
  • packaging/check.sh nfpm: the deb and the rpm build, install, pass the installed checks (which now compare integrations/ byte for byte) and remove cleanly. packaging/check.sh brew and nix: results to follow in this description.

Wiring smoke (unscored, not graded)

Pending. The smoke is queued behind the shared live lock (~/k3dgpu-bench.lock), with throwaway cluster k3dgpu-eas-smoke, its own state dir and kubeconfig, deleted on every exit. It settles three questions: (1) does any bring-your-own task need sudo in the bastion, (2) do agent-sandbox v0.5.0's CRDs install on k3s v1.37, and (3) does the bastion reach the API server through the k3d kubeconfig. The result will be recorded here before merge.

Coordination

Not verified

The agent-toolkit layer has never run on k3s or k3d. Kubernetes 1.37 is outside what EAS tests, since it pins kubectl v1.34.3. Unless the smoke above says otherwise, no install has run from this code. F3 in #249 is the scored run.

tests/eas_test.sh gains a cli section. It runs the launcher's eas functions
and eas.sh against stub docker, kubectl, k3d, git, yq and k3d-gpu fit, with
the real eas-config.py, so the facts the launcher gathers are the facts the
generator reads, and a recording overlay.py. It covers the dry run (no
docker or kubectl change), cluster creation, the bastion's docker run, the
order of eas.sh's steps, ref resolution, secrets kept off argv and out of
logs, refusals, AIGatewayRoute finalizers before teardown (EAS issue #71),
status, the sandbox up refusal, and the bastion's pins against pins.json.

tests/eat_test.sh covers eat-check.sh's EAS mode: the eg-gateway address,
Host litellm.<domain>, a virtual key minted from litellm-master-key and
deleted after, and no key on curl's argument list in either mode.
tests/packaging_test.sh and tests/image_pins_test.sh cover integrations/eas
in the packages and in images.lock.
k3d-gpu eas runs Intel's es_auto_installer.sh (enterprise-ai-solutions)
against a k3d-gpu cluster in its bring-your-own-cluster mode, from a
bastion container, for the platform and inference layers and, with
--agent-toolkit, EAT 2.0 on top.

eas up gathers read-only facts in the schema eas-config.py reads, resolves
any ref that is not a pins.json profile or a commit with git ls-remote, runs
eas-config.py, and either prints the overlay decisions (--dry-run) or builds
the bastion and runs integrations/eas/eas.sh in it. eas.sh clones EAS at the
commit, applies the overlay, sets the layer manifest's revs, runs init,
merges the edit files into env/k3d-gpu and runs install with the image pins
and the overrides as the last extra vars. configure is never run. When the
cluster is absent, eas up runs k3d-gpu up first.

The bastion (integrations/eas/bastion.Dockerfile) is Ubuntu 24.04 at the EAT
bastion's digest with kubectl v1.37.2, helm v3.22.1 and yq v4.54.1 checked by
sha256, a hash-locked venv of EAS's requirements.txt at 97f6157 (ansible
12.3.0, kubernetes 35.0.0) and kubernetes.core 6.3.0 by sha256 over the copy
ansible bundles. It runs as root because es_auto_installer.sh's ensure_sudo
calls sudo -k under set -e unless EUID is 0 (#L866-L875); eas.sh hands the
state dir back to the invoking uid:gid on exit.

eas down clears AIGatewayRoute finalizers before teardown platform --force
(EAS issue #71). eas status prints the layer, pins, overlay decisions, EAS's
own status, the gateway and the pods that are not Ready. eas check runs
eat-check.sh in its new EAS mode: eg-gateway's address with Host
litellm.<domain>, a LiteLLM virtual key minted from litellm-master-key and
deleted after, then the unchanged sandbox step. eat-check.sh now hands keys
to curl as header files and request bodies on stdin in both modes.

sandbox up refuses while EAS's sandbox-router-deployment exists. The EAT
sandbox router and Python runtime images are pinned by digest in
integrations/eas/images.yaml, which images.lock now covers. The packages
ship integrations/eas, the overlay's patch drafts and eat-check.sh under
share/k3d-gpu/integrations/, the layout overlay.py's patch lookup needs.
known-issues.json records the upstream issues this path works around.
eas-config.py now gives a model whose KV cache it lowers an args_append
(fit's --max-model-len) next to its env. eas.sh merges the env into the
catalog entry as before and puts args_append after the entry's args, taking
each flag it names (and its value) out of the current args first. init
never reseeds models.yaml, so a second eas up replaces the flag instead of
repeating it, and no args_append key reaches the catalog. Checked with yq
v4.54.1 on EI 3e8a244's models.yaml, twice in a row.
…es it

packaging/check.sh brew installs the formula's url, the last aur-v release
tarball, which has no integrations/eas yet, so an assert_path_exists on it
fails until the next release. The install lines stay; brew skips the empty
glob with a warning on today's tarball.
A jq wrapper on the stub PATH logs each argv and runs the real jq. In EAS mode the
virtual key reached jq's argv when the check deleted it at exit (--arg k), visible to
ps for the life of that jq.
… is deleted

The /key/delete body was built with jq --arg, which put the run's virtual key on an
argument list. jq now reads it from env.VKEY. tests/eat_test.sh: 87 passed, 0 failed.
The base branch claude/eas-eas-config landed on main in #262. The one conflict was
the header comment of tests/eat_test.sh, which now lists both the EAS mode checks from
this branch and main's deploy_guardrails refusal.
…ailing)

The wiring smoke on cluster k3dgpu-eas-smoke stopped at the overlay, in --dry-run on the
host and in eas up in the bastion: git read-tree --index-output=/tmp/eas-overlay-*/index
said "fatal: unable to write new index file". git renames the index into that path from a
lock file next to the checkout's own index, so the path must be on the checkout's
filesystem. /tmp was a tmpfs with the state dir on disk, and in the bastion /tmp is the
container's while the state dir is a bind mount. The case runs with TMPDIR on /dev/shm
(or /var/tmp), wherever that is another filesystem than the test's work dir.
overlay.py now builds its scratch index with GIT_INDEX_FILE and a plain git read-tree.
git writes that file through a lock in the same directory, so the index can live in
/tmp whatever filesystem the EAS checkout is on. On the smoke's real checkout (state dir
on disk, /tmp a tmpfs) the decisions are 0002 apply, 0006 apply, 0003, 0004 and 0005
skip, as the F3 table predicts.
The second wiring smoke got through the platform and inference layers and 581 tasks of
the agent toolkit, then stopped at enterprise-inference's llm_services task "Enumerate
autodeploy models from the catalog" (roles/llm_services/tasks/deploy_models.yaml at
3e8a244): it pipes yq into jq on the installer host, and the bastion had no jq
("/bin/bash: line 2: jq: command not found", rc 127). EAS's configure does not install
jq either.
enterprise-inference 3e8a244 pipes yq into jq on the installer host to pick the models to
deploy (roles/llm_services/tasks/deploy_models.yaml#L108-L114), and EAS's configure
installs no jq. jq-1.8.2 (tag commit 34f7186, released 2026-06-20) is the newest; its
jq-linux-amd64 sha256 is the one jq's sha256sum.txt and GitHub's asset digest both give.
pins.json's bastion profile, the docs table and the README name it. In the new bastion,
EI's exact pipeline on the smoke's real models.yaml prints qwen2-5-coder-7b and
qwen3guard.
…ing)

The second wiring smoke's eas down exited 0 in 217 s, and it deleted k3d-gpu's own
node-feature-discovery namespace. EAS resolves teardown components by the same enabled
flags as install (filter_plugins/resolve_components.py), and eas.sh down passed none of
the overrides, so nfd_enabled fell back to its default of true and EI's nfd teardown
removed the namespace. Down must pass the image pins and the overrides as install does,
and refuse when the overrides file is gone.
eas.sh down now runs teardown platform with -e @k3d-gpu-images.yaml -e
@k3d-gpu-overrides.yaml, as install does, so the components eas up left to k3d-gpu (its
NFD) are left alone at teardown too. It refuses, before any teardown, when the
overrides file is gone. EAS passes these extra vars to its single BYO pass last
(es_auto_installer.sh#L1508-L1509 at 97f6157). docs/intel/eas.md says so.
88plug added a commit that referenced this pull request Oct 30, 2026
Written before any F3 step, after #260's second wiring smoke. keycloak-admin-secret holds
a 5-byte username next to its 24-byte password; the username is an account name, and a
count of it found 717 lines in the smoke's logs, which would grade any run REFUTED with
no secret shown. Step 7 counts the password key only and records that the username was
not counted. Step 7's own log, written while it counts, is left out of the count.
@88plug
88plug changed the base branch from claude/eas-eas-config to main October 30, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant