diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 00bf2c28..f8dc763a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,6 +47,7 @@ jobs: - run: tests/image_test.sh - run: tests/host_setup_test.sh - run: tests/eat_test.sh + - run: tests/eas_test.sh - run: tests/platforms_doc_test.sh - run: tests/reserved_cpus_test.sh - run: tests/experimental_test.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 3cd4ce42..7db356ff 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -93,6 +93,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57wb | H57db's hypothesis on wildct: inside EAT's banking flow (B24) with Gemma 4 26B-A4B QAT, k3d-gpu auto on the P-cores delivers far more tasks within budget than EAT stock CPU (vLLM CPU bf16 on the QAT bf16 weights). | Written 2026-10-20: not runnable here as registered (disk and memory), so INCONCLUSIVE (not run) is expected (0.95). | H57db's design and gates; runs only if at a hold's start / has >= 15 + 70 GB available (51.6 GB of bf16 weights plus images) and MemAvailable >= 100 GiB; otherwise INCONCLUSIVE (not run), with both readings recorded and no bf16 download. | PENDING | | F2a | Open WebUI in a k3d-gpu cluster reaches a public remote endpoint by name: `k3d-gpu quickstart --remote https://gaudi.farmgpu.net` (no hostAliases) gives a chat through Open WebUI's own API that returns. An earlier run saw `Temporary failure in name resolution` for gaudi.farmgpu.net from a pod. | Pre-registered 2026-10-17, before measuring: the in-cluster failure is a cluster DNS gap (CoreDNS forwarding to an upstream the node container cannot use), not the endpoint; once fixed, one `POST /api/chat/completions` returns `finish_reason=stop` with non-empty content (0.7). | Fresh cluster `k3dgpu-f2-*` (scratch KUBECONFIG, node CPUs 8-15), `nslookup` from a pod before any change, CoreDNS Corefile and node `/etc/resolv.conf` read; root cause fixed in the product if it is a k3d-gpu config gap (regression test first); then one chat through Open WebUI's API with the generated admin account, max_tokens <= 64, temperature 0.6. SUPPORTED: chat returns. REFUTED: no chat returns after the fix. Run (2026-10-17, i9-13900T, k3s v1.35.5, node CPUs 8-15): the prediction's mechanism was wrong. CoreDNS forwarding works (node resolv.conf is k3d's 192.168.32.1 DNAT to Docker's 127.0.0.11; busybox and glibc lookups from pods resolve). The failure is a startup race: `k3d-gpu up` returned on node Ready, when the CoreDNS Deployment was 1 s old (image still pulling); a glibc pod started then got `gaierror(-3, 'Temporary failure in name resolution')` for 11 s (4 tries), first success at 13 s. Fix: `up` waits for the CoreDNS rollout; after it, the same pod resolved on its first try (0.0 s). Then `quickstart --remote https://gaudi.farmgpu.net` with no hostAliases: Open WebUI listed `zai-org/GLM-5.3` and one `POST /api/chat/completions` returned finish_reason=stop, content 'OK', 0.41 s; 0 name resolution errors in its log. | SUPPORTED | | F2b | EAT's LiteLLM gateway serves the remote model registered by `integrations/eat/eat-remote.sh`, with the key only as an env reference. | Pre-registered 2026-10-17, before measuring: one completion through LiteLLM returns (0.7); the key string appears 0 times in the gateway DB dump, `kubectl describe` of the gateway Deployment/pods/Secret, and events (0.9). | `eat-remote.sh --remote https://gaudi.farmgpu.net --model zai-org/GLM-5.3` against EAT's genai-gateway in the same cluster, then one chat through LiteLLM (max_tokens <= 64, temperature 0.6), then `grep -c` of the key over `pg_dump`, `kubectl describe` and `kubectl get events -A`. SUPPORTED: completion returns and 0 key hits. REFUTED: any key hit, or no completion. Run (2026-10-17, same host, EAT release-1.0.3-hotfix, LiteLLM v1.83.14-stable.patch.2): first with eat-remote.sh as shipped, REFUTED: the chat came back 401 from the endpoint. LiteLLM resolves `os.environ/` only for config-file models; a `/model/new` model is loaded from its database with the reference string as the api_key (proxy_server.py `_add_deployment` decrypts only). Fixed eat-remote.sh to write a config-file `model_list` entry instead and delete the stale database entry; rerun: `eat-check.sh` passed (chat, tool call get_weather, sandbox printed 385), and the key (raw and base64) had 0 hits in `pg_dumpall` of the gateway's Postgres (169 KB), `kubectl describe` of the gateway namespace's Deployments, pods, Secrets and ConfigMaps, the Deployment YAML and `kubectl get events -A`. The verdict is for the fixed script. | SUPPORTED | +| F3 | `k3d-gpu eas up --agent-toolkit` takes a fresh, default k3d-gpu cluster on one non-Xeon host to Intel EAS's platform, inference and agent-toolkit (EAT 2.0) layers at the pinned upstream commits with no hand edits, and its own check and teardown work. | Pre-registered 2026-10-29, before any F3 step ([F3 pre-registration](#f3-pre-registration-i9-13900t-2026-10-29)): on the i9-13900T at the `main` profile pins (EAS `97f6157`, enterprise-inference `3e8a244`, EAT `release-2.0.0` `036b662`), `eas up` exits 0 with every PLAY RECAP `failed=0`; its choices and patch decisions equal the pre-registered table (`istio_platform: k3d`, MetalLB off, `best-effort`, NFD off, `require_amx: false`, k3s CIDRs, Kata off, accelerator `cpu`, agent sandbox on; 0002 and 0006 variant a applied, 0003, 0004 and 0005 skipped) and the model pick follows the fit rule; every pod not in phase Succeeded turns Ready; `eas check` passes through the EAS gateway; no secret value appears in a log or in sampled argv; `eas down` exits 0 within 15 min (0.35). | One fresh cluster `k3dgpu-f3-eas` from plain `k3d-gpu up`, then `eas up --agent-toolkit --dry-run`, `eas up --agent-toolkit`, a Ready wait, `eas check`, `eas status`, a secret scan, `eas down` and `k3d-gpu down`, each once, inside the host lock, from the first `main` commit that holds `k3d-gpu eas`. SUPPORTED, REFUTED and INCONCLUSIVE are fixed in the section. | PENDING | | H27x | H27's hypothesis on an AMX host (H27's "re-open for an AMX host"): on the Xeon 6741P (Granite Rapids, AMX), the <=60 s on-box probe (`bench/amx-autoquant/probe.sh`) picks the weight precision for agents better than the static int4 default. Same weights (`OpenVINO/Qwen3-4B-{int4,int8,fp16}-ov` at H27's commits), same parts (a) fidelity, (b) value, (c) cost. A replication: the original model is kept. | Written 2026-10-20, before any H27x measurement. (a) holds (0.85). (b) REFUTED (0.65, outcomes O1 + O2): the quality gate is a property of the weights (NLL at `INFERENCE_PRECISION_HINT=f32`), so int4 fails it again and the pick is int8, and decode stays bandwidth-bound, so int8 stays slower than int4 on the turn (outcome O2 again). (c) holds (0.9). Numbers: decode int4 25-45, int8 18-35, fp16 (bf16 on AMX) 10-22 tok/s; TTFT on 396 tokens 0.15-1.0 s for every arm; turn int4 2-4.5 s, int8 2.5-6 s, fp16 4-9 s. Priors: O1 0.05, O2 0.6, O3 0.05, O4 (int8 beats int4) 0.15, O5 0.15. | H27's row, outcome table and amendments A1-A3, unchanged except as listed in [the cross-host pre-registration](#cross-host-pre-registration-h25x-h27x-h28x-h34x-on-the-xeon-6741p-2026-10-20): OVMS containers on CPUs 0-23,48-71 (the CPUs outside `load.py`'s load set), `typical` only, weights downloaded at H27's pinned commits. Scorer `bench/amx-autoquant/summarize.py`. Run 2026-10-20 ([H27x run](#h27x-run-xeon-6741p-2026-10-20)): load MATCHED, int4 failed the gate again (dNLL +0.072), the probe and the reference both picked fp16 (bf16 on AMX; 5/5 rounds over int8, 3/3 repeats, 4 s per probe), and fp16's 4.23 s median turn is 1.24 s slower than the int4 default's 2.99 s (int4 faster in 5/5 rounds): (a) and (c) hold, (b) fails. | REFUTED | | H34x | H34's hypothesis on the Xeon 6741P: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, under matched background load. A replication: the original models are kept. | Written 2026-10-20, before any H34x measurement. At `typical`: median paired B/A decode tok/s 1.40 (1.15-1.70) and median paired B/A package J/token 0.75 (0.60-0.95); SUPPORTED by H34's rule (tok/s >= 1.20 with its CI above 1.0, J/token <= 0.85 with its CI below 1.0) at 0.6, REFUTED 0.25 (the draft's share of a faster target step grows), INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, unchanged except as listed in the cross-host pre-registration: cluster `k3dgpu-x-h34`, `K3D_GPU_VENDORS=none`, node container on CPUs 0-23,48-71, crescent amendment C1, `typical` only. Run 2026-10-20 ([H34x run](#h34x-run-xeon-6741p-2026-10-20)): load MATCHED, 12/12 pairs; B/A decode tok/s 0.758 (CI 0.688-0.809) and package J/token 1.241 (CI 1.168-1.347), B slower and costlier in 12/12 pairs (A 37.2, B 28.0 tok/s): the draft slows decode on this Xeon. | REFUTED | | H25x | H25's hypothesis (T part) on the Xeon 6741P: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: the original model and prefix shape are kept. | Written 2026-10-20, before any H25x measurement. Added-agent median TTFT OFF 1.2 s (0.5-2.5), ON 0.15 s (0.06-0.4), ratio 8x (4.5-14), the ratio's 95% CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.75; the risk is fixed per-request overhead, which a 5-10x faster prefill leaves as a larger share of the ON time. M (memory per agent) not measured, as in H25 run 4. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds, unchanged except as listed in the cross-host pre-registration: cluster `k3dgpu-x-h25`, `K3D_GPU_VENDORS=none`, node container on CPUs 0-23,48-71 (`NODE_CPUS`), crescent amendment C1, `typical` only. Run 2026-10-21 ([H25x run](#h25x-run-xeon-6741p-2026-10-21)): load MATCHED (amendment 27), no round void; added agent's first chunk 0.245 s ON vs 1.724 s OFF (7.05x, CI 6.94-7.13, n = 15 per arm), c/OFF 0.99, every round pair >= 4x: T SUPPORTED; M not measured (A3). | SUPPORTED | @@ -9446,6 +9447,125 @@ integrations/eat/eat-check.sh # expect: ==> E scripts/k3d-gpu down; bench/load/load.py stop ``` +### F3 pre-registration (i9-13900T, 2026-10-29) + +Written 2026-10-29 and committed before any F3 step. F3 is a functional check of +`k3d-gpu eas` ([docs/intel/eas.md](docs/intel/eas.md)): one scored run, graded only by +what is written here. Nothing below is edited after the run starts; a change is a dated +amendment written before the step it changes. + +**Why.** Gate 2 ran EAS's platform and inference layers on k3d once, by hand, at EAS +`e8f3ebc` and enterprise-inference `aaca1e5` +([upstream-eat.md](docs/intel/upstream-eat.md#test-plan)). The agent-toolkit layer (EAT +2.0, EAT #19) has not run on k3s or k3d in this repo's records. `k3d-gpu eas` makes every +k3d choice from what it reads in the cluster and applies the k3d patches as an overlay, +all read from source. F3 checks that whole path on one host, with nothing done by hand. + +**When.** F3 runs from the first `origin/main` commit that holds this section, the +`k3d-gpu eas` command (`cmd_eas` in `scripts/k3d-gpu`, `integrations/eas/`) and the +`eat-up.sh` move to `release-1.1.0`; that SHA is the run header's `k3d-gpu commit`. The +`eas` CLI PR's unscored wiring smoke on a throwaway cluster comes first and is recorded in +that PR, not here; anything it changed in the product is in that commit. If +`integrations/eas/pins.json` at that commit names other SHAs than the pins below, F3 does +not start until an amendment names the new ones. + +- **Pins (profile `main`).** EAS `97f6157d6a72f7173c698c474d4e51d2c79f1fd6`, + enterprise-inference `3e8a24486b6ee812adf2bf452f8ba5751c16c8bd`, EAT `release-2.0.0` + `036b66234801e9bd950afd90830821859780d668`. No `--eas-ref`, `--ei-ref`, `--eat-ref` or + `--model`. +- **Host.** The shared i9-13900T (8 P-cores, 16 E-cores, 32 CPUs, 125.4 GiB MemTotal, no + AMX, no AVX-512, UHD 770). Its sysctls stay as they are; on 2026-10-29 they were + `fs.inotify.max_user_instances=1024` and `max_user_watches=524288`, below EAT's + 8192/1048576, and `eas up` is expected to warn and print the line, not apply it. No + synthetic background load: F3 checks function, not speed. Other tenants keep running; + `uptime`, `MemAvailable` and the count of running containers are recorded at the start + and end of the lock. +- **One fresh cluster.** `K3D_GPU_CLUSTER=k3dgpu-f3-eas`, created by this run and deleted + after it, through its own kubeconfig (never `~/.kube/config`). Plain `k3d-gpu up`: no + `K3D_GPU_K3S_ARGS`, `K3D_GPU_BALLOONS` or `K3D_GPU_NODE_LABELS`. `HF_TOKEN` is set in the + environment if the operator has one, and recorded only as set or unset. Every step runs + once; nothing is retried; no file in the EAS state dir and no object in the cluster is + edited by hand at any point. +- **Steps, each timed (wall s) and logged in order, exit code recorded.** (0) Before the + lock: `docker build` of the node image at the deploy commit, and `git ls-remote` of the + three pins (recorded; a moved branch head does not matter, the SHAs are pinned). + (1) `k3d-gpu up`. (2) `k3d-gpu eas up --agent-toolkit --dry-run`; `choices.txt` and + `overlay.txt` copied into the record. (3) `k3d-gpu eas up --agent-toolkit`, full log, + with the bastion image tag it built or reused. (4) `kubectl wait --for=condition=Ready + pod -A --field-selector=status.phase!=Succeeded --timeout=30m`, output verbatim, then + every Job's condition on its own line and the pod list. (5) `k3d-gpu eas check`, output + verbatim. (6) `k3d-gpu eas status`, recorded, not graded. (7) Secret scan, below. + (8) `k3d-gpu eas down`, then `kubectl get ns` recorded. (9) `k3d-gpu down`. During + steps 3, 5 and 8, `ps -eo args` is sampled every 2 s into a scan file. +- **Secret scan (step 7).** The values are read from the cluster: Secret + `litellm-master-key` (namespace `litellm`), Secret `sandbox-router-auth` + (`agent-sandbox-system`), Secret `keycloak-admin-secret` if the install made one, and + `HF_TOKEN` if set. Each value, raw and base64, is counted with a fixed-string `grep -c` + over the step logs, the `ps` samples and every file under + `${STATE_DIR}/eas-k3dgpu-f3-eas/logs/`. Only the counts are recorded; no value is + printed, and a file with a hit is deleted after counting. +- **Lock.** `flock -w 21600 /live.lock`, at most 180 min from taking it to + releasing it. Step 3 is stopped 100 min after it starts. The cluster is deleted before + release, also on failure. + +**Predicted choices on this host (from step 2, graded).** Each is a value in `choices.txt` +or the overrides, not a wording: + +| Fact `eas up` should read | Choice | +|---|---| +| a k3d-gpu cluster | `existing_kubernetes` set to the mounted per-cluster kubeconfig | +| node CNI `bin_dirs` `["/usr/bin"]` | `istio_platform: k3d` | +| ServiceLB on | `metallb_enabled: false` | +| k3d-gpu's baked hybrid balloons DaemonSet | `kubernetes_cpu_policy: best-effort` | +| k3d-gpu's NFD in `node-feature-discovery` | `nfd_enabled: false` | +| no AMXBF16 label | `require_amx: false` | +| no CIDR flags in `k3s.io/node-args` | `kubernetes_cluster_cidr: 10.42.0.0/16`, `kubernetes_service_cidr: 10.43.0.0/16` | +| BYO cluster | `kata_containers_enabled: false` | +| `gpu.intel.com/i915` allocatable | `kubernetes_accelerator: cpu`, with the line pointing at `k3d-gpu agent up` | +| no agent-sandbox CRDs | `agent_sandbox_enabled: true` | +| 24 cores; inotify limits as read at step 2 | a core-count warning; an inotify warning with the `sysctl` line if either limit is below 8192/1048576; the host unchanged | +| overlay at EAS `97f6157` | 0002 apply, 0006 apply (variant a), 0003 skip, 0004 skip, 0005 skip | + +The model is recorded and graded against the rule, not a name: EAT's default needs about +131 GiB, more than this node has, so the pick must be the first of `qwen2-5-coder-7b`, +`llama-3-1-8b` that `k3d-gpu fit` says fits, with the node allocatable memory and the fit +JSON it came from recorded. Predicted: `qwen2-5-coder-7b`. + +**Grading, fixed now.** + +- **SUPPORTED** only if all hold: steps 1, 2 and 3 exit 0 and every `PLAY RECAP` line in + step 3's log reads `failed=0`; step 2's choices and overlay decisions equal the table + above, and step 3 applied the same; the model pick follows the rule; step 4 exits 0; + step 5 exits 0 (its chat through the EAS gateway returns HTTP 200 with non-empty + content, and its sandbox step passes); every count in step 7 is 0; step 8 exits 0 within + 15 min; and no step needed a hand edit, a retry, or a flag not named here. +- **REFUTED** if any of: step 1, 2 or 3 exits non-zero, or the overlay STOPs; a choice or + decision differs from the table (recorded; the run goes on so the rest is observed); + step 4 ends non-zero with a pod owned by a Deployment, StatefulSet, DaemonSet, + ReplicaSet or Sandbox not Ready; step 3 stopped at 100 min with a pod Pending on + insufficient CPU or memory, or in CrashLoopBackOff; step 5 fails while every such pod + is Ready; any count in step 7 above 0; step 8 non-zero or over 15 min. +- **INCONCLUSIVE** if: the rule picks no model (`llm_services_autodeploy_models: []`) + because `fit` says none fits at that moment, so no chat is possible (`MemAvailable` and + the fit JSON recorded); an image or model download fails with a registry or Hugging Face + error (HTTP 429 or 5xx, recorded verbatim); step 3 is stopped at 100 min while every pod + not Ready is still pulling an image or downloading a model; step 4 ends non-zero and + every pod not Ready is a Job pod; or the lock runs out before step 8. +- **Precedence** (written 2026-10-29 in review, before any F3 step). A REFUTED condition + that follows only from an INCONCLUSIVE one grades INCONCLUSIVE: step 3's non-zero exit + when it was stopped at 100 min while every pod not Ready was still pulling an image or + downloading a model; a failure of any step caused only by the recorded registry or + Hugging Face error; and step 5's failure when the rule picked no model. Every other + REFUTED condition, a choice or decision that differs from the table above included, + grades REFUTED whatever else happened. + +**Prediction.** SUPPORTED. Prior P(SUPPORTED) = 0.35. The open risks, none observed yet: +agent-sandbox v0.5.0 on Kubernetes 1.37 (EAS tests 1.34.3); the CPU requests of the +whole stack, Qwen3Guard's 8 included, on a 32-CPU node; helm v3.22.1 with +`kubernetes.core` 6.3.0; a BYO task that needs `sudo` inside the bastion; inotify at 1024 +instances with this many pods; EAS #71 at teardown; and host memory taken by other +tenants. + ### H2 run (i9-13900T, 2026-10-16) ```text diff --git a/README.md b/README.md index b112ab4f..07552052 100644 --- a/README.md +++ b/README.md @@ -1403,9 +1403,18 @@ the in-process stub in `bench/test_agent_trace.py`. node through EAT's own brownfield path, on a non-Xeon client CPU. Its vLLM pod joins the hybrid balloons policy's `performance` balloon through EAT's own chart annotation, so EAT's CPU balloons stay off and `hybrid` stays on. It deploys the -pinned tag `release-1.0.3-hotfix` (the latest, `release-1.1.0`, adds OpenShell; -`eat-up.sh` refuses `deploy_openshell=on` because its pinned tag has no OpenShell -role). See [docs/platforms.md](docs/platforms.md). +EAT 1.x tag `release-1.1.0` with the guardrail off; `eat-up.sh` refuses +`deploy_openshell=on` because OpenShell has not been tried on k3d-gpu. See +[docs/platforms.md](docs/platforms.md#intel-enterprise-agent-toolkit-eat). + +EAT 2.0 installs through Intel's AI for Enterprise Solutions (EAS) installer instead. +`k3d-gpu eas up --agent-toolkit` runs it against the k3d-gpu cluster in EAS's +bring-your-own-cluster mode, choosing each k3d setting from the cluster and applying the +k3d patches EAS has not merged yet. It covers EAS's platform, inference and agent-toolkit +layers on the Xeon CPU path, where EAS and EAT ship them today; EAS's Enterprise RAG +layer is not wired in yet, and an Arc or Gaudi card stays with `k3d-gpu agent up`. It has +not run end to end yet: F3 in [EXPERIMENTS.md](EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29) is PENDING. +See [docs/intel/eas.md](docs/intel/eas.md). ### Experimental features (auto-adapting) diff --git a/docs/intel.md b/docs/intel.md index 539091ee..940c387f 100644 --- a/docs/intel.md +++ b/docs/intel.md @@ -135,14 +135,25 @@ moves a row to verified. Please send its output. balloons policy) onto Kubernetes. It targets Xeon servers. We read it at [`0ecf67a`](https://github.com/intel/enterprise-agent-toolkit/tree/0ecf67a); a k3d-gpu cluster is a candidate bring-your-own target for a single developer -machine. [`integrations/eat/eat-up.sh`](../integrations/eat/eat-up.sh) deploys the -pinned tag `release-1.0.3-hotfix` -([`0d3438d`](https://github.com/intel/enterprise-agent-toolkit/tree/0d3438d7119de44cafc5903f66665c0b623e3333); -[tags](https://api.github.com/repos/intel/enterprise-agent-toolkit/tags), checked -2026-10-28). The latest, `release-1.1.0`, adds OpenShell -([PR #14](https://github.com/intel/enterprise-agent-toolkit/pull/14), -`deploy_openshell=off` by default); `eat-up.sh` refuses `deploy_openshell=on` because -its pinned tag has no OpenShell role. Pins and upstream changes, Release 1.1.0 +machine. There are two paths, one per EAT generation: + +- **EAT 1.x.** [`integrations/eat/eat-up.sh`](../integrations/eat/eat-up.sh) deploys the + tag `release-1.1.0` + ([`0c726f5`](https://github.com/intel/enterprise-agent-toolkit/tree/0c726f576595971c53cc45b94e08ba969beb4a4a); + [tags](https://api.github.com/repos/intel/enterprise-agent-toolkit/tags), checked + 2026-10-29) through EAT's own brownfield path, with the guardrail off and the vLLM CPU + image pinned by digest. `release-1.1.0` adds OpenShell + ([PR #14](https://github.com/intel/enterprise-agent-toolkit/pull/14), + `deploy_openshell=off` by default); `eat-up.sh` refuses `deploy_openshell=on` because + OpenShell has not been tried on a k3d-gpu cluster. +- **EAT 2.0.** EAT [PR #19](https://github.com/intel/enterprise-agent-toolkit/pull/19) + moved the toolkit into Intel's EAS installer (`es_auto_installer.sh install + agent-toolkit`). `k3d-gpu eas up --agent-toolkit` runs that installer against the + cluster, with every k3d choice made from what it reads there: + [intel/eas.md](intel/eas.md). It has not run end to end yet; F3 in + [EXPERIMENTS.md](../EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29) is PENDING. + +Pins and upstream changes, Release 1.1.0 ([PR #21](https://github.com/intel/enterprise-agent-toolkit/pull/21)) included: [platforms.md](platforms.md#intel-enterprise-agent-toolkit-eat). @@ -177,20 +188,29 @@ i9-13900T [ledger:H18]. No Xeon run backs it yet. EAT's defaults are read from `release-1.0.3-hotfix`: `docs/prerequisites.md`, `docs/single-node-deployment.md`, `docker/README.md` and `core/helm-charts/` [provenance:P303]. The k3d-gpu column is what the launcher does today; where it has -run on hardware, the row says which host. +run on hardware, the row says which host. `release-1.1.0`, which `eat-up.sh` deploys now, +moves the vLLM CPU image to `v0.28.0` and turns the Qwen3Guard guardrail on by default +([platforms.md](platforms.md#intel-enterprise-agent-toolkit-eat)). EAT 2.0 keeps the same +hardware minimums +([prerequisites.md#L5-L9](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/prerequisites.md#L5-L9)) +and installs through EAS. | | EAT by default | With k3d-gpu | |---|---|---| | Hosts | Xeon. Minimum 48 cores and 32 GB RAM, recommended 96+ cores and 64 GB; Docker mode asks for 96+ cores and 128 GB | Any Intel host it detects: client Core and Core Ultra, Xeon, Arc, Gaudi (detection and config fixture-tested per family, [platforms.md](platforms.md)). EAT ran end to end on one i9-13900T with no AMX and no AVX-512, n=1 [ledger:H1b] | | OS | Ubuntu 22.04 / 24.04, x86_64 | Any Linux with Docker that runs k3d; `host-setup` prints pinned driver steps for Ubuntu 24.04/26.04, Fedora, Arch and Manjaro | | Cluster setup | Kubespray onto the host, or brownfield on an existing cluster; the `ei-*-eligible` labels come only from the Kubespray path | One `k3d-gpu up`; `K3D_GPU_NODE_LABELS` sets EAT's labels, `K3D_GPU_BALLOONS=external` hands the NRI policy to EAT [ledger:H14] | +| EAT 2.0 install | EAS `es_auto_installer.sh install agent-toolkit`, onto Kubespray or an existing cluster (`existing_kubernetes`) | `k3d-gpu eas up --agent-toolkit`: the existing-cluster mode, with the k3d settings and patches chosen from the cluster ([intel/eas.md](intel/eas.md)). Not run yet; F3 is PENDING | | Engines | vLLM CPU (`vllm/vllm-openai-cpu:v0.24.0`, bf16), 9 model profiles in the single-node table (7 chat, 1 embedding, 1 reranker). An OVMS Helm chart ships but no default path deploys it: `openvino/model_server:2025.4`, plus an unpinned `latest-py` tag for its model-export containers | `agent up` picks per host: vLLM on a Gaudi, OVMS 2026.4.1 on an XMX GPU, else OVMS on the P-cores; vLLM XPU (`v0.31.0`) or llm-scaler on Arc by flag. Its images are pinned by digest in `images.lock` | | Accelerators | None requested: no `gpu.intel.com`, `habana.ai` or `/dev/dri` on the default path | Intel GPU, NPU, Gaudi, DSA, IAA and QAT plugins started per detected device, and `up` fails unless each is schedulable | | CPU placement | NRI balloons sized for Xeon NUMA; on a hybrid CPU EAT's reservation needs the patch in [`contrib/eat/`](../contrib/eat) | Balloons baked for hybrid P/E and multi-NUMA hosts; vLLM on P-cores and agent services on E-cores through `contrib/eat/` values | -Not covered by either column: EAT's XPU path for Arc B60/B70, which shipped in -`release-1.1.0`, not in the pinned tag -([platforms.md](platforms.md#eat-pull-requests-checked-2026-10-28)). +Not covered by either column: EAT's XPU path for Arc B60/B70. It ships in +`release-1.1.0`, but `eat-up.sh` deploys the CPU path +([platforms.md](platforms.md#eat-pull-requests-checked-2026-10-28)), and EAT 2.0 leaves +Arc out ("Not in this release: ... Intel® Arc™ (XPU) deployment", +[README.md#L36](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/README.md#L36)). +On k3d-gpu an Arc card is served by `k3d-gpu agent up`. **In progress.** A campaign on one Xeon 6741P (Granite Rapids, AMX) host with 4x Arc Pro B70 checks four things: that `detect` and `up` pick up the Xeon and the diff --git a/docs/intel/eas.md b/docs/intel/eas.md new file mode 100644 index 00000000..0bf4b4b1 --- /dev/null +++ b/docs/intel/eas.md @@ -0,0 +1,306 @@ +# Intel EAS on k3d-gpu (`k3d-gpu eas`) + +> **Status: built from upstream source, not yet run end to end.** The scored run is +> F3 in [EXPERIMENTS.md](../../EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29), +> pre-registered on 2026-10-29 and PENDING. Until F3 has run, nothing on this page says +> that EAS works on k3d-gpu. The EAT 2.0 layer has never run on k3s or k3d in this +> repo's records: Gate 2 covered EAS's platform and inference layers only +> ([upstream-eat.md](upstream-eat.md#test-plan)). + +Intel® AI for Enterprise Solutions ([intel/enterprise-ai-solutions](https://github.com/intel/enterprise-ai-solutions), +EAS) installs Intel's AI stack in layers with one script, `es_auto_installer.sh`. Since +EAT PR [#19](https://github.com/intel/enterprise-agent-toolkit/pull/19) and EAS PR +[#66](https://github.com/intel/enterprise-ai-solutions/pull/66), the Enterprise Agent +Toolkit (EAT 2.0) is one of those layers: `es_auto_installer.sh install agent-toolkit`. +EAS has a bring-your-own-cluster mode, `existing_kubernetes`, that skips Kubespray; a +k3d-gpu cluster is such a cluster. + +`k3d-gpu eas` drives that mode: it reads the cluster, picks every EAS setting that a k3d +cluster needs, applies the k3d patches EAS has not merged yet, and runs the installer +from a pinned bastion container. EAT 1.x (the `deploy-agentic-stack.sh` releases) stays on +[`integrations/eat/eat-up.sh`](../../integrations/eat/README.md). + +## What "full EAS" covers + +| Layer | `es_auto_installer.sh` | `k3d-gpu eas up` | +|---|---|---| +| platform (Istio, Envoy Gateway, cert-manager, PostgreSQL, Keycloak, observability) | `install platform` | always | +| inference (model-manager and its vLLM CPU models, KServe, Envoy AI Gateway, NFD; models behind a Keycloak JWT by default, `auth_type: jwt`) | `install inference` | always | +| agent-toolkit (EAT 2.0: Redis Stack, pgvector, agent sandbox; sets `auth_type: litellm_vkey`, which brings LiteLLM with its guardrail and hybrid router, and Langfuse) | `install agent-toolkit` | `--agent-toolkit` | +| erag (Intel Enterprise RAG) | `install erag` | not yet, see [Limits](#limits) | + +LiteLLM and Langfuse are inference-layer components that EAS installs only when +`auth_type` is `litellm_vkey` +([EI components.yaml#L42-L53](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/components.yaml#L42-L53); +EI's default is `jwt`, +[config.yaml#L44](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/config.yaml#L44)), +and the agent-toolkit layer sets it +([EAT config.yaml#L24](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L24)). +Gate 2 served its model at `inference.` with a Keycloak token +([upstream-eat.md, probe 5](upstream-eat.md#test-plan)). + +The platform, inference and agent-toolkit layers run on the Xeon CPU path. +`kubernetes_accelerator: "cpu"` is the only value with behaviour in EAS today +([global_config.yaml#L106](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/configs/defaults/global_config.yaml#L106); +"Intel GPUs: intel_gpu_operator ... is not available on this branch yet", +[nvidia_gpu_operator/tasks/install.yaml#L20](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/nvidia_gpu_operator/tasks/install.yaml#L20)), +and EAT 2.0 lists "Intel® Arc™ (XPU) deployment" as not in the release +([README.md#L36](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/README.md#L36)). + +## Commands + +```bash +k3d-gpu eas up [--agent-toolkit] [--eas-ref R] [--ei-ref R] [--eat-ref R] \ + [--model NAME] [--dry-run] [--explain] [-- ] +k3d-gpu eas status # EAS status, pins, overlay decisions, gateway IP, pods not Ready +k3d-gpu eas check # with --agent-toolkit: chat through the EAS gateway with a minted LiteLLM key, then the sandbox step +k3d-gpu eas down # EAS teardown; the cluster stays (k3d-gpu down removes it) +``` + +- The cluster is `K3D_GPU_CLUSTER` and its own kubeconfig, `${STATE_DIR}/kubeconfig-`. + When the cluster does not exist, `eas up` runs `k3d-gpu up` first, so EAS is one command + on an Intel host. +- `--dry-run` reads the cluster, prints every choice and patch decision, and writes the + files it would use under the state dir. It changes nothing in the cluster. +- Every automatic choice prints as `fact -> choice [source]`, with a short source + (`repo@sha7 path:line`). `--explain` prints the full permalink instead. +- Arguments after `--` go to `ansible-playbook`, for example + `k3d-gpu eas up --agent-toolkit -- -e '{"llm_services_autodeploy_models": []}'` to + install without EAT's chat model. + +What `eas up` does, in order: + +1. Reads the cluster, read-only: node labels and allocatable CPU, memory and devices; the + `k3s.io/node-args` annotation (ServiceLB, CIDRs); any `nri-resource-policy-balloons` + DaemonSet and its owner; NFD; the agent-sandbox CRDs; EAT 1.x namespaces; the server + version, rootless Docker, host `MemAvailable` and inotify limits; and + `k3d-gpu fit --model --engine vllm --device cpu --json` per model candidate. +2. Writes the choices: an ansible extra-vars file loaded last (`k3d-gpu-overrides.yaml`), + the `existing_kubernetes` line in the env's `global_config.yaml` (the installer reads it + there with `yq`, + [es_auto_installer.sh#L1342-L1358](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1342-L1358)), + the `models.yaml` edits model-manager reads, the layer manifest's `rev:` rewritten to + SHAs, and `choices.txt`. +3. Builds or reuses the bastion image (Ubuntu 24.04, kubectl, helm, yq, a hashed Python + lock and `kubernetes.core`), tagged by the hash of its Dockerfile and lock. +4. In the bastion (host network, your uid and gid, the kubeconfig read-only): clones EAS + at the pinned SHA, applies the [overlay](#patch-overlay), pins the layer manifest to + SHAs ([`_checkout_rev` accepts one](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L658-L693)), + then runs `init --env k3d-gpu` and + `install --env k3d-gpu -- -e @/eas-state/k3d-gpu-overrides.yaml`. + `configure` is skipped: it needs a terminal or `--force`, `sudo` and apt + ([#L318-L352](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L318-L352)), + and every tool it installs is already in the image. The installer reuses a venv that has + `ansible-playbook` ([#L801-L806](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L801-L806)). +5. Prints the gateway: the address of the Gateway `eg-gateway` in `envoy-gateway-system` + (its `status.addresses`). No Service is named `eg-gateway`: EAS sets no Service name, so + Envoy Gateway names the proxy's LoadBalancer Service `envoy-` plus the Gateway's + namespace, name and a hash + ([gateway v1.7.1 resource_provider.go#L299-L304](https://github.com/envoyproxy/gateway/blob/3f75d826cc1dbd3d66632abbd45f96388d226c1d/internal/infrastructure/kubernetes/proxy/resource_provider.go#L299-L304)); + k3s ServiceLB gives it the node IP. With `--agent-toolkit` it also prints a ready + `curl --resolve litellm.:443:` line. It never edits `/etc/hosts`. + +## Automatic choices + +Each row is one line of `choices.txt`. Sources are pinned to the commits in [Pins](#pins). + +| Fact read from the cluster | Choice | Source | +|---|---|---| +| A k3d-gpu cluster | `existing_kubernetes: /kube/config`, the per-cluster kubeconfig mounted read-only | [EAS global_config.yaml#L14](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/configs/defaults/global_config.yaml#L14); [es_auto_installer.sh#L1460-L1466](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1460-L1466) | +| k3d-gpu node image: containerd CNI `bin_dirs` is `["/usr/bin"]` | `istio_platform: k3d` (stock k3s needs `k3s`); needs overlay 0002 | [upstream-eat.md, Gate 2 probes 1 and 2](upstream-eat.md#test-plan) | +| ServiceLB on (no `--disable=servicelb` in `k3s.io/node-args`) | `metallb_enabled: false` | [EAS components.yaml#L100-L102](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/configs/components.yaml#L100-L102) | +| A balloons DaemonSet already runs (k3d-gpu baked it for a hybrid or multi-NUMA host), or Docker is rootless | `kubernetes_cpu_policy: best-effort` | [EAS preflight.yaml#L212-L270](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/playbooks/includes/preflight.yaml#L212-L270); Gate 2 probe 6 | +| No NRI policy on the node, NRI available (`K3D_GPU_BALLOONS=external`, or nothing baked) | `kubernetes_cpu_policy: nri-balloons`; needs overlay 0003 | same; [ledger:H14] for EAT's chart values on such a cluster | +| k3d-gpu's NFD runs in `node-feature-discovery` (a k3s manifest, not a Helm release) | `nfd_enabled: false`: EI looks only for a Helm release of that name and would install a second NFD | [EI roles/nfd/tasks/install.yaml#L15-L25](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/roles/nfd/tasks/install.yaml#L15-L25) | +| `feature.node.kubernetes.io/cpu-cpuid.AMXBF16=true` on the node | `require_amx: true` in `models.yaml` | [EI models.yaml#L20](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/model_manager/models.yaml#L20) | +| No AMXBF16 label | `require_amx: false`, so model pods are not left Pending on the AMX node affinity | [EI core/renderer.py#L90-L104](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/model_manager/core/renderer.py#L90-L104); Gate 2 probe 5 | +| No AMX | `VLLM_CPU_SGL_KERNEL` left as shipped: vLLM v0.28.0 checks AMX itself | [vLLM utils.py#L238-L244](https://github.com/vllm-project/vllm/blob/2cf0a6915ce544dc493a0990f2ea38d81601128a/vllm/model_executor/layers/utils.py#L238-L244) | +| No `--cluster-cidr` or `--service-cidr` in `k3s.io/node-args` | `kubernetes_cluster_cidr: 10.42.0.0/16`, `kubernetes_service_cidr: 10.43.0.0/16` (EI's LiteLLM falls back to `10.233.64.0/18` for the pod network it admits, which EAT turns on with OpenShell and asks for on a BYO cluster) | [k3s server.go#L139-L144 at v1.37.1+k3s1](https://github.com/k3s-io/k3s/blob/356c025475bbefaa09cf0d7e48e8e2adbbde57db/pkg/cli/cmds/server.go#L139-L144); [EI litellm defaults#L65](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/roles/litellm/defaults/main.yaml#L65); [EAT config.yaml#L61-L64](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L61-L64) | +| A BYO cluster | `kata_containers_enabled: false`, set explicitly (EAS's default is already `false`): the role fails on BYO | [EAS kata_containers/tasks/install.yaml#L9-L22](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/kata_containers/tasks/install.yaml#L9-L22) | +| `gpu.intel.com/*` or `habana.ai/gaudi` allocatable | `kubernetes_accelerator: cpu` stays, and the line says "serve the device with k3d-gpu agent up" | [EAT README.md#L36](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/README.md#L36); [EAS nvidia_gpu_operator install.yaml#L20](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/nvidia_gpu_operator/tasks/install.yaml#L20) | +| `--agent-toolkit`, no agent-sandbox CRDs | `agent_sandbox_enabled: true`: EAT installs agent-sandbox `v0.5.0` | [EAT config.yaml#L54-L56](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L54-L56) | +| `--agent-toolkit`: EAT's router image is the floating `:latest-main` | `agent_sandbox_router_image` and `agent_sandbox_python_runtime_image` pinned by digest in the overrides | [EAT agent_sandbox defaults#L27-L28](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/roles/agent_sandbox/defaults/main.yaml#L27-L28) | +| `--agent-toolkit`: node allocatable memory and `fit` cover EAT's default (about 131 GiB: `qwen2-5-coder-14b` 96, Qwen3Guard 16, router 2, the rest 17) | keep the default model | [EAT single-node-deployment.md#L84](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/single-node-deployment.md#L84) | +| They do not | the first EAT-documented 64 GiB model that `fit` says fits: `qwen2-5-coder-7b`, then `llama-3-1-8b`; its `VLLM_CPU_KVCACHE_SPACE` comes from `fit` | same line | +| Nothing fits | `llm_services_autodeploy_models: []`, with the reason | [EAT config.yaml#L32-L35](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L32-L35) | +| `--model NAME` | that model, whatever the fit says | | +| `--agent-toolkit` | `guardrails_enabled` and `hybrid_router_enabled` keep EAT's defaults (on) | [EAT config.yaml#L25-L28](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L25-L28) | + +## Refusals + +`eas up` stops before it changes the cluster when: + +| Condition | Message | Why | +|---|---|---| +| ServiceLB is off (`--disable=servicelb`, the EAT 1.x advice) | recreate the cluster without `--disable=servicelb` | the gateway's Envoy Service needs a LoadBalancer address, which k3s ServiceLB gives on a k3d node; MetalLB on k3d's Docker network is not part of this path | +| EAT 1.x namespaces from `eat-up.sh` exist (`genai-gateway`, its `ingress-nginx`) | remove the EAT 1.x stack first | both stacks would own LiteLLM and ingress | +| k3d-gpu's agent-sandbox v1.0.8 runs (`k3d-gpu sandbox up` was used) and `--agent-toolkit` is set | run `k3d-gpu sandbox down` first | EAT applies agent-sandbox v0.5.0's manifests, which would replace v1.0.8's CRDs and controller with an older release | +| `--layer erag` | ERAG is not wired into `k3d-gpu eas` yet; its Intel GPU flavour also needs an EI `vllm-xpu` runtime that no EI branch has | see [Limits](#limits) | +| A patch in the overlay neither applies nor is found merged or fixed | names the patch, the upstream PR that moved its hunk, and "regenerate" | see [Patch overlay](#patch-overlay) | + +The other direction holds too: `k3d-gpu sandbox up` refuses while EAS's +`sandbox-router-deployment` exists. + +## Patch overlay + +The k3d patches for EAS are the upstream drafts in +[integrations/eat/upstream/patches/](../../integrations/eat/upstream/patches/), one copy for +both uses. The only overlay-only patch is the `release-1.2` form of 0006, +[integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch](../../integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch). +[`integrations/eas/overlay.json`](../../integrations/eas/overlay.json) lists, per patch, its +variants, when it is needed (`always`, `eas_owns_nri`, `multi_node`, `never`), a probe that +recognises an upstream fix done another way, the knob `eas up` writes, and a hint naming the +upstream PR known to move its hunk. [`integrations/eas/overlay.py`](../../integrations/eas/overlay.py) +reads it. + +At the pinned EAS commit each patch gets one decision, printed as `fact -> choice [source]`: + +- **skip**: not needed on this cluster; +- **drop-merged**: a variant reverse-applies (`git apply -R --check`): upstream took the patch; +- **drop-fixed**: the probe matches and the knob is still in the file it matched: upstream + fixed it another way. A probe match with the knob gone is a STOP; +- **apply**: the first variant whose `git apply --check` passes; +- otherwise **STOP** (exit 3), before anything is written or the cluster is touched. The line + names the upstream PR that moved the hunk and says to regenerate the patch. + +| Patch | Needed when | Probe for an upstream fix | `main` profile (EAS `97f6157`) | `v1.2.0-rc.1` profile (EAS `5c27fb3`) | +|---|---|---|---|---| +| 0002 Istio `global.platform` | always | `roles/istio/**` sets `global.platform` from a variable; knob `istio_platform` | apply | apply | +| 0006 Envoy control-plane affinity | always | an `Exists` term on the control-plane key in a file with no exact `""` selector, or `roles/envoy_gateway/vars/k3s.yaml` (the per-distribution layout of EAS #50) | apply, variant a (`roles/envoy_gateway/tasks/controller.yaml`) | apply, variant b (the EnvoyProxy task in [`install.yaml`](https://github.com/intel/enterprise-ai-solutions/blob/5c27fb39a95eecec41aa4e29a1188984e9c726f3/roles/envoy_gateway/tasks/install.yaml#L199-L219)) | +| 0003 nri-plugins chart pin `v0.15.0` | `eas_owns_nri`: `kubernetes_cpu_policy: nri-balloons` | `chart_version:` in `roles/nri_cpu_balloons/tasks/install_advanced.yaml` | skip on a baked-balloons cluster; apply on `K3D_GPU_BALLOONS=external` | same | +| 0004 BYO node topology labels | `multi_node` | `roles/kubernetes/tasks/install.yaml` runs `apply_node_topology.yaml` | skip (one k3d-gpu server) | skip | +| 0005 single-node k3d docs page | `never` (docs only) | none | skip | skip | + +`overlay.json` records these decisions as `verified` cases; `overlay.py --check-verified` +re-checks them against an EAS clone, and +`integrations/eat/upstream/checks/apply-check.sh --overlay` does it on a fresh clone. On +2026-10-29 every patch applied on EAS `main` `97f6157`; on `v1.2.0-rc.1` 0002, 0003 and 0004 +applied and 0005 and 0006 did not (0006's file, `controller.yaml`, does not exist on +`release-1.2`, which does not contain EAS #42), which is why 0006 has a second variant. + +Open EAS PRs that change a decision when they merge (`overlay.py --watch`, checked 2026-10-29): + +| EAS PR | Effect on the overlay | +|---|---| +| [#50](https://github.com/intel/enterprise-ai-solutions/pull/50) and [#52](https://github.com/intel/enterprise-ai-solutions/pull/52) (OpenShift) | 0006 no longer applies: the selector moves into `_eg_k8s_provider`, still `control-plane: ""`. That is a STOP naming #50. Once a k3s vars file lands, 0006 becomes drop-fixed | +| [#61](https://github.com/intel/enterprise-ai-solutions/pull/61) (Headlamp SSO) | 0002 fails on its `global_config.yaml` hunk context: a STOP naming #61, so 0002 needs regenerating. 0004 (skipped on one node) and 0005 (docs, never applied) fail there too. 0006 still applies | + +## Pins + +`integrations/eas/pins.json` (schema `k3d-gpu.eas-pins/v1`) holds flat rows +`{name, repo, ref, sha, date, source}`. Each SHA was the head of its ref on 2026-10-29 +(`git ls-remote`). + +| Profile | EAS | EI (enterprise-inference) | EAT | +|---|---|---|---| +| `main` (default) | `main` [`97f6157`](https://github.com/intel/enterprise-ai-solutions/commit/97f6157d6a72f7173c698c474d4e51d2c79f1fd6) (#69) | `main` [`3e8a244`](https://github.com/intel/enterprise-inference/commit/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd) (#44) | `release-2.0.0` [`036b662`](https://github.com/intel/enterprise-agent-toolkit/commit/036b66234801e9bd950afd90830821859780d668) (the #19 merge; same tree as `agentic-toolkit-migration` `fb59459`) | +| `v1.2.0-rc.1` (`--eas-ref v1.2.0-rc.1`) | tag `v1.2.0-rc.1` [`5c27fb3`](https://github.com/intel/enterprise-ai-solutions/commit/5c27fb39a95eecec41aa4e29a1188984e9c726f3) | `release-1.1` [`170eb3a`](https://github.com/intel/enterprise-inference/commit/170eb3abb1956de0787727f21e356aef2b13ff24) | `release-2.0.0` `036b662` | + +- The `main` triple is what EAS `main` itself clones today: inference `rev: "main"` and + agent-toolkit `rev: "agentic-toolkit-migration"` + ([repos.agent-toolkit.yaml#L12 and #L21](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/configs/repos/repos.agent-toolkit.yaml#L9-L24)). + The pins freeze that combination. +- The `v1.2.0-rc.1` triple is the component table of the + [v1.2.0-rc.1 release notes](https://github.com/intel/enterprise-ai-solutions/releases/tag/v1.2.0-rc.1), + Intel's pre-release vehicle for EAT 2.0. +- Any other `--eas-ref`, `--ei-ref` or `--eat-ref` is resolved to a SHA with + `git ls-remote` when `eas up` runs, printed, and saved in the state dir. A floating ref + never reaches the installer. + +Bastion tools (each binary checked against its upstream checksum file at build time): + +| Tool | Pin | Why this version | +|---|---|---| +| kubectl | v1.37.2 | the minor of the node's k3s (`K3S_TAG` v1.37.1-k3s1); EAS's own v1.34.3 ([#L52](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L52)) is three minors off, outside the [kubectl skew policy](https://kubernetes.io/releases/version-skew-policy/#kubectl) | +| helm | v3.22.1 | latest v3; EAS's floor is v3.20.2 ([#L53](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L53)). Helm v4 is untested with `kubernetes.core` 6.3.0 | +| yq | v4.54.1 | EAS's floor is v4.53.2 ([#L51](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L51)) | +| ansible, kubernetes (Python) | 12.3.0, 35.0.0, hashed | newest inside EAS's ranges `ansible>=10.0,<13.0`, `kubernetes>=29.0,<36.0` ([requirements.txt](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/requirements.txt)) | +| kubernetes.core | 6.3.0 | EAS's [collections/requirements.yml](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/collections/requirements.yml) | + +To move a pin: + +1. Read the new head: `git ls-remote https://github.com/intel/ `. +2. Edit that one row of `integrations/eas/pins.json`: `sha`, `date`, `source`. +3. Read the overlay decisions at the new EAS SHA: `k3d-gpu eas up --dry-run`, or + `integrations/eat/upstream/checks/apply-check.sh --overlay` on a fresh clone. A STOP + names the upstream PR that moved a hunk; regenerate that patch, or add a variant, and + update the `verified` cases in `overlay.json` before the pin lands. +4. Run `tests/eas_test.sh` and update this page's tables. + +The rows are flat on purpose, so a repo-wide pins registry can take them as they are. + +## EAT's minimums on a smaller host + +EAT's prerequisites assume a Xeon server. `eas up` checks each one and warns; it never +changes the host. + +| EAT asks for | Source | What `eas up` does below it | +|---|---|---| +| 48 cores minimum, 96+ recommended | [prerequisites.md#L7](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/prerequisites.md#L7) | warns and goes on; CPU requests that do not fit leave pods Pending, which `eas status` lists | +| 32 GB RAM minimum, 64 GB recommended | [prerequisites.md#L8](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/prerequisites.md#L8) | sizes the model from node allocatable memory and `fit` (the default install requests about 131 GiB; the fallbacks 64 GiB each, [single-node-deployment.md#L84](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/single-node-deployment.md#L84)) | +| `fs.inotify.max_user_instances = 8192`, `max_user_watches = 1048576` | [prerequisites.md#L21-L34](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/docs/prerequisites.md#L21-L34) | warns and prints the `sysctl` line to run | + +## Known issues + +`integrations/eas/known-issues.json` holds these rows as `{id, upstream, symptom, workaround, source, checked}`. + +| Issue | Symptom | What `k3d-gpu eas` does | +|---|---|---| +| EAS [#71](https://github.com/intel/enterprise-ai-solutions/issues/71) (open) | `teardown platform` hangs on the Envoy AI Gateway CRD uninstall once a model was deployed | `eas down` clears the `AIGatewayRoute` finalizers first, then runs `teardown platform --env k3d-gpu --force` (teardown refuses a non-terminal run without `--force`, [#L1486](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1481-L1487)) | +| `require_amx: true` in EI's catalog | on a host without AMX every model pod stays Pending | writes `require_amx: false` when the node has no AMXBF16 label | +| kubectl skew | EAS installs kubectl v1.34.3, three minors behind k3s v1.37 | the bastion carries kubectl v1.37.2 | +| inotify minimum | file watchers fail with `too many open files` under the kernel default | warns and prints the `sysctl` line | +| 48-core minimum | under it, CPU requests may not fit | warns; `eas status` lists Pending pods | +| `--skip-models` in EI's docs ([config.yaml#L97](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/config.yaml#L97)) | EAS's installer has no such flag and stops on it ([#L1312](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1312)) | nothing to replace on the inference layer: EI's seeded config already has `llm_services_deploy_models: false` ([config.yaml#L98](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/config.yaml#L98)). With `--agent-toolkit`, EAT's chat model and the Qwen3Guard guard model deploy whatever that key says ([EI llm_services install.yaml#L143-L161](https://github.com/intel/enterprise-inference/blob/3e8a24486b6ee812adf2bf452f8ba5751c16c8bd/roles/llm_services/tasks/install.yaml#L143-L161)); `k3d-gpu eas up --agent-toolkit -- -e '{"llm_services_autodeploy_models": []}'` skips the chat model ([EAT config.yaml#L34](https://github.com/intel/enterprise-agent-toolkit/blob/036b66234801e9bd950afd90830821859780d668/config.yaml#L34)), and the guard model stays while the guardrail is on | + +## State and secrets + +- State lives in `${STATE_DIR}/eas-/` (mode 0700): the EAS checkout, `env/k3d-gpu`, + logs, `choices.txt`, `overlay.txt` and the resolved pins. +- The Keycloak admin password is generated by EAS + ([keycloak defaults#L28](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/keycloak/defaults/main.yaml#L28)). + `HF_TOKEN` is passed to the bastion by name when it is set. No secret is put on a + command line or in a log. +- `eas check` (after `--agent-toolkit`, the layer that brings LiteLLM) mints a LiteLLM + virtual key from the `litellm-master-key` Secret (namespace `litellm`), uses it once + against the gateway, and deletes it. + +## Limits + +- **Xeon CPU path only.** That is what EAS and EAT ship for a cluster like this one + (sources [above](#what-full-eas-covers)). An Arc GPU or a Gaudi on the node is not used by + EAS; serve it with `k3d-gpu agent up`. Registering such an engine behind EAS's LiteLLM, + as [`eat-remote.sh`](../../integrations/eat/README.md) does for EAT 1.x, is a follow-up. +- **ERAG is not wired in yet.** EAS wires Intel Enterprise RAG in as a further layer + ([repos.erag.yaml](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/configs/repos/repos.erag.yaml#L22-L28)), + and `k3d-gpu eas up` does not install it. Its default flavour, `chatqna`, is a Xeon (CPU) + deployment + ([deployment/README.md#L3](https://github.com/intel/enterprise-rag/blob/319eb4652565fed42de9d601c61f9862b3759dfb/deployment/README.md#L3)), + so it is on the same CPU path as the other layers and is the follow-up: pins, a test + fixture and a run. Its Intel GPU flavour, `chatqna_intel_gpu`, is marked EXPERIMENTAL + and needs `kubernetes_accelerator: intel-gpu` with an EI vLLM-XPU runtime + ([config.yaml#L8-L15](https://github.com/intel/enterprise-rag/blob/319eb4652565fed42de9d601c61f9862b3759dfb/deployment/pipelines/chatqna_intel_gpu/config.yaml#L8-L15), + ERAG `main` `319eb46`, tag `v3.0.0` `25812cb`). No branch of enterprise-inference has + that runtime: all 33 branches carry only `openvino` and `vllm` under + `model_manager/runtimes` (read 2026-10-29), so that flavour cannot run anywhere yet. +- **One node.** Patch 0004 (topology labels) is only needed on more than one node, and + `eas up` targets the single k3d-gpu server. +- **Not yet observed.** The EAT 2.0 layer on k3s or k3d; agent-sandbox v0.5.0 on + Kubernetes 1.37 (EAS tests 1.34.3); the CPU requests of the whole stack on a 32-CPU + desktop; helm v3.22.1 with `kubernetes.core` 6.3.0; and whether any BYO task needs `sudo` + inside the bastion. The installer itself probes `sudo -n true` before every `install` and, + without passwordless sudo, runs `ansible-playbook` with `--ask-become-pass` + ([es_auto_installer.sh#L866-L875](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L866-L875), + [#L1179](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1179), + [#L1423](https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/es_auto_installer.sh#L1423)), + which a bastion with no terminal cannot answer. The `eas` CLI PR's wiring smoke settles + these on a throwaway cluster; F3 is the scored run. + +## Related + +| If you want to… | Go to | +|---|---| +| Run EAT 1.x (`release-1.1.0`) on k3d-gpu | [integrations/eat](../../integrations/eat/README.md), [intel.md](../intel.md#run-intel-enterprise-agent-toolkit-on-one-machine) | +| See the upstream patch drafts and their checks | [upstream-eat.md](upstream-eat.md) | +| See EAT's PR history and pins | [platforms.md](../platforms.md#intel-enterprise-agent-toolkit-eat) | +| See the F3 pre-registration | [EXPERIMENTS.md](../../EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29) | diff --git a/docs/intel/upstream-eat.md b/docs/intel/upstream-eat.md index e4bd1ca5..3fd0381d 100644 --- a/docs/intel/upstream-eat.md +++ b/docs/intel/upstream-eat.md @@ -5,12 +5,35 @@ Drafted 2026-10-17. Heads re-read on 2026-10-19 with `gh api` and `git ls-remote`: EAT is unchanged and #19 is still a draft PR; EAS `main` moved from `7c02ca2` to `e8f3ebc`. Re-read on 2026-10-20: no head in the table moved. EAT #21 (Release 1.1.0) opened on 2026-10-19; see [its impact](#release-110-pr-21-impact). Re-read on 2026-10-28: #21 merged on 2026-10-22 and shipped as `release-1.1.0` (`0c726f5`). #19 merged on 2026-10-27 into the `release-2.0.0` branch, not `main`, at final head `fb59459` (merge `036b662`); no tag carries it. The EAT docs patch below now needs a re-base ([see below](#eat-patch-full-content)). EAS `main` moved from `e8f3ebc` to `890e66e`, where every EAS patch and the series still pass `apply-check.sh`. EAS #40 was closed unmerged on 2026-10-22, a minute after EAS #66 ("Integrate Agentic Toolkit", open) was opened. enterprise-inference `main` moved from `aaca1e5` to `3e8a244`; Gate 2 has not been re-run on it. Re-read on 2026-10-29: EAS #66 merged into `main` (`5767995`), where every EAS patch and the series still pass `apply-check.sh`. Its `configs/repos/repos.agent-toolkit.yaml` clones EAT at the `agentic-toolkit-migration` branch, not a tag, and it adds no `intel_gpu_plugin` role. -| Repo | Ref | SHA | -|---|---|---| -| intel/enterprise-agent-toolkit (EAT) | PR #19 head, "Migration Agentic Toolkit into Unified Convention" | `9d842f5c3b88ad2ec6ea5f7d66a003718bdac828` | -| intel/enterprise-agent-toolkit (EAT) | `main` | `0ecf67a8331e8b502ef41e08bf6d2881416178c5` | -| intel/enterprise-ai-solutions (EAS) | `main` | `e8f3ebc927a7659081f6604d157591bddfca40a9` | -| intel/enterprise-inference | `main`, cloned by `init inference` in Gate 2 | `aaca1e505a8f7947cd455d2dea5f433213a95cad` | +Re-read a second time on 2026-10-29, for [`k3d-gpu eas`](eas.md), with `git ls-remote`, `gh api` and read-only clones: + +- EAS `main` moved from `5767995` to `97f6157` (#69, a preflight retry). Every EAS patch and the series still pass `apply-check.sh` there. +- EAS tag [`v1.2.0-rc.1`](https://github.com/intel/enterprise-ai-solutions/releases/tag/v1.2.0-rc.1) (`5c27fb3`, a pre-release published 2026-10-27) is Intel's vehicle for EAT 2.0. Its release notes pin enterprise-inference `release-1.1` at `170eb3a` and EAT `release-2.0.0` at `036b662`. On it, 0002, 0003 and 0004 apply, and 0005 and 0006 do not: the `release-1.2` branch does not contain EAS #42 (`88fd2ae`), so the EnvoyProxy task is still in `roles/envoy_gateway/tasks/install.yaml#L199-L219`. The overlay in `k3d-gpu eas` carries a second 0006 variant for that file; 0005 is docs and is never applied. +- Open EAS PRs that conflict with the pack: #50 and #52 (OpenShift) move the EnvoyProxy selector into `_eg_k8s_provider`, still as `control-plane: ""`, so 0006 fails on both heads. #61 (Headlamp SSO) breaks 0002's `global_config.yaml` hunk context, 0004 and 0005; 0006 still applies on it. +- What a k3d BYO install of EAT 2.0 needs, read from source at EAS `97f6157`, EI `3e8a244` and EAT `036b662`, not run: + - **LiteLLM**: EI installs LiteLLM and Langfuse only with `auth_type: litellm_vkey`; EI's default is `jwt` (Keycloak, as in Gate 2 probe 5), and EAT's layer config sets `litellm_vkey`. So LiteLLM, its guardrail and the hybrid router come with the agent-toolkit layer. + - **agent-sandbox**: EAT installs agent-sandbox `v0.5.0` itself. A cluster where `k3d-gpu sandbox up` already installed v1.0.8 would get older CRDs and controller applied over it. + - **NFD**: EI's `nfd` role skips only when a Helm release named `node-feature-discovery` exists. k3d-gpu's NFD is a k3s manifest, so EI would install a second one unless `nfd_enabled: false`. + - **require_amx**: EI's catalog defaults to `require_amx: true`, an AMXBF16 node affinity, so on a host without AMX every model pod stays Pending (as in Gate 2 probe 5). + - **pod CIDR**: EI's LiteLLM falls back to `10.233.64.0/18` when `kubernetes_cluster_cidr` is unset, for the pod network it admits once `litellm_allow_pod_network` is on (EAT turns it on with OpenShell and then asks for the real pod CIDR on a BYO cluster). k3s defaults to `10.42.0.0/16`. + - **Kata**: the `kata_containers` role fails on any BYO cluster; it has to stay off. + - **NRI owner**: on a default k3d-gpu cluster on a hybrid or multi-NUMA host k3d-gpu owns the balloons policy, so EAS needs `kubernetes_cpu_policy: best-effort` (Gate 2 probe 6). + - **memory**: EAT's default install requests about 131 GiB, so a 128 GB node needs one of EAT's 64 GiB models. + - **version skew**: EAS installs kubectl v1.34.3 and tests Kubernetes 1.34.3; a k3d-gpu node runs k3s v1.37.1, three minors away. + - **teardown**: EAS issue #71 (open): `teardown platform` hangs on the Envoy AI Gateway CRD uninstall once a model was deployed. +- ERAG (`intel/enterprise-rag`, `main` `319eb46`, tag `v3.0.0` `25812cb`) is a further EAS layer. Its default flavour, `chatqna`, is a Xeon (CPU) deployment, the same path as the other layers; `k3d-gpu eas` does not install it yet. Its Intel GPU flavour `chatqna_intel_gpu` is EXPERIMENTAL and needs `kubernetes_accelerator: intel-gpu` with an EI vLLM-XPU runtime; none of the 33 enterprise-inference branches has that runtime. + +| Repo | Ref | SHA | Used for | +|---|---|---|---| +| intel/enterprise-agent-toolkit (EAT) | `release-2.0.0`, the merge of PR #19 | `036b66234801e9bd950afd90830821859780d668` | `k3d-gpu eas --agent-toolkit` (both profiles). Same tree as `agentic-toolkit-migration` `fb594597efd95f97376c7920a9d59ed54562f19c`, #19's final head | +| intel/enterprise-agent-toolkit (EAT) | PR #19 draft head | `9d842f5c3b88ad2ec6ea5f7d66a003718bdac828` | base of the EAT docs patch below | +| intel/enterprise-agent-toolkit (EAT) | `main` when the pack was cut | `0ecf67a8331e8b502ef41e08bf6d2881416178c5` | base of the issue draft; `main` is now `4a2e930` | +| intel/enterprise-ai-solutions (EAS) | `main` | `97f6157d6a72f7173c698c474d4e51d2c79f1fd6` | `apply-check.sh` (2026-10-29), `k3d-gpu eas` profile `main` | +| intel/enterprise-ai-solutions (EAS) | tag `v1.2.0-rc.1` | `5c27fb39a95eecec41aa4e29a1188984e9c726f3` | `k3d-gpu eas` profile `v1.2.0-rc.1` | +| intel/enterprise-ai-solutions (EAS) | `main` on 2026-10-19 | `e8f3ebc927a7659081f6604d157591bddfca40a9` | Gate 1 and Gate 2 below | +| intel/enterprise-inference | `main` | `3e8a24486b6ee812adf2bf452f8ba5751c16c8bd` | `k3d-gpu eas` profile `main` | +| intel/enterprise-inference | `release-1.1` | `170eb3abb1956de0787727f21e356aef2b13ff24` | `k3d-gpu eas` profile `v1.2.0-rc.1` | +| intel/enterprise-inference | `main`, cloned by `init inference` in Gate 2 | `aaca1e505a8f7947cd455d2dea5f433213a95cad` | Gate 2 below | ## Where the change belongs @@ -216,6 +239,8 @@ shellcheck --severity=error es_auto_installer.sh **Gate 2. Live probes on one throwaway k3d cluster.** **Status: all six ran on 2026-10-20 (UTC) on one i9-13900T host. Probes 1 and 2 found a defect in 0005, now fixed; probe 5 needed one deviation.** EAS `main` `e8f3ebc` with all five patches applied, from an Ubuntu 24.04 bastion container on the host network (`configure` supports Ubuntu and RHEL only; this host runs Manjaro). Each run took the shared live lock, used one cluster named `k3dgpu-gate2`, and deleted it afterwards. +Gate 2 covered EAS's platform and inference layers only, on EAS `e8f3ebc` and enterprise-inference `aaca1e5`. It did not install the agent-toolkit layer, which did not exist in EAS then. EAT 2.0 on k3d stays unverified until F3 ([EXPERIMENTS.md](../../EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29)) has run: `k3d-gpu eas up --agent-toolkit` on one fresh cluster, graded as pre-registered. + | # | Probe | Cluster | Result | |---|---|---|---| | 1 | containerd's CNI `bin_dirs` | stock `rancher/k3s:v1.34.12-k3s1` | `["/var/lib/rancher/k3s/data/cni"]` only; `/bin` also holds the plugins, but containerd does not search it. Istio's **k3s** profile matches, its k3d profile does not | @@ -327,5 +352,6 @@ Refs intel/enterprise-ai-solutions#. | If you want to… | Go to | |---|---| | See the patch pack and its static checks | [integrations/eat/upstream/README.md](../../integrations/eat/upstream/README.md) | +| Install EAS (EAT 2.0) on k3d-gpu with these patches as an overlay | [eas.md](eas.md) | | See what was measured and how | [technical-report.md](technical-report.md) | | See EAT on k3d-gpu today | [docs/intel.md](../intel.md) | diff --git a/docs/platforms.md b/docs/platforms.md index 4c59d819..273076a1 100644 --- a/docs/platforms.md +++ b/docs/platforms.md @@ -218,38 +218,48 @@ Simulation fixture: `tests/fixtures/xeon-dmr` (topology borrowed from ## Intel Enterprise Agent Toolkit (EAT) -EAT is a software stack, not a hardware family, so it has no fixture. Its pins: +EAT is a software stack, not a hardware family, so it has no fixture. It reaches k3d-gpu +two ways, one per EAT generation. Its pins: -- `integrations/eat/eat-up.sh` deploys the pinned tag `release-1.0.3-hotfix` - ([`0d3438d`](https://github.com/intel/enterprise-agent-toolkit/tree/0d3438d7119de44cafc5903f66665c0b623e3333)). - The latest tag is [`release-1.1.0`](https://github.com/intel/enterprise-agent-toolkit/releases/tag/release-1.1.0) +- **EAT 1.x:** `integrations/eat/eat-up.sh` deploys the tag + [`release-1.1.0`](https://github.com/intel/enterprise-agent-toolkit/releases/tag/release-1.1.0) ([`0c726f5`](https://github.com/intel/enterprise-agent-toolkit/tree/0c726f576595971c53cc45b94e08ba969beb4a4a), published 2026-10-22, [tags](https://api.github.com/repos/intel/enterprise-agent-toolkit/tags)), the merge of [PR #21](https://github.com/intel/enterprise-agent-toolkit/pull/21). Its tree, - `04969afa`, is the same as that of #21's head `c86c42b`. + `04969afa`, is the same as that of #21's head `c86c42b`. It moved from + `release-1.0.3-hotfix` + ([`0d3438d`](https://github.com/intel/enterprise-agent-toolkit/tree/0d3438d7119de44cafc5903f66665c0b623e3333); + [compare](https://github.com/intel/enterprise-agent-toolkit/compare/0d3438d7119de44cafc5903f66665c0b623e3333...0c726f576595971c53cc45b94e08ba969beb4a4a), + 204 files) with three changes on our side: the Qwen3Guard guardrail (on by default in + 1.1.0) and the other new components (vLLM Semantic Router, OpenShell, Kata, Ceph, + rack-scale mode) stay off; the vLLM CPU image is `vllm/vllm-openai-cpu:v0.28.0` by + digest (1.1.0's `xeon-values.yaml` moved to v0.28.0); and `eat-up.sh` refuses + `deploy_openshell=on`. The move was read from source; H1b's + end-to-end run used `release-1.0.3-hotfix` [ledger:H1b]. `bench/eatflow` keeps its own + EAT pins (`bench/eatflow/agent.lock`). +- **EAT 2.0:** `k3d-gpu eas up --agent-toolkit` installs EAT through Intel's EAS installer + (`es_auto_installer.sh install agent-toolkit`) on the cluster. It pins EAS `main` + `97f6157`, enterprise-inference `main` `3e8a244` and EAT `release-2.0.0` `036b662` in + `integrations/eas/pins.json`, with EAS `v1.2.0-rc.1` as an opt-in profile. Choices, + patches, refusals and limits: [intel/eas.md](intel/eas.md). It has not run end to end + yet; F3 in [EXPERIMENTS.md](../EXPERIMENTS.md#f3-pre-registration-i9-13900t-2026-10-29) + is PENDING. - `docs/intel.md`, `contrib/eat/` and `integrations/eat/upstream/` read `main` at [`0ecf67a`](https://github.com/intel/enterprise-agent-toolkit/tree/0ecf67a8331e8b502ef41e08bf6d2881416178c5). -`eat-up.sh` and `brownfield.sh` stay on `release-1.0.3-hotfix` while pre-registered -ledger rows pin EAT stock at `0d3438d` (H50, H50o and H51, and `bench/apb/PROTOCOL.md` -through `eat-up.sh`). Moving them to `release-1.1.0` still needs the two checks -described below: `brownfield.sh` has to export `DEPLOY_GUARDRAILS=off`, and the vLLM -CPU image has to be re-pinned to `v0.28.0` by digest. A fresh run like H1b's then has -to pass on it [ledger:H1b]. - `release-1.1.0` ships NVIDIA OpenShell as an opt-in component -(`deploy_openshell=off` by default, [PR #14](https://github.com/intel/enterprise-agent-toolkit/pull/14)). -The pinned tag has no OpenShell role, so `eat-up.sh` stops on a config that sets -`deploy_openshell=on` instead of letting the tag skip the key. Upstream lists a CNI -that enforces `NetworkPolicy` and best-effort Landlock as OpenShell requirements -([docs/openshell.md](https://github.com/intel/enterprise-agent-toolkit/blob/0ecf67a8331e8b502ef41e08bf6d2881416178c5/docs/openshell.md)). -k3d-gpu adds no OpenShell checks while `eat-up.sh` stays on the pinned tag. +(`deploy_openshell=off` by default, [PR #14](https://github.com/intel/enterprise-agent-toolkit/pull/14); +`core/lib/components/openshell-controller.sh`). OpenShell has not been tried on a +k3d-gpu cluster, so `eat-up.sh` stops on a config that sets `deploy_openshell=on`. Upstream +lists a CNI that enforces `NetworkPolicy` and best-effort Landlock as OpenShell requirements +([docs/openshell.md](https://github.com/intel/enterprise-agent-toolkit/blob/0c726f576595971c53cc45b94e08ba969beb4a4a/docs/openshell.md)). Upstream work that changes this integration: - [EAT PR #19](https://github.com/intel/enterprise-agent-toolkit/pull/19) replaces `deploy-agentic-stack.sh` with `es_auto_installer.sh install agent-toolkit`. It merged - on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no tag carries it yet. + on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no EAT tag carries it yet; + EAS's pre-release `v1.2.0-rc.1` pins it. `k3d-gpu eas` is the EAT 2.0 path. The status of each EAT PR is in the table below. - [enterprise-inference PR #22](https://github.com/intel/enterprise-inference/pull/22) merged on 2026-10-22 as "Integrate Agentic toolkit" (`4bc7fed`), the inference-layer @@ -267,14 +277,16 @@ Upstream work that changes this integration: Read with `gh`, `git ls-remote` and a read-only clone only. EAT `main` is `4a2e930`: `release-1.1.0` plus the architecture diagram from [#27](https://github.com/intel/enterprise-agent-toolkit/pull/27). -`release-1.0.3-hotfix` still points at `0d3438d`. +`release-1.0.3-hotfix` still points at `0d3438d`. The #19 and #21 rows were re-read on +2026-10-29, when `eat-up.sh` moved to `release-1.1.0` and `k3d-gpu eas` took over EAT 2.0; +EAT `main` was still `4a2e930`. | PR | Head | What it changes | Impact on k3d-gpu | Action | |---|---|---|---|---| | [#5](https://github.com/intel/enterprise-agent-toolkit/pull/5) Bitnami to SolDevelo images | `fcec7ef`, rebased onto `main` | `bitnami/kubectl:latest` becomes `soldevelo/kubectl:1.29` in the GenAI Gateway restore job, and `bitnamilegacy/minio:2024.12.18` becomes `soldevelo/minio:2024.12.18` in `docker/docker-compose.yml` | None. Neither line runs on our path: `restoreJob.enabled` is `false`, and `eat-up.sh` does not use EAT's Docker Compose mode. Every Bitnami image our path renders pulled on 2026-10-18 (`docker manifest inspect`, amd64 and arm64). See the note below | None. Recheck if a `bitnamilegacy` pull fails | -| [#19](https://github.com/intel/enterprise-agent-toolkit/pull/19) unified convention | Merged 2026-10-27 into `release-2.0.0`, not `main`: final head `fb59459`, merge `036b662`, no tag | Deletes `deploy-agentic-stack.sh`, `core/lib` (the brownfield path with it), `core/playbooks` and `core/roles`; installs through EAS `es_auto_installer.sh`. `release-2.0.0`'s README says Intel Arc (XPU) deployment is not in that release | Ends the path `eat-up.sh` uses, once a tag ships it. The drafted EAT docs patch in [upstream-eat.md](intel/upstream-eat.md) passes `git apply --check` on the draft head `9d842f5` but not on `fb59459` or `036b662`, so it needs a re-base before it is sent | Keep `eat-up.sh` on the tag. Move to the EAS BYO path after a tag that includes #19 | +| [#19](https://github.com/intel/enterprise-agent-toolkit/pull/19) unified convention | Merged 2026-10-27 into `release-2.0.0`, not `main`: final head `fb59459`, merge `036b662`, no tag | Deletes `deploy-agentic-stack.sh`, `core/lib` (the brownfield path with it), `core/playbooks` and `core/roles`; installs through EAS `es_auto_installer.sh`. `release-2.0.0`'s README says Intel Arc (XPU) deployment is not in that release | `eat-up.sh` cannot deploy it: the brownfield path it drives is gone. `k3d-gpu eas up --agent-toolkit` installs it through the EAS bring-your-own-cluster path at `036b662`, the commit EAS `v1.2.0-rc.1` pins too; read from source, F3 PENDING. The drafted EAT docs patch in [upstream-eat.md](intel/upstream-eat.md) passes `git apply --check` on the draft head `9d842f5` but not on `fb59459` or `036b662`, so it needs a re-base before it is sent | EAT 2.0: `k3d-gpu eas up --agent-toolkit` ([intel/eas.md](intel/eas.md)). EAT 1.x stays on `eat-up.sh` | | [#20](https://github.com/intel/enterprise-agent-toolkit/pull/20) OpenClaw operator | `10deb6d`, on #19's branch `agentic-toolkit-migration` (forked at `dab3a98`, before #19's final head) | Optional `openclaw_operator` role (`claw_enabled: false`): operator chart and image `0.38.3` pinned by digest, 3 CRDs (`openclaw.rocks/v1alpha1`). It also adds a `cosign verify` of the chart and the operator image when `cosign` is on the installer's `PATH` (`openclaw_operator_cosign_required: false` by default) | Not reachable from the pinned tag. The operator itself fits one k3d-gpu node: CPU only (100m / 128Mi request), no GPU, no webhook, `kubeVersion >=1.28`, and both images publish amd64. Its example instance is the gap (see below) | None until #19 and #20 reach a tag | -| [#21](https://github.com/intel/enterprise-agent-toolkit/pull/21) Release 1.1.0 | Merged 2026-10-22 as `0c726f5` and released as `release-1.1.0`; head `c86c42b`, 204 files against `release-1.0.3-hotfix` | Keeps the old layout (`deploy-agentic-stack.sh`, `core/lib`); adds Qwen3Guard guardrails (on by default), Kata sandboxes (`kata-fc` Firecracker and `kata-qemu`), vLLM Semantic Router, rack-scale mode, an NVIDIA B300 profile, and an Arc B60/B70 XPU path (`compute_platform=xpu`); OVMS as a Docker engine; vLLM CPU image `v0.24.0` to `v0.28.0`; NRI balloons role reworked into NUMA-aware per-node policies | None on the pinned tag. On `release-1.1.0`: `core/lib/brownfield/` and `deploy-genai-gateway.yml` are unchanged, every function `brownfield.sh` calls still exists, and `gateway-charts.py` runs clean on its charts. Two changes would bite, and the XPU path would overlap what k3d-gpu already installs; all read from source, not run (see below) | Before moving `eat-up.sh` to `release-1.1.0`: turn the guardrail off and re-pin the vLLM image. Validation on 4x Arc Pro B70 is in progress, no results yet | +| [#21](https://github.com/intel/enterprise-agent-toolkit/pull/21) Release 1.1.0 | Merged 2026-10-22 as `0c726f5` and released as `release-1.1.0`; head `c86c42b`, 204 files against `release-1.0.3-hotfix` | Keeps the old layout (`deploy-agentic-stack.sh`, `core/lib`); adds Qwen3Guard guardrails (on by default), Kata sandboxes (`kata-fc` Firecracker and `kata-qemu`), vLLM Semantic Router, rack-scale mode, an NVIDIA B300 profile, and an Arc B60/B70 XPU path (`compute_platform=xpu`); OVMS as a Docker engine; vLLM CPU image `v0.24.0` to `v0.28.0`; NRI balloons role reworked into NUMA-aware per-node policies | `eat-up.sh` deploys this tag. `core/lib/brownfield/` and `deploy-genai-gateway.yml` are unchanged, every function `brownfield.sh` calls still exists, and `gateway-charts.py` runs clean on its charts. Two changes would bite, and the XPU path would overlap what k3d-gpu already installs; all read from source, not run (see below) | Done in `eat-up.sh`: the guardrail and the other new components off, the vLLM CPU image re-pinned to `v0.28.0` by digest. Validation on 4x Arc Pro B70 is in progress, no results yet | | [#22](https://github.com/intel/enterprise-agent-toolkit/pull/22)-[#26](https://github.com/intel/enterprise-agent-toolkit/pull/26) agent toolkit stack | `a627fad`, `a32c5aa`, `9edf77d`, `6dfbcbc`, `7cc4344`, stacked on #20, base `agentic-toolkit-migration` | `Agent`, `AgentGrant` and `Tool` CRDs (`intel-stack.io`) with an agent operator (#22); per-agent memory (#23); sandbox, GitHub and web tools for agents (#24); an agent REST API (#25); a browser console and agent dashboards (#26). Its images are built into an in-cluster registry (`registry.kube-system.svc.cluster.local:5000`, the `kubernetes_registry_enabled` add-on) | Not reachable from the pinned tag. The stack claims sandboxes through `SandboxClaim` at `extensions.agents.x-k8s.io/v1beta1`, the version k3d-gpu's agent-sandbox v1.0.7 serves. Read from source, not run | None until the stack reaches a tag | Bitnami images on the pinned path. `deploy-genai-gateway.yml` runs `helm dependency update` @@ -303,7 +315,8 @@ Rendered with Helm v3.14.0, the pinned charts differ from EAT's only in the 11 ` What #21 (`c86c42b`) changes for `eat-up.sh`, compared with `release-1.0.3-hotfix`. It applies verbatim to `release-1.1.0`, whose tree is `c86c42b`'s. Read from a read-only -clone; nothing here has run against a 1.1.0 stack. +clone; nothing here has run against a 1.1.0 stack. The guardrail and vLLM-image items are +the two the move of `eat-up.sh` to `release-1.1.0` had to handle. - Unchanged: `core/lib/brownfield/` (so the `5 / 1 / yes` prompt answers), `core/playbooks/deploy-genai-gateway.yml` (so the `helm_charts_base_genai{,_trace}` diff --git a/integrations/eas/overlay.json b/integrations/eas/overlay.json new file mode 100644 index 00000000..f44b425a --- /dev/null +++ b/integrations/eas/overlay.json @@ -0,0 +1,164 @@ +{ + "schema": "k3d-gpu.eas-overlay/v1", + "checked": "2026-10-29", + "repo": "intel/enterprise-ai-solutions", + "patch_dirs": ["integrations/eas/patches", "integrations/eat/upstream/patches"], + "rules": { + "always": "every k3d-gpu cluster", + "eas_owns_nri": "only when kubernetes_cpu_policy is nri-balloons: EAS installs the nri-plugins balloons chart only then (configs/components.yaml, nri_cpu_balloons.enabled)", + "multi_node": "only on a cluster with more than one node; k3d-gpu runs one server", + "never": "documentation only; nothing at install time reads it" + }, + "rows": [ + { + "id": "0002", + "what": "istio: pass istio_platform to the Istio chart as global.platform, so istio-cni uses the k3s/k3d CNI paths", + "variants": [ + {"name": "a", "files": ["0002-eas-istio-global-platform.patch"], "for": "main and release-1.2"} + ], + "needed_when": "always", + "fixed_probe": [ + { + "glob": "roles/istio/**", + "regex": "(?m)global\\.platform\\s*=|['\"]platform['\"]\\s*:|global:\\s*\\{\\s*platform\\s*:|^\\s*global:[ \\t]*\\n\\s+platform:", + "knob": "istio_platform", + "what": "an Istio task or default sets global.platform from a variable" + } + ], + "moved": [ + { + "glob": "configs/defaults/global_config.yaml", + "regex": "(?m)^headlamp_enabled:", + "upstream": "https://github.com/intel/enterprise-ai-solutions/pull/61", + "what": "headlamp_enabled now follows velero_enabled, the context of 0002's global_config.yaml hunk" + } + ], + "upstream": { + "filed": false, + "evidence": "https://istio.io/latest/docs/ambient/install/platform-prerequisites/" + } + }, + { + "id": "0003", + "what": "nri_cpu_balloons: pin the nri-plugins balloons chart to v0.15.0", + "variants": [ + {"name": "a", "files": ["0003-eas-pin-nri-balloons-chart.patch"], "for": "main and release-1.2"} + ], + "needed_when": "eas_owns_nri", + "fixed_probe": [ + { + "glob": "roles/nri_cpu_balloons/tasks/install_advanced.yaml", + "regex": "(?m)^\\s*chart_version:", + "knob": null, + "what": "the balloons Helm task passes a chart_version" + } + ], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/containers/nri-plugins/releases/tag/v0.15.0" + } + }, + { + "id": "0004", + "what": "kubernetes: apply the workload-class node labels on multi-node BYO clusters", + "variants": [ + {"name": "a", "files": ["0004-eas-byo-node-topology-labels.patch"], "for": "main and release-1.2"} + ], + "needed_when": "multi_node", + "fixed_probe": [ + { + "glob": "roles/kubernetes/tasks/install.yaml", + "regex": "apply_node_topology\\.yaml", + "knob": null, + "what": "the role's install entry point runs apply_node_topology.yaml itself" + } + ], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/kubernetes/tasks/install.yaml#L23-L25" + } + }, + { + "id": "0005", + "what": "docs: Single-Node Dev on k3d", + "variants": [ + {"name": "a", "files": ["0005-eas-docs-single-node-k3d.patch"], "for": "main"} + ], + "needed_when": "never", + "fixed_probe": [], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/docs/deploy/topologies.md#L189" + } + }, + { + "id": "0006", + "what": "envoy_gateway: select control-plane nodes by label presence (k3s stamps \"true\", kubeadm \"\")", + "variants": [ + {"name": "a", "files": ["0006-eas-envoy-control-plane-affinity.patch"], "for": "main after EAS #42 (roles/envoy_gateway/tasks/controller.yaml)"}, + {"name": "b", "files": ["0006-release-1.2-envoy-control-plane-affinity.patch"], "for": "release-1.2 (roles/envoy_gateway/tasks/install.yaml)"} + ], + "needed_when": "always", + "fixed_probe": [ + { + "glob": "roles/envoy_gateway/**", + "regex": "(?s)\\A(?!.*node-role\\.kubernetes\\.io/control-plane:[ \\t]*(?:\"\"|''))(?=.*?(?PmatchExpressions:\\s*-[ \\t]*key:[ \\t]*[\"']?node-role\\.kubernetes\\.io/control-plane[\"']?\\s+operator:[ \\t]*[\"']?Exists))", + "knob": null, + "what": "a file selects control-plane nodes with a matchExpressions Exists term and has no exact \"\" selector" + }, + { + "glob": "roles/envoy_gateway/vars/k3s.yaml", + "regex": "(?m)^_envoy_gateway_proxy_shape:", + "knob": null, + "what": "a k3s entry in the per-distribution vars layout of EAS #50" + } + ], + "moved": [ + { + "glob": "roles/envoy_gateway/tasks/controller.yaml", + "regex": "_eg_k8s_provider:", + "upstream": "https://github.com/intel/enterprise-ai-solutions/pull/50", + "what": "the EnvoyProxy provider moved into the _eg_k8s_provider var, still an exact \"\" selector (#52 carries the same change)" + } + ], + "upstream": { + "filed": false, + "evidence": "https://github.com/k3s-io/k3s/blob/843c32de489c25ccc1bde53d21c5fd7df23c706e/pkg/server/server.go#L550" + } + } + ], + "verified": [ + { + "profile": "main", + "rev": "97f6157d6a72f7173c698c474d4e51d2c79f1fd6", + "fixture": "tests/fixtures/eas/trees/97f6157", + "cases": [ + {"policy": "best-effort", "nodes": 1, + "expect": {"0002": "apply:a", "0003": "skip", "0004": "skip", "0005": "skip", "0006": "apply:a"}}, + {"policy": "nri-balloons", "nodes": 3, + "expect": {"0002": "apply:a", "0003": "apply:a", "0004": "apply:a", "0005": "skip", "0006": "apply:a"}} + ] + }, + { + "profile": "v1.2.0-rc.1", + "rev": "5c27fb39a95eecec41aa4e29a1188984e9c726f3", + "fixture": "tests/fixtures/eas/trees/5c27fb3", + "cases": [ + {"policy": "best-effort", "nodes": 1, + "expect": {"0002": "apply:a", "0003": "skip", "0004": "skip", "0005": "skip", "0006": "apply:b"}}, + {"policy": "nri-balloons", "nodes": 3, + "expect": {"0002": "apply:a", "0003": "apply:a", "0004": "apply:a", "0005": "skip", "0006": "apply:b"}} + ] + } + ], + "watch": [ + {"ref": "origin/main", "what": "EAS main tip"}, + {"ref": "origin/release-1.2", "what": "EAS release-1.2 tip"}, + {"ref": "pull/50/head", "what": "open: OpenShift platform layer, moves the EnvoyProxy provider into _eg_k8s_provider"}, + {"ref": "pull/52/head", "what": "open: OpenShift ownership labels, stacked on #50"}, + {"ref": "pull/61/head", "what": "open: Headlamp SSO, changes 0002's global_config.yaml context and 0004/0005"} + ] +} diff --git a/integrations/eas/overlay.py b/integrations/eas/overlay.py new file mode 100755 index 00000000..c4c39b47 --- /dev/null +++ b/integrations/eas/overlay.py @@ -0,0 +1,469 @@ +#!/usr/bin/env python3 +"""The k3s/k3d patch overlay for an intel/enterprise-ai-solutions (EAS) checkout. + +For each row of overlay.json it decides, at one EAS commit, and prints the +decision as `fact -> choice [source]`: + + skip the row's needed_when rule is false for these facts + drop-merged a variant reverse-applies (git apply -R --check): upstream has it + drop-fixed a fixed_probe regex matches a file its glob selects, and the knob + the generator writes appears in that file (no knob: the match + alone). A match whose knob is gone is a STOP: upstream fixed it + under another name, and the generator's knob would do nothing. + apply the first variant, in order, that applies (git apply --check) + STOP none of the above: the patch has to be regenerated. The line + names the upstream change that moved the hunk: a `moved` hint of + the row when one matches, else the last commit on the patch's files. + +Decisions read the commit (--rev, default HEAD) through a scratch index, never +the working tree, so they do not change once the overlay is applied. --apply +patches the working tree of the checkout, only when --rev is HEAD and no row +STOPs: every chosen patch the working tree lacks, in one `git apply`. A patch +it already carries (a re-run) is left as it is; a patch's files in any other +state STOP the run before anything is written. + +Patch files are looked up in the overlay.json patch_dirs in order +(integrations/eas/patches, then the upstream drafts in +integrations/eat/upstream/patches), so a draft is never copied. + +Usage: + overlay.py --eas DIR --policy POLICY --nodes N [--rev REV] [--apply] + [--json FILE|-] [--explain] + overlay.py --eas DIR --check-verified every overlay.json `verified` case at its sha + overlay.py --eas DIR --watch decisions at every `watch` ref present in DIR + +POLICY is the kubernetes_cpu_policy the generator chose (nri-balloons, +kubelet-static or best-effort); N is the cluster's node count. Exit 0 when +nothing STOPs, 3 on a STOP (nothing applied), 1 when --check-verified finds a +moved decision, 2 on bad input. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(os.path.dirname(HERE)) +POLICIES = ("nri-balloons", "kubelet-static", "best-effort") +RULES = { + "always": lambda f: True, + "eas_owns_nri": lambda f: f["policy"] == "nri-balloons", + "multi_node": lambda f: f["nodes"] > 1, + "never": lambda f: False, +} +# The k3d-gpu cluster: one server, its own balloons policy kept (--watch). +K3D_GPU_FACTS = {"policy": "best-effort", "nodes": 1} + + +class Bad(Exception): + """Input the overlay cannot work with (exit 2).""" + + +def glob_re(glob): + """A path glob as a regex: ** spans directories, * and ? stay within one.""" + out, i = "", 0 + while i < len(glob): + if glob.startswith("**/", i): + out, i = out + "(?:.*/)?", i + 3 + elif glob.startswith("**", i): + out, i = out + ".*", i + 2 + elif glob[i] == "*": + out, i = out + "[^/]*", i + 1 + elif glob[i] == "?": + out, i = out + "[^/]", i + 1 + else: + out, i = out + re.escape(glob[i]), i + 1 + return re.compile(out + r"\Z") + + +def first_hunk(path): + """The first hunk of a patch: (file, {"old"|"new": (stated start, lines)}, leading context).""" + target, start, sides, ctx, changed = None, None, {"old": [], "new": []}, 0, False + with open(path, encoding="utf-8") as f: + for line in f: + if start is None: + if line.startswith("+++ "): + target = line[4:].rstrip("\n") + target = target[2:] if target.startswith("b/") else target + m = re.match(r"@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))?", line) + if m and target: + start = {"old": int(m.group(1)), "new": int(m.group(3))} + want = {"old": int(m.group(2) or 1), "new": int(m.group(4) or 1)} + continue + if all(len(sides[s]) >= want[s] for s in sides): + break + tag, body = line[:1], line[1:] + if tag == " ": + sides["old"].append(body) + sides["new"].append(body) + ctx += 0 if changed else 1 + elif tag in ("-", "+"): + sides["old" if tag == "-" else "new"].append(body) + changed = True + elif tag != "\\": + break + if start is None: + raise Bad("no hunk in %s" % path) + return target, {s: (start[s], sides[s]) for s in sides}, ctx + + +def landing(eas, path, side): + """(file, line) where a patch's first change sits in the commit: the hunk's `side` + lines ("old" before applying, "new" once applied) located nearest the line the + hunk header states, as git apply locates them.""" + target, sides, ctx = first_hunk(path) + stated, lines = sides[side] + have = eas.text(target).splitlines(keepends=True) + hits = [i + 1 for i in range(len(have) - len(lines) + 1) if have[i:i + len(lines)] == lines] if lines else [] + at = min(hits, key=lambda i: abs(i - stated)) if hits else stated + return target, max(at + ctx, 1) + + +def touched(paths): + """Files a set of patches change.""" + out = [] + for p in paths: + with open(p, encoding="utf-8") as f: + for line in f: + if line.startswith("+++ ") and line[4:].strip() != "/dev/null": + t = line[4:].strip() + t = t[2:] if t.startswith("b/") else t + if t not in out: + out.append(t) + return out + + +class Eas: + """Read-only views of one EAS commit, plus the working-tree apply.""" + + def __init__(self, path, rev): + self.path = os.path.abspath(path) + top = self.git("rev-parse", "--show-toplevel", check=False) + if top.returncode or os.path.realpath(top.stdout.strip()) != os.path.realpath(self.path): + raise Bad("%s is not the top of a git checkout" % path) + self.sha = self.resolve(rev) + if not self.sha: + raise Bad("%s: no commit %s" % (path, rev)) + self._files = None + self._text = {} + self._tmp = tempfile.TemporaryDirectory(prefix="eas-overlay-") + self.index = os.path.join(self._tmp.name, "index") + self.git("read-tree", "--index-output=" + self.index, self.sha) + + def git(self, *args, env=None, check=True): + e = dict(os.environ, **(env or {})) + r = subprocess.run(["git", "-C", self.path] + list(args), capture_output=True, text=True, env=e) + if check and r.returncode: + raise Bad("git %s: %s" % (" ".join(args), r.stderr.strip())) + return r + + def resolve(self, rev): + r = self.git("rev-parse", "--verify", "-q", rev + "^{commit}", check=False) + return r.stdout.strip() if r.returncode == 0 else None + + def files(self): + if self._files is None: + out = self.git("ls-tree", "-r", "-z", "--name-only", self.sha).stdout + self._files = [p for p in out.split("\0") if p] + return self._files + + def text(self, path): + if path not in self._text: + r = self.git("cat-file", "blob", "%s:%s" % (self.sha, path), check=False) + self._text[path] = r.stdout if r.returncode == 0 else "" + return self._text[path] + + def check(self, patches, reverse=False): + """git apply --check at the commit, through the scratch index.""" + args = ["apply", "--cached", "--check", "--whitespace=nowarn"] + (["-R"] if reverse else []) + patches + return self.git(*args, env={"GIT_INDEX_FILE": self.index}, check=False).returncode == 0 + + def last_commit(self, paths): + r = self.git("log", "-1", "--format=%h %s", self.sha, "--", *paths, check=False) + return r.stdout.strip() + + def worktree(self, patches, reverse=False, check=True): + args = ["apply", "--whitespace=nowarn"] + (["--check"] if check else []) + (["-R"] if reverse else []) + patches + return self.git(*args, check=False).returncode == 0 + + +def load_spec(path): + try: + with open(path, encoding="utf-8") as f: + spec = json.load(f) + except (OSError, ValueError) as e: + raise Bad("%s: %s" % (path, e)) from None + if spec.get("schema") != "k3d-gpu.eas-overlay/v1": + raise Bad("%s: schema %r, want k3d-gpu.eas-overlay/v1" % (path, spec.get("schema"))) + for row in spec["rows"]: + if row["needed_when"] not in RULES: + raise Bad("row %s: needed_when %r is not one of %s" % (row["id"], row["needed_when"], ", ".join(RULES))) + for v in row["variants"]: + v["paths"] = [patch_path(spec, f) for f in v["files"]] + for p in row["fixed_probe"] + row["moved"]: + try: + p["re"], p["glob_re"] = re.compile(p["regex"]), glob_re(p["glob"]) + except re.error as e: + raise Bad("row %s: regex %r: %s" % (row["id"], p["regex"], e)) from None + return spec + + +def patch_path(spec, name): + for d in spec["patch_dirs"]: + p = os.path.join(ROOT, d, name) + if os.path.isfile(p): + return p + raise Bad("patch %s is in none of %s" % (name, ", ".join(spec["patch_dirs"]))) + + +def rel(path): + return os.path.relpath(path, ROOT) + + +def line_of(text, pos): + return text.count("\n", 0, pos) + 1 + + +def probe(eas, row): + """(state, file, line, knob): state "fixed", "knob-missing" or None.""" + missing = None + for p in row["fixed_probe"]: + for path in eas.files(): + if not p["glob_re"].match(path): + continue + text = eas.text(path) + m = p["re"].search(text) + if not m: + continue + at = m.start("at") if "at" in p["re"].groupindex and m.start("at") >= 0 else m.start() + knob = p.get("knob") + if knob is None or re.search(r"(? %s [%s]" % (d["fact"], d["choice"], d["permalink"] if explain else d["source"]) for d in decisions] + + +def run(args, spec): + if args.policy not in POLICIES: + raise Bad("--policy %r: want one of %s" % (args.policy, ", ".join(POLICIES))) + if args.nodes < 1: + raise Bad("--nodes %d: a cluster has at least one node" % args.nodes) + facts = {"policy": args.policy, "nodes": args.nodes} + eas = Eas(args.eas, args.rev) + if args.apply and eas.sha != eas.resolve("HEAD"): + raise Bad("--apply patches the working tree, so --rev must be HEAD (%s is not)" % args.rev) + decisions = decide(spec, eas, facts) + stop = any(d["decision"] == "STOP" for d in decisions) + if args.json: + doc = {"schema": "k3d-gpu.eas-overlay-decisions/v1", "repo": spec["repo"], "rev": eas.sha, + "facts": facts, "stop": stop, "decisions": decisions} + if args.json == "-": + json.dump(doc, sys.stdout, indent=2) + print() + else: + with open(args.json, "w", encoding="utf-8") as f: + json.dump(doc, f, indent=2) + f.write("\n") + if args.json != "-": + print("# overlay at eas@%s (%s), kubernetes_cpu_policy=%s, nodes=%d" + % (eas.sha[:7], args.rev, facts["policy"], facts["nodes"])) + print("\n".join(text_lines(decisions, args.explain))) + if stop: + print("STOP: the overlay does not fit eas@%s; regenerate the rows above that say STOP. Nothing was applied." + % eas.sha[:7], file=sys.stderr) + return 3 + if args.apply: + return apply(args, eas, decisions) + return 0 + + +def apply(args, eas, decisions): + """Each chosen patch the working tree lacks, in one git apply; one it already + carries is left alone (a re-run). Any other state of a patch's files: STOP.""" + todo, done = [], [] + for d in decisions: + if d["decision"] != "apply": + continue + patches = [os.path.join(ROOT, f) for f in d["files"]] + if eas.worktree(patches): + todo += patches + elif eas.worktree(patches, reverse=True): + done += patches + else: + todo = None + break + if todo is None or (todo and not eas.worktree(todo)): + bad = touched([os.path.join(ROOT, f) for d in decisions if d["decision"] == "apply" for f in d["files"]]) + print("STOP: the working tree of %s differs from eas@%s in %s; reset it to the commit (git checkout -f %s) " + "and run again. Nothing was applied." % (args.eas, eas.sha[:7], ", ".join(bad), eas.sha[:7]), + file=sys.stderr) + return 3 + if todo and not eas.worktree(todo, check=False): + raise Bad("git apply failed in %s after its --check passed" % args.eas) + if todo: + print("applied %s to %s" % (" ".join(os.path.basename(p) for p in todo), args.eas)) + if done: + print("already applied in %s: %s" % (args.eas, " ".join(os.path.basename(p) for p in done))) + if not todo and not done: + print("nothing to apply to %s" % args.eas) + return 0 + + +def check_verified(args, spec): + bad = 0 + for v in spec["verified"]: + try: + eas = Eas(args.eas, v["rev"]) + except Bad as e: + print("FAIL %s %s: %s (fetch it)" % (v["profile"], v["rev"][:7], e)) + bad += 1 + continue + for c in v["cases"]: + decisions = decide(spec, eas, {"policy": c["policy"], "nodes": c["nodes"]}) + got = {d["id"]: d["brief"] for d in decisions} + label = "%s %s %s nodes=%d" % (v["profile"], eas.sha[:7], c["policy"], c["nodes"]) + moved = ["%s want %s, got %s" % (i, c["expect"].get(i), got.get(i)) + for i in sorted(set(got) | set(c["expect"])) if got.get(i) != c["expect"].get(i)] + summary = " ".join("%s=%s" % kv for kv in sorted(got.items())) + if moved: + bad += 1 + print("FAIL %s: %s" % (label, "; ".join(moved))) + else: + print("ok %s: %s" % (label, summary)) + for line in text_lines(decisions, args.explain): + print(" " + line) + return 1 if bad else 0 + + +def watch(args, spec): + for w in spec["watch"]: + try: + eas = Eas(args.eas, w["ref"]) + except Bad: + print("# %s: not in this clone (%s)" % (w["ref"], w["what"])) + continue + print("# %s = eas@%s (%s), kubernetes_cpu_policy=%s, nodes=%d" + % (w["ref"], eas.sha[:7], w["what"], K3D_GPU_FACTS["policy"], K3D_GPU_FACTS["nodes"])) + print("\n".join(text_lines(decide(spec, eas, K3D_GPU_FACTS), args.explain))) + return 0 + + +def main(argv): + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("--eas", required=True, help="EAS git checkout") + ap.add_argument("--rev", default="HEAD", help="commit to decide at (default HEAD)") + ap.add_argument("--policy", help="kubernetes_cpu_policy the generator chose") + ap.add_argument("--nodes", type=int, help="nodes in the cluster") + ap.add_argument("--apply", action="store_true", help="apply the chosen patches to the working tree") + ap.add_argument("--json", metavar="FILE", help="also write the decisions as JSON to FILE (- for stdout only)") + ap.add_argument("--explain", action="store_true", help="full permalinks instead of short sources") + ap.add_argument("--check-verified", action="store_true", help="check every overlay.json verified case") + ap.add_argument("--watch", action="store_true", help="decisions at every overlay.json watch ref") + ap.add_argument("--spec", default=os.path.join(HERE, "overlay.json"), help=argparse.SUPPRESS) + args = ap.parse_args(argv) + try: + spec = load_spec(args.spec) + if args.check_verified: + return check_verified(args, spec) + if args.watch: + return watch(args, spec) + if args.policy is None or args.nodes is None: + raise Bad("--policy and --nodes are required") + return run(args, spec) + except Bad as e: + print("overlay: %s" % e, file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch b/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch new file mode 100644 index 00000000..48cf316a --- /dev/null +++ b/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch @@ -0,0 +1,77 @@ +From 0d13b34a8ab3c5b271ab8fad3a82b726c013a4b1 Mon Sep 17 00:00:00 2001 +From: Andrew Mello +Date: Thu, 29 Oct 2026 13:35:38 -0700 +Subject: [PATCH] envoy_gateway: select control-plane nodes by label presence + (release-1.2) + +The release-1.2 form of the main-branch change in +roles/envoy_gateway/tasks/controller.yaml. On release-1.2 the EnvoyProxy +task still lives in roles/envoy_gateway/tasks/install.yaml (EAS #42 +moved it to controller.yaml on main), so the same change is cut against +that file. + +The EnvoyProxy DaemonSet pins envoy to control-plane nodes with +nodeSelector node-role.kubernetes.io/control-plane: "". That matches +kubeadm / kubespray nodes, which carry the label with an empty value, +but k3s sets it to "true" (k3s-io/k3s pkg/server/server.go, +util.ControlPlaneRoleLabelKey), so on a BYO k3s or k3d cluster the +DaemonSet schedules no pods and the gateway has no data plane. + +Use a required node affinity with operator Exists on the same key, and +set nodeSelector to null so a re-install over an older EnvoyProxy (a +JSON merge patch, which keeps every key it does not null) drops the old +selector. Behaviour on kubeadm / kubespray clusters is unchanged; the +toleration is untouched. + +Checked statically: release-1.2 pins envoy_gateway_version 1.7.1, the +same gateway-helm whose v1alpha1 EnvoyProxy CRD includes pod.affinity. +Modelling create (null pruned) and re-install over the old object +(RFC 7386 merge) both leave affinity and no nodeSelector. +--- + roles/envoy_gateway/tasks/install.yaml | 23 ++++++++++++++++------- + 1 file changed, 16 insertions(+), 7 deletions(-) + +diff --git a/roles/envoy_gateway/tasks/install.yaml b/roles/envoy_gateway/tasks/install.yaml +index 9f3d423..72c3444 100644 +--- a/roles/envoy_gateway/tasks/install.yaml ++++ b/roles/envoy_gateway/tasks/install.yaml +@@ -191,11 +191,12 @@ + # MetalLB elects at any given moment: every CP node always has a local + # envoy. Survives CP failover, no hostname pinning, no cp[0] dance. + # +-# nodeSelector keys on the standard control-plane label (empty string is +-# how kubespray / kubeadm stamp it) rather than the taint, because +-# operators can remove the taint; the label is the stable Kubernetes +-# control-plane marker. Toleration is kept for the case where the taint +-# is still in place. ++# Node affinity keys on the standard control-plane label rather than the ++# taint, because operators can remove the taint; the label is the stable ++# Kubernetes control-plane marker. It matches on the label's presence, not ++# its value: kubespray / kubeadm stamp it with "" while k3s stamps ++# it with "true", and an exact-value nodeSelector schedules no envoy there. ++# Toleration is kept for the case where the taint is still in place. + - name: "envoy_gateway | Create EnvoyProxy config (DaemonSet on control-plane nodes)" + kubernetes.core.k8s: + state: present +@@ -211,8 +212,16 @@ + kubernetes: + envoyDaemonSet: + pod: +- nodeSelector: +- node-role.kubernetes.io/control-plane: "" ++ # null drops the nodeSelector an older install left behind: ++ # this update is a JSON merge patch, which keeps omitted keys. ++ nodeSelector: null ++ affinity: ++ nodeAffinity: ++ requiredDuringSchedulingIgnoredDuringExecution: ++ nodeSelectorTerms: ++ - matchExpressions: ++ - key: node-role.kubernetes.io/control-plane ++ operator: Exists + tolerations: + - key: node-role.kubernetes.io/control-plane + operator: Exists +-- +2.53.0 + diff --git a/integrations/eat/upstream/README.md b/integrations/eat/upstream/README.md index 59f72604..2ebf3485 100644 --- a/integrations/eat/upstream/README.md +++ b/integrations/eat/upstream/README.md @@ -8,7 +8,7 @@ These are drafts for Intel's repos. **None of them has been filed, opened or pus - It deletes the toolkit's own installer: `deploy-agentic-stack.sh`, `core/lib/brownfield/`, `core/playbooks/setup-bastion.yml`, `core/playbooks/deploy-istio.yml`, `core/roles/nri_cpu_balloons/`. - It moves the toolkit onto `es_auto_installer.sh install agent-toolkit` in [intel/enterprise-ai-solutions](https://github.com/intel/enterprise-ai-solutions) (EAS) and intel/enterprise-inference. - It depends on intel/enterprise-inference#22 and goes in ahead of intel/enterprise-ai-solutions#40. Both are open. - - Re-read 2026-10-29: #19 merged on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no tag carries it. intel/enterprise-inference#22 merged on 2026-10-22. intel/enterprise-ai-solutions#40 was closed unmerged on 2026-10-22, and intel/enterprise-ai-solutions#66 ("Integrate Agentic Toolkit"), opened a minute before, merged on 2026-10-29. Every patch here still passes `checks/apply-check.sh` on EAS `main` `5767995`. + - Re-read 2026-10-29: #19 merged on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no tag carries it. intel/enterprise-inference#22 merged on 2026-10-22. intel/enterprise-ai-solutions#40 was closed unmerged on 2026-10-22, and intel/enterprise-ai-solutions#66 ("Integrate Agentic Toolkit"), opened a minute before, merged on 2026-10-29. Every patch here still passes `checks/apply-check.sh` on EAS `main` `5767995`, and on `97f6157` (#69), the `main` head later that day. On the EAT 2.0 pre-release `v1.2.0-rc.1` (`5c27fb3`), 0005 and 0006 fail: the `release-1.2` branch still has the EnvoyProxy task in `roles/envoy_gateway/tasks/install.yaml`, which the overlay's release-1.2 form of 0006 covers. Open EAS PRs #50 and #52 break 0006, and #61 breaks 0002, 0004 and 0005. `k3d-gpu eas` applies these drafts through [the k3d-gpu overlay](#the-k3d-gpu-overlay) ([docs/intel/eas.md](../../../docs/intel/eas.md#patch-overlay)). - The platform roles that matter for k3s (istio, envoy_gateway, kubernetes, nri_cpu_balloons) already live in EAS `main`. **EAS is the target for every patch here.** - There is no toolkit (EAT) patch. Its k3s platform-detection gap sits in `brownfield_deployment()`, which no entry point calls: it is sourced at [deploy-agentic-stack.sh#L631](https://github.com/intel/enterprise-agent-toolkit/blob/0ecf67a8331e8b502ef41e08bf6d2881416178c5/deploy-agentic-stack.sh#L631) and called only from inside itself, and #19 deletes the file. The issue draft lists it as an observation only. @@ -50,6 +50,18 @@ The patches carry no `Signed-off-by:`. EAS's CONTRIBUTING.md requires a DCO sign git am --signoff integrations/eat/upstream/patches/0002-eas-istio-global-platform.patch ``` +## The k3d-gpu overlay + +Until upstream carries them, k3d-gpu applies these drafts itself to the EAS checkout it installs from. [integrations/eas/overlay.json](../../eas/overlay.json) lists each patch with its variants, when it is needed (`always`, `eas_owns_nri`, `multi_node`, `never`), a probe for an upstream fix done another way, and the knob the generator writes. [integrations/eas/overlay.py](../../eas/overlay.py) decides per patch at the pinned EAS commit and prints `fact -> choice [source]`: + +- `skip`: not needed on this cluster. +- `drop-merged`: the patch reverse-applies. +- `drop-fixed`: the probe matches and the knob is still there. +- `apply`: the first variant that applies. +- `STOP`, before anything changes: it names the upstream change that moved the hunk and says to regenerate. + +The drafts here stay the only copy. The one overlay-only patch is the release-1.2 form of 0006, [integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch](../../eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch); on that branch the EnvoyProxy task is still in `install.yaml`. + ## Static validation | Check | Result | @@ -63,11 +75,14 @@ git am --signoff integrations/eat/upstream/patches/0002-eas-istio-global-platfor | 0004 behaviour, [checks/eas-byo-topology.sh](checks/eas-byo-topology.sh) (stub `kubernetes.core`) | upstream: BYO 3 nodes → no labels (FAIL); patched: server `platform`, agents `inference`. BYO with one node already labelled → that node untouched; BYO with node list or patch denied → install succeeds, no labels; managed with patch denied → still fails. BYO 1 node, topology off, managed single- and multi-node all unchanged | | 0006 behaviour | the patched `EnvoyProxy` validates against the v1alpha1 CRD in gateway-helm v1.7.1, which includes `pod.affinity` and does not mark `nodeSelector` nullable | | 0006 re-install, [checks/eas-envoy-reinstall.py](checks/eas-envoy-reinstall.py) | models kubernetes.core 6.3.0 updating a custom resource (JSON merge patch, RFC 7386) over the object the unpatched task created. Without `nodeSelector: null` the old selector survives next to the affinity (FAIL); with it, gone. Fresh install: the null is pruned on create | +| 0006 release-1.2 variant, same script on EAS `release-1.2` `5c27fb3` before and after the variant | fresh install and re-install over the unpatched object both leave `nodeSelector: None` and the `Exists` affinity (2026-10-29). release-1.2 pins `envoy_gateway_version: "1.7.1"`, the gateway-helm the 0006 CRD check used | +| Overlay decisions, `checks/apply-check.sh --overlay` on a fresh clone (2026-10-29) | EAS `main` `97f6157`: 0002 and 0006 apply (0006 variant a), 0003 skips unless the policy is nri-balloons, 0004 skips on one node, 0005 always skips. `release-1.2` `5c27fb3`: the same, with 0006 variant b. Open PRs: #50 and #52 STOP 0006 (the EnvoyProxy provider moves into `_eg_k8s_provider`), #61 STOPs 0002 (`headlamp_enabled` in its `global_config.yaml` context) | How to rerun the checks (an EAS clone; `ansible-core` and `jmespath` for the topology check; PyYAML for the envoy check): ```bash integrations/eat/upstream/checks/apply-check.sh ../enterprise-ai-solutions +integrations/eat/upstream/checks/apply-check.sh --overlay # clones EAS itself; needs the network integrations/eat/upstream/checks/eas-byo-topology.sh integrations/eat/upstream/checks/eas-envoy-reinstall.py ``` diff --git a/integrations/eat/upstream/checks/apply-check.sh b/integrations/eat/upstream/checks/apply-check.sh index cc8429bc..dc9cfa57 100755 --- a/integrations/eat/upstream/checks/apply-check.sh +++ b/integrations/eat/upstream/checks/apply-check.sh @@ -4,16 +4,44 @@ # that the series applies in order with git am. The clone is copied, never # modified. # +# With --overlay it prints the decisions of integrations/eas/overlay.py, the +# overlay `k3d-gpu eas up` applies, against a real EAS clone: every +# overlay.json `verified` case at its pinned sha (fails when a recorded decision +# moved), then every `watch` ref (branch tips and open upstream PRs) for +# k3d-gpu's own facts. Without a clone it clones intel/enterprise-ai-solutions; +# a given clone is copied first. Fetches branches and PR heads, so it needs the +# network and stays out of CI. +# # Usage: integrations/eat/upstream/checks/apply-check.sh +# integrations/eat/upstream/checks/apply-check.sh --overlay [] set -uo pipefail -EAS=${1:?usage: $0 } PATCHES="$(cd "$(dirname "$0")/../patches" && pwd)" +ROOT="$(cd "$(dirname "$0")/../../../.." && pwd)" WORK=$(mktemp -d) trap 'rm -rf "${WORK}"' EXIT FAILS=0 +if [ "${1:-}" = --overlay ]; then + if [ -n "${2:-}" ]; then + cp -a "$2" "${WORK}/eas" + else + git clone -q https://github.com/intel/enterprise-ai-solutions.git "${WORK}/eas" || exit 2 + fi + git -C "${WORK}/eas" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' \ + '+refs/pull/*/head:refs/remotes/pull/*/head' || exit 2 + echo "## verified pins (integrations/eas/overlay.json)" + python3 "${ROOT}/integrations/eas/overlay.py" --eas "${WORK}/eas" --check-verified || FAILS=$((FAILS + 1)) + echo + echo "## watch refs, k3d-gpu facts" + python3 "${ROOT}/integrations/eas/overlay.py" --eas "${WORK}/eas" --watch || FAILS=$((FAILS + 1)) + [ "${FAILS}" -eq 0 ] + exit +fi + +EAS=${1:?usage: $0 | --overlay []} + # fresh: a throwaway copy with the clone's HEAD checked out clean. fresh() { rm -rf "${WORK:?}/eas" diff --git a/mkdocs.yml b/mkdocs.yml index 7177a2b6..acbcbaad 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,6 +36,7 @@ nav: - Crescent removed models: hosts/crescent-removed-models.md - Integrations: - GPUStack: integrations/gpustack.md + - Intel EAS (EAT 2.0): intel/eas.md - Design: - Speculative decoding: design/speculative-decoding.md - Moat: moat-design.md diff --git a/tests/eas_test.sh b/tests/eas_test.sh new file mode 100755 index 00000000..6c27dc3c --- /dev/null +++ b/tests/eas_test.sh @@ -0,0 +1,437 @@ +#!/usr/bin/env bash +# Checks for integrations/eas without a cluster. With no argument every section +# runs; name sections to run only those (tests/eas_test.sh overlay). +# +# overlay integrations/eas/overlay.py, the k3s/k3d patch overlay, on synthetic +# git repos built from verbatim EAS files (tests/fixtures/eas/trees, +# EAS main 97f6157 and release-1.2 5c27fb3): +# - the fixtures are byte-identical to upstream (blob shas in SOURCES) +# - overlay.json resolves every patch, upstream drafts first-class, no copies +# - main: 0002 and 0006 apply, 0003 only for nri-balloons, 0004 only on +# more than one node, 0005 never; the recorded `verified` cases hold +# - patches already in the tree: drop-merged (git apply -R --check) +# - the fix done another way with the knob present: drop-fixed +# - the fix done another way with the knob renamed: STOP +# - the hunk moved as open EAS #50/#52 move it: STOP naming #50 +# - release-1.2: 0006 variant b applies, 0006a and 0005 do not +# - --apply changes the checkout only when no row STOPs, and a re-run is a no-op +# - every decision prints as `fact -> choice [source]`; --explain gives permalinks +# Needs bash, git, python3. + +set -uo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +PASS=0 +FAILS=0 +WORK=$(mktemp -d) +trap 'rm -rf "${WORK}"' EXIT + +ok() { PASS=$((PASS + 1)); echo "ok $1"; } +fail() { FAILS=$((FAILS + 1)); echo "FAIL $1"; } + +# check