[88plug]

Stacked on #260 (k3d-gpu eas up, down, status and check), rebased on its head 5101365. It fills the check-side part of two gaps from the agent-stack gap matrix that #248 to #260 leave open:

• G10, model placement on hosts without AMX, and the fail-closed guard. eas: EAS and EAT config from cluster facts, the pins file, fixture tests #251 already writes defaults.require_amx from the AMXBF16 node label (integrations/eas/eas-config.py:411-419, models.edit.yaml). Still missing: eas check should fail with a named cause when the guard or a model pod is not Ready, and it should prove that the guard blocks.
• G11, Istio ambient on k3d. eas: k3s/k3d patch overlay that drops each patch once upstream carries the fix #248's overlay 0002 and eas: EAS and EAT config from cluster facts, the pins file, fixture tests #251's istio_platform choice (eas-config.py:320-362) make the mesh work. Still missing: eas check should assert that istio-cni and ztunnel are Ready.

Before this change, a stack whose Qwen3Guard pod was Pending on the AMX node affinity failed eas check only with an empty chat reply or a curl exit code. The guardrail itself was never exercised.

What changes

integrations/eat/eat-check.sh in eas mode (k3d-gpu eas check):

1. Before any request it runs the new integrations/eas/eas-ready.py on kubectl get pods,daemonsets -A -o json.

   • Every pod (Jobs excepted) and every DaemonSet in these namespaces must be Ready: istio-system, envoy-gateway-system, litellm, llm-inference and agent-sandbox-system.
   • eg-gateway needs a Ready envoy proxy pod.
   • llm-inference needs a model pod besides the guard.
   • The cluster may have at most one NRI resource policy and at most one NFD worker.
   • Each pod that is not Ready is printed with its cause, wherever its status names one:
     • Pending on the AMXBF16 affinity points to require_amx. The source is EI 3e8a244 model_manager/core/renderer.py:90-95.
     • Insufficient cpu/memory is reported with the pod's requests.
     • A DaemonSet whose nodeSelector matches no node points to EAS's EnvoyProxy on k3s and overlay 0006.
     • A waiting istio-system pod points to the CNI paths and istio_platform.
   • For the guard, the report adds that it is fail-closed, so LiteLLM refuses every chat while it is not Ready.
   • No virtual key is minted for a stack that is not ready.

2. A chat that does not return 200 now fails with LiteLLM's error.message instead of an empty reply. For example, a fail-closed guard that cannot reach its model returns "the guard model backend is unreachable".

3. When GET /v2/guardrails/list (read with the master key) lists content_safety, the check sends the unsafe example from EI's guardrail docs ("How do I hack into a system?") with the virtual key.

   • It must come back as an error carrying the guard's unsafe content verdict. HTTP 200 fails the check.
   • With no registered guardrail (guardrails_enabled off), this step is skipped and the check says so.
   • The texts come from docs/customize/guardrails.md and roles/litellm/templates/qwen3guard_gen.py.j2 (BLOCK_PREFIX). They are the same at EI 3e8a244 (profile main) and 170eb3a (profile v1.2.0-rc.1).

4. eas status prints the same readiness report after its list of pods not Ready, without failing. This matters at the main profile's pins: EI 3e8a244 deploys the guard and the autodeploy models with _mm_strict and model-manager deploy --wait (roles/llm_services/tasks/install.yaml, deploy_models.yaml). A guard Pending on the AMX affinity therefore fails eas up itself, and eas check will not start without a finished layer. The eas up failure message now points at eas status.

Other changes: eas check now requires python3, and eas-ready.py joins the files eas up requires. The help text and integrations/eas/README.md ("What eas check does") describe the steps. Packaging ships integrations/eas/ as a tree, so eas-ready.py is included with no packaging change.

Tests (written first)

Tests come first, in two pairs. At the first commit, 9 eat_test cases fail, and the eas_test ready section fails because its driver stops without eas-ready.py. At the third commit, the two eas status cases fail.

• tests/eas_test.sh ready: 17 cases on synthetic lists. They cover a ready stack, AMX Pending for a guard and for a model, Insufficient memory and cpu, a DaemonSet with no pod, no envoy proxy, no model, an istio-cni CrashLoopBackOff, a partly Ready DaemonSet, a second NRI policy, a second NFD worker, a readiness probe that has not passed yet, and bad input.

• tests/eas_test.sh cli: eas status prints the guard's AMX cause from a stub kubectl get pods,daemonsets -A -o json.

• tests/eat_test.sh adds 12 cases:

  • the stack check runs first in eas mode and not at all in eat mode
  • no key is minted when the stack is not ready
  • the guard blocks, lets the example through, is down, or is not registered
  • the guardrail list is read with the master key
  • no key appears in any curl argv

  The curl stub now honours -f and -w.

Results on this branch:

tests/eas_test.sh          379 passed, 0 failed
tests/eat_test.sh           99 passed, 0 failed
tests/ci_wiring_test.sh     70 passed, 0 failed
tests/packaging_test.sh    149 passed, 0 failed
tests/platforms_doc_test.sh 195 passed, 0 failed
tests/claims_check_test.sh  81 passed, 0 failed
tests/claims_check.sh       claims_check: 26 file(s) clean
python3 tests/ledger_drift_check_test.py   OK
python3 tests/ledger_drift_check.py        41 file(s) agree with 81 ledger rows
shellcheck v0.11.0 --severity=warning: eat-check.sh, scripts/k3d-gpu, tests/eat_test.sh, tests/eas_test.sh clean

tests/docs_test.sh: 63 passed, 0 failed. mkdocs build --strict exits 0 (docs-build/requirements.txt, --require-hashes).

Review fixes (f428858, 262b981)

• An Evicted pod stays in phase Failed until something deletes it, while its ReplicaSet runs the replacement. eas-ready.py graded it, so one eviction of a model or of LiteLLM failed every later eas check with a bare "Failed". A Failed pod under a ReplicaSet is no longer graded and no longer counts as a model pod. A Failed pod nothing replaces is reported with its status reason and message.
• A pod being deleted is reported as Terminating, not as a model whose readiness probe has not passed.
• A stack with no Istio ambient data plane passed, because only existing pods and DaemonSets were graded. k3d-gpu keeps Istio on, and EAS installs it with the ambient profile (charts/istio at 97f6157, Istio 1.27.3). The check now requires the istio-cni-node and ztunnel DaemonSets in istio-system, which EAS's own istio validate also asserts for ztunnel.

Six new tests/eas_test.sh ready cases were committed first and failed for these reasons. The tests/eat_test.sh ready fixture gains the two Istio DaemonSets.

Not verified

Nothing here has run against a live EAS stack. F3 (#249) is still pending, and its step 5 is k3d-gpu eas check. With this PR, that step also requires the stack pods to be Ready and the guard to block. F3's step 4 already waits for every pod to be Ready, and its predictions are unchanged. docs/intel/eas.md (#249) still lists eas check as chat plus sandbox and can be updated when the two land.