eas: eas check names why the stack is not ready, and checks the guardrail (G10, G11)

integrations/eas/README.md

@@ -11,7 +11,7 @@ the pinned commit. Gaudi and Arc stay with `k3d-gpu agent up`.
 ```bash
 k3d-gpu eas up --dry-run --agent-toolkit   # facts, choices and overlay decisions; changes nothing
 k3d-gpu eas up --agent-toolkit             # creates the cluster first if it is absent
-k3d-gpu eas check                          # chat, tool call and a sandbox run through EAS's LiteLLM
+k3d-gpu eas check                          # stack readiness, then chat, tool call, guard and sandbox through EAS's LiteLLM
 k3d-gpu eas status
 k3d-gpu eas down                           # teardown platform; the cluster stays
 ```
@@ -59,6 +59,36 @@ State per cluster: `~/.local/state/k3d-gpu/eas-<cluster>/` (mode 0700): the EAS
 its `env/k3d-gpu/` and `ext/`, `facts.json`, `pins.resolved.json`, `eas-config.py`'s files,
 `overlay.txt` and `overlay-decisions.json`, `kubeconfig`, `layer` and `logs/`.
 
+## What `eas check` does
+
+`eas check` needs the agent toolkit (LiteLLM and the sandbox come with it). It runs
+`integrations/eat/eat-check.sh` with `EAT_MODE=eas`:
+
+1. `eas-ready.py` on `kubectl get pods,daemonsets -A -o json`, before any request. Every
+   pod and DaemonSet in `istio-system`, `envoy-gateway-system`, `litellm`, `llm-inference`
+   and `agent-sandbox-system` must be Ready. A Job's pods are not graded, nor a Failed pod
+   under a ReplicaSet (kubelet keeps an Evicted pod; its replacement is graded). Istio
+   ambient's `istio-cni-node` and `ztunnel` DaemonSets must exist, eg-gateway must have a
+   Ready envoy proxy pod, `llm-inference` must hold a model pod besides the guard, and the
+   cluster must have at most one NRI resource policy and one NFD worker. Each pod that is
+   not Ready is printed with its cause where its status names one: Pending on the AMXBF16
+   node affinity (EI's `defaults.require_amx: true` on a cluster without AMX), Pending on
+   Insufficient cpu or memory (with the pod's requests), a DaemonSet whose nodeSelector
+   matches no node (EAS's EnvoyProxy on k3s, overlay 0006), an `istio-system` pod waiting
+   (istio-cni and ztunnel chain into the node's CNI paths, `istio_platform`). The guard
+   model is fail-closed: while it is not Ready, LiteLLM refuses every chat, and the check
+   says so. No key is minted for a stack that is not ready. `eas status` prints the same
+   report, without stopping, after its list of pods not Ready: the place to look when
+   `eas up` fails on a model or the guard that never turns Ready.
+2. A LiteLLM virtual key, minted with the master key and deleted at exit.
+3. A chat completion and a `get_weather` tool call through eg-gateway. Any status but 200
+   fails with LiteLLM's `error.message`.
+4. When `GET /v2/guardrails/list` lists `content_safety` (EI's guardrail, on with the agent
+   toolkit unless `guardrails_enabled` is off): the unsafe example of EI's guardrail docs,
+   "How do I hack into a system?", must come back as an error whose message names the
+   content-safety guardrail and unsafe content. HTTP 200 fails the check.
+5. The model writes Python, the agent sandbox router runs it, and it must print 385.
+
 ## Files
 
 | File | Role |
@@ -69,6 +99,7 @@ its `env/k3d-gpu/` and `ext/`, `facts.json`, `pins.resolved.json`, `eas-config.p
 | `images.yaml` | the agent toolkit's floating images by digest (sandbox router `latest-main`, Python runtime v0.5.0) |
 | `known-issues.json` | upstream issues this path works around, one flat row each |
 | `eas-config.py`, `pins.json` | the choice generator and the pins it reads |
+| `eas-ready.py` | the stack readiness check `eas check` runs first (stdin: `kubectl get pods,daemonsets -A -o json`) |
 | `overlay.py`, `overlay.json`, `patches/` | the k3s/k3d patch overlay |
 
 ## Not verified

integrations/eas/eas-ready.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""Is the EAS agent stack ready to serve, and if not, why: the check `k3d-gpu eas check`
+runs (through eat-check.sh EAT_MODE=eas) before it sends its first request.
+
+  kubectl get pods,daemonsets -A -o json | eas-ready.py
+
+Reads the List on stdin and grades the namespaces a chat through LiteLLM and a sandbox
+run depend on. Every pod there must be Ready, except a Job's (model downloads finish
+and stay behind) and a Failed one under a ReplicaSet (kubelet keeps an Evicted pod
+until it is deleted; the ReplicaSet runs its replacement, which is graded). Every
+DaemonSet there must schedule a pod and have all of them Ready, and Istio ambient's
+istio-cni-node and ztunnel must exist. Then: at least one Ready envoy proxy pod for
+eg-gateway, at least one model pod, at most one NRI resource policy and at most one
+NFD worker cluster-wide.
+
+Each problem prints with its cause where the pod status names one:
+  - Pending on the AMXBF16 nodeAffinity: EI's models.yaml defaults.require_amx is true
+    (model_manager/core/renderer.py:90-95 at EI 3e8a244) on a cluster with no AMX node
+  - Pending on Insufficient cpu or memory: the pod's requests do not fit
+  - a DaemonSet that schedules no pod: its nodeSelector matches no node
+  - a pod in istio-system not Ready: istio-cni and ztunnel need the node's CNI paths
+  - a pod being deleted: Terminating; a Failed pod: its status reason and message
+The guard model is fail-closed (EI roles/litellm/templates/qwen3guard_gen.py.j2):
+while it is not Ready, LiteLLM refuses every chat request.
+
+Exit 0 when the stack is ready, 1 with a problem, 2 on input that is not a List.
+Python standard library only.
+"""
+
+import json
+import sys
+
+AMX_LABEL = "feature.node.kubernetes.io/cpu-cpuid.AMXBF16"
+# What each namespace serves, for the report. EAS 97f6157 and EI 3e8a244 defaults:
+# istio_namespace, the envoy gateway, the litellm role, models.yaml defaults.namespace,
+# and EAT 036b662's agent sandbox.
+STACK = {
+    "istio-system": "Istio ambient",
+    "envoy-gateway-system": "Envoy Gateway",
+    "litellm": "LiteLLM",
+    "llm-inference": "the models",
+    "agent-sandbox-system": "the agent sandbox",
+}
+EG_LABEL = "gateway.envoyproxy.io/owning-gateway-name"
+GUARD_PREFIX = "qwen3guard"  # EI litellm role default guardrail_model
+# Istio ambient's data plane: EAS 97f6157 charts/istio (profile ambient, subcharts cni and
+# ztunnel); EAT enrols agent-sandbox-system in ambient.
+AMBIENT = ("istio-cni-node", "ztunnel")
+
+
+def owned_by(obj, kind):
+    return any(o.get("kind") == kind for o in obj["metadata"].get("ownerReferences") or [])
+
+
+def graded(pod):
+    """False for a pod that says nothing about the stack now: one that Succeeded, a Job's,
+    and a Failed one under a ReplicaSet, which has already started its replacement."""
+    phase = (pod.get("status") or {}).get("phase")
+    if phase == "Succeeded" or owned_by(pod, "Job"):
+        return False
+    return not (phase == "Failed" and owned_by(pod, "ReplicaSet"))
+
+
+def required_affinity_keys(spec):
+    terms = (((spec.get("affinity") or {}).get("nodeAffinity") or {})
+             .get("requiredDuringSchedulingIgnoredDuringExecution") or {}).get("nodeSelectorTerms") or []
+    return {e.get("key") for t in terms for e in t.get("matchExpressions") or []}
+
+
+def requests(spec):
+    got = {}
+    for c in spec.get("containers") or []:
+        for k, v in ((c.get("resources") or {}).get("requests") or {}).items():
+            got.setdefault(k, v)
+    return ", ".join(f"{k} {v}" for k, v in sorted(got.items()) if k in ("cpu", "memory")) or "none"
+
+
+def condition(status, kind):
+    for c in status.get("conditions") or []:
+        if c.get("type") == kind:
+            return c
+    return {}
+
+
+def is_ready(pod):
+    st = pod.get("status") or {}
+    if st.get("phase") != "Running":
+        return False
+    cs = st.get("containerStatuses") or []
+    return bool(cs) and all(c.get("ready") for c in cs) and condition(st, "Ready").get("status", "True") == "True"
+
+
+def cause(pod):
+    """(state, cause) for a pod that is not Ready."""
+    ns = pod["metadata"]["namespace"]
+    spec, st = pod.get("spec") or {}, pod.get("status") or {}
+    if pod["metadata"].get("deletionTimestamp"):
+        return ("Terminating", "a rollout or a deletion is under way: re-run when it is done")
+    sched = condition(st, "PodScheduled")
+    if sched.get("status") == "False":
+        msg = sched.get("message") or sched.get("reason") or "unschedulable"
+        if AMX_LABEL in required_affinity_keys(spec) and "affinity" in msg:
+            return (f"Pending: {msg}",
+                    f"it needs a node labelled {AMX_LABEL}=true and none is: env/k3d-gpu/models.yaml "
+                    "defaults.require_amx is true on a cluster without AMX. k3d-gpu eas up writes it from the "
+                    "node labels; re-run it, or set defaults.require_amx: false there")
+        short = [r for r in ("cpu", "memory") if f"Insufficient {r}" in msg]
+        if short:
+            return (f"Pending: {msg}",
+                    f"the node lacks free {' and '.join(short)} for its requests ({requests(spec)}): "
+                    "a smaller model (k3d-gpu eas up --agent-toolkit --model NAME), or free the node")
+        if "selector" in msg or "affinity" in msg:
+            return (f"Pending: {msg}", f"nodeSelector {json.dumps(spec.get('nodeSelector') or {})} and its "
+                    "node affinity match no node")
+        return (f"Pending: {msg}", "")
+    for c in st.get("containerStatuses") or []:
+        w = (c.get("state") or {}).get("waiting")
+        if w:
+            state = f"{w.get('reason', 'waiting')}: {w.get('message', '')}".rstrip(": ")
+            if ns == "istio-system":
+                return (state, "istio-cni and ztunnel chain into the node's CNI paths, which istio_platform "
+                        "picks (k3d: /bin, k3s: /var/lib/rancher/k3s/data/cni); see choices.txt and overlay 0002")
+            return (state, "")
+    if st.get("phase") == "Failed":
+        return (": ".join(x for x in ("Failed", st.get("reason"), st.get("message")) if x), "")
+    if st.get("phase") == "Running":
+        return ("Running, readiness probe not passed yet", "a model loads for minutes: wait and re-run")
+    return (st.get("phase") or "unknown phase", "")
+
+
+def check(items):
+    problems = []
+    pods = [i for i in items if i.get("kind") == "Pod"]
+    sets = [i for i in items if i.get("kind") == "DaemonSet"]
+    count = 0
+    for p in pods:
+        ns, name = p["metadata"]["namespace"], p["metadata"]["name"]
+        if ns not in STACK or not graded(p):
+            continue
+        count += 1
+        if is_ready(p):
+            continue
+        state, why = cause(p)
+        if ns == "llm-inference" and name.startswith(GUARD_PREFIX):
+            why = ("the guardrail model; it is fail-closed, so LiteLLM refuses every chat while it is not "
+                   "Ready" + (". " + why if why else ""))
+        problems.append(f"{ns}/{name} ({STACK[ns]}): {state}" + (f" -> {why}" if why else ""))
+    for name in AMBIENT:
+        if not any(d["metadata"]["namespace"] == "istio-system" and d["metadata"]["name"] == name for d in sets):
+            problems.append(f"istio-system/{name}: no DaemonSet -> Istio ambient's data plane is missing (EAS "
+                            "installs istio-cni-node and ztunnel with charts/istio's ambient profile)")
+    for d in sets:
+        ns, name = d["metadata"]["namespace"], d["metadata"]["name"]
+        if ns not in STACK:
+            continue
+        st = d.get("status") or {}
+        want, ready = st.get("desiredNumberScheduled", 0), st.get("numberReady", 0)
+        if want == 0:
+            sel = ((d.get("spec") or {}).get("template") or {}).get("spec", {}).get("nodeSelector") or {}
+            hint = " (EAS's EnvoyProxy selects control-plane nodes by the value \"\", k3s stamps \"true\": " \
+                   "overlay 0006)" if ns == "envoy-gateway-system" else ""
+            problems.append(f"{ns}/{name}: DaemonSet schedules no pod -> nodeSelector {json.dumps(sel)} "
+                            f"matches no node{hint}")
+        elif ready < want:
+            problems.append(f"{ns}/{name}: {ready}/{want} Ready")
+    if not any(p["metadata"]["namespace"] == "envoy-gateway-system"
+               and (p["metadata"].get("labels") or {}).get(EG_LABEL) == "eg-gateway" and is_ready(p) for p in pods):
+        problems.append("envoy-gateway-system: no Ready envoy proxy pod for eg-gateway -> the gateway has no "
+                        "data plane, so nothing reaches LiteLLM")
+    models = [p for p in pods if p["metadata"]["namespace"] == "llm-inference"
+              and not p["metadata"]["name"].startswith(GUARD_PREFIX) and graded(p)]
+    if not models:
+        problems.append("no model pod in llm-inference -> llm_services_autodeploy_models may be [] (k3d-gpu fit "
+                        "found no model that fits; see choices.txt), or the install stopped before model-manager")
+    nri = sorted(f"{d['metadata']['namespace']}/{d['metadata']['name']}" for d in sets
+                 if d["metadata"]["name"].startswith("nri-resource-policy-"))
+    if len(nri) > 1:
+        problems.append(f"{len(nri)} NRI resource policies ({', '.join(nri)}) -> one per node only: "
+                        "kubernetes_cpu_policy must be best-effort beside k3d-gpu's balloons policy")
+    nfd = sorted(f"{d['metadata']['namespace']}/{d['metadata']['name']}" for d in sets
+                 if d["metadata"]["name"] in ("nfd-worker", "node-feature-discovery-worker"))
+    if len(nfd) > 1:
+        problems.append(f"{len(nfd)} NFD workers ({', '.join(nfd)}) -> nfd_enabled must be false beside "
+                        "k3d-gpu's NFD")
+    return count, problems
+
+
+def main():
+    try:
+        doc = json.load(sys.stdin)
+    except ValueError as e:
+        print(f"eas-ready: stdin is not JSON: {e}", file=sys.stderr)
+        return 2
+    if not isinstance(doc, dict) or doc.get("kind") != "List" or not isinstance(doc.get("items"), list):
+        print("eas-ready: stdin is not a kubectl List (kubectl get pods,daemonsets -A -o json)", file=sys.stderr)
+        return 2
+    count, problems = check(doc["items"])
+    if problems:
+        print(f"stack not ready ({len(problems)}):")
+        for line in problems:
+            print("  " + line)
+        return 1
+    print(f"stack ready: {count} pods Ready in {', '.join(STACK)}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())