[88plug]

Runtime and surface canary for engine and node images (freshness part P5).

No CI job built the node image, started a cluster, or pulled and served an engine. Every engine path in the tests was a stub, so an engine bump from a bot or a person merged without ever running. This adds .github/workflows/canary.yml (name Canary), which checks each bump at runtime before it merges.

This PR's own Canary run on 5be20ca (run 40980204951) is green for all 15 engines, each with verdict pass, hard true and exact true. That covers node, llama-cpu, ovms-cpu, vllm-cpu and every surface engine.

What it checks

Kind	Engines	Check
runtime	vllm-cpu, ovms-cpu, llama-cpu; eat-vllm-cpu (informational)	Pull the pinned image and serve a pinned Qwen3-0.6B the way the launcher runs it. Then one completion (HTTP 200, finish_reason stop or length, tokens > 0, /Paris/), one stream (2 or more chunks, then [DONE]), and the server's version against the pinned tag. A golden mismatch keeps hard and clears exact.
surface	vllm-xpu, llm-scaler, gaudi-vllm, llama-sycl, llama-vulkan, llama-full, dlstreamer, sglang-xeon, npu-umd, ovms, plus the runtime engines first	Every flag and VLLM_* name that the launcher and console/model_opt.py emit (canary/flags.sh) exists in the image. They are read from the image's own --help, or for vLLM statically from arg_utils.py, entrypoints/**/cli_args.py and envs.py inside it, without importing them. The constants derived from the image still match: console/plan.py's vLLM versions, the launcher's vver, NPU_UMD_DRIVER against the image's intel-level-zero-npu, and NPU_UMD_FW_DATE against its firmware.
node	node	Build the root Dockerfile (26.04). Run k3d cluster create canary --image ... --wait with the k3d packaging/install.sh pins (sha256-checked) and the arguments the launcher passes on a host without NVIDIA. Within 5 minutes: node Ready, kube-system pods up, every baked manifest present and every DaemonSet they define.

How it runs

• Triggers. It runs nightly at 03:29, on pull requests touching the pin paths, and on dispatch (engines: ids or auto; attempt).
• Selection. canary/engines.toml locates each pin in the tree and never copies it. Each locator must match exactly one ref, in either the name@sha256 or the name:tag@sha256 form. In auto mode, canary/plan.py selects an engine only when, against the merge base, any of these changed: its resolved ref, its table, its golden, a derived constant, or a flag that flags.sh names for it. A Dockerfile, share/ or k3s change selects the node. A change to the harness selects everything.
• Report. The report job writes canary-result.json (artifact canary-result) with exactly the interface schema. It is green only when every selected non-informational engine has verdict pass with hard true.
• Permissions and secrets. Contents are read-only at workflow level, there are no secrets and no Docker Hub login, and only hosted runners are used. One job has actions: write: it dispatches the workflow again (attempt + 1, at most 3) when an engine is isa-skip.
• flags.sh coverage. flags.sh attributes each flag literal in model_opt.py and fit.py to its engine through the engine == "X" tests around it. A literal that no rule attributes fails flags.sh, so a new emitter cannot go unchecked.

Fixes from the first hosted run

The first Canary run (40960775877, on 0163539) failed three engines. None of them was a real break; each was a gap in the canary.

Engine	What it reported	Cause and fix
node	canary/node.sh: Permission denied	The script was committed without its executable bit. It is now executable, and tests/canary_spec_test.sh checks every canary/ script the workflow runs.
node (second run)	the NVIDIA device plugin pod in CrashLoopBackOff	A plain k3d cluster create on a runner without NVIDIA starts the baked plugin under the nvidia runtime with no driver. The launcher never does that: on a host without NVIDIA it labels the node nvidia.com/gpu.present=false, which the plugin's affinity excludes, and it always passes --no-lb and disables traefik. node.sh now passes the same three arguments (NODE_ARGS). The spec test checks that scripts/k3d-gpu still passes each one word for word. Both Intel stacks stay enabled, so every baked manifest is still applied.
sglang-xeon	--tp not in the image	bench/h54 passes --tp 1. sglang's parser is a plain argparse.ArgumentParser, which allows abbreviations, so --tp is --tp-size, the only option it begins. For sglang, a long flag now also matches when it begins exactly one option in the help, and the detail names the expansion. vLLM flags still match exactly, so a renamed flag cannot hide behind a longer one.
llm-scaler	console/plan.py says 0.26.0, the image's package is 0.26.1.dev0+g568afb3a1.d20260907.xpu	Commit 568afb3a1 is vLLM's v0.26.0 tag. setuptools-scm names a tree at a tag with uncommitted changes X.Y.(Z+1).dev0+g<commit>.d<date>, so the image is v0.26.0 with Intel's patches, and the derived 0.26.0 is right. release() now reads that form as the tag. A build made commits after a tag (.dev1 and up) still mismatches.

The vLLM CPU goldens (exact was null) are now recorded from that run. run.sh also logs the runner's CPU model and count, so a golden can name the CPU it was recorded on.

Spike (this PR's own canary runs)

Run 40980204951 on 5be20ca is all green. It took 14 min 45 s from the first job start (00:31:22 UTC) to the end of the report (00:46:07). The critical path was vllm-cpu on an AVX2-only runner. Runner queueing comes on top of that when the PR's CI runs at the same time.

Runtime

Engine	Image	Runner CPU	Ready	Job	Result
vllm-cpu	vllm/vllm-openai-cpu:v0.32.0	EPYC 7763, avx2 only	329 s	536 s	pass, exact
vllm-cpu (run 40979011371)	same	EPYC 9V74, avx512f	127 s	214 s	pass, exact
eat-vllm-cpu	vllm/vllm-openai-cpu:v0.24.0	EPYC 9V74, avx512f	109 s	204 s	pass, exact
ovms-cpu	openvino/model_server:2026.4.1-gpu, CPU plugin, int8	EPYC 9V74	4.1 s	39 s	pass, exact
llama-cpu	ghcr.io/ggml-org/llama.cpp:server-b11960, Q8_0	EPYC 7763	2.0 s	46 s	pass, exact

• vLLM CPU serves on hosted runners, so it is not isa-skip. The settings were:
  • --dtype=bfloat16, --max-model-len=2048 and --max-num-seqs=1;
  • VLLM_CPU_KVCACHE_SPACE=1 and a 2 GiB shm, which gave a KV cache of 9,344 tokens.
• vLLM uses _C_AVX2 on an AVX2-only runner. There its engine init (profile, KV cache, warmup) took 290 s.
• The vLLM golden holds on both ISAs. The text ' Paris. The capital of France is also' came back on the AVX2 runner and on the AVX-512 runner.
• OVMS and llama.cpp match the goldens recorded on a workstation. Their text was ' Paris. The capital of Italy is Rome'.

Node. The image built in 37 s, and the cluster was created in 10 s. It was ready 50 s later, with 11 baked manifests, 11 DaemonSets and every kube-system pod up. The job took 94 s.

Surface. Each row gives the image size against free disk.

Engine	Compressed	Estimated on disk	Job	Method
llm-scaler	7.78 GB	27.2 GB	257 s	static: vllm serve --help=all needs a device
vllm-xpu	5.18 GB	18.1 GB	166 s	static, as above
llama-sycl	4.34 GB	15.2 GB	93 s	help
gaudi-vllm	2.61 GB	9.1 GB	85 s	help
sglang-xeon	2.01 GB	7.0 GB	74 s	help; --tp is --tp-size
llama-full	1.14 GB	4.0 GB	57 s	convert_hf_to_gguf.py --help, llama-quantize --help
dlstreamer	0.86 GB	3.0 GB	47 s	gst-inspect-1.0 per element
npu-umd	0.40 GB	1.4 GB	19 s	help, deb version, firmware date
ovms	0.40 GB	1.4 GB	20 s	help
llama-vulkan	0.31 GB	1.1 GB	26 s	help

• Disk. Every runner had about 93.9 GB free on docker's root, not the 14 GB the spec assumed. So no image was too-big, and the toolchain cleanup in run.sh never ran. The too-big path stays, covered by tests/canary_test.sh.
• Runner CPUs seen. They were AMD EPYC 7763 (avx2), EPYC 9V74 and 9V45 (some with avx512f), and Xeon Platinum 8573C (avx512f, once amx_tile).

Mutation checks

• Throwaway pull request throwaway: canary mutation check (not for merge) #263, now closed and its branch deleted. It pointed llama-cpu's pin at server-b99999@sha256:0…01 and targeted this branch, so only Canary ran.
  • Run 40979086697 selected only llama-cpu and went red: pull failed after 3 attempts: ... not found. The report failed.
• Fixture mutation. Renaming --served-model-name in the vLLM fixture turns tests/canary_surface_test.py red (6 of 35 fail; rerun on 5be20ca).
• The new tests against the old surface.py. tests/canary_test.sh fails exactly the two hosted false reds: sglang --tp and llm-scaler 0.26.0.

Tests

Suite	Result
tests/canary_test.sh	61 passed
tests/canary_spec_test.sh	108 passed
tests/canary_surface_test.py	35 passed

The three suites are wired into ci.yml (job canary-tests).

• canary_test.sh runs canary/run.sh against tests/openai_stub.py and tests/fake_ovms.py behind a docker shim. It covers:

  • a good server, empty content, a stream without [DONE], a version mismatch, and a golden mismatch (pass, hard true, exact false);
  • a server that never becomes ready (fails in about 3 s), a probe that writes no result, the OVMS pull step, and a vLLM surface pass, missing flag and unparseable tree;
  • SIGILL without and with avx512f, too-big, pull retries, an informational failure, and model fetch then cache;
  • run.sh --surface for sglang with the image's real help and _version.py: a pass through the --tp abbreviation, and a fail once a second option begins with --tp;
  • llm-scaler at 0.26.1.dev0+g568afb3a1.d20260907.xpu, which passes, and at .dev4, which fails naming console/plan.py.

• canary_spec_test.sh covers:

  • every locator on the real tree, both ref forms, and a duplicate refused;
  • 40-hex revisions, goldens, and non-empty flags per engine;
  • derived constants, and a flags.sh failure on an unattributed literal;
  • auto planning against fixture diffs and against a git base;
  • the report schema and exit codes, and the workflow's shape;
  • every script the workflow runs is executable, and node.sh's NODE_ARGS are ones the launcher passes.

• canary_surface_test.py runs the parser on real inputs:

  • vLLM v0.32.0's real arg_utils.py, envs.py and launchers/cli_args.py;
  • the pinned images' real help output, sglang v0.5.22-xeon's now included.

  A present flag passes, a removed flag fails, VLLM_USE_V1 is absent and fails, and broken trees are unparseable. For argparse abbreviations, a unique prefix passes; an ambiguous one, a prefix of no option and a new option that makes it ambiguous all fail, and vLLM takes exact names only. The setuptools-scm cases are a patched tag build, a build after a tag, and no scm node.

• Other suites, all green.

  • tests/claims_check.sh, tests/ledger_drift_check.py and tests/ci_wiring_test.sh (72 passed).
  • From the first round: scripts/images-lock check and pins, tests/image_pins_test.sh and tests/packaging_test.sh.
  • Also from the first round: tests/bench_test.sh, tests/bench_vs_test.py, bench/admission/test_run.sh, and tests/tune_test.sh with TUNE_E2E_FULL=1.
  • CI on 5be20ca is green.

• Lint. actionlint 1.7.12 (hack/ci-images.env) and shellcheck v0.11.0 are clean.

Facts verified at build time

• Actions. Each is the latest release, resolved with gh api on 2026-10-29:

  Action	Version	Commit
  actions/checkout	v7.0.1	3d3c42e5
  actions/cache	v6.1.0	55cc8345
  actions/upload-artifact	v7.0.2	cf430e03
  actions/download-artifact	v8.0.2	9000827c

• Models. Each pin is the HF HEAD, checked again on 2026-10-29:

  Model	Revision
  Qwen/Qwen3-0.6B	c1899de289a04d12100db370d81485cdf75e47ca
  OpenVINO/Qwen3-0.6B-int8-ov	ac8e917fb8350f528ada41b02c60ea3a2853955f
  ggml-org/Qwen3-0.6B-GGUF	13ac06624948e4d8589cb4b6f90100837342fcb8 (Q8_0)

• k3d v5.9.0. k3d cluster create --help lists --wait (default true) and --timeout, and the binary's sha256 matches packaging/install.sh.

• vLLM v0.32.0 (commit 5df63a72).

  • It moved the frontend flags to vllm/entrypoints/launchers/cli_args.py, so the parser globs entrypoints/**/cli_args.py.
  • On a CPU without AVX-512, its CPU platform imports vllm._C_AVX2 (vllm/platforms/cpu.py).

• vLLM v0.26.0 is commit 568afb3a13806beb53bb2e6bd518269357b237c0.

• setuptools-scm. In vcs_versioning/_version_schemes/_standard.py, guess_next_dev_version returns the tag only when the build is exact, meaning distance 0 and not dirty. Otherwise it returns the next version with .dev<distance>.

• sglang v0.5.22-xeon.

  • prepare_server_args builds argparse.ArgumentParser(prog="sglang serve"), which allows abbreviations. The image runs Python 3.12.15.
  • In the image, --tp 1 parses to tp_size 1, and --t 1 gives "ambiguous option".
  • The sglang docs at that tag write --tp 2.
  • The version comes from sglang/_version.py (0.5.22), because the distribution is sglang-cpu.

• OVMS 2026.4.1.

  • It serves GET /v3/models, GET /v2 (version 2026.4.1.cfc9ac721) and /v3/completions with top_k and seed.
  • Its --pull keeps an existing non-git model directory ("Skipping download and using existing files"). The run therefore sets HF_ENDPOINT to an unreachable address.

• NPU user space. The image's intel-level-zero-npu is 1.35.0.20260722-29947505341~ubuntu24.04, and the firmware vpu_37xx_v1.bin is built 20260630. Both equal NPU_UMD_DRIVER and NPU_UMD_FW_DATE.

Known limits

• vLLM XPU and llm-scaler are checked statically. vllm serve --help=all raises "Failed to infer device type" without a device, so the check reads their flags from the installed sources. Their env names always come from envs.py.
• An abbreviation is judged against the help output. An option hidden from help (argparse.SUPPRESS) could make --tp ambiguous to argparse but not to the canary.

Owner actions

None are needed for this part. Until canary.yml is on main, two things do not work, because GitHub requires workflow_dispatch workflows to be on the default branch:

• the isa-skip re-dispatch;
• engines= dispatch.

Pull-request and scheduled runs are unaffected.