ci(canary): runtime and surface canary for engine and node images

.github/workflows/canary.yml

@@ -0,0 +1,237 @@
+name: Canary
+
+# Every engine or node-image bump, from a bot or a human, checked at runtime before it
+# merges (canary/README.md):
+#   runtime  vllm-cpu, ovms-cpu, llama-cpu (and eat-vllm-cpu, informational): pull the
+#            pinned image, serve a pinned 0.6B model the way the launcher would, one real
+#            and one streamed completion, the version the server reports;
+#   surface  every engine image: the flags and env names the launcher emits exist in it,
+#            and the constants k3d-gpu derives from it still match (no device needed);
+#   node     build the node image, start a k3d cluster on it, everything baked comes up.
+# A pull request or `engines=auto` runs the engines whose pin, flags, settings or golden
+# differ from merge-base(origin/main, HEAD), or from the pull request's base branch
+# (canary/plan.py matrix); the schedule runs all.
+# The report job writes canary-result.json (artifact canary-result) and is green only when
+# every selected engine's verdict is pass with hard true.
+#
+# Read-only: no secrets, no registry login (public images), GitHub-hosted runners only.
+# The re-dispatch job alone may write: it starts this workflow again (attempt + 1, at most
+# 3) when an engine was isa-skip, the runner's CPU lacking an instruction set it needs.
+
+on:
+  schedule:
+    - cron: '29 3 * * *'
+  workflow_dispatch:
+    inputs:
+      engines:
+        description: 'engine ids from canary/engines.toml, comma separated, or auto (the engines that differ from main)'
+        type: string
+        default: auto
+      attempt:
+        description: 'attempt number: an isa-skip re-dispatches with attempt + 1, at most 3'
+        type: number
+        default: 1
+  pull_request:
+    paths:
+      - Dockerfile
+      - scripts/k3d-gpu
+      - 'share/**'
+      - console/moe_predict.py
+      - console/plan.py
+      - console/model_opt.py
+      - console/fit.py
+      - bench/h54/h54.py # sglang_args, the sglang flags the surface check reads
+      - 'canary/**'
+      - packaging/install.sh
+      - integrations/eat/vllm-values.yaml
+      - .github/workflows/canary.yml
+
+permissions:
+  contents: read
+
+concurrency:
+  group: canary-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  plan:
+    name: Plan
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    outputs:
+      runtime: ${{ steps.plan.outputs.runtime }}
+      surface: ${{ steps.plan.outputs.surface }}
+      node: ${{ steps.plan.outputs.node }}
+      selected: ${{ steps.plan.outputs.selected }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0 # merge-base(origin/main, HEAD)
+          persist-credentials: false
+      # auto: against merge-base(origin/<the pull request's base, else main>, HEAD)
+      - id: plan
+        env:
+          EVENT: ${{ github.event_name }}
+          ENGINES: ${{ inputs.engines }}
+          BASE: ${{ github.base_ref || 'main' }}
+        run: |
+          mode=auto
+          [ "$EVENT" != schedule ] || mode=all
+          python3 canary/plan.py matrix --mode "$mode" --engines "${ENGINES:-auto}" --base "origin/$BASE" \
+            --github-output "$GITHUB_OUTPUT"
+
+  runtime:
+    name: Runtime (${{ matrix.id }})
+    needs: plan
+    if: needs.plan.outputs.runtime != '[]'
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.plan.outputs.runtime) }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      # The model at its pinned revision; canary/run.sh downloads it on a miss.
+      - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ${{ matrix.cache_path }}
+          key: ${{ matrix.cache_key }}
+      - name: canary/run.sh
+        env:
+          ID: ${{ matrix.id }}
+          CANARY_FREE_DISK: '1'
+          CANARY_OUT_DIR: ${{ runner.temp }}/canary-out
+        run: canary/run.sh "$ID"
+      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        if: always()
+        with:
+          name: canary-part-${{ matrix.id }}
+          path: ${{ runner.temp }}/canary-out/
+          if-no-files-found: ignore
+          retention-days: 14
+
+  surface:
+    name: Surface (${{ matrix.id }})
+    needs: plan
+    if: needs.plan.outputs.surface != '[]'
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      max-parallel: 4
+      matrix:
+        include: ${{ fromJSON(needs.plan.outputs.surface) }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - name: canary/run.sh --surface
+        env:
+          ID: ${{ matrix.id }}
+          CANARY_FREE_DISK: '1'
+          CANARY_OUT_DIR: ${{ runner.temp }}/canary-out
+        run: canary/run.sh --surface "$ID"
+      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        if: always()
+        with:
+          name: canary-part-${{ matrix.id }}
+          path: ${{ runner.temp }}/canary-out/
+          if-no-files-found: ignore
+          retention-days: 14
+
+  node:
+    name: Node image
+    needs: plan
+    if: needs.plan.outputs.node == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - name: canary/node.sh
+        env:
+          CANARY_OUT_DIR: ${{ runner.temp }}/canary-out
+        run: canary/node.sh
+      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        if: always()
+        with:
+          name: canary-part-node
+          path: ${{ runner.temp }}/canary-out/
+          if-no-files-found: ignore
+          retention-days: 14
+
+  report:
+    name: Report
+    needs: [plan, runtime, surface, node]
+    if: always() && needs.plan.result == 'success'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    outputs:
+      isa_skip: ${{ steps.report.outputs.isa_skip }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
+        with:
+          pattern: canary-part-*
+          merge-multiple: true
+          path: ${{ runner.temp }}/parts
+      - id: report
+        env:
+          SELECTED: ${{ needs.plan.outputs.selected }}
+          SHA: ${{ github.event.pull_request.head.sha || github.sha }}
+          REF: ${{ github.head_ref || github.ref_name }}
+        run: |
+          mkdir -p "$RUNNER_TEMP/parts"
+          rc=0
+          python3 canary/plan.py report --selected "$SELECTED" --parts "$RUNNER_TEMP/parts" --sha "$SHA" --ref "$REF" \
+            --out canary-result.json --summary "$GITHUB_STEP_SUMMARY" --github-output "$GITHUB_OUTPUT" || rc=$?
+          echo "rc=$rc" >> "$GITHUB_OUTPUT"
+      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        with:
+          name: canary-result
+          path: canary-result.json
+          retention-days: 30
+      - name: Green only when every selected engine passed
+        env:
+          RC: ${{ steps.report.outputs.rc }}
+        run: exit "${RC:-1}"
+
+  redispatch:
+    name: Re-dispatch isa-skip engines
+    needs: report
+    if: >-
+      always() && needs.report.outputs.isa_skip != '' &&
+      (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      actions: write
+    steps:
+      - env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          REF: ${{ github.head_ref || github.ref_name }}
+          ENGINES: ${{ needs.report.outputs.isa_skip }}
+          ATTEMPT: ${{ inputs.attempt || 1 }}
+        run: |
+          if [ "$ATTEMPT" -ge 3 ]; then
+            echo "::warning::$ENGINES isa-skip on attempt $ATTEMPT of 3: no runner with the instruction set it needs; the canary stays not green"
+            exit 0
+          fi
+          gh workflow run canary.yml --repo "$REPO" --ref "$REF" -f engines="$ENGINES" -f attempt="$((ATTEMPT + 1))"

.github/workflows/ci.yml

@@ -483,3 +483,22 @@ jobs:
       - run: python3 tests/known_issues_test.py
       - run: python3 tests/ledger_freshness_test.py
       - run: hack/known-issues check
+
+  # Part canary (.github/workflows/canary.yml runs the engines themselves): canary/run.sh
+  # against stub engines and a docker shim, canary/node.sh against scripted pod states, the
+  # surface parser on vLLM v0.32.0's real sources and the pinned images' help output,
+  # flags.sh, the plan and the report. No image is pulled.
+  canary-tests:
+    name: Canary harness (stub engines, surface parser, plan, report)
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - run: tests/canary_test.sh
+      - run: tests/canary_node_test.sh
+      - run: tests/canary_spec_test.sh
+      - run: python3 tests/canary_surface_test.py

canary/README.md

@@ -0,0 +1,98 @@
+# Canary
+
+Every engine or node-image bump, whether a bot or a person opened it, is checked at
+runtime before it merges. Engines move fast: a vLLM bug today is fixed tomorrow, and a
+flag the launcher passes today is renamed tomorrow. Before this canary, no CI job built the
+node image, started a cluster, or pulled and served an engine. Every engine path in the
+tests was a stub.
+
+`.github/workflows/canary.yml` (name `Canary`) runs at 03:29 UTC every night, on pull
+requests that touch a pin or the launcher, and on dispatch (`engines`: ids or `auto`;
+`attempt`). It runs only on GitHub-hosted runners. It has read-only permissions and no
+secrets.
+
+| Kind | Engines | What it proves |
+|---|---|---|
+| runtime | `vllm-cpu`, `ovms-cpu`, `llama-cpu`; `eat-vllm-cpu` (informational) | The pinned image serves a pinned Qwen3-0.6B the way the launcher runs it. It returns one completion (HTTP 200, `finish_reason` stop or length, tokens > 0, the text matches `/Paris/`) and one stream (at least 2 chunks, then `[DONE]`), and the server reports the version its tag names. |
+| surface | `vllm-xpu`, `llm-scaler`, `gaudi-vllm`, `llama-sycl`, `llama-vulkan`, `llama-full`, `dlstreamer`, `sglang-xeon`, `npu-umd`, `ovms` (and the runtime engines first) | Every flag and env name the launcher and `console/model_opt.py` emit (`canary/flags.sh`) exists in the image, and the constants derived from it still match: `console/plan.py`'s vLLM versions, and the NPU driver and firmware date. No device is needed. |
+| node | `node` | The root Dockerfile builds. `k3d cluster create --image` it with the k3d `packaging/install.sh` pins and the arguments the launcher passes on a host without NVIDIA (no load balancer, no traefik, the node labelled `nvidia.com/gpu.present=false`). Within 5 minutes the node is Ready, every baked manifest is in place, every DaemonSet they define exists, and every pod in every namespace (kube-system, NFD, the device plugins) is Running and ready or Succeeded; then the pods hold for 20 s with no container restarting. A plugin that needs hardware gets no pod on a hosted runner (its node selector), so a pod still Pending at the timeout, such as one whose image does not pull, fails. |
+
+## Files
+
+| File | Role |
+|---|---|
+| `engines.toml` | One table per engine. Its `locator` points at the pin in the tree (never a copy of it), plus the model and revision, the launcher-equivalent arguments, the golden, the version probe, the flag sources and the derived constants. |
+| `plan.py` | Resolves the pins, plans the job matrix and merges the report. |
+| `run.sh` | Runtime and surface: disk, CPU flags, model, size, pull, surface, serve, probe, ISA. |
+| `probe.py` | The HTTP checks against a serving engine. |
+| `surface.py` | The flag, env and derived-constant check. It reads the engine's own `--help`, or for vLLM statically parses `arg_utils.py`, `entrypoints/**/cli_args.py` and `envs.py` inside the image, without importing them. |
+| `flags.sh` | Lists the names k3d-gpu emits, per engine family. |
+| `node.sh` | The node canary. |
+| `goldens/<id>.json` | `{prompt, regex, exact, recorded_with}` for each runtime engine. |
+
+## How names and versions compare
+
+- A flag matches exactly; for vLLM, `--a_b` and `--a-b` are one flag.
+- sglang's parser is argparse with abbreviations allowed, and `bench/h54` passes `--tp`.
+  So for sglang, a long flag also matches when it begins exactly one option (`--tp` is
+  `--tp-size`). Once a second option begins with it, argparse calls it ambiguous, and so
+  does the canary. vLLM flags never match by prefix, so a renamed flag cannot hide behind
+  a longer one.
+- A derived constant matches the image's version by release (`0.32.0` = `0.32.0+cpu`).
+  A setuptools-scm build of a tag with patches applied, `X.Y.(Z+1).dev0+g<commit>.d<date>`,
+  is release `X.Y.Z`: intel/llm-scaler-vllm 1.5 reports `0.26.1.dev0+g568afb3a1.d20260907.xpu`,
+  which is vLLM v0.26.0's commit with Intel's patches. A build commits after a tag
+  (`.dev1` and up) matches nothing.
+
+## Which engines run
+
+`schedule` runs every engine.
+
+A pull request or `engines=auto` runs the engines that differ from
+`merge-base(origin/main, HEAD)` in any of these:
+
+- the resolved image reference;
+- their `engines.toml` table;
+- their golden;
+- a derived constant;
+- a flag `flags.sh` names for them.
+
+The node engine also runs when the Dockerfile, `share/` or `packaging/install.sh` change. A
+change to the harness itself runs every engine.
+
+`engines=a,b` runs exactly those engines.
+
+## Verdicts (`canary-result` artifact, `canary-result.json`)
+
+```json
+{"sha": "...", "ref": "...", "engines": {"<id>": {"kind": "runtime|surface|node",
+  "pinned_ref": "...", "verdict": "pass|fail|isa-skip|too-big|unparseable",
+  "hard": true, "exact": true, "cpu_flags": ["avx2"], "detail": "..."}}}
+```
+
+Only `verdict` pass with `hard` true is green. The report job fails unless every selected
+non-informational engine is green.
+
+| Value | Meaning |
+|---|---|
+| `exact` false | The text differs from the golden, but the hard checks passed. It is still green, but it is not evidence for an automatic merge. |
+| `isa-skip` | The runner's CPU lacks an instruction set the engine needs: no `avx512f`, and the engine's `isa_signal` in its log, or SIGILL. The workflow dispatches itself again, up to `attempt` 3. Hosted runners vary in CPU, so a later attempt may land on one with AVX-512. |
+| `too-big` | The image needs more disk than the runner has, even after the toolchains are deleted. |
+| `unparseable` | A help output or a source the surface check could not read. It is red, never skipped. |
+
+## Recording a golden
+
+`exact` is the completion text the hosted runner produced with temperature 0, top_k 1 and
+seed 0. Take it from the `detail` (`text '...'`) of a green run, and write it with the
+image, model and CPU it was recorded on (the job log's `cpu:` line names the CPU). A `null`
+`exact` means nothing is recorded yet. The vLLM goldens come from an AVX2-only runner; a
+runner with AVX-512 runs other kernels and may produce other text. That clears `exact`,
+which sends the bump to review, but it is never red.
+
+## Adding an engine
+
+1. Add an `[[engine]]` with a `locator` that matches exactly one reference.
+   `tests/canary_spec_test.sh` checks it against the tree.
+2. If the launcher emits flags for a new family, teach `flags.sh` its source. A flag
+   literal in `console/model_opt.py` or `console/fit.py` that no rule attributes to an
+   engine fails `flags.sh`.