Repository navigation
batch: land #276, #270, #275, #252, #253, #258, #265, #268, #254, #273 - #277
Merged
Merged
Conversation
…it and nft_log for H10 The box owner approved loading and persisting the fourteen netfilter modules that H10 named after smoke3. They were loaded on wildct at 21:16:54Z with three hard dependencies and appended to /etc/modules-load.d/k3dgpu-wildct.conf, with kernel.modprobe left empty. The host's services, its published endpoints, Prometheus's targets and the firewall's rule counts and hashes read the same before, at once after and three minutes after the load. Step 1 then ran from main at e5748cd. H1-H9 pass; H10 reads "NFPROBE missing: NFLOG limit". iptables-nft writes -m limit and -j NFLOG as nftables' native limit and log expressions, which need nft_limit and nft_log; neither is loaded, and a probe in a throwaway network namespace with the host's nft fails the same way. smoke4 did not run. Loading those two modules, or restoring kernel.modprobe, is the box owner's call. Amendment R4 records this; RESUME.md names nft_limit and nft_log in H10's fail column.
No CI job built the node image, started a cluster, or pulled and served an engine: every engine path in the tests was a stub, so an engine bump (bot or human) merged without ever running. The canary checks each bump at runtime before it merges. canary.yml (name Canary) runs nightly at 03:29, on pull requests that touch a pin, the launcher or the canary, and on dispatch (engines: ids or auto; attempt). It is read-only, uses no secrets and runs only on GitHub-hosted runners; one job may write, to dispatch the workflow again after an isa-skip. - runtime (vllm-cpu, ovms-cpu, llama-cpu; eat-vllm-cpu informational): pull the pinned image, serve a pinned Qwen3-0.6B the way the launcher runs it, then one completion (200, finish_reason stop or length, tokens, /Paris/), one stream (2+ chunks, then [DONE]) and the version the server reports against the pinned tag. A golden mismatch keeps hard and clears exact. - surface (every engine image): each flag and VLLM_* name the launcher and console/model_opt.py emit (canary/flags.sh) exists in the image, read from its own --help or, for vLLM, statically from arg_utils.py, cli_args.py and envs.py inside it; and the constants derived from the image (plan.py's vLLM versions, the NPU driver and firmware date) still match. - node: build the Dockerfile, k3d cluster create on it with the k3d release packaging/install.sh pins, and see the node Ready, kube-system up, every baked manifest and every DaemonSet they define. canary/engines.toml points at each pin with a locator (exactly one match, name@sha256 or name:tag@sha256), never a copy of it. In auto mode canary/plan.py runs only the engines whose pin, flags, settings, golden or derived constants changed against the merge base; the report job writes the canary-result artifact (canary-result.json) and is green only when every selected engine passes with hard true. Tests: tests/canary_test.sh (run.sh against tests/openai_stub.py and tests/fake_ovms.py behind a docker shim), tests/canary_spec_test.sh (the locators on the real tree, flags.sh, planning against fixture diffs and a git base, the report, the workflow's shape) and tests/canary_surface_test.py (the parser on vLLM v0.32.0's real sources and the pinned images' help output), wired into ci.yml.
run.sh read a missing probe.json as no failures, so a probe.py that crashed before writing its result passed. Once the engine is started, a pass now needs the probe's result; tests/canary_test.sh covers a probe that cannot parse its timeout.
…pen alarms A PR that GITHUB_TOKEN opens or pushes gets pull_request runs held for approval (#240 sat with CI action_required), while a workflow_dispatch run needs none. ci.yml now takes workflow_dispatch, and hack/bot-ci-dispatch runs it on each same-repo autobump/* branch opened by github-actions[bot] that has no run at its head, re-running a first-attempt failure once. It keeps one bot ci.yml run in flight (BOT_CI_MAX_INFLIGHT) and defers while a human PR's CI is queued, since one run is about 39 jobs against a 20-40 job pool. The three nightly PR jobs call it after create-pull-request, and the new hourly 'Bot sweep' workflow (job ci-fill) fills in what they deferred. hack/autobump-branch-reset generalises the digest job's stale-branch step to all three PR jobs: a bump branch whose workflow files differ from main's is deleted before create-pull-request, whose force-push GitHub would refuse. The merge gate's checks move to hack/gate_lib.sh. ci_status reads ci.yml at a SHA through a pull_request run or a dispatch run on the PR branch; once the dispatched run is green, held entries of 'CI' and 'Auto-merge Dependabot PRs' no longer block. base_ok ignores release-only bot commits on main, so the three release commits per launcher release no longer force a sync. The gate prints every open ci-alarm issue and refuses (exit 4) a PR that changes .github/ or shipped paths while one is not acked; ledger, docs, bench and test PRs are never refused. ci.yml gains a last job, alarms, which warns on every open ci-alarm issue and on a missing, disabled or quiet Watchdog workflow, and never fails. dependabot-auto-merge.yml is deleted: it ran gh pr merge --auto on an unprotected main, merged nothing in 500 runs, and held a run on every bot PR.
…top on an unreadable run pool
…g builds match The first hosted canary run failed three engines, none of them a real break. canary/node.sh was committed without its executable bit, so the node job stopped with "Permission denied". It is now executable, and tests/canary_spec_test.sh checks every canary/ script the workflow runs. sglang-xeon reported --tp missing. bench/h54 passes --tp 1, and sglang's parser is a plain argparse.ArgumentParser that allows abbreviations: in the v0.5.22-xeon image --tp parses to tp_size 1, as sglang's own docs write it. For sglang, a long flag now also matches when it begins exactly one option in the help, and the detail names the expansion. vLLM flags still match exactly. llm-scaler reported vLLM 0.26.1.dev0+g568afb3a1.d20260907.xpu against the tree's 0.26.0. Commit 568afb3a1 is vLLM's v0.26.0 tag, and setuptools-scm names a tag built with uncommitted changes X.Y.(Z+1).dev0+g<commit>.d<date>, so the image is v0.26.0 with Intel's patches and the derived constant is right. release() now reads that form as the tag. A build commits after a tag still mismatches. The vLLM CPU goldens are recorded from the same run, and run.sh logs the runner's CPU model and count so a golden can name it. The sglang help output and version files are new fixtures, taken from the pinned image.
…thout NVIDIA The node canary built the image and started a cluster, but failed on the baked NVIDIA device plugin: on a runner without NVIDIA its pod starts under the nvidia runtime with no driver and crash-loops. The launcher never lets that happen. On a host without NVIDIA its cluster create labels the node nvidia.com/gpu.present=false, which the plugin's affinity excludes, and it always passes --no-lb and disables traefik. node.sh now passes those three arguments (NODE_ARGS). Both Intel stacks stay enabled so every baked manifest is still applied and checked. tests/canary_spec_test.sh checks that scripts/k3d-gpu still passes each argument word for word, so the canary cannot drift from the launcher.
…d renders in the hold's cluster) The H59, H59d and H59w pre-registration requires a runner amendment with tests merged before the first hold. This adds it. bench/h59/run.py holds the arm table, the hold orders, and host profiles for crescent (Xeon 6741P), the desktop (i9-13900T) and wildct (i7-13700). It renders each arm with `k3d-gpu agent manifest --model` in the hold's cluster (K3D_GPU_CLUSTER set, revisions pinned, inherited agent variables dropped), stages it against the campaign's model directory, and refuses an arm whose render is not the registered one: images, model paths, flags, the P-core balloon. It also writes crescent's load verdict under H54's void rule and runs the prefix-reuse phase. hold.sh, hold-run.sh and arm.sh run the registered holds: performance holds 1-3 at N=1, N=4 (and N=16 on crescent), TTFT and prefix reuse, the quality hold, and the spares. Their records are in the layout bench/h59/h59.py scores. prep.sh downloads the pinned model files and builds the drafter with the render's own conversion Job. bench/wildct/row.sh gains h59w and h59w-prep. bench/h57d/arm.sh rendered Od without K3D_GPU_CLUSTER. Since #183 that renders through context k3d-gpu, which a hold's kubeconfig does not hold, so Od lost its P-core label and got 1 CPU (H57dp's deviation). It now passes the hold's cluster. H57d amendment R2 records the fix for the H57w rows, which have not run. EXPERIMENTS.md: H59 amendment R1 (mechanics only: the runner, the flags read at each engine's pin, staging, holds, host profiles, the engine rule as merged), and a report for the owner on what the desktop load gate reads. On the H57d and H57dp records, the foreign P-core load over the N=4 phases moves with the arm, not the host, while the arms' idle windows match. No gate is changed. H57d amendment R2. Links from the H59, H59d, H59w and H57wa rows; no prediction changes. Tests: bench/h59/test_run.py renders both arms through the real launcher against the fit fixtures, with a kubectl stand-in that answers only for the hold's context. It runs arm.sh against a local stub server and scores four desktop holds with h59.py end to end (SUPPORTED on a 2x-faster L). It also covers crescent's N=16 phase and void re-run, the gate, the prefix phase, hold.sh's guards and the shim. bench/h57d/test_h57d.py renders Od through arm.sh and reads the P-core label and 16 CPUs; it failed before the fix. tests/wildct_hold_test.sh covers the h59w rows. The new suite is wired into CI.
…uses its controller Intel's Enterprise Agent Toolkit installs agent-sandbox v0.5.0 into agent-sandbox-system, the namespace k3d-gpu's v1.0.8 manifest also uses. Before this change `k3d-gpu sandbox up` applied its manifest over that controller and `sandbox down` deleted the namespace and the CRDs, which deletes every Sandbox, SandboxClaim, SandboxTemplate and SandboxWarmPool in the cluster. sandbox up now reads the controller Deployment and the four CRDs first. The controller is k3d-gpu's when it carries app.kubernetes.io/managed-by=k3d-gpu, or runs the image pinned in share/agent-sandbox.yml (or the v1.0.7 image earlier releases installed unlabelled) with no other managed-by value or Helm release. After an install, up labels the Deployment, the namespace and the CRDs with that key. Another installer's controller, or a CRD with another installer's label or Helm release, stops up before anything is applied, and the error names the image or the CRDs. sandbox up --adopt keeps the existing controller: it checks that the SandboxTemplate and SandboxWarmPool CRDs serve extensions.agents.x-k8s.io/v1beta1, skips the manifest and the label, waits for the CRDs and the rollout, then applies the E-core profile (the --remote profile too). With no controller present it is an error. --adopt goes with up only. sandbox down still deletes the profile first. It deletes the namespace, the CRDs and the controller only when the controller is k3d-gpu's, no CRD carries another installer's marks, no sandbox object exists outside namespace agent-sandbox, and no pod other than the controller's runs in agent-sandbox-system. Otherwise it prints what it kept and why, and exits 0.
The nightly build was red for nine nights with nobody told, and release 0.1.76 left a tag with no image after a Docker Hub 429. This adds a watchdog that files an issue per finding and closes it once the finding clears, and makes a release unable to cancel another or to stop half way without a retry. hack/ci_alarm keeps one issue per alarm key (label ci-alarm, assigned to the owner with an @mention); a changed body comments, an unchanged one comments again only after 24 h. When the repository has issues disabled it prints a ::error annotation and the step summary and exits 3. hack/watchdog (.github/workflows/watchdog.yml: hourly at :47 and after CI, Docs and every release workflow completes on main) raises stale:<file> and red:<file> for scheduled workflows (failed jobs, the failed log up to its last error, a signature hint, one re-dispatch per failed run), disabled:<file>, ci-missing and ci-main for main (dispatching ci.yml once when HEAD has no run), release:<ver> when the newest tag older than 45 min lacks its Docker Hub or GHCR image, its .deb, .rpm and SHA256SUMS, or its AUR update (one release re-dispatch when the Hub image is missing), bot-pr-stuck for autobump PRs over 72 h without green CI, and simulate on demand. Its own errors become watchdog-error. The tests replay the October red nights and the 0.1.76 tag from the real run list, jobs and failed log. build-and-push.yml queues each release version on its own, builds each variant in up to three attempts, and pushes only the versioned tags; the floating tags move afterwards, from hack/release-tags, only for the highest aur-v* tag, and aur-deploy.yml never moves AUR back. hack/release-paths is the one list of shipped paths (now including console/, images.lock and the shipped bench and oracle files): auto-release.yml repeats it as its push paths and release_cas.sh reads it, and a mode=changed dispatch releases only when one of them changed. Every job has a timeout, every workflow a permissions block, and no cron fires at minute 0 or 30.
The watchdog re-dispatched a red scheduled workflow, or a release with no Docker Hub image, and only then wrote the record that stops the next re-dispatch into the alarm. With Issues disabled (this repository today) the record can never be written and every watchdog run exits 1, so each run re-dispatched build-and-push.yml, the missing release and watchdog.yml itself, and each dispatched watchdog run did the same, without end. A failed alarm write or a failed alarm listing lost the record the same way. The record now goes into the alarm before the dispatch. A run that cannot read or file the alarm dispatches nothing: with Issues disabled the alarm is only annotated, after a failed listing the alarm is left as it is so its record survives, and a failed write stops the dispatch. A failed dispatch rewrites the alarm without the record, so the next run tries again. workflow_run's branches filter matches the triggering run's head branch name, which a fork's pull request can also call main, so watchdog.yml now also requires the run to come from this repository. bot-pr-stuck skips cross-repository PRs, so a fork's autobump/* branch cannot hold it open. The AUR guard in aur-deploy.yml failed the deploy when the AUR RPC could not be read (pipefail made the command substitution fail the step), the opposite of its comment; an unreadable RPC now lets the deploy go ahead.
… holds node.sh checked only kube-system, and let a pod a baked DaemonSet owns stay Pending. NFD runs in its own namespace, so an NFD bump whose master crash-looped passed, and an intel-gpu-plugin pod whose image did not pull stayed Pending and passed too. Device plugin, NFD and DRA bumps auto-merge on the node canary, so a pass has to mean the pods run. Every pod in every namespace must now be Running with its containers ready, or Succeeded. A plugin that needs hardware gets no pod on a hosted runner (its node selector keeps it off), so a pod that exists must run, and one still Pending at the timeout fails. Once up, the pods must hold for CANARY_NODE_SETTLE seconds (default 20) with the same restart counts, so a container that crashes after it started is caught; one restart while the cluster settles still passes. tests/canary_node_test.sh runs node.sh against a docker, k3d and curl shim with scripted pod states: up, NFD crash-looping, a plugin pod stuck Pending, a late pod, a flapping container, one restart, a missing DaemonSet, no pods and a failed cluster create. It is wired into the canary-tests job.
canary/plan.py reads bench/h54/h54.py (flags.sh takes sglang-xeon's flags from its sglang_args), but canary.yml's pull_request paths left it out, so a change there selected sglang-xeon in the plan yet never started the workflow. tests/canary_spec_test.sh now derives every file the plan reads (BASE_PATHS and each engine's locator, derived, flags_source and node paths) and checks that the pull_request paths cover each one.
F3 already has its row and pre-registration on main (from #249), so no second F3 row is added and its prediction is not touched. R1 is written before any F3 step. It takes the desktop holds' lock (~/k3dgpu-bench.lock) before the scratchpad live.lock, so F3 never overlaps H55, H57dp or H59d. It also names the node image tag, the run's own state directory and kubeconfig, and how the ps samples are stored and scanned. No step, ceiling or grade changes.
CI on this branch went red once main moved EAT's vLLM to v0.28.0: tests/canary_spec_test.sh edited `tag: "v0.24.0@`, which no longer matched, so the plan selected nothing and the case failed. The same holds for every engine pin the tests named. The stubs in tests/canary_test.sh reported llama.cpp b11960, vLLM 0.32.0, sglang 0.5.22 and llm-scaler's 0.26.0, and the spec test edited or asserted sglang's v0.5.22-xeon tag, vllm-xpu's 0.32.0, the OVMS pin's form and the llama-cpu model revision. A tree with those pins bumped the way a freshness bump moves them failed 17 cases in canary_test.sh and 7 in canary_spec_test.sh, so every llama.cpp, vLLM CPU, sglang, vLLM XPU, llm-scaler or EAT bump would have been red in CI before its canary ran. The stubs now report the version each pin names (canary/plan.py show): the llama.cpp build, the vLLM CPU and EAT tags, the OVMS tag when the pin has one, sglang's tag written into a copy of the v0.5.22 fixture site, and llm-scaler's setuptools-scm version built from the derived constant. The spec test's edits match any value, the reference-form case rewrites the OVMS pin to both forms whatever the tree holds, vllm-xpu's derived constants must agree on one value, and the cache key is checked against the engine's model and revision. An edit that changes nothing in the tree now fails as such, instead of reading as "selected nothing". Both suites pass on this branch, on it merged with main, and on a copy with the llama.cpp, vLLM CPU, sglang, vLLM XPU, llm-scaler, EAT and OVMS pins and the llama-cpu model revision bumped.
…dment R5 The box owner approved the netfilter module class for this campaign. nft_limit and nft_log, which iptables-nft needs for k3s's limit and NFLOG rules, were loaded on wildct at 05:02:48Z and appended to /etc/modules-load.d/k3dgpu-wildct.conf, with kernel.modprobe left empty. The host's containers, start times, restart counts, published port codes and firewall line counts read the same before, at once after and two minutes after the load. The resume procedure then ran from main at cf49f0e. Release 0.1.81 (the newest, launcher and share/ equal to main's) was installed and its node image pulled by digest; the campaign's 0.1.79 image was removed. Step 1 passed, with H10 reading NFPROBE clean on the 0.1.81 image. The wiring smoke smoke4 passed: up in 30 s, the node Ready and every pod ready after 16 s, no restarts, no GPU plugin pod, no /dev hostPath, no GPU resource in the node's allocatable, and down rc 0. No cluster, container or process of the campaign was left, and the host's services read as before. Amendment R5 records this; RESUME.md marks the smoke done and points at H52w as the next step.
main added the known-issues job at the end of ci.yml (#256) where this branch adds canary-tests, so the two conflicted and gate_merge.sh's sync stops on a ci.yml conflict. Both jobs are kept as they are: known-issues as main has it, then canary-tests at the end of the file.
… out of workflows pins/classes gives every pin in the repo a class: ours (the default, always the newest stable release), eat-stock (tracks Intel EAT's releases), ledger (a ledger row's runner or records) or historical (a finished run's record). A ledger line is an arm while any of its rows is PENDING in the EXPERIMENTS.md Hypotheses table and historical once every row has a verdict, so a verdict landing needs no reclassification. historical-ref lines hold finished runs' exact pins that live runners keep. hack/pin-classes answers class, effective, excludes and coverage. hack/pin-guard enforces the ledger rules on a diff: a historical record is never modified, deleted or renamed; a historical-ref string keeps its count; an arm's pin tokens change only with a dated amendment heading in the same diff that names a PENDING row of the line or one of its aliases; an autobump branch touches neither .github/ nor pins/classes, and EXPERIMENTS.md only on autobump/amend-*; an unclassified pin-bearing file under bench/ or oracle/ fails. A new ci.yml job, pin-guard, runs it with the tests and hack/pin-classes coverage. scripts/images-lock: repin and the new drift subcommand leave ledger, historical and historical-ref files alone; update --only FILE... resolves and checks only the references those files source and keeps the other entries, so one moved digest elsewhere no longer fails a bump; http() retries 429 and 5xx up to five times and honours Retry-After; pins fails on a digest-only reference in an ours or eat-stock file. The digest-only refs in the launcher, share/ and CI now carry the tag that resolves to the same digest (checked against each registry); no digest changed. Consumers that compare or cache by image (plan cells, tune fingerprints, autoquant cache, oracle bundles and release checks) compare name plus digest, so their behaviour is unchanged. Tool versions the workflows installed inline (Prometheus, kubeconform, Chrome for Testing, jsonschema, helm v4, shellcheck, packaging) move to hack/ci-tools.env, which each job loads into GITHUB_ENV, so a token without the workflows permission can move every pin.
…s is on (failing) With deploy_guardrails=on, EAT release-1.1.0's deploy-guardrails.yml installs the vllm chart with core/helm-charts/vllm/guardrail-values.yaml, which names vllm/vllm-openai-cpu:v0.28.0 by tag only (0c726f5 lines 115 and 133-135). These tests ask brownfield.sh to hand that playbook, and only it, a copy of EAT's charts whose guardrail-values.yaml carries the digest vllm-values.yaml already pins for the same image, and to stop before anything deploys when the two disagree. The stub EAT now calls deploy-guardrails.yml the way fresh-install.sh:419 and guardrails-controller.sh do, and its charts carry a guardrail-values.yaml cut from 0c726f5. Ten cases fail: the copy is never made, and no mismatch stops the deploy.
…ass the pin gate On Ubuntu /bin/sh is dash, whose echo turns the \n in the CRD read's jsonpath into a line break. The stub then logged that one call as two lines, and the fresh-cluster case that checks the first three kubectl calls failed in CI while it passed on a host where /bin/sh is bash. The stub now logs and matches with printf '%s\n'. scripts/images-lock pins reads STUB_CONTROLLER_IMAGE=<ref> as an image reference and failed on the two tag-only v1.0.7 fixtures. Those cases now take the tag from the pinned V107 reference through V107_TAG.
EAT release-1.1.0 installs its Qwen3Guard guard with deploy-guardrails.yml, which
runs helm on {{ remote_helm_charts_base }}/vllm with that chart's
guardrail-values.yaml (0c726f5 lines 115 and 133-135). That file names
vllm/vllm-openai-cpu:v0.28.0 by tag only, so with deploy_guardrails=on the guard
pulled whatever the tag pointed to, while every other image eat-up.sh deploys is
pinned by digest.
In stack mode, with the guard on, brownfield.sh now copies EAT's core/helm-charts
to ~/k3d-gpu-guard-charts before anything deploys. The new guard-values.py sets the
copy's guard image tag to the digest vllm-values.yaml already pins for the same
image. It stops the deploy if EAT's guard names another repository or tag, or if
the vLLM values carry no digest. The ansible-playbook wrapper hands that copy to
deploy-guardrails.yml alone, as the extra var remote_helm_charts_base, which takes
precedence over the playbook's vars_files. Nothing else in that play reads the
variable; the utils tasks it includes do not either. The play runs on the
bastion itself (brownfield_deployment.sh:212-222 puts localhost, ansible_connection
local, in kube_control_plane), so the copy's path is where helm runs. A marker file and a check
after the deploy catch a guard playbook that bypassed the wrapper, as for the
gateway, and the wrapper refuses a guard call that has no copy.
Checked against 0c726f5: helm template of the vllm chart with the playbook's
values renders the same manifests from EAT's charts and from the copy except for
the one image line, which now ends in
@sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647
(re-read from Docker Hub with scripts/images-lock digest). The guard has not run on
a k3d-gpu cluster; the shipped config keeps it off and F4 runs with it off.
One more eat_test case covers the wrapper refusing a guard call with no copy; it
fails with that line removed.
A ledger line listed only the rows whose ledger sections name its path, so a runner another row runs or imports turned historical when its own rows finished while the reusing row was still PENDING. #262 landed the H57dp and H57de verdicts and bench/h57d/** went historical while wildct's H57w rows still run it through bench/wildct/row.sh; the guard then refused #268's H57d amendment R2 edit of bench/h57d/arm.sh as a historical edit. A scan of every ledger runner's code for the bench/ paths it calls found the reuses, and each line now lists the reusing rows and their amendment prefix: bench/h57d (the H57w rows; h57d.py also H59), bench/h54/h54.py with its tests and GSM8K items (H59, H60), bench/prefill-guard's cpufreq.sh and prompt.txt (H55), crescent's serving.py (the cross-host rows, H60, H59), bench/xhost/fetch_models.py (the wildct rows), bench/load (the desktop, crescent and cross-host rows and H59w) and bench/shared-prefix (H59). The rest of each tree keeps its own rows, and the H57dp and H57de records get their own historical line now that the H57d tree is an arm again. tests/pin_classes_test.py checks each reuse: the reusing file still names the path, the path's line lists the row, and the path is an arm while the row is PENDING.
Written before any F3 step, after #260's second wiring smoke. keycloak-admin-secret holds a 5-byte username next to its 24-byte password; the username is an account name, and a count of it found 717 lines in the smoke's logs, which would grade any run REFUTED with no secret shown. Step 7 counts the password key only and records that the username was not counted. Step 7's own log, written while it counts, is left out of the count.
…ference scripts/images-lock pins reads `AGENT_OVMS_IMAGE="openvino/model_server` in the two seds that rewrite the OVMS pin's form as an image reference with no digest, which failed tests/image_pins_test.sh. The seds now match the name by pattern, so nothing in them reads as a reference.
…ted holds, pinned test images The images.lock check failed on three tag-only image strings in bench/h59/test_run.py. They are now derived from the pinned references in run.py and the test's node image, so no unpinned literal is left in the file. hold.sh compared the host's CPU by substring, so an i7-13700K passed as the i7-13700 and an i9-13900TE as the i9-13900T. The engine rule matches a row by the exact model number (console/model_opt.py class_tokens), so the host check now uses that function (run.py cpu). A spare could repeat a hold that spare 5 had already measured, and h59.py reads spare 6 over spare 5, so a measured record could be replaced. Spare 6 now runs only after spare 5, and a spare repeats a hold only while the record standing for it has an arm not measured. A hold interrupted before it recorded its rc could run again under the same number. hold.sh now writes `started` once the floors pass, refuses a hold that has one, and spare-ok counts it as run. test_run.py now checks the server's request log against the registered counts and settings (3 warmup, 8 at N=1, 16 at N=4, 4 TTFT requests at 16 tokens on H54's prompt, 6 prefix agents, GSM8K 0-49 at 1024 tokens with thinking off, unique prompts, the 900 s boot limit). Before this a runner with 6 N=1 requests, 32-token TTFT requests or a 600 s boot limit passed every test. The new guard tests failed on the previous commit. Amendment R1 records the spare order, the `started` file and the CPU class rule, and cites OVMS's `--task` option at src/cli_parser.cpp:324, where it is defined (:62 reads it). No prediction, statistic, gate, cap or hold count changes.
…'s idle window The owner chose to gate on the idle window (2026-10-29). As H54 amendment R4 did on crescent, H57d's load gate now reads each arm's mean foreign P-core load over its 60 s idle windows of the performance holds; amendment S1's reading over the N=4 phases is still recorded beside it and no longer gates. On the desktop's records the N=4 reading moves with the arm (H57d spread 2.61, H57dp 2.70, limit 2.00) while the idle windows match (1.46 and 0.21). run.py gains idle_gate and the idle-gate command, whose record h59.py score --load reads; the desktop and wildct profiles name load_rule h57d-idle. test_run.py covers the gate, its CLI, the end-to-end desktop row and recomputes the cited spreads from bench/h57d/results. No prediction, prior, arm, statistic, threshold or hold changes.
# Conflicts: # .github/workflows/ci.yml
# Conflicts: # .github/workflows/ci.yml
… runs bench/h62 landed on main after this branch was cut, so hack/pin-classes coverage failed on the merged tree with three unclassified pin-bearing files (h62.py, qemu.Dockerfile, test_h62.py). bench/h62/** is now a ledger line for H62, and H62 joins the bench/eatflow live.sh, crescent and harness lines whose files its runner calls (pairs.py, live.sh, crescent/hold.sh and gpu-serving.sh).
# Conflicts: # .github/workflows/ci.yml # .github/workflows/dependabot-auto-merge.yml # tests/ci_wiring_test.sh
# Conflicts: # .github/workflows/ci.yml
# Conflicts: # .github/workflows/ci.yml # tests/ci_wiring_test.sh
… reads release-tags workflow_checks.py wants the watchdog to listen to every workflow it can name, and #258 and #253 add Canary and Bot sweep. #265's no-version-literal check flags the 26.04 24.04 loop #252 added to the release job; the loop now takes the variants release-tags prints, which are the same two.
… images by digest The watchdog finds every scheduled workflow, and #258 and #253 add canary.yml and bot-sweep.yml, which the fake gh in tests/watchdog_test.sh had no route for; the healthy setup now answers them with a green run. #265 renders tagged digest refs (openvino/model_server:2026.4.1-gpu@sha256:...), and #268's H59 runner compared the image string to the untagged ref; it now compares the digest-pinned ref with the tag dropped, which names the same image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch merge of reviewed PRs, each merged with --no-ff so its commits stay intact. Code first, then the bench and ledger rows:
Integration:
for v in 26.04 24.04loop ci: alarm on red or missing automation, and harden the release #252 added to the release job. The loop now reads the variants from hack/release-tags output withwhile read, which gives the same two v2604 and v2404 outputs.Local checks on the batch head: