[88plug]

The nightly "Auto-update, Build & Release" failed nine scheduled runs in a row (2026-10-20 to 10-28) and nobody was told: GitHub mailed only the last editor of the cron line. Release 0.1.76 left tag aur-v0.1.76 with no image after a Docker Hub 429. A single concurrency group meant one pending release could replace another, and a console-only merge cut no release. This PR is part P1 (alarm-release) of the freshness spec.

Owner action needed before this is useful

The repository has Issues turned off (has_issues: false; gh issue list answers "the '88plug/k3d-gpu' repository has disabled issues"). Every alarm is an issue, so turn Issues on in Settings > General > Features. I did not change it because repository settings are off limits for this work.

Until Issues are on, the watchdog cannot file anything, and it re-dispatches nothing (a re-dispatch is recorded in its alarm first). Instead, hack/ci_alarm prints a ::error annotation and the alarm body to the step summary and exits 3, and hack/watchdog exits 1. Every watchdog run is red, so the gap stays loud.

What changes

hack/ci_alarm open|close|list

• Keeps one issue per alarm key, titled [ci-alarm] <key>: <title> and labelled ci-alarm (the label is created with --force).
• Assigns the issue to $ALARM_OWNER (default 88plug) and puts an @mention in the body.
• When the body changes, it replaces the issue body and comments. An unchanged body comments again only after 24 h.

hack/watchdog and .github/workflows/watchdog.yml

• Runs hourly at :47, and on workflow_run for CI, Docs and the four release workflows, on main only. It always checks out main, never the code of the triggering run.
• Alarm keys it opens:
  • stale:<file> (no green scheduled run within cron period + 6 h)
  • red:<file> (run URL, failed jobs, the last 60 log lines up to the last ##[error], and the first hint from hack/ci_alarm_signatures.tsv; one re-dispatch per failed run id)
  • disabled:<file>
  • ci-missing (ci.yml is dispatched once when HEAD is 30 min old and has no CI run; the alarm opens at 2 h)
  • ci-main
  • release:<ver>: the newest tag older than 45 min must have its Docker Hub and GHCR images, its .deb, .rpm and SHA256SUMS, and its AUR update. A missing Docker Hub image triggers one build-and-push.yml -f release=<ver> dispatch. Older broken tags from the last 14 days are listed for information, and superseded release alarms close.
  • bot-pr-stuck
  • simulate
• Every re-dispatch is recorded in its alarm before it runs, and the record stops the next one. A run that cannot read or file the alarm (Issues disabled, a failed listing, a failed write) dispatches nothing; a failed dispatch is not recorded, so the next run retries.
• workflow_run starts it only for runs from this repository's main (a fork's PR branch can also be called main), and bot-pr-stuck ignores forks' autobump/* branches.
• Its own gh, registry or AUR errors become watchdog-error; the other checks still run.

Release hardening (build-and-push.yml)

• The concurrency group is now build-and-push-${{ inputs.release && format('release-{0}', inputs.release) || 'nightly' }}.
• Each variant builds in up to 3 attempts with the same inputs. Attempts 1 and 2 continue on error, with a 2 min and a 4 min wait after them; attempt 3 fails the job.
• The builds push the versioned tags only. After both variants are pushed, hack/release-tags (checked out from main) copies the floating tags (latest, ubuntu26.04, ubuntu24.04, <k3s>-ubuntu*) with docker buildx imagetools create. It does so only when the version is the highest aur-v* tag on origin at that moment.
• The refusal to overwrite a published version is kept.
• Because releases now build in parallel, aur-deploy.yml skips a version older than the one AUR already has. It also gets permissions: contents: read.

Release paths

• hack/release-paths is the single list of shipped paths. It now adds console/**, images.lock and every bench and oracle file that nfpm or the formula ships.
• auto-release.yml repeats the list as literal push paths, and a test keeps the two equal. release_cas.sh reads the file.
• New dispatch input mode:
  • force (default): today's behaviour.
  • changed: runs as a push, so it releases only when a shipped path changed. Each mode queues separately.

Workflow hygiene

• Every job has a timeout-minutes of about twice its observed maximum (10 min floor; console 40, detect and tune 30, capacity 25, release 60).
• Every workflow has a permissions block.
• The nightly cron moves from 0 6 to 17 6.
• ci.yml gains workflow_dispatch, which the watchdog needs, and an ops-tests job at the end.

Tests (written first, each seen failing for the missing piece)

Suite	Result	What it covers
tests/ci_alarm_test.sh	13/13	fake gh issue store: open, same body within 24 h, changed body, 24 h reminder, close, list, owner override, bad input, issues disabled (exit 3), gh error
tests/watchdog_test.sh	30/30	real gh shapes from the October nights (tests/ops/fixtures/); see below
tests/release_tags_test.sh	6/6	newest vs older release, sort -V, unreadable remote, bad arguments
tests/auto_release_cas_test.sh	26/26	adds: console, bench/vs.py, oracle schema and images.lock changes release; EXPERIMENTS.md, an unshipped bench file and docs do not; the filter is read from main's file
tests/ci_wiring_test.sh	110/110	adds tests/ops/workflow_checks.py (PyYAML): timeouts, permissions, cron minutes, watchdog discovery and workflow_run names, per-version group, retry structure, floating-tag step order, release-path equality and nfpm/Formula coverage, mode mapping, AUR guard (run against a fake AUR RPC: newer skips, same/older/unreadable deploy), fork guard on workflow_run
tests/image_test.sh	31/31	release-job contracts moved to the tag script, retries and floating copy
tests/gate_merge_test.sh	31/31
tests/packaging_test.sh	126/126
tests/image_pins_test.sh	43/43
tests/claims_check.sh	27 files clean
tests/claims_check_test.sh	81/81
tests/ledger_drift_check.py	42 files agree with 81 rows
tests/ledger_drift_check_test.py	OK
actionlint 1.7.12 (ACTIONLINT_IMAGE)	clean
shellcheck v0.11.0 at CI severity	clean	new and changed scripts

The tests/watchdog_test.sh cases:

• The October replay alarms on the first night: red:build-and-push.yml with run 39412124435, job repin-images, the "refusing to allow a GitHub App" line and its hint, plus exactly one re-dispatch. The next hour adds no dispatch and no comment. Night 2 re-dispatches once more and opens stale. The green 10-29 run closes both.
• The 0.1.76 shape gives exactly one release=0.1.76 dispatch, then the alarm. A complete 0.1.77 closes it as superseded.
• No re-dispatch it cannot record: with Issues disabled, with alarm writes failing, or after a failed alarm listing, two runs in a row add no dispatch (the PR's first version dispatched 3 per run, watchdog.yml among them, which looped); with Issues on, one each; a failed dispatch is retried by the next run.
• A fork's autobump/* PR is not a stuck bot PR.
• The other cases: stale, disabled workflow, ci-missing (10 min: nothing; 40 min: one dispatch; 2 h 10 min: alarm), ci-main, GHCR-only gap, a running release defers the check, a 73 h bot PR, the simulate_red round trip, a gh error becoming watchdog-error, issues disabled.

Mutation checks: re-dispatching on every run, dropping the 6 h margin, and dispatching CI before 30 min each turn the watchdog suite red.

A read-only run of hack/watchdog against the live repository was also made, with gh writes blocked. It read every real API shape. It confirmed 0.1.79 and 0.1.77 complete and 0.1.76 missing on Docker Hub, on GHCR and as a GitHub release. It stopped at the disabled Issues as described above.

Open after merge (eyes-on)

• Turn Issues on, then run gh workflow run watchdog.yml -f simulate_red=true. It should open [ci-alarm] simulate, assigned to 88plug, and the next scheduled watchdog run should close it.
• Watch the Actions page for hourly Watchdog runs.
• Check whether workflow_run fires for runs that a GITHUB_TOKEN dispatch started. This is unverified; the hourly cron is the backstop.
• Spec deviations:
  • The watchdog also needs pull-requests: read, for gh pr list.
  • A missing GHCR image alone is alarmed but not re-dispatched, because the release job's overwrite guard would stop a rebuild while the Docker Hub tag exists.
  • README.md and LICENSE.md are not release paths, so documentation rides the next release.
• dependabot-auto-merge.yml only gained a timeout, because P2 removes it.