feat(backup): page when a scheduled job is three allowances stale

[Gerrrt]

What changed

A new alert in backup.rules.yaml, ScheduledJobStaleCritical: the same comparison as ScheduledJobStale, against three times the job's own homelab_job_max_age_seconds, severity: critical, with the same human-proof exclusions and a runbook_url to a new row in schedule-maintenance.md § If something goes wrong. An inhibit rule in alertmanager.yaml holds the warning back while the page is up, keyed on homelab_job and instance. Four promtool cases in backup.test.yaml. Eight counted claims move from 157 to 158 and 177 to 178, and deploy-stack.md's Rules-page count to 164.

Why

Found during a repository review. Every rule in backup.rules.yaml is severity: warning, and the routing tree sends a warning to the in-house ntfy only, repeating every twelve hours. A backup that stopped running in week one was therefore a notification on a phone on the same network for as long as nobody acted on it; it never reached the urgent receiver, which only a critical does. Three allowances rather than two because install-timers.sh already pads each allowance for a late run, so two is one missed run and a slow one, and three is a job that has stopped.

Blast radius

prometheus only. One more rule and one more inhibit pair; nothing changes for a job inside its allowance. The existing amtool config routes test assertion urgent severity=critical category=correctness already covers where this routes.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

Run here (no Docker daemon, so promtool test rules and amtool check-config run in CI, not locally):

$ python3 scripts/check_rule_tests.py     # every one of 190 alert rules is selected by a promtool test
$ python3 scripts/check_docs.py           # docs OK — 158 Prometheus + 20 Loki rules, 8 dashboards, 156 panels, 11 assertions
$ grep -c 'alert:' stacks/observability/prometheus/rules/*.yaml   # 158

The test cases: with a 1h allowance the line is 3h, pending until 3h30m (asserted quiet at 2h and 3h20m, firing at 4h with the exact labels and annotations), a job with 100000s to spare stays quiet, and verify-key-backup three allowances stale stays SecretsKeyBackupUnproven's business.

• make validate passes (CI)
• Deployed to the lab and confirmed working (converge on prometheus applies it; make reload is enough)
• Docs updated (docs/observability.md, schedule-maintenance.md, counts in README, security.md, deploy-stack.md, stack README)

🤖 Generated with Claude Code

https://claude.ai/code/session_013Dchg9vmfKeQd7uzRzPcLR

---

Generated by Claude Code

[Copilot]

🟡 Changes recommended

The rule pages weekly and nightly jobs after roughly six periods, while its comments and runbook incorrectly promise three.

3 open findings

• Align runbook paging promises with alert threshold timing · New
• Correct timing examples for allowance-based alert thresholds · New
• Clarify configured allowance versus remaining time · New

What changed in this PR

Adds critical paging when scheduled jobs remain stale for three allowances.

Changes:

• Adds and tests ScheduledJobStaleCritical.
• Inhibits the corresponding warning while critical is active.
• Updates runbooks and alert-count documentation.

File	Description
README.md	Updates alert totals.
docs/​observability.md	Documents the new alert and totals.
docs/​runbooks/​deploy-stack.md	Updates expected rule count.
docs/​runbooks/​schedule-maintenance.md	Adds critical-alert remediation.
docs/​security.md	Updates alert totals.
stacks/​observability/​README.md	Updates Prometheus rule count.
stacks/​observability/​alertmanager/​alertmanager.yaml	Adds warning inhibition.
stacks/​observability/​prometheus/​rules/​backup.rules.yaml	Defines the critical alert.
stacks/​observability/​prometheus/​tests/​backup.test.yaml	Tests timing and exclusions.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Gerrrt]

Secret scan failed on this head, and the failure is not this PR's. The full-history scan runs with --log-opts=--all, so it walks every branch the checkout fetched, and at 05:18 it found a PEM-shaped test fixture in commit 1033681 on the #1187 branch (scripts/check-sops-encrypted.sh:118), a file this PR does not touch. That commit was amended out of #1187 at 05:19 and is no longer reachable from any ref, so no fix needs porting here. Re-running the failed job once to confirm.

---

Generated by Claude Code