diff --git a/README.md b/README.md index f933b79a..83b5d1db 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and - 177 alert rules — 157 metric-based in Prometheus, 20 log-based in Loki — sharing + 178 alert rules — 158 metric-based in Prometheus, 20 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -195,7 +195,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology . ├── stacks/ │ ├── observability/ # the estate's stack on prometheus — ten services -│ │ ├── prometheus/ # config, file_sd targets, 157 alert rules +│ │ ├── prometheus/ # config, file_sd targets, 158 alert rules │ │ ├── alertmanager/ # routing and inhibition │ │ ├── loki/ # single-binary config + 20 LogQL rules │ │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/observability.md b/docs/observability.md index 4cfdaf92..72341bba 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -344,7 +344,7 @@ separates a quiet stream from a stopped one. ## Alerting -177 rules in total: 157 metric-based in `prometheus/rules/`, and 20 log-based in +178 rules in total: 158 metric-based in `prometheus/rules/`, and 20 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -505,7 +505,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -157 rules across twelve files in `prometheus/rules/`: +158 rules across twelve files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -517,7 +517,7 @@ argument and for what to do when it exits 1. | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which an hours pair watches for a stalled renewal instead — `TlsAcmeRenewalLate` at 48h, `TlsAcmeRenewalStalled` at 36h ([#426](https://github.com/Gerrrt/HomeLab/issues/426)) — Grafana and the wiki verified against the lab CA ([#847](https://github.com/Gerrrt/HomeLab/issues/847)), the APC card's self-signed one read but not trusted, Prometheus, Loki and Alertmanager over plain http (the switch UI's probes were removed on 2026-09-06, `targets/blackbox.yaml`), each of the household sites behind `trinity`'s Caddy, verified against the tier's root, paging as `SiteDown` after ten minutes so a `make up` restart does not ([#855](https://github.com/Gerrrt/HomeLab/issues/855)), and — the other way round — that the ingest proxy on `10.0.99.20:9090` and `:3100` still refuses a request with no token (`IngestAuthNotEnforced`, [#182](https://github.com/Gerrrt/HomeLab/issues/182)). The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)). Their certificates' expiry is watched without that pass, from `host.rules.yaml` ([#857](https://github.com/Gerrrt/HomeLab/issues/857)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense. Since [ADR-0055](adr/0055-forward-to-adguard-alone.md) AdGuard is the only forwarder, so `AdGuardNotAnswering` is **critical** at five minutes: the house cannot resolve outside names. `AdGuardNotFiltering` stays a warning, because a filter that fails open is a convenience lost, not an outage. The targets in `targets/blackbox-dns.yaml` are live since 2026-09-28, against AdGuard on `trinity` ([#126](https://github.com/Gerrrt/HomeLab/issues/126), [#404](https://github.com/Gerrrt/HomeLab/issues/404)) | -| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github.com/Gerrrt/HomeLab/issues/496)), whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)), and whether the household's copy has been carried to the holder's drive within ninety days and proved by the holder within a year ([ADR-0073](adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md)) | +| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, and the page: `ScheduledJobStaleCritical` at three of a job's own periods and never under a day, because the warning reaches only the in-house channel and a backup that stopped in week one was otherwise a notification repeating on the same network for as long as nobody acted (2026-10-14) — whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github.com/Gerrrt/HomeLab/issues/496)), whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)), and whether the household's copy has been carried to the holder's drive within ninety days and proved by the holder within a year ([ADR-0073](adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md)) | | `deploy.rules.yaml` | Whether each host that pulls — `prometheus` and `trinity` ([#533](https://github.com/Gerrrt/HomeLab/issues/533)) — is running what the repository says: an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` it is, and a host whose record stopped arriving. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)). Also whether convergence had to start a stack service that something stopped — `DeployServiceRevived` — and whether one has been left stopped for two hours, held or under a backup — `DeployServicesStopped` ([ADR-0087](adr/0087-start-a-stopped-stack-service-from-the-converge-timer.md)) | | `internet.rules.yaml` | Why the internet is slow ([#914](https://github.com/Gerrrt/HomeLab/issues/914)): three scheduled speed tests in a row under 400 Mbit/s down or 100 up, the WAN held above 85% of what it can carry, frames arriving corrupted on the WAN port — `WanReceiveErrors`, the fault found the day these were written, when a cable between the XB7 and `morpheus` cut downloads to ~2 Mbit/s — and `speedtest-tracker` itself going quiet or unreachable. Also records `homelab_wan_bits_per_second` from pf's own em0 counters. Download thresholds are against the ~940 Mbit/s the 1000baseT WAN port can carry, not the plan's 2000. `speedtest-tracker` is left out of `InstanceDown`, so its outage is a warning and not a page | | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the fast, per-interface half; `SuricataLogsStopped` in `loki/rules/security.rules.yaml` is the slow, aggregate half ([#90](https://github.com/Gerrrt/HomeLab/issues/90), [#441](https://github.com/Gerrrt/HomeLab/issues/441)) | @@ -541,7 +541,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is 157 rules of 157 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all eleven +against the broken rule too. Coverage is 158 rules of 158 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all eleven in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`, the three container-state rules from diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index d4b6c992..f2c531d8 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -151,7 +151,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. 2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The - page lists 163: the 157 alert rules this repository counts everywhere + page lists 164: the 158 alert rules this repository counts everywhere else, plus the six recording rules: `homelab_suricata_expected_interface`, `homelab_battery_runtime_seconds`, and the two directions each of diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 5e7a3b68..b2afa8b4 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -849,6 +849,7 @@ expected rather than a second fault. | Symptom | Cause | Fix | | --- | --- | --- | | `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` and `verify-ca-key-backup` until you first verify each key, for `offsite-copy` until the first visit makes the copy, and on `trinity` for `household-copy` and `household-proof` until there is a household holder and the holder has opened it. For anything else, `systemctl start homelab-.service` and read the journal | +| `ScheduledJobStaleCritical` | A job has gone three of its own periods without a success — one and a half of the allowance in the `JOBS` table, which is twice the period; a weekly backup three weeks stale, a nightly one three nights, and never under a day for the hourly collectors — and the `ScheduledJobStale` warning for it has been repeating on the in-house channel unanswered | The same first look as the warning: `systemctl list-timers 'homelab-*'`, then `journalctl -u homelab-.service`. The rows below name the usual cause per job. The page holds the warning back while it is up, so a job that recovers clears both at once | | `OffsiteCopyStale` | Ninety days since the newest sets were last copied to the second recipient's medium and proved there | Mount it and run `make backup-offsite DEST=…` — [`copy-the-backups-offsite.md`](copy-the-backups-offsite.md). `--list`, `--verify-only` and `--prune` do not clear it, on purpose | | `CaKeyBackupUnproven` | No offline copy of `certificates/ca-key.pem` has been proved in ninety days — or ever, or not since the CA was re-minted, which the fingerprint label tells apart | Mount the medium and `make certs-verify-backup KEY=…` ([`back-up-the-ca-key.md`](back-up-the-ca-key.md)). After a re-mint, copy the new key there first; the old copy is refused | | `SecretsKeyRecipientsUnrecorded` | The ninety-day deadline is declared and no recipient has a proof series, so `SecretsKeyBackupUnproven` cannot fire however stale the proof is | `systemctl start homelab-recipient-state.service`. If that unit does not exist the timers predate [#400](https://github.com/Gerrrt/HomeLab/issues/400): `make install-timers` adds it and primes it. On a host with one recipient the first write inherits the old `verify-key-backup` proof rather than starting from never | diff --git a/docs/security.md b/docs/security.md index 255d15da..b4aae0a5 100644 --- a/docs/security.md +++ b/docs/security.md @@ -24,7 +24,7 @@ What this network is actually built to survive: | Someone on the lab segment reaching the SOC's web UIs | **Scoped by address, and accepted.** The Wazuh dashboard (`443`) and the Velociraptor GUI (`8889`) on `odin` are published by the stack's `caddy` service (container `soc-agent-msi`), not by their own containers, and its Caddyfile answers Hicks's `10.0.50.0/24` and refuses everyone else ([#1139](https://github.com/Gerrrt/HomeLab/issues/1139)). Until then both answered all of VLAN 30, the six domain guests included, because traffic between VLAN 30 hosts never crosses pfSense — so no firewall rule could have narrowed them. WireGuard peers are not in the list. The residuals: Hicks is a whole /24 with no per-host source yet ([ADR-0031](adr/0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)), and an address on VLAN 30 can be spoofed, so this narrows who reaches the login pages and does not replace them. And the hop from Caddy to Velociraptor skips certificate verification: the GUI's certificate is signed by the CA inside the gitignored `server.config.yaml` on `odin`, and the hop is container to container on the `soc` bridge. The dashboard hop is verified against the Wazuh generator's root | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 177 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 178 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 3e715b42..213a73dc 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -53,7 +53,7 @@ prometheus/ targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/node.yaml node_exporter scrapes, for the host that runs no Alloy (smaug) targets/blackbox*.yaml probe targets, http, dns and latency — hot-reloaded, no restart - rules/*.rules.yaml 157 alert rules: host, network, ups, containers, blackbox, + rules/*.rules.yaml 158 alert rules: host, network, ups, containers, blackbox, dns, backup, ids, deploy, stack, internet and watchdog tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said diff --git a/stacks/observability/alertmanager/alertmanager.yaml b/stacks/observability/alertmanager/alertmanager.yaml index a9542411..a191a764 100644 --- a/stacks/observability/alertmanager/alertmanager.yaml +++ b/stacks/observability/alertmanager/alertmanager.yaml @@ -193,6 +193,15 @@ inhibit_rules: - source_matchers: [alertname = "ManagementCertificateExpiryImminent"] target_matchers: [alertname = "ManagementCertificateExpiringSoon"] equal: ["endpoint"] + # The backup pair. ScheduledJobStaleCritical is the same comparison as + # ScheduledJobStale at three periods, so from the moment it pages the + # warning says nothing the page does not; without this the in-house channel + # keeps repeating the warning every twelve hours beside it. `homelab_job` + # and `instance`: one job on one host is one stream, and a second stale job + # on the same host keeps its own warning. + - source_matchers: [alertname = "ScheduledJobStaleCritical"] + target_matchers: [alertname = "ScheduledJobStale"] + equal: ["homelab_job", "instance"] receivers: - name: "null" diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 491cbd38..37ac3d84 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -97,6 +97,55 @@ groups: `journalctl -u homelab-{{ $labels.homelab_job }}.service` on {{ $labels.instance }} — docs/runbooks/schedule-maintenance.md. + - alert: ScheduledJobStaleCritical + # The same proof, three periods stale. ScheduledJobStale is a + # warning, and a warning goes to the in-house ntfy and nowhere else + # (alertmanager.yaml): a backup that stopped running in week one was a + # notification on a phone on the same network, repeated every twelve + # hours, for as long as nobody acted on it. The off-network pager is + # the `urgent` receiver, which only a critical reaches. So the same + # comparison, further out, pages. + # + # How far out is written in terms of the allowance, because that is + # the series install-timers.sh publishes, and the allowance is twice + # the period in every case: "one missed run and not two", as the + # JOBS table says. So the warning already means two periods, and one + # and a half allowances is three — a weekly backup three weeks + # stale, a nightly one three nights. Two missed runs after the one + # the warning tolerated is a job that has stopped, not one that + # slipped. + # + # Never under a day, whatever the multiple says. The hourly + # collectors have an allowance of ninety minutes, and a page at a + # quarter past two in the morning because gateway-state has been + # quiet since midnight is the warning's job, not this one's. A day + # stale is the floor at which anything here is worth waking for. + # + # The exclusions are ScheduledJobStale's. The human proofs each have + # a rule that says what to do with a medium; a page telling the + # operator to read a journal that has nothing in it is the advice + # that rule exists to avoid. + expr: | + (time() - homelab_job_last_success_timestamp_seconds{ + homelab_job!~"verify-key-backup|verify-ca-key-backup|offsite-copy|household-copy|household-proof"}) + > on(homelab_job) group_left() clamp_min(1.5 * homelab_job_max_age_seconds, 86400) + for: 30m + labels: + component: backup + severity: critical + category: correctness + annotations: + summary: "{{ $labels.homelab_job }} has not succeeded in {{ $value | humanizeDuration }} — three periods" + description: >- + This job has been stale for three of its own periods, and at least + a day. Either the timer is gone or every run since has failed, and + the warning for it has been repeating on the in-house channel + unanswered. Check + `systemctl list-timers 'homelab-*'` and + `journalctl -u homelab-{{ $labels.homelab_job }}.service` on + {{ $labels.instance }}. + runbook_url: "https://github.com/Gerrrt/HomeLab/blob/main/docs/runbooks/schedule-maintenance.md#if-something-goes-wrong" + - alert: ScheduledJobFailed # Separate from staleness so a failure is visible within the hour rather # than after the threshold expires. A backup failing every night for diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index 4c232c0f..759c05f8 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -73,6 +73,68 @@ tests: `journalctl -u homelab-job-stale.service` on prometheus — docs/runbooks/schedule-maintenance.md. + # --- ScheduledJobStaleCritical: three periods, never under a day ---------- + # + # The same two jobs. job-stale's allowance is 1h, so one and a half of it + # is 90m — and the rule floors at a day, so the line is 24h plus the 30m + # `for`: quiet at 2h (which is what asserts the floor is there), pending at + # 24h20m, firing at 25h. job-fresh's allowance is 100000s, so its line is + # 150000s, about 41h40m, and it is quiet throughout. The second case feeds + # a human-only job far past both lines, which is SecretsKeyBackupUnproven's + # business and not a page to read an empty journal. + - interval: 1m + input_series: + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="job-stale", instance="prometheus"}' + values: "0+0x1560" + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="job-fresh", instance="prometheus"}' + values: "0+0x1560" + - series: 'homelab_job_max_age_seconds{homelab_job="job-stale", instance="prometheus"}' + values: "3600+0x1560" + - series: 'homelab_job_max_age_seconds{homelab_job="job-fresh", instance="prometheus"}' + values: "100000+0x1560" + alert_rule_test: + # Past 90m, which 1.5 × 3600s alone would page on. The floor says no. + - eval_time: 2h + alertname: ScheduledJobStaleCritical + exp_alerts: [] + + # Over the day at 24h, pending until 24h30m. + - eval_time: 24h20m + alertname: ScheduledJobStaleCritical + exp_alerts: [] + + - eval_time: 25h + alertname: ScheduledJobStaleCritical + exp_alerts: + - exp_labels: + alertname: ScheduledJobStaleCritical + homelab_job: job-stale + instance: prometheus + component: backup + severity: critical + category: correctness + exp_annotations: + summary: "job-stale has not succeeded in 1d 1h 0m 0s — three periods" + description: >- + This job has been stale for three of its own periods, and at + least a day. Either the timer is gone or every run since has + failed, and the warning for it has been repeating on the + in-house channel unanswered. Check + `systemctl list-timers 'homelab-*'` and + `journalctl -u homelab-job-stale.service` on prometheus. + runbook_url: "https://github.com/Gerrrt/HomeLab/blob/main/docs/runbooks/schedule-maintenance.md#if-something-goes-wrong" + + - interval: 1m + input_series: + - series: 'homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "0+0x1560" + - series: 'homelab_job_max_age_seconds{homelab_job="verify-key-backup", instance="prometheus"}' + values: "3600+0x1560" + alert_rule_test: + - eval_time: 25h + alertname: ScheduledJobStaleCritical + exp_alerts: [] + # --- ScheduledJobStale: a second host joins one-to-one -------------------- # # trinity declares its own threshold in its own textfile directory (#404