Skip to content

security.md

Latest commit

 

History

History
1180 lines (1080 loc) · 84.7 KB

security.md

File metadata and controls

1180 lines (1080 loc) · 84.7 KB
Raw

Security

The lab is a security project, so the interesting question is not "is it secure" but "what is it defending against, and what is it knowingly not."

Threat model

What this network is actually built to survive:

Threat Control
A compromised IoT device pivoting to a workstation VLAN 20 is terminal — no route to any other segment
A guest on the Wi-Fi enumerating the LAN VLAN 10 is terminal, client isolation on
A smart TV's firmware phoning somewhere unexpected VLAN 40 is terminal outward — egress only, and nothing on it initiates into another segment. Since 2026-09-16 two more-trusted segments reach one host on it on named ports (ADR-0016), which changes what reaches in and not what gets out; the igc0.40 tripwire still reads zero packets
A corporate laptop carrying something in from outside Sits on VLAN 50 but has no management access
A lab VM escaping into the house VLAN 30 reachable only from trusted, never to it; the lab tripwire logs anything that gets past the blocks (#234)
A lost or stolen tunnel peer Accepted. A WireGuard peer is a device with a key and no second factor (ADR-0042); the firewall, not the key, bounds what it reaches — the lab and nothing else — with a /32 pin on the server and a preshared key per peer. Revocation is a wg0.conf edit on phoenix (runbook), proportionate at two or three devices and recorded as the thing that stops being so
An attacker on the lab segment reaching the hypervisor's BMC Accepted. shiva stays on VLAN 30 by decision (ADR-0033), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates
An attacker on the lab segment reaching the hypervisor's management plane Closed at the host, and watched. Saruman's Proxmox firewall admits 8006 and 22 from Hicks and 8006 from the deployment host only (ADR-0014, ADR-0043). It was found disabled and turned on on 2026-09-20 (#566); since #576 homelab_pve_firewall_enabled is read every five minutes and PveFirewallDisabled pages after ten, so a pve-firewall stop left in place is noticed rather than found. PveFirewallPolicyAccept does the same for policy_in left at ACCEPT, which reads as enabled and admits the whole segment
Someone on the lab segment reaching the domain's configuration path Scoped, key-only, and accepted for phoenix itself. Each of ADR-0029's six listens on 22 for ansible/ (ADR-0077). The rule admits 10.0.30.70 alone, sshd refuses passwords, and the only administrators' key is phoenix's, installed by the template and never by hand on a clone. The residual: whoever owns phoenix owns the lab domain. That was already true through its Proxmox token, so it adds reach to nothing. Host keys are not pinned, because every rebuild replaces them
Someone on the lab segment reading the domain's metrics Scoped, unauthenticated, and accepted. windows_exporter listens on 9182 on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits 10.0.30.40 (alexander) alone (build-the-lab-domain.md §7). That rule is the difference between an endpoint a /24 sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns alexander reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as node_exporter on the NAS below
Someone on the lab segment fetching a Velociraptor client package Scoped by address, and accepted. odin's caddy service (container soc-agent-msi) serves the two agent MSIs ansible/ installs, and the two Debian packages garuda installs, on 8448 (#1068, ADR-0092). Its Caddyfile answers the six domain addresses, .50–.55, and garuda's .62, and refuses everyone else, Hicks's route from VLAN 50 included. The Velociraptor MSI and the Velociraptor .deb each carry the client config: the server URL, its CA, and the enrolment nonce. Whoever holds either can enrol a rogue client, which is why #864 took the MSI out of SYSVOL. The residual: an address on this segment can be spoofed, and any of the seven can fetch either package. What the nonce buys is noise in Velociraptor's client list, the same class as the Wazuh enrolment password. Rotating it means a new server config, a rebuilt MSI with the role's pin moved, and the staged .deb deleted so that scripts/stage-agent-msis.sh builds it again. The two Wazuh packages carry no secret
An OpenVAS plugin on garuda reaching a non-target Fenced at layer 2 (ADR-0093). OpenVAS scans the domain's six only. While a scan runs, the scanner runs in a network namespace whose one route to the segment is a macvlan, scan0 at 10.0.30.63, behind an nftables netdev egress hook that drops every frame — IP, ARP (send_arp_request) and raw (send_frame) — to anything but the six. So the AF_PACKET paths that bypass an IP filter are caught here. Proved: a GVM scan of an out-of-scope host raised the egress drop counter with 0 open ports, and an arping from the namespace to an out-of-scope host was dropped while an in-scope host answered. gvmd, notus, redis and mosquitto stay on the host; only MQTT crosses to the namespace, on a veth that routes nowhere else
Someone on the lab segment reaching the SOC's web UIs Scoped by address, and accepted. The Wazuh dashboard (443) and the Velociraptor GUI (8889) on odin are published by the stack's caddy service (container soc-agent-msi), not by their own containers, and its Caddyfile answers Hicks's 10.0.50.0/24 and refuses everyone else (#1139). Until then both answered all of VLAN 30, the six domain guests included, because traffic between VLAN 30 hosts never crosses pfSense — so no firewall rule could have narrowed them. WireGuard peers are not in the list. The residuals: Hicks is a whole /24 with no per-host source yet (ADR-0031), and an address on VLAN 30 can be spoofed, so this narrows who reaches the login pages and does not replace them. And the hop from Caddy to Velociraptor skips certificate verification: the GUI's certificate is signed by the CA inside the gitignored server.config.yaml on odin, and the hop is container to container on the soc bridge. The dashboard hop is verified against the Wazuh generator's root
A range target with a path out It has none — ifrit's targets sit on a bridge with no physical port, on 172.30.30.0/24, which the firewall does not route and on which nothing has a default route at all (ADR-0014, ADR-0017)
Someone with the trusted Wi-Fi key quietly joining Kea's lease log reaches Loki; UnknownDeviceOnTrustedSegment fires the first time a MAC appears on VLAN 50 in seven days (ADR-0019)
Losing visibility of a failure 177 alert rules, 30 days of metrics and logs
Someone on a reachable VLAN silencing an alert to hide a failure Alertmanager binds to 127.0.0.1; silences go through authenticated Grafana
Mains power loss The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12. A pack fitted to mjolnir on 2026-08-28 passed its self-test; the TP-Link carrying prometheus and oracle has been on UPS power since 2026-09-08 (#110); the laptops ride a cut out on their own batteries, which HostBatteryHealthLow in host.rules.yaml now reads — prometheus's cell was replaced on 2026-09-18 and reads 101 % of design, oracle's is the original at 72 %, with its replacement bought on 2026-09-19 and in transit (#531) — and prometheus's runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while oracle's never has been; since the same day the projection is recorded on every cut and pages under thirty minutes (HostBatteryRuntimeLow, #532), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — morpheus, Saruman, neo and smaug, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate. ADR-0049 decides that the firewall's NUT server halts Saruman and smaug first and itself last, and shut-down-on-the-ups.md is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested (#574)
The estate being down while the person who runs it is unavailable Documentation, yes; data, not yet off the estate. ADR-0011 puts the emergency tier on paper; ADR-0023 extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet (#455) — see below

What it explicitly does not defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container image, or a vulnerability in pfSense itself. There is no egress filtering by domain or by port (ADR-0014 says why not for the lab, and the reason generalises), and no MFA on the internal services.

The MFA gap has an expiry now, and it is not one gap. ADR-0022 ends ADR-0008's SSO deferral on a state rather than leaving it open: the first real secret, photo or document in the sensitive tier, any reachability from outside the house, or a third account holder — whichever comes first. The second of those has now fired and the deferral was re-accepted, not ended: ADR-0042 opens a WireGuard path terminating on the lab, which takes ADR-0008's no external exposure premise with it. Nothing in the tier became reachable — it is on Winterfell, and the tunnel reaches the lab only — but the lab's own Grafana on alexander did, and that is one of the three below that cannot carry a factor at all. The first trigger fired on 2026-09-28, with Immich's first real photographs, and the deferral was re-accepted again (ADR-0075): no identity provider, and Immich named as the residual that matters. The third trigger keeps its full force, and so does the second for anything on the tier. The floor is per-application TOTP. It is enrolled on Home Assistant (2026-09-28), Stirling-PDF (2026-09-29), and Vaultwarden and Paperless-ngx (reported 2026-10-02). It does not reach everything. Vaultwarden, Paperless-ngx and Home Assistant can each carry a second factor; Grafana, Immich and AdGuard Home cannot — Grafana OSS has no MFA in any edition, Immich's upstream has declined it and points at OAuth, and AdGuard has one password-only admin account. For those three an identity provider is the only route to a second factor rather than a heavier alternative to one. Miniflux joins them (ADR-0057): it has no TOTP, and its passkeys are a second way to log in rather than a second step, so they are left off. Memos joins them too (ADR-0059): passwords and SSO, nothing between. So does Mealie (ADR-0060): it has OIDC and LDAP and no factor of its own, accepted because recipes are the only thing it holds. So does linkding (ADR-0061), which offers OIDC or a trusted proxy header and nothing of its own. Actual is another (ADR-0062): its only factor beyond the one household password is OpenID, and in password mode every signed-in device shares one session that a password change does not end. Grafana is the only one of the six deployed today, which makes "no MFA" a standing property of the estate rather than a pending piece of work — it closes when an identity provider exists and not before.

Availability of the data is a separate question from access to it, and it now has an answer too. ADR-0008's tier will hold the household's credentials, photographs and documents on one mini PC behind a private CA, so if that box is down and the person who runs it is unreachable, the credentials for recovering everything else sit behind the thing that broke — and step-ca means the browser refuses the handshake rather than degrading. ADR-0023 declines to make any of it highly available and constrains the path instead: nothing the household needs in an emergency may have the estate on its only route. The household's own credentials are recoverable without Vaultwarden; Immich, Paperless-ngx and Actual get an encrypted copy outside the estate whose staleness is visible, off-estate rather than off-host, because oracle shares the rack and the power feed; nothing on the break-glass card depends on a certificate this estate issues; and nothing physical may be operable only through Home Assistant. Those fall due on ADR-0022's triggers — the first real credential, photo or document. The photo trigger fell on 2026-09-28, when 615 photographs reached Immich before the copy existed. The copy is built, and was rehearsed onto the household drive on 2026-10-01, but there is no copy of record until a household holder's key is in household.recipients (#455, What backs Immich up). The copy leaving the house is a new residual: it is the first household data to sit in someone else's building, reduced to an availability problem by encryption at rest with a key that never leaves here, and accepted on that basis. ADR-0073 decided on 2026-10-01 that "here" means the household. The copy goes on a drive the household holder keeps at their own address, and the holder's key stays with them, apart from the drive. The residual this adds is that one burglary at the holder's address could take the drive and the key together. Accepted, because the alternative is a key the holder has to ask this estate for, which is the failure ADR-0023 exists to prevent.

The estate's own backup sets leave the house too, and with a key. Decided 2026-09-20: the newest firewall export, volume set and NAS set are carried onto the offline medium that holds the second age recipient, on its ninety-day visit, from that medium's first visit onward (ADR-0048). That medium is the one place where the estate's ciphertext sits beside a key that opens it — the property ADR-0015 refused oracle, and still does; the argument there turned on a powered, networked host in the same room, and this is an offline medium held off the estate by the technical second. The residual is the disclosure delta on losing it: the key alone already means rotating every credential in the repository, and the sets add the firewall's rule bodies and WAN address, the device password hashes in the export, and grafana.db. Accepted, because the alternative is a successor who holds the key and not the thing it opens.

Intrusion detection has been running on Skids (VLAN 20) since 2026-08-21 and on Degens (VLAN 10) since 2026-09-02, one Suricata process per interface. Suricata sits on morpheus rather than the hypervisor because it is the only device that sees the IoT and guest segments, per ADR-0006. Alerts reach Loki through the firewall's syslog pipe, with classification, priority and interface parsed into labels; SuricataHighPriorityAlert and SuricataAlertStorm are armed against them per interface, and the homelab-security dashboard charts them next to the firewall's own block decisions. runbooks/enable-suricata.md covers the setup and the tuning.

Three limits, stated rather than implied:

  • It is alert-only. Block Offenders is off on both interfaces and stays off until a fortnight of understood alerts on each, and probably not on VLAN 20 even then — an auto-block there can take out a camera or the alarm hub. On VLAN 10 it would hit a guest's device whose owner cannot be told why.
  • It watches two segments. Skids and Degens; the rest are unwatched. WAN deliberately never will be.
  • It sees plaintext only. Suricata cannot inspect inside TLS, so the useful signal is DNS, SNI, JA3 and the diminishing share of traffic still in the clear.

Suricata dying is detected as of 2026-09-03, and Suricata going mute as of 2026-09-12. A quiet IDS and a stopped one produce identical log output for an hour at a time, so SuricataStopped in prometheus/rules/ids.rules.yaml reads the firewall's process table over SNMP and fires per declared interface within ten minutes (#90). It proves the process is alive, not that it is inspecting anything. SuricataLogsStopped in loki/rules/security.rules.yaml covers the rest of that gap slowly: it fires once both interfaces have been silent for nine hours, a window read from 22 days of the stream whose longest silence was 75 minutes, and it cannot see one interface going quiet on its own — the guest segment is silent for days at a time (#441). Re-measured on 2026-09-17 over 27.7 days, the worst aggregate silence was 80 minutes and the guest segment alone crossed nine hours seven times, so the window holds and the decision to aggregate is what keeps the rule quiet. The runbook's test alert is still the only proof that it detects.

A limit none of the three names, and no rule here closes: Suricata is not a protocol logger, and the lab is out of reach. SNI, TLS fingerprints and certificate metadata — the ground the plaintext limit gives up — are Zeek's (JA4+ since #776, by ADR-0069), and east-west traffic between the lab's domain guests crosses no router, so morpheus never sees a packet of it (#437, ADR-0006). When that sensor is built its logs stay on alexander and never reach 10.0.99.20, which ADR-0007 requires and ADR-0020 enforces by giving the lab no Alertmanager for a ruler to deliver to. So the absence rule this section describes has no Zeek equivalent and will not get one: whether that sensor is still running is answered on the hypervisor, as guest state crossing under ADR-0028, by the homelab_zeek_mirror_active gauge #437 builds alongside the mirror. That mirror is tc on Saruman's existing Linux bridge, not Open vSwitch, and the gauge reads 1 only while every port of the bridge mirrors to the sensor and packets arrive (ADR-0068); ZeekMirrorInactive pages when it does not. It proves the mirror, not the Zeek process inside the guest, and says so.

Device joins are detected as of 2026-09-04, from the DHCP server rather than from the wireless. morpheus ships Kea's lease log to Loki, and the first lease a MAC takes on Hicks or Winterfell in seven days raises an alert — warning on the trusted segment, critical on management, where nothing has joined in the 13 days of logs the rules were written against. The eero cloud was rejected as the source: it is a two-minute poll of Amazon for an event the firewall logs in the same second, and it is unavailable exactly when the WAN is (ADR-0019).

Three limits here too. It sees leases, not associations — a device with a static address never asks, and never appears. It sees the segment, not the radio: on Hicks it cannot tell a new laptop on the cable from a new phone on the Wi-Fi, and it never knows which access point. And the noise floor is private MAC addresses: iOS and Android rotate them per network, so a rotation reads as a new device, which is accepted rather than filtered out because filtering it would blind the rule to the addresses an intruder would present.

Segmentation

Default deny holds for Switch LAN, Winterfell (99), ImaginationLAN (30), CasaBonita (40), Skids (20) and Degens (10). Each blocks every other segment explicitly before its egress rule, and the narrow exceptions that exist — SNMP to the iLO and to the switch, SSH to the firewall, the enumerated passes from Hicks into management — are listed in network.md and in ADR-0013.

It does not hold for Hicks (50), and since 2026-09-08 that is a decision rather than an omission. Hicks reaches management on a named list above a logged block, and reaches all of ImaginationLAN by a rule that says so — the lab is meant to be reachable from trusted workstations, and cannot be narrowed until the lab exists and its list does. ADR-0031 records both, and the two deviations it deliberately left in place: the lab rule is TCP-only, with the catch-all carrying the rest, and the DNS and NTP passes to 10.0.99.1 carry nothing because Hicks resolves at its own gateway.

The switch LAN was the other exception and is not any more. Since 2026-09-02 that interface carries six logged blocks — one per VLAN — above an egress rule renamed Allow internet, with DNS and NTP to the gateway the only passes above them (#229); since 2026-09-06 nothing on Winterfell can reach it either (ADR-0025).

IPv6 terminates at the WAN and is not carried inside the estate. That was never written down and is now measured rather than assumed (#353, 2026-09-07): the WAN interface holds one global v6 address and a working v6 default route — the firewall itself reaches the v6 internet in about 11 ms — while no VLAN interface has a global v6 address, and neither radvd nor dhcpd6 is running. So nothing inside is offered IPv6 and nothing inside can prefer a route that does not work, which is the failure a half-configured v6 stack usually causes. The estate is addressed in RFC1918 IPv4 by decision, and this paragraph is that decision.

The WAN_DHCP6 gateway's monitor answers, since 2026-09-30. From 2026-09-07 it reported 100% loss while the link was fine. dpinger was monitoring the ISP gateway's link-local address, which does not answer ICMPv6 echo, while traffic went through that same gateway. make gateway-state collects both the status pfSense reports and whether each family actually reaches the internet, and GatewayMonitorUnreliable fired on that disagreement for three weeks.

The fix was a firewall change. The gateway's Monitor IP now points at 2606:4700:4700::1111, the address the collector's own v6 probe uses. The next collection read status 1, 14 ms, and forwarding 1. The monitor now measures the uplink, so a real v6 outage shows as GatewayDown and not as the same alert that fired for a working link.

IPv6 is where the segmentation pattern is not finished, and both halves of the gap are worth naming because the documents recorded only one of them for a while. The switch LAN interface still carries pfSense's stock Default allow LAN IPv6 to any with no IPv6 blocks above it, so on paper it reaches every VLAN. Going the other way, CasaBonita, ImaginationLAN, Skids and Degens each carry paired inet/inet6 blocks toward the six VLAN macros and neither toward the switch LAN, so their IPv6 catch-all reaches it. Both are latent rather than live for the same reason — igc0 has only a link-local address, which does not route — and both are divergences from the pattern rather than paths anything can use today. #353 owns whether IPv6 is carried at all.

Every claim in the four paragraphs above is checked against the running firewall rather than reviewed. docs/firewall-claims.yaml states the posture in the form pfctl can answer and make check-firewall diffs it against the live ruleset; make check-docs asserts this section names the same segments the claims file does. Neither can run in CI's sandbox — the ruleset is not in this repository and deliberately never will be — so the first runs at deploy time and on a timer. ADR-0026 records why that is the shape, and #363 is what built it.

Winterfell is the half that has since been narrowed. On 2026-09-02 the Hicks interface gained ten host- and port-scoped passes into 99 and a logged Block access to Winterfell under them, so Hicks now reaches management on an enumerated list — SSH and ping to the segment, the firewall's admin UI, resolver and NTP, the wiki, Grafana, and the UPS card — and nothing else. network.md holds the list and is the document to read for it. ADR-0013 read the ruleset the day before that landed and describes the wider state; it is left as written, per ADR-0001, and this section is where the current posture lives.

This section previously said "three exceptions", ADR-0002 said two, and ADR-0008 said five. All three were counts, and a count cannot express "reachable because a catch-all was reached". ADR-0013 supersedes ADR-0002 and replaces the count with a list; the audit behind it is recorded there. A document that overstates a control is worse than one that admits the exception, and this one overstated it for as long as it was a number.

Everything else — IoT, media, guest — gets internet and nothing more.

That sentence is about what those segments initiate, and two decisions now reach into them without touching it. ADR-0016 writes passes into CasaBonita for the NAS and ADR-0035 one into Skids, and they are at different stages — this paragraph said neither was created, and half of that stopped being true on 2026-09-16. CasaBonita's exist. Hicks reaches 10.0.40.30 on 443 and 8096, and 10.0.99.20 reaches it on 9100 and 22: four host- and port-scoped passes above Block access to CasaBonita, verified in position with pfctl rather than read off the web UI, where an appended rule looks present while matching nothing. ADR-0016 wrote three and four exist, because the Hicks pass is split rather than carrying a port list, and the 22 was inert until 2026-09-19, when build-the-nas.md §6.2 switched SSH on for the backup pull — TrueNAS ships it disabled (ADR-0040), and ADR-0045 is the decision that turned it on: key authentication only, one unprivileged read-only user, reachable from 10.0.99.20 alone by the rule that already existed. The residual it leaves, accepted, is one more service on the NAS with one more key that reads it — a key that lives on the host already holding the estate's age identity, and reads a directory that includes Jellyfin's users' password hashes — and, since Audiobookshelf was deployed on 2026-09-29, its users' hashes too, in the same pull (ADR-0050), and since Navidrome's on 2026-09-30, the hashes in its navidrome.db (#141). Two more are for the phones on Hicks, and both exist. The issues that proposed Audiobookshelf and Navidrome each said no new rule would be needed; the Hicks passes are per port, so each needs one. Allow 4533 to smaug, for Navidrome (#141), was created on 2026-09-22 ahead of the service, and what could be proved without a listener was: its position above the block, read from morpheus, and the monitoring host still refused on 4533. That Hicks reaches it was proved at the deploy on 2026-09-30 (build-the-nas.md §6.6). vlan50 net → 10.0.40.30:13378, for Audiobookshelf, was created by §6.5 and a Hicks workstation reached it on 2026-09-29 — ADR-0050 calls it the fifth; it is the seventh. Neither adds a residual 8096 did not already have: each first admin is made before a phone is pointed at it, and the libraries behind both are mounted read-only. One more is for workstations, and it writes. vlan50 net → 10.0.40.30:445, Allow SMB to smaug, lets a Hicks workstation mount the media share (ADR-0051). It was created on 2026-09-23, and a Hicks workstation has mounted the share through it. Its residual is the one write access always carries: a workstation holding the password can delete or encrypt the library, and erebor/media is not snapshotted. The library is replaceable (ADR-0008), so this is accepted. What limits it is the user: samwise is a second SMB user, typed on workstations and never stored on a television, so the televisions' bilbo and the workstations' credential are revoked apart. The same pass puts an SMB login prompt in front of everything on Hicks, the corporate laptop included, as 443 already puts the TrueNAS one. One more is from ImaginationLAN, and it exists. 10.0.30.110 → 10.0.40.30:2049, Allow NFS from Saruman to smaug, lets the hypervisor mount erebor/iso, the ISO store Packer builds templates from (ADR-0072). It was created on 2026-10-01. Saruman mounts the share through it, and alexander, on the same segment, is refused. Its residual is the one NFS with sec=sys always carries: the export trusts an address. Anything on VLAN 30 that took Saruman's address could replace an installer, and every template built from it afterwards would carry the change. The control is not the rule. It is a daily hash of every file on the store against the list in scripts/collect-iso-store-state.sh, run on Saruman, which pages IsoChecksumMismatch if a listed ISO changes and warns on any file the list does not name. phoenix, which runs Packer, holds read-only PVEAuditor on the store and cannot write to it. The gap this leaves is up to a day between a replacement and the page; a template built inside that day is the one to suspect. Root on Saruman maps to pippin, which owns that one directory, so the hypervisor is not root on the NAS. network.md holds the current list. Skids has one, since 2026-09-28: 10.0.99.40 → 10.0.20.20:80,443/tcp — Home Assistant to the Hue bridge, the one device on that segment with a local API — above the block that has stood between 99 and 20 since the segments existed (#134, ADR-0035). Nothing on Skids can start a connection through it; the #223 tripwire read 0 after pairing.

One of CasaBonita's six is a monitoring pass, and it carries a residual worth naming. 10.0.99.20 → 10.0.40.30:9100 lets Prometheus scrape node_exporter on the NAS (#256), and that endpoint is unauthenticated. The firewall rule is what stops other segments reading it; nothing stops CasaBonita itself, because the televisions and the game consoles share that broadcast domain and the firewall never sees those packets — the same property that lets them reach Jellyfin, working the other way. node_exporter has no write API, so the exposure is disclosure of the host's shape: filesystems, uptime, load. Accepted, and the same class as the unauthenticated ports the observability stack publishes. Audiobookshelf's 13378 is reachable the same way and is not the same residual: everything there but its health and status endpoints wants a login.

The row ADR-0008wrote as 99 → 20 is narrower than it read: one host to one device on two ports, with the twenty other devices on Skids still unreachable from anywhere, and both segments still initiating nothing. The tripwire below is the check that the second half holds — waiting on Skids, and since 2026-09-16 actually checking on CasaBonita.

That sentence is now checked rather than asserted. Four tripwire rules sit below the block rules that stop each cross-segment path and above the → any egress rule: three on the terminal interfaces (#223), pass + log for <terminal net> → Internal_Segments, and a fourth on ImaginationLAN (#234, decided by ADR-0014), pass + log for <lab net> → House_Segments. The lab's rule points at a different alias on purpose: Internal_Segments names the lab itself, so against it every DNS query from the lab to its own gateway logs as a crossing — 1,239 such lines in the rule's first three days after 2026-09-06, none of them real, which is why #234 is not closed by the rule existing. They log nothing while the design holds, and cost nothing; if one ever logs a line, TerminalSegmentReachedInternalNetwork or LabSegmentReachedInternalNetwork fires on it. Before them the first of those alerts matched action="pass" against a firewall that logged only blocks, so it could not fire for any input — the control was described here and not actually watched. Note that a firewall restore from a backup older than 2026-09-01 drops them silently; the restore runbook checks for them.

On CasaBonita that tripwire is now the only control watching the segment, and since 2026-09-16 there is something on it worth watching. smaug at 10.0.40.30 is scraped rather than pushing, by the same ADR-0016 decision that keeps the segment terminal outward: it runs no Alloy agent, and with the agent go its logs, because Loki has no pull — ingest is a push or it is nothing — so the only rule that would centralise them is the 40 → 99:3100 the ADR refused, and refusing it is the residual recorded in SECURITY.md (#255). Its SMART attributes were the same residual until ADR-0047 put them under the scrape instead — a root cron job in TrueNAS's UI runs the estate's collector from a copy on erebor/apps, which root writes, frodo reads and the SMB share does not reach, and the exporter serves a world-readable .prom with no serial numbers in it; the disclosure that adds to the unauthenticated 9100 is the host's disk models and their health, which is accepted for the same reason the port is. Its patch state is deliberately not collected — an appliance with no apt to ask — and the ADR records that (#483). Suricata does not watch igc0.40 either; it runs on Skids and Degens only. So the tripwire is what is left, and it is worth being exact about what that buys: TerminalSegmentReachedInternalNetwork fires on a pass from 10, 20 or 40 toward 30, 50, 99 or the switch LAN, which is a compromise leaving the segment. Anything smaug does that stays on CasaBonita, or goes straight out to the internet, is not merely unalerted — it is unrecorded.

There is no alert for that and there cannot be one, which is a decision and not an omission. A rule reading "no logs from smaug" is an absence rule, and an absence rule needs a declared expectation to measure the silence against — homelab_suricata_expected_interface is that pattern, and SuricataLogsStopped only became writable once somebody measured what a normal silence looks like: 22 days of stream, a worst gap of 75 minutes, nine hours chosen as seven times it. Here the expected volume is zero and always was, so silence is the design working and no threshold separates it from a compromise. The two collection-completeness rules are no help for the same reason. JournalSourceStopped and LogEntriesDropped are both keyed on an agent that exists, and check_loki_coverage.py takes its denominator from the hosts Loki is already hearing from — so a host that has never shipped a line is absent from the question rather than failing it. The firewall and the tripwire are the control. This paragraph is the record that there is nothing else.

Segmentation used to do more work here than it should have had to. Prometheus and Loki published unauthenticated ingest ports for the agents to use, so anything that could route to 10.0.99.20:9090 or 10.0.99.20:3100 could read and write the metric and log stores, and delete logs, without a credential. That is exactly the failure ADR-0002 predicted when it recorded that "a compromised workstation reaches Winterfell". The firewall narrowed who "anything" was: since 2026-09-02 Hicks reaches 10.0.99.20 on 3000 only, and 10.0.30.110 on ImaginationLAN has one explicit pass for Saruman's agent. But a control that depends on one un-reviewed rule ordering is not authentication.

Since #182 it is authenticated (ADR-0067). Prometheus and Loki are on 127.0.0.1. The host's address on both ports is an ingest proxy with an allowlist:

  • an agent token, one per host, may push and do nothing else;
  • the reader token may query and do nothing else;
  • the admin API, the lifecycle endpoints and Loki's delete API are refused to every token.

Reaching the port now grants nothing. The firewall still narrows who can try, and the two layers are independent: a bad rule ordering no longer hands over the stores, and a stolen token still has to come from a segment that can route. The blackbox probes in targets/blackbox.yaml ask the published address for a query and a delete with no token, and IngestAuthNotEnforced pages if either is ever answered by anything but the proxy's 401.

Since #764 both ports serve TLS under the estate CA (ADR-0086), and every client verifies the chain, so no token crosses VLAN 99, or Saruman's pass from VLAN 30, in cleartext. What remains is deliberate, and SECURITY.md records it: loopback is unauthenticated, for a local user who already holds the SOPS key.

The lab's stores got the same treatment second, and needed it more (#834). alexander published its Prometheus and Loki to all of VLAN 30, unauthenticated, from 2026-09 (#436). VLAN 30 is the segment that exists to hold attackers. From there, anything could:

  • POST /-/quit to the lab's Prometheus;
  • forge series through remote-write;
  • delete log ranges through Loki's live delete API, in the store that holds the segment's own evidence.

Authored, and live only after its rollout. Until the ordered steps in stacks/lab/README.md have run and their three curls return 401, the exposure above is still the running state: none of these hosts converges. The fix is the estate's proxy, moved one segment down: the stores on loopback, and stacks/lab/Caddyfile on 10.0.30.40 with one token each for odin, phoenix and fenrir. Those tokens are the lab's own, in secrets/lab.sops.yaml. A lab host never holds an estate token, and deploy-agent.sh refuses to decrypt the estate's file for a lab target.

Two residuals are recorded rather than fixed:

  • The tokens are plain HTTP on a segment built for ARP spoofing, so they can be stolen by anyone in a position to spoof. A stolen token can still only push, or read; it cannot quit or delete. The estate's proxy answered this with TLS (#764, ADR-0086); the lab's has not been asked yet.
  • Nothing pages if the proxy is bypassed. The lab has no blackbox exporter and no Alertmanager (ADR-0020), so its version of IngestAuthNotEnforced is a step in the rollout check, not a rule. Making the lab's failures reach the estate is #858.

What has been taken off the firewall's shoulders is Alertmanager. It had no off-host client, so it now binds to 127.0.0.1 and reaching VLAN 99 no longer lets anyone silence an alert; see ADR-0012.

The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya white-noise machine. Every one of those is a network-connected computer running firmware nobody outside its vendor has audited, several with no update mechanism at all. Treating them as untrusted is not paranoia; it is the only assumption consistent with what they are.

Secrets

  • Credentials are encrypted with SOPS + age and committed in encrypted form. See secrets/README.md.

  • The private key lives at ~/.config/sops/age/keys.txt on prometheus, the host that deploys the estate's stacks, and is never in the repository. (ADR-0043 calls phoenix "the deployment host" too. phoenix builds machines, holds no age key, and is covered below.) That host's disk is not encrypted — see below.

  • That key is the single point of failure for every encrypted secret here, so it is copied off the host and the copy is proven to decrypt with make secrets-verify-backup KEY=<copy> — which refuses to run against the live key and blanks the environment first, because the obvious hand-typed equivalent passes even for an unrelated keypair. See runbooks/back-up-the-age-key.md.

  • scripts/render-config.sh decrypts at deploy time into gitignored files. Nothing writes a plaintext secret into a tracked path.

  • make secrets-edit hardens $EDITOR before handing it the decrypted file, so the editor cannot persist the plaintext in an undo file, swap file or backup that sops does not shred. See secrets/README.md.

  • CI runs gitleaks with rules specifically for SNMP communities, inline Grafana passwords, PEM private keys, age secret keys and unencrypted OpenTofu state. It separately asserts that every secrets/*.sops.yaml is genuinely encrypted, and that no state, plan or .terraform/ is tracked.

  • phoenix is a secret-bearing host, and holds no age key. Its secrets are all mode 600 and owned by its operator:

    • the Proxmox API token phoenix@pve!builder;
    • the Windows build password, which is also a fresh clone's Administrator password until #448 rotates it;
    • tofu/'s state passphrase, all three in ~/.config/proxmox/phoenix.env (ADR-0074 §4);
    • the encrypted state itself, tofu/state/lab.tfstate (ADR-0076);
    • the lab domain's three, also in phoenix.env: LAB_ADMIN_PASSWORD, which is Domain Admin, LAB_DSRM_PASSWORD (ADR-0077), and LAB_POPULATION_SEED, from which every population password is derived (ADR-0078). The seed adds no reach: the first of the three already owns the domain.

    The state is the one that would have been quiet. A Terraform state holds every value a provider touched in cleartext, the cloud-init password of every guest among them. OpenTofu encrypts it, enforced = true refuses a plaintext write, and scripts/check-tofu-state-encryption.sh proves both, in CI against a canary and on phoenix against the real file. The passphrase is escrowed in secrets/tofu.sops.yaml to the estate's two recipients, so phoenix wrote that copy and cannot read it. What protects all four on the host is file permissions and the token's scope: PhoenixBuilder on named paths, never at /. That scope is the sentence to argue with, because whoever holds the token can create and destroy guests.

  • The alert path's credentials are in SOPS on two hosts, and one of them crosses the CA boundary on purpose. Since #136 Alertmanager delivers to the tier's own ntfy. ntfy is deny-all, and the users, access list, token and topic names are all NTFY_* keys in secrets/sensitive.sops.yaml. Alertmanager's copy of the token is ALERTMANAGER_NTFY_TOKEN in the observability file. It is sent as a header and never as ntfy's ?auth= parameter, because Caddy's access log records the URL and Alloy ships that log to Loki. To verify the leaf, the estate's Alertmanager trusts the tier's root, which ADR-0037 otherwise keeps apart from the estate's. The trust is scoped: a ca_file on those three webhook configs and on one blackbox module, not the container's trust store. Nothing else on prometheus is asked to believe that root. The urgent and security channels also keep their ntfy.sh topics, where the name alone is the credential, as before.

  • One service on the sensitive tier keeps its credentials outside SOPS, by necessity and on the record. Home Assistant obtains device credentials through its own pairing flows — the Hue application key, the Ring token — and writes them to its store inside the home-assistant-config volume; no environment variable or rendered file is a way to hand them in. So the rule above covers what that stack takes from outside, and the tier's most numerous credentials are protected instead by the disk-encryption decision #404 makes and by the encrypted volume archive — which therefore carries live credentials, as grafana-data already does. ADR-0035 records the deviation and what would retire it.

  • The media tier keeps its admin credential outside SOPS too, and has no secrets file at all. Jellyfin's admin is created by its own setup wizard and kept as a hash in jellyfin.db under its /config bind mount, and Audiobookshelf's root is created on its first-run screen and kept as a hash in absdatabase.sqlite (ADR-0050); no environment variable or rendered file is a way to hand either in, so stacks/media has no secrets/media.* and no .sops.yaml rule — on purpose, decided on #528, and not because the stack was deployed by hand. The plaintext is in the operator's password manager, where pfSense's and iLO's already are. What protects it is a hash at rest, VLAN 40's terminal property, the 50 → 40 passes being the only way in, and ADR-0008's blast radius — a media server whose data is replaceable. Navidrome (#141) is the same class: its admin is made by navidrome user create --admin in the container, prompted for on the TrueNAS shell, and kept as a hash in navidrome.db on the bind mount backup-nas.sh pulls. What would retire it: a service on that tier taking a credential from outside. Then the tier gets secrets/media.sops.yaml and a rule of its own under ADR-0020, and because smaug runs no sops the key would be the operator's rather than the host's — a decision for that day and not before. Recovering the admin without the password is in stacks/media/README.md.

  • The sensor stack has had a secrets file since #834, and it holds one key. Until then it had none, because it had no secrets: Zeek takes no credential, and fenrir's Alloy pushed to the lab's unauthenticated Loki and Prometheus. It was brought up with docker compose rather than make up (#437, ADR-0068).

    • What retired that is the trigger this entry named: a service there taking a credential from outside. The lab's stores now sit behind an ingest proxy, and fenrir needs a token to push.
    • So it took the shape this entry prescribed. secrets/sensor.sops.yaml has a rule of its own above the catch-all, and make secrets-init STACK=sensor on fenrir writes its key there. The stack is now brought up with make up STACK=sensor.

  • The scratch stack's secrets exist only on its guest, and are never committed. diabolos is built for one investigation and destroyed at the end of it (#438, ADR-0071). It has its own .sops.yaml rule above the catch-all, so it can never open the estate's credentials or odin's. The rule's REPLACE_WITH_SCRATCH_AGE_PUBLIC_KEY placeholder is permanent in git. Each incarnation runs make secrets-init STACK=scratch on itself, which writes its key into its own checkout and encrypts secrets/scratch.sops.yaml there, and .gitignore keeps that file out of git. The key has no backup, unlike every other key in ADR-0024, on purpose: the key, the secrets and the data they protect are destroyed together by qm destroy 161 --purge.

  • The BloodHound stack's key is one recipient on one guest, and has no backup (#451, ADR-0081).

    • How it is held. eden's rule sits above the catch-all, with a placeholder until the guest exists. So the host holding a map of the lab domain's weaknesses opens its own four secrets and nothing else.
    • Why it gets no second recipient, unlike ADR-0024's other keys. What the key guards is all re-issuable:
      • the database's password, which guards data that is itself one collection run from rebuilt;
      • a first admin, which matters only on the first start;
      • a session signing key;
      • an ingest token, whose other copy is on alexander.
    • What losing it costs. New values, a fresh data disk and a collection run. check_sops_rules.py's list of stacks whose key guards data therefore leaves it out.
    • The graph itself never leaves the guest. Not to a backup, and not to VLAN 99. Its Alloy ships the guest's telemetry to alexander, not the graph.

Known historical exposure

This repository previously committed real credentials. Removing them from HEAD does not remove them from history, and anything ever pushed to a public repository must be treated as compromised:

What Where Status
SNMP community shared across all four devices snmp.yaml, from commit ee3d443 (now rewritten) Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. morpheus, mjolnir and shiva verified answering the new community and refusing the old. neo answers the new one; whether it still holds the previous one is unverified, not confirmed. Its agent serves GETBULK to any community of sixteen characters or fewer without consulting the table (measured 2026-09-12), which is what the 2026-09-06 and 2026-09-09 sightings of the old string and the stock public and private actually were; those measurements are withdrawn. Over GET, which it does check, both stock strings and a junk string are refused — the previous community's row is the one thing still unmeasured, because that string was purged from history and is not to hand. Accepted risk, see SECURITY.md and the runbook
Grafana admin / admin with anonymous Admin access compose file Fixed: password from SOPS, anonymous auth disabled
Decrypted secrets in editor undo files ~/.local/state/nvim/undodir/, written by make secrets-edit Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. make secrets-edit now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis
Alertmanager webhook URL and the MokerLink SNMP community a local Claude Code session transcript under ~/.claude/projects/ Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the neo residual below all over again. That reasoning is spent — the overwrite it was afraid of was done on 2026-09-12 and persisted — and the value retires with the MokerLink hardware, which is replaced by a switch commissioned with v2c off entirely; no value the MokerLink held is carried across
Passphrase-encrypted TLS private keys certificates/, added in efb2632, deleted in 647d90a Purged from history, and the CA replaced — see runbook. Anything that trusted the old CA must be re-pointed at the new one

CI scans both the working tree and the full history, with no ignore file. Both must be clean unconditionally.

There was a .gitleaksignore listing nine historical findings, each annotated with what it was and why it was still there. It was an acknowledgement, not a fix, and it existed because a CI job that is permanently red for a known reason gets ignored — and then a genuinely new leak goes unnoticed alongside it. The purge removed what it acknowledged, so the file was deleted. A history scan that passes with no exceptions is the evidence the purge worked.

Everything above sits on an unencrypted disk

Every entry in that table is something that happened once. This one is a standing property of the host, which is why it is stated separately rather than added as a sixth row.

Measured on prometheus (10.0.99.20):

  • /dev/mapper holds control and ubuntu--vg-ubuntu--lv and nothing else — no LUKS anywhere. The root filesystem is plain ext4 on LVM.
  • /boot and the EFI partition are likewise plain.
  • /swap.img is 4 GiB, unencrypted, on that same root filesystem, and in use. Anything the stack has held in memory can have been paged into it.

Since #114 the containers no longer contribute to that last line. Every service in stacks/observability/compose.yaml sets memswap_limit equal to its mem_limit, which is memory.swap.max = 0 — the stack cannot page out at all. Left unset, memswap_limit defaults to twice mem_limit, so setting a memory limit without one would have increased what reaches this file: a constrained container spills to swap rather than being killed, and Grafana's admin password, Alertmanager's receiver URLs and the SNMP community string are all resident in those processes. The measured cost of switching it off is small — peak container_memory_swap over the fourteen days that sized the limits was 70 MiB for Prometheus, 17 for Alloy, 12 for Loki and under 5 for the rest.

This narrows the exposure; it does not close it. Swap is only one of the ways memory reaches the disk, the host's own processes still swap freely, and anything already paged out before this change was deployed is still in /swap.img. The files below remain the substance of this section.

So the age private key at ~/.config/sops/age/keys.txt, the rendered artefacts under snmp-exporter/.rendered/ and alertmanager/.rendered/, and stacks/observability/.env — the last three hold plaintext by design, because something has to hand the containers a usable credential — are protected by nothing but file permissions. They are all mode 600 and owned by robo, which is the right setting and is also the entire control. Permissions are enforced by the running kernel; they mean nothing to a disk read on another machine.

This is accepted, not scheduled. The threat model above already excludes an attacker with physical access to the rack, and this is that exclusion restated where it actually bites. Full-disk encryption on a headless host has its own failure mode — either a passphrase nobody is present to type after a power cut, or a key stored on the same machine, which is most of the way back to where this started.

It is recorded because it changes the severity of things that would otherwise look minor. The undo-file leak above is the worked example: three community strings at mode 664 in ~/.local/state/nvim/undodir/ were a real finding because the disk beneath them is readable. On an encrypted disk that is a much smaller problem. Neither fact is interesting alone.

The UPS reported a battery it did not have

Until 2026-08-28 mjolnir had no battery installed. Its Network Management Card nonetheless reported 100% state of charge, 48.0 VDC, a battery temperature, an hour of runtime, a 2030 replacement date, and upsAlarmsPresent = 0. Every one of those values was derived rather than measured.

The single honest signal it emits is the self-test result. The management card rendered it as Refused — internal fault; over SNMP it is upsTestResultsSummary = 4 (aborted), from the standard UPS-MIB the apc_ups module already walks. No extra OIDs were needed to see it.

Any alert rule keyed on charge, runtime or alarm count therefore could not fire, no matter how bad things got. UpsSelfTestFailed and UpsBatteryUnproven in ups.rules.yaml key on the self-test instead, and are the only two rules in that file that can detect this condition.

This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it.

A pack was fitted on 2026-08-28, and that closed this. A card that cannot see a pack which is present — badly seated, or faulty out of the box — emits the same five fabricated values as one sitting over an empty bay, so a healthy-looking dashboard distinguished nothing. Only a passing self-test and readings that have left the pre-fit baseline do, and both now hold: upsTestResultsSummary went 4 (aborted) to 1 (donePass), upsBatteryVoltage left 480 for a float reading that varies, and the runtime estimate no longer sits on exactly 63. The silence on UpsSelfTestFailed was deleted the same day rather than left to expire in September, so that rule is live again.

Two things outlast the fix. Stored metrics older than 2026-08-28 are the fabricated values rather than measurements, so a dashboard or query whose range crosses that date is reading fiction on one side of it. And the card's test schedule is on but unwatched. Read off the NMC on 2026-09-03, upsAdvTestDiagnosticSchedule is 8 (biweeklySinceLastTest), which is what keeps 1 from being a frozen last-known result — and as of #249 that is confirmed rather than assumed. The apc_ups module now walks 1.3.6.1.4.1.318.1.1.1.7.2 alongside the standard UPS-MIB, and two rules watch it: UpsSelfTestScheduleOff fires the moment the schedule reads unknown, at-turn-on or never, and UpsSelfTestStale fires when the last-test date has not moved in 21 days.

That closes a gap worth naming for its shape rather than its size. The missing battery pack was visible in a MIB already walked — upsTestResultsSummary = 4 needed no new OIDs — and the missing schedule would not have been. upsTestResultsSummary would have held 1 forever and every rule would have stayed correctly quiet; UpsBatteryUnproven could not have caught it either, because it matches 6 (noTestsInitiated) and this card reads 1. A healthy-looking metric with nothing underneath it is the failure this estate keeps finding, one level up each time.

Why SNMPv2c is still a weak point

The devices are polled with SNMPv2c, which transmits the community string in cleartext, and whoever can see the poll's last hop can read it off a single packet. Two mitigations are in place, and the third is decided per device rather than for the fleet:

  • Done: each device has its own community, confirmed live on all four, so one captured packet no longer grants read access to the whole fleet. The switch does still accept its own previous community as well — an accepted residual, recorded in SECURITY.md.
  • Done: SNMP is reachable only on the management VLAN, the switch-management LAN and — for the iLO — the lab segment, and only from the monitoring host.
  • SNMPv3 authPriv, per poll (ADR-0036). The iLO's poll is the one that matters: it is delivered into ImaginationLAN, where ADR-0014 puts the attack VM on purpose, so a guest there that ARP-spoofs 10.0.30.10 reads the community every minute. shiva has been polled over v3 with SHA and AES since 2026-09-20, and its SNMPv1 Request switch is off — the iLO refuses its own former community over v2c, proved with snmp-verify.sh --old the same day, so that poll is no longer readable from the lab segment. The UPS card followed on the same procedure — its profile created the same day, its poll over authPriv from 2026-09-21, its SNMPv1 access switched off the same day and its former community proved refused. Both devices ADR-0036 names are done, and neither answers a community at all now. So two of the four polls are encrypted, and the two that are not both ride on Winterfell. The firewall cannot move without losing what it is polled for: bsnmpd is the only daemon that serves the pf MIB, and pfSense writes no v3 user for it — checked on the box on 2026-09-09. The switch stays on v2c; its agent answers v3 on the wire, and whether its UI can create a user is the unchecked half. Which devices have actually moved is recorded on #85 and in the runbook.

These communities are read-only, but "read-only" on a firewall means the complete state table and interface topology. They are credentials — except on the switch, where the agent does not check a short one on GETBULK at all (SECURITY.md has the measurement). The two that stay on v2c ride on Winterfell only, where anything that can sniff is already on the segment that holds the firewall's admin UI and the monitoring host.

The switch's management UI is HTTP, and stays that way

neo serves its management UI on port 80 and nothing on 443 — no TLS listener, and no way to import a certificate. Checked against the device on 2026-09-04 and decided in ADR-0018, which gave the switch a name (neo.matrix.elysium) and closed the certificate half of #97 as unavailable rather than pending.

So the switch admin password crosses the wire in cleartext, and it is worth being precise about where: through neo itself, which is the device the password protects. A mirrored port or a foothold on the switch sees the credential to the switch. This is the same shape as the SNMP argument above and a sharper version of it, because this credential is read-write.

What holds it: the password is unique to the device, and only Hicks and Winterfell can reach 10.7.7.0/24 at all (ADR-0013). What does not hold it: anything on the device, which is now carrying its fourth firmware limit after the undeletable community row, an SNMP agent that serves GETBULK to any community of sixteen characters or fewer (measured 2026-09-12, #84), and an SNMPv3 user page it may or may not have (ADR-0036). A TLS management interface belongs in the selection criteria whenever this switch is replaced — and the replacement that has one, a MikroTik CRS326, was bought 2026-09-13 (hardware.md, #444), with delivery estimated 2026-09-23. This section changes when it is racked, not before — that date is a courier estimate rather than an arrival, and the swap takes the whole house offline and shares a rack visit, so it buys a window to pick rather than a date this closes on.

Hardening applied to the stack

  • Anonymous Grafana access disabled; sign-up disabled; admin password from SOPS.

  • Every service except Alloy and speedtest-tracker runs as a non-root uid. speedtest-tracker's image is built on s6-overlay, which must start as root and as PID 1; s6 then runs nginx, php-fpm and the test itself as the operator's ${RENDER_UID}. Its capabilities are cut to the six that drop needs (#914, measured in compose.yaml), and no-new-privileges holds. Prometheus and blackbox-exporter run as nobody (65534), Loki and Grafana as their own unprivileged UIDs (10001, 472), and Alertmanager and snmp-exporter as the operator's ${RENDER_UID} because each mounts a 0600 file that scripts/render-config.sh wrote. blackbox-exporter needs its user: written out because its image sets none — it was uid 0 until #330, which is how it went unnoticed for as long as it did. Alloy is uid 0 on purpose and is covered separately below: it reads the host's logs, and the capability drop rather than the uid is what bounds it.

  • speedtest-tracker saturates the WAN for ~20 seconds every 30 minutes, to Ookla's servers, from VLAN 99 — the same deliberate egress path as the probes below, and far larger: a test at the port's ~940 Mbit/s moves around a gigabyte each way (#914). What leaves is test traffic; what Ookla learns is the public address and its speed. Its UI on :8443 wants the app's own login, but /prometheus and /api/healthcheck on the same port answer without one, and the first discloses the ISP's name and the latest result — nothing a client on a segment that can reach 10.0.99.20 could not measure for itself.

  • Outbound probes leave VLAN 99 for the public internet, and that is a deliberate new egress path rather than a side effect (#166). blackbox-exporter makes TCP connects to 1.1.1.1:443 and 8.8.8.8:53 every 30 seconds so the estate can answer "is the internet slow, and is it us or the ISP". They are connects only — nothing is sent, nothing about the estate is disclosed beyond the fact that this address exists, which the ISP knows anyway. tcp_connect rather than an ICMP prober specifically so that blackbox-exporter keeps cap_drop: [ALL]: an icmp module needs CAP_NET_RAW, and handing one container back a capability the rest of the stack just gave up is a poor trade for a measurement TCP already provides.

  • Miniflux polls the internet from trinity, on the same kind of path (#147, ADR-0057). Every subscribed feed is fetched about once an hour, and that is the service working. What it may not do is fetch inward: FETCHER_ALLOW_PRIVATE_NETWORKS=0 refuses every private, loopback and link-local address after DNS, measured against a container by name and against localhost. That flag is the only thing between a feed URL and the rest of Winterfell, because hosts on one segment do not cross the firewall to reach each other. It is written out in compose.yaml rather than defaulted for that reason. Its REST API, which takes the admin's password without the login page, is off.

  • Mealie fetches the page a user pastes, from trinity (#146, ADR-0060). It fetches on request, not on a timer. It does not fetch inward: its safehttp transport refuses private, loopback, link-local, reserved and CGNAT addresses after DNS. That was measured against a recipe page on the compose network, by name and by address, and against 127.0.0.1, 10.0.99.1, 10.0.99.20:9090 and 169.254.169.254. None of those requests arrived. HTTP_ALLOW_LIST is what would open it, so it is written out empty in compose.yaml.

  • snmp-exporter is never published to a host interface — it is reachable only on the compose network.

  • Alertmanager binds to 127.0.0.1 only. It is unauthenticated, and a silence is how monitoring gets switched off — quietly, since the record lives in the system being switched off. Nothing off-host used the port; silences are reached through Grafana.

  • Prometheus and Loki bind to 127.0.0.1 only, since #182. Their off-host clients (three agents, Homepage and Home Assistant) reach them through the ingest proxy on 10.0.99.20, which wants a token and serves each token one role. See ADR-0012 for which ports are published and ADR-0067 for what a client must prove.

  • The Alloy debug UI binds to 127.0.0.1 only.

  • The Docker socket is no longer mounted into Alloy (#193). It was, marked :ro, which was worth less than it looked: read-only applies to the socket file, not to the API behind it, and anything that can talk to that API can start a container with the host filesystem mounted read-write. Alloy now reaches the API through docker-socket-proxy, which allowlists the reads its three components need — containers, images, info, version, events and networks — and refuses everything else. Verified rather than assumed: POST /containers/create through the proxy returns 403 Forbidden, while GET /containers/json works.

    This moves the trust boundary rather than removing it. The proxy container holds the socket now. What it buys is that Alloy — the component with a network listener, a read-only mount of the whole rootfs and by far the largest attack surface here — no longer has a path to POST. It is not "the socket is now safe".

    The lab's three Docker guests got the same proxy in #836. Until then, alexander, odin and fenrir each handed Alloy the socket itself, and this paragraph named only oracle, so it was wrong about three hosts. Two of those guests hold keys: alexander the lab's only age key, and odin the SOC's key and Velociraptor's CA. check_compose_health.py now fails on any docker.sock mount in a compose file outside the proxy image, which is how they would have been caught.

    The proxy was not the whole fix, and review of #836 said why. Every Alloy also mounts the host's / at /rootfs:ro for node metrics, and that carries /rootfs/run/docker.sock with it. A read-only mount does not stop connect() on a socket, and Alloy runs as uid 0, which owns it, so any Alloy, the estate's behind #193's proxy included, could reach the full API around the proxy with no capability at all. /rootfs/run is now an empty tmpfs in all four. smaug's node-exporter gets the same mask at /host/run, although as uid 65534 it could not open the socket anyway. The check now fails three shapes:

    • the socket itself, under either spelling, /var/run or /run;
    • its directory;
    • the host's / with no mask over <target>/run.

    oracle's agent has its own proxy too. It was the last Alloy holding the socket. It is started by deploy-agent.sh as a docker run, not from a compose file, so no compose check could see it. The script now starts alloy-socket-proxy beside it, with the same image and allowlist, on a private network. Its /rootfs/run is an empty tmpfs, because /:/rootfs:ro otherwise carries the socket past the proxy: a read-only mount does not stop connect(), and Alloy is the socket's owner. check_image_pins.py refuses any docker run in the repository, other than the proxy's (traced by image), that reaches the socket. It looks for three things:

    • the socket under either spelling;
    • its directory;
    • the host's / with no --tmpfs over <target>/run. docker.alloy still falls back to the socket when DOCKER_API is unset, so an agent deployed before this keeps working until its next deploy-agent.sh run.

    "Refuses everything else" was not true of /containers until 2026-10-10. The image's rule for CONTAINERS=1 is a prefix match on /containers, so it also allowed four GETs that POST=0 does not touch: archive, which returns any file from any container, export, which returns a whole container filesystem, top and changes (Tecnativa/docker-socket-proxy#182, unfixed upstream). The daemon serves them as root, so a client trusted only to list containers could read through the proxy what cap_drop keeps it from reading through /rootfs. All four answered 200 on prometheus.

    • Every proxy now runs the repository's HAProxy config, stacks/observability/docker-socket-proxy/haproxy.cfg. It is upstream's template with that one rule replaced by the three /containers paths the clients send, read from three days of the proxy's own request log. That covers the six compose stacks and deploy-agent.sh's alloy-socket-proxy.
    • check_socket_proxy.sh boots the pinned image on it in CI. Alloy's paths must answer, and archive, export, top, changes, stats, attach and the .. and encoded-slash spellings must be refused by the proxy itself. It also fails when an image bump changes upstream's template under the committed copy.
    • check_compose_health.py fails a proxy that does not mount the config or is not started on it, because either one alone falls back to the image's template without any visible change.

  • Alloy holds no capabilities. It runs as uid 0 with cap_drop: [ALL] and no-new-privileges, so root inside it is subject to file permissions like any other user, and joins only the group that owns /var/log/syslog so the auth and syslog sources stay readable (#188). scripts/deploy-agent.sh applies the same flags to every Docker host it deploys to, so oracle's agent is no longer the privileged copy it was until #88; on Saruman the native package runs as its own unprivileged alloy user.

  • Every service runs with a read-only root filesystem (#186). Writable state is confined to the named volumes and, where a process genuinely needs scratch space, to a sized tmpfs: 64 MiB on Grafana, which extracts plugins into /tmp, and 32 + 16 MiB on Alloy for /tmp and /root/.cache. The sizes are set rather than defaulted because a tmpfs is memory, and the default is half of host RAM — an unbounded one would be a second memory budget beside the mem_limit each service already has.

    This was established by running, not by reading: a missing tmpfs is a crash loop that restart: unless-stopped retries forever, so each service was booted read-only from its pinned digest with its real mounts before the line was written. Grafana got the full soak the issue asked for — login, all three datasources, all eight dashboards fetched by uid, an annotation written and a folder created. Alloy is included too: #186 proposed skipping it as performative next to privileged: true, and that reason left with #188.

  • The other six services hold no capabilities either. Each carries cap_drop: [ALL] and no-new-privileges:true (#187). For five of them this is belt and braces and the honest description matters: they already ran non-root with an effective set of zero, so what the drop clears is the bounding set — the capabilities a process could regain, not any it held. What that buys is that a future image bump shipping a setuid or file-capability binary cannot be used to gain one, which no-new-privileges independently forecloses. None of the six pinned images contains such a binary today; each container's filesystem was exported and scanned for the setuid and setgid bits. blackbox-exporter was the exception when #187 landed: it was running as uid 0 with the full effective set, so there the drop removed something real. #330 then gave it a non-root uid as well, which puts it in the same belt-and-braces position as the other five — the position to be in, and not a reason to drop either setting.

  • Every service runs under a real init (init: true) and a chosen task ceiling (pids_limit, 512; 1024 for Alloy) rather than the inherited systemd default of 9056. This was not theoretical: Grafana's https healthcheck was leaking two unreaped ssl_client children every 30 seconds and would have exhausted the inherited ceiling about three days after each start (#71).

  • Prometheus carries a byte ceiling as well as a time one, so a change that quietly multiplies the series count cannot consume the disk unnoticed.

  • All images are pinned to explicit versions, so an upstream compromise cannot arrive silently via :latest. Dependabot proposes the bumps; CI validates them.

  • Grafana telemetry and update checks disabled.

  • Stirling-PDF's analytics, PostHog, Scarf and update checks disabled, and its heap dump on OOM too, because a dump would put the document being processed on a disk (ADR-0063).

Alloy no longer runs privileged: true. It never needed it: cgroup: host is what makes cAdvisor see the host's cgroups, and dropping every capability changed no container metric and cost six unused series — node_rapl_*_joules_total and node_cpu_{core,package}_throttles_total, which read root-only sysfs and which nothing here references. Before that change Alloy ran as uid 0 with the full capability set, a read-only mount of /, and could therefore read ~/.config/sops/age/keys.txt directly. It can no longer.

What remains is the Docker socket, and it is the larger half. Read access to that API is enough to create a container with / mounted read-write, which is root on the host and the age key with it — so the paragraph in SECURITY.md saying file permissions are all that protect the plaintext artefacts is true of every process on the host except a compromised Alloy. Putting the socket behind a proxy that permits only the handful of GETs cAdvisor and the log discovery actually use is tracked separately; the privilege reduction above is defence in depth, not a closed door.

A writable directory inside Alloy's read scope

Scheduling the maintenance jobs (#77) put /var/lib/node_exporter/textfile_collector on the host, owned by the deploying user and read by Alloy's textfile collector through the /rootfs mount that already existed. It is 0755 with 0644 files, and it has to be: Alloy runs as root with every capability dropped, so it obeys the mode like anyone else and a 0600 file would simply be invisible to it.

The contents are four numeric gauges per job — timestamps, a duration and an exit code. Nothing secret is written there, and nothing decrypted passes through it. What it does create is a path from write access as that user to arbitrary metric names and label values in Prometheus, since the collector will parse whatever it finds. Two things bound that: only the deploying user can write, and scripts/run-scheduled.sh constrains the one operator-supplied field to ^[a-z][a-z0-9-]{0,30}$ before it becomes either a filename or a label value.

Anyone who can write there can already run the jobs themselves, so this adds no privilege — but it is a new file-backed input to the metrics pipeline, and that is worth stating rather than discovering.

What this repository deliberately does not publish

Being able to describe a network precisely is useful; publishing a complete fingerprint of a house is not. Withheld on purpose:

  • Full MAC addresses. Truncated to the OUI, which keeps the useful information (vendor, and therefore what the device is) and drops the unique identifier. Full MACs enable device tracking and, on some networks, MAC-based access control bypass. This is also why the unknown-device rules hold no allowlist: the set of known devices is whatever Loki has seen in the last seven days, so nothing in this repository has to enumerate them.
  • Owner-linked device names. Personal devices are listed by role (laptop-01) rather than by person, and a child's bedroom is not labelled.
  • Camera-to-room mapping. Knowing there are seven cameras is fine. Knowing which one covers which door is a physical-security detail.
  • The WAN address, firewall rule bodies, and Wi-Fi configuration.
  • The WireGuard endpoint and its listen port, for the same reason and with the same instinct: a repository that withholds the WAN address and then publishes the port a VPN answers on has withheld nothing. The design, the peer subnet and the rules are published (ADR-0042); where to send a packet is not. That now includes the dynamic DNS hostname, its provider, the account and the update token (ADR-0044): naming the provider narrows the search for the name to one zone, and the token lives in config.xml, which this repository already keeps out of the tree.

The public IP was already redacted in the original inventory — the rest of this is the same instinct applied consistently.

Rack patch-cable colours are published, and that is a deliberate exception worth defending rather than an oversight. It is the same shape of information as the camera-to-room mapping above — a physical-security detail — but not the same reach. A room mapping is useful remotely: you learn which camera covers which door straight off this page, without ever approaching the house. A cable colour is useful only to someone already standing at the rack, who can see the cables, read the switch port labels and reach the firewall's console port regardless. It tells an attacker nothing their position has not already given them, and it tells a maintainer a great deal. See ADR-0009.