From 3955ceda828f90270e94c4a453fc21c6d9a0b829 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 10 Oct 2026 15:28:19 -0700 Subject: [PATCH 1/3] docs(lab): garuda's SOC enrolment as run, phase 2 Runbook section 11 records section 12 as run on 2026-10-10: staging on odin, both packages verified on garuda, the Wazuh agent enrolled as 009 Active with the password on standard input, the Velociraptor client connected (7 connections server-side), and WazuhAgentsNotConnected expecting garuda and quiet. Changelog entry for the day. Refs #921. Co-Authored-By: Claude Opus 5.5 --- docs/changelog.md | 12 ++++++ .../runbooks/build-the-analyst-workstation.md | 42 +++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/docs/changelog.md b/docs/changelog.md index 2cf1ae7c..abef5fbc 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,18 @@ docstring gives: it is a record, not a claim about now. ## 2026-10-10 +- **`garuda` is enrolled in `odin`'s Wazuh and Velociraptor** + ([#921](https://github.com/Gerrrt/HomeLab/issues/921) phase 2, + [ADR-0092](adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md), + [`build-the-analyst-workstation.md`](runbooks/build-the-analyst-workstation.md) + §11). It is the first Linux guest in either. + - **Wazuh.** Agent 4.14.8, enrolled as agent 009, Active. The password went + in on standard input, and its hash matched `odin`'s. + - **Velociraptor.** Client 0.77.3, from a `.deb` the server built. It is + connected, and the server counts 7 client connections. + - **The lab.** `WazuhAgentsNotConnected` now expects `garuda` and is + quiet. + - **`garuda` enrols in `odin`'s Wazuh and Velociraptor: authored** ([#921](https://github.com/Gerrrt/HomeLab/issues/921) phase 2, [ADR-0092](adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)). diff --git a/docs/runbooks/build-the-analyst-workstation.md b/docs/runbooks/build-the-analyst-workstation.md index e8818eed..9f4e4197 100644 --- a/docs/runbooks/build-the-analyst-workstation.md +++ b/docs/runbooks/build-the-analyst-workstation.md @@ -522,6 +522,48 @@ the console password. - **Not run: the desktop line.** It needs the console from Hicks, and the console password Garrett sets at a prompt (§4). `user@968.service` (`lightdm`) still fails on every boot. +- **§12, phase 2, 2026-10-10** + ([#1163](https://github.com/Gerrrt/HomeLab/pull/1163), + [ADR-0092](../adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)). + - **`odin`:** + - The checkout was pulled to `main`, and `make up STACK=soc` left all + seven services healthy. + - `stage-agent-msis.sh` ran as root through the guest agent, because + `barnabas` has no passwordless sudo. It re-verified both MSIs, matched + the Wazuh `.deb` to its pin, and built + `velociraptor-client_0.77.3_amd64.deb` (sha256 + `b89a412865058fc6b3025b78791b59215bae087c445ffdd9dd0e05df4ef5b1cc`). + - **Fetching.** `8448` answered `garuda` (200; it was 403 before the + reload). + - Both packages verified on `garuda`: the Wazuh one against the pin, the + Velociraptor one against the hash read from `odin` over SSH. + - **Wazuh.** + - `dpkg -i` with `WAZUH_MANAGER`, `WAZUH_AGENT_NAME=garuda` and + `WAZUH_AGENT_GROUP=default`, and no password. + - The password, `LAB_WAZUH_REGISTRATION_PASSWORD` from `phoenix.env`, went + into `authd.pass` on standard input. Its hash matched `odin`'s + `authd.pass` before the agent started. + - The agent logged "Valid key received" and connected to `1514`. `odin` + listed it as **009, `garuda`, Active**. + - `authd.pass` was then removed. + - **Velociraptor.** + - `dpkg -i` gave `velociraptor_client` enabled and active, with an + established connection to `odin:8000`. + - The server's `client_comms_current_connections` read 7 (the six and + `garuda`), and `frontend_enroll_response` counted 1, `garuda`'s. + - A `query` against `clients()` from a second process on the server + returned nothing, and was not pursued. The GUI check from Hicks is + Garrett's. + - **The lab.** `alexander` was pulled to `main` (the rule change), and + `make up STACK=lab` reloaded Prometheus. + - `homelab_wazuh_agent_active{agent="garuda"}` first read 0: the + collector had run six seconds after the agent connected. It read 1 at + the next run. + - `WazuhAgentsNotConnected` was briefly pending for `garuda`, then + inactive. + - **Cleanup.** The downloaded packages were deleted from `garuda`, since + the Velociraptor `.deb` carries the enrolment nonce. Installed: + `wazuh-agent 4.14.8-1` and `velociraptor-client 0.77.3`. - **Agents.** `guest-exec` timed out on both `garuda` (162) and `phoenix` (170) after commands carrying non-ASCII text or heredocs. `scripts/qga-resync.py` fixed each at once. Keep agent commands ASCII, and From b6d099166cf5b242251df6a860c8c9589093eb06 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 14 Oct 2026 08:50:46 -0700 Subject: [PATCH 2/3] =?UTF-8?q?docs(runbook):=20record=20the=20Kali=20temp?= =?UTF-8?q?late=20epoch-expiry=20fix=20in=20analyst=20=C2=A711=20as-run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the degraded-state follow-up to §11: root cause (Kali's install layer ships ~14 service accounts expired at the epoch; only lightdm surfaces via user@.service), the live fix on garuda, the preseed late_command fix, and the 910 rebuild + smoke + fresh-clone verification. Corrects the two earlier notes that said user@968 still fails. The 902 sibling check is owed. Refs #921, #1194, #1195, #1196 Co-Authored-By: Claude Opus 4.8 --- .../runbooks/build-the-analyst-workstation.md | 36 ++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/docs/runbooks/build-the-analyst-workstation.md b/docs/runbooks/build-the-analyst-workstation.md index bed32fd0..6f67d1e4 100644 --- a/docs/runbooks/build-the-analyst-workstation.md +++ b/docs/runbooks/build-the-analyst-workstation.md @@ -472,12 +472,15 @@ the console password. - swap is `/swapfile` (4 GB); - cloud-init reports `done`; - SSH and `core doctor` work (exit 0). - - **One failed unit, on every boot before and after the fix:** + - **One failed unit, on every boot (fixed 2026-10-14, see below):** `user@968.service`, for `lightdm`. - The package creates that account with an expiry of 1970-01-02, so PAM refuses its user manager, and the system reads `degraded`. - `lightdm` itself is `active`. - - §9's desktop check from Hicks decides whether it matters. + - Resolved by the template fix dated 2026-10-14 below + ([#1194](https://github.com/Gerrrt/HomeLab/issues/1194), PR + [#1195](https://github.com/Gerrrt/HomeLab/pull/1195)); it was not + lightdm-only, but ~14 Kali service accounts. - **§6, 2026-10-10** ([#1144](https://github.com/Gerrrt/HomeLab/pull/1144)). - **garuda's tools:** `age` 1.3.2 from Kali, and `sops` 3.9.4's release binary (checksum OK) at `alexander`'s version. The repository is at @@ -520,8 +523,33 @@ the console password. 2.5 minutes everything above passed again with nothing typed, including 500 journal lines in Loki over 3 minutes. - **Not run: the desktop line.** It needs the console from Hicks, and the - console password Garrett sets at a prompt (§4). `user@968.service` - (`lightdm`) still fails on every boot. + console password Garrett sets at a prompt (§4). At the time, `user@968.service` + (`lightdm`) failed on every boot; that `degraded` state was fixed on + 2026-10-14 (see below). +- **§9 follow-up — the `degraded` state, 2026-10-14** + ([#1194](https://github.com/Gerrrt/HomeLab/issues/1194), PR + [#1195](https://github.com/Gerrrt/HomeLab/pull/1195)). + - **Root cause:** not lightdm-specific. Kali's package postinsts ship ~14 + service accounts (`lightdm`, `_gvm`, `inetsim`, `nm-*`, `dnsmasq`, + `tcpdump`, `mysql`, `pcscd`, `rtkit`, `Debian-snmp`, `systemd-network`, + `systemd-timesync`, `_dhcpcd`) with their shadow account-expiry at the + epoch (field 8 == `1`). `lightdm` is the only one `systemd` starts a + `user@.service` for, so it is the only one whose expired account + reaches PAM and fails `224/PAM`; the rest are latent. Not from our build + (`grep` of `packer/`, `scripts/`, `tofu/`, `stacks/` for + `chage`/`usermod -e`/`--expire` finds nothing) — it comes from Kali's + install layer. + - **Fixed live on garuda:** `chage -E -1` on every UID<1000 account at the + epoch, then `systemctl start user@968.service` → `running`, 0 failed units. + - **Fixed in the template:** a `late_command` step in + `packer/kali/preseed.cfg.pkrtpl` clears the epoch expiry at install time, + so fresh clones of 910 (and 902) are not born `degraded`. + - **Verified:** 910 rebuilt (`packer build -force`, `BUILD_EXIT=0`), + `packer-smoke.sh 910` passed, and a throwaway full-clone booted + `system: running` with 0 failed units and 0 accounts at the epoch. + - **Still owed:** the same check on 902 (ifrit's Kali, same preseed) when it + is next built — [#1196](https://github.com/Gerrrt/HomeLab/issues/1196), + tied to [#790](https://github.com/Gerrrt/HomeLab/issues/790). - **§12, phase 2, 2026-10-10** ([#1163](https://github.com/Gerrrt/HomeLab/pull/1163), [ADR-0092](../adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)). From e6a9e2563644efcfa1c0f304ce037a4d91ed0d54 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 14 Oct 2026 09:27:53 -0700 Subject: [PATCH 3/3] docs(runbook): scope the 902 guarantee and mark the grep as pre-fix (review) Address review on #1197: the late_command fix is only realised on a rebuilt template, so limit the "not born degraded" claim to a rebuilt 910 and note 902 keeps the defect until rebuilt (#1196); phrase the grep-finds-nothing evidence as a pre-fix observation, since PR #1195 added a chage to the preseed. Co-Authored-By: Claude Opus 4.8 --- docs/runbooks/build-the-analyst-workstation.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/runbooks/build-the-analyst-workstation.md b/docs/runbooks/build-the-analyst-workstation.md index 6f67d1e4..3aa8c336 100644 --- a/docs/runbooks/build-the-analyst-workstation.md +++ b/docs/runbooks/build-the-analyst-workstation.md @@ -535,15 +535,18 @@ the console password. `systemd-timesync`, `_dhcpcd`) with their shadow account-expiry at the epoch (field 8 == `1`). `lightdm` is the only one `systemd` starts a `user@.service` for, so it is the only one whose expired account - reaches PAM and fails `224/PAM`; the rest are latent. Not from our build - (`grep` of `packer/`, `scripts/`, `tofu/`, `stacks/` for - `chage`/`usermod -e`/`--expire` finds nothing) — it comes from Kali's - install layer. + reaches PAM and fails `224/PAM`; the rest are latent. Not from our build: + at diagnosis (before this fix) a `grep` of `packer/`, `scripts/`, `tofu/`, + `stacks/` for `chage`/`usermod -e`/`--expire` found nothing — it comes from + Kali's install layer. (The `late_command` added below now puts a `chage` in + the preseed, so that grep matches afterwards.) - **Fixed live on garuda:** `chage -E -1` on every UID<1000 account at the epoch, then `systemctl start user@968.service` → `running`, 0 failed units. - **Fixed in the template:** a `late_command` step in `packer/kali/preseed.cfg.pkrtpl` clears the epoch expiry at install time, - so fresh clones of 910 (and 902) are not born `degraded`. + so fresh clones of a rebuilt 910 are not born `degraded`. 902 (ifrit's + Kali) shares the same preseed but keeps the defect until it too is rebuilt + ([#1196](https://github.com/Gerrrt/HomeLab/issues/1196)). - **Verified:** 910 rebuilt (`packer build -force`, `BUILD_EXIT=0`), `packer-smoke.sh 910` passed, and a throwaway full-clone booted `system: running` with 0 failed units and 0 accounts at the epoch.