zbctl

Safe LAB ⇄ HPC mode switching for the nodes of a small Zero-Trust Beowulf cluster.

A worker node in a lab is both a student desktop (GDM, DHCP, campus network) and an HPC compute node (Slurm, isolated VLAN, static address). zbctl moves one node between those two identities transactionally: it verifies the real network first, changes services only after that, verifies again, rolls back on any failure, and records everything in an audit log. It never claims a mode it could not prove.

	LAB mode	HPC mode
Network	Student_Nw (VLAN 100), DHCP	HPC (VLAN 10), static address
GDM (desktop)	running	stopped
slurmd	stopped	running, registered with the controller

The physical switch port decides the network. zbctl never configures the switch. The operator moves the cable, then runs sudo zbctl mode lab|hpc; zbctl verifies what is really connected.

Why it is built this way

• Observation over belief. The state file is only a record of what zbctl did. status and every decision use the live system; a stale or corrupt record can make the reported state worse, never better.
• Transactional transitions. PRECHECK → PREPARE → NETWORK_APPLY → NETWORK_VERIFY → SERVICE_APPLY → SERVICE_VERIFY → CLUSTER_VERIFY → COMMIT. Any failure restores the previous mode; if that fails too the node is marked DEGRADED, never READY.
• Wrong-cable protection. mode hpc is rejected up front if the Student_Nw gateway is answering on the interface (and vice versa) — nothing is changed.
• No bypass. There is deliberately no --force; unknown flags are rejected.
• Small attack surface. Local only, root required for changes, no shell (argv arrays), a fixed allowlist of executables, secrets redacted from audit/diagnostics output, advisory locking against concurrent runs.
• Idempotent. Re-running mode hpc on a healthy HPC node verifies and exits without touching anything.

flowchart LR
    A[Cable moved by operator] --> B[sudo zbctl mode hpc]
    B --> C{PRECHECK<br/>wrong network?}
    C -- yes --> X[Reject, no changes]
    C -- no --> D[Apply + verify network]
    D -- fail --> R[Rollback]
    D -- ok --> E[Stop GDM / start slurmd]
    E --> F[Verify services + Slurm registration + health]
    F -- fail --> R
    F -- ok --> G[COMMIT + audit]
    R --> H{Previous mode restored?}
    H -- yes --> I[FAILED, previous mode kept]
    H -- no --> J[DEGRADED]

Loading

Use

zbctl config validate            # check /etc/zbctl/zbctl.yaml
zbctl status                     # node, mode, state, network, services, Slurm, health, last transition
zbctl network | services | health | diagnostics     # read-only, no root needed
sudo zbctl mode lab              # after cabling to a Student_Nw port
sudo zbctl mode hpc              # after cabling to an HPC port
zbctl version

Options: --config, --state-dir, --log-dir, --json. Exit codes: 0 ok · 1 failed/rejected · 2 usage · 3 unhealthy/DEGRADED.

Install

Download a release archive (linux/amd64 or linux/arm64), verify it against SHA256SUMS, then:

tar xzf zbctl-<version>-linux-amd64.tar.gz && cd zbctl-<version>
sudo ./scripts/install.sh            # installs /usr/bin/zbctl, example config, logrotate, systemd timer files
sudoedit /etc/zbctl/zbctl.yaml       # values measured on the node: docs/ENVIRONMENT.md
zbctl config validate && zbctl diagnostics

Or build from source (Go ≥ 1.22, no third-party modules): make → bin/zbctl (static, -trimpath, reproducible). Details: docs/INSTALL.md · docs/PLATFORMS.md.

Project status — honest edition

Current release line: 0.x (alpha). See CHANGELOG.md and docs/ROADMAP.md.

Area	Status
Config, state machine, adapters, audit, CLI — unit tests incl. all failure paths	✅ passing in CI
Installer, config validate, status, network, services, health, diagnostics on a real Ubuntu 24.04 worker	✅ verified
Distro handling: GDM gdm3 vs gdm, NetworkManager-managed check	✅ verified on Ubuntu · ⏳ AlmaLinux host test pending
mode hpc / mode lab end-to-end on real hardware (needs a working Slurm worker)	⏳ not yet run
Slurm registration against a real controller	⏳ pending (cluster Slurm not installed yet)
Physical-failure tests (wrong cable, power loss mid-transition)	⏳ pending

Nothing in the ⏳ rows should be assumed to work until a release note says it was tested.

Repository layout

cmd/zbctl            entry point
internal/cli         commands, status / health / diagnostics output
internal/config      strict YAML-subset loader + validation (no external deps)
internal/mode        the transactional state machine (depends only on interfaces)
internal/network     NetworkManager adapter, `ip -j` parsing
internal/services    systemd adapter        internal/slurm     scontrol adapter
internal/health      observation + health   internal/osinfo    distro detection
internal/audit, state, system, fail, domain
configs/ systemd/ scripts/                   example config, units, install scripts
docs/                                        install, platforms, environment, testing, 20 ADRs

Documentation

Installation · Supported platforms · Environment checklist · Testing & failure model · Architecture decisions · Releasing · Roadmap · Security policy · Contributing

License

MIT