feat(tenant): one SSH port for all tenants through sshpiper (#80)

README.md

@@ -147,7 +147,7 @@ All modules are idempotent and support `--dry-run`.
 | `storage` | RAID/NVMe directory setup, symlinks |
 | `network` | UFW firewall, port rules (SSH port always allowed) |
 | `monitoring` | Opt-in Prometheus node exporter and DCGM GPU exporter, plus an optional Prometheus/Alertmanager/Grafana hub that reloads on rule changes |
-| `tenant` | Opt-in host prep for `rootfiles tenant`: GPU device guard timer, daily tenant password sync, fstrim, operator sudoers rule, tenant volume mount units |
+| `tenant` | Opt-in host prep for `rootfiles tenant`: GPU device guard timer, tenant password and key sync, fstrim, operator sudoers rule, tenant volume mount units, optional sshpiper single-port SSH routing |
 
 Modules run in this order; `users` precedes `ssh` so declared operators exist before password auth is turned off.
 

docs/releases/v0.17.0.md

@@ -0,0 +1,13 @@
+Tenant containers can now share one SSH port: with sshpiper enabled, `ssh -p 2222 <tenant>.<user>@host` reaches any tenant.
+
+### New
+- `modules.tenant.sshpiper` installs a pinned, checksum-verified sshpiper as `rootfiles-sshpiper.service` and routes `<tenant>.<user>` to each tenant's own sshd. Passwords are checked by the tenant's sshd, which also enforces the change of a temporary password; keys in the user's `~/.ssh/authorized_keys` inside the container work after the next sync. Repeated failed logins ban the address for an hour. `rootfiles tenant status` and `create` print the routed command. With `modules.network.ufw` the port is admitted in the firewall; an active UFW without it, or a port held by another process, stops `apply` before sshpiper starts. (#81)
+
+### Improved
+- `rootfiles tenant sync` runs every 2 minutes instead of daily and no longer takes the global lock, so it cannot fail while `rootfiles apply` runs. (#81)
+- Reading accounts and keys from a tenant container is limited in size and time, so a tenant cannot stall a sync or an import. (#81)
+
+### Action needed
+- To use single-port SSH, set `modules.tenant.sshpiper.enabled: true` and run `sudo rootfiles apply --module tenant` (or `rootfiles fleet apply`). Existing tenants need `sudo rootfiles tenant apply <tenant>` and `sudo rootfiles tenant restart <tenant>` once before key logins work through the shared port. (#81)
+
+**Full changelog**: https://github.com/STAIxBWLB/rootfiles/compare/v0.16.0...v0.17.0

docs/specs/2026-10-20-tenant-containers-design.md

@@ -161,9 +161,10 @@ Implementation starts only on the owner's go-ahead.
 - Config lives under `modules.tenant` (`enabled`, `root`, `ops_group`). The default operator group is `rootfiles-tenant-ops`.
 - The container mounts the volume subdirectories with `--mount type=bind` instead of a named Docker volume `tenant-<t>`. A bind mount fails when its source is missing, so a container cannot start on the bare mount point of an unmounted volume and write past the quota. The named volume added nothing else.
 - The volume image is mounted by a systemd `.mount` unit with `loop,discard,nodev,nosuid`, `Before=docker.service` and `WantedBy=docker.service`. `nosuid,nodev` and a root-only `<root>/<t>` keep host users away from setuid files a tenant leaves in its volume.
-- On a container's first boot the stored password hashes replace whatever the image ships. On later boots they only seed accounts without a usable password, so a password changed inside a container survives a plain restart even before the daily sync.
+- On a container's first boot the stored password hashes replace whatever the image ships. On later boots they only seed accounts without a usable password, so a password changed inside a container survives a plain restart even before the next sync.
 - Tenant names cannot end in `-prev`, the suffix of rollback containers. Containers carry a `rootfiles.tenant.gpus` label; a failed apply returns the GPU ledger to the GPUs of the container left running.
-- The daily sync is `rootfiles tenant sync`, run by `rootfiles-tenant-sync.timer`. `tenant list` prints tenant names.
+- The sync is `rootfiles tenant sync`, run by `rootfiles-tenant-sync.timer` (every 2 minutes since #80, without the global lock). `tenant list` prints tenant names.
+- Single-port SSH through sshpiper followed in #80; its design is in that issue and in `docs/tenant.md`.
 - A recreate refuses while `tenant-<t>-prev` still exists, so an earlier rollback copy is never discarded silently.
 - `tenant ops` refuses every flag and clears `ROOTFILES_*` variables before loading the host's applied config, because the sudoers wildcard would otherwise let an operator point `--config` at a spec they control.
 - Import requires the legacy container to be running (accounts, hashes and the overlay merged directory are read from it) and keeps the tenant's passwords by copying the hashes.

docs/tenant.md

@@ -21,13 +21,38 @@ sudo rootfiles apply --module tenant
 The module installs:
 
 - `rootfiles-gpu-guard.timer`: every minute, restores GPU access of running `--gpus` containers whose device filter a `systemctl daemon-reload` rebuilt without their `/dev/nvidia*` nodes (`systemctl set-property --runtime ... DeviceAllow`). Tenant containers use the nvidia runtime and do not need it; it covers the other GPU containers on the host.
-- `rootfiles-tenant-sync.timer`: daily `rootfiles tenant sync`, which saves the password hashes of running tenants.
+- `rootfiles-tenant-sync.timer`: every 2 minutes, `rootfiles tenant sync` saves the password hashes of running tenants and refreshes their sshpiper keys and routes. It needs root but not the global lock, so it never fails because an `apply` is running.
 - `fstrim.timer`, so space deleted inside a volume returns to the host.
 - `/etc/sudoers.d/rootfiles-tenant-ops`: `%<ops_group> ALL=(root) NOPASSWD: /usr/local/bin/rootfiles tenant ops *`, validated with `visudo`.
 - A mount unit per existing tenant volume (rewritten after an OS reinstall).
 
 GPU tenants need the `nvidia` module (Docker's `nvidia` runtime).
 
+## One SSH port for every tenant (sshpiper)
+
+```yaml
+modules:
+  tenant:
+    sshpiper:
+      enabled: true
+      port: 2222       # default; must not be the host's SSH port
+      # address: 0.0.0.0, version: v1.6.1 (pinned; the archive SHA-256 is checked)
+```
+
+`sudo rootfiles apply --module tenant` installs sshpiperd with its yaml and failtoban plugins under `/usr/local/lib/rootfiles/sshpiper/<version>` and runs `rootfiles-sshpiper.service`. Users then reach any tenant through one port by naming the tenant in the user name:
+
+```bash
+ssh -p 2222 est.dev@dgx02      # user dev of tenant est
+```
+
+- Passwords go to the tenant's own sshd, which checks them and enforces the change of a temporary password.
+- Keys: sshpiper checks the client key against the user's `~/.ssh/authorized_keys` inside the container (copied by the 2-minute sync; only plain key lines, options are dropped), then logs in to the container with a host-only key. A tenant created before sshpiper was enabled needs `rootfiles tenant apply <t>` and `rootfiles tenant restart <t>` once before key logins work.
+- sshpiper checks each container's host key against the keys rootfiles stores. Clients see sshpiper's own host key on port 2222.
+- failtoban bans an address for an hour after 5 failed logins.
+- Firewall: with `modules.network.ufw`, the network module admits the port. If UFW is active without that, or its state cannot be read, `check` reports the port as blocked and `apply` stops before starting sshpiper. A port held by another process is refused the same way, also when changing the port of a running sshpiper.
+- The direct per-tenant ports keep working. Disabling sshpiper removes the service and the route files.
+- `rootfiles tenant status` and `create` print the routed command for each account.
+
 ## Create a tenant
 
 ```bash

internal/cli/runtime.go

@@ -37,7 +37,7 @@ var mutatingCommands = [][]string{
 	{"tenant", "create"}, {"tenant", "apply"}, {"tenant", "user", "add"},
 	{"tenant", "passwd"}, {"tenant", "reset"}, {"tenant", "gpu"}, {"tenant", "image"},
 	{"tenant", "port"}, {"tenant", "volume", "resize"}, {"tenant", "restart"},
-	{"tenant", "import"}, {"tenant", "ops"}, {"tenant", "sync"},
+	{"tenant", "import"}, {"tenant", "ops"},
 }
 
 func markMutating(root *cobra.Command) {

internal/cli/tenant.go

@@ -195,6 +195,9 @@ func newTenantCreateCmd() *cobra.Command {
 						return err
 					}
 				}
+				for _, r := range m.Routes(s) {
+					fmt.Fprintln(cmd.OutOrStdout(), "Through sshpiper:", r)
+				}
 				return nil
 			})
 		},
@@ -564,9 +567,14 @@ host's applied configuration. Every attempt is audited.`,
 func newTenantSyncCmd() *cobra.Command {
 	return &cobra.Command{
 		Use:   "sync",
-		Short: "Save the password hashes of every running tenant (daily timer)",
-		Args:  cobra.NoArgs,
+		Short: "Save password hashes and refresh sshpiper keys of every tenant (2-minute timer)",
+		Long: `Runs without the global lock, so the timer never fails because an apply is
+running: it only rewrites each tenant's derived files. It still needs root.`,
+		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			if geteuid() != 0 {
+				return fmt.Errorf("'%s' must run as root", cmd.CommandPath())
+			}
 			_, m, err := tenantEnv(cmd)
 			if err != nil {
 				return err
@@ -579,7 +587,7 @@ func newTenantSyncCmd() *cobra.Command {
 			for _, t := range names {
 				s, err := m.Load(t)
 				if err == nil {
-					err = m.SyncShadow(cmd.Context(), s)
+					err = m.Sync(cmd.Context(), s)
 				}
 				if err != nil {
 					fmt.Fprintf(os.Stderr, "✗ %s: %v\n", t, err)

internal/config/config.go

@@ -51,6 +51,36 @@ type TenantConfig struct {
 	// OpsGroup members may run `sudo rootfiles tenant ops` for the tenants
 	// that list them as operators.
 	OpsGroup string `yaml:"ops_group,omitempty"`
+	// SSHPiper routes `ssh <tenant>.<user>@host -p <port>` to the tenants
+	// through one port.
+	SSHPiper SSHPiperConfig `yaml:"sshpiper,omitempty"`
+}
+
+// SSHPiperConfig runs sshpiperd for single-port tenant SSH.
+type SSHPiperConfig struct {
+	Enabled bool   `yaml:"enabled"`
+	Port    int    `yaml:"port,omitempty"`    // default 2222
+	Address string `yaml:"address,omitempty"` // default all addresses
+	Version string `yaml:"version,omitempty"` // default DefaultSSHPiperVersion
+}
+
+// DefaultSSHPiperVersion is the sshpiper release rootfiles installs.
+const DefaultSSHPiperVersion = "v1.6.1"
+
+// ListenPort returns the port, 2222 when unset.
+func (p SSHPiperConfig) ListenPort() int {
+	if p.Port == 0 {
+		return 2222
+	}
+	return p.Port
+}
+
+// Release returns the version, DefaultSSHPiperVersion when unset.
+func (p SSHPiperConfig) Release() string {
+	if p.Version == "" {
+		return DefaultSSHPiperVersion
+	}
+	return p.Version
 }
 
 // DataRoot returns the tenant root, /raid/tenants when unset.

internal/config/validate.go

@@ -133,6 +133,17 @@ func (c *Config) Validate() error {
 	if t := c.Modules.Tenant; t.Root != "" && (!validTenantRoot.MatchString(t.Root) || filepath.Clean(t.Root) != t.Root || t.Root == "/") {
 		add("modules.tenant.root: %q must be a clean absolute path of letters, digits, '_', '.', '-' and '/'", t.Root)
 	}
+	if sp := c.Modules.Tenant.SSHPiper; sp.Enabled || sp.Port != 0 || sp.Address != "" {
+		checkPort("modules.tenant.sshpiper.port", sp.Port)
+		if sp.Port != 0 && sp.Port == c.SSH.Port || sp.ListenPort() == 22 && (c.SSH.Port == 0 || c.SSH.Port == 22) {
+			add("modules.tenant.sshpiper.port: %d is the host's SSH port", sp.ListenPort())
+		}
+		if sp.Address != "" {
+			if _, err := netip.ParseAddr(sp.Address); err != nil {
+				add("modules.tenant.sshpiper.address: %q is not an IP address", sp.Address)
+			}
+		}
+	}
 	if g := c.Modules.Tenant.OpsGroup; g != "" && (!validAccountName.MatchString(g) || len(g) > 32) {
 		add("modules.tenant.ops_group: invalid group name %q", g)
 	}

internal/module/network.go

@@ -23,6 +23,9 @@ func (m *NetworkModule) requiredPorts(ctx context.Context, rc *RunContext) []int
 	for _, p := range sshPorts(ctx, rc) {
 		set[p] = true
 	}
+	if sp := rc.Config.Modules.Tenant.SSHPiper; sp.Enabled {
+		set[sp.ListenPort()] = true // tenant SSH through sshpiper
+	}
 	ports := make([]int, 0, len(set))
 	for p := range set {
 		ports = append(ports, p)

internal/module/tenant.go

@@ -44,6 +44,7 @@ func NewTenantManager(rc *RunContext, out io.Writer) *tenant.Manager {
 		ReserveGPUs: func(ctx context.Context, t string, gpus []int) error {
 			return ReserveTenantGPUs(ctx, rc, t, gpus)
 		},
+		Piper: newPiper(rc),
 	}
 }
 
@@ -82,7 +83,7 @@ WantedBy=timers.target
 `, 0o644},
 		{unit(tenantSyncUnit + ".service"), `# Managed by rootfiles
 [Unit]
-Description=rootfiles: save tenant password hashes
+Description=rootfiles: save tenant password hashes and sshpiper keys
 After=docker.service
 
 [Service]
@@ -91,12 +92,11 @@ ExecStart=` + fleetSudoBinaryDefault + ` tenant sync
 `, 0o644},
 		{unit(tenantSyncUnit + ".timer"), `# Managed by rootfiles
 [Unit]
-Description=rootfiles daily tenant password sync
+Description=rootfiles tenant password and key sync
 
 [Timer]
-OnCalendar=daily
-RandomizedDelaySec=1h
-Persistent=true
+OnBootSec=2min
+OnUnitActiveSec=2min
 
 [Install]
 WantedBy=timers.target
@@ -159,7 +159,11 @@ func tenantChanges(ctx context.Context, rc *RunContext) ([]Change, error) {
 			add("Install volume mount unit of tenant "+s.Name, "write "+mgr.UnitFor(s.Name))
 		}
 	}
-	return changes, nil
+	piper, err := piperChanges(ctx, rc)
+	if err != nil {
+		return nil, err
+	}
+	return append(changes, piper...), nil
 }
 
 func loadTenantSpecs(rc *RunContext) ([]*tenant.Spec, error) {
@@ -257,6 +261,9 @@ func (m *TenantModule) Apply(ctx context.Context, rc *RunContext) (*ApplyResult,
 		}
 		note("installed tenant operator sudoers rule for %" + group)
 	}
+	if err := applyPiper(ctx, rc, note); err != nil {
+		return nil, err
+	}
 	if rc.DryRun {
 		res.Changed = false
 	}