From a79d84ec08c50bbb8a0fe5e6e9db3ad438ad7ad9 Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Tue, 13 Oct 2026 07:22:00 +0200 Subject: [PATCH 1/2] deploy: run ftwcloudd at cloud.ftw.energy on one t4g.small in Stockholm bootstrap.sh brings up an Amazon Linux 2023 host through SSM: swap, core dumps and sshd off, Docker as on the relay host, and ftw-cloud-deploy, which builds the checked-out commit and starts a new container. The service key is made on the host and copied to tmpfs before Docker starts. deploy/README.md records the running host: ids, how to update, check and tear it down, and the cost. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- README.md | 4 +- deploy/README.md | 159 +++++++++++++++++++++++++++++++++++++------- deploy/bootstrap.sh | 134 +++++++++++++++++++++++++++++++++++++ 4 files changed, 274 insertions(+), 25 deletions(-) create mode 100644 deploy/bootstrap.sh diff --git a/CLAUDE.md b/CLAUDE.md index 5f88b21..9f0971f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,7 +35,7 @@ maintains it. - Keep the service small. The budget is about 3,000 lines of Go; tiles go first if it is overrun. - Nothing is deployed from here without the owner. `deploy/` describes the - intended host; it does not run itself. + running host; it does not run itself. - Keep task breakdowns and agent notes out of the repository. ## Check diff --git a/README.md b/README.md index cbd5fca..44d20dc 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,9 @@ This is `ftwcloudd`, the service. The decision is [ADR 0008](https://github.com/srcfl/ftw/blob/master/docs/adr/0008-ftw-cloud.md) and the wire format is the normative [FTW Cloud protocol](https://github.com/srcfl/ftw/blob/master/docs/protocol-cloud.md), -both in `srcfl/ftw`. Both are proposed; nothing here is deployed. +both in `srcfl/ftw`. Both are proposed. The service runs at +`cloud.ftw.energy` ([deploy/README.md](deploy/README.md)), but no box sends +to it yet. ## What works today (C2) diff --git a/deploy/README.md b/deploy/README.md index 97fe1ba..0788f0f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,39 +1,152 @@ # Running FTW Cloud -Proposed, not deployed. ADR 0008 in `srcfl/ftw` chose one `t4g.small` in AWS -eu-north-1 (Stockholm), in an AWS account of its own, apart from the relay and -escrow. The account is not decided yet; nothing here creates anything. +FTW Cloud runs on one `t4g.small` in AWS eu-north-1 (Stockholm) and answers +at `https://cloud.ftw.energy`. No box sends to it yet: release builds of Core +name no service, and a home can turn FTW Cloud on only from a Core built with +the uplink (srcfl/ftw#1595), after its owner decides to. + +## The host + +| What | Value | +|---|---| +| Account | `976075258475`, shared with other Sourceful work. FTW Cloud moves to an account of its own when it has real users (ADR 0008). | +| Region | eu-north-1, zone eu-north-1a, default VPC | +| Instance | `i-0ecac4cbc6b5f8adc`, `t4g.small`, Amazon Linux 2023 arm64, IMDSv2 required, no key pair | +| Disk | `vol-0760cad0e8551e5e4`, 20 GB gp3, encrypted with `aws/ebs` | +| Elastic IP | `13.50.84.149` (`eipalloc-0085f3e68b6f36da4`) | +| Security group | `sg-098992ad382ecf94a` `ftw-cloud`: 443/tcp from anywhere, nothing else in | +| Instance role | `ftw-cloud-host`, with `AmazonSSMManagedInstanceCore` and nothing else | +| DNS | `cloud.ftw.energy` A `13.50.84.149` in the Cloudflare zone `ftw.energy`, grey cloud (record `22d28943eea9f6b8584214e1a8256ef1`) | +| Branch | `main`, checked out in `/opt/ftw-cloud` | +| Service public key | `BG6EvBxFzSHPVsQOES6PYaKFo5vHQmte9lJ6D5OBIT9JgcBrJAOxAucdqKXxS76pToQ7orI1Ivsno0OtxbMuliQ` | + +Every resource carries the tag `Project=ftw-cloud`. ## Shape - One instance, one container (`compose.yml`), no standby. A rebuilt host starts empty and boxes fill it again when they reconnect. -- `ftwcloudd` ends TLS 1.3 itself and gets its certificate by ACME - (TLS-ALPN-01 on 443). **No TLS-terminating proxy in front**: box and service - bind the handshake to the TLS session, so Caddy, an ALB or an orange - Cloudflare cloud would break every box's connection, on purpose. An NLB in - TCP mode is fine. DNS for `cloud.ftw.energy` is a grey-cloud A record. -- The service key comes from Secrets Manager into `/run/ftw-cloud/service.key` - (tmpfs) at boot. Its public half is compiled into Core - (`cloudlink.ServiceKeys`); Core carries up to two, so it can rotate across a - release. -- The volume holds the SQLite registry (home ids, box keys, signed rosters, - tombstones) and the ACME cache. No readings. Use an encrypted EBS volume. +- `ftwcloudd` ends TLS 1.3 itself and gets its certificate from Let's Encrypt + by TLS-ALPN-01 on 443, so port 80 stays shut. **No TLS-terminating proxy in + front**: box and service bind the handshake to the TLS session, so Caddy, + an ALB or an orange Cloudflare cloud would break every box's connection, on + purpose. An NLB in TCP mode is fine. That is also why the DNS record is + grey. +- The host builds the image from its checkout and tags it with the commit + (`ftw-cloud:<12 hex>`); `/etc/ftw-cloud/compose.env` names it for + `compose.yml`. Images built in CI and deployed by digest come later. +- The Docker volume `ftw-cloud_ftw-cloud-state` holds the SQLite registry + (home ids, box keys, signed rosters, tombstones) and the ACME cache, on the + encrypted root disk. No readings. No backup. + +## The service key + +`ftw-cloud-deploy` makes the key on the host the first time it runs +(`ftwcloudd keygen`) and keeps it at `/var/lib/ftw-cloud/service.key`, root +only, on the encrypted disk. At boot, `ftw-cloud-key.service` copies it to +`/run/ftw-cloud/service.key` (tmpfs) before Docker starts, and `compose.yml` +mounts that copy. + +The key never leaves the host, and nothing backs it up. Its public half, +above, goes into Core's `cloudlink.ServiceKeys`. A new host makes a new key, +and Core then needs a release that carries it; Core carries up to two keys so +it can rotate across a release. + +ADR 0008 keeps the key in Secrets Manager. That needs a role that may read a +secret, and this host's role may only talk to SSM. Move the key to Secrets +Manager before a Core release carries its public half. ## Host rules -- Swap off (`swapoff -a`, no swap in fstab) and core dumps off (the compose - file sets `ulimits: core: 0`; `ftwcloudd` also marks itself not dumpable). +- Swap off (`swapoff -a`, no swap in fstab) and core dumps off: the kernel + pipes cores to `/bin/false`, systemd-coredump stores nothing, `compose.yml` + sets `ulimits: core: 0`, and `ftwcloudd` marks itself not dumpable. Household readings exist only in this process's memory. -- Security group: 443 from anywhere. Nothing else; administration is through - SSM, with every session logged to a separate account. -- Outbound only to AWS endpoints, Let's Encrypt and, from C4, APNs. -- Deploy by image digest, built in CI from a reviewed commit. The running - build's source is at `https://cloud.ftw.energy/source`. +- Nothing comes in but 443. There is no SSH and no key pair; administration + is through SSM. Session logs to a separate account are not set up yet. +- Outbound is open, because the host builds its own image (Docker Hub, + gcr.io, the Go module proxy, GitHub). Narrow it to AWS endpoints, Let's + Encrypt and, from C4, APNs once images come from CI. +- The running build's source is at `https://cloud.ftw.energy/source`. + +## Bringing it up + +`bootstrap.sh` is the whole host. It is idempotent: it turns off swap and +core dumps, installs Docker, Compose and buildx as the relay host does, +clones this repository to `/opt/ftw-cloud`, installs the two systemd units and +`/usr/local/sbin/ftw-cloud-deploy`, and runs that. + +```bash +aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \ + --document-name AWS-RunShellScript \ + --parameters "$(jq -n --rawfile s deploy/bootstrap.sh '{commands: [$s]}')" +``` + +The DNS record must exist before the first TLS connection, since that is when +`ftwcloudd` asks Let's Encrypt for a certificate. + +## Updating + +```bash +aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \ + --document-name AWS-RunShellScript \ + --parameters 'commands=["/usr/local/sbin/ftw-cloud-deploy"]' +``` + +or, in a session (`aws ssm start-session --region eu-north-1 --target +i-0ecac4cbc6b5f8adc`), `sudo ftw-cloud-deploy`. It pulls the checked-out branch, +builds, and starts a new container. Every box reconnects within seconds. + +It always makes a new container, even when only a mounted file changed. The +relay ran into this: a file mounted on its own, like the service key, keeps +the copy the container started with after the file changes on the host. + +## Checking it + +```bash +curl -s https://cloud.ftw.energy/healthz +``` + +`ok` means DNS, the certificate and the service all work. +`https://cloud.ftw.energy/source` returns the source archive. +`/v1/app` and `/v1/now` answer 501 until C3 and C5. + +## Cost + +eu-north-1 on-demand prices, October 2026: + +| Item | Price | Per month | +|---|---|---| +| `t4g.small` | $0.0172 an hour | $12.56 | +| 20 GB gp3 | $0.0836 a GB-month | $1.67 | +| Public IPv4 (the Elastic IP) | $0.005 an hour | $3.65 | +| **Total** | | **about $18** | + +Traffic from boxes comes in, which AWS does not charge for. Traffic out to +phones (C3) is free for the first 100 GB a month across the account. SSM costs +nothing. CPU credits run unlimited, which costs extra only if the CPU averages +more than 20% over a day. + +## Tearing it down + +```bash +cf dns records delete 22d28943eea9f6b8584214e1a8256ef1 -z ftw.energy +aws ec2 terminate-instances --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc +aws ec2 wait instance-terminated --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc +aws ec2 release-address --region eu-north-1 --allocation-id eipalloc-0085f3e68b6f36da4 +aws ec2 delete-security-group --region eu-north-1 --group-id sg-098992ad382ecf94a +aws iam remove-role-from-instance-profile --instance-profile-name ftw-cloud-host --role-name ftw-cloud-host +aws iam delete-instance-profile --instance-profile-name ftw-cloud-host +aws iam detach-role-policy --role-name ftw-cloud-host \ + --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore +aws iam delete-role --role-name ftw-cloud-host +``` + +The disk goes with the instance, and with it the registry and the service +key. Nothing else holds FTW Cloud data. ## Not here yet APNs (C4), the app session (C3), widget tokens (C5) and statistics (C7–C8) add secrets, egress and storage when they land. The erase test, canary homes -and the external heartbeat that pages Sourceful belong with the first -deployment. +and the external heartbeat that pages Sourceful belong with the first homes. diff --git a/deploy/bootstrap.sh b/deploy/bootstrap.sh new file mode 100644 index 0000000..764517c --- /dev/null +++ b/deploy/bootstrap.sh @@ -0,0 +1,134 @@ +#!/bin/bash +# Bring up an FTW Cloud host. Amazon Linux 2023, arm64, run as root through +# SSM. Safe to re-run: everything below is idempotent, and a re-run is also +# how the host's own tooling gets updated. +set -euxo pipefail + +export HOME=${HOME:-/root} + +# Readings live only in ftwcloudd's memory. No swap may page them to disk and +# no core file may carry them out. +swapoff -a +sed -i '/\sswap\s/d' /etc/fstab +cat > /etc/sysctl.d/99-ftw-cloud.conf <<'SYSCTL' +kernel.core_pattern = |/bin/false +fs.suid_dumpable = 0 +SYSCTL +sysctl --system >/dev/null +install -d /etc/systemd/coredump.conf.d +printf '[Coredump]\nStorage=none\nProcessSizeMax=0\n' > /etc/systemd/coredump.conf.d/ftw-cloud.conf +printf '* hard core 0\n' > /etc/security/limits.d/ftw-cloud.conf + +# No SSH, not even behind the security group. Administration is through SSM. +systemctl disable --now sshd.service + +# Docker, Compose and buildx, as the relay host does it. +dnf install -y docker git +systemctl enable --now docker + +install -d /usr/local/lib/docker/cli-plugins +case "$(uname -m)" in + aarch64) PLUGIN_ARCH=arm64 ; COMPOSE_ARCH=aarch64 ;; + *) PLUGIN_ARCH=amd64 ; COMPOSE_ARCH=x86_64 ;; +esac +curl -fsSL "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-${COMPOSE_ARCH}" \ + -o /usr/local/lib/docker/cli-plugins/docker-compose +chmod +x /usr/local/lib/docker/cli-plugins/docker-compose +# sed reads to the end. head or grep -m1 closes the pipe early, and under +# pipefail that fails the script. +BUILDX_URL=$(curl -fsS https://api.github.com/repos/docker/buildx/releases/latest \ + | grep -o "https://[^\"]*linux-${PLUGIN_ARCH}" | sed -n 1p) +curl -fsSL "$BUILDX_URL" -o /usr/local/lib/docker/cli-plugins/docker-buildx +chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx +docker compose version +docker buildx version + +if [ ! -d /opt/ftw-cloud ]; then + git clone https://github.com/srcfl/ftw-cloud /opt/ftw-cloud +fi + +install -d -m 0700 /var/lib/ftw-cloud /etc/ftw-cloud + +# The service key rests on the encrypted root volume, readable by root only. +# At boot, before Docker starts, a copy goes to tmpfs, where compose.yml +# mounts it into the container. +cat > /etc/systemd/system/ftw-cloud-key.service <<'UNIT' +[Unit] +Description=FTW Cloud service key into tmpfs +Before=docker.service +ConditionPathExists=/var/lib/ftw-cloud/service.key + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/bin/install -d -m 0700 /run/ftw-cloud +ExecStart=/usr/bin/install -m 0400 -o 65532 -g 65532 /var/lib/ftw-cloud/service.key /run/ftw-cloud/service.key + +[Install] +WantedBy=docker.service +UNIT + +# Come back after a reboot without anyone logging in. +cat > /etc/systemd/system/ftw-cloud.service <<'UNIT' +[Unit] +Description=FTW Cloud +Requires=docker.service ftw-cloud-key.service +After=docker.service ftw-cloud-key.service network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +RemainAfterExit=yes +WorkingDirectory=/opt/ftw-cloud/deploy +ExecStart=/usr/bin/docker compose --env-file /etc/ftw-cloud/compose.env up -d +ExecStop=/usr/bin/docker compose --env-file /etc/ftw-cloud/compose.env down + +[Install] +WantedBy=multi-user.target +UNIT + +# One command updates the host: pull, build, run. The checkout's branch +# decides what runs. +cat > /usr/local/sbin/ftw-cloud-deploy <<'DEPLOY' +#!/bin/bash +# Build the commit checked out in /opt/ftw-cloud and run it. Every box +# reconnects within seconds; there is nothing to drain. +set -euo pipefail +export HOME=${HOME:-/root} +cd /opt/ftw-cloud +git pull --ff-only +rev=$(git rev-parse --short=12 HEAD) +docker build --pull -t "ftw-cloud:$rev" . + +# The key is made once, on this host, and never leaves it. Its public half +# is what Core carries in cloudlink.ServiceKeys. +if [ ! -s /var/lib/ftw-cloud/service.key ]; then + docker run --rm --user 0:0 --network none -v /var/lib/ftw-cloud:/out \ + "ftw-cloud:$rev" keygen -out /out/service.key +fi +systemctl enable ftw-cloud-key.service ftw-cloud.service +systemctl restart ftw-cloud-key.service + +printf 'FTW_CLOUD_IMAGE=ftw-cloud:%s\n' "$rev" > /etc/ftw-cloud/compose.env +cd deploy +# Always a new container. A file mounted on its own, such as the key, keeps +# the copy the container started with even after the file on the host +# changes; only a new container reads it again. +docker compose --env-file /etc/ftw-cloud/compose.env up -d --force-recreate + +for img in $(docker image ls ftw-cloud --format '{{.Repository}}:{{.Tag}}'); do + [ "$img" = "ftw-cloud:$rev" ] || docker image rm "$img" +done +docker image prune -f +docker builder prune -f --filter until=168h +echo "ftw-cloud:$rev running" +DEPLOY +chmod 0755 /usr/local/sbin/ftw-cloud-deploy + +systemctl daemon-reload +/usr/local/sbin/ftw-cloud-deploy + +# Show that the host rules hold. +swapon --show +sysctl kernel.core_pattern +docker compose --env-file /etc/ftw-cloud/compose.env -f /opt/ftw-cloud/deploy/compose.yml ps From 0d4e5da2345a30f0553e5bf3bf3ac42a44350f4e Mon Sep 17 00:00:00 2001 From: Fredrik Ahlgren Date: Tue, 13 Oct 2026 07:50:53 +0200 Subject: [PATCH 2/2] deploy: mask sshd, since cloud-init starts it at every boot A reboot test showed sshd listening on 22 again: cloud-init.service wants it, so disabling the unit is not enough. Masking it keeps it off across reboots. The security group already dropped 22. Co-Authored-By: Claude Opus 5.5 --- deploy/bootstrap.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/deploy/bootstrap.sh b/deploy/bootstrap.sh index 764517c..117314b 100644 --- a/deploy/bootstrap.sh +++ b/deploy/bootstrap.sh @@ -20,7 +20,8 @@ printf '[Coredump]\nStorage=none\nProcessSizeMax=0\n' > /etc/systemd/coredump.co printf '* hard core 0\n' > /etc/security/limits.d/ftw-cloud.conf # No SSH, not even behind the security group. Administration is through SSM. -systemctl disable --now sshd.service +# cloud-init wants sshd at every boot, so disabling it is not enough. +systemctl mask --now sshd.service # Docker, Compose and buildx, as the relay host does it. dnf install -y docker git