deploy: run ftwcloudd at cloud.ftw.energy in Stockholm

CLAUDE.md

@@ -35,7 +35,7 @@ maintains it.
 - Keep the service small. The budget is about 3,000 lines of Go; tiles go
   first if it is overrun.
 - Nothing is deployed from here without the owner. `deploy/` describes the
-  intended host; it does not run itself.
+  running host; it does not run itself.
 - Keep task breakdowns and agent notes out of the repository.
 
 ## Check

README.md

@@ -8,7 +8,9 @@ This is `ftwcloudd`, the service. The decision is
 [ADR 0008](https://github.com/srcfl/ftw/blob/master/docs/adr/0008-ftw-cloud.md)
 and the wire format is the normative
 [FTW Cloud protocol](https://github.com/srcfl/ftw/blob/master/docs/protocol-cloud.md),
-both in `srcfl/ftw`. Both are proposed; nothing here is deployed.
+both in `srcfl/ftw`. Both are proposed. The service runs at
+`cloud.ftw.energy` ([deploy/README.md](deploy/README.md)), but no box sends
+to it yet.
 
 ## What works today (C2)
 

deploy/README.md

@@ -1,39 +1,152 @@
 # Running FTW Cloud
 
-Proposed, not deployed. ADR 0008 in `srcfl/ftw` chose one `t4g.small` in AWS
-eu-north-1 (Stockholm), in an AWS account of its own, apart from the relay and
-escrow. The account is not decided yet; nothing here creates anything.
+FTW Cloud runs on one `t4g.small` in AWS eu-north-1 (Stockholm) and answers
+at `https://cloud.ftw.energy`. No box sends to it yet: release builds of Core
+name no service, and a home can turn FTW Cloud on only from a Core built with
+the uplink (srcfl/ftw#1595), after its owner decides to.
+
+## The host
+
+| What | Value |
+|---|---|
+| Account | `976075258475`, shared with other Sourceful work. FTW Cloud moves to an account of its own when it has real users (ADR 0008). |
+| Region | eu-north-1, zone eu-north-1a, default VPC |
+| Instance | `i-0ecac4cbc6b5f8adc`, `t4g.small`, Amazon Linux 2023 arm64, IMDSv2 required, no key pair |
+| Disk | `vol-0760cad0e8551e5e4`, 20 GB gp3, encrypted with `aws/ebs` |
+| Elastic IP | `13.50.84.149` (`eipalloc-0085f3e68b6f36da4`) |
+| Security group | `sg-098992ad382ecf94a` `ftw-cloud`: 443/tcp from anywhere, nothing else in |
+| Instance role | `ftw-cloud-host`, with `AmazonSSMManagedInstanceCore` and nothing else |
+| DNS | `cloud.ftw.energy` A `13.50.84.149` in the Cloudflare zone `ftw.energy`, grey cloud (record `22d28943eea9f6b8584214e1a8256ef1`) |
+| Branch | `main`, checked out in `/opt/ftw-cloud` |
+| Service public key | `BG6EvBxFzSHPVsQOES6PYaKFo5vHQmte9lJ6D5OBIT9JgcBrJAOxAucdqKXxS76pToQ7orI1Ivsno0OtxbMuliQ` |
+
+Every resource carries the tag `Project=ftw-cloud`.
 
 ## Shape
 
 - One instance, one container (`compose.yml`), no standby. A rebuilt host
   starts empty and boxes fill it again when they reconnect.
-- `ftwcloudd` ends TLS 1.3 itself and gets its certificate by ACME
-  (TLS-ALPN-01 on 443). **No TLS-terminating proxy in front**: box and service
-  bind the handshake to the TLS session, so Caddy, an ALB or an orange
-  Cloudflare cloud would break every box's connection, on purpose. An NLB in
-  TCP mode is fine. DNS for `cloud.ftw.energy` is a grey-cloud A record.
-- The service key comes from Secrets Manager into `/run/ftw-cloud/service.key`
-  (tmpfs) at boot. Its public half is compiled into Core
-  (`cloudlink.ServiceKeys`); Core carries up to two, so it can rotate across a
-  release.
-- The volume holds the SQLite registry (home ids, box keys, signed rosters,
-  tombstones) and the ACME cache. No readings. Use an encrypted EBS volume.
+- `ftwcloudd` ends TLS 1.3 itself and gets its certificate from Let's Encrypt
+  by TLS-ALPN-01 on 443, so port 80 stays shut. **No TLS-terminating proxy in
+  front**: box and service bind the handshake to the TLS session, so Caddy,
+  an ALB or an orange Cloudflare cloud would break every box's connection, on
+  purpose. An NLB in TCP mode is fine. That is also why the DNS record is
+  grey.
+- The host builds the image from its checkout and tags it with the commit
+  (`ftw-cloud:<12 hex>`); `/etc/ftw-cloud/compose.env` names it for
+  `compose.yml`. Images built in CI and deployed by digest come later.
+- The Docker volume `ftw-cloud_ftw-cloud-state` holds the SQLite registry
+  (home ids, box keys, signed rosters, tombstones) and the ACME cache, on the
+  encrypted root disk. No readings. No backup.
+
+## The service key
+
+`ftw-cloud-deploy` makes the key on the host the first time it runs
+(`ftwcloudd keygen`) and keeps it at `/var/lib/ftw-cloud/service.key`, root
+only, on the encrypted disk. At boot, `ftw-cloud-key.service` copies it to
+`/run/ftw-cloud/service.key` (tmpfs) before Docker starts, and `compose.yml`
+mounts that copy.
+
+The key never leaves the host, and nothing backs it up. Its public half,
+above, goes into Core's `cloudlink.ServiceKeys`. A new host makes a new key,
+and Core then needs a release that carries it; Core carries up to two keys so
+it can rotate across a release.
+
+ADR 0008 keeps the key in Secrets Manager. That needs a role that may read a
+secret, and this host's role may only talk to SSM. Move the key to Secrets
+Manager before a Core release carries its public half.
 
 ## Host rules
 
-- Swap off (`swapoff -a`, no swap in fstab) and core dumps off (the compose
-  file sets `ulimits: core: 0`; `ftwcloudd` also marks itself not dumpable).
+- Swap off (`swapoff -a`, no swap in fstab) and core dumps off: the kernel
+  pipes cores to `/bin/false`, systemd-coredump stores nothing, `compose.yml`
+  sets `ulimits: core: 0`, and `ftwcloudd` marks itself not dumpable.
   Household readings exist only in this process's memory.
-- Security group: 443 from anywhere. Nothing else; administration is through
-  SSM, with every session logged to a separate account.
-- Outbound only to AWS endpoints, Let's Encrypt and, from C4, APNs.
-- Deploy by image digest, built in CI from a reviewed commit. The running
-  build's source is at `https://cloud.ftw.energy/source`.
+- Nothing comes in but 443. There is no SSH and no key pair; administration
+  is through SSM. Session logs to a separate account are not set up yet.
+- Outbound is open, because the host builds its own image (Docker Hub,
+  gcr.io, the Go module proxy, GitHub). Narrow it to AWS endpoints, Let's
+  Encrypt and, from C4, APNs once images come from CI.
+- The running build's source is at `https://cloud.ftw.energy/source`.
+
+## Bringing it up
+
+`bootstrap.sh` is the whole host. It is idempotent: it turns off swap and
+core dumps, installs Docker, Compose and buildx as the relay host does,
+clones this repository to `/opt/ftw-cloud`, installs the two systemd units and
+`/usr/local/sbin/ftw-cloud-deploy`, and runs that.
+
+```bash
+aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \
+  --document-name AWS-RunShellScript \
+  --parameters "$(jq -n --rawfile s deploy/bootstrap.sh '{commands: [$s]}')"
+```
+
+The DNS record must exist before the first TLS connection, since that is when
+`ftwcloudd` asks Let's Encrypt for a certificate.
+
+## Updating
+
+```bash
+aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \
+  --document-name AWS-RunShellScript \
+  --parameters 'commands=["/usr/local/sbin/ftw-cloud-deploy"]'
+```
+
+or, in a session (`aws ssm start-session --region eu-north-1 --target
+i-0ecac4cbc6b5f8adc`), `sudo ftw-cloud-deploy`. It pulls the checked-out branch,
+builds, and starts a new container. Every box reconnects within seconds.
+
+It always makes a new container, even when only a mounted file changed. The
+relay ran into this: a file mounted on its own, like the service key, keeps
+the copy the container started with after the file changes on the host.
+
+## Checking it
+
+```bash
+curl -s https://cloud.ftw.energy/healthz
+```
+
+`ok` means DNS, the certificate and the service all work.
+`https://cloud.ftw.energy/source` returns the source archive.
+`/v1/app` and `/v1/now` answer 501 until C3 and C5.
+
+## Cost
+
+eu-north-1 on-demand prices, October 2026:
+
+| Item | Price | Per month |
+|---|---|---|
+| `t4g.small` | $0.0172 an hour | $12.56 |
+| 20 GB gp3 | $0.0836 a GB-month | $1.67 |
+| Public IPv4 (the Elastic IP) | $0.005 an hour | $3.65 |
+| **Total** | | **about $18** |
+
+Traffic from boxes comes in, which AWS does not charge for. Traffic out to
+phones (C3) is free for the first 100 GB a month across the account. SSM costs
+nothing. CPU credits run unlimited, which costs extra only if the CPU averages
+more than 20% over a day.
+
+## Tearing it down
+
+```bash
+cf dns records delete 22d28943eea9f6b8584214e1a8256ef1 -z ftw.energy
+aws ec2 terminate-instances --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc
+aws ec2 wait instance-terminated --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc
+aws ec2 release-address --region eu-north-1 --allocation-id eipalloc-0085f3e68b6f36da4
+aws ec2 delete-security-group --region eu-north-1 --group-id sg-098992ad382ecf94a
+aws iam remove-role-from-instance-profile --instance-profile-name ftw-cloud-host --role-name ftw-cloud-host
+aws iam delete-instance-profile --instance-profile-name ftw-cloud-host
+aws iam detach-role-policy --role-name ftw-cloud-host \
+  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
+aws iam delete-role --role-name ftw-cloud-host
+```
+
+The disk goes with the instance, and with it the registry and the service
+key. Nothing else holds FTW Cloud data.
 
 ## Not here yet
 
 APNs (C4), the app session (C3), widget tokens (C5) and statistics (C7–C8)
 add secrets, egress and storage when they land. The erase test, canary homes
-and the external heartbeat that pages Sourceful belong with the first
-deployment.
+and the external heartbeat that pages Sourceful belong with the first homes.

deploy/bootstrap.sh

@@ -0,0 +1,135 @@
+#!/bin/bash
+# Bring up an FTW Cloud host. Amazon Linux 2023, arm64, run as root through
+# SSM. Safe to re-run: everything below is idempotent, and a re-run is also
+# how the host's own tooling gets updated.
+set -euxo pipefail
+
+export HOME=${HOME:-/root}
+
+# Readings live only in ftwcloudd's memory. No swap may page them to disk and
+# no core file may carry them out.
+swapoff -a
+sed -i '/\sswap\s/d' /etc/fstab
+cat > /etc/sysctl.d/99-ftw-cloud.conf <<'SYSCTL'
+kernel.core_pattern = |/bin/false
+fs.suid_dumpable = 0
+SYSCTL
+sysctl --system >/dev/null
+install -d /etc/systemd/coredump.conf.d
+printf '[Coredump]\nStorage=none\nProcessSizeMax=0\n' > /etc/systemd/coredump.conf.d/ftw-cloud.conf
+printf '* hard core 0\n' > /etc/security/limits.d/ftw-cloud.conf
+
+# No SSH, not even behind the security group. Administration is through SSM.
+# cloud-init wants sshd at every boot, so disabling it is not enough.
+systemctl mask --now sshd.service
+
+# Docker, Compose and buildx, as the relay host does it.
+dnf install -y docker git
+systemctl enable --now docker
+
+install -d /usr/local/lib/docker/cli-plugins
+case "$(uname -m)" in
+  aarch64) PLUGIN_ARCH=arm64 ; COMPOSE_ARCH=aarch64 ;;
+  *)       PLUGIN_ARCH=amd64 ; COMPOSE_ARCH=x86_64 ;;
+esac
+curl -fsSL "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-${COMPOSE_ARCH}" \
+  -o /usr/local/lib/docker/cli-plugins/docker-compose
+chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
+# sed reads to the end. head or grep -m1 closes the pipe early, and under
+# pipefail that fails the script.
+BUILDX_URL=$(curl -fsS https://api.github.com/repos/docker/buildx/releases/latest \
+  | grep -o "https://[^\"]*linux-${PLUGIN_ARCH}" | sed -n 1p)
+curl -fsSL "$BUILDX_URL" -o /usr/local/lib/docker/cli-plugins/docker-buildx
+chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx
+docker compose version
+docker buildx version
+
+if [ ! -d /opt/ftw-cloud ]; then
+  git clone https://github.com/srcfl/ftw-cloud /opt/ftw-cloud
+fi
+
+install -d -m 0700 /var/lib/ftw-cloud /etc/ftw-cloud
+
+# The service key rests on the encrypted root volume, readable by root only.
+# At boot, before Docker starts, a copy goes to tmpfs, where compose.yml
+# mounts it into the container.
+cat > /etc/systemd/system/ftw-cloud-key.service <<'UNIT'
+[Unit]
+Description=FTW Cloud service key into tmpfs
+Before=docker.service
+ConditionPathExists=/var/lib/ftw-cloud/service.key
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/bin/install -d -m 0700 /run/ftw-cloud
+ExecStart=/usr/bin/install -m 0400 -o 65532 -g 65532 /var/lib/ftw-cloud/service.key /run/ftw-cloud/service.key
+
+[Install]
+WantedBy=docker.service
+UNIT
+
+# Come back after a reboot without anyone logging in.
+cat > /etc/systemd/system/ftw-cloud.service <<'UNIT'
+[Unit]
+Description=FTW Cloud
+Requires=docker.service ftw-cloud-key.service
+After=docker.service ftw-cloud-key.service network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+WorkingDirectory=/opt/ftw-cloud/deploy
+ExecStart=/usr/bin/docker compose --env-file /etc/ftw-cloud/compose.env up -d
+ExecStop=/usr/bin/docker compose --env-file /etc/ftw-cloud/compose.env down
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+
+# One command updates the host: pull, build, run. The checkout's branch
+# decides what runs.
+cat > /usr/local/sbin/ftw-cloud-deploy <<'DEPLOY'
+#!/bin/bash
+# Build the commit checked out in /opt/ftw-cloud and run it. Every box
+# reconnects within seconds; there is nothing to drain.
+set -euo pipefail
+export HOME=${HOME:-/root}
+cd /opt/ftw-cloud
+git pull --ff-only
+rev=$(git rev-parse --short=12 HEAD)
+docker build --pull -t "ftw-cloud:$rev" .
+
+# The key is made once, on this host, and never leaves it. Its public half
+# is what Core carries in cloudlink.ServiceKeys.
+if [ ! -s /var/lib/ftw-cloud/service.key ]; then
+  docker run --rm --user 0:0 --network none -v /var/lib/ftw-cloud:/out \
+    "ftw-cloud:$rev" keygen -out /out/service.key
+fi
+systemctl enable ftw-cloud-key.service ftw-cloud.service
+systemctl restart ftw-cloud-key.service
+
+printf 'FTW_CLOUD_IMAGE=ftw-cloud:%s\n' "$rev" > /etc/ftw-cloud/compose.env
+cd deploy
+# Always a new container. A file mounted on its own, such as the key, keeps
+# the copy the container started with even after the file on the host
+# changes; only a new container reads it again.
+docker compose --env-file /etc/ftw-cloud/compose.env up -d --force-recreate
+
+for img in $(docker image ls ftw-cloud --format '{{.Repository}}:{{.Tag}}'); do
+  [ "$img" = "ftw-cloud:$rev" ] || docker image rm "$img"
+done
+docker image prune -f
+docker builder prune -f --filter until=168h
+echo "ftw-cloud:$rev running"
+DEPLOY
+chmod 0755 /usr/local/sbin/ftw-cloud-deploy
+
+systemctl daemon-reload
+/usr/local/sbin/ftw-cloud-deploy
+
+# Show that the host rules hold.
+swapon --show
+sysctl kernel.core_pattern
+docker compose --env-file /etc/ftw-cloud/compose.env -f /opt/ftw-cloud/deploy/compose.yml ps