deploy: images built in CI, run by digest; service key in Secrets Manager

.github/workflows/image.yml

@@ -0,0 +1,70 @@
+name: Image
+
+# Builds the host's image (linux/arm64) from every commit on main and pushes
+# it to ghcr.io/srcfl/ftw-cloud, a public package the host pulls without a
+# credential. A pull request builds the image and pushes nothing. Nothing
+# here deploys: the owner picks a digest from a run on main and runs
+# ftw-cloud-deploy with it (deploy/README.md).
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  image:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+      id-token: write
+      attestations: write
+    env:
+      PUSH: ${{ github.event_name == 'push' }}
+    steps:
+      - uses: actions/checkout@v5
+      - uses: docker/setup-buildx-action@v4
+      - if: env.PUSH == 'true'
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - id: build
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          platforms: linux/arm64
+          push: ${{ env.PUSH == 'true' }}
+          tags: ghcr.io/srcfl/ftw-cloud:${{ github.sha }}
+          build-args: REVISION=${{ github.sha }}
+          # ftw-cloud-deploy reads the revision label to check out the
+          # matching compose.yml and to refuse an image built off main.
+          labels: |
+            org.opencontainers.image.source=https://github.com/srcfl/ftw-cloud
+            org.opencontainers.image.revision=${{ github.sha }}
+            org.opencontainers.image.licenses=AGPL-3.0-only
+          # One plain manifest, so the digest names exactly one image. The
+          # provenance goes to GitHub's attestation store below.
+          provenance: false
+          sbom: false
+      # Anyone can check that a digest came from this workflow on a commit
+      # in this repository: gh attestation verify oci://<image> -R srcfl/ftw-cloud
+      - if: env.PUSH == 'true'
+        uses: actions/attest-build-provenance@v4
+        with:
+          subject-name: ghcr.io/srcfl/ftw-cloud
+          subject-digest: ${{ steps.build.outputs.digest }}
+      - if: env.PUSH == 'true'
+        name: The digest to deploy
+        env:
+          DIGEST: ${{ steps.build.outputs.digest }}
+        run: |
+          {
+            echo "Built \`${GITHUB_SHA}\` as \`ghcr.io/srcfl/ftw-cloud@${DIGEST}\`."
+            echo
+            echo "Deploy it with \`ftw-cloud-deploy ${DIGEST}\` (deploy/README.md)."
+          } >> "$GITHUB_STEP_SUMMARY"

Dockerfile

@@ -4,13 +4,21 @@
 # AGPL-3.0 asks that everyone who uses the service over a network can get the
 # code running it.
 
-FROM golang:1.26-alpine AS build
+# Go cross-compiles, so CI builds the arm64 image on any runner without
+# emulating arm64.
+FROM --platform=$BUILDPLATFORM golang:1.26-alpine AS build
+ARG TARGETOS TARGETARCH
+# The commit, for X-Source-Revision. CI passes it; .git stays out of the
+# build context, so Go cannot read it itself.
+ARG REVISION=
 WORKDIR /src
 COPY go.mod go.sum ./
 RUN go mod download
 COPY . .
 # No cgo: the SQLite driver is pure Go, so the binary runs on a static base.
-RUN CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o /out/ftwcloudd ./cmd/ftwcloudd
+RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath \
+      -ldflags "-s -w -X github.com/srcfl/ftw-cloud/internal/server.Revision=${REVISION}" \
+      -o /out/ftwcloudd ./cmd/ftwcloudd
 RUN mkdir -p /out/share && tar -czf /out/share/source.tar.gz \
       --exclude=./.git --exclude=./state . \
     && mkdir -p /out/state

README.md

@@ -23,7 +23,8 @@ to it yet.
   never written to disk.
 - The registry (SQLite) holds the home id, the box key and the signed roster.
   A box that turns FTW Cloud off gets its home deleted and a signed receipt.
-- `/source` serves the source of the running build, as AGPL-3.0 asks.
+- `/source` serves the source of the running build, as AGPL-3.0 asks, and
+  names its commit and image digest.
 
 Not built yet, with a marked place in the code:
 

deploy/README.md

@@ -15,12 +15,13 @@ the uplink (srcfl/ftw#1595), after its owner decides to.
 | Disk | `vol-0760cad0e8551e5e4`, 20 GB gp3, encrypted with `aws/ebs` |
 | Elastic IP | `13.50.84.149` (`eipalloc-0085f3e68b6f36da4`) |
 | Security group | `sg-098992ad382ecf94a` `ftw-cloud`: 443/tcp from anywhere, nothing else in |
-| Instance role | `ftw-cloud-host`, with `AmazonSSMManagedInstanceCore` and nothing else |
+| Instance role | `ftw-cloud-host`, with `AmazonSSMManagedInstanceCore` and the inline policy `ftw-cloud-service-key` ([`host-policy.json`](host-policy.json)): read one secret, nothing else |
+| Secret | `ftw-cloud/service-key` in Secrets Manager, eu-north-1, encrypted with `aws/secretsmanager` |
 | DNS | `cloud.ftw.energy` A `13.50.84.149` in the Cloudflare zone `ftw.energy`, grey cloud (record `22d28943eea9f6b8584214e1a8256ef1`) |
-| Branch | `main`, checked out in `/opt/ftw-cloud` |
+| Image | `ghcr.io/srcfl/ftw-cloud`, public, run by digest; `/etc/ftw-cloud/compose.env` names the running one |
 | Service public key | `BG6EvBxFzSHPVsQOES6PYaKFo5vHQmte9lJ6D5OBIT9JgcBrJAOxAucdqKXxS76pToQ7orI1Ivsno0OtxbMuliQ` |
 
-Every resource carries the tag `Project=ftw-cloud`.
+Every AWS resource carries the tag `Project=ftw-cloud`.
 
 ## Shape
 
@@ -32,29 +33,37 @@ Every resource carries the tag `Project=ftw-cloud`.
   an ALB or an orange Cloudflare cloud would break every box's connection, on
   purpose. An NLB in TCP mode is fine. That is also why the DNS record is
   grey.
-- The host builds the image from its checkout and tags it with the commit
-  (`ftw-cloud:<12 hex>`); `/etc/ftw-cloud/compose.env` names it for
-  `compose.yml`. Images built in CI and deployed by digest come later.
+- CI builds the image. The `Image` workflow
+  ([`image.yml`](../.github/workflows/image.yml)) builds `linux/arm64` from
+  every commit on `main`, pushes it to `ghcr.io/srcfl/ftw-cloud` tagged with
+  the commit, and records its provenance in GitHub's attestation store. The
+  host builds nothing and needs no credential to pull.
+- The host runs one image, named by its digest, never by a tag.
+  `/opt/ftw-cloud` is checked out at the commit that image was built from and
+  supplies `compose.yml`.
 - The Docker volume `ftw-cloud_ftw-cloud-state` holds the SQLite registry
   (home ids, box keys, signed rosters, tombstones) and the ACME cache, on the
   encrypted root disk. No readings. No backup.
 
 ## The service key
 
-`ftw-cloud-deploy` makes the key on the host the first time it runs
-(`ftwcloudd keygen`) and keeps it at `/var/lib/ftw-cloud/service.key`, root
-only, on the encrypted disk. At boot, `ftw-cloud-key.service` copies it to
-`/run/ftw-cloud/service.key` (tmpfs) before Docker starts, and `compose.yml`
-mounts that copy.
+The key lives in Secrets Manager as `ftw-cloud/service-key`, in PEM. At boot,
+before Docker starts, `ftw-cloud-key.service` runs
+`/usr/local/sbin/ftw-cloud-key`, which fetches it into
+`/run/ftw-cloud/service.key` (tmpfs, owned by the container's user, mode
+`0400`), and `compose.yml` mounts that file. The key is never on disk and
+never in the image. `ftw-cloud-deploy` fetches it again, so a new version
+takes effect on the next deploy or reboot.
 
-The key never leaves the host, and nothing backs it up. Its public half,
-above, goes into Core's `cloudlink.ServiceKeys`. A new host makes a new key,
-and Core then needs a release that carries it; Core carries up to two keys so
-it can rotate across a release.
+The role may read this one secret and write nothing. A rebuilt host reads the
+same key, so Core needs no new release for it. The public half, above, goes
+into Core's `cloudlink.ServiceKeys`.
 
-ADR 0008 keeps the key in Secrets Manager. That needs a role that may read a
-secret, and this host's role may only talk to SSM. Move the key to Secrets
-Manager before a Core release carries its public half.
+To rotate, ship a Core release that carries both public keys first (Core
+carries up to two), then put the new key into the secret. Writing the secret
+needs `secretsmanager:PutSecretValue`, which the role lacks: grant it for the
+change and take it away after. The key moved from the host's disk to the
+secret the same way.
 
 ## Host rules
 
@@ -64,17 +73,32 @@ Manager before a Core release carries its public half.
   Household readings exist only in this process's memory.
 - Nothing comes in but 443. There is no SSH and no key pair; administration
   is through SSM. Session logs to a separate account are not set up yet.
-- Outbound is open, because the host builds its own image (Docker Hub,
-  gcr.io, the Go module proxy, GitHub). Narrow it to AWS endpoints, Let's
-  Encrypt and, from C4, APNs once images come from CI.
-- The running build's source is at `https://cloud.ftw.energy/source`.
+- Outbound is open. The host now needs only GitHub and `ghcr.io` (the image
+  and `compose.yml`), AWS endpoints (SSM, Secrets Manager), Let's Encrypt
+  and, from C4, APNs. Narrowing it is still to do.
+- The running build's source is at `https://cloud.ftw.energy/source`, with
+  its commit in `X-Source-Revision` and the image digest in `X-Source-Image`.
 
 ## Bringing it up
 
-`bootstrap.sh` is the whole host. It is idempotent: it turns off swap and
-core dumps, installs Docker, Compose and buildx as the relay host does,
-clones this repository to `/opt/ftw-cloud`, installs the two systemd units and
-`/usr/local/sbin/ftw-cloud-deploy`, and runs that.
+The secret and the role's policy come first; they outlive the host:
+
+```bash
+aws secretsmanager create-secret --region eu-north-1 --name ftw-cloud/service-key \
+  --description 'FTW Cloud service key, PEM' --tags Key=Project,Value=ftw-cloud
+aws iam put-role-policy --role-name ftw-cloud-host --policy-name ftw-cloud-service-key \
+  --policy-document file://deploy/host-policy.json
+```
+
+A secret without a value needs a key in it before the host can start (see
+the service key, above).
+
+`bootstrap.sh` is the rest of the host. It is idempotent: it turns off swap
+and core dumps, installs Docker and Compose, clones this repository to
+`/opt/ftw-cloud`, and installs `ftw-cloud-key`, the two systemd units and
+`/usr/local/sbin/ftw-cloud-deploy`. On a host that already runs an image from
+CI, it runs that image again with the new tooling. A new host waits for
+`ftw-cloud-deploy` with a digest.
 
 ```bash
 aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \
@@ -87,15 +111,33 @@ The DNS record must exist before the first TLS connection, since that is when
 
 ## Updating
 
+Every commit on `main` gets an image. Take its digest from the summary of the
+`Image` workflow's run, or from the `Digest:` line of:
+
+```bash
+docker buildx imagetools inspect ghcr.io/srcfl/ftw-cloud:<commit>
+```
+
+To check that the digest came from this repository's CI:
+
+```bash
+gh attestation verify oci://ghcr.io/srcfl/ftw-cloud@sha256:<digest> -R srcfl/ftw-cloud
+```
+
+Then run it:
+
 ```bash
 aws ssm send-command --region eu-north-1 --instance-ids i-0ecac4cbc6b5f8adc \
   --document-name AWS-RunShellScript \
-  --parameters 'commands=["/usr/local/sbin/ftw-cloud-deploy"]'
+  --parameters 'commands=["/usr/local/sbin/ftw-cloud-deploy sha256:<digest>"]'
 ```
 
 or, in a session (`aws ssm start-session --region eu-north-1 --target
-i-0ecac4cbc6b5f8adc`), `sudo ftw-cloud-deploy`. It pulls the checked-out branch,
-builds, and starts a new container. Every box reconnects within seconds.
+i-0ecac4cbc6b5f8adc`), `sudo ftw-cloud-deploy sha256:<digest>`. It pulls the
+image, refuses one whose commit is not on `main`, checks out that commit in
+`/opt/ftw-cloud`, fetches the key, starts a new container and deletes the old
+image. Every box reconnects within seconds. To go back, run it with the
+earlier digest.
 
 It always makes a new container, even when only a mounted file changed. The
 relay ran into this: a file mounted on its own, like the service key, keeps
@@ -105,10 +147,12 @@ the copy the container started with after the file changes on the host.
 
 ```bash
 curl -s https://cloud.ftw.energy/healthz
+curl -sI https://cloud.ftw.energy/source
 ```
 
-`ok` means DNS, the certificate and the service all work.
-`https://cloud.ftw.energy/source` returns the source archive.
+`ok` means DNS, the certificate and the service all work. `/source` returns
+the source archive, and its headers name the commit (`X-Source-Revision`) and
+the image digest (`X-Source-Image`), which should match what CI built.
 `/v1/app` and `/v1/now` answer 501 until C3 and C5.
 
 ## Cost
@@ -120,11 +164,13 @@ eu-north-1 on-demand prices, October 2026:
 | `t4g.small` | $0.0172 an hour | $12.56 |
 | 20 GB gp3 | $0.0836 a GB-month | $1.67 |
 | Public IPv4 (the Elastic IP) | $0.005 an hour | $3.65 |
+| Secrets Manager, one secret | $0.40 a month | $0.40 |
 | **Total** | | **about $18** |
 
 Traffic from boxes comes in, which AWS does not charge for. Traffic out to
 phones (C3) is free for the first 100 GB a month across the account. SSM costs
-nothing. CPU credits run unlimited, which costs extra only if the CPU averages
+nothing, and reading the secret at each boot and deploy costs next to
+nothing. The public image on `ghcr.io` is free. CPU credits run unlimited, which costs extra only if the CPU averages
 more than 20% over a day.
 
 ## Tearing it down
@@ -137,13 +183,17 @@ aws ec2 release-address --region eu-north-1 --allocation-id eipalloc-0085f3e68b6
 aws ec2 delete-security-group --region eu-north-1 --group-id sg-098992ad382ecf94a
 aws iam remove-role-from-instance-profile --instance-profile-name ftw-cloud-host --role-name ftw-cloud-host
 aws iam delete-instance-profile --instance-profile-name ftw-cloud-host
+aws iam delete-role-policy --role-name ftw-cloud-host --policy-name ftw-cloud-service-key
 aws iam detach-role-policy --role-name ftw-cloud-host \
   --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
 aws iam delete-role --role-name ftw-cloud-host
+aws secretsmanager delete-secret --region eu-north-1 --secret-id ftw-cloud/service-key
 ```
 
-The disk goes with the instance, and with it the registry and the service
-key. Nothing else holds FTW Cloud data.
+The disk goes with the instance, and with it the registry. Secrets Manager
+keeps the service key for 30 days after `delete-secret`, so a mistaken
+teardown can still be undone, and then deletes it. Nothing else holds FTW
+Cloud data; the images on `ghcr.io` hold only code.
 
 ## Not here yet