feat(cloud): follow FTW Cloud to the shard that holds the home, and spread the redial after a restart

.changeset/cloud-shard-redirect.md

@@ -0,0 +1,5 @@
+---
+"ftw": patch
+---
+
+The box follows FTW Cloud to the instance that holds its home. When the service closes the uplink with 4421, the box dials the host named in the close reason once, only if it is `cloud.ftw.energy` or one label under it, and keeps that host with its link so a restart goes there first. It goes back to `cloud.ftw.energy` after a second redirect in a row, or after three failed dials over at least 30 s, so a shard's restart does not send it away. When the service closes with 4503 before a deploy, the box redials at a random moment within the spread the service names, at least 1 s, instead of all boxes coming back within half a second; a 4503 before `hello_ok` counts as a failure and grows the backoff. The box lists `shard` in `hello.caps`. The service still proves itself with the TLS certificate and the service key the box was built with, so a redirect grants nothing.

docs/protocol-cloud.md

@@ -6,7 +6,9 @@ or the phone. The bytes live in `go/protocol/cloudwire`, with golden vectors
 in its `testdata`: `cloud-vectors.json` for the uplink,
 `cloud-app-vectors.json` for one whole app session, its
 [tunnel](#tunnel) messages and a pairing proof, and
-`cloud-once-vectors.json` for one [one-shot read](#one-shot-read).
+`cloud-once-vectors.json` for one [one-shot read](#one-shot-read), and
+`cloud-shard-vectors.json` for which hosts a client may follow and how it
+reads a restart's spread ([Shards and restarts](#shards-and-restarts)).
 `srcfl/ftw-cloud`, the phone and the watch replay them. `cloudwire.View` is the phone's
 [freshness](#freshness) rules in Go, and the cases listed there are its tests.
 
@@ -52,7 +54,9 @@ for both live in `go/protocol`, under Apache-2.0
 - No proxy may end TLS in front of the service, because that breaks the
   handshake binding. A plain TCP load balancer is fine.
 - Reconnects back off with full jitter from 0.5 s to 60 s. The attempt
-  counter resets after 60 s connected.
+  counter resets after 60 s connected. Two closes are not failures: 4421
+  names the instance that holds the home, and 4503 asks the box to come back
+  within a spread (see [Shards and restarts](#shards-and-restarts)).
 - The service sends a WebSocket ping every 15 s. The box's read deadline is
   45 s and its write deadline 10 s.
 - WebSocket compression is off.
@@ -700,6 +704,8 @@ nothing about which check failed.
 | 4409 | The roster changed what this device may read | Reconnects at once, once; then retries with backoff |
 | 4429 | The home has as many sessions as the service allows, as at the relay | Retries with backoff |
 | 1013 | The phone fell too far behind | Retries with backoff and gets a fresh snapshot |
+| 4421 | Another instance holds the home. The reason is its host | Follows once and keeps the host for the home (see [Shards and restarts](#shards-and-restarts)) |
+| 4503 | The service is about to stop. The reason is the spread in milliseconds | Redials at a random moment within the spread, without backoff |
 
 - **Backoff** is full jitter from 0.5 s to 60 s, as on the uplink. It also
   covers a lost socket.
@@ -904,7 +910,10 @@ through the relay instead, and prices and the plan wait for the phone.
 
 ## Extension token
 
-- 32 random bytes, base64url. The service stores its SHA-256, never the token.
+- The home id, a dot, and 32 random bytes in base64url. The service stores
+  the SHA-256 of the random bytes, never the token, and refuses a token whose
+  home is not the one its row is bound to. The home id lets any instance tell
+  which shard holds the token. To the extension the token is opaque.
 - Bound to one roster entry and valid for 7 days. The app renews it at each
   foreground and background refresh.
 - It dies when the roster drops the device, the device's epoch changes, an
@@ -1129,6 +1138,78 @@ app's, so the grant screen claims no more than an owner's session.
   After 90 days without a valid roster it deletes the home as in a purge.
 - **A new home** that sends no roster within 24 hours is deleted.
 
+## Shards and restarts
+
+Every home lives on exactly one instance of the service, its shard. Its box,
+phones, watch and extensions all meet there, so nothing about a home crosses
+between instances: no shared pub/sub and no shared storage, and readings stay
+off disk. `cloud.ftw.energy` is the default host. With one shard it is the
+only host and nothing below happens.
+
+**Wrong shard.** An instance that gets a connection or a request for a home
+it does not hold names the host that does:
+
+| Where | When the service knows the home | Answer |
+|---|---|---|
+| `/v1/box` | After `hello`, from the key that signed it, before the registry is touched | Close 4421, reason the host |
+| `/v1/app` | From `?home=`, before message 1 is read | Close 4421, reason the host |
+| `/v1/app/once` | From `?home=`, before the body is read | 421, the host in `Ftw-Shard`, empty body |
+| `/v1/now` | From the home id in the token | 421, the host in `Ftw-Shard`, empty body |
+
+A client follows a wrong-shard answer only to `cloud.ftw.energy` itself or
+one DNS label of lowercase letters, digits and hyphens directly under it,
+such as `s2.cloud.ftw.energy` (`cloudwire.ShardHostOK`). No port, userinfo,
+uppercase or trailing dot; `cloud-shard-vectors.json` holds the cases every
+client must decide alike. A phone that follows `Ftw-Shard` on `/v1/now`
+carries its bearer token there. The redirect carries
+no trust: the box still checks the TLS certificate for that host and
+`hello_ok` against the service keys it was built with, and the phone still
+runs Noise IK to the service Noise key it carries.
+
+- The client keeps the host for that home. The box stores it with its link,
+  so a restart dials the shard first.
+- It follows once. A second wrong-shard answer before it is connected again
+  sends it back to `cloud.ftw.energy` after backoff, where it may follow once
+  more.
+- After 3 dials in a row that fail at a kept host over at least 30 s, the
+  client forgets it and starts again at `cloud.ftw.energy`. The window
+  outlasts a shard's restart, so a restart does not send its clients to
+  the default host.
+- A box lists `shard` in `hello.caps` when it follows 4421 and spreads its
+  redial after 4503. A second shard holds homes only once their boxes list
+  it.
+
+**Directory.** Each instance holds the same directory: the shards' hosts, a
+shard for each of 256 buckets of homes, and single homes moved on their own.
+A home's bucket is the first byte of SHA-256 of `ftw.cloud.shard\0` and the
+home id. Moving homes copies their registry rows to the new shard, then
+updates the directory on every instance; the tool for it comes with the
+second shard. Their clients follow the redirect at their next connection,
+and extension tokens keep working, because the rows hold their hashes and
+each token names its home.
+
+Assistants' MCP and OAuth clients belong to other parties and follow neither
+4421 nor 421. `/mcp` and `/oauth/*` therefore stay on `cloud.ftw.energy`.
+Once there is a second shard, that instance forwards each call to the shard
+that holds the home; the client never sees another host.
+
+**Restart.** An instance that is about to stop, for a deploy or a move, stops
+accepting connections and closes every socket with 4503. The reason is a
+spread in milliseconds, in decimal, which the service sets from how many
+sockets it holds and how fast the next instance can take them. The client
+redials at a uniformly random moment within the spread, from 1 s to 60 s,
+and does not count the close as a failed attempt. A missing or unreadable
+reason means 5 s. A 4503 that comes before `hello_ok`, or before message 2
+on `/v1/app`, is a failed attempt: the backoff grows, so a service that
+sends every client away at once cannot hold them in a loop. The service
+sends no `box.unreachable` for boxes it closed this way.
+
+A command in flight when the service stops may be lost with the socket,
+before or after the box acted on it. A phone treats a command that got no
+answer as unknown, not failed: it shows the box's state once the session is
+back, and sends the command again only if the state says it was not
+applied.
+
 ## Service state
 
 Per home, the service keeps in memory the newest envelope of each class, the
@@ -1188,6 +1269,9 @@ constants, so `srcfl/ftw-cloud` need not spell them by hand.
 | What | Value |
 |---|---|
 | Reconnect backoff, box and phone | 0.5 s to 60 s, full jitter |
+| Redial after 4503 | Uniform within the reason's spread, 1 s to 60 s; 5 s without one; backoff when it comes before `hello_ok` or message 2 |
+| Wrong-shard redirects followed | One per connection; a kept host is forgotten after 3 failed dials over at least 30 s |
+| Shard buckets | 256 |
 | Phone retry after 4403 | Half to all of min(15 s × 2ⁿ, 5 min) |
 | Ping, read deadline, write deadline | 15 s, 45 s, 10 s, on both sockets |
 | Period per class | 5 s; 1 s while watched; `lp` at least 2 s |

go/internal/cloudlink/conn.go

@@ -8,6 +8,8 @@ import (
 	"errors"
 	"fmt"
 	"math/big"
+	"net"
+	"net/url"
 	"slices"
 	"sync"
 	"time"
@@ -45,25 +47,28 @@ func (l *Link) dialer() *websocket.Dialer {
 		HandshakeTimeout:  dialTimeout,
 		TLSClientConfig:   l.tlsConfig(),
 		EnableCompression: false,
+		NetDialContext:    l.opts.NetDial,
 	}
 }
 
 // once dials, serves until the connection ends and says how long to wait.
 func (l *Link) once(ctx context.Context, snap grantSnapshot) (time.Duration, error) {
 	dialCtx, cancel := context.WithTimeout(ctx, dialTimeout)
-	conn, resp, err := l.dialer().DialContext(dialCtx, l.opts.Endpoint, nil)
+	conn, resp, err := l.dialer().DialContext(dialCtx, l.endpoint(), nil)
 	cancel()
 	if resp != nil && resp.Body != nil {
 		_ = resp.Body.Close()
 	}
 	if err != nil {
+		l.hostFailed()
 		return l.backoff(), fmt.Errorf("dialling FTW Cloud: %w", err)
 	}
 	defer conn.Close()
 
 	connectedAt := time.Now()
 	err = l.serve(ctx, conn, snap)
 	l.mu.Lock()
+	up := l.connected
 	l.connected = false
 	if time.Since(connectedAt) >= backoffCap {
 		l.attempt = 0
@@ -72,9 +77,134 @@ func (l *Link) once(ctx context.Context, snap grantSnapshot) (time.Duration, err
 	if errors.Is(err, errLinkEnded) || errors.Is(err, errClosed) {
 		return 0, nil
 	}
+	// The service's two closes that are not failures: another instance
+	// holds the home, or this one is about to stop.
+	var closed *websocket.CloseError
+	if errors.As(err, &closed) {
+		switch closed.Code {
+		case cloudwire.CloseWrongShard:
+			if l.followShard(closed.Text) {
+				return 0, nil
+			}
+		case cloudwire.CloseRestart:
+			wait := spread(cloudwire.RestartSpreadMs(closed.Text))
+			if !up {
+				// A service that sends every box away before hello_ok is
+				// not restarting once; the backoff grows as for a failure.
+				return max(wait, l.backoff()), nil
+			}
+			l.mu.Lock()
+			l.attempt = 0
+			l.mu.Unlock()
+			return wait, nil
+		}
+	}
+	if !up {
+		l.hostFailed()
+	}
 	return l.backoff(), err
 }
 
+// spread is a uniformly random wait within ms milliseconds.
+func spread(ms int64) time.Duration {
+	if ms <= 0 {
+		return 0
+	}
+	n, err := rand.Int(rand.Reader, big.NewInt(ms*int64(time.Millisecond)))
+	if err != nil {
+		return time.Duration(ms) * time.Millisecond / 2
+	}
+	return time.Duration(n.Int64())
+}
+
+// endpoint is the uplink at the host the service last named for this home,
+// or at Endpoint's own host. A kept host that is not under Endpoint's, as
+// after a build moved the service, is not dialled.
+func (l *Link) endpoint() string {
+	l.mu.Lock()
+	host := ""
+	if l.link != nil {
+		host = l.link.Host
+	}
+	l.mu.Unlock()
+	u, err := url.Parse(l.opts.Endpoint)
+	if host == "" || err != nil || !cloudwire.ShardHostOK(u.Hostname(), host) {
+		return l.opts.Endpoint
+	}
+	if port := u.Port(); port != "" {
+		host = net.JoinHostPort(host, port)
+	}
+	u.Host = host
+	return u.String()
+}
+
+// followShard keeps the host a wrong-shard close named, once. A second
+// wrong-shard close before the box is connected again, or a host off the
+// service's domain, sends the box back to Endpoint's host after backoff,
+// where it may follow once more.
+func (l *Link) followShard(host string) bool {
+	base := ""
+	if u, err := url.Parse(l.opts.Endpoint); err == nil {
+		base = u.Hostname()
+	}
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.link == nil {
+		return false
+	}
+	if l.shardHops > 0 || !cloudwire.ShardHostOK(base, host) {
+		l.shardHops = 0
+		l.keepHostLocked("")
+		return false
+	}
+	l.shardHops = 1
+	l.keepHostLocked(host)
+	l.log.Info("FTW Cloud: another instance holds this home; following", "host", host)
+	return true
+}
+
+// hostFailed counts a dial or handshake that failed at a kept host. After
+// cloudwire.ShardHostFailures in a row over at least
+// cloudwire.ShardHostFailWindowMs the box forgets it, and Endpoint's host
+// names the right one again. The window outlasts a shard's restart, so a
+// box does not leave its shard for the default host because of one.
+func (l *Link) hostFailed() {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.link == nil || l.link.Host == "" {
+		return
+	}
+	now := l.now()
+	if l.hostFails == 0 {
+		l.hostFailedSince = now
+	}
+	l.hostFails++
+	window := time.Duration(cloudwire.ShardHostFailWindowMs) * time.Millisecond
+	if l.hostFails >= cloudwire.ShardHostFailures && now.Sub(l.hostFailedSince) >= window {
+		l.log.Info("FTW Cloud: the kept host stopped answering; starting again at the default host")
+		l.keepHostLocked("")
+	}
+}
+
+// keepHostLocked stores the host the uplink dials. Forgetting it starts a
+// new chain at Endpoint's host, where one redirect may be followed again.
+// The caller holds l.mu.
+func (l *Link) keepHostLocked(host string) {
+	l.hostFails = 0
+	if host == "" {
+		l.shardHops = 0
+	}
+	if l.link.Host == host {
+		return
+	}
+	rec := *l.link
+	rec.Host = host
+	if err := saveJSON(l.opts.KV, linkKey, rec); err != nil {
+		l.log.Warn("could not store the FTW Cloud host", "err", err)
+	}
+	l.link = &rec
+}
+
 // backoff is full jitter from 0.5 s to 60 s.
 func (l *Link) backoff() time.Duration {
 	l.mu.Lock()
@@ -172,6 +302,7 @@ func (l *Link) serve(ctx context.Context, conn *websocket.Conn, snap grantSnapsh
 	tunOut, tunBulk := make(chan []byte, tunQueue), make(chan []byte, tunBulkQueue)
 	l.mu.Lock()
 	l.connected = true
+	l.shardHops, l.hostFails = 0, 0
 	if l.opts.Tunnel != nil {
 		l.tunOut, l.tunBulk = tunOut, tunBulk
 	}
@@ -366,7 +497,7 @@ func (w *writer) send(snap grantSnapshot, t string, body any) error {
 // tunnel's own budget, and was counted. Noise transport survives a lost
 // message, so a session goes on after ErrTunDropped.
 var (
-	ErrNoUplink   = errors.New("cloudlink: no uplink to FTW Cloud")
+	ErrNoUplink         = errors.New("cloudlink: no uplink to FTW Cloud")
 	ErrTunDropped error = tunDropped{}
 )
 

go/internal/cloudlink/grant.go

@@ -38,6 +38,9 @@ type linkRecord struct {
 	// Ending means the box asked the service to delete the home and waits
 	// for the receipt. The key stays until it arrives.
 	Ending bool `json:"ending,omitempty"`
+	// Host is the instance of the service that holds this home, as a
+	// wrong-shard close named it. Empty is Endpoint's own host.
+	Host string `json:"host,omitempty"`
 }
 
 // KV is the part of the state store the grant needs.

go/internal/cloudlink/link.go

@@ -28,6 +28,7 @@ import (
 	"errors"
 	"fmt"
 	"log/slog"
+	"net"
 	"os"
 	"slices"
 	"strings"
@@ -169,6 +170,8 @@ type Options struct {
 	ServiceKeys []*ecdsa.PublicKey
 	RootCAs     *x509.CertPool
 	Tick        time.Duration
+	NetDial     func(ctx context.Context, network, addr string) (net.Conn, error)
+	Now         func() time.Time
 }
 
 // Link is the box's one uplink to FTW Cloud.
@@ -207,8 +210,14 @@ type Link struct {
 	lpSeq      uint64
 	resyncAt   time.Time
 	attempt    int
-	purgedAt   int64
-	rosterDue  bool
+	// shardHops counts wrong-shard closes since the box was last
+	// connected, and hostFails dials in a row that failed at the kept
+	// host (linkRecord.Host), since hostFailedSince.
+	shardHops       int
+	hostFails       int
+	hostFailedSince time.Time
+	purgedAt        int64
+	rosterDue       bool
 
 	// events wait for the writer; eventsDropped counts those that never
 	// left. closing is set by Bye, after which the link does not dial
@@ -256,6 +265,9 @@ func New(opts Options) (*Link, error) {
 	if opts.Started.IsZero() {
 		opts.Started = time.Now()
 	}
+	if opts.Now == nil {
+		opts.Now = time.Now
+	}
 	if opts.Tick <= 0 {
 		opts.Tick = 250 * time.Millisecond
 	}
@@ -966,11 +978,14 @@ func (l *Link) caps(snap grantSnapshot) []string {
 	if l.opts.Tunnel != nil {
 		out = append(out, cloudwire.CapTun)
 	}
-	return out
+	return append(out, cloudwire.CapShard)
 }
 
 // tlsConfig is TLS 1.3 only, with the system roots unless a test or a dev
 // build adds its own.
 func (l *Link) tlsConfig() *tls.Config {
 	return &tls.Config{MinVersion: tls.VersionTLS13, RootCAs: l.opts.RootCAs}
 }
+
+// now is the wall clock, or a test's.
+func (l *Link) now() time.Time { return l.opts.Now() }