Skip to content

feat(cloud): follow FTW Cloud to the shard that holds the home, and spread the redial after a restart - #1645

Merged
frahlg merged 6 commits into
masterfrom
feat/cloud-shard-redirect
Oct 15, 2026
Merged

frahlg merged 6 commits into
masterfrom
feat/cloud-shard-redirect

Conversation

@frahlg

@frahlg frahlg commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Paired with srcfl/ftw-cloud#17 (branch feat/scale), which pins go/protocol at this branch.

Why

FTW Cloud must be able to grow past one instance without a client release, and a deploy must not make every box and phone redial in the same half second. Today a box backs off 0 to 0.5 s after any close, so a restart is a reconnect storm, and there is no way to tell a box that another instance holds its home.

What

Wire (go/protocol/cloudwire/shard.go, docs/protocol-cloud.md "Shards and restarts")

  • CloseWrongShard 4421 on /v1/box and /v1/app. The close reason is the host that holds the home.
  • StatusWrongShard 421 with the host in Ftw-Shard, for /v1/app/once and /v1/now.
  • CloseRestart 4503. The reason is a spread in milliseconds. The client redials at a random moment within it, and the close does not count as a failure.
  • ShardHostOK: a client follows only to cloud.ftw.energy or one label under it.
  • CapShard: a box that follows lists shard in hello.caps.
  • The extension token now carries the home id (<home>.<secret>). No app reads /v1/now yet, so nothing breaks.

Core (go/internal/cloudlink)

  • The box follows one 4421 and stores the host in its link record, so a restart dials the shard first.
  • A second 4421 in a row, or three failed dials at the kept host, sends it back to cloud.ftw.energy after backoff.
  • After 4503 the box redials within the spread and resets its attempt counter.
  • The redirect grants nothing: the box still checks the TLS certificate for the new host and hello_ok against the service keys it was built with.

Tests

  • go test -race ./internal/cloudlink/ ./internal/appuplink/, and go test ./cloudwire/ in go/protocol.
  • New tests cover each case against two fake instances on one certificate:
    • the redirect is followed once and kept across a restart;
    • an off-domain host is refused;
    • two instances naming each other do not loop;
    • a kept host that stops answering is forgotten;
    • five 4503s in a row do not grow the backoff.

Review

A review (Sonnet) of both PRs found two things on this side, fixed in 4969254:

  • A stale hop count. After the box forgot a dead host, the default host's next redirect counted as a second hop, which cost one more backoff of up to 8 s. Forgetting a host now starts a new chain. TestAfterForgettingAHostTheNextRedirectIsFollowed fails without the fix.
  • A kept host from an earlier build. A host saved under an earlier build's endpoint was still dialled. endpoint() now uses a kept host only if it is the endpoint's own host or one label under it.

Owner's review

Fixed in 26426a6 and 7f40a0e:

  • The spread has a 1 s floor. RestartSpreadMs returns 1 s to 60 s, so a reason of 0 no longer makes a tight loop.
  • A 4503 before hello_ok counts as a failed attempt. The backoff grows instead of resetting (TestARestartBeforeHelloOKKeepsTheBackoff).
  • A box keeps its host through a shard restart. It forgets the host only after 3 failed dials over at least 30 s (ShardHostFailWindowMs). Before, three failures took about 3.5 s, so a 4 s restart sent boxes to the default host (TestAShardRestartDoesNotMakeTheBoxForgetItsHost).
  • testdata/cloud-shard-vectors.json is shared by every client. Swift, Kotlin and TypeScript replay it: which hosts to follow (no port, userinfo, uppercase, trailing dot or missing dot; 29 cases) and how to read a spread (12 cases).
  • Docs:
    • a command in flight at a drain is unknown to the phone, not failed;
    • /mcp and /oauth stay on cloud.ftw.energy, which forwards to the home's shard once there is a second shard;
    • moving homes needs a tool that comes with the second shard.

Not here

  • The iOS, Android and web clients do not follow 4421 or honour 4503 yet. iOS reads the close code and reason, but its host is a constant, and an unknown code gets backoff from 0 to 0.5 s.
  • With one shard, none of this triggers. A second shard holds homes only after their boxes list shard and the apps follow 4421.

🤖 Generated with Claude Code

…d spread the redial after a restart

The wire for sharding FTW Cloud and for deploys without a reconnect storm:

- cloudwire: CloseWrongShard 4421 (reason: the host), CloseRestart 4503
  (reason: spread in ms), StatusWrongShard 421 with Ftw-Shard for the
  HTTP routes, CapShard, ShardHostOK (the host must be cloud.ftw.energy or
  one label under it), RestartSpreadMs.
- cloudlink: the box follows 4421 once, keeps the host in its link record,
  forgets it after a second redirect in a row or three failed dials, and
  redials within the spread after 4503 without growing its backoff.
- docs/protocol-cloud.md: "Shards and restarts", the close codes, the
  extension token's home prefix and the numbers.

Paired with srcfl/ftw-cloud (feat/scale).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg and others added 2 commits October 15, 2026 19:59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…host must sit under the endpoint

- Forgetting the kept host resets the hop count, so the default host's
  next redirect is followed at once instead of costing another backoff
  (up to 8 s). The test fails without the fix.
- endpoint() dials a kept host only if it is the endpoint's own host or one
  label under it, so a host kept under an earlier build's endpoint is not
  dialled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frahlg
frahlg marked this pull request as ready for review October 15, 2026 18:20
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T18:23:50.144420Z 4969254 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

frahlg and others added 3 commits October 15, 2026 22:35
… box forgets its shard, and shard vectors for every client

From the owner's review of #1645:
- RestartSpreadMs is 1 s to 60 s; a reason of 0 no longer gives a tight
  loop.
- A 4503 before hello_ok is a failed attempt: the backoff grows instead of
  resetting, so a service that turns every box away cannot hold them in a
  loop.
- A kept host is forgotten after 3 failed dials over at least 30 s
  (ShardHostFailWindowMs), which outlasts a shard's restart; three failures
  took about 3.5 s before.
- testdata/cloud-shard-vectors.json: which hosts a client may follow (no
  port, userinfo, uppercase, trailing dot or missing dot) and how it reads
  a spread, for Swift, Kotlin and TypeScript to replay.
- docs: a command in flight at a drain is unknown, not failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on the default host

- A kept host is forgotten after 3 failed dials over at least 30 s; the
  spread is 1 s to 60 s; a 4503 before hello_ok or message 2 grows the
  backoff; cloud-shard-vectors.json is named with the other vectors.
- A command in flight at a drain is unknown to the phone, not failed.
- /mcp and /oauth stay on cloud.ftw.energy, which forwards to the home's
  shard once there is a second one: MCP and OAuth clients follow no 4421
  or 421.
- Moving homes needs a tool that comes with the second shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant